Design Visual Search

Someone is looking at a photo of a living room they like. They don't have words for the lamp in the corner, they don't know the brand, and a text search box is useless to them because they can't describe what they want. So they draw a box around the lamp instead, and the product's job is to return Pins of lamps that look like that one. That is visual search, and it is the system that no other company's interview prepares you for. Pinterest runs more than 600 million visual searches a month across three surfaces. Flashlight searches from a region you crop inside an existing Pin. Lens searches from a live camera photo of something in the real world. Shop-the-Look finds buyable products inside a scene.

Most people hear "image search" and picture Google Images, so they reach for an inverted index of tags and captions. Cut that instinct. The query here is pixels, so there is no text to look up. The spine is different. Turn the query image into a vector that encodes what it looks like. Turn every Pin image in the corpus into the same kind of vector ahead of time. Then find the corpus vectors nearest the query's. Image in, embedding out, nearest-neighbor search over billions of embeddings, then a re-rank. Lead with that and everything else, the camera, the crop, the shoppable products, is a variation on the same pipeline.

The Pinterest angle

Say the words "search by embedding, not by keyword" in the first minute and you have framed the round correctly. A candidate who starts indexing alt-text and board titles has quietly turned visual search back into text search, and an interviewer hears that they don't believe the query is an image. This design is hard, and Pinterest's least-copyable system, because similarity lives in pixel space. Two lamps that share no words and no metadata still need to come back together because they look alike. Commit to the embedding-and-ANN spine before you draw a second box, and the metadata becomes what it should be, a re-rank signal instead of the index.