Design a Scene Similarity System

An engineer working on the driver hits a rare failure. Say the planner hesitates behind a cyclist walking a bike across a crosswalk. The engineer wants every other time the fleet saw something like it. The match is the same situation, at any location and any timestamp. That request is the whole problem. You are handed petabytes of recorded driving and asked to return the scenes that resemble a given one. It has to be fast enough that the engineer keeps working instead of filing a batch job and coming back tomorrow.

Say plainly why the obvious approaches fail, because ruling them out is what points at the real design. You cannot grep it: the interesting signal lives in camera pixels, lidar returns, and agent trajectories, not in text tags someone remembered to write. You cannot scan it: the corpus is far too large to compare a query against every scene per request. And you cannot hand-label your way out, because there is no finite list of scene types to bucket into, and the situations that matter most are the rare ones nobody thought to name. So the design has two halves that reinforce each other. A model that maps a scene to a vector where near means similar, and an index that finds the near vectors without touching the rest of the corpus.

The Waymo angle

Here, similar is something a model has to learn. No database predicate captures it. Waymo improves the driver by finding more examples of the situations it handles badly. Scene retrieval is part of how the fleet gets safer, more than a nice-to-have internal tool. That sets the bar. The system has to capture similarity a human would agree with. A pedestrian stepping off a curb at dusk is like another pedestrian stepping off a curb at dusk, even when the street, the car, and the weather differ. And it has to do that over a corpus measured in millions of miles. Get the learned metric right and the index is standard machinery.