Design the Autopilot Data Engine

Understanding the problem

Start with the one fact that shapes everything else. This system is a loop. More cars on the road surface more rare situations, and those become training data. The data makes the next model better, and the better model drives more of the fleet. Driving more of the fleet surfaces the next tier of rare situations. Each turn compounds. That is the flywheel, and the system exists to make it spin faster and cheaper. The hard part is the fuel: real driving footage arrives as an unshippable firehose of video, and almost none of it is worth keeping. A car driving straight down an empty highway teaches the model nothing. The signal lives in the rare frames, and finding them is the game.

The Tesla angle

Tesla's edge in self-driving is not one clever network, it is the machine that keeps improving the network, and this design is that machine. In January 2026 Tesla reframed its mission around AI, autonomy, and robotics. The data engine is where that bet is won or lost. A competitor can copy an architecture, but not a fleet of five million cars mining the long tail every day. Tesla has described this loop in public talks, so the shape is well attested even where exact cadences are not. Treat the numbers below as illustrative teaching figures rather than audited internals. The instinct is the real signal. The interviewer wants to see you reason about which few seconds are worth the bandwidth and how a regressed model is kept away from the fleet, not hear you name a network. If you find yourself designing a generic data lake, you have missed that the fleet is a sensor you can query and the release gate is a safety system.

Functional requirements

Keep the core to the loop itself, and leave the widening to the interviewer. Six stages carry the design, and they form a ring rather than a line. The car detects and captures a hard moment. It uploads just that clip. The clip gets labeled. Labeled clips are curated into a dataset. A candidate model trains on that dataset. The candidate is validated and rolled out, which changes what the fleet does next and feeds the ring again. Two stages deserve naming on purpose: capture is where the bandwidth budget is spent, and rollout is where safety is on the line.

Non-functional requirements

Trace each target back to what the loop is for. Bandwidth is the scarce resource, so the system is judged first on how little it uploads to learn a given thing, not on how much it collects. Label quality and dataset balance outrank raw volume, because a million redundant sunny-highway clips move the model less than a thousand well-labeled hard ones. Then there is the constraint that does not bend: safety. A regressed model reaching millions of cars is a recall-shaped event, so the release gate and the rollback path are first-class, non-negotiable parts of the design. Privacy sits next to safety: the car uploads only what the owner has permitted. And freshness matters more than it does for a warehouse. The loop should close in days, so a failure mode found on Monday can be fixed and validated by Friday, not next quarter.

• ~5M+ cars in the fleet. The distributed sensor the loop runs on. Illustrative and approximate.
• ~8 cameras per car. Plus the IMU and GPS traces. Raw multi-camera video is the thing you cannot ship.
• ~PB/day raw footage if you uploaded it all. Far past what any cellular network could carry. An order-of-magnitude estimate.
• ~10 s length of a triggered clip. Tens of MB with all cameras. The unit the fleet uploads.

Sizing the firehose

Put the bandwidth numbers down, because the entire design bends around them. Take each figure as an order-of-magnitude sketch, sized to teach rather than to audit. Start from raw video. Five million cars, each driving on the order of an hour a day, each with eight cameras, produces multi-camera footage measured in tens of gigabytes per hour per car even after on-car compression. Multiply it out and the fleet generates on the order of hundreds of petabytes of raw footage a day. Uploading that over cellular is physically impossible. It would be a waste even if it were free, because the overwhelming majority is a car driving straight down an empty road. Now price the alternative. A triggered clip is about ten seconds across the cameras plus the sensor traces, tens of megabytes, and a car fires only a handful worth keeping per day. That is a reduction of three or four orders of magnitude, and it is the difference between an impossible system and a working one.

• Raw footage: ~hundreds of PB/day. 5M cars times multi-camera video. The number that kills "upload everything" in one line.
• Triggered clip: ~tens of MB. a ~10 s multi-camera window plus the sensor traces. The unit that fits the budget.
• Upload budget: a few clips/car/day. bounded on purpose, so a mistuned trigger cannot swamp the network or the label queue.
• Loop latency: days, not months. how fast a newly found failure mode becomes a shipped fix. The flywheel's real metric.

One relationship in that grid decides the shape of the design. Store everything and you drown in petabytes of redundant highway; store nothing and the loop has no fuel. So the job is to spend a tight upload budget only on the frames that carry new information, then convert those frames into labeled data fast enough that the loop closes in days. Everything that follows is in service of those two moves: choosing what to upload, and turning it into a model you can safely ship. Choosing what to upload is the first deep dive, and it is the one the interviewer is waiting to hear.

The set-up

Core entities

Fix the vocabulary before drawing anything. Five objects carry the design, and the one to nail down first is the model version, because it is what makes the loop safe to run at all. Every deployed model is an immutable, numbered snapshot bound to the exact dataset and config that produced it. Any model on the road can be traced back to its training data and rolled back to the version before it. That single property is what turns "ship a new model to millions of cars" from a gamble into a reversible operation.

Pin the stores next, since the interviewer will want each one named. Clips are large immutable blobs read in bulk by training jobs, so they live in an object store, an S3-style system internally, keyed by clip id. Labels and the derived features live in a label and feature store that can be queried and joined to clips. Datasets are versioned manifests over that store, a data-lake pattern where a dataset is a pinned list of clip and label ids rather than a copy. Models live in a registry that keeps the binary, the lineage, and the eval scores together. Four roles, four stores, and keeping them separate is what lets each one be good at its own job: cheap bulk blobs, queryable labels, reproducible datasets, and traceable models.

API

Keep the API narrow and built around the two moments that matter: a car uploading a triggered clip, and an operator shipping a model. The clip upload is a server-accepted push from the car rather than a request it waits on, because the car fires and moves on. The model calls are operator or pipeline actions that gate what reaches the fleet. Point back to these by name in the walkthrough and the dives.

Two calls carry the design's spine. POST /clips is the narrow throat every uploaded clip passes through. The budget check and the consent check belong there, so a runaway trigger or a revoked opt-in is stopped at the door. POST /fleet/triggers looks minor but it is the "queryable sensor" idea made concrete: an engineer who needs more examples of a rare sign publishes a trigger and the fleet starts answering. And the pairing of rollout with rollback is deliberate; a rollout may only start behind the regression gate, and a rollback may always fire. Ship a good model slowly and pull a bad one instantly: that asymmetry is the safety story in miniature.

High-level design

Follow the clip around the ring, one stage at a time. Capture and trigger happen on the car. Ingest lands the clip and drops an event. Auto-labeling turns the clip into labels, with humans on the hard cases. Curation assembles a balanced, versioned dataset. Training produces a candidate. Evaluation gates it, and deployment ships it in shadow first and then over the air, which changes what the fleet captures next and closes the ring. Each box in the diagram carries a role color: blue is the Fleet vehicles, violet the pipeline services, amber the Clip event log, green the four stores. The walkthrough tags each component the same way.

1) Trigger and capture on the car. Fleet vehicles run the shipping model to drive, and alongside it hundreds of cheap trigger classifiers plus a shadow model predicting silently. When a trigger fires, a rare object, a hard brake, a disengagement, or the shadow model disagreeing with what the driver did, the car pulls the buffered ten-second window and queues it for upload. Nothing has left the car yet except a decision that this moment is interesting. The bottleneck being deliberately handled here is bandwidth, and how the car decides is the first deep dive.

2) Upload and ingest. The car pushes the clip to Clip ingest via POST /clips. Ingest authenticates the car, checks it is under its daily budget and that the owner permits this data, writes the blob to Clip store, and drops a record on the Clip event log. The upload is at-most-once from the car's point of view and cheap to retry: if the link drops, the car re-queues and sends later. What is punted here is what makes a clip worth accepting in the first place, which the trigger deep dive owns.

3) Label the clip. The Auto-labeler consumes the event, reads the clip from Clip store, and runs heavy offline models to produce labels, which it writes to Label store. Because it runs offline, it can use the entire clip and even later frames, so it labels more accurately than the car could in real time. Clips where the auto-labeler is unsure, or where labels disagree, are routed to human reviewers. How to split auto from human is the second deep dive.

4) Curate a dataset. The Dataset curator reads labeled clips and drops near-duplicates. It balances the mix across conditions and scenarios so the model does not drown in easy highway miles, and pins the selection into a versioned manifest in Dataset store. A dataset is a list of ids, not a copy, so it is cheap to define many and reproduce any. The punt here is how curation and training interact, which the training deep dive takes up.

5) Train a candidate. The Training cluster trains a new network on a pinned dataset and registers the result as an immutable version in Model registry, stamped with its dataset manifest and config. It is a candidate, held short of release. Being a version with full lineage is what will make the next two steps, validation and rollback, possible.

6) Validate, then deploy. The Eval harness scores the candidate against a frozen regression suite and the newly mined cases; only if it holds the line does it hand the version to the Deploy controller. The controller runs it in shadow mode on real driving, compares it to the shipping model, and only then stages an over-the-air rollout, cohort by cohort, ready to roll back. The rolled-out model drives the fleet slightly better, which changes which moments trigger next, and the ring turns. The two threads left dangling, how the gate works and how shadow-then-ship stays safe, are the last two deep dives.

Technology choices

None of these boxes is a system until you name one. Set a recognizable tool in each slot, hold the line on the category, and know how it strains under this loop's particular skew, because the follow-ups all surface there.

• Clip store is an object store (S3 or an internal equivalent): immutable clip blobs keyed by clip id, written once and read in bulk. This category is right because clips are large, never mutated, and streamed by training jobs rather than queried field by field, which is exactly what object storage is cheap at. The failure follow-up is cost and lifecycle: petabytes accumulate, so clips age into colder tiers and clips whose value has been extracted can expire, while the labels that summarize them stay hot.
• Clip event log plus Clip ingest is the ingest spine: a partitioned log, Kafka, in front of a stateless ingest tier, partitioned by VIN. A log is right because uploads are bursty and labeling should not backpressure onto the car; clips wait durably while consumers drain at their own rate. The failure story is a trigger pushed fleet-wide that suddenly fires everywhere at once: the per-car budget at ingest and a cohort scope on the trigger keep the burst bounded instead of unbounded.
• Label store and Dataset store are the feature and label layer plus dataset versioning: a queryable label store joined to clips, with datasets expressed as pinned manifests over it (a data-lake pattern). Manifests rather than copies are the right trade because a dataset is defined far more often than the raw data changes, so pinning ids keeps runs reproducible without duplicating petabytes. The failure mode is label drift, handled by tracking each label's producer and version so a bad auto-labeler release can be found and re-run.
• Training cluster is a distributed GPU cluster, Tesla's Dojo as the concrete example, running data-parallel training on pinned datasets. A dedicated cluster is right because training a large vision network on curated fleet data is throughput-bound and runs continuously, so utilization and data-loading bandwidth are the design constraints. The failure follow-up is a run that silently underperforms: every run is tied to its dataset manifest and config, so a regression is reproducible rather than a mystery.
• Eval harness and Deploy controller plus Model registry are the release path: a scoring service over a frozen suite, a staged OTA controller, and an immutable versioned artifact store. Immutability is right because safety needs a model on the road to be traceable and reversible, so every version carries its lineage and eval scores and a rollback is a pointer flip. The subtlety to name is that this gate is an ML gate: it compares per-scenario metrics against the shipping model, not a generic health check, which is the next-to-last deep dive.

Interviewers commonly probe one contrast. The clip store and the label store both describe the same driving moments, so why not one system? They are different animals. The clip store holds heavy, write-once pixels read in bulk and almost never queried by attribute. The label store holds small, structured, queryable annotations that curation slices and joins constantly, and that get rewritten as a better auto-labeler reprocesses old clips. Collapse them and you either force bulk pixel storage to answer "give me every clip with a cyclist at dusk," which it is terrible at. Or you put constantly-updated labels in a write-once blob store, which loses them. Name the brand so the interviewer knows you have run one, then defend the category, because the category is what the follow-up tests.

Potential deep dives

The remaining points are won here. An ML-systems loop almost always pushes on the same four seams: what you upload, how you label it, how you train and gate, and how you ship without hurting anyone. These are the seams the problem tests.

1) You cannot upload the video, so what do you upload?

This is the decision the design rests on, and the one the interviewer is waiting to hear you reason through. The fleet generates hundreds of petabytes of raw footage a day and a tight upload budget of a few clips per car. The question is which few. Two naive strategies fail in opposite directions, and the strong answer is a third thing that treats the fleet as something you can query.

2) Labeling at fleet scale: auto-label, human in the loop

A clip is worthless until it has ground truth attached, and labeling is where the loop most easily chokes. The clips you collected are, by construction, the hard ones, which is precisely where labels are most expensive to get right and most dangerous to get wrong. The instinct to reach for is a split by difficulty, with machines carrying the volume and humans reserved for where they change the answer.

3) From clips to a model: curation, distributed training, and the regression gate

With labeled clips flowing, the temptation is to feed them straight into training. Resist it. Two bookends around training separate a working loop from a churn of wasted GPU hours. A curation step decides what the model sees. A gate decides whether the result is allowed anywhere near the fleet. The training itself is the least interesting part to an interviewer; the discipline around it is the signal.

4) Shipping safely: shadow mode, versioning, and rollback

A model that passes the gate still is not safe to trust with the wheel, because the regression suite is a sample and the real world is not. Here the design has to earn the word "safe." It mirrors the fleet OTA instinct: a bad push to millions of cars is a recall, far beyond a rollback of a web deploy. The staff-level move is to validate on real driving before granting any control, and to make every step reversible.

Step back and the four deep dives are one machine. Triggers and shadow mode choose the few seconds worth uploading. Auto-labeling with humans on the hard cases turns those seconds into ground truth. Curation and the regression gate turn ground truth into a model that is provably not worse. Shadow-then-ship deployment puts that model on the road without betting lives on it, and its disagreements become the next triggers. Each pass makes the fleet drive a little better, which changes what is rare, which sharpens what gets collected. That is the flywheel. It is a moat because it compounds with fleet size: more cars means more of the long tail per day, and no amount of cleverness substitutes for having driven the roads.

What is expected at each level

The surface is wide, and the level you interview at decides how much of it you must land. Tesla's system design round scales hard with seniority, and reports point to Autopilot and AI roles pushing the ML-systems depth harder than the fleet-infrastructure designs. Treat the rungs as directional: a working loop at mid, the two hard decisions owned at senior, and the closed-loop and safety story driven above that.

Mid-level engineer

Breadth is the priority, roughly 80/20. Get one coherent ring standing: cars capture events, upload clips, clips get labeled, a dataset trains a model, the model ships back, and the loop repeats. The load-bearing insight to arrive at unprompted is that you cannot upload all the video, so the car has to decide what is worth sending. The trigger-versus-shadow-mode nuance and the regression gate can stay lightly sketched here. The giveaway is when the interviewer's "how does the car know what to upload" pushes you to on-car detection instead of streaming everything. The bar: a flywheel you can defend end to end, with the interviewer softening the ML edges.

Senior engineer

Depth steps up, roughly 60/40. This is where you have to stand alone. Get to trigger classifiers plus shadow mode for selective upload on your own, and frame the fleet as a queryable sensor rather than a data dump. Land the labeling split, auto-label the bulk and route hard cases to humans, and explain why the hard cases are exactly where auto-labels are least trustworthy. Present the regression suite as a gate rather than a report, and argue why per-scenario comparison beats aggregate accuracy. Raise the bandwidth constraint before being asked and treat it as the axis the design turns on. The bar: a near-complete loop where you found the upload problem and the labeling trap yourself and had concrete, defensible answers.

Staff and beyond

Depth runs across the full ring now, roughly 40/60. Own the closed loop as a system. Shadow mode is both the final validator and a trigger source. Dataset versioning and lineage let any model trace to its data. The regression suite grows with every fleet-found failure. Own the safety story, shadow before canary before wide, immutable versions, instant rollback, and defend blocking a candidate that is better on average but regresses one safety-critical scenario. Carry the operational metrics as distinct levers: loop latency, trigger precision, label-quality drift, and the per-car upload budget. You run it; the interviewer corrects rather than saves the round. The bar: you hold selective capture, the labeling economics, the regression gate, and the reversible deploy as separate, defensible calls, and you keep bandwidth, label quality, and safety as independent constraints across every refinement.