Design a Telemetry System

Understanding the problem

Set the register first, because it decides how you answer. This design isn't about serving a billion users or sizing a cache tier. The vehicle flies once and the flight lasts minutes. What matters is data that arrives correct, ordered, complete enough to trust, and fast enough to act on. The functional surface is small: sample, downlink, decode, store, watch, command. The depth is all reliability: never silently dropping a frame that matters, trusting a flight-critical decision when one computer might be wrong, and keeping sensor-to-console latency bounded.

One honest caveat so the framing is right. A SpaceX system-design round is usually candidate-led and light on the prompt. The interviewer hands you a scenario in a sentence or two and expects you to drive it, ask the questions, and set the scope. There is no single canonical telemetry prompt. The design below is representative, built from how SpaceX flies software, and it's here for the reasoning it exercises. Don't expect this exact system to be your question.

Picture the shape before any code. Many vehicle subsystems sample sensors and hand readings to a set of flight computers that run in triple redundancy. A judge takes the majority view so a single faulty computer cannot steer the picture. The agreed state is framed and sent over an RF downlink to a ground station, where an ingest and decode stage turns raw frames into timestamped samples. Those samples fan two ways: a live console stream that operators read now, and a time-series store that feeds anomaly alerting and, later, a post-flight archive. Commands travel the same link back up. One idea runs through all of it: trust nothing that has not been made redundant or checked.

The SpaceX angle

SpaceX flies software the way it builds hardware, so the design conversation grades reliability judgment over diagram breadth. The flight stack is C++ on a customized real-time Linux, and a failing flight computer gets redundancy instead of a retry. SpaceX runs a pattern it calls Actor-Judge: three computers propose a result and a judge takes the majority, so a single fault is outvoted. The same instinct shows up in testing, on hardware-in-the-loop rigs where a flight computer is killed at random mid-simulation to force the failover. A strong telemetry answer treats a lost frame and a lying sensor as cases you design for, and it reaches for redundancy and voting before scale. Say out loud where losing data is unacceptable and where a best-effort sample is fine.

Functional requirements

Keep the core to the flow and let the interviewer widen it. The system has to ingest high-rate telemetry from many subsystems, each producing its own channels of measurements. It has to move that data from the vehicle to the ground over a radio link that is lossy and bandwidth-limited. On the ground it has to decode each frame back into structured samples and stamp each with a trustworthy time, so a later reconstruction knows exactly when a thing happened. It has to store those samples twice over: streamed to live mission consoles during flight, and written to a durable store for post-flight analysis. It has to watch the stream for anomalies and raise alerts fast enough to matter. And it has to carry commands from the ground back up to the vehicle over the same link.

Non-functional requirements

Reason through each target, because these draw the design. No silent data loss comes first and outranks the rest. Telemetry is the flight team's only window into a vehicle they can't touch, and it's the evidence a post-flight investigation runs on. A frame that goes missing without anyone noticing is the worst outcome there is. A radio link will drop packets, so the bar is a system that is lossless in effect for the data that matters, or knows exactly which samples it's missing. A gap the team can see is recoverable; a gap nobody sees turns every downstream conclusion into a guess.

Bounded latency is the second target, and as in any real-time system, the tail matters more than the average. An operator watching a tank pressure needs the console to reflect reality within a tight, predictable delay. A value that's usually fresh but sometimes seconds stale is worse than one that's always a beat behind, because the team stops trusting it. So the live path holds a bounded delay from sensor to screen. Anything that could pause without bound, a blocking write, an allocation, a slow disk, goes onto the durable side, where a little more latency is fine.

Redundancy and determinism are the third target, and they explain why the design leans the way it does. Flight-critical state can't rest on one computer being correct, so the vehicle runs several and votes, and a disagreeing computer is flagged. Parts of the path that must react within a hard deadline need a deterministic schedule. That's why flight software runs on a real-time kernel and avoids anything with unbounded timing. Scope it honestly, too: not every byte needs the same guarantee. Flight-critical channels get redundancy and lossless handling, high-rate diagnostic channels can be best-effort, and saying which is which is half the design.

• No frame silently lost: lossless or known-gapped. The data that matters is either delivered or the system knows precisely which samples are missing and shows the gap. A visible gap is recoverable; an invisible one turns every later conclusion into a guess. Sequence numbers on every frame make loss detectable.
• Latency is a distribution: tail, not average. An operator trusts a console that is consistently a beat behind over one that is usually instant and occasionally seconds stale. The live path holds a bounded, predictable delay; anything with unbounded timing is pushed onto the durable side.
• Trust nothing un-voted: triple redundancy, majority wins. Flight-critical state is proposed by several computers and a judge takes the majority, so a single fault is outvoted rather than believed. The disagreeing computer is flagged and never quietly followed. A failing part is answered with redundancy.
• Not every byte is equal: guarantee by channel. Flight-critical channels get redundancy and lossless handling; high-rate diagnostic channels can be best-effort. Deciding which measurements need which guarantee, and saying so, is half the design and the honest part of scoping it.

One framing choice steers everything downstream, so state it out loud. There are two hard problems here, and they deserve separate reasoning. The delivery problem is getting data to the ground complete and in order, over a lossy link and under a firehose of samples. It lives in framing, sequence numbers, at-least-once with dedup, and backpressure. The correctness problem is trusting a flight-critical value when a computer or sensor might be wrong, and it lives almost entirely in redundancy and the vote. Sketch the pipeline, then spend the design on those two.

The pipeline, stage by stage

Lay the pipeline out from the sensor to the console and back, because that shape is the design. Read the diagram by role. Blue is an endpoint, the vehicle subsystems at one end and the mission operators at the other. Violet is a component the system runs. Green is state it keeps. Amber is the branch that leaves the happy path, the moment a frame arrives stale or a sequence number is missing and the system flags a gap instead of guessing.

Name the stages and what each owns, because the interviewer grades the boundaries as much as the code. The subsystems produce the firehose. The three flight computers and the judge turn several possibly-wrong opinions into one trusted value before anything is sent. The downlink frames and sequences the data so loss on the radio link is detectable. Ingest decodes, timestamps, and dedups, and raises a gap the moment a sequence is missing. From there the sample fans to a live stream built for bounded latency and to a durable store built for completeness. Alerting watches the durable side, and the uplink carries acknowledged commands back. Everything on the fast path is one of these, and everything that can tolerate more latency, the archive, the heavy analysis, hangs off to the side.

Redundancy and the Actor-Judge vote

This section separates a reliable design from a hopeful one, so slow down. A single flight computer is a single point of failure, and the failures are real. A cosmic ray flips a bit in memory, a sensor reads garbage, a software fault produces a wrong answer that looks plausible. You can't power-cycle a vehicle in flight and try again, so run several computers and cross-check them instead of trying to make one perfect. SpaceX flies a pattern it calls Actor-Judge: several computers act, each proposing a result, and a judge compares the proposals and takes the majority. A value two of three agree on is accepted; the odd one out is marked suspect and can be isolated.

Three works and two doesn't for a simple reason. Two computers can detect a disagreement, but they can't resolve it, because nothing tells you which one is right. Three give you a majority. A single fault is outvoted, so the system keeps flying on the correct value instead of halting because it noticed a problem. That's the difference between fault-detecting and fault-tolerant, and for a vehicle in flight it decides everything. The vote is cheap, a comparison across three proposals, and it runs on every flight-critical value on a deterministic schedule.

// Actor-Judge: three flight computers propose a value, the judge takes
// the majority. A single faulty computer is outvoted and flagged, not
// believed. Runs on every flight-critical channel, on a real-time schedule.
struct Proposal {
    ComputerId computer;
    Value      value;      // the flight-critical value this computer computed
    bool       healthy;    // self-reported: did this computer complete cleanly
};
// Returns the trusted value, or asks for a safe state when no majority exists.
VoteResult judge(const std::array<Proposal, 3>& proposals) {
    // Count agreement among the computers that reported healthy this cycle.
    for (const auto& a : proposals) {
        if (!a.healthy) continue;
        int agree = 0;
        for (const auto& b : proposals)
            if (b.healthy && values_match(a.value, b.value)) ++agree;
        if (agree >= 2) {                 // two-against-one or unanimous
            flag_dissenters(proposals, a.value);   // isolate the odd computer
            return VoteResult::accept(a.value);
        }
    }
    // No majority: do NOT pick one and hope. Command a safe state and alarm.
    return VoteResult::safe_state();
}

Two details matter here. First, the judge counts only computers that reported healthy this cycle. A computer that knows it faulted should recuse itself, so you never have to override its vote later. Second, the all-disagree case is a real branch. With no majority, the system commands a known safe state and raises an alarm instead of guessing, because a deliberate safe state is recoverable and a confident wrong answer is not. Expect the interviewer to probe that fallback and what the vehicle does when redundancy runs out.

The SpaceX angle

Raise failure before the interviewer does. Say that a single computer can be wrong in a way its output won't reveal, and that three computers with a majority vote make that fault detectable and survivable. Then go straight to the ugly cases: what the judge does when all three disagree, how the computers stay in lockstep, and how you'd prove the failover works. That last one has an honest answer. You prove it by injecting faults and killing a computer mid-run on a test rig until the survivors carry the flight every time. A candidate who reasons about a board browning out at three in the morning, beyond the clean case, shows the judgment the flight-software bar is set to find.

No silent loss: sequence, dedup, and backpressure

With the vote settled, the second hard problem is getting the data down complete, and it lives on the ground side. The link is the enemy: radio loses frames, a vehicle handed between ground stations can deliver the same frame from two receivers, and the sample rate can exceed what any one consumer drains. The fix has the shape every reliable pipeline takes. Every frame carries a monotonic sequence number and a vehicle timestamp at the source. Ingest tracks the next sequence it expects, so a duplicate is dropped, an in-order frame is applied, and a skip becomes a detectable gap. Delivery is at-least-once by design, and dedup on the sequence number makes it safe to apply each frame exactly once.

// Every downlinked frame is sequenced and timestamped at the source, so
// the ground can dedup and detect loss. At-least-once + dedup = each frame
// applied exactly once; a skipped sequence is flagged, never interpolated.
struct Frame {
    uint64_t seq;          // monotonic per link; the dedup and gap key
    uint64_t vehicle_ns;   // source timestamp; trusted over arrival time
    ChannelId channel;
    Sample    sample;      // one measurement (or a batch of them)
};
class Ingest {
    uint64_t expected_seq_ = 0;
public:
    void accept(const Frame& f) {
        if (f.seq < expected_seq_) return;              // duplicate: already applied
        if (f.seq > expected_seq_)                      // a frame was lost on the link
            mark_gap(expected_seq_, f.seq);             // show it; do NOT fill it in
        expected_seq_ = f.seq + 1;
        publish_live(f);                                // low-latency console path
        store_durable(f);                               // durable, complete record
    }
};

Backpressure is the other half of no-loss, and you should decide how the system degrades before it degrades by accident. Two consumers pull from ingest, and they have opposite needs. The console wants the freshest value, so a lagging console gets the latest reading and misses the ones in between; that never stalls ingest or affects another operator. The durable store must not skip anything, so it's buffered and absorbs bursts. If it ever can't keep up, shed the lowest-priority diagnostic channels first and alarm loudly, and never silently drop a flight-critical one. Saying which channels are sheddable and which are sacred is the reliability decision an interviewer wants to hear.

Two paths: consoles now, analysis later

The most useful structural decision is splitting the live path from the durable path. They optimize for opposite things, and one component serving both does each badly. The live path puts a trustworthy value on a console within a bounded delay. It's an in-memory publish that a slow reader can't stall, carrying the latest state instead of a complete history. The durable path keeps the record complete: a write-optimized time-series store keyed by channel and vehicle time that absorbs the full firehose and never drops a sample it accepted. Fanning each sample to both at ingest, instead of reading the console off the store, keeps a slow disk from adding latency to the flight team's screen.

• Live path: latest state, bounded delay. An in-memory stream consoles subscribe to. It carries the freshest value instead of a complete log, and a lagging console is served the latest sample rather than every intermediate one. Built so no reader can stall it and no blocking write sits on it.
• Durable path: complete record, write-optimized. A time-series store keyed by channel and vehicle time that absorbs the whole firehose. It carries a little more write latency than the console tolerates, in exchange for the completeness the archive and any investigation depend on.
• Timestamp at the source: vehicle clock, not arrival. Each sample is stamped on the vehicle and that time is authoritative, so a reconstruction knows when a thing happened regardless of link delay, retransmit, or which ground station received the frame. Arrival time is only a lower bound.
• Archive is immutable: evidence, not a cache. Once the flight is down the durable samples become the frozen record an investigation runs on, gaps marked explicitly where frames were lost. Completeness is the contract, which is why no-silent-loss is the top target rather than a nice-to-have.

Time is the subtle requirement inside all of this. Every sample is stamped on the vehicle at the instant it's measured, and everything downstream orders and correlates on that vehicle time. Arrival time on the ground won't do. Link delay varies, frames retransmit, and two ground stations can hand the same vehicle back and forth, so arrival time is noisy while source time is exact. The interviewer may push on getting one trustworthy clock across subsystems. The honest answer is that the vehicle carries a synchronized time base and stamps at the sensor, so the ground never guesses the order of events.

Alerting, anomalies, and the command uplink

Watching the stream and acting on it closes the loop, with the same reliability posture. Alerting runs over the durable store, off the sample path, so a heavy check across a recent window never adds latency to a console. It works on two levels. Hard limits fire immediately when a channel crosses a red line. Expected-behavior checks raise a softer flag when a value drifts out of its normal profile for this phase of flight. An alert the team sees a second too late didn't help, so the limit checks are cheap and run continuously, while the heavier profile analysis is allowed to run slightly behind.

The command uplink is telemetry in reverse, and it carries the strongest guarantee in the system, because an error costs the most here. A command must reach the vehicle, must not be applied twice, and must not be silently lost. So each command is sequenced, made idempotent by that sequence, and acknowledged. The ground sends, the vehicle applies once and confirms, and the ground retries until it sees the acknowledgment. A diagnostic telemetry channel can be best-effort; an uplinked command can't. Build the acknowledgment and idempotency in from the start instead of bolting on a retry later, so a dropped or duplicated command never becomes an incident.

The SpaceX angle

Showing you understand what the telemetry is for signals more than component fluency. This system is the flight team's only window into a vehicle they can't reach, and every post-flight conclusion rests on it. A channel that goes stale without a marker, a command applied twice, or a flight-critical sample shed quietly under load is a mission-level failure. The interviewer watches whether you keep blocking writes off the console path, timestamp at the source, split sheddable diagnostic channels from sacred flight-critical ones, and make the uplink acknowledged and idempotent. Those choices show you grasp that this is the nervous system of the flight. Frame it as a live picture other people bet a vehicle on, and you read as someone who could own a piece of a real flight-software stack.

What the round rewards

The bar here is reliability judgment and clear scoping, and the round is candidate-led, so drive it. Take the one-line prompt and set the scope yourself. Ask what's flight-critical and what's diagnostic, state that no-silent-loss and bounded latency are the targets that matter, and sketch the pipeline before diving. Then spend real time on the two hard problems: the vote that makes a flight-critical value trustworthy, and the framing, dedup and backpressure that make delivery lossless in effect. Reach for redundancy before scale, name the failure cases out loud, and say what the system does when a part breaks.

Common questions