Design a Low-Latency Market Data System

Understanding the problem

This is a low-latency low-level design question. The interesting part is what each message costs on the machine, and whether the firm's book stays a faithful copy of the exchange's book under a flood of updates. Then there is whether the worst-case latency stays close to the median instead of spiking whenever the feed bursts. Scaling to a billion users is beside the point. The functional surface is small: receive, decode, apply, publish. The depth is all underneath. Keep the hot path free of allocation and locks, and lay the book out so a mutation stays in cache. Two problems make a feed handler more than a parser: arbitrating the redundant lines and recovering from a lost packet.

Picture the shape before any code. An exchange publishes its book updates as a sequenced multicast stream. A kernel-bypass receive path pulls the packets off the wire without a trip through the kernel. A line arbitrator merges the two redundant feeds and forwards each sequenced message once. A decoder turns the exchange's wire format into an internal update, which is applied to the order book the firm keeps in memory. Every change becomes an entry on an SPSC ring that the strategies read. One direction, one hot thread, and a data layout chosen so the machine keeps up with the feed.

The Hudson River Trading angle

Set the register first, because it decides how you should answer. Hudson River Trading's strategies quote and trade automatically. The round trip from a market-data packet arriving to a decision leaving the box is counted in microseconds, some of it in nanoseconds. A design conversation about the feed handler is therefore not a capacity-planning exercise. Nobody wants you to shard the feed across a thousand machines or size a cache tier. HRT's own hiring writing is blunt that the failure they see most is a shallow grasp of fundamentals. They steer away from burst-of-insight puzzle questions in favor of whether you understand what the machine is doing. So the panel watches whether you pick a data structure and justify it by its behavior in cache. It watches whether you keep the hot path free of allocation and locks, and whether you treat book correctness and the latency tail as what matters. Answer in cycles, cache lines, and lost packets and you are speaking the room's language.

Functional requirements

Keep the core to the pipeline and let the interviewer widen it. The system has to receive the exchange's multicast messages off the wire. It has to arbitrate the two redundant lines the exchange sends, so that a message carried on both is applied exactly once and a message missing from one is still delivered by the other. It has to notice when a sequence number is skipped, meaning a packet was lost on both lines, and recover the missed updates rather than corrupt the book. It has to decode each message from the exchange's wire format into a normalized internal update and apply it to the order book it maintains. Then it publishes the resulting change to every strategy that consumes the feed.

Non-functional requirements

Reason through each target, because these are what draw the design later. Latency is the property everything bends around, and it is not the average that matters. A firm trading on this feed feels the tail. Take the one message in a million that took an extra microsecond because a buffer reallocated or a page faulted. That message made its price stale while everyone else's was fresh. So aim for a tight distribution. Receiving, decoding, and applying a message should each run in a small constant number of cycles. The worst case should sit close to the median instead of spiking when the feed bursts at the open or on a news print.

Correctness of the book is the second target, and on a feed handler it is not free. The exchange sends incremental updates, so the firm's book is only right if every update is applied, in order, exactly once. Multicast over the network can lose a packet or deliver duplicates, which is why the exchange sends two lines and stamps every message with a sequence number. The system has to be lossless in effect: it either applies each sequenced update or knows it has a gap and stops trusting the book until it has recovered the missing messages. A book that has quietly skipped an update is worse than a slow book, because the firm keeps trading on a picture that is wrong.

Determinism is the third target, and it is why the design leans away from anything that adds variance. No allocation on the hot path, because a call into the allocator is an unbounded pause waiting to happen. No lock on the shared state, because a contended lock turns a fixed-cost operation into one whose cost depends on another thread. No blocking system call in the loop, which is why the receive path bypasses the kernel and the thread busy-polls rather than sleeps. And scope it honestly, as low-level design demands. One feed is owned by one core. Throughput across many symbols or many venues comes from running many handlers side by side instead of parallelizing inside one.

• Latency is a distribution: tail, not average. The message that took an extra microsecond is the one that left a price stale. The target is a tight spread from median to worst case, so each stage is a small fixed cost that does not spike when the feed bursts.
• The book must never silently drift: lossless or known-gapped. Every sequenced update is applied in order exactly once. Otherwise the system knows it has a gap and recovers before trusting the book again. A quietly-skipped update is worse than a slow one.
• Nothing allocates on the hot path: pre-sized pools only. Orders come from a preallocated pool, price levels from a fixed array, and the rings are fixed buffers. The steady state touches only memory that already exists, so no operation waits on the allocator.
• One feed, one core: run many, not one big one. A single feed is owned by one busy-polled thread, so the hot path holds no lock. Throughput across symbols and venues comes from many handlers on many cores, which keeps each one deterministic.

State one framing choice out loud, because it steers everything downstream. There are two hard problems here and they should be reasoned about separately. Getting the bytes off the wire and the book laid out in memory is the latency problem. Applying exactly the right updates in exactly the right order is the correctness problem, and it lives almost entirely in line arbitration and gap recovery. Sketch the pipeline, then spend the design where the difficulty is, in those two places plus the memory layout that keeps the common path fast.

The pipeline, stage by stage

Lay the pipeline out as a straight line from the wire to the strategy, because that shape is the design. Read the diagram by role. Blue is an endpoint at the edge, the exchange upstream and the strategies downstream. Violet is a component we run on the hot path. Green is state we keep in memory. Amber is the one branch that leaves the fast path, the moment a gap sends the system into recovery.

Name the stages and what each owns, because the interviewer grades the boundaries as much as the code. The receive stage gets bytes off the wire with no kernel in the way. The arbitrator turns two lossy lines into one trustworthy ordered stream and raises a flag the moment it cannot. The decoder normalizes the wire format so nothing downstream knows which venue it came from. The book applies updates and holds the state. The publisher hands each change to consumers over a ring it owns alone. Everything on the fast path is one of these, and everything slow, recovery, capture, connectivity, hangs off to the side.

Sequence gaps and A/B line arbitration

This is the section that separates a feed handler from a parser, so slow down on it. The exchange sends its book as a stream of increments, each stamped with a sequence number that goes up by one. The firm's book is correct only if it applies every increment, in order, once. Multicast gives no such guarantee on its own: a packet can be lost, and the same packet can arrive twice. The exchange's answer is redundancy plus sequencing. It sends two independent lines carrying the identical numbered stream, and it publishes a recovery channel for the rare message that both lines drop.

Arbitration is the logic that turns those two lines into one trustworthy stream. The handler tracks the next sequence number it still owes its consumers. When a message arrives on either line, three things can be true. Its number is below what the handler expects, meaning the other line already delivered it, so it is a duplicate and gets dropped. Its number is exactly what the handler expects, so it is the next message, and it passes through and the expectation advances. Or its number is above what the handler expects, which means the awaited message was missing from both lines, and now there is a real gap. The first two cases are the whole common path, and they are cheap. The third is the case worth designing around.

// The A and B lines carry the SAME sequenced stream on two paths.
// The arbiter forwards each sequence number exactly once and flips
// into recovery the moment one is missing from both lines.
class LineArbiter {
    uint64_t expected_seq_ = 0;   // the next sequence number we still owe consumers
    bool     recovering_   = false;
public:
    // Called for every message off either line. Returns true when the
    // message should be applied to the book now.
    bool accept(uint64_t seq) {
        if (recovering_) { buffer_during_recovery(seq); return false; }
        if (seq < expected_seq_)  return false;  // the other line already delivered it
        if (seq == expected_seq_) {              // exactly what we were waiting for
            ++expected_seq_;
            return true;
        }
        // seq > expected_seq_: the awaited message was lost on BOTH lines.
        // Do not guess and do not plow on; recover off the hot path.
        recovering_ = true;
        request_recovery(expected_seq_, seq);    // snapshot / retransmit channel
        buffer_during_recovery(seq);
        return false;
    }
};

Two details are worth saying by name. First, a small reorder tolerance is normal. The two lines are independent, so a later sequence number can arrive a beat before the one you are waiting for. A short buffer sorts that out without declaring a false gap. The line above treats "ahead" as a gap for clarity, but a production handler waits a bounded moment before it gives up on the awaited message. Second, recovery is a state, not an instant. While it runs, live messages keep coming and have to be held, and the handler is only back to normal once it has replayed the buffer up to the live sequence. Getting that catch-up right, so the book is exactly correct at the seam between recovered and live, is the point an interviewer will probe.

The Hudson River Trading angle

On this loop the correctness of the book carries as much weight as the speed. HRT's strategies trade automatically on the picture this system produces. A book that has quietly dropped a cancel shows a wrong price, and the firm will act on it before anyone notices. Raise loss before the interviewer does. Name that multicast is lossy, and explain why the exchange sends two lines and sequences every message. Design the gap path deliberately instead of assuming the network is clean. Picture a candidate who says "I detect the skip, I stop trusting those symbols, I recover from the snapshot channel, and I define what the strategy sees meanwhile." That candidate has thought about what happens when the feed misbehaves at three in the afternoon. The firm is hiring for that judgment.

The hot path: pools and a cache-friendly book

With arbitration settled, the steady state is short, because the layout does the work. The book the handler keeps is the same shape as any fast order book. Each side is a flat array of price levels indexed by tick, and each level holds a FIFO of resting orders. An id map turns a cancel or modify into a constant-time unlink. The orders live in a preallocated pool, so applying an update never calls the allocator. Read an update, index the level, splice a node or adjust a quantity, publish the delta. Nothing searches, nothing sorts, nothing allocates once the handler is running.

// An order is an intrusive list node: its links live inside the struct,
// so resting it costs no separate allocation and the node plus its links
// share a cache line. Hot fields sit together, near the front.
struct Order {
    uint64_t id;          // exchange order id, the key in the id-to-node map
    uint32_t qty;         // remaining quantity, adjusted on partial fills
    int32_t  price_tick;  // price in whole ticks, the index into the level array
    Order*   prev;        // intrusive links within this price level's FIFO
    Order*   next;
};
// One price level: aggregate resting quantity and the head and tail of
// the FIFO of orders at this price.
struct PriceLevel {
    uint64_t total_qty = 0;
    Order*   head = nullptr;
    Order*   tail = nullptr;
};

// The firm's copy of one symbol's book, rebuilt from the exchange's
// increments. Two arrays of price levels indexed by tick, an id map for
// O(1) cancel and modify, and a pool so applying an update never mallocs.
class Book {
    std::array<PriceLevel, PRICE_SLOTS> bids;   // buy side, index = price tick
    std::array<PriceLevel, PRICE_SLOTS> asks;   // sell side
    std::unordered_map<uint64_t, Order*> by_id; // exchange id -> node, O(1) cancel
    ObjectPool<Order> pool;                     // preallocated, no hot-path malloc
    int32_t best_bid = -1;
    int32_t best_ask = PRICE_SLOTS;
public:
    void on_add(uint64_t id, Side side, int32_t tick, uint32_t qty);
    void on_cancel(uint64_t id);                // resolve id, unlink in O(1)
    void on_exec(uint64_t id, uint32_t filled); // exchange printed a trade
};

The whole hot path fits in one loop on one core, and writing it out is the clearest way to show it holds no lock, makes no system call, and allocates nothing. Spin on the card, pull a packet, walk its messages through the arbiter, decode what survives, apply it, and publish the change. A packet carries many updates, so the per-packet fixed cost amortizes over a batch, which is one reason the design prefers a few larger operations over many tiny ones.

// The entire fast path on one core: busy-poll the card, arbitrate the
// two lines, decode, apply to the book, publish the delta. No syscalls,
// no locks, no allocation in the steady state.
void FeedHandler::run() {
    for (;;) {                                  // busy-poll: never sleep
        Packet* p = nic_.poll();                // kernel-bypass receive, no syscall
        if (!p) continue;                       // nothing yet, spin again
        for (RawMsg m : p->messages()) {        // one packet carries many updates
            if (!arbiter_.accept(m.seq)) continue;   // duplicate or gap: skip it
            Update u = decoder_.normalize(m);   // venue wire form -> internal update
            book_.apply(u);                     // O(1) mutation, cache-warm
            out_ring_.push(book_.delta(u));     // hand the change to a consumer
        }
        nic_.release(p);
    }
}

One thread, busy-poll, and no locks

This is the section the round is really testing, because the instinct to reach for threads is where latency designs go wrong. The book is shared mutable state under a flood of updates, so the reflex is to protect it with a lock and let several threads work the feed in parallel. On a latency-critical path that reflex is the bug. A lock turns a fixed-cost operation into one whose cost depends on what another thread is doing. It is cheap uncontended and a stall when contended. Threads collide at the moment the feed bursts, so the lock adds cost when you can least afford it. Worse, it adds variance, and variance is the tail the whole design exists to flatten.

The tradeoff is honest and worth stating. Removing the lock removes contention instead of managing it. Each stage is a fixed cost and the tail stays tight. Ordering falls out for free, because one thread applies updates in the order it reads them. Parallelizing one feed cannot be done cheaply given the ordering constraint, so throughput comes from partitioning. Assign each feed, or each group of symbols, to its own core and run many single-threaded handlers side by side. A machine carries many feeds while every individual book stays deterministic. The cost is that one very hot feed is bounded by one core's speed. So every stage on that core is kept to the cheapest operations there are: integer compares, array indexing, and list splicing. The thread also spins instead of sleeping, since a wakeup from the kernel would cost more than the work it woke up to do.

Memory, cache, and huge pages

The data structures were chosen for their memory behavior, so make that reasoning explicit, because it is the register the round lives in. A cache miss to main memory costs on the order of a hundred nanoseconds, while an add or a branch costs a fraction of one. On the hot path a single miss can dwarf all the arithmetic around it. That one fact drives almost every layout choice: keep the data a message touches contiguous and small enough to stay in cache, and avoid designs that chase a pointer to somewhere cold. The array of price levels keeps the top of book in a few cache lines. The intrusive links keep an order and its pointers on one line. The pool keeps order nodes packed together, so walking a level's queue touches warm memory.

• Contiguous over pointer-chasing: arrays beat trees. The price-level array keeps the top of book in a handful of cache lines, so working near the best price stays hot. A tree of price nodes scattered across the heap misses the cache on every hop, and the misses, not the comparisons, are the cost.
• Intrusive links, pooled nodes: no wrapper allocation. Links live inside the Order struct and nodes come from a pool, so an order and its list pointers share a line and pooled orders sit near each other. Traversing a level touches packed memory instead of scattered heap.
• Cache-line separation on the ring: kill false sharing. The producer and consumer indices of each SPSC ring sit on their own cache lines, so the two threads never invalidate each other. Sharing one line would trade a nanosecond handoff for a coherence stall.
• Huge pages shrink the page-table walk: fewer TLB misses. Backing the book and pools with 2 MiB or 1 GiB pages instead of 4 KiB means far fewer TLB entries cover the working set. A translation miss, and the dependent memory loads that resolve it, then happen far less often on the hot path.

Spell out false sharing. It is a favorite probe, and it is invisible in the code. Two variables that logically belong to different threads can land on the same 64-byte cache line. Then every write by one thread invalidates the other thread's copy, forcing a reload across cores even though neither touched the other's data. The fix is padding: put the ring's head and tail indices on their own lines. The usual mechanical-sympathy habits apply too. Keep branches predictable so the pipeline is not flushed, and prefetch the next order node while working the current one. Prefer a few larger operations over many tiny ones so fixed costs amortize.

The Hudson River Trading angle

HRT's own engineering writing dwells on this layer, and its onsite goes under the hood here. Go a level deeper than the diagram on this part of the answer. The published low-latency material returns again and again to the cost of address translation. A virtual address has to become a physical one, and an ordinary 4 KiB page can leave that translation missing from the TLB. Resolving the miss walks a multi-level page table: several dependent memory loads before the real access even starts. Backing the hot data with huge pages, 2 MiB or 1 GiB instead of 4 KiB, means a handful of TLB entries cover the whole working set and that walk almost never happens. The onsite probes the same instincts from the other side. Expect questions like what happens when you call malloc, how a virtual call compiles and what a devirtualized one saves, and how a container grows its storage. The round is built to find fluent reasoning at that level, in TLB entries, page-table loads and cache lines as well as big-O. That is why this design is written in C++ with the allocator kept off the hot path.

Publishing to strategy consumers

Every change to the book has to reach the strategies, because they price and decide against exactly this feed and it must never disagree with the book the handler holds. The trap is to publish inline: encode the update and push it to wherever it is going from inside the hot loop. That puts serialization, and possibly I/O, on the critical path. The time to process a message now depends on how fast the slowest consumer or destination drains. The design exists to avoid that coupling. The fix is the same lock-free handoff used on the way in, in reverse.

The handler writes each book change as a compact record to a per-consumer SPSC ring and goes straight back to the wire. Each strategy drains its own ring on its own core, so one slow reader cannot stall the handler or any other reader. A strategy that falls behind is served a conflated top of book: the current best bid and ask, without every intermediate delta. It catches up on the state that matters instead of dragging the producer down. Anything genuinely slow, encoding for an external destination, capture to disk for later analysis, monitoring, is drained off yet another ring by a separate thread. The principle holds across the whole design: the one thread whose latency is money does only the work that must happen now, and everything that can wait is moved off it.

The Hudson River Trading angle

Showing you understand what the feed is for signals more than data-structure fluency. Every automated decision the firm makes downstream is only as good as the book this handler publishes, and several strategies read the very same feed. The trading system depends on the handler emitting one change per mutation and keeping publication off the hot path. It also depends on a slow reader taking a conflated view instead of back-pressuring the source. The interviewer watches you keep encoding and capture off the critical path, publish over a lock-free ring per consumer, and reason about what a lagging strategy should see. They want to know if you grasp that this component sits at the head of everything the firm trades. Frame it as a live structure under a firehose whose output other systems act on. You will read as someone who could own a piece of a real trading stack.

What the round rewards

The bar here is mechanical sympathy, determinism, and correctness of the book, not scale. The design conversation reads as a discussion, so narrate the mechanism plainly. Lay out the pipeline from wire to ring and scope it to one feed on one core. Then spend real time on the two places the difficulty lives. Get the line arbitration and gap recovery right, because a feed handler that cannot survive a lost packet is not a feed handler. Defend the book's layout by its behavior in cache, and design the lock away with a single busy-polled owner and lock-free rings. Expect the panel to push down into the memory and systems layer, because that under-the-hood fluency is what this loop is built to test.

Common questions