Design a Market Data System

Picture what lands on the wire before you commit to any structure. An exchange broadcasts, it does not answer questions. A continuous stream of small binary messages arrives over UDP multicast: an order rested at this price for this size, an order was cancelled, a trade printed, an auction imbalance shifted. Each record is a few dozen bytes, each carries a sequence number, and they arrive back to back at a rate that spikes hard when the market moves. The system stands at the wire, reads that stream, and turns it into something the rest of the firm can reason about. Four jobs. Take the raw feed in, decode it into one model, and rebuild the book. Then keep the running statistics on top of the book fresh, which is the one people forget.

Decoding means one model for the whole world, so be precise about what "one model" buys you. Every venue speaks its own dialect. Nasdaq sends ITCH, CME sends a Simple Binary Encoding feed, the consolidated options feed carries its own layout, and their symbols and price scales all differ. Nothing downstream should care which venue a quote came from. So the system decodes each proprietary format into a canonical record: a firm-wide instrument id, a side, a price held as a fixed integer, a size, a message type, and a timestamp. The lever here is the type of that record. If the canonical form is a variant that is either a valid, fully-typed message or an explicit error, a half-filled record cannot exist to trip up a strategy two layers away.

Rebuilding the book means the system keeps state, it does not just forward messages. From the stream of adds, cancels, and modifies it maintains a live order book per instrument: resting bids ranked high to low, asks low to high, the size sitting at each price. But most consumers do not read the raw book on every tick. They read functions of it: the top of book, a moving average of the mid, VWAP over a window, rolling volatility.