Design the Metrics System

Understanding the problem

"Design a monitoring service like Datadog or SignalFx, assume an agent on every host" is the Datadog system design round in one sentence, and it is the most-asked design prompt in the loop. The shape is concrete. Millions of hosts each run a small agent that emits metrics every ten to fifteen seconds: a name like request.latency, a set of tags like service:web, region:us-east, host:i-0abc, a numeric value, and a timestamp. Our job is to ingest that firehose, store it cheaply, then answer queries like "p95 latency for service=web over the last hour, broken down by region" in under a second. Everything downstream is shaped by that volume.

Get the vocabulary down before anything else; every later choice hangs on it. A single metric point is the four-tuple of name, tags, value, and timestamp. A time series is the identity formed by a name plus a specific tag set: request.latency{service:web, region:us-east, host:i-0abc} is one series, and the same metric reported from a different host is a different series. Storage is organized around series, not individual points, and a query selects a set of series by their tags before aggregating over a time window. Fix that early. The whole design rotates on the series and its tag set, read over a window.

The Datadog angle

This isn't "something like Datadog." It's Datadog's actual product, and the person across the table built it. A hand-wave that would pass elsewhere stands out fast. They know where the bodies are buried: the cardinality of the tag index, the cost of storing raw points forever, the impossibility of averaging percentiles. This is also where seniors get sunk, the candidates who clear both coding rounds and then never get concrete on design. So reach for the real mechanisms. Agents, a Kafka buffer, rollups, a per-series sketch, the way someone who has read the engineering blog would. The scale number isn't background. It's the prompt.

Functional requirements

Hold the scope to three verbs: ingest, store, query. The agent reports points, we durably accept them, and a user runs a time-range query that filters by tags and aggregates. The one aggregation we must get right is the percentile, because the round leans on it and because it is the part that breaks naive designs. Everything else is below the line.

Non-functional requirements

Put a number on each one, because these decide the architecture. Write volume is the dominant constraint. We accept on the order of a billion points per second at peak, and we must never drop the firehose when storage hiccups, so a buffer has to decouple ingest from the write path. Reads are far rarer than writes but latency-sensitive: a dashboard query should return in well under a second even when it touches millions of series. Freshness can be relaxed to a few seconds. A point reported now showing up on a graph a handful of seconds later is fine, which buys us an asynchronous pipeline. And percentiles are allowed to be approximate, accurate to a percent or two, which is the single relaxation that makes the design affordable.

Cost is a first-class requirement here, because storing every raw point at this volume is the fastest way to bankrupt the design. We will trade exactness for money in two places, downsampling old data and approximating percentiles, and we flag both now so the deep dives can cash them in. Availability matters more than strong consistency: a metrics backend that is briefly behind is acceptable, but the worst possible moment to be down is the middle of an outage. So we accept being eventually-correct rather than blocking writes to stay consistent. Stay up first. Be exactly right second.

• ~100T Points per day. ~1.2B points/sec at peak.
• < 1 s Query latency. Even across millions of series.
• ~Few sec Freshness. Async pipeline is fine.
• Approx OK Percentiles. Accurate to a percent or two.

The set-up

Core entities

Name the data the system passes around first, then draw. Four carry the design:

The store under the rollups is a time-series store. We will name Cassandra as the concrete pick. The access pattern is a perfect fit, and it is the database the original Datadog prompt explicitly asks you to reach for. Every write is an append keyed by series and time. Every read is a contiguous range scan over one series within a window. There is essentially no update-in-place. We partition by series_id so one series lives together and a range scan is sequential, and we order within a partition by timestamp so a time-range read is one continuous sweep. Cassandra's wide-row, write-optimized, horizontally-sharded model is built for exactly this, and it sustains the write rate by spreading partitions across many nodes. The tag index is a separate store, an inverted index we discuss in its own deep dive, because mixing the "which series" lookup into the time-ordered store would wreck both access patterns.

API

Two endpoints carry everything. The agent batches many points into one POST every interval rather than one request per point, because a billion tiny HTTP calls a second would melt any gateway. The query endpoint takes a metric and a tag filter over a time range, plus the aggregation to apply, and returns a series of timestamped values, already rolled up to a resolution that suits the range.

The POST returns 202 Accepted on purpose, where a 200 OK would overpromise. It tells the agent we durably buffered the batch, and says nothing about whether the batch is queryable yet. That single status code is the contract that lets us put a buffer between ingest and storage, which is what survives the firehose. The query parameters preview the three deep dives that matter: agg=p95 forces the percentile question, filter forces the tag index, and from/to forces the rollup-and-retention story. We build each below.

High-level design

Go requirement by requirement, standing up the simplest working version before any hardening. The colors below sort components by role. Blue is the Agents on every host. Violet is the services: the Ingest gateway, the Write consumers, and the Query service. Amber is the Kafka buffer that absorbs the firehose. Green is the Time-series store and the Tag index. The walkthrough tags each component the same way.

1) Ingest the firehose. The Agents batch points and POST them every interval to the Ingest gateway, a stateless fleet that authenticates and lightly validates and does nothing slow. It writes each point straight to the Kafka buffer keyed by series id and returns 202. The gateway never touches a database, so it scales out flatly and stays cheap. And because the agent batches, the gateway sees a manageable rate of fat requests rather than a billion tiny ones. The naive version we are deliberately not building yet would have the gateway write straight to the store synchronously, which collapses the instant the store hiccups. Deep dive four explains exactly why the buffer is not optional.

2) Buffer the write firehose in Kafka. A queue earns its box here because the firehose is bursty and the store cannot be allowed to apply backpressure all the way to the agents: bursts have to wait somewhere that is not the request path. We partition the Kafka buffer by series id so every point of one series lands on the same partition in timestamp order. That ordering lets a single consumer aggregate a window with no cross-partition coordination. Kafka holds points durably for a retention window, so a slow or restarting store loses nothing. What we punt on here is duplicates: at-least-once delivery means a point can be replayed, and a percentile or a count that double-counts is wrong. Deep dive four settles the dedupe.

3) Roll up and store. The Write consumers read each partition and do the aggregation the design is built around. Rather than store every raw point, a consumer groups a series' points into a per-minute window and writes one Time-series store row holding the window's sum, count, min, max, and a percentile sketch. The first time it sees a series it also registers that series in the Tag index. The store is partitioned by series id and ordered by time, so this is an append. We are punting two big questions. Deep dive one is how rollups and retention keep 100T points a day from exploding the bill. Deep dive two is how a sketch lets us serve p95 without keeping raw values.

4) Serve the query. A dashboard hits the Query service with a metric and a tag filter over a time range, plus the aggregation to apply. It first asks the Tag index which series ids the filter selects, intersecting the posting lists for service:web and region:us-east. Then it range-scans those series' rollups from the Time-series store at a resolution that fits the range. An hour query reads per-minute rollups, a month query reads per-hour ones. It merges the per-window aggregates across series and returns the result. It reads rollups, never raw points. Reads are rare. Writes are the firehose. The punt is the index itself: resolving a filter over millions of series fast, and surviving high-cardinality tags, is deep dive three.

Technology choices

• Ingest gateway is a stateless service fleet (any language, Datadog’s real agent and intake are Go) behind a load balancer. It holds no state, so when one falls over the load balancer routes around it and a new one spins up cold. The only failure that matters is the gateway outrunning Kafka, which it can’t for long because writing to a partition is a fast append; if a Kafka broker is unavailable the gateway writes to another in-sync replica.
• Kafka buffer: Kafka is the event log that decouples ingest from storage. Partitioned by series id so one series is one ordered stream; replicated across brokers so a broker failure promotes an in-sync replica and the consumer resumes from its committed offset with no lost points. Consumer lag, how far behind the latest offset the consumers are, is the freshness dial: rising lag means the store is struggling, and we can scale consumers or shed load before dropping anything.
• Write consumers are a consumer group, one set of partitions per consumer, that does the rollup-and-sketch work. If a consumer dies, its partitions are reassigned and the replacement replays from the last committed offset, which is why the rollup write has to be idempotent (keyed by series plus window) so a replay overwrites rather than double-counts. This is the work-queue-versus-event-log distinction in action: Kafka is an event log we can rewind and replay, not a queue that deletes on read.
• Time-series store: Cassandra (or Datadog’s columnar Husky on object storage). Wide-row and write-optimized: partition key is the series id, clustering key is the timestamp, so a write is an append and a time-range read is one sequential scan. Horizontally sharded across nodes to sustain the write rate; replication factor three for durability and to survive a node loss. The probe it invites, "what if a hot series overwhelms one partition," is answered by sharding a hot series across sub-partitions by time bucket.
• Tag index is an inverted index from tag to series ids, the same idea as a search index, served from a store tuned for set intersection (Elasticsearch-style postings, or a custom index). Its failure mode is not a crash but cardinality: a tag with millions of distinct values bloats the postings and slows intersection, which deep dive three handles head-on. The brand is less important than the category; what matters is that it answers "which series match this filter" without a full scan.

Potential deep dives

At Datadog the deep dives decide the level, and most of the signal sits here. The interviewer locks onto one part of the pipeline and pushes until the cost or the correctness gives way. The four below recur the most. Each is scored against the same tiers, and each opens on the number that forces the design.

1) How do we store 100T points a day without the cost exploding?

Start with the arithmetic, because it is the whole reason this design is hard. Take two million hosts, each emitting on the order of 8,640 distinct series, reporting every fifteen seconds. A day has 86,400 seconds, so a series reports 86,400 / 15 = 5,760 times a day. Multiply: 2,000,000 hosts × 8,640 series × 5,760 points ≈ 1.0 × 10^14. That is about 100 trillion points a day. Average that over the day and it is 100T / 86,400s ≈ 1.2 billion points a second. If each raw point costs roughly 12 bytes (a timestamp and a value), 100T points is about 1.2 petabytes a day uncompressed, and storing six months of that as raw rows is hundreds of petabytes. Nobody pays that bill. The data almost nobody queries at one-second resolution months later is not worth keeping that way.

What forces the upgrade is the observation that resolution and retention are inversely valuable: recent data is read at fine resolution, old data is read coarsely or not at all. So we do the aggregation once, on the write path, and we age data down through resolution tiers instead of keeping it raw.

The scaling writes lesson covers absorbing and pre-aggregating a high write volume, and Cassandra and DynamoDB covers the wide-row, time-partitioned store the rollups live in.

The Datadog angle

Rollups and retention tiers are how the product gets priced, not just built. Datadog keeps metrics at full resolution for a short window and downsamples older data into coarser rollups. Retention is a literal line item on the bill. A candidate who lands on tiered retention is describing the real cost model, not a textbook one. Put it bluntly: you don't store 100T points a day, you store rollups of them, and you keep raw data only as long as someone might debug with it. That's metrics economics the way the team lives it, where every byte kept for a year gets multiplied by trillions of series.

2) How do we serve p95 without storing every raw value?

This is the percentile-over-time-series question, asked almost word for word, and it is the one naive designs get wrong twice. A p95 for one series over one minute means: of all the latency values in that minute, the value below which 95% of them fall. Two facts make it hard. First, at this volume a single busy series can record millions of values a minute, so keeping every value to sort later is exactly the cost we just refused to pay. Second, and less obvious, you cannot average percentiles. If host A's p95 is 100ms and host B's p95 is 200ms, the combined fleet's p95 is not 150ms. A percentile is a position in a sorted distribution, and positions do not add. So we need a structure that summarizes a distribution cheaply and, crucially, can be merged across hosts and windows.

What we need is a small, fixed-size summary of a distribution that gives an approximate percentile and, the key property, can be merged: combine two hosts' summaries and read the fleet's percentile off the result. That structure is a histogram-style sketch, and it ties straight back to the coding round.

This is the same structure the latency histograms coding round builds by hand: bucket counts, log-scale buckets for the tail, a percentile read as a running-sum scan until you cross a rank, and merging two histograms by adding their counts. The coding round implements one host's sketch; this design distributes it across the fleet. Saying "this is the histogram from the coding round, made mergeable and rolled up over time" connects the two rounds in a way that lands.

The Datadog angle

The percentile sketch is literally Datadog's, and it bridges straight to the coding loop. DDSketch is a Datadog invention with a published relative-error guarantee, built precisely because you can't average percentiles across a fleet and can't afford to store every value. The product ingests latency as distributions, merges them across hosts and time, and serves p50, p95, and p99 off the merged sketch. Name a mergeable sketch, explain why averaging p95s is invalid, tie it back to a Premium question coding problem, and you've handled the hardest idea in the metrics backend. Of the four dives, this is the one that pushes a senior answer toward staff.

3) How do we serve a tag-filtered query fast over millions of series?

A query like service:web AND region:us-east has to find which series match before it can read any data, and the count is the problem. With two million hosts each carrying thousands of series, there are on the order of hundreds of millions to billions of active series. Checking each one's tags against the filter is a full scan of billions of records for every dashboard refresh, which blows the sub-second budget instantly. We need to go from filter to matching series ids without looking at the ones that do not match. A full scan is off the table.

Elasticsearch covers inverted indexes and posting-list intersection, the same machinery a search engine uses to map a term to its documents.

The Datadog angle

Cardinality is the most famous failure mode in the product. "High-cardinality tags blow up the index" is not a textbook footnote here. It is the thing that drives Datadog's custom-metric pricing and the number-one way customers accidentally run up a bill. Two things matter. Do you see that a tag like user_id multiplies the series count and the index size without bound? And would you defend the system by metering or rejecting it at ingest? Bring up cardinality yourself, and you've named the one place this exact system breaks in production.

4) How do we handle the write firehose without dropping points?

Back to the rate: roughly 1.2 billion points a second on average, and far higher in bursts when a fleet redeploys or an incident makes everything noisy at once. The store cannot absorb that directly, and the one thing a metrics backend must never do is silently drop points during the exact incident an engineer is staring at the dashboard for. So the question is how ingest stays up and lossless when the store is slow or briefly down. Losing points is not allowed.

One follow-up the interviewer reliably reaches for: partitioning by series id is right for ordering, but it can create a hot partition. A handful of extremely busy series, say a metric on a load-balancer tier that every request touches, can pile far more volume onto their partitions than onto others. One overloaded partition then slows the consumers that own it. The fix is to spread a hot series across sub-partitions, hashing on series id plus a small time bucket so its load fans out, then merging the sub-partition rollups back together at write time. The same hot-key reasoning shows up on the read side, where one viral series is read far more than the rest. A small read cache in front of the Query service absorbs that.

The scaling writes lesson covers buffering a high write rate, and Kafka covers partitioning and consumer groups, plus the offset replay and at-least-once delivery that force idempotent consumers.

The Datadog angle

A Kafka buffer in front of the metrics store is the architecture the prompt itself names, and the one Datadog runs. The prompt often spells out the toolbox: "describe the libraries or frameworks, Kafka, Cassandra." So Kafka as the firehose buffer and Cassandra-style storage answers the question on its own terms. Datadog runs one of the largest Kafka deployments anywhere, precisely to decouple ingest from storage at this scale. And the buffer isn't there for throughput alone. It's there so a store hiccup during an incident never costs you the metrics you need most at that exact moment.

What is expected at each level

That is a lot of surface for one pipeline. How far you take it scales with the level you are targeting, and the round leans on depth, because the reject that recurs is a strong coder who never gets concrete about design.

Mid-level

Breadth first. Get the pipeline right end to end. Agents batch and POST, a gateway writes to a buffer, consumers roll points up into per-minute aggregates and store them, and a query reads those rollups over a time range. The one cost idea to land is that you store rollups, not every raw point, and you should be able to say why a per-minute aggregate is cheaper than the raw rows it replaces. The interviewer will drive the percentile and tag-index depth. The bar: a clean, working pipeline with rollups, where you can explain why keeping raw points forever is too expensive. It is fine if the sketch and the cardinality cases need prompting.

Senior

Breadth first, then real depth you reach on your own. Two problems should surface without prompting. First, percentiles: you cannot store every value, and you cannot average per-host p95s, so you keep a small mergeable sketch per window instead. Second, the tag filter: an inverted index from tag to series ids, intersected as posting lists, is what turns a filter into a set of series without scanning them all. Defend Kafka as a deliberate buffer too, not decoration. Consumer lag is the freshness dial you watch, and idempotency is how you survive a replay. The bar: you find the percentile and tag-filter bottlenecks without being led to them, take each to a Great tier, and explain why you picked the buffer and the time-series store.

Staff+

By the staff bar the scope widens, and the cardinality fight is yours to drive. Treat cardinality as a budgeted resource: you cap or reject runaway tags right at ingest. Own the retention tiers too. Raw data ages into per-minute rollups and then per-hour ones, and that is how you keep the cost in line. Drive the mergeability of the sketch as the one property that makes a fleet-wide percentile possible at all. Then widen out. Add multi-region ingest, where each region runs its own buffer and store, and treat cost as a lever you pull at every layer. Under overload, you shed low-priority metrics on purpose rather than letting points drop silently, and you call that a design decision, not an accident. The bar: you lead and the interviewer only nudges, never rescues, and you reason about cost and cardinality end to end. The percentile sketch and the retention tiers stay at the center of your design, which is where a strong answer keeps them.