Design an Event Analytics Pipeline

Understanding the problem

Every product team wants the same picture. Which features do people actually use, how does usage move through the day, how many distinct people showed up. A product-analytics pipeline is what draws that picture. An SDK sits inside the app, and every meaningful action fires a small event: a button tapped, a report exported, a workflow run. Those events arrive by the billions, and the pipeline folds them into counts a dashboard can read while they are still warm. It never stops taking events. And it can never scan them all at read time, so the counting has to happen on the way in.

Two things make this harder than a counter behind an API. The first is volume. Events arrive faster than any one machine can accept them, so ingestion has to batch at the edge and spread the load wide, and the raw stream is far too big to keep query-ready. The second is counting itself. Networks are flaky, so the SDK retries, and a retry sends the same event again. Increment naively and every retry inflates the number. A count that a customer sees, or gets billed on, has to reflect each real action once and only once, no matter how many times the wire delivered it.

Put numbers down before drawing anything. Aim at ten million people using the product on a busy day, each firing on the order of two hundred events. That's roughly two billion events a day and around a hundred thousand a second at peak. Each event is small, a few hundred bytes: an id, who did it, what they did, plus the moment it happened. The product wants three views of every metric, rolled up to the fifteen-minute window, then the hour, then the day. It wants the newest bucket fresh within seconds. One tension runs through the whole design. The live tiles on a dashboard can be approximate and a hair behind. The totals that feed billing, and the numbers a customer reports to their own boss, cannot be. A count is allowed to be late. It is not allowed to be wrong.

The Rippling angle

Rippling runs this interview as a depth test. The box diagram is the easy part and they know it, so the follow-ups drive straight at the counting. An event arrives twice, now what? Here that question sits at the center of the round. Rippling sells itself as the one system of record across HR, IT, and payroll, and product usage gets measured to power adoption dashboards and, for some modules, what a customer pays. A pipeline that double-counts under retries reports more active users than exist and bills for usage that never happened. So bring the mechanism, dedup by event id, and the willingness to defend it while the interviewer pokes at the edges. What keys the dedup. How long you remember an id. What a duplicate that shows up a day late does to the count.

Functional requirements

Keep the core tight and name what you are setting aside. The pipeline ingests a high-volume event stream, counts events into per-metric buckets at three grains, serves those counts to dashboards fast, and guarantees each event lands in the count once. Everything past that you name and skip.

Non-functional requirements

Attach a target to each non-functional axis, because these are what the design is judged against. Throughput has to absorb a hundred thousand events a second at peak without dropping or stalling, which forces batching at the edge and a partitioned queue behind it. Freshness means the newest bucket is queryable within a few seconds, so a nightly batch alone will not do. Correctness has two tiers, so state them plainly. The official counts are exactly-once, every event reflected a single time across retries and reprocessing. The live readout may be approximate and settle a moment later. Durability means an accepted event survives a consumer crash, because the queue holds it until a processor has folded it into a rollup that is safely stored.

One line runs under all of it, so say it now. Freshness wants to publish a bucket the instant its window passes. Exactness wants to wait, because a retried or late event might still change that bucket. They pull apart, and the design does not pretend otherwise. It resolves them with a fast path that dedups and rolls up in near real time for the live number, and a slower recompute over the retained raw events that produces the authoritative total. The dial that never moves is single-counting. A tile can flicker and settle. A billed total cannot count one tap twice.

• Ingest throughput: ~100K events/sec peak. Roughly 2B events a day, about 23K a second on average, bursting near 4x toward business hours. Batching at the SDK and a partitioned queue absorb it; one node could not.
• Freshness: Newest bucket in ~seconds. The live rollup is readable within seconds of an event landing. A batch-only path cannot meet that, which is why the stream computes the fast number at all.
• Correctness: Exactly-once counts. Each event counted once across SDK retries and queue redeliveries. Live tiles may be approximate; billed and reported totals must be exact.
• Grains: 15 min / 1 hr / 1 day. Three rollup resolutions per metric. The fifteen-minute grain is computed from the stream; the coarser two fold up from it rather than rescanning raw.

The set-up

Core entities

Name the handful of things the pipeline moves before drawing a box. Four carry the design.

One split holds the rest up. The durable, ordered record of every event in the Kafka queue, tiered into Object storage for long retention, is the source of truth, replayable from disk if anything downstream falls over. The Rollup processor holds only transient dedup and counting state, rebuildable from the log. The rollups in the Rollup store are derived data, cheap to recompute by replaying raw events through the same logic. An event lands in a durable log, a stateful processor dedups it and folds it into buckets, and a serving store answers reads. That shape is the standard product-analytics topology, the one behind tools like Amplitude, and naming it up front tells the interviewer you know what you are building.

API

Two interfaces matter and they sit at opposite ends. The SDK pushes events into the front through a batched collect call that returns once the batch is durably queued. Readers pull finished counts from the back through a thin query surface over the rollups. Dedup, bucketing, the rollup cascade: all of that lives in the middle and never surfaces in either interface.

Two fields carry the contract. The eventId on track is the idempotency key, and every later exactly-once claim traces back to it. The client sets it once and holds it across retries. The pipeline uses it to recognize a duplicate. The getUnique response is deliberately an estimate. Counting distinct users exactly would mean holding every user id in every bucket. Those sets don't add up when you fold a fifteen-minute count into an hour, so the design trades a little accuracy for a mergeable sketch. One field pins the correctness story. The other pins where correctness is knowingly approximate.

High-level design

Walk it the way an event moves, and build the simplest thing that counts correctly before optimizing. A batch of events is accepted and appended to a durable queue. A processor reads the queue, drops duplicates by event id, and folds each survivor into its buckets at the fifteen-minute grain. Those rollups are written to a serving store and cascaded up into hourly and daily grains. A slower job replays the raw log to recompute exact totals. Color marks each role in the diagram. Blue is the SDK and the Dashboard. Amber is the Kafka queue. Violet is the Ingest API, the Rollup processor, the Query service, and the Batch recompute. Green is durable data: the Dedup store, the Rollup store, and Object storage.

Follow an event through the pipeline. On the write side, the SDK batches events and posts them to the Ingest API, which appends them to Kafka and returns. Kafka partitions by user id, so every copy of a given event, and every event for one user, lands on one partition in order. The Rollup processor consumes a partition, checks each event id against the Dedup store, drops the ones it has already seen, and folds the rest into fifteen-minute buckets it writes to the Rollup store. Kafka also tiers its raw records into Object storage. In the background, the Batch recompute replays that raw log and overwrites the Rollup store with exact totals. The Query service reads finished rollups for the Dashboard.

1) A batch lands in the durable queue first. The SDK does not fire one request per event. It buffers events and flushes a batch every few seconds or every fifty events, whichever comes first. That single choice turns a hundred thousand events a second into a couple of thousand HTTP requests a second, which is the difference between an ingest tier you can afford and one you cannot. The Ingest API does almost nothing on the hot path: validate, stamp a receive time, append to Kafka, acknowledge. Kafka replicates each record before the ack, so an accepted batch is durable the moment track returns. Partitioning by user id earns its keep twice. It spreads the firehose across brokers, and it puts every duplicate of an event on one partition, which is what lets a single consumer dedup locally instead of coordinating across the cluster.

2) The processor dedups, then folds by event time. The Rollup processor reads its partition in order. For each event it asks the Dedup store one question: have I seen this id? A hit means a retry or a redelivery, and the event is dropped. A miss means a real action. The processor records the id and folds the event into every bucket it belongs to, choosing buckets by the event's own timestamp instead of the wall clock. Count and sum buckets just increment. A distinct-user bucket updates a sketch. The fifteen-minute grain is the only one computed from raw events; the hour and the day fold up from it. State lives in memory and on local disk for speed, checkpointed to Object storage so a crash resumes without recounting.

3) Rollups serve the fast path; recompute is the slow truth. A finished fifteen-minute rollup is upserted into the Rollup store keyed by its bucket, where the Query service reads it within a second or two for the Dashboard. The fast path is close but not perfect. A duplicate that outran the dedup window, or an event that arrives hours late, can leave a bucket slightly off. So the Batch recompute periodically replays the full raw log from Object storage, counts distinct event ids per bucket exactly, and overwrites the serving rows in place. Both paths write the same bucket key, so the exact number replaces the estimate instead of stacking on top of it. Fresh comes from the stream. Exact comes from the replay.

Back-of-the-envelope

Do the arithmetic, because two of these figures pick the architecture rather than decorate it. Start at the top. Ten million daily users firing about two hundred events each is two billion events a day. Spread over 86,400 seconds that averages roughly twenty-three thousand a second, and usage clumps into working hours, so peak runs near four times that, call it a hundred thousand a second. At a few hundred bytes an event, a hundred thousand a second is about fifty megabytes a second flowing in. Now apply batching. Fifty events to a request drops the ingest tier from a hundred thousand requests a second to about two thousand, which a modest fleet behind a load balancer handles without breaking a sweat. The batching is what makes the write path affordable.

The storage split falls out of the next number. Two billion events at a few hundred bytes is roughly a terabyte of raw events a day, and past a year that is several hundred terabytes. You do not keep that query-ready. Raw events live in cheap object storage for replay and recompute, retained as far back as the exact recount might reach. The rollups are a different animal, and here is the payoff of pre-aggregating. A count is keyed by the metric, the dimensions, the grain, the time bucket, so the number of rollup rows grows with distinct metric combinations and time buckets. Event volume doesn't move it. A dashboard asking for a feature's daily active count over a month reads about thirty rows. The same question against raw events would scan billions. The rollup store exists to close that gap.

• Peak ingest: ~100K events/sec. 10M daily users times ~200 events is ~2B a day, ~23K a second average, bursting ~4x to peak. About 50 MB a second at a few hundred bytes each.
• Request rate: ~2K req/sec. Batching ~50 events per request collapses 100K events a second into ~2K HTTP requests a second, the gap between an affordable ingest tier and an overwhelmed one.
• Raw storage: ~1 TB/day. ~2B events at a few hundred bytes. Hundreds of TB a year, so raw lives in cheap object storage for replay, not in the hot store.
• Rollup reads: ~30 rows, not billions. A month of a daily metric is ~30 precomputed rows. Rollup count grows with metric-and-bucket cardinality, not event volume, which is why pre-aggregation wins.

Technology choices

So far the boxes are categories. In the room you name a concrete tool for each and defend it, because the follow-up never changes: what happens when this piece dies mid-stream? Below, why each kind of tool fits and how the design degrades when it fails.

• Kafka is the ingest queue (Kinesis or Pulsar fill the same slot). A partitioned, replicated log fits because the firehose is bursty and the consumer is not infinitely fast. Events need somewhere durable to wait outside the request path and the processor's memory. Partitioning by user id spreads load and co-locates an event's duplicates on one partition. When a broker dies, a replica is promoted and the consumer resumes from its committed offset, so the cost is a short pause, never a lost or double-counted event.
• The Rollup processor is a stream processor, Flink or Spark Structured Streaming. Both hand you keyed state and checkpointing out of the box, on top of partitioned consumption. On a bare consumer loop you would reinvent the dedup state and get crash recovery wrong, so don't hand-roll it. When an instance dies, the framework restarts the task from its last checkpoint, reloads the dedup and counting state, and rewinds the offsets to match. No bucket is lost and none is counted twice.
• The Dedup store is a fast key-value store keyed by event id, either Redis or an embedded RocksDB colocated with the processor. It answers "seen this id?" in well under a millisecond and expires ids after a bounded window so it does not grow forever. Colocating it with the processor, sharded the same way as the partitions, keeps the check local. If it is lost, recovery replays from the checkpoint, and the batch recompute is the backstop for anything the warm store missed.
• The Rollup store is a columnar or time-series store like ClickHouse or Druid. The read pattern is aggregate lookups keyed by metric and time, range scans for a series included. These engines are built for that while ingesting a steady rollup stream. A row store like Postgres would buckle under the scan volume. Writes are upserts keyed by bucket, so a recompute overwrites in place. If it falls behind on writes, the rollups queue in the log behind it and drain once it recovers, with nothing lost in the meantime.
• Object storage is S3, GCS, or the like, holding the tiered raw log and the processor checkpoints. It is the right home because it is cheap per terabyte and effectively bottomless, and it survives the loss of any single machine, which a local disk does not. A failed checkpoint write just falls back to the previous good one and replays a few more seconds of the log, which is exactly the safety the durable queue was put there to provide.
• The Query service is a thin stateless API in front of the rollup store, scaled out behind a load balancer. It holds no state, so a dying instance sheds its in-flight reads to a sibling and nothing has to recover. It does no aggregation either, since the stream and batch already computed every answer. Its work is fast range reads, with a short read cache in front to absorb a dashboard hammering the same popular metric.

The names are the least of it. What carries is the category and its defense. Put a partitioned replayable queue out front and a stateful processor that dedups and counts in the middle. Behind them sit a fast columnar store for keyed reads and cheap object storage holding the raw log and checkpoints. One separation runs under all of it. The durable plane is the queue and the raw archive, ordered and replayable, true across any failure. The serving plane is the dedup state and the rollups, fast and derived and rebuildable from the log. Say which component makes an event durable, which makes a read fast, and why the recompute can always rebuild the exact count. That line marks a senior answer.

Potential deep dives

The deep dives carry most of the signal. An interviewer picks a piece and pushes until something gives. Three come up on this design more than any others, and each traces back to the same split the design started from: fast and approximate against slow and exact.

1) Idempotent counting: dedupe by event id

Start with where the duplicates come from, because the fix only makes sense once you see the sources. The SDK retries on a timeout, so the same batch can arrive twice. The queue guarantees at-least-once delivery, so a consumer that crashes after processing but before committing its offset re-reads and reprocesses. And the recompute deliberately replays old events. Three independent sources, one symptom: the same event, more than once. Count on arrival and every one of them inflates the number. This deep dive is how to count each real event a single time without slowing the hot path to a crawl.

The gap is narrow but real, and the way to close it is not a bigger dedup window. It is a second path that does not depend on remembering anything. Make the authoritative count a recompute over the raw events, where dedup is a property of the data itself.

Be ready to pin down what the id actually is, because that is the first thing the interviewer tightens. The id is generated on the client and attached before the first send, so every retry of an event carries the identical id. A server-assigned id would be useless here, since the server sees a retry as a fresh arrival and stamps a new one. A hash of the payload is tempting and wrong: two distinct taps with the same properties would collide into one and undercount real actions. There is a cheap win on top of it. Give each batch its own id as well. A flaky-network SDK retries whole batches, and a batch id lets you drop the retry in a single check instead of event by event.

The recent-ids set is also a lot of memory, and that is the next place the questions go. A hundred thousand events a second across a two-day window is on the order of ten billion ids, far too many to hold as raw keys. Two moves shrink it. Sharding the set by partition means each consumer remembers only its own slice, which the user-id partitioning already arranges for free. Then a Bloom filter fronts the exact set: a small probabilistic structure that answers "definitely new" or "possibly seen" in a handful of bits per id. A "definitely new" is the overwhelming common case and skips the backing lookup entirely; only a "possibly seen" pays for a real check. The filter costs a sliver of false-positive rate, harmlessly re-verifying a few genuine events, and buys back an order of magnitude in memory.

The Rippling angle

This is the follow-up Rippling leans on hardest, and it rhymes with their signature coding round. The transactional key-value store they love to hand candidates is the same instinct in miniature: get the semantics exactly right when an operation might be applied more than once. Here the interviewer will not accept "exactly-once" as a word. They will ask what keys the dedup, and the answer is a client-generated event id, stable across retries, not a server timestamp and not a payload hash two distinct events could share. They will ask how long you remember an id, and the answer names a bounded window plus a recompute for anything past it. Then comes the one that separates the levels: an event shows up a day late, after its id has aged out, so how does the count stay right? Having the recompute ready, and being able to say why the fast path alone cannot be trusted with a billed number, is what a senior answer sounds like.

2) The multi-granularity rollup

The product wants counts at the fifteen-minute grain, the hour, the day, and the naive reading is to compute each one on its own. Do not. Two questions decide this deep dive. Do you aggregate ahead of time or at query time? And once you pre-aggregate the fine grain, how do you get the coarse ones without rescanning everything?

Pre-aggregation wins for the dashboard, so the question becomes how to build three grains without paying for three scans. Use a cascade. Compute the fifteen-minute buckets once from the stream. Fold four consecutive fifteen-minute buckets into an hour, and twenty-four hours into a day. Because a count is a sum and sums are associative, folding the fine grain up gives exactly the same answer as counting the coarse grain from scratch, at a tiny fraction of the work. The day grain never touches a raw event. It touches twenty-four numbers.

Distinct counts break the cascade, and this is the detail that shows depth. You cannot add fifteen-minute unique-user counts to get the hourly unique count, because a user active in three of those buckets would be counted three times. Sums fold. Distinct counts do not. The fix is to store a mergeable sketch per bucket, a HyperLogLog, instead of a bare number. Sketches merge without double-counting, so the hour's sketch is the union of its four fifteen-minute sketches, and the answer is a distinct-count estimate with a small, known error. That tradeoff is exactly why getUnique returns an estimate rather than an exact figure.

There is a limit to how far pre-aggregation scales, and it is the dimensions. Every dimension you let a metric be sliced by multiplies the bucket count. Feature by account by platform by country is a product, and materializing every combination at every grain explodes the row count past what it saves. So do not materialize the whole cube. Pre-aggregate the handful of dimension combinations the dashboards actually ask for, the common slices. Answer the rare ad-hoc slice with a slower query-time group-by over recent raw events. That hybrid keeps the rollup store bounded to the queries that matter while still serving the long-tail question, just not instantly. Naming that boundary, and refusing to pre-aggregate everything, is the senior instinct the interviewer is listening for.

The grains also decide what you keep and where, which is the hot-versus-cold question on the serving side. Fine buckets are precise but many, and their value decays. Nobody reads last month at fifteen-minute resolution the way they read it by day. So age them out. Keep the fifteen-minute grain hot for a week or two, fold everything older into hourly and daily buckets, then drop the fine grain past its window. Daily rollups are tiny and live cheaply for years. The raw events sit colder still in object storage, reached only by the recompute. The fast store stays small and quick where it matters, on the recent windows a dashboard actually hits, without losing the long-range history a trend line needs.

3) Scale and late-arriving events

Two pressures test this design under real traffic: raw volume, and events that show up long after they happened. Take volume first, quickly, because the architecture already answers most of it. The queue absorbs bursts, so a spike becomes consumer lag that drains rather than dropped data. The ingest tier is stateless and autoscales on request rate. Consumers scale out to the partition count, so provisioning the partitions widely up front is what allows the fan-out a peak needs. The rollup store shards by time and dimension. None of that is exotic. The interesting pressure is time itself.

Two mechanics inside the volume half are worth saying out loud, since they are the usual follow-ups. Backpressure comes first. When a spike outruns the processor, events do not swell its memory, they wait in the queue as consumer lag, a number you watch and autoscale on. Lag climbs, you add consumers up to the partition count. Lag drains, you scale back down. The queue turns a burst into a bounded delay instead of dropped data. Then skew. Partitioning by user id is even until one key is not, and a single giant account, or a firehose event like a page view, can pin one partition and starve its consumer. The fix is the standard one. Salt the hot key into several sub-keys so its load fans across many tasks, then sum the partial counts downstream, which is exact because a count is associative. A light running-frequency estimate spots the hot key, and you salt only the few that cross a threshold, so the common path pays nothing.

A phone goes into a tunnel at 10:00 and surfaces at 14:00, and it uploads four hours of events with their original timestamps. This is the late-event problem, and it opens with a clock question: which time defines the bucket? Ingest time is easy and wrong. Bucket by when the event arrived and that 10:00 tap lands in the 14:00 window, smearing the metrics for both. Event time is correct and harder. The event carries its own timestamp, so it belongs in the 10:00 bucket, but that bucket was closed and served hours ago.

One last question the recompute invites: how often, and over how much? A full pass over months of raw events every hour would cost far more than it returns. So make it incremental. On a frequent cadence, recompute only the recent partitions that could still be shifting: the last few days the fast path might have gotten slightly wrong. Save a full historical pass for the rare case of a fixed bug in the counting logic. Because the recompute is idempotent and bucket-keyed, running it over a narrow recent window is safe and cheap, and it converges the live estimates onto the exact totals a few hours behind. Fresh from the stream, exact from the replay, and the replay only ever reprocesses what might have moved.