Design a Streaming Aggregation Pipeline

A streaming aggregation pipeline is the thing Databricks sells, so they hand it to you straight. Events arrive in a continuous firehose, a click, a metric sample, a sensor reading, a log line, millions of them a second. The pipeline chews that stream into useful aggregates and serves them while they are still fresh. Think how many events per key in the last minute, the running top-k busiest keys, or a per-minute count a dashboard can read two seconds after the event happened. It never stops; it never sees the whole input at once. And it has to keep producing answers while the data keeps coming.

This is the shape under real-time metrics, fraud scoring, and live dashboards. The trap is picturing a database with a fast loop over it. A stream is not a table you query. It is an unbounded sequence you get one pass at as it flows by. Two facts make it hard. The input never ends, so you cannot wait for "all the data" before answering; you cut the stream into finite windows and emit one result per window. And events arrive late and out of order. A phone was offline, a network hiccuped, and the event that belongs in the 10:00 window shows up at 10:03. Getting the answer right turns on a question the wall clock cannot settle: when is a window truly finished?

Anchor on numbers before drawing anything. Target a peak ingest of roughly one million events per second, each event a small record of a few hundred bytes carrying a key, a value, and a timestamp. The pipeline computes aggregates over one-minute windows keyed by something with high cardinality, say a few million distinct keys live at once. Freshness has to stay tight: a result for a window should be queryable within a couple of seconds of that window closing; the headline tension is freshness against correctness. The live numbers can be approximate and a hair behind, but the official totals the business reports on have to be exact, with every event counted once and only once.