Design a Monitoring and Alerting System

Understanding the problem

"Design a monitoring and alerting system" lands on the whiteboard underspecified. Start by saying what it has to do, before you name a single database. A monitoring platform takes in a firehose of health signals from every service in a fleet. It reduces that firehose into numbers you can reason about over time. It decides when those numbers mean something is wrong, and it tells a human before a customer notices. Four responsibilities, four layers: ingest, aggregate, detect, alert. Keep them separate and the design holds up under every follow-up. Blur them, and the detector fires on half-written windows while the alerting path pages the same incident a thousand times.

The trap is answering with a picture of a dashboard you've used. Resist it. A candidate who can name a monitoring product's panels but can't build the aggregation layer from scratch falls apart fast. The interviewer asks what happens when ten thousand agents report a spike in the same second, and there's no answer. Build the general pipeline first, from ingestion through aggregation and detection to alerting, and reason about correctness and load at every seam. The monitoring domain then adds real constraints. The signal never stops, the interesting moments arrive as bursts, and the system exists to stay quiet until it shouldn't.

The Zscaler angle

A platform in the traffic path lives or dies on its own observability, so this design sits closest to what Zscaler runs. Zscaler operates an inline cloud-security service. Enterprise traffic goes through a globally distributed proxy fleet, gets inspected, and is allowed or blocked before it reaches its destination. A fleet like that emits health signals from every node in every data center. Catching a regional latency creep or an unhealthy enforcement node within seconds keeps the service smooth. Catching it minutes later means an outage a customer feels. So the conversation keeps circling back to scale, tight latency and staying calm under a flood. Clever detection formulas matter less. Say the shape out loud before you optimize anything. Ingest a global stream, window it, evaluate it, and alert once per incident. A fancier anomaly model impresses less than a pipeline that stays up while the fleet it watches has a bad day.

Bounding the requirements first

Spend a minute bounding the problem. A monitoring system sprawls the moment you start naming stores without a scope. State the functional surface plainly. The platform ingests metrics from a large fleet of services. Each metric is a named series with a value and a timestamp, arriving continuously and in bursts. It aggregates samples into fixed time windows, so downstream logic reasons about a rate or average over a minute instead of single points. It evaluates rules against those windows to decide when a series is unhealthy: a static threshold first, a simple anomaly check beyond it. It raises alerts when a rule trips, and it collapses a storm of related breaches into one alert an on-call human can act on. And it never silently loses a signal. Anything malformed or undeliverable lands in a dead-letter path.

Then draw the line between what you build now and what you name and defer. Full anomaly detection is a research field of its own. Deriving a seasonality model or forecast band in the first pass guarantees you run out of time before the skeleton is clean. Model the platform so those pieces have an obvious home, and say so. Ingestion, windowed aggregation, threshold evaluation and deduplicated alerting are the core you build. Richer detection models, the long-term historical store for dashboards, and multi-region federation get named, given a seam, and left for the follow-up. If "swap the static threshold for a learned band later" changes one component, the design is good. If it means rewriting ingestion, it isn't.

• The grading axis: calm under a flood. The interesting moments arrive as bursts, so a design that is fast on average but stalls when the fleet spikes fails. Size the pipeline for the storm. Say what it does when input arrives faster than a consumer can drain it.
• Reduce before you reason: windows, not raw points. A rule that fires on a single sample is noisy and fragile. Fold samples into fixed time windows first, so detection reasons about a rate or an average over an interval. The window is the unit the whole platform thinks in, and everything downstream reads it, never the raw firehose.
• One incident, one alert: dedup is the product. A real fault trips the same rule on thousands of samples across many nodes. An alert per breach buries the on-call human and trains them to ignore the pager. Collapsing related breaches into one alert per incident, with a cooldown, makes the system usable as well as correct.
• Never a silent drop: dead-letter everything. A malformed sample or an alert the channel could not deliver must not vanish. Route it to a dead-letter path where it can be inspected and replayed. For a platform whose job is noticing trouble, losing the signal that trouble happened is the worst possible failure.

The signal model, before the boxes

Start from the types. This round rewards a clean data model far more than a busy diagram. Three nouns carry the whole platform. A Sample is one measurement: a series name, a numeric value and a timestamp, nothing else. A Window is a fixed slice of time for one series. It holds the samples that fell inside it, reduced to a summary: a count, a sum, a max, whatever the rule needs. A Rule is a predicate over a window: a threshold, a minimum breach count, and what to do when it trips. Underline those three and the layers are halfway written.

One decision makes the model correct. A Window is a derived summary keyed by series and interval. It isn't a growing bag of every point you ever saw. A monitoring platform can't keep every raw sample forever and reason over all of them, since the volume is enormous and mostly uninteresting. So aggregation folds each sample into its window the moment it arrives, keeping a bounded summary per interval. That one choice buys two things. Detection reads a small, fixed-size summary instead of scanning a firehose. And memory stays flat however many samples a window receives, which you need when a spike sends a million points into one minute.

// A Sample is one IMMUTABLE measurement: which series, what value, and when.
// It carries no identity beyond the series name. A sample is a fact that
// already happened, so it is folded into a window, never edited in place.
interface Sample {
  series: string   // e.g. "proxy.latency_ms" for one node or region
  value: number    // the measured quantity
  timestamp: number // event time, in milliseconds, from the source clock
}
// A Window is a DERIVED, bounded summary of one series over one fixed interval.
// It folds samples in as they arrive and keeps only the aggregate the rules
// need, so its size never grows with the number of samples it receives.
interface Window {
  series: string
  start: number    // interval start; the window covers [start, start + width)
  count: number    // how many samples landed here
  sum: number      // running total, for an average without keeping points
  max: number      // running max, for a peak-based rule
}
// A Rule is a PREDICATE over a window plus what to do when it trips. Keeping
// the rule separate from the window is what lets the same aggregation feed a
// static threshold today and a learned band later without touching ingestion.
interface Rule {
  series: string
  threshold: number   // the value a breach is measured against
  minBreaches: number // how many breaching samples in the window before it fires
  cooldownMs: number  // how long one alert suppresses re-firing for the same series
}

Defend two details in that model out loud. A Window keeps a running count, sum and max, never the samples themselves. It can answer "what was the average" or "what was the peak" while holding a handful of numbers, which keeps aggregation bounded under a burst. And a Rule owns the breach threshold and the cooldown, but not the window it reads, so deduplication and detection stay separate concerns. Keep those apart and the correctness argument fits in a sentence: fold samples into windows, test windows against rules, and let the rule decide when a breach becomes an alert.

The pipeline end to end

Now lift the signal model into a running system and draw it. The architecture is the seams between the layers and the order data moves through them. Read the diagram by role. Blue is the outside world handing you signal or waiting to be paged. Indigo is a service that transforms. Amber is the durable stream everything hangs off. Green is state the platform owns, including the dead-letter path. Follow the numbered edges for one full path. Health signals land at the ingest gateway, which appends them to the stream. The aggregator folds them into windows, and the evaluator tests those windows against rules. The alert manager deduplicates the breaches and routes one alert to the on-call responder. Anything malformed or undeliverable goes to the dead-letter queue instead of being dropped.

Ingestion and windowing under a flood

Assume the input will spike, because it will. A fleet reports at a steady background rate until something goes wrong. Then a whole region reports trouble in the same second. That burst is the workload the platform exists for, so ingestion has to be built for it. The gateway validates and appends to a durable stream without processing inline. That decouples how fast signal arrives from how fast it's drained. When the aggregator falls behind, the stream absorbs the backlog and the consumer catches up from its offset. A spike costs a little latency and no lost signal.

Ordering matters where a window gets folded. The gateway partitions by series, keeping every sample for one series on one ordered lane while different series flow in parallel. The aggregator folds each sample into the fixed window its timestamp lands in, keeping a running summary. Two realities complicate the fold, and each needs a named answer. Samples arrive slightly late, so a window stays open for a short grace period past its end before it's sealed and evaluated. That catches stragglers without waiting forever. And source clocks skew, so windowing keys off the sample's event time instead of arrival time. A sample too far outside any open window is late data. It gets dead-lettered, never folded into the wrong interval.

Detection: threshold first, anomaly second

The evaluator runs on sealed windows, and the honest starting point is a static threshold. A rule says the series is unhealthy when enough samples in the window breached a fixed value. For example, latency over a limit on at least some minimum count of samples in the minute. That simple rule catches most real faults. It's trivial to reason about and easy for a human to trust, and an alert nobody trusts is an alert nobody acts on. State the threshold rule first and show it working. Reach for something more sophisticated only when the interviewer asks how you'd catch a fault that never crosses a fixed line.

Anomaly detection is the natural second layer. Keep it simple and behind the same seam. The evaluator already has the completed window and the series' recent history from the store. So a lightweight check compares the current window against a rolling baseline. Flag it when it deviates from the recent mean by more than a few standard deviations, or when a rate drops toward zero where it usually isn't. That catches a regression a static threshold would miss, like a latency that doubled but is still under the hard limit. It doesn't pretend to be a forecasting system. Say out loud that both rules read the same window summary and emit the same kind of breach. The alerting path downstream never needs to know which rule tripped.

Alerting: dedup, cooldown, and a dead-letter path

Pull the alerting story together. On this platform, restraint is the design. A breach isn't an alert. One fault trips the same rule on thousands of samples across many nodes in a region. Turn each breach into a page and the monitoring system becomes the thing on-call engineers mute. The alert manager collapses that storm. It fires one alert per series per incident, keyed on the series and the rule. It suppresses re-firing for a cooldown period, so a fault that keeps breaching produces one actionable page. Deduplication is the most important behavior in the alerting path. Here a candidate either states a clean invariant or hand-waves.

Delivery is the other half. It can fail, so it needs the same discipline as ingestion. Say the alert manager routes a page and the channel is down or rejects it. The alert can't just disappear, because the platform exists to preserve the fact that something is wrong. So an undeliverable alert gets retried. If it still can't be delivered, it goes to the dead-letter queue for inspection and replay. The same path catches the malformed sample the gateway couldn't parse upstream. The room wants to hear you say where the signal goes when it can't be processed or delivered, so the platform never goes blind.

• Dedup key: series plus rule, per incident. Breaches are deduplicated on the series and the rule that fired, so all the samples of one fault map to one alert. Key on something finer, like the individual sample or node, and the pager floods again. Key too coarse and a second real fault hides. The key is the incident, and the alerting design depends on getting it right.
• Cooldown: one page, then quiet. After an alert fires for a series, the manager suppresses re-firing for a cooldown window. A fault that breaches every second produces one page instead of sixty a minute. The cooldown turns a storm into a single actionable signal, and it resets only after the series has been quiet.
• Undeliverable alerts: retry, then dead-letter. A page the channel could not accept gets retried and, failing that, routed to the dead-letter queue. Losing the alert that trouble happened is the worst failure a monitoring platform can have. So the delivery path gets the same never-drop guarantee as ingestion.
• Recovery: resolve, do not just stop. An alert should clear when the series returns to health. The manager tracks the active alert and emits a resolve once the breaches have stopped long enough. An alert that fires but never resolves leaves the on-call human guessing whether the fault is still live.

The runnable core: a windowed threshold alerter

The interviewer often compresses the whole pipeline into one class you implement. For a monitoring system, that's the windowed threshold alerter. It folds samples into a trailing window, fires when enough of them breach a threshold, and won't re-fire the same alert during a cooldown. It's the correctness story in miniature. The trailing window is the aggregation layer, the breach count is detection, and the cooldown is deduplication made concrete. The boundary case is a storm of breaches that must produce exactly one alert. There, a candidate either states a clean invariant or hand-waves.

The problem below keeps the windowing and the dedup and drops the anomaly math, since that's what runs. A sample here is a series name, a timestamp and a value. That's enough to define a trailing window (breaches within the last width of time), a fire rule (enough breaches, not in cooldown), and the active set (series currently under an alert). The exercise is handling window eviction and cooldown suppression cleanly, just as a deduplicating cache comes down to the duplicate key.

1

Windowed threshold alerter: fire once per incident

Medium

Problem

A monitoring platform folds a stream of samples into a trailing time window per series and raises an alert when enough samples breach a threshold, but it must fire once per incident, not once per breaching sample. Implement the alerter:

- Alerter(window, threshold, minBreaches, cooldown) creates an alerter. A sample breaches when its value is at least threshold. A series fires when at least minBreaches breaching samples fall within the trailing window of width window. After a series fires, it is suppressed until more than cooldown time has passed since it last fired.
- observe(series, ts, value) folds one sample seen at time ts. Samples arrive in non-decreasing ts order. If value is below threshold it is not a breach and nothing fires, return false. Otherwise record the breach, drop breaches older than the trailing window (a breach at time s stays only if s is greater than ts - window), and if the window now holds at least minBreaches breaches and the series is not within its cooldown, fire: record the fire time and return true. Otherwise return false.
- active(ts) returns the series currently under an alert, meaning those whose last fire was no more than cooldown ago (ts - lastFired is at most cooldown), sorted ascending.
- fired() returns the total number of alerts fired so far.

Window eviction and cooldown suppression are the exercise. A breach that fell out of the trailing window must not count. A storm of breaches within one cooldown must produce exactly one alert, never one per sample.

Input encoding. A test case is a JSON array of operations. The first entry is ["Alerter", [window, threshold, minBreaches, cooldown]]; each later entry is [method, [args]]. The expected array holds each operation's return, with null for the constructor.

Input: Alerter(10,100,3,20); observe("cpu",0,150); observe("cpu",2,150); observe("cpu",4,150); observe("cpu",6,150); fired()

Output: the third observe returns true, the fourth returns false; fired() returns 1

Three breaches at times 0, 2, 4 all fall within the trailing window of width 10, so the third one trips the rule and fires. The fourth breach at time 6 also has three-plus breaches in window, but the series fired at time 4 and 6 minus 4 is within the cooldown of 20, so it is suppressed. One alert, not two.

Input: Alerter(10,100,3,20); observe("net",0,200); observe("net",20,200); observe("net",40,200); fired()

Output: every observe returns false; fired() returns 0

The three breaches are 20 apart, wider than the window of 10, so each new breach evicts the previous one and the window never holds more than one breach at a time. The rule needs three, so nothing ever fires.

How this uses the pattern

Keep, per series, a deque of breaching timestamps and the last fire time. observe drops breaches older than the trailing window. It checks the count against minBreaches and gates a fire on the cooldown. active filters series by how recently they fired; fired is a running total.

Windowed threshold alerter: fire once per incident

Python

Focus Mode

Loading...

OutputSubmissions

Run

Submit

Click "Run" to execute your code

Test Cases

Input: [["Alerter",[10,100,3,20]],["observe",["cpu",0,150]],["observe",["cpu",2,150]],["observe",["cpu",4,150]],["observe",["cpu",6,150]],["fired",[]],["active",[10]],["active",[30]]]Expected: [null,false,false,true,false,1,["cpu"],[]]

Input: [["Alerter",[10,100,3,20]],["observe",["net",0,200]],["observe",["net",20,200]],["observe",["net",40,200]],["fired",[]]]Expected: [null,false,false,false,0]

Input: [["Alerter",[10,100,2,20]],["observe",["mem",0,50]],["observe",["mem",1,50]],["observe",["mem",2,150]],["observe",["mem",3,150]],["fired",[]]]Expected: [null,false,false,false,true,1]

Input: [["Alerter",[10,100,3,20]],["observe",["cpu",0,150]],["observe",["cpu",1,150]],["observe",["cpu",2,150]],["observe",["cpu",3,150]],["observe",["cpu",30,150]],["observe",["cpu",31,150]],["observe",["cpu",32,150]],["fired",[]]]Expected: [null,false,false,true,false,false,false,true,2]

Input: [["Alerter",[10,100,2,20]],["observe",["b",0,150]],["observe",["a",0,150]],["observe",["b",1,150]],["observe",["a",1,150]],["active",[5]],["fired",[]]]Expected: [null,false,false,true,true,["a","b"],2]

Need a hint? (3 available)

The Zscaler angle

This is the correctness core of the monitoring design, reduced to what runs. Zscaler cares that a fault tripping the same rule thousands of times produces one page. An on-call human drowning in duplicate alerts is worse off than one with none. The interviewer watches whether the window eviction and the cooldown suppression survive the test cases. A stale breach must stop counting, and a storm within one cooldown fires once. Pushing to a list proves nothing. State the fire-once-per-incident invariant before you code it. Then drive the boundary cases yourself: breaches spread wider than the window, a storm inside the cooldown, and a re-fire only after the cooldown has fully elapsed.

Take it further

A natural extension puts the platform back around the alerter. First, resolution. Track the active alert and emit a resolve when a series goes quiet long enough, so an alert clears instead of only firing. Then the anomaly layer. Compare the window against a rolling baseline instead of a fixed threshold and breach on a deviation. That means a running mean and variance per series in place of a raw count. Then durability. The in-memory window and cooldown don't survive a restart, so the real alerter rebuilds recent state by replaying the tail of the durable stream. Be ready to say what each costs. Resolution needs a quiet-period timer per series. The anomaly check needs a per-series baseline. Durability turns the alerter into a consumer that can resume from an offset.

Show solution & walkthrough

Backpressure, failure, and scale

A pipeline that works in a demo and buckles when the fleet spikes fails this round, so spend real time on scale and how it breaks. Scale here is mostly about keeping ingestion, aggregation, detection and alerting from dragging each other down. Raw requests per second is the easy part. The durable stream decouples the stages, so each scales on its own axis. Ingestion fans out across partitions keyed by series, and aggregation runs a consumer per partition. Detection and alerting read the far smaller stream of sealed windows and breaches. The log separates them, so a heavy detection pass never slows the fold of a live sample.

Then walk the failure modes out loud. On a live platform they're routine. An agent misbehaves and floods one series. The gateway rate-limits per source, and the partition contains the blast radius so one noisy node can't starve the rest. A consumer lags behind a burst. The stream buffers, applying backpressure, and the consumer resumes from its offset, trading a little latency for keeping live signal. A clock skews. Windowing keys off event time, and wildly out-of-range samples get dead-lettered instead of folded into the wrong minute. The alert channel itself goes down. Undeliverable alerts get retried and then dead-lettered, because the platform must never lose the fact that something was wrong. Name these before the interviewer does, and you read as senior.

• Backpressure: buffer, do not drop. When a consumer falls behind a burst, the durable stream absorbs the backlog and the consumer catches up from its offset. The platform trades a little latency to keep every live signal. That is the right trade, because the burst is when you most need the data.
• Noisy source: rate-limit and contain. A single misbehaving agent can flood one series. The gateway rate-limits per source, and partitioning keeps the flood on its own lane. The blast radius stays at one series, so one broken node cannot blind the platform to the rest of the fleet.
• Clock skew: event time, dead-letter the rest. Source clocks drift, so windows key off the sample event time. A sample too far outside any open window gets dead-lettered instead of folded into the wrong interval. Misattributing signal to the wrong minute is worse than flagging the sample for review.
• Channel outage: retry, then never lose it. The paging channel can be down, so an undeliverable alert is retried and then routed to the dead-letter queue. A monitoring platform that loses the alert about an outage during that outage has failed at its one job. Delivery gets a never-drop guarantee.

The Zscaler angle

Now put the cloud-security skin on. Everything above is a general monitoring pipeline, on purpose. The interviewer wants to see you build one, and reciting a product won't do. The Zscaler flavor shows you understand what the platform is for. The fleet is a globally distributed proxy sitting inline in customer traffic. Signal volume is enormous and the latency budget for noticing trouble is tight. So aggregation reduces before it reasons, and detection runs on windows instead of points. Availability can't lapse, because the monitored system sits in the critical path of every customer request. The pipeline has to stay up while the thing it watches falls over, hence the durable stream, backpressure, and dead-letter guarantees. And a thousand alerts for one regional fault are worse than useless to an on-call team already fighting an incident. That's why deduplication is the centerpiece. Say those things on top of a clean general design, and you sound like someone who could own the platform that watches the platform.

What the round rewards

The bar here is a clean pipeline, correctness under a flood, and restraint in what reaches a human. Raw traffic scale matters less. Whichever monitoring prompt you draw (a metrics platform, a log-analytics pipeline, an alerting service), the moves are the same. Bound the problem and name the four layers before you draw a box, so ingest, aggregate, detect and alert stay separate. Model the types first. Make a window a bounded, derived summary so aggregation stays flat under a spike. Reduce before you reason: detection runs on windows and starts with a static threshold before any anomaly model. Treat deduplication as the product, so one incident becomes one alert with a cooldown. And name the failure modes yourself: a source that floods, a consumer that lags, a clock that skews, a channel that goes down. Give them a dead-letter path so the platform never goes blind. The cloud-security skin then sits naturally on top.