Design a Messaging & Notification System

Understanding the problem

This system has a quiet half and a loud half. The quiet half is a developer wiring up the API: a handful of test sends, a webhook URL, a check that the code arrives. It asks almost nothing of the machine. The loud half is production, and it is where the interview lives. Millions of messages an hour. Carriers that throttle and drop. A two-factor code that is worthless if it lands a minute late, and a customer billed for every send and furious about a double one. The loud half is where one simple-looking API call turns into a hard distributed-systems problem.

Name the hard part before drawing a box. Sending one SMS is a solved problem; a single call to one carrier's API does it. The work is everything around that call. It is accepting a submit and never losing it, even when a worker dies between the accept and the send. It is delivering through a carrier that is rejecting half your traffic this minute. It means refusing to send the same text twice when a retry fires. It means turning away load you can't carry instead of melting, and reporting back to a sender who is already gone. Get those right and the SMS itself is the easy part.

The Twilio angle

A single message submit is the small door into Twilio's entire delivery spine, and the strong candidates walk through it. The literal prompt is one SMS. The same system carries WhatsApp messages, fires voice-call notifications and pushes email, because they share one shape. Accept something from a sender, deliver it through an unreliable downstream, report back what happened. Say that out loud early. Then plant the flag on what makes Twilio's version hard, which is that Twilio does not own the last mile. The carriers do, and they throttle, fail and answer slowly whenever they please. A design that assumes the downstream is reliable has skipped the entire reason Twilio exists as a company.

Functional requirements

Scope to the path that carries the weight and name the rest to set it aside. Six things make the core. A sender submits a message, or a batch, over the API. The system validates it, rate-limits it and accepts it durably. It delivers each message through a carrier, retrying and failing over when the carrier balks. It guarantees at-least-once delivery deduped by an idempotency key, so a real submit always goes and a retry never doubles it. It tracks each message's status as the carrier reports back. And it delivers that status to the sender over a signed webhook. Everything else you name and drop.

Non-functional requirements

These are the requirements that pick the design, so put a number on each. Delivery is near-real-time but not instant, and it is asynchronous. A login code should reach the handset within seconds. Nobody is blocked at a checkout waiting on the API, though, so the submit call can return the moment the message is safely accepted, and the slow work of reaching a carrier happens after. That one fact, that the caller does not wait for delivery, is what separates this from a payments API and what lets the whole pipeline be a queue.

What must not slip is the pair of guarantees that pull against each other. A submitted message must not vanish, so delivery is at-least-once and survives a crash. And a message must not go twice, so the pipeline is idempotent. A dropped two-factor code locks a user out of their account. A doubled marketing text is a compliance incident and a second charge on the bill. Holding both at once, never lost and never duplicated, across carriers you do not control, is the spine of the whole design.

Now the shape of the load. Assume hundreds of thousands of sending accounts and, at peak, on the order of tens of thousands of submits a second as two-factor surges and marketing campaigns overlap. The raw submit rate is not the scary number, though. Two things are. A single broadcast can expand from one API call into millions of messages in a burst. And every one of those waits on a carrier that takes hundreds of milliseconds to seconds to accept it, then returns its delivery receipt seconds to minutes later. The system is sized by the in-flight backlog those two facts create, not by the submits per second at the front door.

• Peak submit rate: ~tens of thousands/sec. Two-factor surges and marketing campaigns overlap and burst. Each submit is a cheap write plus an enqueue, so the accept tier scales flat behind a load balancer.
• Broadcast fan-out: 1 call to millions. A single send-to-my-list request expands server-side into one message per recipient. This amplification, not the request rate, is the burst the queue has to absorb.
• Carrier path: async, seconds to accept. A carrier takes hundreds of ms to seconds to accept, then returns a delivery receipt seconds to minutes later. The result comes back on its own clock and out of order.
• Delivery guarantee: at-least-once, deduped. A real submit always reaches a carrier and a retry never doubles the send. Near-real-time is the bar, not milliseconds, which is what lets the pipeline be asynchronous.

The set-up

Core entities

Name the objects a message is built from before drawing anything. Four carry the design, and the one that quietly does the most work is the token bucket, because it runs on every submit before anything durable happens.

The whole design leans on one line. A submit is a promise to accept the message durably and deliver it reliably soon, which is easier to guarantee than sending right now. Splitting accept from deliver is what lets the front door stay fast under a burst while the back of the system grinds through carriers at their own pace. Every design choice below follows from that split.

The stores fall out of the work each part does. The durable home of the messages is a wide-column store like Cassandra, keyed by message id and indexed by account, because the reads are point lookups by id and by account, never joins. The rate-limit buckets and the idempotency keys live in Redis, because they are tiny, hot and touched on every single submit. The message queue and the webhook queue both ride Kafka, the backbone a Java and AWS shop like Twilio already runs. The Redis, Kafka and Cassandra lessons cover why each store fits its job.

API

The surface a sender touches is small, and it keeps the fast accept path apart from the slow delivery. Submitting a message is a write that returns immediately. Reading a message status is a plain lookup. The carrier's delivery receipts are not a public endpoint at all; they arrive as a stream the status service consumes. And the status callback is an outbound POST the sender receives, signed so they can trust it.

Two things on that surface carry the design. The idempotency key on the submit makes at-least-once safe end to end. Every hop after it can retry, and one submit must still cost the customer one text. The other is the signed status callback. It carries the delivery result to a sender who left milliseconds after asking. The signature tells them the result is real and not a forged POST to their public URL.

High-level design

Walk one message through the system, the simplest correct version first. The diagram colors each box by role. The sender and the external carrier gateways are blue. The services are violet: the ingest API, the sender workers, the status service and the webhook dispatcher. The message queue and the webhook queue on Kafka are amber. The message store and the rate-limit buckets are green.

1) A sender submits. The sender calls the ingest API with a destination, a body and an idempotency key. It takes a token from the account's rate-limit bucket, claims the idempotency key as it writes the message down as accepted in the message store, then enqueues the job on the message queue. Then it answers 202. The caller is done in milliseconds and nothing has been sent yet, by design.

2) A worker delivers. A sender worker pulls the job off the message queue, picks a carrier route for that destination and hands the message to the carrier gateway. It records the attempt in the message store. If the carrier accepts, the status moves to sent. If the carrier throttles or times out, the worker backs off and retries, and after a few misses it fails the message over to a backup route.

3) The carrier reports back. Delivery is asynchronous, so the answer comes later and on its own schedule. The carrier returns a delivery receipt, delivered or failed with a reason, and the status service consumes it. The service updates the message's status in the message store and puts a status event on the webhook queue.

4) The sender hears the result. The webhook dispatcher pulls each status event, signs it with the account's secret and POSTs it to the sender's callback URL. Webhook delivery is itself at-least-once, so it retries a callback endpoint that is down and the sender dedups on the message id. The loop closes. The sender submitted a message and learned, a beat later, that it reached the handset.

Size the system

Work the arithmetic, because it sets the whole story. Start with the accept path. Hundreds of thousands of accounts submit at a peak of tens of thousands a second, and each submit is a token check, a key claim and an enqueue. That is cheap and stateless, so the ingest tier is a flat fleet behind a load balancer, sized to the submit rate and nothing more.

The number that sizes the fleet is downstream. A worker that sends a message does not get an instant answer; it waits on a carrier for hundreds of milliseconds, sometimes seconds, and the delivery receipt trails by seconds to minutes more. So each in-flight message holds a worker slot far longer than the submit took to accept. A broadcast that dumps millions of messages onto the queue at once builds a deep backlog. Size the worker fleet and the queue for that backlog and that carrier latency, because a spike here is one API call becoming millions of slow, blocking sends.

• Retained messages: billions of records. Every message keeps a status record with its history, retained for the sender to query and for billing. Point reads and writes by id, which is what a wide-column store is built for.
• In-flight per worker: carrier-latency bound. A send blocks on a carrier for hundreds of ms to seconds, so the worker slot is held far longer than the submit was. Queue depth, not submit rate, sets the fleet size.
• Backlog spike: 1 broadcast to millions. One list send dumps millions of jobs on the queue at once. The durable queue absorbs the burst so the accept path never blocks on the delivery path behind it.
• Load shed: 429 past the ceiling. Past a per-account quota or the global limit, the system rejects fast rather than growing an unbounded queue that hides the overload and misses every deadline.

Technology choices

Each box is a real category of tool, and Twilio is a Java and Spring shop on AWS, so the picks below are the ones the interviewer is listening for. Name a concrete tool, say why that kind of tool fits the job, and keep the failure question in your pocket.

• Kafka carries the message queue and the webhook queue. Partition the message queue by account so one customer's two-million-recipient blast rides its own partitions and cannot starve a bank's login codes stuck behind it. Because the log is durable and consumption is at-least-once, a worker that dies mid-send resumes from its last committed offset and redelivers, which is safe only because the sink dedups. The Kafka and async processing lessons cover the queue guarantees under this.
• Redis holds the rate-limit buckets and the idempotency keys. Both are hot-path, point-access state that every submit touches, so they belong in memory rather than in the durable store. A submit is one atomic pair of operations: take a token or reject, then claim the dedup key or skip. The handling contention lesson covers the atomic claim that makes both safe under concurrency.
• Cassandra is the durable home of the messages, keyed by message id for the status read and indexed by account for the sender's history view. The workload is billions of small records, written once and updated a few times as the status moves. They're read by point lookup and never joined, the shape a wide-column store serves and a relational one strains under.
• The ingest API, the sender workers and the status services are stateless Spring services, scaled by adding instances and consumers as load grows. Every piece of state lives in the datastores, so any instance handles any submit or any job, and a box dying costs only its in-flight work, which redelivery replaces.
• The carriers are external and unreliable, reached over SMPP or an HTTP API, and they set the terms: they throttle, they fail, they answer slowly. The delivery half treats every send as best-effort and routes around a bad carrier rather than trusting any single one. That single fact, more than any traffic number, is what shapes the retries, the failover and the honest at-least-once guarantee.

Potential deep dives

The deep dives carry most of the grading. The interviewer takes one property of the system and pushes on it until it bends, and for a messaging platform there are four places it bends. Each is a real Twilio interview prompt in its own right, from the SMS-delivery design to the blunt rate-limit question about ten thousand requests a second.

1) At-least-once delivery and idempotency

This is the crux, and the guarantee the whole platform is judged on. A submit can be retried when the sender's connection blips. The queue can redeliver a job after a worker crash. A worker can wake up unsure whether its last send went out. Every one of those wants to send again, and sending again means the customer's user gets a second login code for one request. Two failures pull against each other here: losing the message, and sending it twice. Both are unacceptable. The design has to shut out both at once.

The handling contention lesson covers the atomic claim the sink uses to turn a duplicate send into a no-op, the same primitive the rate limiter leans on below.

2) Retries, failover and the dead-letter queue

A worker hands a message to a carrier, and the carrier is the least reliable box in the design. It throttles you when you push too fast. It returns a 500 that means try again and a 400 that means it never will. It goes down in one country while the rest of the world is fine. The delivery half has to keep trying without hammering a carrier that is already struggling, and without letting one bad destination block every other message behind it.

• Back off with jitter. Space each retry further out and randomize it. Otherwise a fleet of workers reacting to the same carrier blip retries in one synchronized wave and knocks the carrier back down the instant it recovers.
• Classify the error before retrying. A carrier's transient 500 or a rate-limit response is worth another attempt; a permanent rejection like an invalid number or a blocked-content code is not. Retrying a permanent failure wastes the queue and delays real traffic, so a permanent error goes straight to failed.
• Fail over to a backup route. Most destinations are reachable through more than one carrier, so after a few failed attempts on the primary, re-route the message to a secondary. This is the difference between a carrier outage being your problem or the customer's.
• Circuit-break a dead carrier. When a carrier's error rate crosses a threshold, trip a breaker and stop sending to it for a cool-down, routing everything to alternates. Occasional probe sends test when it is healthy again. It stops the fleet from pouring millions of doomed messages into a carrier that is simply down.
• Dead-letter what exhausts its retries. A message that has failed every route and every attempt is parked in a dead-letter queue with its error trail, not dropped. It can be inspected, replayed after a fix, or surfaced to the sender as a hard failure through the status webhook.

The Twilio angle

Twilio's product is reliable delivery across carriers it does not own, so the routing logic in this deep dive is the business itself. Anyone can call one carrier's API. Customers pay a communications platform for the layer that notices a carrier throttling in one country. It fails their messages over to a second route and trips a breaker on a gateway that has gone dark, without the customer writing a line of it. Say that in the room. Name the error classification, the per-carrier circuit breaker and the failover route, and resist the shortcut of treating the carrier as a reliable pipe. On this system the carrier is the unreliable part, and most of the engineering goes into working around it.

3) Rate limiting and shedding load

Twilio asks this one almost verbatim: design a service that takes ten thousand requests a second and rejects everything past that. Behind the blunt version are two real jobs. One is fairness, because each account has a plan that allows so many messages a second and one customer's campaign must not starve everyone else. The other is self-protection, because the platform as a whole can only push so much to carriers, so past that ceiling it has to turn work away on purpose rather than fall over. Both are the same primitive, a rate limiter, applied at two different scopes.

4) Delivery status and the status webhook

The sender is long gone by the time the carrier says whether the message landed. The submit returned 202 in milliseconds; the delivery receipt shows up seconds to minutes later, on a different connection entirely. The status webhook is how that late answer gets back to the sender's application. Three problems stack up: reliability, ordering and trust. Take them one at a time.

• Deliver the callback at-least-once. Put every status event on a durable webhook queue, POST it to the sender's URL, and retry a failing endpoint with backoff before dead-lettering it. The sender's server can be down for a deploy; the delivery result must survive that and arrive when they come back.
• Make the sender dedupe. At-least-once means the same status can arrive more than once, so stamp each callback with the message id and the status and have the sender treat a repeat as a no-op. The same idempotency discipline that protects the send path protects the sender consuming your callbacks.
• Do not promise an ordering you cannot keep. Carrier receipts arrive out of order, so a delivered can land before the sent that logically preceded it. Stamp each status with the carrier's timestamp or a sequence number and keep only the latest, so a stale earlier status never overwrites a newer one in your store or theirs.
• Sign every callback. The sender's URL is public, so anyone could POST a fake delivered to it. Sign each callback with a secret shared with the account, an HMAC over the payload, so the sender can verify it came from you and reject a forgery. Without the signature the status stream is trivially spoofable.

The Twilio angle

For a communications platform the status webhook is a product the customer builds on. The sender's own retry logic, their fraud checks, their delivery dashboards and their per-message billing all run off these callbacks, so the callback contract carries as much weight as the text itself. That raises the bar on it. It has to be at-least-once so a status is never lost, and signed so it can't be forged. Its error code must be specific enough for the customer's code to tell a bad number from a carrier outage. A candidate who designs the send path with care and waves at the status path has designed half the system. The delivery result is the half the customer's engineers integrate against.

What is expected at each level

That is a lot of surface for one prompt, and how much of it you have to land scales with the level. Twilio gates this round: lighter for an early engineer, a real bar from the senior band up, a genuine grilling at staff. One Twilio note before the tiers. Several Twilio design rounds lean low-level, asking for the API contract and message schema as much as the boxes. State the submit API and the message model precisely here, the way the API-design round rewards.

Software Engineer 2

Breadth carries it. Get to a correct spine: a sender submits over the API, the message lands on a durable queue, a worker delivers it to a carrier, and a status flows back. Say why the submit cannot send inline, because a carrier is slow and a burst is large, so the queue absorbs both. Know that delivery is at-least-once and needs an idempotency key so one submit is not sent twice. The rate limiter and the carrier failover are the interviewer's to raise. The bar is a clean end-to-end path and the instinct that delivery, not the API call, is the hard part.

Software Engineer 3 and Senior

Cover the breadth, then own idempotency and delivery. You are expected to reach the durable-queue-plus-idempotent-claim design without prompting, and to explain why exactly-once through a carrier is off the table. Build the retries with backoff, a dead-letter queue and failover to a backup carrier. Build the rate limiter as a token bucket that sheds with a 429 rather than an unbounded buffer. Treat a double-sent text as a real failure. The bar is a design that stays correct under a carrier outage and a traffic spike, with each store justified as a deliberate pick.

Staff and above

Depth everywhere the system bends, and a widening frame. Lead on the carrier layer: the failover routing, the per-carrier circuit breaker and the error classification that decides what is even worth a retry. Design the sharded rate limiter with a global shed, and the status webhook with its ordering, its signing and its at-least-once contract. Reason about the consistency between the fast Redis state and the durable message store, and how it is kept from drifting. Then widen the frame: the same spine carries voice, email and push, so design the delivery and status platform once, not a one-off for SMS. The bar is that reliability, no double-send and overload all stay central at once while you drive and the interviewer redirects.