Design a Rate Limiter

Understanding the problem

"Design a rate limiter" sounds small and turns into a real distributed-systems problem, which is why it tops the list of Microsoft design prompts. The job states easily. For every incoming request, decide in a millisecond or two: allow or reject. On a reject, return HTTP 429 with a Retry-After header so a well-behaved caller knows to back off. The limiter protects the API from overload and keeps one client's burst from starving everyone else. The hard part is the word "distributed." The decision runs across a fleet of API servers, so the count of what a client has spent can't sit on any one of them. It has to be shared, and sharing a fast-changing counter across a fleet is where this gets interesting.

Pin the goal before drawing anything. Abuse detection, authentication, and billing all live in other layers. What this system does is smooth load. A client gets a steady rate of requests plus a little room to burst. Past that we shed their excess, so the shared API stays healthy for everyone behind it. The limiter has to be far cheaper than the work it guards, since it runs on the hot path of every single call. And it has to honor the limit across the whole fleet even though no single server sees all the traffic.

The Microsoft angle

This is the top design ask at Microsoft, and it usually arrives in Azure framing. The version you hear is "design throttling for an Azure API" or "rate-limit a service behind API Management." The mechanics underneath are the generic ones every interviewer knows: a counter in Redis bucket store checked at the API gateway. Microsoft gates this round at Senior (L63+) and in Azure orgs, and it moves your level more than your hire/no-hire. They reach for this prompt constantly. Over-prepare it.

Functional requirements

Keep the scope tight. The whole round turns on getting one decision right under load, so breadth buys little. The core is a per-request allow-or-reject decision, enforced per client across the whole fleet, with a clear 429 and a Retry-After when we reject. Controlled bursts are in scope, since real API traffic is spiky and a rigid limit would punish honest callers. Everything else sits below the line.

Non-functional requirements

These requirements decide the design, so reason through each one with a target. Latency first. The limiter sits on every request, so its own check has to cost a millisecond or two at most. A slow limiter is itself an outage. Correctness is next, and it is subtle. A perfect count is not the goal. Admitting a handful of requests over the limit during a burst is fine. What sinks the design is the limit failing wholesale, the fleet collectively admitting many times the intended rate because each server counted on its own. The bar is "globally close to the limit," not "exactly the limit."

Availability carries a real tension worth flagging now. The limiter depends on shared state. When that state goes unreachable, the textbook options are both bad. Fail open and allow the traffic, and you risk an overloaded API. Fail closed and reject it, and you risk the limiter taking down the very API it protects. The last deep dive comes back to this, because the answer is "neither, exactly." Fairness is the other first-class requirement, and it bites in practice. One large client running a spike must not eat all the shared capacity. If it does, smaller clients see 429s on traffic they had every right to send.

• ~1-2 ms Limiter budget. It runs on every request.
• Per client key Isolation unit. One bucket per API key.
• Global Enforcement. Across the whole server fleet.
• 429 + Retry-After Reject response. A retryable signal to back off.

The set-up

Core entities

Name the handful of objects the system moves around before drawing anything. Three carry the design:

The store under the bucket state is Redis, and the choice is load-bearing, so be ready to defend it. We need one shared counter per client that every API server can read and update in well under a millisecond. Across the fleet that happens hundreds of thousands of times a second. A relational database is out. A disk-backed store with a row lock per client would be slower than the work it guards, which defeats the point. Redis keeps the bucket in memory, keyed by client id, so a check is a single round trip to a fast in-memory store. And a bucket is tiny and expendable in a way a real record never is. Lose one on a node failure and the client briefly gets a fresh full bucket, a harmless over-admission rather than lost data. We shard the keyspace across Redis nodes by client id once one node is not enough, since a bucket is only ever touched under its own client key anyway. On Azure this is the same shape as Azure Cache for Redis; on the diagram it stays "Redis" because that is the name an interviewer recognizes.

API

The limiter is a check the gateway runs against every incoming request before doing real work, plus a header on the response. So the contract has two halves: what the limiter exposes internally, and what the caller sees on a reject.

The cost parameter is worth noticing. Requests are not all equal, so an expensive call can pull more than one token from the bucket. That lets the same limiter throttle by the work a request does, not only by how many requests arrive. On the wire there are two outcomes that matter: a served request, or a 429 carrying Retry-After. That header is the contract with the caller. A good client waits the named seconds and resends, so a 429 costs it a short delay instead of a hard failure. The deep dives build the limiter that produces it.

High-level design

Walk the request path one piece at a time, and build the simplest version that works before hardening it. The shape is small. The gateway runs the same limiter check against one shared store before any request reaches the service behind it. In the diagram, color marks each component's role. Blue is the Client. Violet is the API gateway and the Limiter check running inside it. Green is the shared Redis bucket store. Amber is the 429 + Retry-After response that turns a rejected caller back. The walkthrough tags each component the same way.

1) Receive and identify the client. The Client sends a request with its API key, and the API gateway resolves that key to a client id. This is the unit of isolation: every client gets their own bucket, so one client's traffic can never spend another's allowance. The gateway does no real work yet, and it does not forward anything until the check clears. Skip the gate and you let load pile up unchecked, which is exactly how one busy client melts the shared API for everyone, so we gate first.

2) Run the limiter check. The Limiter check inside the gateway takes the client id and the request's cost, usually one token, and asks the shared store whether a token is available. This is the entire hot-path decision, and it has to be a single fast round trip. Anything slower than the work it guards is a tax on every call. One thing we punt on here: the check reads and updates a shared counter. If two gateway nodes run it against the same bucket at the same instant without care, both can see a free token and both admit. Making that update atomic is the core of the design and the first deep dive.

3) Check and decrement the bucket. Against the Redis bucket store, the check does three things as one indivisible step. First it refills the bucket by how much time has passed since the last refill (rate times elapsed seconds, capped at the bucket size). Then, if there is at least one token, it subtracts the request's cost and reports allowed, and otherwise reports denied. The bucket is just a count and a timestamp, so this is cheap. Running all three steps atomically inside Redis is what stops two nodes from racing. Pull the read into the app and write back separately and the race reopens. The punt here is fairness under stress. Per-client buckets isolate clients from each other, yet a hot fleet still needs to protect its most important traffic, which is deep dive three.

4) Allow, or reject with Retry-After. If the bucket had a token, the gateway forwards the request on to the service, and the client's remaining count drops by one. If the bucket was empty, the API gateway returns the 429 + Retry-After response immediately, naming how many seconds until the bucket refills enough for one token. The caller waits that long and resends, so the 429 costs a short delay and never a dropped request. The last thing to handle is what happens when the shared store itself is slow or down. The limiter must not become the outage, and that is the final deep dive.

Potential deep dives

Most of the interview signal lives here. The interviewer picks one piece and pushes hard, because the distributed, under-load behavior is the entire point of this round. Below are the ones that come up most on this prompt, each graded against the tiers an interviewer scores you on.

1) Which algorithm, and why token bucket?

API traffic is bursty but bounded. A client sits quiet, then fires a cluster of calls when a job runs or a page loads, then goes quiet again. A good limiter has to absorb that burst up to a point and then hold a steady sustained rate. The algorithm we pick decides how well it does both. Work up through the options, because the popular simple one has a sharp edge.

The Redis lesson covers the in-memory counters and atomic operations these algorithms lean on.

The Microsoft angle

The fixed-window boundary burst is the classic follow-up here, so don't get caught by it. Pick token bucket and say why it fits an API in particular. Traffic is bursty but bounded. The bucket absorbs the burst, the refill caps the sustained rate, and the state is a couple of bytes per client. Then name the trap yourself: a fixed window leaks twice the limit at the boundary, so the number you advertise isn't the number you enforce. Flag that on your own and you read like someone who has run a real throttle in production.

2) The limiter runs on 100 gateway nodes. How do we not over-admit?

This is the deep dive the design exists for. The decision runs on a fleet, call it 100 gateway nodes, and they all enforce the same client's limit. The question is how they share the count without either drifting wildly over the limit or racing each other into admitting too much. The arithmetic makes the failure concrete: if each of 100 nodes keeps its own local allowance, the fleet collectively admits 100 times the intended rate. Work up through it, because the obvious shared-counter fix still has a race hiding in it.

-- KEYS[1] = bucket key for this client
-- ARGV: capacity, refill_rate (tokens/sec), now (sec), cost
local capacity    = tonumber(ARGV[1])
local refill_rate = tonumber(ARGV[2])
local now         = tonumber(ARGV[3])
local cost        = tonumber(ARGV[4])
local state  = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local tokens = tonumber(state[1])
local ts     = tonumber(state[2])
-- first time we see this client: start with a full bucket
if tokens == nil then
  tokens = capacity
  ts     = now
end
-- lazy refill: add rate * elapsed, capped at capacity
local elapsed = math.max(0, now - ts)
tokens = math.min(capacity, tokens + elapsed * refill_rate)
local allowed = 0
if tokens >= cost then
  tokens  = tokens - cost      -- take the token(s)
  allowed = 1
end
redis.call('HMSET', KEYS[1], 'tokens', tokens, 'ts', now)
-- let the bucket expire if untouched, so idle clients cost no memory
redis.call('EXPIRE', KEYS[1], math.ceil(capacity / refill_rate) + 1)
-- retry_after = seconds until one token refills (0 when allowed)
local retry_after = 0
if allowed == 0 then
  retry_after = (cost - tokens) / refill_rate
end
return { allowed, retry_after }

The handling contention material and the Redis lesson cover atomic server-side operations and why a read-modify-write across the network races where a single scripted operation does not.

The Microsoft angle

The atomic check is the difference between a limiter that works and one that quietly lies about its limit. The interviewer's favorite escalation on this prompt is "two of your nodes hit the same client's bucket at the same microsecond, walk me through it." A candidate who keeps the read and the write in application code walks straight into the over-admission race. One who names the Lua script (or GCRA) as the fix has internalized that a shared counter and an atomic counter are different animals. Most distributed-limiter bugs live right there, at the seam where a network round trip splits a check from its action. Reason cleanly through that race and the rest of the design falls into place.

3) Fairness across client tiers, and graceful degradation under fleet stress

Per-client buckets already buy the base fairness: one client's burst spends only their own tokens, so a single spiking client cannot drain a small client's allowance. But two harder questions sit on top. Clients are not equal, so a large paying account should get a bigger bucket than a free-tier one. And when the whole fleet is hot, not just one client, isolation alone is not enough. Every client can honor their own limit and the total can still add up to more than the API can serve. We need to protect the most important traffic when the system as a whole is under stress.

Tiering is the easy half. The bucket size and refill rate live in the LimitPolicy, looked up per client, so a bigger tier simply gets a larger N and a higher r. Tuning a row of config does it. No code moves, and the mechanism stays uniform. Everyone runs the same token-bucket script, only the numbers differ. The harder half is fleet-wide stress, and the answer layers more limiters on the request rate limiter we already built.

The Microsoft angle

Raise fairness and graceful degradation yourself, before anyone asks. Azure runs giant enterprise tenants and tiny developers on the same API surface, so "one tenant's spike must not 429 everyone else" is real operational ground. Name the layers: a request rate limiter, a concurrent request limiter, a fleet shedder, a worker shedder. Then explain that the shedders favor critical traffic when the fleet is hot. Treat degradation as the design's job, not an edge case bolted on at the end, and you sound like someone who has operated one of these.

4) What happens when Redis itself is the bottleneck or down?

The limiter now depends on Redis on the hot path of every request, which means Redis is a single point that can take the whole API down with it. The interviewer will push here, because a limiter that becomes the outage is worse than no limiter. There are two questions: what we do when Redis is unreachable, and how we keep one Redis node from becoming a bottleneck as the fleet grows.

Start with the unreachable case, because the obvious framing is a false binary. Failing open means allow all traffic when Redis is down, which risks overloading the API exactly when it may already be unhealthy. Failing closed means reject all traffic, which turns a Redis blip into a full API outage, the limiter killing the thing it protects. Neither extreme survives contact with production. A better answer is a local approximate fallback that degrades gracefully instead of picking one of the two cliffs.

The Redis lesson covers replication, failover, and why an in-memory store is the right fit for hot, expendable counters like these.

The Microsoft angle

Fail open and fail closed are both wrong, so refuse the binary. Treat the limiter as infrastructure that stays up even when its dependency wobbles. That's why the local fallback and the "cheaper than the work it guards" rule matter. Reach for a degraded mode instead of either cliff, then name Redis as a dependency to harden with sharding and replicas. Candidates who treat the store as a magic always-up box are the ones who get burned. An Azure-org interviewer is watching for exactly this: do you harden the thing your hot path leans on, or assume it never breaks.

What is expected at each level

Microsoft gates the distributed design round at Senior (L63+) and in Azure orgs, so the depth expected scales hard with level. Here is how much you need to land, by the level you are interviewing for.

SDE II (L61/L62)

Mostly breadth, and at this level a full distributed round is not guaranteed. Get to a working single-node limiter: a token bucket per client, a request takes a token, an empty bucket returns 429 with Retry-After. You should be able to explain why token bucket suits bursty API traffic and contrast it with a fixed window's boundary problem. The interviewer drives the distributed and failure parts. The bar: a clean account of one token bucket producing a correct allow-or-reject and a sensible 429, even if the multi-node and Redis-outage cases need prompting.

Senior (L63/L64)

This is where the design round is really gated, so breadth then real depth on the distributed core. You are expected to reach the shared-state answer on your own. That means the bucket in Redis, the over-admission race when two nodes read-modify-write the same bucket, and the atomic Lua script that closes it. You should pick token bucket and defend it against the alternatives, and add per-client buckets sized by tier for fairness. The bar: you find the over-admission race yourself and resolve it with an atomic check-and-decrement. You justify Redis as a deliberate choice for a hot, expendable counter rather than reaching for a database.

Principal (L65+)

Depth across the hard parts and a widening scope. Expect to lead on the layered limiter stack and on load shedding by request criticality, so the writes that matter survive a fleet-wide spike. Expect the fail-open-versus-fail-closed tension and the local approximate fallback that resolves it. Expect Redis as a dependency to harden through sharding and replicas. And expect the rule that the limiter must be cheaper than the work it guards. You raise fairness and degradation before anyone asks, and you keep driving while the interviewer redirects rather than rescues. The bar: surviving a Redis outage and protecting critical traffic under stress sit at the center of your design, never tacked on as edge cases. And you reason about the fleet's behavior end to end.