Design a Rate Limiter & API Gateway

Understanding the problem

IBM reaches for this design once a candidate is senior enough to get a system-design round at all, and the choice is deliberate. IBM's world is hybrid cloud and microservices. An enterprise runs a few hundred services across its own data center and a public cloud, with an API gateway as the front door. The rate limiter is the part of that gateway that decides who gets through and who waits. Nothing here runs at consumer-firehose scale. Strip it down and you have a counter per client that must stay correct while a fleet of gateway replicas hits it at once, and must stay up when something underneath it fails. IBM cares far more about how gracefully the design degrades than about how many billions of requests it can survive.

The round usually starts small, a limiter for one service, and then grows under you, with something like two hundred services behind the gateway, then twenty gateway replicas, then Redis going down. At some point expect the interviewer to drop from the architecture straight into code and ask for the token-bucket check, or the class that holds the counter. That drop from high-level design to low-level implementation inside one question is common here, so know the refill math well enough to write it cold.

The IBM angle

IBM's business is hybrid cloud and enterprise integration, so a rate limiter behind an API gateway is home turf, and the interviewer knows the failure modes cold. They want to know what your gateway does when one of the two hundred services behind it starts timing out, or when the shared counter it depends on goes briefly unreachable. A heroic requests-per-second number barely registers. Candidates who have run OpenShift or a service mesh in production tend to reach for the resilience patterns by name, and that fluency reads as real experience. Operated a gateway? Say so. And treat fail-open versus fail-closed as a choice you made on purpose, with the reason ready.

Functional requirements

Keep the core small and let the interviewer pull more out of you. A client calls the gateway with an API key. The gateway resolves which tier that key belongs to, checks whether the client has budget left, and either forwards the request to the right service or rejects it. A rejection is an HTTP 429 with a Retry-After header telling the caller how long to wait. Two requirements carry the weight. The limit holds even though many gateway replicas serve the same client. And the rejection is honest: Retry-After reflects when a token actually frees up.

Non-functional requirements

Put a number on each target. A rate limiter is judged on a few concrete promises, and added latency comes first. The check sits on the hot path of every request, so the budget is one to two milliseconds at p99, which by itself rules out heavy work per call. Correctness across the fleet is next. A client that bought, say, one hundred requests a second should see one hundred. One hundred times the replica count is a bug. Availability is where the design earns its grade. The limiter must never be what takes the gateway down. If the shared counter is slow or unreachable, the gateway keeps serving under a fallback you picked on purpose, and it does not fail every request.

One piece of slack is worth pocketing now. The count can be slightly approximate. An enterprise client on a hundred-a-second plan will not open a ticket because the gateway let through a hundred and three during a one-second spike, or held it to ninety-eight. That few percent pays off later. It's what lets you keep a network round-trip off the hot path, with each replica serving from a small local reservation and reconciling with the shared counter in the background.

• ~50K/s aggregate request rate. Peak traffic across all clients through the gateway; enterprise scale, not a consumer firehose. Illustrative teaching figure.
• ~20 gateway replicas. The gateway runs as a horizontal fleet behind a load balancer, so any replica can serve any client, which is exactly why the counter has to be shared.
• 1-2 ms added latency budget (p99). The check runs on every request, so it has to be near free; this budget is what rules out a network hop per call.
• 429 + Retry-After the rejection contract. An over-limit request gets a 429 and a Retry-After the caller can trust, so a well-behaved client backs off instead of hammering.

Where the cost falls

Sketch the load before the architecture. One asymmetry shapes the whole design. Admitting a request is cheap. Doing it correctly across twenty replicas, on the hot path, without a round-trip per call, is not. The figures are illustrative, so derive them out loud. A good interviewer redoes the arithmetic with you. Take a silver-tier client on six thousand requests a minute. Six thousand divided by sixty is one hundred requests a second. Now picture the gateway as twenty replicas behind a load balancer that sprays that client's calls evenly. If each replica keeps its own local counter set to the full hundred a second, the client can actually push twenty times a hundred, two thousand a second. Every replica thinks it is fine. The limit is only real if the twenty replicas share one view of the count. That single fact forces most of the design.

• The check: on every request. the limit decision runs inline before routing, up to ~50,000 times a second in aggregate, so it lives on a one-to-two millisecond budget and cannot afford heavy work per call.
• The shared count: one view across replicas. twenty replicas each holding a local full-limit counter overshoots by 20x. The count must be shared, and that sharing is where the round-trip cost and the hot-key risk live.
• The limiter down: must not sink the gateway. the counter store can blip. If the gateway fails every request when it does, the limiter has become a single point of failure for two hundred services at once.
• A downstream down: one bad service, not all. one of the ~200 backends can hang. Without isolation, its slow calls pile up and consume every gateway thread, so a single sick service starves the other 199.

Two lines in that grid set the agenda. Sharing the count makes the limit correct, and it also tempts you into a network round-trip per request that blows the latency budget. Keeping the limiter and the downstreams from taking the gateway down is the enterprise-resilience half. IBM leans on that half hardest. So the design has to hold three properties at once. The check stays cheap. The count is shared but not perfectly exact. And a failure anywhere underneath degrades to something sensible instead of a cascade.

The set-up

Core entities

List the moving parts before you draw an arrow. There are only three. The whole limiter is a little state per client plus the policy that governs it.

The interviewer will ask where each piece lives, so pin the stores next. The counter lives in a fast in-memory store, Redis being the usual pick, keyed by the client key so each bucket is one entry. Redis is right here because the counter is touched on every request and has to answer in well under a millisecond, which a disk-backed database cannot promise. The Tier policy lives in ordinary config, cached in each gateway replica and refreshed occasionally, because it changes rarely and is read constantly. The gateway itself holds no durable state. Any replica can serve any request, and losing one just means the load balancer routes around it.

API

The external surface is thin. A rate limiter is mostly one decision applied to every call. Do write down the internal atomic check against the counter store, though, since the whole correctness story lives in that one operation. You'll refer to these by name later.

Two of these carry the design. The proxied call is the product, and an honest 429 is what makes clients trust the limiter instead of routing around it. If Retry-After doesn't reflect when a token actually frees up, clients ignore it and retry immediately, which defeats the point of a limiter. The internal allow check is the other one. It turns twenty independent replicas into one correct limit, and making it atomic is most of the job. The deep dives either make that check more correct or make it cheaper.

High-level design

Follow one request from a client to a backend. An API clients call arrives at the API gateway carrying an API key. The gateway resolves the key to a client and a tier by reading the Tier policy it keeps cached. It runs the atomic budget check against the Rate-limit store, and on a pass forwards the request to the right one of the Microservices behind it. On a fail it returns a 429 straight back to the caller without ever touching a backend. The diagram colors each box by role: blue for the API clients, violet for the API gateway and the Microservices it fronts, and green for the two stores it reads.

1) The request arrives and the key is resolved. An API clients call reaches the API gateway with an API key in the Authorization header. The gateway verifies the key and looks up its tier in the Tier policy, which it keeps cached, so this step adds no network hop. Now it knows the client identity and which bucket size applies. Anything slow stays off the hot path on purpose. No per-request database read, no call out to an identity service. Either one would blow the latency budget on every request.

2) The gateway checks the budget. With the tier in hand, the gateway runs the atomic allow check against the Rate-limit store. That one operation refills the client's token bucket based on how much time has passed, tries to remove a token, and returns whether the request is allowed along with how many tokens remain. The limit is only correct if this is atomic across the fleet. That is the first place the design gets pushed, and the second deep dive owns it.

3) An admitted request is routed. If a token was available, the gateway forwards the request to the right one of the Microservices, adds the X-RateLimit headers to the response, and streams the answer back. The common case stays cheap: a cached tier lookup, one fast counter check, then a proxy. What gets punted for now is a slow or dead downstream. One hanging backend must not take the gateway with it, and the fourth deep dive deals with that.

4) A rejected request gets an honest 429. If the bucket was empty, the gateway never touches a backend. It returns a 429 with a Retry-After computed from the bucket's refill rate, the time until the next token appears, plus the X-RateLimit headers. A good client reads Retry-After and waits. A bad one retries immediately and gets another 429, which costs the backends nothing because the reject happens at the gateway. Which counting algorithm makes that Retry-After trustworthy is the first deep dive.

5) When the machinery underneath blips. Everything above assumes the Rate-limit store answers quickly and the Microservices are healthy. Neither is guaranteed. The store can be briefly unreachable, and any of the two hundred backends can hang. In those moments the gateway can fail open to a local fallback when the store blips, trip a circuit breaker on a sick service, and isolate its threads. That is the enterprise-resilience core, and the last deep dive covers all of it.

Technology choices

A box on the diagram is just a label until you name a real tool, say why that kind of tool fits, and know how it breaks under this load. Put a popular pick in each slot. The follow-ups will be about how each one fails.

• Rate-limit store is an in-memory key-value store: Redis, or a compatible managed one. In-memory is right because the counter is read and written on every request and has to answer in well under a millisecond, which a disk-backed database cannot. The primitive that matters is a Lua script, which Redis runs atomically. It refills the bucket, spends a token, and writes the new state in one indivisible step, so two gateway replicas hitting the same key can never both take the last token. The failure mode to name is Redis as a hot single node at fifty thousand checks a second. The fixes are sharding the keyspace by client key and, in the next dive, serving most requests from a local reservation so the store is not touched on every call.
• API gateway is a real gateway, not something you hand-roll: Envoy or NGINX, or a managed API gateway on top of one. This kind of tool exists precisely to do auth, routing, rate limiting, and request and response transforms in one place, so two hundred services do not each reimplement them. Rate limiting is a first-class feature in both, backed by exactly the token-bucket and shared-counter machinery here. The failure mode is the gateway itself becoming a bottleneck or a single point of failure. The answer is the horizontal fleet behind a load balancer, which is also why the counter has to be shared.
• The resilience layer is a library, not a diagram box: Resilience4j, or Envoy's own circuit breaking, for the breakers, timeouts, and bulkheads that keep a sick downstream contained. They are configuration on the gateway rather than new services. The failure they answer is a slow backend consuming every gateway thread, and the mechanism is a per-downstream connection pool plus a breaker that fails fast once error rates cross a threshold. The fourth deep dive is where this earns its place.
• For the estate itself, a service mesh such as Istio on OpenShift is where a lot of enterprise IBM deployments live. The mesh gives you sidecar rate limiting for east-west traffic between services, while the gateway handles north-south traffic from outside. Name the split honestly: the gateway is the front door for external clients, the mesh handles service-to-service limits inside, and both lean on the same counting ideas.

The IBM angle

Map the design onto IBM's hybrid-cloud stack when the interviewer asks, and be specific. IBM shops run OpenShift, and in production a lot of this lives in a service mesh riding on it. Envoy sidecars do per-service limits and the mesh handles retries and timeouts between services, with an API gateway at the edge for outside traffic. If you have worked with OpenShift or Envoy, name it and say what each piece did. None of these products hands you a correct distributed limit for free. You get the token-bucket primitive and the circuit-breaker config. The tier model, the shared-counter behavior and the fail-open decision are still yours to design.

Potential deep dives

A rate-limiter round usually pushes on four corners: the counting algorithm, keeping the count correct across a fleet, where the limiter sits in the gateway, and what happens when something underneath fails. IBM spends the most time on the last one. It gets the most room below.

1) Which counting algorithm actually holds the line?

Start with the counting. The algorithm you pick decides how fair the limit feels and how honest the Retry-After is. Anchor on a silver client at six thousand requests a minute, which is one hundred a second. Every algorithm answers the same question: given the requests that already arrived, is this next one allowed? The naive answers have real flaws. Walk from the worst to the best, because that is the reasoning the interviewer wants to watch.

Sliding-window counters are already cheap, so the jump to the last tier is about the missing burst. Real API clients are bursty on purpose. A batch job wakes up, fires a hundred calls, and goes quiet for a minute, and its average stays well under the limit the whole time. A pure rate limit punishes that pattern anyway. You want a limiter that says yes to a short burst as long as the sustained rate holds. A token bucket has exactly that shape, and it is the practical default real gateways ship.

2) How do twenty gateway replicas share one limit?

A single gateway makes the token bucket trivial: the two numbers live in local memory and every request updates them in order. Reality is a fleet. The gateway runs as roughly twenty replicas behind a load balancer, and a client's requests spray across all of them with no affinity. Say each replica keeps its own local bucket sized to the full hundred a second. The client can push a hundred through each of twenty replicas at once, two thousand a second, and every replica still reads itself as under the limit. So the count has to be shared. Sharing it correctly inside a one-to-two millisecond budget is the hard part.

3) Where in the gateway does the limiter belong?

Now zoom out from the counter to the pipeline around it. An API gateway does several jobs in sequence, and the order changes what the limiter can do. A gateway authenticates the caller, resolves its identity and tier, applies the rate limit, routes to a backend, and transforms the request and response along the way. Put the limiter in the wrong place in that chain and it either cannot see the tier it needs or wastes work on requests it should have dropped for almost nothing. The enterprise version of the question is sharper still: with two hundred services behind the gateway, do you centralize the limit or let each service run its own?

The IBM angle

The two-hundred-services framing is IBM's, and it is a setup for the partial-failure question. An IBM interviewer will often say it out loud: two hundred services behind the gateway is two hundred failure points and two hundred rate-limit policies, so how do you keep it sane? They listen for three things. The gateway centralizes the policy, so there's one place to reason about it. The limit runs after identity, so tiers are real. And the one they actually care about: a misbehaving service can't take the gateway or the others down with it. Land the centralized, ordered pipeline, then pivot straight into how the gateway contains a sick downstream. That pivot answers the question they were really asking.

4) What happens when the limiter or a downstream is down?

The whole round was building toward this dive. Two things underneath the gateway can fail, and they fail differently. The rate-limit store can go slow or unreachable, and any of the two hundred backends can hang or start erroring. Either way the gateway keeps serving the healthy traffic it can and contains the damage. The patterns that do that are the enterprise-resilience vocabulary, and the interviewer wants to hear them used correctly.

Take the rate-limit store first: Redis is briefly unreachable, or slow enough that waiting on it would blow the latency budget. The gateway has two honest options, and picking one is a real decision. Fail open means that if the limiter does not answer within a tight timeout, the gateway allows the request. Availability is preserved, but the rate limit is gone exactly when the store is struggling, so a flood could pour through at the worst moment. Fail closed means that if the limiter is unreachable, the gateway rejects. The backends stay protected, but a brief Redis blip becomes a total outage where every request gets a 429, which is often a worse failure than the one you were guarding against.

Put a tight timeout on the Redis call, a few milliseconds, so a slow store never adds latency to every request. Then fail open, but not to no limit at all: fall back to the per-replica local token bucket at a conservative fraction of the real limit. When the shared store is gone, each replica still enforces something, so you neither drop all traffic nor remove all protection, and when Redis recovers the replicas resume the shared count. Which way you lean depends on what the limit protects. A limit guarding a fragile paid downstream or a hard cost ceiling leans toward fail-closed or a strict local fallback, while pure abuse-prevention on a robust service leans open. Say which, and say why.

Sizing that local fallback is its own small decision. Divide the real limit by the replica count and you are back to the exact math that made local-only wrong, so lean lower. Give each replica a fraction that, summed across the fleet, stays comfortably under the true limit. You would rather under-admit for the few seconds Redis is away than let the flood through. When the store returns, the replicas drop the fallback and resume the shared count on their next check. The fallback is a safety floor for a few bad seconds.

Now the other failure. A request was admitted, the gateway routes it, and the backend hangs. This is where one sick service among two hundred can quietly take down the gateway itself, because a gateway thread waiting on a dead backend is a thread that serves no one else. A gateway has only so many threads. Four patterns, layered, keep one bad service contained.

The IBM angle

This dive is why IBM leads its senior loop with the rate limiter, and it is the clearest place to sound like someone who has run this in production. A consumer-scale interview rewards a clever counting trick. IBM's enterprise register rewards the answer to what happens when a piece of a two-hundred-service estate falls over at three in the morning. Use the vocabulary precisely: a timeout releases a thread, and a circuit breaker fails fast to give a service room. Jittered backoff stops a retry herd, and a bulkhead keeps one drowning service from taking the whole pool. Say fail-open-to-a-local-fallback for the limiter and bulkhead-plus-breaker for the downstreams, explain why each degrades gracefully, and you've given the resilience-first answer this interview is built to find.

What is expected at each level

There's a lot of surface here, and your level decides how much of it you cover before anyone asks. Read the rungs as a gradient: mid gets a working limiter, senior owns the shared counter and the fail-open decision, and staff carries the full partial-failure story and the operability around it.

Mid-level engineer

Favor breadth, call it eighty to twenty. Reach one coherent design: a gateway in front of the services, a token bucket per client key, tiers mapped to bucket sizes, and a clean 429 with Retry-After. Get the token-bucket check itself onto the board, since this interviewer often drops from the diagram into the code, so be ready to write tokens = min(burst, tokens + elapsed * rate) and explain it. The phase that earns the pass is the fleet question. When they ask what happens with twenty replicas, reason toward a shared counter rather than a local one, even if you need a nudge to get there. A working limiter whose algorithm and 429 contract you can write out clears this rung. Expect the interviewer to fill in the distributed and failure details.

Senior engineer

Now depth matters, roughly sixty to forty, and this is the rung the round is tuned to. Raise the hard facts yourself: a local-per-replica counter overshoots by the fleet size, so the count must be shared and updated atomically, and the shared store must not become a single point of failure. Reach the token-bucket-in-Redis answer with an atomic Lua script, and then earn the local-reservation optimization by naming the round-trip cost that forces it. Make the fail-open versus fail-closed call out loud. Defend which way you lean, and why. You should find the twenty-times overshoot and the limiter-as-single-point-of-failure problems on your own and answer each concretely. Clearing this rung means a near-finished limiter whose distributed correctness and its main failure mode you surfaced without being asked.

Staff and beyond

Depth dominates across the whole picture, roughly forty to sixty. Own the partial-failure story end to end. The limiter fails open onto a conservative local bucket. The two hundred downstreams get timeouts, circuit breakers, jittered backoff, and bulkheads, each explained by the specific cascade it prevents. Carry the ordered gateway pipeline, the north-south gateway versus east-west mesh split, and the sharding and hot-key handling for the counter store. Show the arithmetic that takes a naive twenty-times overshoot back down to a bounded few-percent slack. Know how an operator changes a tier, watches for a client pinned at its limit, and sees a breaker open. You set the direction. The interviewer only nudges. The bar here is holding the cheap check, the shared count, the gateway ordering, and the failure containment as four distinct levers, and keeping added latency and graceful degradation intact through every refinement.