Design LLM Inference Serving

Understanding the problem

The phrasing is a hint that the answer lives inside the GPU. There, one batched matmul over 100 requests costs almost the same as a matmul over a single request, so the problem reduces to keeping that batch full. Take the literal reading too. A user sends a prompt. A model the size of a small datacenter's worth of weights answers it token by token, and you do that for millions of concurrent users without lighting money on fire. The tempting first answer is a stateless service in front of a model that you load-balance, and it dies on the first follow-up. Generating text is a loop. That loop holds growing per-request state on the GPU, and the cost of that state is what the design lives or dies on.

A single inference is two workloads stitched together. The model reads your entire prompt in one pass, which is heavy parallel math. Call it prefill; it is compute-bound. Then it generates the answer one token at a time, each new token conditioned on everything before it. Each step does little math but drags the whole cached context out of memory. Call it decode; it is memory-bandwidth-bound. Different bottlenecks, different scaling behavior. Candidates report that the model itself is treated as a black box for this round, so you do not need its architecture or training math. You need to see the distributed-systems problem under the AI framing: queuing, batching under a hard memory limit, and routing token streams back to the right caller.

The Anthropic angle

Anthropic keeps how it serves Claude private, so build the answer from how production LLM inference works in the open, and say so once. The prefill and decode split, the KV cache, PagedAttention, continuous batching: these come from the public serving literature and systems like vLLM and TensorRT-LLM, not from anything Anthropic has shipped about its own stack. Frame it as the standard way frontier inference is served, not as a claim about Claude's internals. By candidate reports the round rewards exactly this kind of honesty about where public knowledge stops. An interviewer who works on this system every day trusts a candidate who marks the edge of the public record far more than one who invents inside detail. One first-party glimpse does exist: Anthropic's postmortem of three recent issues says Claude runs on AWS Trainium, NVIDIA GPUs and Google TPUs, with short- and long-context requests routed to separate server pools.

Functional requirements

Keep the surface small and let the interviewer widen it. The core is: take a prompt, generate a completion, stream it back as it is produced, and do that for many tenants at once under a latency target.

Non-functional requirements

Reason through the targets, because they are what make this design unusual. A user feels two separate latency numbers. Time-to-first-token is how long after they hit enter before anything appears, dominated by queueing plus prefill. Inter-token latency is the gap between streamed tokens once it starts, the how-fast-does-it-type feeling, set by how long each decode step takes. A system can be excellent at one and miserable at the other, so commit to both.

Cost is not a footnote on this system. It is the objective. GPUs are the dominant expense and they are scarce, so the number that matters is tokens per second per GPU, which is the same as cost per token. Almost every clever thing in the design exists to push more tokens through the same silicon. Favoring the change that lowers the cost number over the one that looks good in a diagram is Anthropic's preference for working over clever showing up in an infra round. Throughput and latency then pull against each other. Batching more requests together raises throughput but slows every token in the batch, so you are always tuning a dial between them. The workload makes it harder: a request can be fifty tokens or fifty thousand, and you do not learn the output length until it stops.

The memory ceiling

You will not have Anthropic's real numbers, and you should not pretend to. Reason from orders of magnitude and from the one quantity that controls capacity here: GPU memory. Work it as a budget. The weights take a fixed slice, the live KV caches take whatever is left, and dividing the leftover by the cache cost per request gives the concurrency ceiling. The point of the estimate is to show that memory runs out first during decode, well before compute does.

Pin a concrete model so the arithmetic is real. Take a 70-billion-parameter model in fp16: about 140 GB of weights, more than a single 80 GB card holds, so it splits across two GPUs with tensor parallelism. On that pair of cards, weights eat ~140 GB and you are left with roughly 20 GB per replica for the cache. Each token of context stores attention keys and values for every layer, which for a model this size is around 300 KB per token. So 20 GB holds about 60,000 tokens of cache total, shared across every in-flight request. At a 2,000-token average context that is only ~30 concurrent requests on the replica, and that ceiling is set by memory, not by compute the GPU could in principle do. That single number is why "100 requests cost about the same as 1" is true only until the cache fills, and the rest of the design is about pushing that fill line higher.

• Weights on the GPUs: ~140 GB, fixed. A 70B model in fp16 is roughly 140GB of weights, more than one 80GB card holds, so it is sharded across two or more GPUs with tensor parallelism. That reservation never moves. On a pair of 80GB cards you spend ~140GB on weights and have ~20GB left per replica, and that leftover is exactly what the growing KV caches fight over.
• KV cache per token: ~300 KB. Each token stores the attention keys and values for every layer. For a 70B model with grouped-query attention that is roughly 300KB per token. The ~20GB of free memory therefore holds about 60,000 tokens of cache total across all requests. At a 2,000-token average context that is only ~30 concurrent requests per replica, which is the line PagedAttention exists to push higher.
• Prefill vs decode: one pass vs ~25 ms/token. Prefill runs every prompt token through the model in a single forward pass with high arithmetic intensity, so the compute units saturate. Decode then runs one token per step at maybe 20 to 40ms each, re-reading the full 140GB of weights and the whole cache from memory while doing little math. Same model, opposite bottleneck: prefill is compute-bound, decode is memory-bandwidth-bound.
• Throughput per replica: ~1,000+ tokens/s. At ~25ms per decode step one request streams ~40 tokens per second, and batching ~30 of them together pushes the replica past 1,000 tokens per second on the same weight read. That product, tokens per second per GPU, is the cost number the business pays. Almost every later choice exists to raise it without breaking the latency budget.

The shape that falls out: a fixed fleet of memory-constrained GPUs, two phases with opposite bottlenecks, per-request state that grows as it runs, and two separate latency promises. That is the argument for everything downstream. Paged memory for the cache, batching at the token step, a demux that maps batched output back to synchronous callers, and a recovery story for when a GPU drops mid-batch all follow from it. Build it in that order.

The set-up

Core entities

Name the objects before you draw. The one that surprises people is the KV cache, because it is mutable per-request state sitting on the GPU, which is the opposite of the stateless service you might expect.

One relationship carries the design. GPU memory is shared between the model weights and the live KV caches of every in-flight request. The weights are fixed, so concurrency is decided entirely by how efficiently you pack the caches. That single constraint is why a virtual-memory trick for the cache and a token-granular scheduler are not optimizations bolted on later. They are the architecture.

API

Anthropic's public surface is one streaming endpoint, POST /v1/messages, and the streaming is the interesting part: the response is a sequence of chunks over a held-open connection, each sent the moment it is produced. The internal calls are where the design lives, between the gateway, the scheduler, and a model replica.

High-level design

Trace one request all the way through. It hits the caller-facing gateway, gets tokenized, and lands in the scheduler's queue. From there it runs prefill on a model replica, which fills the KV cache. Then it enters the decode loop that emits one token per step, and each token is streamed back to the caller waiting on its request id. The diagram codes each box by role: blue for the caller, violet for the serving stages, green for the KV cache memory, amber for the token stream.

1) Admit, then tokenize. A prompt reaches the gateway, which authenticates the caller and checks the token quota before letting it through. The gateway then makes the one decision that protects everything behind it: admit the request now, or hold it in the queue. GPUs are finite, so the front door pushes back when the pool is full rather than let work pile onto an overloaded replica. Admitted requests get tokenized on the CPU, which is cheap and entirely off the GPU hot path. The prompt's token count falls out of that step as the first input the scheduler uses to size the work. Each request also gets an id here, which the demux later uses to find the right connection. Admission without that sizing is how one tenant starves the rest, which the fairness deep dive picks up.

2) Prefill the prompt. The scheduler places the request on a replica and runs prefill. That is one forward pass over every prompt token. It saturates the GPU's compute, produces the first output token, and writes the KV cache for the whole prompt at once. Prompt length shows up here. A 50-token prompt prefills in a blink. A 2,000-token prompt is a much heavier matmul and a visibly slower first token, since time-to-first-token is mostly queueing plus this pass. Nothing here is incremental: the prompt is processed in parallel, then the cache is full and the request is ready to generate. The hidden cost is that a long prefill monopolizes the GPU and stalls every other request's decode on that replica, which the disaggregation deep dive fixes.

3) Decode in a loop. Now the request enters decode. Each step feeds in the last token, reads the full KV cache plus all ~140 GB of weights out of memory, computes the next token, and appends its keys and values. The arithmetic is tiny; the memory traffic dominates, so a step runs in roughly 20 to 40 ms, and that number is the inter-token latency the user feels. The request does not own the GPU alone. The scheduler runs many requests' decode steps together as one batch, so a single weight read serves them all, which is the literal reason 100 requests can cost about the same as one. It refills emptied slots from the queue on every step. Two questions hang over this loop. How do you pack the cache so more requests fit (the KV-cache deep dive)? And how do you keep the batch full when outputs finish at different lengths (the continuous-batching deep dive)?

4) Demux, stream, and stop. Output tokens come off the batch tagged by request id. The scheduler routes each one to the per-request channel its caller is streaming from, so the batched internals collapse back into 100 clean synchronous responses. The first token reaches the caller after a few hundred milliseconds, and each one after it lands every 20 to 40 ms, faster than a person reads. The loop ends on a stop token or the length cap, and a client disconnect cancels it early. At that moment the request's cache pages are freed and handed straight to whoever is waiting in the queue. Freeing fast matters more than it looks: those pages are the scarcest thing in the system, and a slot held a second too long is concurrency you paid for and never used.

Technology choices

Every box is a category until you name something and defend the trade-off. These are the public building blocks; use them and be ready for the follow-up behind each.

• The model server is a specialized inference engine, where vLLM is the recognizable open one and TensorRT-LLM and TGI are its peers. A generic web service cannot do this job, and the reason is everything in this lesson. A normal RPC server treats each call as stateless and independent. Here every request carries a growing KV cache on the GPU, the batch is rebuilt at the token step rather than per call, and the cache memory is handed out in pages. None of that exists in a plain service, and those are precisely the mechanics that set throughput. The probe to expect is "why not put the model behind your usual load balancer and autoscale?" The answer is that load balancing distributes stateless requests, while this request is a stateful loop holding the scarcest resource in the system. The scheduling that matters happens inside one replica rather than across many.
• A frontier model is served as a tensor-parallel replica across several GPUs over NVLink, because the weights do not fit on one card and a per-layer all-reduce needs the fast intra-node link. Tensor parallelism splits each layer's matmul across the GPUs in the group and stitches the results back every layer, so the group runs as one logical model. For models too large even for one node, pipeline parallelism splits the layers across machines, at the cost of bubble overhead where stages wait on each other. The failure probe is what happens when one GPU in the group dies. Because the shards are one unit, the whole replica goes down rather than a fraction of it. So the recovery story is replica-level: drain its in-flight requests, restart the group, and let the scheduler route around it while a warm spare absorbs the load.
• The KV cache is managed with PagedAttention, the idea vLLM made standard: fixed-size cache blocks handed out on demand, with a per-request block table mapping logical token positions to whichever physical pages hold them. It is the virtual-memory trick applied to attention state. The alternative, reserving one contiguous max-length block per request, wastes most of the memory that caps concurrency, and paging reclaims it, which roughly multiplies how many requests fit on the same GPU. The probe is what happens when the page pool runs dry mid-generation, since admission let in more work than the cache can hold. Then you either preempt a request and evict its pages, recomputing or swapping them back later, or refuse new admissions until pages free. So the allocator and the scheduler are really one system.
• Streaming is server-sent events over HTTP, with the gateway holding the connection open while the replica produces tokens and the scheduler pushing each one to the right request channel the moment it is decoded. The alternative is buffering the whole completion and returning it at the end. That throws away the latency benefit of incremental generation and makes a long answer feel frozen, so it is the wrong default for an interactive product. The probe is the caller that disconnects mid-stream because the user closed the tab. That disconnect has to propagate back through the gateway to the scheduler so generation stops and the cache pages free immediately, since a request nobody is reading is pure waste on the scarcest resource. Offline and bulk workloads can take the buffered path instead, which is one reason to run a separate throughput-tuned batch tier.

The Anthropic angle

The KV cache is where this round is won or lost, so go there early and be specific about why decode is memory-bound. A generic answer treats serving as "put the model behind a load balancer and autoscale," which is the right shape for a stateless service and the wrong shape for this. What makes LLM serving its own discipline is that each request carries growing state on the GPU and that decode spends its time moving memory while doing little math. Name the cache as the capacity constraint, name PagedAttention as how you pack it, and name continuous batching as how you keep the GPU busy. The model is framed as a black box on purpose. The signal is whether you can find the distributed-systems problem underneath. An interviewer on this team knows within two minutes whether you understand where the cost really lives.

Potential deep dives

The signal is in the four problems below. How do you manage cache memory? How can 100 requests cost about the same as one? How do batched internals route back to synchronous callers when a GPU dies mid-batch, and which latency number are you optimizing? Each one separates a recited answer from a built one.

1) The KV cache: why decode is memory-bound, and how to pack it

Start with why decode is memory-bound at all, because the whole design rests on it. A decode step does very little arithmetic, since it predicts one token. To do it, though, the GPU must read all of the model weights and the entire KV cache for the request out of memory. The math finishes long before the memory transfer does, so the expensive compute units sit idle waiting on bandwidth. That is the definition of memory-bound, and it is why the answer to "how do I go faster" is almost always "move less memory or keep more requests busy," not "add FLOPs."

Then the capacity question. The cache for one request grows by one token's worth of keys and values per step, across every layer, so a long conversation is a large and growing chunk of GPU memory. With the weights already pinned, the leftover memory divided by the per-request cache cost is your concurrency ceiling. How you allocate that memory is the design.

The Anthropic angle

One phrase carries this deep dive: the cache is paged like virtual memory. It names the very failure mode it fixes. Contiguous per-request reservation wastes the memory that caps concurrency, and pages reclaim it. Follow with prefix sharing: one prefill of a shared system prompt, its pages reused read-only across requests, and you have shown you see the second-order win past the first. This lines up with how Anthropic says it judges design work, rewarding the simplest approach that holds. Paging is a small, well-understood idea borrowed from operating systems, and it buys the largest single jump in requests-per-GPU. Hand-waving "use a KV cache" without the allocation story leaves the interviewer unsure you know why the cache is the bottleneck in the first place.

2) Continuous batching: why 100 requests can cost about the same as 1

This is the deep dive the signature prompt is really about. Batching amortizes the memory reads. Pull the weights out of memory once, apply them to many requests at the same step, and the per-token cost falls far below serving one request at a time. The weight read, not the math, is the dominant cost. That is precisely why 100 requests can take close to the wall-clock time of one. The catch is unique to generation. Requests in a batch finish at wildly different lengths: one stops after 20 tokens, another runs to 800, and you do not know which is which up front. How you handle that mismatch is the central throughput question.

3) The async-to-sync demux, and surviving a GPU that dies mid-batch

Here is the twist Anthropic interviewers reportedly love, and it is the one the box diagram hides. Each of the 100 callers holds one ordinary synchronous streaming request and expects only its own tokens. Internally those 100 are enqueued, batched, and run together on a GPU, so a single decode step produces a vector of next-tokens for the whole batch at once. Something has to take that batched output and route each token back to the exact connection that is waiting on it, in order, without ever leaking one caller's tokens onto another caller's stream. That mapping from async batch back to synchronous caller is the demux, and forgetting it is how a tidy diagram falls apart on the first follow-up.

The mechanism is request correlation. The gateway tags each admitted request with an id and opens a per-request response channel, an in-memory queue or stream the SSE handler drains. Every token the batch emits carries its request id, and the scheduler pushes it onto that request's channel, where the caller's open connection picks it up. Ordering per request is guaranteed because decode is sequential within a request even though it is parallel across the batch. Backpressure rides on the same structure. If a slow client cannot drain its channel, that channel fills, and the scheduler can pause or cancel that request rather than let it back up the whole batch.

The Anthropic angle

One batched GPU, a hundred callers, each holding a private synchronous stream: the async-to-sync demux is the seam this round keeps pulling at. The serving internals are shared industry knowledge: prefill, decode, paging. Reach this part early; it is unmistakably Anthropic's. How does a token find its connection? How does ordering hold? What happens to those streams when the GPU under them dies? Walk the request-id correlation, the per-request channel, and the re-enqueue-and-resume path. Do that and you have answered the failure-mode questions before they are posed, which is the proactive reasoning candidates report the round rewards.

4) Disaggregation, autoscaling on tokens, and multi-tenant fairness

Three things separate a senior answer here, and all three come from this being a shared, streamed, memory-bound system where request-response intuition breaks down. The first is that prefill and decode fight each other on a shared GPU. A prefill is a heavy compute burst, and while it runs, the decode steps of every other request on that replica stall, so a long prompt arriving makes everyone else's text stutter. The fix the field has moved toward is disaggregation: a pool of GPUs dedicated to prefill and a separate pool dedicated to decode. A request is prefilled on the prefill pool, its KV cache is shipped across a fast interconnect to a decode worker, and it generates there. Now a compute burst never interrupts a memory-bound decode loop, and you size the two pools independently, which matters because a sea of short prompts is prefill-heavy while long generations are the reverse. The cost is shipping caches between machines, real bandwidth that can fail, so disaggregation earns its place only at scale; below that, chunked prefill on one pool is simpler and fine.

• Scale on token load, not request count. One "request" can be 50 tokens or 50,000, so autoscaling on requests-per-second is meaningless here. Scale on signals that reflect real work: queue depth, KV-cache utilization across the pool, and tokens per second. When cache utilization runs hot or the queue backs up, add replicas; when it drains, retire them.
• Replicas are slow to start, so scale ahead. Bringing up a new replica means loading tens or hundreds of gigabytes of weights into GPU memory, far slower than a stateless container. Reactive autoscaling arrives too late, so you keep a warm buffer of capacity and scale on a leading indicator, well before the pool is saturated. The same warm buffer is what absorbs a replica that dies mid-batch.
• Two latency numbers, not one. A single p99 is nearly useless for a streaming generator. Time-to-first-token is the wait before anything appears, dominated by queueing plus prefill, so it worsens with long prompts and a backed-up queue. Inter-token latency is the gap between streamed tokens, set by decode-step time, so it worsens as you grow the batch. They trade off directly, so commit to a target for each, and know that "make it cheaper" usually means "make inter-token latency a little worse," a product decision as much as an infra one.
• Cost per token is the north star. Batching, paging, disaggregation, quantization to lower-precision weights, speculative decoding to emit several tokens per step: every one of these is about pushing more tokens through the same GPU. Frame placement and autoscaling against tokens-per-GPU-second, because that is the number the business pays.

What is expected at each level

That is a lot of surface, and how much you have to land depends on the scope the loop is calibrated for. Anthropic titles ICs by scope (Software Engineer, Senior, Staff), not a public L-number, so the bar below is described by how much you self-direct and how deep the failure-mode reasoning goes. This AI-infra design comes up for infra, ML-systems, and applied-AI candidates. Prep vendors report the onsite as a technical loop that gates the rest, so a weak showing here carries more weight than an averaged-out round elsewhere. Treat that gate framing as reported rather than confirmed, and treat the bar as scope, not a number.

Software Engineer (mid-level)

Get the two phases right and resist the stateless-service reflex. Reach a working end-to-end shape: admit, tokenize, prefill the prompt, decode in a loop, stream tokens back. Then say it plainly: prefill is compute-bound, decode is memory-bound, so this is more than a model behind a load balancer. The non-negotiable is naming the KV cache as per-request state on the GPU and recognizing that it is what fills up GPU memory, ahead of compute. Gesture at why batching makes 100 requests cheap. The deeper packing, demux, and scheduling can surface in conversation rather than unprompted.

Senior

Depth now, and this is often where the round sets your level. You drive the KV-cache-as-capacity-constraint argument unprompted, and reach PagedAttention as the way to pack it, explaining the fragmentation it kills. You reach continuous batching as the answer to mismatched output lengths that static batches cannot handle. You explain why 100 requests cost about the same as one in terms of the weight read, and you handle the async-to-sync demux when the interviewer pushes on it. You separate time-to-first-token from inter-token latency and reason about the throughput-versus-latency dial. The bar is a near-complete serving design where you found the memory and batching structure yourself.

Staff and beyond

You drive the conversation end-to-end with no steering, which Anthropic calls out explicitly at this scope. A few things should come from you, not the interviewer. Disaggregate prefill from decode, and justify it from their opposing bottlenecks. Raise the demux and a clean GPU-died-mid-batch recovery before you are asked. Autoscale on token load, and keep warm capacity because replicas start slowly. Make the tenant first-class so one caller cannot set the median, and tie it all back to cost per token. You also reason about how the system degrades: a cache pool exhausted mid-generation, a long-prompt flood spiking everyone's inter-token latency, a hot tenant, an agent client that never stops. For each one you name the lever without prompting, whether that is preemption, chunked prefill, a priority tier, or speculative decoding. The bar is that every stage is a defended choice and the trade-offs are yours to own.