Design an LLM Serving System

Understanding the prompt

The prompt often arrives thin, sometimes as bare as "design an LLM serving system," and it's Scale's own problem. Scale is a data-for-AI company that lives inside models. You won't be scaling a photo feed or sizing a cache tier for a chat app. You'll be serving a large language model to real traffic, fast enough for whoever is waiting, with throughput that holds under load and a cost per call the product can afford. Say early, out loud, that a slow and expensive model call sits at the center, and size everything else around it.

In a classic prompt the hard part is fan-out, sharding, and surviving a read spike. Here a request is cheap to route and cheap to store. The model's answer is where the money goes. Producing one token means running billions of parameters across a GPU, and a response needs hundreds of them. So the limits are tokens per second across the fleet, GPU memory per in-flight request, the wait for the first token, and the price of every token. The same reasoning carries over to the agent-over-documents prompt, and to the serving layer under any Scale pipeline.

The Scale AI angle

The bar here is inference economics, so size the model call and its cost before you draw a box. The system is graded on whether a model call comes back correct, fast enough, and cheap enough at the customer's volume. In practice you need tail latency that holds up on a saturated GPU, and a way to batch many requests through one forward pass without starving whoever is waiting on the first token. Then the token bill has to stay sane as traffic grows. Talk in tokens per second, GPU memory, batching, caching, and dollars per request.

Scoping the requirements first

Don't name GPUs yet. The prompt is thin on purpose, and a minute spent bounding it keeps the rest coherent. Clients send a prompt and expect a completion, usually streamed token by token so the first word shows up fast. At Scale those clients are often agents and internal products. Some requests are interactive, with a person or an agent step waiting. Others are bulk jobs running behind them. The system serves one or more large models, keeps them warm on scarce GPUs, and answers within a latency target while holding a throughput target and a cost ceiling.

Then separate what you handle now from what you name and defer. Skip the exact model architecture, the precise quantization scheme, and the full autoscaling policy in the first pass, but model the skeleton so those pieces have an obvious home. The serving path starts with one large model and grows into a fleet of models and tenants. Batching starts naive. Continuous batching arrives once the basic loop works, and cost control grows from caching into routing and quantization. Naming the deferral is the judgment the round looks for. Then pin the non-functional bar out loud, because every later decision answers to the numbers below.

• The clock that matters: time to first token, then between tokens. An interactive user judges an LLM on how fast the first token appears and how smoothly the rest streams. A single round-trip number misses that. State a target out loud, on the order of a few hundred milliseconds to first token and a steady stream after. A bulk job trades all of that away for throughput. The latency SLO decides how aggressively you can batch.
• The unit of work: tokens per second, not requests per second. One request is hundreds of units of work, because a completion is generated one token at a time. Size the fleet in tokens per second it must generate, since a request for 50 tokens and one for 2,000 cost forty times as much on the same hardware. QPS hides that, and a web prompt never makes you see it.
• The constraint that binds: GPU memory caps the batch. GPUs are scarce and expensive, and memory usually runs out before compute. The model weights sit in it, and every in-flight request holds a growing KV-cache in it, so memory decides how many requests batch together. Say this early. It explains why batching and cache management are the load-bearing moves.
• The number that never leaves: cost per request is a first-class SLO. Every token spent is money, and GPU hours dominate the bill, so cost runs through the whole conversation. Treat dollars per thousand tokens as a target alongside latency and throughput. Name where the spend goes and how you cap it.

Estimation, with illustrative numbers

Do the arithmetic out loud before any architecture, and call every number a back-of-envelope illustration. None of them are real internal figures. Nobody is checking whether you memorized a GPU spec sheet. The interviewer wants to watch you go from an assumed traffic shape to a GPU count and a memory budget, because those two numbers decide the rest of the design. State the assumptions and keep the numbers round. A candidate might work the traffic like this.

Assume on the order of:
  peak arrival        ~50 requests/sec, ~500 in flight at once
  prompt (input)      ~1,000 tokens          (the "prefill" work)
  completion (output) ~500 tokens streamed   (the "decode" work)
 
Generation throughput the fleet must sustain:
  50 req/s x 500 output tokens = ~25,000 output tokens/sec
 
One busy large-model GPU, in a batch, decodes on the order of
  ~1,000 to 3,000 output tokens/sec (very model- and hardware-dependent)
 
  -> 25,000 / ~2,000 = on the order of ~12 GPU-workers of decode
     headroom, before any caching or routing cuts the load

GPUs are the cost center, so anything that cuts tokens generated pays back directly in hardware, like caching a repeated answer or routing a simple request to a smaller model. The less obvious result is that a single GPU hits its memory limit before its throughput limit. Work that one explicitly. It explains the whole batching design.

Model weights (illustrative, a ~70B-parameter model):
  fp16   ~2 bytes/param  -> ~140 GB   (spans several GPUs)
  int8   ~1 byte/param   -> ~70 GB    (quantization halves it)
  -> weights alone consume most of one or more GPUs, always resident
 
KV-cache, the per-request state that grows with context:
  every token attends to all prior tokens, so their keys/values
  are cached in GPU memory for the life of the request
  assume on the order of ~hundreds of KB per token for a big model
  1,000-token context -> on the order of ~hundreds of MB per request
 
  -> with weights resident, the leftover memory divided by the
     per-request KV-cache is what caps how many requests batch at once
  -> KV-cache, not compute, is the ceiling on concurrency

Decode throughput bounds tokens per second. The KV-cache memory left over after the weights load bounds concurrency. They pull against each other. A bigger batch uses the GPU efficiently and raises tokens per second, but every request in it holds KV-cache memory, so memory caps the batch long before compute does. Hold both limits in your head and every later call about batching, quantizing, and caching has a reason you can say out loud.

The high-level architecture

With the limits named, the architecture is the wiring that keeps the GPU fed without overrunning its memory. Colors mark roles. Blue is the actor at the edge, the client apps and agents that send prompts and consume streamed tokens. Amber is the queue that decouples bursty arrivals from the scheduler. Green is the state the system owns: model weights, the KV-cache in GPU memory, the caches, and the metrics. Violet is the services that admit, route, batch, and run the work.

The inference path: prefill, decode, and batching

On a worker, a model call runs in two phases that behave nothing alike. Prefill comes first. The whole input prompt goes through the model in one compute-heavy pass, parallel across the input tokens, and builds the KV-cache the rest of the request reuses. Decode follows, one output token at a time. Each step is cheap in compute but bound by memory bandwidth, since it reads the whole model and the growing KV-cache to produce a single token. Prefill sets time to first token. Decode sets the stream after it. Because they stress the GPU so differently, a naive server that handles one request at a time wastes most of the hardware.

Batching fixes that, and continuous batching is the version to name. Static batching collects a fixed group and runs it to completion, so when requests finish at different lengths the longest one holds the whole batch hostage and the GPU sits partly idle. Continuous batching works at the token step. Every step runs all active requests together. When one finishes it's evicted, and a waiting request takes its slot. The batch can only grow as large as KV-cache memory allows, so memory is the scheduler's hard limit and cache management ends up setting throughput.

Caching prefixes and whole responses

Caching is how you stop paying twice for the same work, and an LLM server has two caches that cut very different costs. The response cache sits at the gateway and stores whole completions keyed by the exact request, model, and decoding parameters. When an identical, deterministic request returns, it comes back instantly and spends zero tokens. It applies only to safe cases, low or zero temperature with no per-user personalization. On Scale's agent traffic, where one templated prompt runs across many documents, those cases are common and a full model call disappears.

The prefix cache sits on the worker and cuts a subtler cost. Agent and evaluation prompts often share a long common head, a fixed system prompt, a block of instructions, a retrieved document that many questions ask about, followed by a short unique tail. Without caching, prefill over that head is redone on every request. The prefix cache stores the head's KV-cache, so the worker loads it and prefills only the unique tail. Time to first token and GPU cost both drop. Keep the two straight in the room. The response cache removes the whole call. The prefix cache removes only the repeated prefill.

Serving cost controls

GPU hours dominate the bill. Every cost lever is some way of generating fewer tokens, on cheaper hardware, without losing more quality than the workload can spare. A strong answer stacks several of them and says what each one gives up. Stacked, they're what separate a demo-priced system from a production-priced one.

• Cache first: remove the call, then the prefill. The response cache removes a whole model call for repeated deterministic prompts. The prefix cache removes repeated prefill for shared prompt heads. Both cost only memory and pay back directly in GPU hours, so reach for them first. The risk is staleness, bounded by caching only deterministic, non-personalized work.
• Route by difficulty: small model for the easy majority. Not every request needs the largest model. A router sends short classifications and simple rewrites to a smaller distilled model and saves the frontier model for hard reasoning. This is often the biggest lever after caching. Misrouted requests risk quality, so tune the router conservatively and measure accuracy per route.
• Quantize the weights: more batch per GPU. Serving at int8 or int4 shrinks the resident weights, freeing memory for a bigger KV-cache and a larger batch. Throughput rises and cost per token falls. The trade is a small, measurable quality loss, so quantize tolerant traffic and keep a full-precision path for quality-critical calls.
• Batch the patient work: throughput over latency for bulk jobs. Bulk evaluation and offline generation have nobody waiting, so they run at maximum batch size behind the interactive traffic, getting the most tokens per GPU-second. They give up latency on purpose. That also keeps them out of the way of requests that can't wait.

The Scale AI angle

Design the cost controls into the architecture, because at Scale the bill is a product constraint. Treat the token bill the way a consumer engineer treats latency, as a number the design answers to at every step. The gateway meters and caps tokens per tenant, the response cache removes repeated calls, and the router sends the easy majority to a cheaper model. Quantization fits more batch on each GPU, and the metrics pipeline accounts for every token. Name each as a measured trade, a defined slice of quality or latency for a defined saving. Send every request to the biggest model, bring up cost only when asked, and you've designed a demo.

Reliability: timeouts, fallbacks, and autoscaling

A model call is slow, heavy, and occasionally stuck, and the reliability story has to fit that. Every request carries a deadline. A generation that blows past it gets cut off with a partial result or a graceful error, because a runaway request otherwise holds KV-cache memory the whole batch needs. Blind retries are dangerous, since each one is another full, expensive generation, so retry only fast pre-generation failures and never automatically re-run a completed call. When a large-model worker is overloaded or a model version is failing, fall back to the smaller model or a cached or degraded response so the product keeps answering.

Then handle load at the fleet level. Admission control at the gateway is the first defense. When the queue is deep and GPUs are saturated, shed or delay low-priority work so interactive traffic keeps its latency. Autoscaling is the slower lever, because GPU scaling isn't instant. A new worker loads tens of gigabytes of weights before it serves a token, so scale on queue depth and utilization ahead of demand and keep a warm buffer. Admission control handles the spike. Predictive autoscaling handles the trend.

Bottlenecks and tradeoffs

A happy-path answer is a weak one. Say where this system strains and what you gave up to relieve it. The pressure points all come from serving a model on scarce GPUs, which generic web-scale prep never touches. Raising them before the interviewer does reads as senior.

• Throughput versus latency: the batch-size dial. A bigger batch uses the GPU efficiently and raises tokens per second. It also lengthens each request's wait per step, so pushing throughput can hurt the interactive latency SLO. Tune batch size to protect time to first token for interactive traffic while bulk jobs run at maximum batch. That's why the queue and scheduler are priority-aware.
• Cost versus quality: the routing and quantization dial. Routing to a smaller model and quantizing the weights both cut the bill, and both can cost accuracy on requests that needed the full model at full precision. Measure the trade. Route conservatively, quantize only tolerant traffic, and keep a full-quality path for critical calls, with per-route accuracy watched in the metrics.
• KV-cache memory: the real concurrency ceiling. GPU memory caps concurrency before compute does. The resident weights plus every in-flight KV-cache fill it, and a few very long contexts can starve the batch. Manage the cache actively by paging KV in blocks, reusing shared prefixes, and bounding max context. You accept complexity to fit more requests per GPU.
• Cold starts and scarce GPUs: warm buffer versus idle cost. GPUs are expensive and slow to warm, since weights take tens of gigabytes to load, so scaling can't react instantly to a spike. Keep a warm buffer of headroom that costs money while idle. You buy it on purpose so a burst doesn't blow the latency SLO while a cold worker loads.

What the interviewer probes

Follow-ups tend to push on those pressure points, and the answers are already in your design. If asked how you hold time to first token on a saturated GPU, point to admission control, priority queuing, and an interactive-tuned batch size. "The bill is doubling, cut it without a rewrite" gets the cache, router, and quantization levers as a stack of measured trades, and "what caps concurrency?" gets the KV-cache-versus-weights math and continuous batching. A new model version or a second tenant's own model is handled by the warm-load weights store, the router, and per-tenant metering. If the prompt bends toward an agent over documents, add retrieval and a token budget in front of the serving core you already built.

The Scale AI angle

Reason from cost and correctness. The round checks whether you can make a system built around a large, slow, expensive, fallible model work in production. A diagram rehearsed for a consumer app won't get you there. Put the model call at the center. State its job, then size tokens per second and GPU memory, flagging the numbers as illustrative. Decide what must be fast for a person and what can run as batch before you draw anything. Keep the token bill in view, say the caching, routing, and quantization trades out loud, and design reliability for a call you can't cheaply retry.