Design a GPU Cloud System

Understanding the prompt

CoreWeave rents GPU compute to AI labs, so the design round hands you the company's actual problem. Forget sizing a photo feed or a chat cache. You are given a pool of physical machines, each carrying several GPUs, wired together by a high-speed fabric. Your job is to turn that pool into a cloud. Accept jobs from many tenants, place each job on free GPUs, keep the expensive hardware busy, and operate the fleet safely as machines fail and get cycled. The prompt is thin and under-specified on purpose. The first move that reads well is to name the resource and its economics before naming any service.

Say the constraint plainly. A GPU sitting idle burns money every second, and a distributed training run may hold hundreds of GPUs for days. So the design is graded on how fully you keep the fleet utilized and how safely you place and protect work on it. Requests per second are not the measure. Two facts about the workload shape everything downstream. A large training job is a gang: it wants its whole set of GPUs at once, because the workers run a synchronized collective and one missing worker stalls all the rest. And those GPUs are not interchangeable, because the job's speed depends on the workers sitting close on the fabric so the collective stays fast. Utilization, gang placement, and fabric locality are the three ideas the whole answer hangs on.

The CoreWeave angle

The fleet is the asset, so utilization and safe placement are the grade, ahead of throughput. Picture the meter running. Every GPU in the pool costs a fortune whether it computes or idles, and a customer has been paying for a training run that has been going for three days across four hundred GPUs. That reframes the design away from anything you rehearsed for a web app. The weighty questions are operational. How do you pack jobs onto the fleet without stranding fragments too small to use? How do you place a distributed job so its collective does not choke on the network? And how do you take a machine out for maintenance without murdering the run sitting on it? Aim the whole conversation there. Talk fluently about capacity, gang scheduling, fabric topology, and protecting live work, and you are answering the question CoreWeave is asking.

Scoping the requirements first

Resist the urge to start naming Kubernetes components. Spend a minute bounding the problem and the design stays coherent instead of sprawling. State the functional surface. Tenants submit jobs, and a job declares a shape: how many GPUs, of which type, for roughly how long, at what priority, as one gang or as independent workers. The system places each job on free GPUs that satisfy that shape, runs it, reports its state, and reclaims the GPUs when it finishes or is preempted. Operators drain and reboot machines for maintenance without breaking running jobs. That is the core loop, and it is enough to design something real.

Then draw the line around what you handle now and what you name and defer. Do not try to specify the exact bin-packing heuristic, the full multi-dimensional fair-share policy, or the precise fabric routing in the first pass. Sketch the skeleton so each piece has an obvious home, then say so out loud. State that scheduling starts with a simple fit-and-pack pass and grows into topology-aware gang placement with preemption. Fairness starts with per-tenant quotas and grows into weighted share. Both have a named place to live; naming the deferral is the judgment the round rewards. Pin the non-functional bar too, because for this system it carries most of the weight.

• The scarce resource: utilization, not QPS. The binding constraint is how fully the fleet stays busy. An idle GPU is money burned, so the design is judged on packing work densely onto the pool and reclaiming capacity fast. Requests per second are secondary. Size the fleet and its utilization first, out loud, and let it drive the rest.
• Gang, all-or-nothing: whole job or none. A distributed training run is a synchronized gang: it needs every GPU it asked for at the same time, because one missing worker stalls the collective and idles the rest. Placing half a gang wastes the half you placed, so scheduling is all-or-nothing, which is the constraint a generic queue never has.
• Multi-tenant: fairness and isolation. Many labs share one pool, so no tenant can starve the others or see their data. The scheduler enforces per-tenant quotas and a fair share of contended capacity, and placement isolates each job on its own GPUs. Naming this early keeps utilization from turning into a free-for-all.
• Fabric locality: placement is topological. GPUs are not interchangeable. A gang runs a collective over InfiniBand, and its speed depends on the workers sitting close on the fabric, ideally one rail or one rack. Placement has to respect topology, which is the concern a web-scale prompt, where any server is as good as any other, never forces you to confront.

Estimation, with illustrative numbers

Before architecture, do the accounting out loud, and frame every number as a back-of-envelope illustration rather than a CoreWeave figure. Nobody expects you to have memorized the real fleet size. The interviewer is checking whether you can reason from an assumed pool to the pressures that decide the design. How many GPUs, how big a job can be, how much idle capacity costs, and how fat the fabric is. State the assumptions, keep the numbers round, and let them steer the choices. A candidate might reason as follows.

Assume, for a single cluster:
  GPUs per node       ~8 GPUs, one high-speed NIC per GPU
  nodes               ~5,000
  ------------------------------------------------------------
  ~40,000 GPUs in the pool (real fleets run much larger)
 
Job shapes to place (illustrative):
  small inference     1 to 8 GPUs, minutes to hours
  mid training        64 to 256 GPUs, hours to days
  large training      512 to 2,048 GPUs, days to weeks, one gang
 
Cost pressure:
  every idle GPU-hour is paid-for capacity earning nothing
  target very high utilization, well above 90%
  -> a few points of stranded capacity is a large bill

Two conclusions fall straight out of that. First, packing density is the whole economic game. At forty thousand GPUs, leaving even a few percent stranded in fragments too small to hold a gang is a serious recurring cost, so the scheduler is judged on how little it wastes. Second, the fabric is a first-class dimension of placement, because a large gang's speed lives or dies on it.

Collective bandwidth (illustrative):
  a training step ends in an all-reduce over the whole gang
  each worker's NIC moves hundreds of Gbps over InfiniBand
  a gang spread across the fabric hits slower cross-rack hops
  -> place the gang close: same rail or rack where possible
 
Checkpoint pressure (illustrative):
  a large model checkpoints model + optimizer state
  tens to hundreds of GB, written every ~15 to 30 min
  -> checkpoint I/O must not stall training, and a restart
     from the last checkpoint must cost minutes, not the run

Those two numbers set the spine of the design. Utilization pushes the scheduler toward dense packing and reclaiming capacity aggressively. Fabric bandwidth pushes it toward keeping each gang topologically tight, which fights dense packing, because the tightest placement is not always the one that wastes the least. Holding that tension in view, pack densely but place gangs close, is what makes the later decisions have obvious answers.

The high-level architecture

With the constraints named, the architecture is the wiring that enforces them, so draw it and read it by role. Blue is an actor: the AI lab that submits work and the physical GPU fleet that runs it. Amber is the queue that holds pending jobs so the scheduler places from a batch instead of racing arrivals. Green is the state the system owns: the authoritative record of the fleet, its topology, and every allocation, plus the object store that feeds datasets and holds checkpoints. Violet is the services that move and place work. Follow the flow top to bottom. A job enters through the control-plane API into the pending queue. The gang scheduler reads fleet topology from the state store and binds a whole gang through Kubernetes. Node agents launch the pods on GPUs and report health. A fleet-health controller drains and reboots machines by updating the same state the scheduler reads.

The scheduler: gang, packing, preemption, topology

The scheduler is where this design stops resembling anything generic, so spend your time here. A web load balancer answers one small question, which healthy server takes the next request. This scheduler answers a harder one. Given a queue of jobs and a live map of a fragmented fleet, which specific GPUs does each gang land on, all at once? The answer has to keep the pool busy and the fabric fast. Walk the four forces it balances, because the interviewer will push on each.

Put those together and a placement pass looks like this. Take the highest-priority pending job and find candidate GPU sets that fit its shape. Score them by how tightly they sit on the fabric and how little fragmentation they leave behind. If none exists, decide whether preempting lower-priority work opens a viable set. Commit the winner atomically, update the cluster state, and move to the next job. The loop is simple to state and subtle to tune, which is exactly why the round uses it: it forces you to reason about scarce, non-interchangeable resources under a hard all-or-nothing constraint.

The CoreWeave angle

Defend the gang and the fabric, because that is what a GPU cloud gets wrong when it treats jobs like web requests. A tempting wrong answer places GPUs greedily, one worker at a time onto whatever is free, the way a load balancer hands out requests. On this workload that quietly destroys value: you strand half-placed gangs, you scatter a training job across the cluster so its all-reduce crawls, and you fragment the pool until no large job fits. Say the opposite out loud. A gang is placed atomically or not at all. It is placed close on the fabric even when a looser spot would pack marginally denser, because a fast collective is worth more than a point of packing. Then make preemption honest by leaning on checkpoints, so reclaiming a GPU costs the preempted job minutes instead of its whole run. A candidate who protects the gang and the fabric first, then optimizes utilization around them, is reasoning like someone who has operated this hardware.

Fleet health and batch reboots

Machines fail constantly at fleet scale, and maintenance never stops, so operating the pool is a first-class part of the design rather than an afterthought. This is also the one CoreWeave system-design prompt candidates have reported, designing batch reboots across a machine fleet, so treat it as the heart of the answer. The hard requirement is blunt: no in-flight customer job dies because you cycled a machine underneath it. Everything in the health path exists to honor that.

Walk the maintenance loop in the order that keeps it safe. The node agents report health continuously, so a GPU throwing correctable errors, a flapping NIC, or a firmware update due surfaces as a signal before it becomes a surprise outage. The health controller decides which machines to cycle and then moves them through a strict sequence rather than rebooting on a timer.

Storage for datasets and checkpoints

Storage is easy to wave away and expensive to get wrong, because a GPU waiting on data is a GPU wasted, and idle GPUs are the one thing this design cannot tolerate. The store has two jobs that pull in different directions. On the read side it has to feed training datasets to many GPUs in parallel at high throughput. On the write side it has to absorb periodic checkpoint bursts without stalling the very jobs it protects. Name both and show they do not fight for the same path.

Bottlenecks and tradeoffs

A design that only walks the happy path is a weak answer, so name where this system strains and what you traded to relieve it. The pressure points come from the utilization-and-placement shape of a GPU cloud, and generic web scale does not produce them. Calling them out before the interviewer does reads as senior.

• Utilization versus fairness: pack hard, but share. Packing purely for utilization lets a few large tenants dominate the pool; enforcing strict fairness leaves capacity idle. The tradeoff is priority and weighted fair-share layered on top of packing, so the fleet stays busy while no tenant starves. No setting maxes both at once. Say so out loud.
• Fragmentation versus gangs: defend the big blocks. Filling every gap with small jobs maximizes instantaneous utilization but shatters the fleet so no future gang fits. The tradeoff is to sometimes leave a whole node free, accepting short-term idle GPUs, to keep space for the large all-or-nothing jobs that pay the most. Optimizing this minute can starve the next hour.
• Preemption cost: reclaim, but not for free. Preempting a low-priority job frees capacity fast, but the preempted job loses everything since its last checkpoint and has to re-queue. The tradeoff is bounded by the checkpoint interval and by preemption policy: reclaim aggressively enough to keep utilization high, gently enough that low-priority work still makes progress.
• Locality versus flexibility: close is faster, tighter. Insisting every gang sit on one rail makes collectives fast but makes placements scarce and can leave a job queued behind a locality it cannot get. The tradeoff is a locality preference with fallbacks, tight when the fabric allows, looser when the alternative is an idle gang, rather than a hard rule that strands work.

What the interviewer probes

The round escalates by pushing on the pressure points, and the strong answers are already latent in the design if you built it in this order. Expect the pushes to cluster. On gang scheduling: what happens when a gang cannot fully fit, which the atomic all-or-nothing commit and the queue answer, the job waits rather than half-placing. On fragmentation: how do you stop the pool decaying into unusable fragments, which the packing policy and defending whole nodes answer. On maintenance: how do you reboot a fleet without killing a three-day run, which the cordon-drain-checkpoint sequence in capacity-aware waves answers. On the fabric: why does placement care about topology at all, which the bandwidth-bound collective answers. And on evolution: what changes when GPU types are mixed or a new fabric generation arrives? The state-driven, topology-aware scheduler absorbs it by treating capability and locality as data instead of assumptions baked into code.

The CoreWeave angle

Reason from the hardware and the design defends itself. This round checks whether you can look at a pool of physical machines wired together for training and build something operable on top. Pattern-matching to a reference design you rehearsed for a consumer app will not get you there. So make the fleet the first thing you talk about. Count the GPUs, state that an idle one costs money, decide how big a gang can be, and only then draw the scheduler. Place distributed jobs atomically and close on the fabric, and pack to defend the big free blocks. Preempt with checkpoints as your safety net; cycle machines in drained, capacity-aware waves that never kill a live run. Start from the physics of a GPU fleet: scarcity, all-or-nothing placement, fabric bandwidth, and safe operations. A candidate who does that, instead of reaching for a generic traffic-scaling template, reads as someone who could own a piece of the systems CoreWeave runs.