Design a Real-Time Multiplayer Game Server

Understanding the problem

A Roblox experience is a creator-built 3D game (an obstacle course, a role-play town, a tycoon) that a group of players joins together. Each running copy of that experience is an instance, and its server is the authority. It holds the true positions of every player and object, steps the physics forward on a fixed clock, and tells each client what the world looks like. Roblox interviewers point the system design round straight at their own platform instead of a generic web service. The multiplayer backend is the deepest of the gaming shapes, so treat it as the flagship real-time design.

The whole job sits between a player pressing a key and every other player in that instance seeing the result, over an unreliable network, fast enough that nobody notices the round trip. This is not a feed or a chat app. There is no leisurely read path to cache. The world changes on every tick, the update has to reach a client within tens of milliseconds, and the client code running on a kid's laptop cannot be trusted to report the truth. Three constraints drive everything else: latency, a fixed simulation clock, and an authoritative server.

The Roblox angle

The server owns the truth because the client is running untrusted code written by a creator and played by a child, so trusting the client would hand exploiters the game. Anyone can open Roblox Studio, script an experience in Luau (Roblox's typed Lua dialect), and run a modified client. Suppose the client could say "I moved here, I hit that player, I collected that coin" and the server just believed it: cheating would be trivial, and the youngest players would be the easiest to grief. So the server simulates the world itself and treats each client message as a request, not a fact. State the authoritative-server model in the first two minutes and tie it to the young user base you can't trust by default. That shows you understand why a Roblox game server looks nothing like a stateless API tier.

Functional requirements

Keep the scope on one instance and the loop that keeps it in sync. A player is already routed to a server (matchmaking and placement are their own design); here we take the connection from there. The server accepts inputs, runs the simulation, replicates state to everyone, and keeps each client's view believable despite lag. Everything else is a thread the interviewer can pull.

Non-functional requirements

Reason each target from the feel of the game. Latency is the defining constraint. A player expects their own avatar to react the instant they press a key, and to see other players move within a frame or two. So the end-to-end budget from input to visible result sits in the low tens of milliseconds locally and roughly a hundred milliseconds for remote players. That budget is what forces prediction on the client and a nearby server. The simulation runs on a fixed clock instead of on request. The server steps physics at a steady tick whether or not input arrived (Roblox replicates player characters at about 20 Hz, a step every 50 milliseconds). A shared world needs one authoritative timeline everyone reconciles against.

Consistency here is unusual. The server is strongly consistent with itself, since it holds the one true state. Every client, though, is a little behind and a little ahead of it at once. It is behind because snapshots take time to arrive, and ahead because it predicts its own next move before the server confirms it. That gap is the whole design, not a bug to remove. Availability is per instance: if one instance dies, only its players drop, and they can be re-placed into a fresh one, so the blast radius is a single game session rather than the platform. And it all has to hold under Roblox's real peaks. Tens of millions of players are in instances at once, and a single hit experience can pull tens of millions into the same game.

• Tick rate: ~20 Hz, a step every 50 ms. The server advances the simulation on a fixed clock. Roblox replicates player characters around 20 Hz and server-driven projectiles and NPCs at 10 to 20 Hz, so a snapshot goes out roughly every 50 milliseconds regardless of when input arrives.
• Latency: Tens of ms local, ~100 ms remote. Local movement must feel instant, which is why the client predicts. Remote players can lag by a round trip, which is why the client interpolates between the snapshots it receives instead of snapping.
• Instance size: Tens of players. A typical experience caps an instance in the tens (often around 30 to 50), some push to hundreds. This bound is what keeps per-tick replication affordable, and raising it is a real deep dive.
• Consistency: Server-authoritative, clients reconcile. The server holds the one true state. Every client is intentionally behind on remote objects and ahead on its own, and corrects toward the server whenever the two disagree.

The set-up

Core entities

Name the objects before drawing. Four carry the design, and the one that does the real work is the snapshot, the per-tick description of the world that the server ships to each client.

The split that organizes everything is upstream inputs versus downstream snapshots. Upstream, a client sends small, frequent input messages: a movement intent, a fired action, a request to interact. These are proposals, and the server validates each one against its own state before applying it. Downstream, the server sends snapshots: after each tick it computes what changed and pushes the relevant slice to each client. The two halves are asymmetric on purpose. Input packets are tiny and one per client. Snapshots are the heavy traffic and fan out to everyone. That is why almost all the optimization effort goes into what a snapshot contains.

API

The surface between a client and its server is small and lives on a persistent connection, not a request per action. Roblox exposes this to creators as RemoteEvents, the client-to-server and server-to-client messaging primitive, and each RemoteEvent carries a real replication cost, so the interface is deliberately thin. The structure worth showing is the loop: inputs flow up as proposals, the server steps the world, and snapshots flow down at the tick rate.

The pair to dwell on is sendInput and snapshot. The ackSeq field ties them together. Each input carries a sequence number, and every snapshot echoes back the last sequence the server has processed. That one number is what lets a client know which of its predicted moves the server has confirmed and which are still outstanding, which is the entire basis of reconciliation later. Treat the connection as a continuous exchange of proposals and authoritative snapshots, not a set of REST calls, and the rest of the design follows.

High-level design

Draw the smallest version that shows the loop and the edge-versus-core split, then go deep on replication and prediction, because those are the parts that make this Roblox. A player connects to a game server in the nearest region, then the loop runs. The client sends inputs, and the server validates them and steps the authoritative simulation on its tick. It replies with a snapshot of the relevant world. The client renders its own avatar from its prediction and draws everyone else by interpolating between the snapshots it receives. The heavy machinery (economy, identity, moderation) sits in the core and is only touched when play needs it.

Color marks the role. Blue is the player client, predicting and rendering. Violet is the services, the edge game server that owns the simulation and the region router that points a client at the nearest one. Amber is the replication channel, the reliable-UDP transport carrying snapshots down. Green is state, the per-instance session state, the asset CDN the client loads the world from, and the core services behind them.

1) Connect to the nearest instance. A player who has been matched into an instance asks the region router to resolve it, and the router returns the address of a game server in the region closest to the player. This is a latency decision made once at join: a server near the player shrinks the round trip that everything downstream has to hide. A player in Sao Paulo lands on a South American edge instead of reaching across to a core data center. The client also pulls the static world (the place file, meshes, avatar assets) from the asset CDN here, so the gameplay connection never carries art. What is punted is how the instance was chosen and provisioned in the first place, which the matchmaking design owns.

2) Send inputs as proposals. Many times a second the client samples local input (a movement direction, a jump, a fired action) and sends it up as a small packet stamped with a sequence number. The game server does not trust these packets. Each one is a request, not a result. It validates each input against its own authoritative state, then applies it. Is the move physically possible from where the server thinks the player is? Is the action allowed, and is the target in range? An input is a few dozen bytes, one per client per tick, so upstream traffic is tiny compared to what comes back. What is deferred is what the client does in the meantime, since waiting for confirmation would make the game feel sluggish, and prediction handles that gap.

3) Step the simulation on a fixed tick. Independently of when inputs arrive, the game server advances the world on a steady clock, around 20 Hz, a step every 50 milliseconds. On each tick it applies the validated inputs, runs physics, resolves collisions, and fires game events, producing the one authoritative snapshot of reality that everyone will reconcile against. The fixed clock matters because a shared world needs a single timeline: if the server stepped only when input showed up, different clients would disagree about when things happened. What is kept off this hot path is anything durable or cross-instance (writing a purchase, saving progress), which is punted to the core so the tick never blocks on a network round trip.

4) Replicate the relevant world back. After stepping, the server computes what each client needs and pushes it down the replication channel as a snapshot, usually a delta against what the client last acknowledged, not the full world. It sends each client only the entities relevant to it, not everything in the instance, which is what keeps per-tick bandwidth bounded as the world grows. The client renders its own avatar from its prediction and draws remote players by interpolating between the snapshots it has. The snapshot echoes the last input sequence the server processed, which is the hook reconciliation uses. The deep dives take this apart, starting with why the server has to own the truth at all.

The Roblox angle

Where a component runs is a real answer at Roblox, because the platform is physically split into an edge that holds game state and a core that holds money and identity. Roblox runs on the order of two dozen edge data centers that host the game servers. A small number of core data centers run the website, recommendations, safety filters, the virtual economy, and publishing. The rule is clean: the simulation lives at the edge, close to the player, where a 50-millisecond tick is affordable. Anything that needs a single strong authority (a Robux balance, an account, moderation of record) lives in the core. When the interviewer asks where a piece runs, placing the tick loop at the edge and the purchase authorization in the core, and explaining why each belongs there, is the differentiator most candidates miss.

Size the tick loop

A few numbers set the shape of both directions of traffic. Start from one client's downstream. Say a busy instance shows a player about 40 relevant entities at once. Each entity's state (an id, a position, a velocity, an orientation) packs into roughly 24 bytes. That is about 1 KB of world per snapshot. At the 20 Hz tick that is 20 KB per second, near 160 kbps down to each player. This is why snapshots are deltas and why area-of-interest trimming is not optional. Send the full world instead of the relevant slice and this number grows with the size of the map rather than staying flat.

Now scale it to an instance. With around 50 players each pulling roughly 20 KB per second, one instance pushes on the order of 1 MB per second, roughly 8 Mbps, out of a single game server. Upstream is trivial by comparison. A few dozen bytes of input per client per tick is well under 100 KB per second for the whole instance, so the engineering pressure sits on the downstream snapshot. Then scale to the fleet. Roblox has peaked around 30.6 million concurrent players. At tens of players per instance, that is on the order of hundreds of thousands of live instances spread across the edge data centers. A single record-setting experience alone drew over 21 million simultaneous players. The takeaway the numbers force is that the cost is per-tick fan-out of state, so every decision pushes toward sending less, less often, to fewer clients.

• Per-player downstream: ~160 kbps. About 40 relevant entities at ~24 bytes each is ~1 KB per snapshot, times 20 Hz. This is why snapshots are deltas and trimmed to an area of interest rather than the full world.
• Per-instance downstream: ~1 MB/s out of one server. Roughly 50 players times ~20 KB per second. Upstream inputs are under 100 KB per second for the whole instance, so the snapshot fan-out is the whole cost.
• Tick budget: 50 ms per step. At 20 Hz the server has 50 milliseconds to apply inputs, run physics, and build every snapshot. Anything durable or cross-instance is punted off this window to the core.
• Fleet: Hundreds of thousands of instances. Peaks near 30.6M concurrent players at tens per instance. One hit experience alone has drawn 21M-plus simultaneous players, which is why capacity has to burst.

Technology choices

On this board every node is a category rather than a fixed product, and most map onto tools Roblox runs in production. Pick a well-known one, justify the category it stands for, and rehearse the failure case, since the interviewer spends the back half there.

• The replication channel is UDP with a thin reliability layer rather than TCP, which is what Roblox uses under its RemoteEvent primitive. A datagram transport fits because a late position update is worthless. By the time TCP retransmitted a dropped snapshot, a newer one has made it stale, so head-of-line blocking would add latency for data nobody wants. The design layers reliability selectively, marking a handful of events (a score change, an item pickup) as guaranteed-delivery while ordinary movement stays fire-and-forget. The failure to reason about is heavy packet loss. With deltas, a client that misses several snapshots drifts, so the protocol lets it request a full resync and the server sends an absolute snapshot to re-anchor it.
• The game server is a stateful, single-threaded-per-instance simulation pinned to one machine at the edge, not a horizontally scalable stateless tier. It holds the authoritative world in memory because a 50-millisecond tick cannot round-trip to a database, and one instance maps to one process so the physics step has a consistent view without locking. It scales out by running more instances, not by sharing one instance across servers. The failure to plan for is the process dying mid-session. An instance is a self-contained failure domain, so only its players drop, and they are re-placed into a fresh instance. Anything durable (saved progress, inventory) was already written through to a datastore, not trapped in memory.
• The asset CDN is a standard content delivery network (edge caches fronting object storage) serving the versioned place file, meshes, textures, and avatar parts. A CDN fits because millions download the same popular experience and the assets are immutable once published. Caching them near players is far cheaper than streaming from origin, and it keeps static art off the gameplay channel. The failure to name is a cache miss storm when a brand-new experience goes viral. A shield or origin tier collapses the region's misses into a single origin fetch. Immutable, content-hashed asset URLs mean a version can be cached effectively forever with no invalidation problem.
• The core services are the strongly consistent systems in Roblox core data centers, the virtual economy, identity, catalog, and moderation, reached only when play needs them. A separate, more consistent tier fits because money and identity cannot tolerate the eventual, per-instance looseness that game state runs on: a Robux balance needs one authority and exactly-once writes. The failure to handle is the core being slow or unreachable mid-game. The tick loop must never block on it, so a purchase is an async authorize call with a timeout. If the core cannot answer, the action is deferred or declined so it never stalls the simulation everyone else is in.
• The edge fleet itself is organized into self-contained failure domains that Roblox calls cells. A deployment tool automatically stripes services across them, and cloud bursting spins up virtual edge capacity on public cloud for weekend peaks. Cells fit because a fault in one is contained like a fire door instead of cascading across the region. The resiliency stack around them (adaptive concurrency control, circuit breakers, retry shedding, and a chaos-testing platform) keeps a 30-million-player Saturday from turning one bad node into an outage.

Name the brand, defend the category. The genuinely Roblox pieces here are the RemoteEvent-over-reliable-UDP channel, the edge game servers organized into cells with cloud bursting, and the edge-versus-core split that keeps the economy in the core. The CDN and the datastore write-through are standard components doing a familiar job. The through-line comes from the requirements. The server owns the truth and the client is never trusted. Every choice bends toward getting authoritative state to nearby players within a tick.

Potential deep dives

The deep dives are the round. A Roblox interviewer picks one box and drills. Expect the push on why the server has to be authoritative, on what actually goes into a snapshot each tick, and on how prediction and reconciliation hide latency without letting clients lie. Then region placement and surviving a crash, and what breaks when you try to pack more players into one instance. Take them in that order.

1) Why the server owns the truth

The core question is who decides what happened when two players disagree. Walk it from the naive version up, because the right answer depends on Roblox's specific reality, that the client is untrusted code played by children, and you have to say why.

The Roblox angle

The reason authority is non-negotiable at Roblox, and optional in a trusted-console game, is entirely about who is on the other end of the wire. Roblox experiences are made by outside creators in Studio and played by an audience that skews young, on ordinary personal devices where the client can be inspected and modified by anyone curious enough. There is no trusted console, no signed game binary, no assumption that the player is an adult who will not exploit a child. So the server cannot afford to believe the client about anything that touches the shared world or the economy. Frame authority as the enforcement layer that protects the youngest players from a modified client. That ties a standard game-networking pattern to the specific thing Roblox worries about.

2) What to replicate each tick

The server steps the world 20 times a second and has to tell every client what changed, without melting the network. The question is what goes into a snapshot. From the capacity math, sending the full world to everyone grows with the size of the map. So the whole game is sending less.

Three techniques stack, and a strong answer names all three rather than stopping at one. First, deltas: instead of the absolute state of every entity, the server sends only what changed since the client's last acknowledged snapshot, so a mostly-static world costs almost nothing to replicate. The client's ackSeq tells the server which baseline to diff against, and a client that has drifted too far (after heavy packet loss) can ask for a full absolute snapshot to re-anchor. Second, quantization. Positions and orientations do not need full floating-point precision on the wire. Pack a coordinate into a couple of bytes and an orientation into a couple more, and each entity shrinks to roughly a couple dozen bytes with no visible loss.

Third, and most important, area-of-interest (relevance) filtering: a client is only sent the entities near its player, not the whole instance. The server keeps a spatial index of the world (a grid or similar). Each tick it gathers the entities within a player's view radius, so a 200-player world still costs each client only the 30 or 40 things it can see. This is what makes per-player bandwidth flat in the size of the map instead of linear. Priorities layer on top: a fast-moving nearby projectile updates every tick, a distant slow object every few ticks, so the tick budget is spent where the player will notice. New content ties in here too, since avatar parts and assets stream from the CDN and only their dynamic state replicates, keeping art off the gameplay channel entirely.

The Roblox angle

Roblox's own scaling pain shows why relevance filtering is the right instinct: their friends-on-a-leaderboard feature was expensive because it fanned reads out across a user's entire friend list. The fix was to stop reading what did not matter. They shipped a GetFriendsWhoPlayed call that reads only the friends who played the experience, not every friend. Area-of-interest replication does the same thing: compute the relevant subset instead of the whole set. In a design round, say "only send each client the entities in its area of interest" and tie it to Roblox's habit of trimming fan-out. That shows you understand how the platform keeps consumer-scale features affordable, beyond a game-networking checklist.

3) Client prediction and server reconciliation

Server authority buys fairness but costs a round trip: if the client waited for the server to confirm every move, a player 100 milliseconds away would feel their own avatar lag behind their fingers. The question is how to make local control feel instant while the server still owns the truth.

The answer is client-side prediction with server reconciliation, and it hangs on the sequence numbers from the API. When the player presses a key, the client applies the move locally right away, so the avatar responds within a frame. It stamps that input with a sequence number and keeps it in a pending list. The input also goes to the server. The server validates and applies it on its next tick, then in its snapshot echoes back the last sequence it processed (ackSeq). When that snapshot arrives, the client compares the server's authoritative position for its player against where its own prediction had put it at that same input. If they agree, which is the normal case, the player never notices. If they disagree (the server rejected or altered the move, or a collision the client did not predict), the client snaps to the server's position. Then it replays all the still-pending inputs on top of that, so it stays responsive yet ends up consistent with authority.

Remote players get the opposite treatment: entity interpolation. A client cannot predict what another player will do. Snapping other avatars to each new snapshot would look jerky at 20 Hz. So it renders them a fraction of a second in the past and interpolates between the last two snapshots it has. That trades a tiny, unnoticeable delay for smooth motion. For fast interactions where that delay would matter (did my shot hit a moving player), the server does lag compensation. It rewinds the world to the tick the firing client actually saw, checks the hit against those historical positions, then resolves it, so a well-aimed shot on a laggy connection still counts. Prediction hides your own latency, interpolation hides everyone else's, and lag compensation reconciles the two when they collide.

The Roblox angle

This is exactly the vocabulary Roblox uses to describe its own engine, so reaching for it by name signals you know the platform and not just generic multiplayer theory. Roblox replicates player characters automatically at around 20 Hz and uses client-side interpolation between snapshots for server-driven objects. Its networking model is built around client prediction and server reconciliation over a reliable-UDP transport. An interviewer who works on the engine will notice whether you say "the client predicts locally, keeps pending inputs, and replays them after the server acknowledges" versus waving at "sync the state." The precise chain is hardest to fake and shows depth most clearly. Predict then reconcile for the local player, interpolate for remote players, and lag-compensate for hit detection.

4) Region-based instances and surviving a crash

Two operational realities decide this dive. Players are spread around the planet. Game servers are ordinary machines that fail. The question is how placement keeps latency low and how one instance dying does not take the experience down.

Placement is a latency decision made at join. The prediction-and-interpolation machinery exists to hide the round trip, so the cheapest help is to shorten the round trip itself, so an instance runs on a game server in the edge data center nearest its players. Roblox runs game servers across roughly two dozen edge data centers worldwide for exactly this reason, and the region router hands a joining player the closest one. Players who want to be together (a party) go into the same instance, in a region that is a reasonable compromise for the group. That is a matchmaking concern this design consumes but does not solve. The point to make is that region placement is not a nicety: it directly sets the latency budget every other technique in this lesson is working within.

Crash survival rests on the instance being a self-contained failure domain. An instance holds its authoritative world in memory on one server. If that process dies, that world is gone, but only that one. The blast radius is a single game session, not the experience. The players who dropped are re-placed into a fresh instance and rejoin. Durable state (saved progress, inventory, currency) must not be lost. The server writes it through to a datastore or the core as it changes, so a crash costs at most the last few seconds of in-progress play. Roblox goes further at the fleet level with cells, self-contained failure domains it likens to fire doors. Services are striped across them so a fault stays contained, and cloud bursting adds virtual edge capacity for weekend peaks. Name the tradeoff plainly: keeping the hot world in memory is what makes a 50-millisecond tick possible, and write-through plus re-placement is the price of accepting that a single instance can vanish.

5) Fitting more players into one instance

The instance-size cap in the requirements is not arbitrary, and an interviewer will push on it: why tens of players and not thousands, and what would it take to raise it. The answer exposes what actually limits a single simulation.

Two costs grow with player count, and they grow at different rates. Replication fan-out grows roughly with the number of players times how many entities each can see, so without relevance filtering it trends toward quadratic (everyone seeing everyone). That is precisely why area-of-interest trimming is the first lever: cap what each player sees and the cost grows with the visible neighborhood, not the whole roster. The second cost is the simulation itself: one instance steps physics single-threaded on one machine within a 50-millisecond budget, and more players and entities mean more collisions and interactions to resolve per tick. Push past the budget and the server cannot keep 20 Hz. The tick rate sags, and the whole experience feels worse for everyone.

So raising the cap is about relieving whichever ceiling you hit first. If replication is the limit, tighter area-of-interest, lower update priorities for distant entities, and more aggressive delta and quantization buy headroom. If simulation is the limit, partition the world. Split a large map into zones, run each zone as its own instance or simulation shard, and hand a player off at the boundary. No single tick has to simulate everyone. So a massive experience is really many bounded instances stitched together, not one server holding thousands of players. Each instance stays inside the failure-domain and tick-budget model the rest of the design depends on. The honest framing is that a single authoritative instance trades scale for consistency and latency, and you buy more players by adding instances, not by making one instance bigger.

What is expected at each level

How much you need to land scales with the level. Roblox's ladder is IC-based and has no Staff title (Senior is IC3, Principal is IC5), so the bars below map to Software Engineer, Senior Software Engineer, and Principal rather than the numbered codes. System design carries more weight as the level rises, and senior loops often add a deep-dive on a system you actually built, so pace yourself for depth.

Software Engineer

Get the loop right and defend it. Frame the server as authoritative, draw inputs going up as proposals and snapshots coming down at a fixed tick, and explain why the client is never trusted. Name that Roblox replicates around 20 Hz, that clients predict their own movement, and that the server validates inputs before applying them. The interviewer will drive the harder parts. Keep it simple and correct. The bar is a coherent skeleton, connect to nearest server, send inputs, step on a tick, replicate relevant state, that you stand behind with a nudge or two.

Senior Software Engineer

Depth carries it, and the replication and latency-hiding pieces are where you earn the level. Raise deltas, quantization, and area-of-interest filtering unprompted and explain why they keep per-player bandwidth flat in the size of the map. Walk the full prediction-and-reconciliation chain using the sequence numbers, predict locally, keep pending inputs, replay them after the server acknowledges, and interpolate remote players. State the edge-versus-core split and put the tick loop at the edge and the economy in the core on purpose.

The bar is a near-complete design where you found the hard parts, the per-tick fan-out and the round-trip you have to hide, on your own. Most senior candidates are sorted right here.

Principal

Lead across the whole system. Reason about the instance-size ceiling and which cost (replication fan-out or single-threaded simulation) you hit first, and how zoning a world into stitched instances gets you past it. Handle failure concretely: an instance as a self-contained domain, write-through for durable state, and re-placement on crash, and connect it to how Roblox organizes the fleet into cells with cloud bursting for peaks. Place the boundaries cleanly so you are not redesigning all of Roblox: where this design ends and matchmaking, the economy, and chat moderation begin. The bar is that you drive the room while the interviewer redirects rather than rescues.