Design a Delivery Service

Understanding the problem

A delivery platform starts life as one application. One Spring Boot process holds the booking, the courier assignment, the live tracking, and the customer messages, and it reads and writes a single Postgres database for all of it. That is fine at a steady dozen bookings a second. It stops being fine when a promotion points fifty times that load at one process. Now a slow query behind a tracking screen can stall a booking, and a stuck text-message vendor can hold a database connection a checkout needed. The task is to carve the monolith into services that fail on their own and scale on their own, then follow one delivery from booked to delivered across them.

The arc of a delivery is short to say. A Customer books a package to go from a pickup to a dropoff and pays for it. The platform finds an available Courier near the pickup and offers them the job. The customer watches the courier cross the map until the package is delivered, and both sides get a message at each step. Four jobs live inside that one sentence, and today one process does all four. The whole design is deciding where to draw the lines between them.

Functional requirements

Keep the scope tight and let the interviewer widen it. The core is the lifecycle of one delivery: a customer books and pays, the platform assigns a courier, everyone tracks the package to the door, and both sides are kept informed. Everything else is a neighbor system that can wait.

Non-functional requirements

Reason through each target, because the requirements are what draw the service lines later. The booking is the consistency-critical object. Once a customer sees "booked" and a card is charged, that delivery cannot quietly vanish, cannot be charged twice, and cannot have two couriers show up for it. So the canonical delivery record is strongly consistent, and every step through its lifecycle happens once. We would rather reject a state change and retry it than let a delivery skip a step or run one twice.

Dispatch is the opposite kind of problem, and saying so out loud scores. Picking which courier gets a delivery tolerates being a beat behind and tolerates being approximate. A slightly stale view of who is free near the pickup costs at most a wasted offer that the next courier picks up. That relaxation is exactly what lets dispatch run as its own asynchronous service instead of blocking the booking. Availability matters across the board, because a checkout that times out during the promotion is lost revenue, but when correctness and availability collide on the delivery record, correctness wins.

Now put numbers down, because the prompt is about the surge. Say the platform runs about a million deliveries a day in normal times. That is roughly twelve per second on a flat average, and bookings bunch into the daytime, so call the normal peak forty per second. The promotion multiplies the peak by fifty, which lands around two thousand bookings per second for a few hours. A delivery row is about a kilobyte, so even at the surge peak the booking writes come to a couple of megabytes a second, which one Postgres primary absorbs without breaking a sweat. The write rate is not the hard part.

Two other things are. Couriers ping their location constantly: at the surge, say two hundred thousand couriers online each pinging every five seconds, which is forty thousand writes a second, roughly a thousand times the booking rate. Tracking reads fan out on top of that: a delivery lasts about forty minutes, so two thousand bookings per second means on the order of five million deliveries being watched at once. The firehose of pings and the wall of tracking reads are where this system bends, and neither belongs on the booking database.

• ~2,000/s Bookings at surge. 50x the ~40/s normal peak; ~2 MB/s of writes, inside one Postgres primary.
• Exactly 1 Charge and courier per delivery. The invariant the round tests.
• ~40K/s Courier location pings. ~1000x the booking rate; lands in an in-memory geo cache, never the SQL primary.
• ~5M Deliveries watched at once. Tracking reads served from a cache and replicas, off the booking path.

The JPMorgan angle

A JPMorgan panel decides early whether you understand this prompt, and the tell is your first move. Carving the platform along the seams that genuinely differ reads as strong. Reaching for a memorized web-scale template reads as someone who missed that the prompt is a monolith taken to fifty times its load, which is the pragmatic decomposition the round rewards. So the winning move is to name the four jobs the one process does today and split them by how they genuinely differ, then defend each line in plain terms. A billion-user feed diagram is the wrong rehearsal; a lead who has broken up a monolith wants a clean split of booking, dispatch, tracking and notification. Add a sentence on why each line falls where it does.

Breaking the monolith into services

A service boundary belongs where two pieces of the system genuinely differ. Nouns on a whiteboard are a poor guide to it. Four differences are worth cutting along: consistency needs, scaling curve, failure blast radius, and ownership. When two pieces of the monolith disagree on all four, keeping them in one process forces the strict piece to inherit the weak piece's problems. That is the reasoning to say out loud, because it is the reasoning the grader is checking. Four services come out of it.

Two supporting pieces hold the services together; an API gateway sits at the edge as the single front door. It authenticates the caller, routes to the right service and enforces rate limits. No service redoes that work, and the surge can be shed in one place. And an Event log, a durable partitioned queue, is the spine between the services. The order service writes one event per delivery transition, and the three downstream services each consume that log at their own pace. That log is what turns four tightly-wired modules into four services that fail independently, because a consumer falling behind becomes lag on the log rather than a stalled booking.

Here is the shape end to end. Read the colors as roles: blue is an actor at the edge, violet is a service we run, amber is the event log, and green is a datastore.

1) Book the delivery. The Customer calls the API gateway, which routes to the Order service. The order service asks the Payment service to charge the card once under an idempotency key. Only after a successful authorization does it write the Delivery store row in state booked and emit a booked event on the Event log. It returns the delivery id right away; this is the one strict, synchronous stretch of the whole design, and it is small. At forty bookings a second normally and two thousand at the surge peak, one Postgres primary partitioned by region keeps up. Everything after this can be slow and the delivery still holds, because the row is already durable.

2) Assign a courier, asynchronously. The Dispatch service reads the booked event off the log rather than being called inline, so booking never waits on courier supply. It queries the Courier geo cache for idle couriers near the pickup, a lookup that answers in microseconds against an in-memory index instead of touching the delivery database. It scores the candidates and offers the job to a good one over a stream. When a Courier accepts, dispatch tells the order service, which moves the delivery to assigned. The offer is safe to retry, and keeping one courier per delivery when two workers race is the second deep dive.

3) Track and notify off the same log. As the courier drives, each milestone is a transition the order service records and emits once: assigned, picked_up, delivered. The Tracking service consumes those events plus the courier's location pings and keeps a hot view per delivery in the Tracking cache, which is what the customer's tracking screen reads. The Notification service consumes the same transitions and sends the push and SMS. Both run off the log at their own pace, so a backed-up notification vendor or a slow analytics job can never wedge a booking or a dispatch. That is the payoff of the split: the strict path is short, and the slow work hangs off a queue behind it.

Data and the hot read

Core entities

Name the objects the system moves around before arguing about stores. Four carry the design, and the delivery is the spine the other three hang off.

One split decides where things live: the Delivery and its Assignment want a strongly consistent, transactional store. One charge and one courier per delivery are exactly what a row lock and a conditional update enforce. That is Postgres, partitioned by region so a region's deliveries sit together and a whole lifecycle is one local transaction; the Courier live location is the opposite. It changes every few seconds across hundreds of thousands of couriers, a write rate that would buckle a transactional database. So it lives in an in-memory geo index in Redis; the high-volume append data, the DeliveryEvent stream, goes to Kafka, which is durable, ordered per delivery, and replayable. Say those homes out loud with the reason attached, because the interviewer will ask where each entity lives and why.

The tracking read is the one that would melt the booking database if you let it, so it earns its own cache. At the surge, on the order of five million deliveries are watched at once. Each open screen polls or holds a stream for the current state, the courier's position and an arrival estimate. None of that is a booking write, and none of it should reach the booking primary. So the Tracking service keeps a hot view per delivery in a Redis Tracking cache keyed by delivery id, updated as events and pings arrive, and serves every tracking read from memory. A cache miss falls back to a read replica of Postgres, never the primary. The pattern is a read-through cache in front of replicas: the primary takes the small strict write traffic, and the enormous read traffic is soaked up by memory and copies.

The API

Keep the surface thin and notice who calls what. The customer books and then subscribes to status. A courier pings location on a high-volume path that lands in the geo cache, and receives offers on a stream. The booking call carries an idempotency key so a phone that retries on a flaky network still ends up with one delivery and one charge.

The JPMorgan angle

These four services split along the four differences, and the nouns barely enter into it. Booking and tracking both touch a delivery, but they sit apart because one is a strict low-volume write and the other is a huge stale-tolerant read. That is the boundary reasoning a JPMorgan lead is grading. When the interviewer asks why dispatch is its own service, the weak answer is "because it does dispatching." A strong answer names the differences. Dispatch needs weaker consistency than booking, runs on a different store and scales on its own curve. In one process, the strict path would inherit the weak path's load and outages. Name the seam and defend it on those terms.

Potential deep dives

Most of the signal lives here, and a JPMorgan interviewer drives at one seam at a time and pushes until you either defend it or admit where it breaks. Four dives carry this design: charging a delivery once, assigning one courier safely, what fails at each seam, and holding the fifty-times surge. Each one is where a clean diagram either survives a hard question or falls over.

1) How does one booking charge once and assign one courier?

This is the invariant the whole round is built to test. A customer taps book, their phone hiccups, the app retries, and now two requests are in flight for one delivery. The failure mode is a double charge and, later, two couriers dispatched for one package. The fix is to make booking idempotent, so the same intent applied twice produces one delivery and one charge. Work up to it from the naive version, because the interviewer wants to see you know why the easy answer fails.

The JPMorgan angle

This is the seam where a bank's grader leans in hardest, because it is money; a delivery platform cannot charge a customer twice or take payment and never deliver. A JPMorgan interviewer will keep asking "and what if that call is retried" until you show the invariant survives. Strong answers are plain and exact: the idempotency key ties the retries together. The unique constraint makes a double-insert fail loudly. The outbox commits the record and its event as one. The machinery stays plain. What you show is that every path a retry can take still lands on one delivery and one charge.

One follow-up almost always comes, so have it ready: the charge that times out in the ambiguous window. The order service calls the Payment service, the processor takes the money, and then the response is dropped on the network before it gets back. Now it cannot tell whether the charge landed, so it can neither safely mark the delivery booked nor safely give up. The idempotency key covers the retry: the next attempt reuses the same key, the processor recognizes it, and it returns the original charge instead of billing again. For the rare case where even the retry cannot confirm, a slow reconciliation job checks recent bookings against the processor's own record. It settles any that disagree, refunding a charge with no delivery or completing a booking whose charge went through. A background sweep like that closes the gap the request path cannot.

2) How does dispatch assign exactly one courier?

Dispatch is stale-tolerant, but the assignment itself is not. Two things can go wrong. Two dispatch workers can pick up the same booked event and both start offering the delivery. One courier can end up assigned twice, or a courier's accept arrives twice and assigns them to the same job twice. The delivery record has to end with exactly one active courier no matter how the offers race. The key idea is simple: offers can be noisy and repeated. The accept cannot, so it lands as a single conditional state change.

A lost confirmation like that forces the next step. Under the conditional accept alone, a retried accept from the same courier can behave in two ways depending on timing. If the delivery is still theirs, the retry is harmless. If in the meantime the offer expired and another courier took it, the retry has to fail cleanly rather than steal the delivery back. So the accept needs to be idempotent by offer, and offers need a lifetime, which is the upgrade below.

The design settled into a clear shape. Offers are cheap, so send them freely. The accept is the one guarded write. Everything upstream of it can be noisy, retried, and duplicated, and the invariant still holds, because a single conditional transition is the only thing that can change who owns the delivery. That is the same idempotency idea from the booking dive, applied to a different seam, and reusing one idea across two seams is what makes the whole design feel coherent instead of ad hoc.

3) What breaks at each seam under failure?

The reason to split the monolith was so that one part failing does not take the rest down, and the interviewer will test that promise seam by seam. Walk each one and say what happens, because the decomposition exists so that most of these degrade instead of failing. The through-line is that the booking path stays up as long as Postgres and the payment processor are up, and everything else recovers from the event log.

• Payment processor down. Booking fails fast and clean. The order service never writes a delivery it could not charge, so there is no half-booked state to reconcile. The customer sees a clear failure and retries, and the idempotency key means the retry that finally succeeds still produces one delivery. Nothing downstream ever recorded the failed attempt.
• Dispatch service down. Bookings keep succeeding and are durable in state booked, because dispatch is off the request path. Deliveries simply queue as unconsumed booked events on the Event log. When dispatch recovers, it resumes from its last offset and works through the backlog, assigning couriers to everything that piled up. The customer waited longer for a courier, but no booking was lost.
• Courier geo cache node lost. Positions are disposable, so this is the gentlest failure. Every online Courier pings again within seconds and the Courier geo cache refills itself. A dispatch during the gap runs against slightly stale or thin data, and the worst case is a wasted offer that the next courier picks up. No delivery and no money is at risk.
• Tracking cache down. Tracking reads miss the Tracking cache and fall back to a read replica of Postgres. Slower and more load on the replicas, but correct, and the booking primary is still shielded. The customer's map lags a little. The booking and dispatch paths do not notice.
• Notification vendor stuck. The Notification service is an asynchronous consumer, so a stuck SMS or push vendor becomes consumer lag, never a blocked booking. Messages retry with backoff, and a send that keeps failing lands in a dead-letter queue after a few attempts for a human to look at. A booking placed during the outage is fine, it just gets its "courier assigned" text late.
• Event log lag or a crashed consumer. A consumer that falls behind or dies rewinds to its last committed offset and replays. Because every downstream effect is idempotent, keyed on the delivery or the offer, a replayed transition does not fire a second notification or a second assignment. Lag shows up as staleness in tracking and delay in notifications, both of which self-heal as the consumer catches up.

The JPMorgan angle

A JPMorgan interviewer rarely asks you to enumerate all six; they pick one seam and push. "The message vendor is timing out at peak, what happens to a booking placed right now?" A strong answer walks the blast radius out loud. The booking commits and charges normally because notification is off the path. The state change lands on the log and the notification lags behind a retry, and the customer gets a late text and a correct delivery. That is the operational judgment a bank hires for, and it is why the decomposition was worth doing. Every seam should have an answer that ends in "degrades" rather than "goes down."

Three moves make every one of those answers possible, and it helps to name them as a set. Keep the durable state in one owner behind each seam, so a failure loses at most an in-flight request and never a committed delivery. Make each downstream effect idempotent and keyed, so a retry or a replay produces the same result once. Put the slow, flaky work behind the Event log, so a struggling consumer becomes lag that catches up rather than a wedge on the booking path. Given those three, most failures turn into a delay a customer barely notices, and the rare one that cannot degrade fails a single booking cleanly with the money still correct.

4) How does this hold fifty times the traffic?

Come back to the surge, because it is the literal prompt. Normal peak is about forty bookings a second, and the promotion multiplies it by fifty. Plan for roughly two thousand bookings a second for a few hours, then add forty thousand courier pings a second and five million tracking screens on top. The point to make first is that the booking writes are not the scary number. Two thousand rows a second at a kilobyte each is a couple of megabytes a second, which one Postgres primary handles. That is the easy part. The pressure is on the read fan-out, the ping firehose, and the sheer connection count, and each has its own lever.

• The API gateway sheds load at the edge. When arriving traffic exceeds what the fleet can serve, the gateway is the one place to throttle. It rate-limits aggressive retries and can shed the cheap, low-priority traffic first, tracking polls and duplicate booking taps, so the expensive strict path stays responsive. Better to reject a small slice of tracking refreshes at the door than to let a thundering herd topple the booking service behind it. That is backpressure, and doing it in one place is a reason the gateway is its own component.
• A load balancer plus stateless services autoscale on their own curves. No service holds session state, so any instance serves any request and you scale a service by adding instances behind the API gateway and a load balancer. The curves differ wildly. At the surge the Tracking service and Dispatch service need many more instances than the Order service. As separate fleets, each scales to its own load, so you never over-provision one giant process for the busiest part. An instance that dies just falls out of the pool.
• The Event log absorbs the burst. Booking emits an event and returns; the downstream consumers take events at whatever rate they can. A spike becomes consumer lag on Kafka rather than dropped work or a stalled booking. Partition the log by delivery id so one delivery's events stay ordered while the whole thing scales out across partitions, and a consumer group adds workers to chew through a backlog. The queue is what converts a fifty-times spike from an outage into a few minutes of catch-up.
• The Tracking cache and read replicas soak up the reads. The five million watchers never reach the primary. They are served from the Redis Tracking cache, with a read replica behind it for misses. Reads scale by adding replicas and cache capacity, entirely separately from the write path. This is the single biggest reason tracking is its own service: its load is a hundred times booking's and it must never contend for the booking primary's resources.
• Postgres scales writes by partitioning, only if it must. At two megabytes a second one primary is fine, so do not shard for show. If a single region's bookings genuinely outgrow one node, the delivery data already partitions cleanly by region, since a courier in one city never serves a pickup in another. That regional shard key means growth spreads across nodes without a cross-shard transaction on the common path.

Put one number on the connection problem, because it is the one the surge makes real. Five million customers holding a live tracking stream is five million open connections, and a single server keeps tens of thousands of them at most. So the tracking tier needs on the order of a few hundred instances at the peak just to hold the sockets, each subscribed to the delivery updates its watchers care about. That is a large fleet, and it is fine, because tracking is stateless and its instances come and go behind the load balancer. The point to make out loud is that this fleet scales entirely on its own, driven by watcher count, with no effect on the forty-booking-a-second write tier next to it. The independence between those two tiers is the dividend the decomposition pays. If tracking still lived inside the monolith, holding five million sockets would starve the very process that has to accept a booking.

Put together, the surge story is the pragmatic decomposition paying off. The gateway holds the door, stateless fleets scale where the load truly sits, the log turns bursts into lag, and the cache keeps the reads off the primary. None of it is exotic. It is the API gateway, load balancer, queue, and cache that a JPMorgan panel names in the prompt, applied exactly where the numbers say they are needed.

What the round rewards

The bar here is judgment over scale. Open on requirements, draw the four service lines and defend where each falls, then go deep on the one seam the interviewer picks. Expect the design to drift into Java and the enterprise stack, and let it, because at JPMorgan the architecture and the language are one conversation.

Common questions