Design an Idempotent Payments API

Understanding the problem

The setup is concrete. A client sends POST /v1/charges to move money, the request reaches the server and the charge succeeds, and then the response is lost on the way back. A timeout. The client now has no idea whether the customer was charged. If it naively retries, it charges twice. If it gives up, it might have taken money for nothing. This one failure, a lost response over an unreliable network, is the whole problem, and idempotency is the answer.

Start by naming what is and is not achievable. Exactly-once delivery over a network is impossible: any message can be lost, and the sender can never tell a lost request apart from a lost response. So we aim for the next best thing. The client retries until it gets an answer, which gives us at-least-once delivery, and the server makes processing idempotent, meaning a repeated request produces the same single result. At-least-once delivery plus idempotent processing gives you effectively-once, the property a payments API really needs. Say that out loud early. It frames every decision that follows.

The Stripe angle

Stripe ships a real, public Idempotency-Key header, and the timed-out-payment question is the prompt this whole round is built on. So put the idempotency key on the write path early, before anyone asks for it. Do that and you sound like you have shipped money code. Stripe's own framing is "correctness before performance, failure modes before happy paths," which means a candidate who opens with capacity math and QPS has already misread the room. Open with the lost response. Stripe's engineering blog post Designing robust and predictable APIs with idempotency covers this exact case: a retry with the same idempotency key gets back the cached result of a request that already succeeded.

Functional requirements

Keep the scope tight. The depth here is one endpoint done correctly, not a wide surface of features. The core is three things. Creating a charge that is safe to retry. Replaying a retried request so it returns the original result instead of re-running. And rejecting misuse of a key. Everything else is below the line.

Non-functional requirements

These are the requirements that decide the design, so reason through each one. Correctness is first and non-negotiable: we must never charge a customer twice for one logical operation, and we must never lose record of a charge that did go through. Two concurrent retries of the same request have to resolve to exactly one charge. After a crash, we must be able to tell whether the money moved and resume safely. Availability matters, but it sits behind correctness here, because for money a wrong answer is worse than a slow one. We would rather make the client retry than risk a double charge.

Scale is the requirement we deliberately relax. Payments are a modest write volume next to a read-heavy product like a video platform. A large processor handles thousands of charges a second at peak, where a read-heavy product would see millions, and each charge is a tiny row. With that much headroom we can pay for strong consistency without flinching. Hold every money amount as an integer count of the smallest currency unit, cents for US dollars, never a floating-point number. Floats lose pennies to rounding, and pennies in a ledger are bugs.

• Exactly 1 Charge per key. Even under concurrent retries.
• At-least-once Client delivery. It retries until it hears back.
• Strong Consistency. Two requests can't both be first.
• Integer cents Money type. Never a float.

The set-up

Core entities

Name the objects the system moves around before drawing anything. There are three that carry the design:

The store under all of this is Postgres, and the choice is load-bearing rather than incidental, so be ready to defend it. We need two requests that arrive at the same instant to never both believe they are the first. That is a strong-consistency property, and a relational database gives it to us through a single mechanism. A unique constraint is a rule the database enforces so that no two rows can share the same value. Put a unique constraint on the pair (account_id, key), and the database itself guarantees that the second insert of a duplicate key fails. An eventually-consistent key-value store cannot promise that, because two replicas can each accept a write and only discover the conflict later, after both have already acted. By then the customer is charged twice. At payments volume we can afford the strongly-consistent store, so we take it. We partition by account_id once it grows, since a key is only ever unique within one account anyway.

API

The client generates a unique key, a UUID, once per logical charge and sends it in the Idempotency-Key header. The rule that makes it all work lives on the client. It reuses that same key on every retry of the same operation, and generates a fresh one only for a genuinely new charge. On the server side, the key is stored paired with the final response, and a replay returns that stored response without touching the card network again.

Four responses, and three of them are the interesting ones. The first call creates the charge and returns 200. A replay with the same key and the same body returns the very same 200 from storage without re-charging. A second request that arrives while the first is still running returns 409, telling the client to back off and retry. And a request that reuses a key with a different body returns 422, because a key is a promise to do one specific thing once. The deep dives build each of these.

High-level design

Walk the create-charge path one requirement at a time, and build the simplest version that works before hardening it. Each box is tinted by its job. Blue for the Client, violet for Payment service, green for the Idempotency store and Ledger, and amber for the one slow external call, the Card network. The walkthrough holds to the same scheme.

1) Receive and identify the request. The Client calls POST /v1/charges with its Idempotency-Key. The Payment service does not start charging yet. First it runs one indexed lookup of the key in the Idempotency store, which costs well under a millisecond. If there is already a completed record for this key, the work is done: it returns the stored response and never calls the Card network. If there is no record, this is a fresh request and we begin. Looking the key up before doing anything is what makes a replay cheap and safe, because the expensive card call is the one step it skips. The version we are deliberately not building yet would jump straight to charging, which is exactly how a retry double-charges. Deep dive one settles who wins when two of them race.

2) Claim the key, then create a pending charge. For a fresh request, the Payment service does two writes in one short database transaction that commits in single-digit milliseconds. It inserts an idempotency record keyed on (account_id, key) and writes a Charge row to the Ledger in the pending state. The unique constraint on the key is doing real work here. If a second copy of the same request is racing us, only one insert can win, and the loser knows immediately that it is a duplicate. The amount is stored as integer cents the moment it lands, so no float ever enters the system. What we are punting on: two retries arriving at the very same instant need a precise rule for who wins and what the loser returns. Deep dive one settles that.

3) Call the card network. With a pending charge recorded, the Payment service submits the charge to the Card network and waits for an authorize-or-decline. This round trip runs hundreds of milliseconds to a couple of seconds, orders of magnitude slower than every database write around it. It is also the dangerous step. It moves money in a system we do not control and cannot undo with a database rollback. By design it runs outside any open database transaction, carrying its own idempotency token so the network can recognize a resubmission. The punt here is the scary one. If our server crashes after the network authorized the card but before we recorded it, did the customer get charged, and can we resume without charging again. Deep dive two works through exactly that, and it is the heart of the design.

4) Record the result and store the response. When the Card network answers, the Payment service does three things in a second short transaction. It updates the Charge to succeeded or failed, writes the double-entry LedgerEntry rows in integer cents, and advances the record's recovery point. Then it stores the final response, status code and body, on the idempotency record and releases the lock. That stored response is a kilobyte or so of JSON kept for the 24-hour retry window. From now on any retry with this key short-circuits at step one and returns that stored response. The last thing to handle is a client that reuses a key for a different charge by mistake, which deep dive three catches with a fingerprint.

Size the write path

You do not open a payments round with queries-per-second. Once correctness is settled, though, a senior candidate puts the numbers on the board, because they explain why we can afford a strongly-consistent store. Payments are a write-modest workload. A large processor handles a few thousand charges a second at peak and on the order of 100 million a day. A read-heavy product like a video feed would see a hundred times that. Each charge fans into about four small row writes: the idempotency record, the pending Charge, and two double-entry LedgerEntry rows. So a few thousand charges a second is only ten to fifteen thousand row writes a second, which one partitioned Postgres primary handles without strain.

Storage splits into two very different stories. The idempotency table is bounded by its 24-hour time-to-live. That is roughly 100 million keys a day, each around 1 to 2 KB once the stored response is attached. So it tops out near 150 GB, and a daily prune holds it there instead of letting it grow forever. The ledger is the opposite. Charge and ledger rows are kept for years to satisfy financial-compliance retention. They are append-only and grow on the order of tens of TB a year, so we partition them by time and roll older partitions to cheaper storage. The point of the arithmetic: the volume is small enough that strong consistency never has to be traded away for throughput. The lifecycle deep dive picks up the prune and the per-merchant hotspot this sizing exposes.

• Write path: ~3K charges/s peak. About 100M charges a day. Each fans into ~4 small row writes, so ~12K writes/s, comfortable for one partitioned Postgres primary.
• Idempotency table: ~150 GB, capped. ~100M keys a day at 1 to 2 KB each, held flat by the 24-hour TTL and a daily prune. Bounded by retention, not by all-time volume.
• Ledger growth: ~30 TB/yr. Charge and double-entry rows kept for years for compliance, append-only. Partition by time and age out the cold partitions.
• Replay cost: 1 indexed read. A retry that hits a completed key is a sub-millisecond point lookup returning the stored response. No card-network call, so replays are effectively free.

Technology choices

Each box on the diagram is a real category of tool. Name a concrete pick, say why that category fits from first principles, and have the failure probe it invites ready, because this round pushes on every dependency you put on the board.

• The idempotency store and the ledger both live in Postgres. We reach for a relational database here for one property above all: a unique constraint on (account_id, key) that the database enforces atomically, so two concurrent retries can never both believe they are first. That single guarantee carries the entire concurrency story, and it comes for free with a strongly-consistent store. Integer-cents columns keep money exact, one transaction groups the record and the pending charge into a single commit, and partitioning by account_id spreads load once one merchant gets hot. The probe is "why not a key-value store like DynamoDB?" Eventually-consistent stores let two replicas each accept the same key and reconcile later, after both have already charged the card, which is the exact double charge we exist to prevent.
• The payment service is a stateless application tier with PgBouncer pooling its database connections. Stateless matters because all the real state lives in Postgres, so any instance can serve any retry and we scale by adding boxes. The pooler matters because of the card network: if every in-flight charge held a connection open across a multi-second card call, a traffic spike would drain the pool and stall the database for everyone. So we keep the card call outside any open transaction and return a fast 409 to a concurrent duplicate rather than block a connection on it. The probe is "what happens under a retry storm?" Throughput is bounded by the pool, which is why the fast-409 choice in deep dive one is a capacity decision and not only a correctness one.
• The card network is an external dependency we cannot roll back, reached through a processor gateway. No database ROLLBACK un-charges a card. So we treat the call as a foreign state mutation and push idempotency one layer down into it. We pass our own token, the charge id, so a resubmission after a crash authorizes the card once and returns the same answer the second time. The probe is "what if the network does not honor the token?" Then we cannot blindly resubmit, so we query it for that token's status before deciding, and we lean on batch reconciliation against its settlement file to catch any disagreement. Real card networks and Stripe's internal abstractions do provide this dedupe, so we state it as the assumption the recovery path rests on.
• The prune and reconciliation jobs run off the request path. A scheduled worker (a cron-driven job) sweeps idempotency rows past their 24-hour TTL so the table stays near 150 GB instead of growing without bound. A heavier batch pipeline (Airflow over the day's ledger) reconciles our charge rows against the card network's settlement file and flags drift. Drift means a charge we recorded as succeeded that the network shows as reversed, or the reverse. Neither touches the hot path, so neither competes with live charges for the Postgres primary. The probe is "how do you know your books match the network's?" Idempotency keeps each request correct in the moment; reconciliation is the slower net that catches the rare case where our record and the network's settlement disagree.

Potential deep dives

The grading concentrates in these dives. The interviewer names a failure mode and pushes on it, since staying correct when things break is what the round is checking. The ones below come up most at Stripe, each measured against the tiers an interviewer scores you on.

1) Two retries arrive at the same time. How do we not charge twice?

A flaky client often fires the retry before the first request has finished, so two copies of the same charge can be in the service at the same instant. This is the race the design has to survive. Both copies look up the key, both see no completed record, and if nothing stops them, both charge the card. The fix has to make exactly one of them win and give the other a safe answer. Work up to it in tiers, because the obvious version has a subtle hole.

The handling contention lesson covers unique constraints, row locks, and the time-of-check race in general, and PostgreSQL covers the storage choice underneath.

The Stripe angle

This is the mechanism behind Stripe's real Idempotency-Key header. Stripe's engineering team published exactly this design: an idempotency-key table in Postgres with a unique key per account. So defend why the strongly-consistent relational store is right here and an eventually-consistent key-value store is wrong. Two replicas that each accept the same key independently is a double charge waiting to happen, and a unique constraint is the cheapest way to make it impossible. Call the constraint the concurrency backbone before they push you to, and it reads like you have run money under load.

2) The server crashed mid-charge. Was the customer charged, and can we resume?

A card network call cannot be taken back. It changes the world in a system we do not own, and no database ROLLBACK reaches it, which is what makes payments genuinely hard. Picture our server dying after the network authorized the card but before we wrote that down. Our database says pending. The customer's statement says charged. The retry must not authorize the card a second time, and it still has to finish the work the dead request left half-done. The fix: break the operation into phases, each ending at a recorded checkpoint we can resume from.

There are four phases. Phase one, in a single short transaction, creates the idempotency record and the pending charge. Phase two calls the card network. That call lives between transactions because it is slow and external. Phase three, in a second short transaction, persists the result and advances the recovery point past the network call. Phase four stores the final response. The recovery point is a marker on the idempotency record naming the last phase that completed. A retry reads it and resumes from there rather than starting over.

The async processing lesson generalizes resuming a multi-step operation after a failure. The pattern of pushing idempotency down into an external call is the same instinct the integration round builds on the client side.

The Stripe angle

Your server dies mid-charge, now what? That is the escalation this correctness-first round leans on. Keep the network call inside one big transaction and you walk straight into the trap: the rollback cannot un-charge the card. The answer that holds up is recovery points plus an idempotent network call. Reach it and you have shown you understand that a database transaction's guarantees stop at the edge of the database. Most payments bugs live right at that edge, where the rollback runs out of reach and the card has already moved.

3) Same key, different request body. What then?

A key is a promise to perform one specific operation once. So what if a client sends a known key with a different body, charging $30 under the key it already used for a $20 charge? Honoring it would be wrong twice: replaying the stored $20 response ignores the new amount, and running the $30 charge breaks the once-per-key guarantee. The fix is to fingerprint the request.

When we first store a key, we also store a fingerprint. The fingerprint is a hash of the canonical request parameters: the method, the path, and the body. The body's fields go in a fixed order so the same request always hashes the same way. On every later request with that key, we recompute the fingerprint and compare. A match means it is a genuine retry of the same operation, and we replay or resume. A mismatch means the client is reusing a key for a different request, which is a bug on their side. We reject it with 422 and never touch the card. The rejection is a feature: it stops one key from silently doing two different things, and it surfaces the client's mistake loudly instead of letting it corrupt money quietly.

4) Key lifecycle and scale: expiry, pruning, and the per-merchant hotspot

Idempotency keys are not kept forever. A retry only makes sense for a short window after the original request, so a key earns a time-to-live, the span it stays valid. Stripe sets that at about 24 hours. Past it, the same key is treated as new. Keys also pile up, one row per charge attempt, so a background sweep prunes expired rows on a schedule, keeping the table from growing without bound. A retry that shows up after the key has been pruned simply starts a fresh charge, which is acceptable precisely because the retry window already closed.

Now the scale question, answered honestly. Payments are a modest write volume, so raw queries-per-second is not the threat; correctness is. The real risk is a single hot merchant, a marketplace running a flash sale, aiming a flood of charges at one part of the store. We already partition the idempotency table by account_id, and that does double duty. It keeps each account's keys together. And it spreads load so one busy merchant's traffic lands on its own partition instead of bottlenecking everyone. A key only has to be unique within an account, so partitioning by account costs us nothing in correctness. Strong consistency stays affordable because the volume is small and each row is tiny, so we never have to trade it away for throughput.

One more layer the best candidates raise: reconciliation. The card network keeps its own record of every authorization, and that record is the ultimate source of truth for what money really moved. A slower batch job runs off the request path. It compares our ledger against the network's settlement file and flags or corrects any drift. An example is a charge we think succeeded that the network shows as reversed, or the reverse. Idempotency keeps individual requests correct in the moment; reconciliation catches the rare disagreement that slips through and keeps the books honest over time.

The Stripe angle

The 24-hour expiry and the ledger reconciliation are both real Stripe mechanics, and a related round designs the ledger directly. Stripe documents that idempotency keys expire after 24 hours, so citing that window tells the interviewer you have read the API. A recurring Stripe design prompt is a high-throughput balance ledger that avoids a per-merchant hotspot, which is the same partition-by-account instinct pointed at the money rows instead of the keys. Raise reconciliation as a first-class safety net, before anyone asks what happens when the books disagree, and you sound like someone who has operated this rather than just sketched it.

What is expected at each level

That is a lot of surface area for one endpoint. Here is how much you need to land, by the level you are interviewing for.

Mid-level

Mostly breadth. Get to a working happy path and one retry. The client sends an idempotency key, the server stores the key with the response, and a retry returns the stored response instead of charging again. You should explain why exactly-once over a network is impossible and why at-least-once plus idempotent processing is the achievable goal. The interviewer will drive the deeper failure modes. The bar: a clean account of how one key makes one retry safe, even if the concurrency and crash cases need prompting.

Senior

Breadth, then real depth on the races. You are expected to reach the concurrency answer on your own. The unique constraint on (account_id, key) is the backbone, paired with a lock so a concurrent duplicate gets a 409 or a clean replay rather than a second charge. You should add the fingerprint to catch a reused key. And you should defend Postgres and the unique constraint without being asked, including why an eventually-consistent store would let two requests both think they are first. The bar: you find the concurrency race yourself and resolve it precisely, and you justify the strongly-consistent store as a deliberate choice.

Staff+

Real depth on the hard failures, and the scope widens. You lead on a wide set of pieces. Recovery points. The foreign state mutation, and why a database transaction cannot wrap the network call. An idempotency token passed down to the network. The key lifecycle with expiry and pruning. Partitioning to avoid a per-merchant hotspot. A multi-region key store. And batch reconciliation against the network's record of truth. You raise idempotency before anyone asks, and you drive the conversation while the interviewer redirects more than rescues. To clear the bar, you treat the crash-mid-charge case as the center of the design rather than a corner of it, and your reasoning about correctness holds end to end.