Design a Payments Ledger

Understanding the problem

Square published "Books," its own immutable double-entry accounting database, and the design here is that system's shape: an append-only journal where every transaction balances to zero, with cached balances derived on top. The job has two halves. Record every money movement (a card-present sale at a Square Reader, a Cash App transfer, a payout to a seller's bank, a refund, an Afterpay installment) with perfect accuracy and a complete audit trail. And answer one question fast: what is this account's balance right now. A ledger that loses a cent, or cannot say where a number came from, has failed at the one thing it exists to do. So the headline requirements are correctness and auditability. Scale is the twist that makes it hard.

Name the hard part up front, because the panel will steer straight at it. A naive ledger keeps one running balance row per account and adds to it on every sale. That holds until a single seller gets hot. A flash sale fires thousands of card taps a second at one merchant's account. Every one of those writes lines up behind the same row. Throughput on that seller collapses to one write at a time, right when the sale needs it most. The same skew shows up on Cash App when a popular $cashtag takes a flood of payments. So we build the correct ledger first, then earn our way to the hot-account fix.

The Block angle

Square Books is a real, documented Block system, and your design should read like it. Block published Books as an immutable double-entry accounting database running on Cloud Spanner and GKE. An append-only journal is the audit log, with cached balances on top. The hard invariant is that every transaction balances to zero, "each cent lost matched with a cent gained." Two moves carry the round. Reach for double-entry bookkeeping and immutable append-only entries without being told. Then spot the hot seller as the scaling threat before the panel pushes you there. This one ledger sits under both Square and Cash App, so it records a card-present tap and a P2P transfer through the same machinery. Every cent stays traceable because that is what economic empowerment, Block's stated purpose, demands of rails people trust with their money.

Functional requirements

Scope this down hard. The depth lives in money moving correctly at scale, and Block's product breadth (Square, Cash App, Afterpay, payouts) only pulls focus off it. The core is three things. Record a money movement so it either lands completely or not at all. Serve an account's current balance quickly. And keep a full history we can audit and reconcile. Anything past that sits below the line.

Non-functional requirements

These decide the design, so take them one at a time rather than reciting adjectives. Correctness comes first and is non-negotiable: money is never created or destroyed by the system, only moved between accounts, and the books always balance to the cent. A money movement is atomic, so all of its entries commit together or none do, and a balance is never left half-updated. The recorded entries are strongly consistent, because they are the source of truth for real money flowing through Square and Cash App. A balance derived from them is allowed to lag a little, which is a relaxation we cash in later. And the whole history is auditable, so any number can be traced back to the movements that produced it.

Scale is where this round bites. Across all of Block's sellers and Cash App customers the aggregate runs on the order of a few thousand transactions a second, a modest number, so raw queries-per-second poses no threat. The threat is skew. Most sellers are quiet while a few go white-hot during a sale and aim a flood of card taps at one account. That single account is where a naive design serializes and falls over. The budget we have to defend, then, is throughput on one hot seller. Hold every amount as an integer count of the smallest currency unit, cents for US dollars, never a floating-point number. Floats lose fractions of a cent to rounding, and a ledger that rounds is a ledger that is wrong.

• Sum to 0 Every transaction. Debits equal credits, always.
• Atomic Money movement. All entries land or none do.
• Append-only Journal. Never update, never delete.
• Integer cents Money type. Never a float.

The set-up

Core entities

Double-entry bookkeeping, the centuries-old accounting practice Books is built on, hands the ledger exactly three pieces to move money between:

The mechanic that makes this trustworthy is double-entry. Every money movement is a transaction made of two or more entries that sum to zero: a debit on one account and an equal credit on another. Money is never created or destroyed, only moved, so the entries of one transaction always net to nothing. A $20 card-present sale of 2000 cents writes +2000 to the seller's balance account and -2000 to a Block clearing account, and the two cancel. Entries are also immutable and append-only. We never UPDATE or DELETE an entry once it is written. A mistake is fixed by a new compensating transaction that moves the money back. That append-only immutability is what makes the ledger auditable, since the history is a complete record of every movement with nothing edited out from under it. Books calls this the journal, and it is the system's audit log.

The store under all of this is Cloud Spanner, the database Square Books actually runs on, and it is a pick you should expect to defend under questioning. We need all the entries of one transaction to commit together and to enforce that they sum to zero. Spanner gives a strongly-consistent, atomic multi-row commit across shards: BEGIN, insert the entries, check the invariant, COMMIT, and it holds even when the accounts live on different shards. The rows are tiny and uniform: an entry is a handful of integers and a currency code. The table is enormous in count but trivial per row, and Spanner shards it horizontally as it grows. We interleave an account's entries under the account so they sit together in one shard, since an account's entries are read and summed together. Hand this to an eventually-consistent key-value store and it cannot promise the entries land atomically and balance, the one guarantee a ledger cannot do without. So the strongly-consistent relational store is a deliberate pick.

API

The interface is deliberately small. A caller posts a balanced transaction with an idempotency key, and reads a balance back. The ledger never exposes a way to edit or delete an entry, because that operation does not exist; a correction is just another posted transaction. The amounts are integer cents on the wire and in storage.

Posting a transaction is the only way money moves. It is rejected with 422 if the entries do not sum to zero, so an unbalanced movement never reaches the database. The same idempotency key replays the original transaction instead of posting a second copy, which ties this design straight to Square's public idempotency-key API. Reading a balance is its own endpoint because, as the deep dives show, computing it is the interesting part. And reversing a transaction writes a new one rather than touching the old, which is how the ledger corrects mistakes without ever rewriting history.

High-level design

Take the post-a-transaction path requirement by requirement. Build the simplest correct version first, then harden it for the hot seller. The diagram colors each box by what it does. The Client is blue, the Ledger service violet, the Journal log and Balances store green, and the Reconciliation job that runs off to the side amber. The steps below keep those same colors.

1) Receive and validate the transaction. The Client calls POST /v1/transactions with a set of entries and an Idempotency-Key. The Ledger service writes nothing yet. First it sums the entries' amount_cents and confirms they net to zero, rejecting with 422 if they do not. That sum runs over a handful of integers, so it costs microseconds, and an unbalanced transaction would create or destroy money, so it has to be turned away before it ever reaches storage. Then it looks up the idempotency key. If this key already posted a transaction, the service returns that original unchanged and stops, which is what makes a timed-out client safe to retry. Validating before any write keeps the books correct for almost no cost. A naive ledger would trust the caller's arithmetic and skip both checks, and that is how the books drift out of balance one bad write at a time. The atomic-commit guarantee behind this is deep dive one.

2) Append the entries atomically. For a fresh transaction, the Ledger service opens one Spanner read-write transaction, inserts every Entry row into the append-only Journal, and commits, so all of a movement's entries land together or none do. Rows are appended and never modified, so two unrelated transactions write their own brand-new rows and never wait on a shared lock. The write path runs wide and parallel, even at four to ten thousand entry inserts a second. The amounts are integer cents the moment they land, so no float ever enters the ledger. What we are punting on lives one store over. When several thousand of those transactions a second all credit the same hot seller's balance during a flash sale, that single account turns into a serialization point. Spreading those writes so no per-seller bottleneck forms is deep dive two, the headline of this design.

3) Update the affected balances. With the entries committed as the source of truth, the Ledger service reflects them into the Balances store so a later read does not re-sum the account's whole history. For a quiet account that is one small write to one row. For a hot seller it can't be a single UPDATE balance SET amount = amount + X on a shared row. Under a flash sale every tap would queue behind that row's lock. Spanner would hotspot on the one shard holding it, and throughput would drop to a few hundred writes a second. So the balance store stays off the critical correctness path. It is derived state, allowed to lag the journal by a moment, and rebuildable from it if it ever drifts. How that balance is structured so the writes spread across many rows is deep dive two, and how a read of it stays fast over millions of entries is deep dive three.

4) Serve balances and reconcile off the side. A GET /v1/accounts/{id}/balance reads from the Balances store and returns a number derived from the journal, which under load may trail it by a few milliseconds. That lag is safe because the journal is the strongly-consistent source of truth and the balance is only a convenience computed from it. When a decision cannot tolerate any lag, a payout that must not overdraw a seller's account, we read the exact balance straight from the journal instead. Off to the side and entirely off the request path, the Reconciliation job streams settled movements out of the Journal. It compares them line by line against the bank and card-network settlement files, which arrive in daily batches. Any drift is corrected with a reversing transaction, never an edit to the past. Deep dive four covers that audit story end to end.

Capacity: what the books absorb

Size it, because the arithmetic sets the sharding and retention story. Across Square sales and Cash App transfers the ledger sees on the order of a few thousand transactions a second. Call it 2,000 sustained, with peaks several times that during retail rushes. These are reasoned estimates rather than a figure Block has confirmed, so hold them loosely. Each transaction is at least two entries, so the Journal absorbs roughly 4,000 to 10,000 row inserts a second. Over a day that is around 350 million new entries. An entry row is a handful of integers and a currency code, call it 250 bytes with indexes. The journal grows by roughly 90 GB a day, on the order of 30 TB a year. That figure only ever climbs, because nothing in the ledger is deleted.

Two consequences fall out of that. First, retention is measured in years. Financial records have to stay queryable for audit and compliance for seven to ten years, so the design never truncates the journal. It shards the entries by account_id. Cold shards, the accounts and date ranges no live balance still needs, age out to cheaper object storage and stay readable. Second, the aggregate rate is comfortable but the skew is not. A flash sale can aim several thousand card taps a second at one seller. A single mutable balance row tops out near a few hundred writes a second under lock contention on one Spanner shard. That gap, thousands wanted against hundreds delivered, is what forces the striped balance in deep dive two.

• Write rate: ~4-10K entries/s. Around 2,000 transactions a second sustained across all sellers and Cash App customers, at two or more entries each. The append-only journal takes 4,000 to 10,000 inserts a second, and several times that at peak around paydays. The aggregate is modest; the skew is the threat.
• Storage growth: ~30 TB/year. Roughly 350M entries a day at about 250 bytes a row with indexes is near 90 GB a day. The journal is append-only and never shrinks, so it climbs steadily into the tens of TB.
• Retention: 7-10 years. Records stay queryable for audit and compliance. The ledger never deletes: it shards by account id and ages cold shards out to object storage that stays readable.
• Hot-seller skew: 1000s/s on one account. A flash sale points several thousand card taps a second at one seller, while a single mutable balance row caps near a few hundred a second on its shard. That gap forces the striped balance.

Technology choices

The stores on that diagram are still categories. For each, land on a specific product and say why a ledger in particular wants it, then keep the failure question close, because a correctness round tests every store you name.

• Cloud Spanner holds the journal and the derived balances, and it is the database Square Books runs on, so the choice is anything but incidental. A ledger needs all the entries of one movement to commit together, and to sum to zero before anything is visible. A Spanner read-write transaction gives you that across shards: BEGIN, insert the entries, check the invariant, COMMIT. The rows are tiny and uniform, a few integers and a currency code. The table is enormous in row count but trivial per row. Spanner shards it horizontally with each account's entries interleaved beside it. The failure probe is the hot seller. Spanner hotspots when a flood of writes lands on one key range, so a single mutable balance row would serialize every writer on one shard. The design keeps the authoritative journal append-only, where inserts never contend, and stripes the derived balance across many rows so the load spreads across shards. The reason it is not a key-value store: an eventually-consistent one lets entries land without the atomic, balances-to-zero promise, and that promise is the whole job here.
• A change stream off Spanner, published to Kafka (Pub/Sub in Block's GCP stack), carries committed movements off the write path. Reconciliation, seller reporting, Cash App's activity feed, and fraud scoring read a stream of entries instead of querying the ledger directly. The append-only journal is already an event log in shape: every row is an immutable fact in commit order, so publishing it as a stream is a natural fit. Consumers replay from an offset, fall behind, and catch up without ever adding read load to the path that is taking thousands of writes a second. The probe here is the dual-write trap, where the service writes to Spanner and the stream separately and one succeeds while the other fails, leaving the two disagreeing. The fix is to derive the stream from the committed journal itself through the change stream, so a movement is published only because it was already durably recorded, never as an independent second write.
• Object storage like Google Cloud Storage plays two roles a relational database should not. It holds the daily settlement files the banks and card networks deliver, large flat files the Reconciliation job streams through and compares against the journal line by line. Cold ledger shards go here too. Entries for closed date ranges that no live balance needs stay queryable for the seven-to-ten-year retention window, without inflating the hot Spanner tier's indexes. Object storage is built for large write-once files read occasionally, which is that workload exactly. The failure question is durability of the audit trail. Object storage of this class gives eleven-nines durability with object versioning, so an archived shard or a settlement file can't be silently lost or rewritten underneath you.
• The ledger service is a stateless app tier on GKE, the same place Books runs, and stateless is the point. Every correctness invariant, the zero-sum check, the atomic commit, the uniqueness of the idempotency key, is enforced inside the Spanner transaction rather than in the process, so no instance holds state another instance needs. Any one of them can serve any request, and you scale by adding pods behind a load balancer. The probe is what happens when an instance dies mid-request. Because nothing commits until the single Spanner transaction does, a crashed pod leaves no half-written movement behind, and the client's retry, carrying the same idempotency key, lands the transaction exactly once.

Potential deep dives

The deep dives carry most of the grading. The panel fixes on one property of the ledger and presses until it bends, since correctness under load is what this round exists to test. The ones below are what Block reaches for most, each scored against the tiers a panel grades you on.

1) How do we guarantee a transfer never half-applies?

A Cash App transfer moves money from one account to another, so it touches at least two accounts at once. If the debit on the sender lands and the credit on the receiver does not, money has vanished and the books no longer balance. Balanced double-entry written atomically prevents this: the two entries are a single unit that either both commit or both fail. Work up to it in tiers. The obvious version leaks money in two different ways.

The handling contention lesson covers atomic transactions and unique constraints in general. Block's idempotent payments API design works the idempotency key all the way through, including the concurrent-retry race that picks exactly one winner.

The Block angle

Integer cents and immutable double-entry are how Block builds money systems, and Books is the proof. Money as integer cents, never a float, is the baseline that says you have moved real money before; using it without being told signals you have shipped a payments system. The idempotency key here is the same one Square's public Payments API requires on CreatePayment, where a repeated key returns the original response and a changed body under a reused key is rejected. Tie them together: the ledger transaction carries an idempotency key for the same reason a Square charge does. The top of the bar is treating corrections as reversing entries rather than edits, which is exactly what Books does. Propose UPDATE-ing a past entry and you have shown you do not yet get why an auditable ledger is append-only.

2) One seller takes thousands of taps a second. How do we not bottleneck on their balance row?

This is the dive the Block prompt is really about, because it is where a flash sale meets a regulated ledger. Size it first. A flash sale can aim several thousand card taps a second at a single seller, and each one wants to update that seller's balance. A database row can be modified by exactly one transaction at a time. Each write takes a lock on the row, and the next waiter blocks until it commits. On Spanner, all of that traffic also lands on the one shard holding the row. So if the balance is one mutable row, those thousands of taps form a single-file queue behind that lock on one hot shard. Effective throughput on that seller drops to a few hundred writes a second right when the sale needs the most. The contention math is straightforward. One row means one writer at a time, and N independent write targets mean roughly N concurrent writers. So the fix is to stop funneling the hot seller's writes through a single row.

The handling contention lesson generalizes the striped-counter pattern. It is the same instinct as splitting a viral post's like counter across keys: spread a single hot write target into many so the lock contention divides N ways.

The Block angle

A merchant runs a flash sale through a Square Reader. Thousands of card-present taps a second, all aimed at one account. That is Block's version of this problem: find the single mutable balance row that serializes every write, and break it up. The answer runs in three moves. Append-only entries kill the write contention. Summing them on read does not scale, which sends you to striped sub-balances for a fast balance without reintroducing the hot row. Then the contention math: one row is one writer, N stripes are N writers, and on Spanner the stripes spread across shards instead of hotspotting one. Say that and it lands as understanding rather than recitation. The same skew hits a popular Cash App $cashtag flooded with payments, and the striped balance fixes it there the same way.

3) How do we read a balance fast when an account has millions of entries?

Striping spreads the writes, but a long-lived seller still accumulates millions of entries. Summing all of them on every read, or even all their stripes back to the beginning of time, gets slower as the account ages. The question is how to serve a current balance in single-digit milliseconds without re-deriving it from the entire history on every read. It comes down to a read-versus-write tradeoff. Whatever work we do to maintain a running balance on the write path is work a read no longer has to do. The bill lands on whichever side we underbuild.

The practical answer combines both with the striping from deep dive two. Keep the striped sub-balances as the fast current number. Then periodically write a snapshot, a checkpoint recording an account's balance as of a specific entry id, which is the same thing Books does with its cached balances. Now even a full rebuild from the journal only has to scan from the last snapshot forward. A balance read becomes a snapshot plus a short tail, which stays fast no matter how old the account is. The journal remains the strongly-consistent source of truth, and the snapshot is just a durable shortcut into it.

The scaling reads lesson covers maintaining derived read models off an authoritative log. The snapshot-plus-tail shape is the same idea as a database checkpoint: replay only the changes since the last durable point rather than the entire history.

4) Auditability, reconciliation, and fixing mistakes

A ledger's reason to exist is that any number can be explained, so this dive is about correctness over time rather than throughput. Two properties carry it. The journal is an immutable append-only log, and the ledger is reconciled against the outside world that really holds the money. The reversing entry is what makes corrections safe.

Because entries are never updated or deleted, a mistake is fixed forward. To undo a wrong 2000-cent transfer, we post a new transaction that moves 2000 cents back, a reversing entry. That leaves the original transfer and its reversal both on the record. The balance nets the two and ends up correct, and since nothing was erased the history stays complete. An auditor can trace every cent of a seller's balance back through the exact movements that produced it, including the mistakes and their corrections. That becomes impossible once a process can quietly edit a past number, which is why UPDATE-ing or DELETE-ing an entry is simply not an operation the ledger offers.

Reconciliation is the second layer, and the best candidates raise it without prompting. The ledger records what Block believes happened, but the card networks and banks hold the authoritative record of what money settled. A batch job runs off the request path and reads the bank's settlement files. It compares them line by line against the journal and flags any disagreement. That could be a sale the ledger shows as settled that the network reversed, or a deposit the bank reports that the ledger missed. A confirmed discrepancy is corrected the only way the ledger allows, with a reversing or adjusting transaction, never by editing the past. The atomic per-transaction guarantees keep each movement correct in the moment; reconciliation catches the rare drift that slips through and keeps the books honest against external reality over time.

The Block angle

Reconciliation against settlement is real Block operations, the layer where the ledger meets the card networks and banks. Block compares its internal journal against bank and network settlement files, since the money truly settles through Visa, Mastercard, and the banking rails rather than inside Square. Treat reconciliation as a first-class part of the design and it reads as operational maturity, the kind a regulated payments company demands. The pairing to make explicit: idempotent atomic writes keep individual movements correct, and reconciliation is the safety net for the disagreements no single write can catch. Reach for reversing entries over edits, the same append-only discipline Books runs on, and you have shown you understand that an auditable system fixes its mistakes forward.

What is expected at each level

That covers a wide surface for one primitive. What you have to land depends on the level you are interviewing for. Block's ladder runs L3 through L8, and the system design round weighs more the higher you go.

L4 (Software Engineer)

At L4, breadth carries it. Get to a correct ledger. Model a money movement as double-entry: two or more entries that sum to zero. Write all the entries of one transaction atomically in a single Spanner transaction, so it never half-applies. You should hold money in integer cents and never a float, and explain why the books must always balance. The scaling and audit cases are the panel's to raise. The bar: you can describe how a balanced, atomic double-entry transaction records money movement correctly, even if the hot-seller and reconciliation cases need prompting.

L5 (Senior)

Cover the breadth, then go deep on the hot seller. You are expected to find the per-seller bottleneck yourself. Spot that a single mutable balance row serializes a hot seller's writes on one lock and hotspots one Spanner shard. Propose append-only entries to remove the write contention, then striped sub-balances summed for the true balance. State the contention math: one row is one writer, N stripes are N writers. You should defend strong consistency on the journal and add the idempotency key so a retried movement posts once, tying it to Square's public API. The bar: you locate the hotspot unprompted and resolve it precisely, and you justify the strongly-consistent journal as a deliberate choice.

L6+ (Staff and up)

Depth across the hard parts and a widening scope. Expect to lead on several fronts at once. Snapshots and cached balances for fast reads. The consistency split between the strongly-consistent journal and the eventually-derived displayed balance. Reconciliation against bank and network settlement as the source of truth over time, sharding the ledger by account, and per-currency accounts so multi-currency balances never mix. You treat corrections as reversing entries on principle, and you drive the conversation while the panel redirects rather than rescues. The bar: you hold the per-seller hotspot and the audit story at the center of the design, and you reason about correctness end to end across Square and Cash App.