Design an Idempotent Webhook Delivery API

A producer, some internal service, wants to tell a subscriber that something happened: a payroll run finished, a device was enrolled, an invoice posted. It calls one endpoint with a payload and a destination, then hands off. From there the service owns the hard part, which is getting that notification to a possibly flaky receiver over a network that drops things. Two words carry the whole design. Idempotent, so a producer that retries a timed-out send never causes a second delivery. And at-least-once, so a delivery that might have failed is tried again until it lands or is given up on. Everything below is those two words made concrete in tables and endpoints.

This is a low-level design, so the interviewer is not after a planet-scale capacity plan. They want the contract and the tables. What exactly does the send endpoint accept and return? What happens on the second identical call? Which table carries the unique constraint that makes de-duplication real, and what is that constraint on? When a delivery fails for the ninth time, where does the message go, and how does someone get it back? The scale is modest on purpose, a few thousand deliveries a second at most, so the pressure sits on correctness under retries and concurrency rather than on sharding.

Anchor a few numbers so the choices carry weight. Picture a service fielding on the order of a few thousand deliveries a second across many subscribers. Individual receivers vary wildly. One is a fast API that acks in twenty milliseconds; another takes several seconds or times out entirely. A delivery is retried for up to about a day before it is declared dead, which fixes the retry schedule. An idempotency key is remembered for around twenty-four hours, long enough to cover any sane client retry loop, after which the key can be reused. None of these numbers force sharding. They force the design to stay correct when a receiver is down for an hour and a producer is retrying the whole time.