Design the LLM API & Rate Limiting

The Claude API caps a key on three axes at once, and the docs say so plainly: requests per minute (RPM), input tokens per minute (ITPM), and output tokens per minute (OTPM). Cross any one and the call comes back with a 429 and a retry-after hint, and every successful response carries the remaining budget back in headers. A short tool-use turn might burn one request and forty tokens. A long document summary burns one request and twenty thousand. The token axes are the ones that bind for real workloads, and they are exactly the ones a request counter cannot see. Anthropic's rate limits docs also say the API uses a token bucket, so capacity refills continuously instead of resetting at fixed intervals.

That input and output tokens get their own per-minute budgets is the detail the whole design turns on. The input is free to measure: the prompt is sitting right there, run it through the tokenizer and count. ITPM enforcement is a normal counter problem.

Output is the hard half. The model decides how long its answer is while it generates, one token at a time, and a request capped at four thousand output tokens might stop after thirty or run to the cap. So the cost against OTPM is unknown at the exact moment you have to decide whether to admit the call. One axis you can count, one axis you have to predict, and they are billed and limited separately. That split earns this its own lesson.

The Anthropic angle

A request limiter charges for something that already happened. The Claude API's OTPM limit charges for tokens that do not exist yet. The move the rest of the design is built on is reserve-then-reconcile, and the ITPM/OTPM split makes it sharper than a single combined token budget would. At admission you count the input exactly and debit it against ITPM, and you estimate the output and reserve that many tokens against OTPM up front. Run the generation. When it finishes you have the true output count, so you settle up.