Tokenizers & Text at Scale

Text goes into a model as integer ids and comes back out as text, and every training run starts by turning a very large corpus into those ids. Anthropic's coding rounds use both ends of that pipeline. One problem is the tokenizer itself, a vocabulary of strings mapped to ids, with an encoder that picks the longest match and a decoder that gives back exactly what went in. The others take a counting job that's easy on one machine and ask you to run it across workers or nodes.

The hard parts are at the edges, like a character the vocabulary never saw, or a word that a byte boundary cuts into bro and wn. In the sharded problem, it's a value spread thinly over many nodes, so no single node's local count shows it winning. Each problem is built around one of those edge cases. A solution that ignores it passes the easy tests and fails the first awkward one.

The Anthropic angle

Anthropic builds the tokenizer its models read through, and the data behind those models is sharded across machines, so both show up in the questions. The tokenizer round usually starts with reading someone else's implementation and finding its bug, so state a property like a lossless round trip before anyone asks. In the scale-out problems, the cost is whatever crosses the network. With a bandwidth limit, you want a worker that returns a small summary, or a node that sends one count per value instead of a million rows.

AnthropicPremium lesson · Coding round

The 3 Anthropic problems are below.

1. 1A Premium questionCoding · Medium
2. 2A Premium questionCoding · Medium
3. 3A Premium questionCoding · Hard

Each with hints, the follow-up and a tested solution.

Get the Anthropic questions

$49 a year · all 100 companies · cancel anytime

From 20,000+ candidate reports, forum threads behind logins and write-ups in Chinese.

Read a free Anthropic lesson firstAlready a member? Sign in