Losses, Sampling & K-Means

These drills are for the OpenAI Research Engineer track. Software engineers won't be asked for them and can skip ahead. Even on the track, they're a second tier: attention from scratch and the model-debug round come up more, so drill those first. Candidates do report sampling routines and classic ML written without a library, and the softmax cross-entropy gradient is the derivation they're expected to know cold. That makes three functions worth drilling: the loss with its gradient, one decoding step under temperature and top-k, and k-means. K-means sits with the classic-ML warm-ups, while the loss and the sampler connect to the transformer questions. Each one fits in about a dozen lines. What makes them hard is one numerical or ordering detail in each, and that detail decides whether your output can be checked against an expected answer. For the loss, you shift by the max before every exponent and take the log as a log-sum-exp. The sampler should return the distribution and leave the random draw as one line on top, and k-means needs a fixed rule for ties and empty clusters.

Spend the most time on the cross-entropy gradient. Candidates report being asked to derive it by hand, and it trips up a lot of them. Writing it as code shows two mistakes a whiteboard version hides: an exponent that overflows, and a full Jacobian you never needed to build. Follow-ups can add top-p sampling after top-k, or k-means++ seeding after plain k-means.

OpenAIPremium lesson · Coding round

The 3 OpenAI problems are below.

1. 1A Premium questionCoding · Medium · Frequently asked
2. 2A Premium questionCoding · Medium
3. 3A Premium questionCoding · Medium

Each with hints, the follow-up and a tested solution.

Get the OpenAI questions

$49 a year · all 100 companies · cancel anytime

From 20,000+ candidate reports, forum threads behind logins and write-ups in Chinese.

Read a free OpenAI lesson firstAlready a member? Sign in