Design an Experimentation Platform

"Design an experimentation platform" lands on the whiteboard as one phrase hiding two very different systems. Split them out loud before you name a datastore. One half is the write path: when a user shows up, decide which version of a feature they get, the control or one of the variants, and make that decision fast, consistent, and honest. The other half is the read path: watch what those users then do, add it up per variant, and decide whether the difference between control and variant is real or noise. Assignment and measurement. Keep them apart and the design stays coherent under every follow-up. Blur them and you end up measuring an experiment whose users were never cleanly split in the first place.

The trap is to start with the dashboard, the part a product manager sees, and treat the bucketing as a coin flip you wave away. It is the opposite. Every number the platform reports depends on assignment being deterministic and sticky. A user who sees the control on Monday and the variant on Tuesday belongs to neither group and quietly poisons both. Build the assignment service so it cannot do that, then build a measurement pipeline that only ever counts users who genuinely saw the thing. Get those two right and the statistics on top are almost mechanical. Get either wrong and the prettiest significance test in the world is computing a p-value for garbage.

The Affirm angle

On most products a bad variant costs you a few percent of clicks. On Affirm's it can cost real money. Affirm is point-of-sale lending. The flows people experiment on are checkout and the approval decision. That is where the company approves a good borrower, declines a bad one, or gets it wrong and lends money it will not get back. So an experiment here is never just a color change. A variant that nudges approval rates up looks like a win on conversion and can be a loss on defaults that only shows up weeks later. That reframes what the platform is for.