Design a News Aggregator

A news aggregator is a reader over the open web. Nobody posts to it. It pulls from a large set of external sources: an RSS feed in one place, a publisher API in another, a scraped homepage where nothing cleaner exists. It turns that raw firehose into one clean, ranked, deduplicated feed. Nobody using it wants the same wire story from forty outlets forty times. They want the story once, with the outlets that carried it hanging off it, ordered so the important and the fresh rise to the top. That collapse, many near-identical articles into one story, is the whole game.

Picture the shape before drawing boxes. Data enters from sources you do not own. You cannot trust them to be fast or well-formed, and some are not even online when you knock. It leaves as a paginated feed a reader scrolls in a couple hundred milliseconds. In between sit three hard problems. You have to fetch from a million sources without hammering any of them or missing breaking news. You have to decide when two articles are the same story, which is a similarity problem with no exact answer. And you have to rank and serve the result fast enough that a spike in readers does not melt the system. Miss any one and the product fails in an obvious way. It reads stale. It clogs with duplicates. Or it simply runs slow.

Anchor on numbers before anything else. Target roughly a million sources, polled on an adaptive schedule so a fast-moving newsroom gets checked every few minutes and a dormant blog every few hours. Those sources produce on the order of ten million new articles a day, a bit over a hundred a second on average and several hundred a second when news breaks. On the read side, assume tens of millions of daily readers pulling their feed several times a day, which peaks near eight thousand feed requests a second. Two tensions fall out of those numbers and steer the design.