Design Tweet Search

It looks like a database query, find the rows whose text contains these words, but a database is the wrong tool here. You cannot scan billions of tweets on every search, and no B-tree on a text column helps you find a word buried in the middle of a sentence. Search is an information-retrieval problem, and its core data structure is the inverted index, a map from each word to the list of tweets that contain it. Build that well and a query becomes a few list lookups instead of a scan. The second half of the problem is the one that makes it specifically Twitter's. A web search engine can afford to rebuild its index every few minutes. Tweet search cannot, because the point is the live conversation, so the index has to absorb new posts and make them findable within seconds.

The X angle

The freshness requirement separates this from a generic search design, and it is why Twitter built its own engine, Earlybird, instead of running stock Lucene. Finding tweets is easy; finding a tweet posted two seconds ago is the hard part. Earlybird is a real-time search system built on a heavily modified Apache Lucene, sharded and replicated, with a result aggregator historically called Blender in front of it. The public engineering write-ups describe the shape to reach for: posting lists in the inverted index run newest-first. A small mutable segment accepts brand-new tweets with no rebuild, and larger immutable segments are tuned for the older corpus. Reaching for Elasticsearch and stopping is a weak finish. Name the standard tool, then spend your time on the freshness path and the ranking, because that is where a search design earns its score. Treat the architecture here as the well-documented public design rather than a claim about X's exact current internals.