Design Log Ingestion & Search

"Design a service like Datadog, and focus on the log part" is one of the two design prompts the round reaches for, asked in almost those words. It often shows up phrased as "design a streaming service" when the interviewer wants the ingest side. The shape is concrete. Millions of hosts and services emit log lines without pause. Each line has a timestamp plus a service name and a host, with a level like error or info. It also carries a free-text message and a bag of tags such as env:prod, region:us-east. Our job is to swallow that stream, store it cheaply, then answer two kinds of read. One is search: "show me error AND timeout for service:checkout over the last fifteen minutes," which spans recent hot data and three-week-old cold data. The other is a Premium question: a stream of new logs matching a filter, scrolling in front of the user as they arrive. The volume is the headline, so we lead with it.

Lock the vocabulary down before drawing; every later choice rides on it. A log event is the structured record above: timestamp, service, host, level, message, tags. The raw event is the blob we keep so a user can read the actual line. An index is a separate structure built over those events so a search does not scan them all. An inverted index, specifically, maps a term like timeout to the set of events that contain it. That split is the spine of the design, so fix it early. The raw events and the index are two different things, stored two different ways, and the single hardest cost decision is how much of the stream we index versus merely store.

The Datadog angle

This isn't "design something like Datadog." This is Datadog Logs, and the person across the table ships it. The design round is where seniors get sunk, the strong coder who turns out thin on design. A hand-wave that passes elsewhere stands out here.