Design Change Streams

Plenty of systems need to know the instant data changes. A full-text search index has to reflect a document the moment it is written. A cache has to drop an entry that just went stale. An analytics warehouse wants every insert and update as it happens, and a second service might want to react to a specific field flipping. The blunt way to get that is to poll: re-query the collection every few seconds and diff the results against last time. Polling is expensive, it lags by the poll interval, and it quietly misses a change that happened and was overwritten between two polls. So you need a push feed of changes that never misses one and never reorders a single document's history.

The good news is that MongoDB already writes down every change it makes. A replica set keeps an oplog, a capped collection on the primary that records every data mutation as an idempotent operation, and secondaries stay current by tailing it and replaying it. That log is the source of truth for replication and failover recovery. It is also exactly the ordered record of changes a CDC consumer wants. So the whole design is less about producing a change feed and more about exposing the existing one safely. You need a resumable cursor over the oplog and a bookmark a consumer can hand back after a crash. You need correct ordering when the log is really one log per shard. And you need a way to serve many subscribers without reading the same oplog a hundred times.

The MongoDB angle

Change streams are a governed read over the same oplog that already drives replication. The strong answer inherits ordering and durability from it instead of reinventing them. The interviewer builds this engine, so they are listening for whether you reach for the oplog rather than a homegrown trigger table or a dual-write. Say plainly that the oplog is a capped collection of idempotent ops and that secondaries already tail it to replicate.

MongoDBPremium lesson · System design round

The MongoDB design, worked end to end.

1. 1The API and data modelSystem design
2. 2The design, drawn out2 diagrams
3. 34 deep divesWith the tradeoffs

Get the MongoDB questions

$49 a year · all 100 companies · cancel anytime

From 276 MongoDB reports, forum threads behind logins and write-ups in Chinese.

Read a free MongoDB lesson firstAlready a member? Sign in