Message Queues & Event Streaming

Not every operation needs to complete before the user gets a response. When a user uploads a video, you don't transcode it to every resolution before responding. You accept the upload, return "processing," and handle the heavy work asynchronously. Message queues and event streaming platforms make this possible.

These technologies appear in nearly every non-trivial system design. They decouple producers from consumers, absorb traffic spikes, and enable event-driven architectures. This lesson covers when and how to use them.

Strip the vocabulary away and a queue buys you three things. Decoupling first: the upload handler does not need to know that a transcoder exists, how many are running, or whether any of them are up. Second is buffering. The producer writes as fast as traffic arrives, the consumer reads as fast as it can sustain, and the difference piles up in the queue instead of in a timeout. The third gets undersold. Retry stops being code you write, because a failed HTTP call is gone unless you built the retry yourself, while a failed message goes back on the queue and comes around again.

What you gave up is the straight answer. The producer learns that the message was accepted and nothing more. Whether the work succeeded, and what the user hears if it never does, are now yours to design. Most of the operational weight of an async system sits in that gap.