Design a Threat-Detection Pipeline

Understanding the prompt

CrowdStrike's coding bar is ordinary LeetCode, so the company shows its hand in the design round. It pulls the prompt straight out of the Falcon product. You get a version of the job CrowdStrike does: take objects in, decide whether they're dangerous, and do it at a volume where the naive approach falls over. The prompt arrives in three phrasings of the same problem. One is bare: design a system that reads files and performs threat detection. One is spelled out: a service that accepts files from users and produces security findings by calling a third-party scanner. One is a take-home: design a VirusTotal-like system with uploads, hashing, blob storage, and an event-driven flow. Open by naming that they converge. Then design the one pipeline all three describe.

So skip the consumer template. This isn't a viral feed where the hard part is fan-out and read scale for a hot timeline. A file comes in, and you must not redo work you've already done. You lean on an external scanner you don't control. The output is a verdict someone acts on, so losing an item or being wrong has consequences a dropped tweet never does. The interesting constraints are throughput on an ingestion path and the cost of scanning the same bytes twice. Add the correctness of a verdict, and what happens when the scanner is slow or down. Aim the whole conversation there.

The CrowdStrike angle

This is a security data pipeline. Dedup, async detection, and an unreliable dependency replace caches and feeds as the load-bearing concerns. In design Twitter, the pressure is absorbing a burst of reads for a viral post. Here it's different in kind. You accept a stream of files, most of which you've seen before. You must land the new ones durably before anything clever runs. You call a third-party scanner that has rate limits and outages. And you emit a malicious-or-clean finding that a customer's response depends on. Expect an asynchronous ingestion-and-detection pipeline and build it around content-hash dedup, and you're speaking this round's language. Reach for a read-cache and a fan-out fan, and you're answering a question nobody asked.

Scoping the requirements first

The prompt is thin on purpose, and a minute spent bounding it keeps the design from sprawling. State the functional surface plainly: a client submits a file or object and quickly gets back an acknowledgement, with no verdict on the spot. The system stores the raw bytes durably and decides whether the file is malicious. It makes that verdict available to whoever asked: by callback, by polling, or in a findings store a dashboard reads. If the same bytes have been seen before, the answer comes back without redoing the work. That's the core, and it's enough to design something real.

Then draw the line between what you build now and what you name and defer. Don't specify the exact detection model, the storage sharding scheme, or the full findings query language on the first pass. Model the skeleton so those pieces have an obvious home, then say so. Detection starts with a third-party scanner plus a few cheap internal signals, and grows into a richer detector fleet. The findings store starts as a keyed lookup by hash and grows into a queryable index. Both have a named place to live. Naming the deferral is the judgment the round looks for. Pin the non-functional bar out loud too, because for this system it decides the shape.

• The expensive resource: a scan, not a request. The costly thing is not serving a request, it is scanning a file: reading the bytes, calling a rate-limited third-party scanner, running detectors. The whole design is organized to spend that cost as rarely as possible, which is why dedup, not caching a read, is the center of gravity.
• Intake and detection differ: fast ack, slow verdict. Accepting a file and landing it durably has to return in milliseconds. Producing a verdict can take seconds or longer because it may wait on an external scanner. These run on different clocks, so the intake path returns before detection has happened and the verdict arrives asynchronously.
• A verdict is acted on: no silent drops. Someone quarantines or clears a file based on the finding, so an item cannot vanish and a verdict cannot be quietly lost. That makes at-least-once processing, idempotent workers, and a dead-letter path core requirements rather than polish added at the end.
• Most files are repeats: dedup is the lever. The same identical binary arrives from many sources, so a large share of submissions are hashes already resolved. Turning a repeat into a single lookup that returns the cached verdict is what makes the scanning bill workable, and it is the insight the generic prompts never force.

Estimation, with illustrative numbers

Before architecture, do the arithmetic out loud, and frame every number as a back-of-envelope illustration rather than a CrowdStrike figure. The interviewer is not checking whether you memorized a real submission rate. They are checking whether you can reason from an assumed load to the decisions it forces. So state your assumptions, keep the numbers round, and let the dedup ratio drive the design. A candidate might reason as follows.

Assume on the order of:
  submissions          ~1e9 files/day    -> ~12k files/sec average, peak a few x
  duplicate ratio      ~90% are a hash already resolved
  genuinely new files  ~10% -> ~1e8/day  -> ~1.2k full scans/sec average
  average file size    ~1 MB
 
New durable bytes per day (the only bytes we must store):
  ~1e8 new files x ~1 MB  ->  assume on the order of ~100 TB/day of new raw blobs
  verdict record          ~1 KB per hash  ->  tiny next to the blobs
  hash -> verdict index    ~1e8 new keys/day, small values, hot lookup

Two conclusions fall out of that math. First, dedup is the design itself. It collapses on the order of a billion submissions a day into a hundred million actual scans. That's a tenfold cut in the one path that costs real money, and the third-party scanner only ever sees a hash it hasn't resolved. Second, the two paths through the system have opposite shapes, so keep them apart in your head.

Cheap path (a repeat, the common case):
  hash the bytes -> look up the verdict store -> return the cached finding
  no blob write, no queue, no scan, answered in milliseconds
 
Expensive path (a genuinely new hash):
  hash -> store raw bytes -> enqueue -> scan (third-party + internal) -> write verdict
  seconds+ of asynchronous work, bounded by the scanner, done once per hash
 
-> the dedup ratio is the single biggest lever on both cost and latency

That contrast is the spine of the whole design. The intake path is throughput-bound and has to return fast for every submission, repeat or not. The expensive detection path runs asynchronously and only for new hashes, bounded by how fast the scanner and detectors can clear the queue. Every later decision, what to cache, what to queue, where to shard, has an obvious answer once you hold those two flows apart.

The high-level architecture

With the two paths named, the architecture is the wiring between them. Draw it and read it by role. Blue is an actor at the boundary: the client that submits files, the third-party scanner the system depends on but doesn't own, and the consumer that reads findings. Amber is the queues that decouple stages and catch failures. Green is the durable state the system owns, the raw blob store and the verdict-and-findings store. Violet is the services that hash, scan, and notify. Follow the flow top to bottom. A submission hits intake, which hashes it and checks the verdict store. A known hash returns at once; a new one is written to the blob store and enqueued. Scanner workers pull the queue, read the bytes, and call the third-party scanner and the internal detectors. They write the verdict and emit a finding. Messages that keep failing fall to a dead-letter queue.

The content-hash dedup spine

This is the heart of the answer, so make it the spine. Compute a content hash of every file at the intake boundary and key everything on it: the stored object, the verdict record, the queue message, the finding. The rule is simple: if you've seen the hash before, you already have its verdict, so return the cached finding and stop. Only a hash you've never resolved earns a blob write, a queue message, and a scan. That one decision turns an impossible scanning bill into a workable one. The same binary flows in from many sources, and re-scanning it every time burns your one expensive resource for no new information.

Getting dedup right means having answers ready for the interviewer's pushes. Use a strong cryptographic hash, so an attacker can't craft two different files with the same digest and smuggle a malicious payload behind a clean verdict. A fast non-cryptographic hash is a false economy here, because collision resistance is a security property as well as a performance one. Keep the hash lookup cheap and hot, since it sits in front of every submission. It lives in a fast keyed store sized for the whole known-hash space, beyond just the new-bytes rate. Treat a verdict as revisable. When a scanner reclassifies a file that was once clean, invalidate or version that hash's entry and let it be re-scanned. That keeps the cache from calcifying a stale answer. You can even warm it from known-good and known-bad hash sets. Then a fresh binary from a trusted vendor or a known-malware family resolves without a scan at all.

The CrowdStrike angle

Dedup by content hash is the load-bearing idea. Reach for it unprompted, then defend it. A file-scanning service can run at this scale because it has seen most of what it sees before. The design that survives the interviewer lets a repeat skip blob storage, the queue, and the third-party call in a single lookup. Arrive at that on your own. Then talk about collision resistance as a security requirement, cache invalidation when a verdict changes, and warming the cache from known sets. That reads as someone who has thought about this class of system. Hash only to name files and re-scan identical bytes every time, and your system's cost can't scale. The round is built to expose that.

Asynchronous processing and delivery guarantees

Behind the queue, the detection path is asynchronous by necessity. A scan can take seconds and may wait on an external service, and you can't hold the intake request open that long. So intake returns once the bytes are durable and enqueued, and the workers do the slow work on their own clock. The guarantees on that queue decide whether someone can trust a verdict or the pipeline quietly loses files. Walk them explicitly.

Notice how these interlock. At-least-once would be dangerous on its own because it duplicates, but idempotency keyed on the hash makes a duplicate a no-op. Retries would be dangerous on their own because a permanent failure retries forever, but the dead-letter path gives a doomed message somewhere to go. The set is the design; naming one without the others is the answer that looks right and fails the follow-up.

The third-party scanner as an unreliable dependency

Detection leans on an external scanner. The interviewer wants to see you treat it as something you don't control, a call that may never return. It has rate limits you can exceed, latency that spikes, and outages that last minutes or hours. The failure being tested is letting one slow dependency stall the entire line. So make the integration a boundary you can degrade behind. Put a per-call timeout on it so a hung request doesn't pin a worker. Retry transient failures with backoff and jitter. Respect its rate limits with a client-side limiter, so you throttle yourself before it throttles you. Wrap it in a circuit breaker that trips when its error rate climbs. When the scanner is clearly sick, you stop sending it traffic and fail fast. The affected files stay pending instead of burning workers on calls that will time out anyway.

Two moves keep the pipeline useful while the scanner is down. First, the verdict cache does double duty. Every third-party answer is stored by hash, so the scanner is called at most once per new file and a repeat never touches it. Its bill and its load track the dedup ratio, not the submission rate. Second, the internal detectors give you a floor. Cheap static signals and known-bad and known-good hash lists can settle the obvious cases without the external call. A scanner outage then degrades confidence on the ambiguous middle instead of halting detection entirely. When the breaker is open, new ambiguous files sit pending, with their bytes safely stored and their messages retained. They drain through as soon as the scanner recovers. An outage costs latency on some verdicts, and no files get lost.

The CrowdStrike angle

Design the scanner integration to fail without taking the pipeline with it. The round probes this boundary hard. The hire instinct assumes the external service will be slow, rate-limited, or down, and makes that survivable. Say out loud that the call is wrapped in a timeout, a bounded retry, a rate limiter, and a circuit breaker. Its answers are cached by hash, so it's hit at most once per new file. Internal detectors keep the system partly functional when it's degraded. Then say what happens to an in-flight file when the breaker trips. It stays pending with its bytes durable and its message retained, and it resolves when the dependency comes back. Make the unreliable dependency a boundary to degrade behind. A step in the middle that everything waits on is the design this prompt is built to catch.

Scale and throughput

Scaling this pipeline mostly means scaling each stage independently, the payoff for keeping the paths apart. The intake tier is stateless behind a load balancer, so it scales horizontally with the submission rate. The only shared thing it touches per request is the hot hash lookup. That's why that store has to be fast and sized for the whole known-hash space. The queue absorbs bursts, so the worker fleet doesn't need provisioning for peak. Workers scale on queue depth, adding capacity as the backlog grows and shedding it as the queue drains. The blob store and verdict store shard by content hash. The hash is uniform, so load spreads evenly, and all state for a given file sits under one key.

• Intake tier: stateless, scales flat. The upload-and-hash path holds no per-file state beyond the shared hash lookup, so it scales horizontally with submissions behind a load balancer. Keep the hash-to-verdict store fast and well-sized, because it sits in front of every request and a slow lookup here is a slow lookup for everyone.
• Worker fleet: scales on queue depth. Detection workers are the expensive, bursty part, so autoscale them on backlog rather than provisioning for peak. The queue turns a spike into depth the fleet works down, and depth becomes the control signal: growing means add workers or the scanner is the bottleneck, draining means scale in.
• Storage sharding: shard by content hash. Both the blob store and the verdict store shard on the hash, which is uniform, so load spreads evenly with no hot key and every artifact for one file lives under a single key. It also means an identical file always lands on the same shard, reinforcing dedup instead of fighting it.
• The real ceiling: the scanner, not your tiers. Past a point the bottleneck is the third-party scanner throughput and rate limits, not your own services. That is why dedup and internal detectors matter for scale and not only cost: every file they resolve without the external call is throughput you did not have to buy from the dependency.

Reliability and observability

In this domain, "we would add monitoring" is weak, and naming the specific signals is strong. A verdict nobody notices is stuck is as bad as a wrong one. Start with reliability. The no-silent-drops promise comes from the ordering: land the bytes durably and enqueue before acknowledging intake. Anything the system accepted is then recoverable, even if a worker dies mid-scan. At-least-once delivery plus idempotent workers keyed on the hash mean a crash or redelivery re-derives the same verdict. Nothing is lost or doubled. The dead-letter queue quarantines and counts a permanently failing item, so it never gets silently retried into oblivion.

Then be concrete about how you'd know the pipeline is healthy. The interviewer is listening for these signals. Queue depth and its rate of change tell you whether detection keeps pace or falls behind. Per-stage latency shows where the line is slow, especially time-to-verdict and third-party call latency. Scan success rate and scanner error rate show whether the external dependency is degrading, and the circuit breaker watches them. The dead-letter count and the number of items stuck pending point to poison messages and an unhealthy scanner. Watch the dedup hit rate too. If it drops, the scanning bill climbs, and a sudden change often means an upstream client stopped deduping or started sending novel files. Alert on the leading signals, rising queue depth and a climbing dead-letter count. That catches a backed-up or failing stage before verdicts go stale.

What the interviewer probes

The round escalates by pushing on the pressure points, and if you built the design this way the answers are already in it. Expect the pushes to cluster. On cost and scale: how does this not cost a fortune? Content-hash dedup collapses repeats into a single lookup, so the scanner is hit once per new file. On the external dependency: what happens when the scanner is slow or down? The timeout, retry, rate limiter, circuit breaker, and internal-detector floor answer it, with in-flight files pending and their bytes durable. On correctness under failure: what if a worker crashes mid-scan or a message arrives twice? At-least-once plus idempotency keyed on the hash covers it. On staleness: what if a clean file is later found malicious? The revisable, versioned verdict re-opens that hash instead of trusting a permanent cache. On operability: how would you know it's broken? The named signals: queue depth, time-to-verdict, scan success rate, dead-letter count, items stuck pending.

The CrowdStrike angle

Reason from the pipeline, and the design defends itself. This round checks whether you can take an unfamiliar, security-flavored data problem and build something sound on it. A template rehearsed for a different company won't pass. So talk about the data flow first. State what a submission carries and hash it at the boundary. Decide what a repeat short-circuits and what a new file earns. Only then draw the async detection stage and the third-party call. On reliability, be concrete about at-least-once, idempotency, and the dead-letter path. Treat the external scanner as a dependency to degrade behind. Start from the real constraints: dedup, async processing, an unreliable scanner, and a verdict someone acts on. That reads as someone who could own a piece of the systems behind Falcon, the bar this round is set to find.