Design a Distributed Job Scheduler

Scheduling work across a cluster is the engine under Databricks' entire product, which is why this comes up so often. It sounds tame. Accept jobs, run them on a pool of machines, report when they finish. The shape of a job and the size of the cluster are what make it real. A job is not one task. It is a graph of tasks with dependencies, a DAG, where some tasks cannot start until others finish. The cluster is thousands of workers that come and go. And the promise is reliability: once a job is accepted, it runs to completion and produces the right result, even when individual workers die mid-task and even when the scheduler itself restarts. Almost every hard part flows from that promise meeting that scale.

Picture the workload concretely so the numbers steer the design. A cluster has on the order of 10,000 workers, each with something like 16 cores and 64 GB of memory. A single large analytics job can fan out into tens of thousands of tasks. At peak the system places thousands of task starts per second, and a task that becomes ready should land on a worker in well under a second. Each task declares what it needs, say two cores and 8 GB, and the scheduler has to fit it onto a worker with that room free. Three facts shape everything from here. Work arrives shaped like a dependency graph, where some tasks gate others. The cluster is large and unreliable, so a worker dying is routine rather than exceptional. And the result has to be correct after any single failure, which means the scheduler can never lose track of which tasks ran and which still owe their output.

The Databricks angle

This is Spark's scheduler, and your interviewer has likely shipped one. Databricks is built on Apache Spark, whose core is this exact design. A job becomes a DAG of stages, and a DAG scheduler turns dependencies into runnable tasks. A task scheduler places those tasks on executors, and the whole thing keeps running while executors are lost and replaced.