Worked Example 15.15 · 15 of 15

Design a distributed job scheduler

📖 4 min read · 10 sections · May 2026

Asked at: Airbnb Uber LinkedIn

The setup

"Build a scheduler for the company. Teams register cron-style jobs; the system runs them at scheduled times across a worker pool; provides history, retry logic, observability. 1M scheduled jobs; ~10K jobs running per minute at peak."

Clarifying questions

Question	Story answer
Job trigger types?	Cron-style recurring, one-shot at time T, delayed N seconds.
What does "run a job" mean?	HTTP POST to a registered endpoint with parameters.
Retry semantics?	Configurable: max attempts, backoff strategy.
Latency tolerance for trigger time?	±2 seconds.
What happens if a worker fails mid-job?	Detect and re-run (lease expiration).

The core architecture

┌─────────────────────────────────┐
│  Schedule Store (Postgres)       │
│    table: jobs                   │
│      (id, schedule, next_run,    │
│       status, retry_policy, ...) │
└──────────┬───────────────────────┘
           │
           ↓ poll for due jobs
┌──────────────────────────────────┐
│  Scheduler (partitioned)         │
│  - Find "due now" jobs           │
│  - Dispatch to worker queue      │
└──────────┬───────────────────────┘
           │
           ↓ enqueue
┌──────────────────────────────────┐
│  Job Queue (Kafka / SQS)         │
└──────────┬───────────────────────┘
           │
           ↓ consume
┌──────────────────────────────────┐
│  Worker Pool                     │
│  - Pull job; lease it            │
│  - Execute (HTTP call)           │
│  - Update status; release lease  │
└──────────────────────────────────┘

Design Decision 1: finding "due now" jobs

Scheduler runs every ~10 seconds. Query:

SELECT id, job_def, next_run_time
FROM jobs
WHERE next_run_time <= now() + interval '30 seconds'
  AND status = 'scheduled'
  AND partition_id IN (my_partitions)
ORDER BY next_run_time
LIMIT 1000
FOR UPDATE SKIP LOCKED;

-- For each: mark as 'dispatched', enqueue to job queue.

Why FOR UPDATE SKIP LOCKED: multiple scheduler instances can run concurrently. Each takes its own batch without blocking on the others.

Partitioning: jobs are sharded by hash(job_id) % N. Each scheduler instance owns specific partitions. Avoids all-instances-contend-on-same-rows. Each partition's load is independent.

Design Decision 2: the lease (failure tolerance)

When worker picks up a job, it claims a lease on it:

UPDATE jobs SET
  status = 'running',
  worker_id = $worker_id,
  lease_until = now() + interval '5 minutes'
WHERE id = $job_id AND status = 'dispatched'

Worker heartbeats every minute to extend the lease. If worker dies:

No heartbeats → lease expires
A background reaper job finds expired leases: UPDATE jobs SET status='scheduled' WHERE lease_until < now() AND status='running'
Next scheduler poll picks this job up again — re-dispatched to a new worker

This is the foundational at-least-once execution model. Combined with idempotent job code, you get reliable execution.

Design Decision 3: retry policy

Jobs can fail. Configurable retry per job:

retry_policy: {
  max_attempts: 5,
  backoff: {
    type: 'exponential',
    base_seconds: 60,
    max_seconds: 3600
  }
}

On failure:
  attempt = current_attempt + 1
  if attempt > max_attempts: status = 'failed', dead-letter
  else: next_run_time = now() + backoff(attempt), status = 'scheduled'

Cross-examination round 1: scheduling precision

Interviewer: Job scheduled for noon UTC. What's the worst-case fire time?

Scheduler polls every 10 sec. Worst case: poll happens at 11:59:55; misses the noon job (only 5 sec until due, not 30 sec). Next poll at 12:00:05; picks up the job; dispatched immediately to worker queue; worker starts within 1-2 sec. Worst case fire time: 12:00:05-07 — about 7 sec after scheduled. Acceptable for the ±2 sec requirement? No — need to tighten. Solutions: poll more frequently (every 1 sec for low-latency jobs) or use timer-wheels (in-memory data structure that fires events at precise times for known-soon jobs).

Cross-examination round 2: hot job problem

Interviewer: One job is scheduled to run every second (1M-times/day cron). Other jobs are quieter. Doesn't this become a hot row?

Yes. Mitigations: (1) Move ultra-frequent jobs to a dedicated tier — a scheduler instance per high-frequency job. (2) Don't store as a "schedule" — instead, the worker for that job loops continuously, reading config for its execution interval. Trade flexibility for performance. (3) Rate-limit job scheduling: cap individual jobs at one execution per minute as a platform policy; users who need higher frequency build their own.

Cross-examination round 3: dependencies between jobs

Interviewer: Job B should run only after Job A succeeds. How?

Three levels of complexity:

Simple: Job A's success-handler enqueues Job B. Decoupled; no native dependency tracking.
Medium: DAG-based scheduling (Job B depends on A; scheduler tracks). Like Airflow.
Advanced: workflow orchestration (Temporal, AWS Step Functions). Job is a long-running workflow with persisted state.

For most cron-style schedulers, simple is sufficient. For complex workflows, use Temporal/Airflow.

Takeaway

Distributed schedulers are the foundational subsystem for any data platform — every team needs cron at scale. The combination of (persistent schedule store + partitioned poller + worker pool with leases + retry policy) is the standard recipe. The hardest part isn't the scheduling itself; it's the failure semantics: what happens when a worker crashes mid-job? Idempotency at the job level, plus lease-based ownership, plus retry policies — together they make a scheduler reliable. Quartz, Cadence, Temporal, Sidekiq, Cron — all variations on this pattern.

The deeper insight: at-least-once execution is the only practical model. Exactly-once is theoretically possible only at high cost and complexity. At-least-once with idempotent job code gets you exactly-once semantics at orders of magnitude lower cost. This pattern applies far beyond schedulers: every reliable distributed execution system is at-least-once + idempotent.