Distributed task queue with retries, dead-letter, and metrics

Resume bullet (when finished)

“Built a Celery-based distributed task queue running on 4 worker nodes processing 12k jobs/min, with idempotent retries, dead-letter routing, Prometheus metrics, and a flake-investigation runbook used in real incidents.”

Locked tech stack

No "choose your language" — analysis paralysis kills completion. Follow the stack to the letter on your first build.

Milestones (6 · ~30h)

1. M1~4h

   Celery skeleton + first job

   FastAPI enqueues, Celery worker consumes. Compose stack: API, worker, Redis, Postgres.

   CHECK BEFORE MOVING ON:

   • Why Celery over RQ or Dramatiq here?
   • Where does the broker live and why?
   $ git commit -m "feat(queue): Celery + first job"
2. M2~5h

   Idempotent retries

   Jobs accept a `job_id`; the worker uses it as a Postgres uniqueness key. Re-running a retried job is a no-op.

   CHECK BEFORE MOVING ON:

   • Why idempotency keys instead of 'check before write'?
   • What's at-least-once vs exactly-once and which do you really have?
   $ git commit -m "feat: idempotency keys + safe retries"
3. M3~5h

   Exponential backoff + max retries

   Failed jobs retry at 2s, 4s, 8s, 16s, 32s with jitter. After 5 fails → dead-letter.

   CHECK BEFORE MOVING ON:

   • Why jitter and not pure exponential?
   • What's the trade-off in choosing max-retries?
   $ git commit -m "feat: exponential backoff with jitter + DLQ"
4. M4~5h

   Dead-letter routing + replay tool

   DLQ stored in Postgres. CLI tool `dlq-tool list / replay / drop`. Reasons captured.

   CHECK BEFORE MOVING ON:

   • Why store the DLQ in Postgres and not Redis?
   • What's the difference between 'replay' and 'drop'?
   $ git commit -m "feat(dlq): Postgres DLQ + replay CLI"
5. M5~6h

   Prometheus + Grafana dashboard

   Throughput, error rate, retry rate, DLQ depth, P95 job duration. Dashboard JSON committed.

   CHECK BEFORE MOVING ON:

   • Which metric do you wake up at 3am for?
   • What's the difference between a counter, a gauge, and a histogram?
   $ git commit -m "ops: Prometheus + Grafana dashboard"
6. M6~5h

   Runbook + load test

   12k jobs/min on 4 workers. Runbook for 'queue depth growing', 'worker crashloop', 'DLQ flooding'.

   CHECK BEFORE MOVING ON:

   • Why does the runbook live in the repo, not Confluence?
   • What's the first command you run when queue depth is growing?
   $ git commit -m "docs: runbook + load test results"

60-second demo storyboard

What you say in the recruiter screen when they ask "tell me about your latest project." Practice it out loud.

1. 0-5s: 'Distributed task queue: 12k jobs/min, idempotent retries, DLQ + replay.'
2. 5-25s: live demo — submit 1000 jobs, watch the Grafana dashboard light up.
3. 25-45s: intentionally crash a worker, show retries on the dashboard, no jobs lost.
4. 45-60s: walk through one DLQ entry and the replay command.

STAR talking points for behavioral round

Production references — how grown-up systems do this

Self-review rubric (before you claim done)