Skip to main content
IncidentPaged 20:05 UTC, Sunday (weekly traffic peak)

Cache stampede — DB CPU spikes every 60 seconds exactly

Service: product-api (Flask + Redis cache + Postgres)

Symptom: DB CPU oscillates 8% → 92% → 8% in a clean 60-second sawtooth. P99 latency jumps from 12ms to 1.4s at each spike. Hot products only.

The code that's running in prod

# product_api.py
# GET /products/<id> — cached in Redis for 60 seconds.

import json


def get_product(product_id, redis, db):
    key = f"product:{product_id}"
    cached = redis.get(key)
    if cached:
        return json.loads(cached)

    # Cache miss — fetch from DB and populate.
    product = db.query(Product).get(product_id)
    redis.setex(key, 60, json.dumps(product.dict()))
    return product.dict()

Log dashboard (last 60 seconds)

20:05:00INFO[api]GET /products/p_42 — cache HIT (rate: 10,400 RPS for last 60s)
20:05:59INFO[redis]EXPIRE fired for key product:p_42
20:06:00INFO[api]GET /products/p_42 — cache MISS (first of 320 concurrent)
20:06:00WARN[db]320 concurrent SELECTs for products WHERE id=p_42 within 87ms window
20:06:00WARN[db]CPU 92%; query_avg 4ms; queue depth 280
20:06:00INFO[api]First request completed in 41ms; wrote cache. Subsequent 319 requests served from DB anyway (already in flight)
20:06:00INFO[api]Subsequent requests now cache HIT (rate: 10,500 RPS resumes)
20:06:01INFO[db]CPU 8%; query_avg 4ms; queue depth 0
20:07:00WARN[db]Same pattern — 318 concurrent queries within 92ms. CPU 90%.
20:08:00WARN[db]Same pattern — 322 concurrent queries within 89ms. CPU 91%.

Pick the root cause

Four candidates. Three are plausible but mistake symptom for cause. Pick what you'd write in the postmortem timeline.