Skip to main content
IncidentPaged 12:00 UTC (deploy window), Monday

post-deploy thundering herd: 8 pods restart, Postgres replica panics

Service: catalog-api (FastAPI + Redis cache, Postgres read replica behind)

Symptom: Every rolling deploy produces a 30-second window of 5xx responses and replica connection-pool saturation. Steady state is fine — the burn is exactly around the deploy. Ops bookmarked the graph.

The code that's running in prod

# catalog/products.py

from fastapi import FastAPI
import redis.asyncio as redis
import asyncpg
import json

app = FastAPI()
cache = redis.Redis(host="cache", decode_responses=True)
pool: asyncpg.Pool  # initialised on startup

@app.get("/products/{product_id}")
async def get_product(product_id: int):
    cached = await cache.get(f"product:{product_id}")
    if cached is not None:
        return json.loads(cached)

    # Cache miss — read through to Postgres.
    async with pool.acquire() as conn:
        row = await conn.fetchrow(
            "SELECT id, name, price_cents, stock FROM products WHERE id = $1",
            product_id,
        )

    payload = dict(row)
    await cache.set(f"product:{product_id}", json.dumps(payload), ex=300)
    return payload

Log dashboard (last 60 seconds)

12:00:05INFO[deploy]catalog-api v2026-06-22-r3 → all 8 pods Ready
12:00:06INFO[redis]new client connections from 8 hosts; cold start
12:00:06INFO[api]GET /products/12 — cache MISS → reading from DB
12:00:06INFO[api]GET /products/12 — cache MISS → reading from DB (4 concurrent for SAME id)
12:00:06INFO[api]GET /products/12 — cache MISS → reading from DB (7 concurrent for SAME id)
12:00:07WARN[pgrep]replication lag spiked to 4.2s
12:00:07ERROR[pgrep]max_connections=200 reached; new connections refused
12:00:08WARN[api]~8,000 concurrent /products requests, ~97% cache miss
12:00:10ERROR[api]P99 latency 8,200ms; error_rate 6.4% (asyncpg.TooManyConnectionsError)
12:00:25INFO[redis]cache fill rate decelerating; hit rate climbing through 70%
12:00:35INFO[api]P99 back to 28ms; cache hit rate 91%; replica recovered

Pick the root cause

Four candidates. Three are plausible but mistake symptom for cause. Pick what you'd write in the postmortem timeline.