Postmortem Replay
Real incident shapes, deconstructed. Read the buggy code that ran in prod, scan the log dashboard ops saw at 3am, pick the root cause from four hypotheses. Reveal the senior's analysis and the fix.
The single hardest senior skill these train: separating the symptom the dashboard shows from the root cause that produced it.
Sibling tools for the same senior reflex: ✨ Refactor Coach — paste any Python, get a senior review. 🎯 Refactor Challenge — guess the senior's top finding before reveal. ⚔️ Diff Battle — accept or reject candidate refactors PR-review style.
Stripe webhook returns 200 on DB write failures
Customers report duplicate charges. Ops sees errors but Stripe says everything was acknowledged.
Open the replay →Idempotency key returns wrong customer's response
Two customers using the same idempotency key receive each other's receipts. A two-line cache bug; twelve weeks to find.
Open the replay →asyncio worker hits 'Too many open files' after 45 minutes
Background poller works fine for ~45 minutes then starts failing. Restart fixes temporarily. The leak isn't where you'd think.
Open the replay →Postgres deadlock on every 100th order
Two endpoints touch the same two rows in opposite orders. Works in dev, fires under prod traffic. Postgres aborts one transaction at random.
Open the replay →Connection pool exhausted during external HTTP call
API works at 50 RPS in staging. Tips over at 30 RPS in prod. The pool isn't too small — connections are trapped in the wrong place.
Open the replay →Logged-out at random — one pod's clock is 8 seconds behind
Users get logged out, log back in, work for a minute, get logged out again. Affects only some requests. The load balancer is innocent.
Open the replay →Cache stampede — DB CPU spikes every 60 seconds exactly
Popular product endpoint is fast 99% of the time. Every TTL boundary, the DB takes 320 concurrent queries in 100ms and tail latency jumps 50×.
Open the replay →OOM-killed every 16 hours — telemetry buffer grew to 2 GB
Worker memory grows quietly over a day, then kernel kills it. Restart resets the clock. The leak isn't in the request handler — it's where you'd never look.
Open the replay →Attacker derives the API key from response time
Auth endpoint compares the key with `==`. The first byte mismatch returns faster than the last. Attacker times the difference and walks the key out one byte at a time.
Open the replay →asyncio service freezes for 600ms whenever a 5 MB upload arrives
Most requests are fast (12ms). One large upload arrives and every concurrent request waits 600ms. The event loop is single-threaded — and that's not a bug, it's a constraint.
Open the replay →tenacity retry loop turns a 30-second upstream slowdown into a self-amplifying outage
Stripe goes slow for 30 seconds. The order-processor's @retry decorator stays calm. The DB pool doesn't.
Open the replay →post-deploy thundering herd: 8 pods restart, Postgres replica panics
After every catalog-api deploy, the read-replica spikes to 8s P99 for 30 seconds. The graph shape is identical every time. Once the cache warms, everything's fine.
Open the replay →config-rotator writes a fresh JSON; readers occasionally see 0 bytes
The writer logs success. The reader logs JSONDecodeError on a file that's clearly non-empty seconds later. Sometimes empty, sometimes truncated mid-string, never the previous version.
Open the replay →Got an incident shape you want turned into a replay? Email us — blameless retros only; we sanitise everything before publishing.