IncidentPaged 09:14 UTC, Tuesday
Stripe webhook returns 200 on DB write failures
Service: payment-events webhook handler
Symptom: ~3% of charge.succeeded events recorded in our DB don't match Stripe's records.
The code that's running in prod
# webhook_handler.py
# Receives charge.succeeded events from Stripe.
processed = set()
def handle_webhook(event, db):
"""Returns (status_code, body). Stripe retries on non-2xx."""
if event["id"] in processed:
return (200, "already processed")
# Mark this event as processed so duplicate webhooks no-op.
processed.add(event["id"])
try:
db.write(event["id"], event["data"])
return (200, "OK")
except IOError as e:
print(f"[ERROR] write failed for {event['id']}: {e}")
return (200, "OK") # avoid retry storm
Log dashboard (last 60 seconds)
Pick the root cause
Four candidates. Three are plausible but mistake symptom for cause. Pick what you'd write in the postmortem timeline.