Skip to main content
IncidentPaged 09:14 UTC, Tuesday

Stripe webhook returns 200 on DB write failures

Service: payment-events webhook handler

Symptom: ~3% of charge.succeeded events recorded in our DB don't match Stripe's records.

The code that's running in prod

# webhook_handler.py
# Receives charge.succeeded events from Stripe.

processed = set()

def handle_webhook(event, db):
    """Returns (status_code, body). Stripe retries on non-2xx."""
    if event["id"] in processed:
        return (200, "already processed")

    # Mark this event as processed so duplicate webhooks no-op.
    processed.add(event["id"])

    try:
        db.write(event["id"], event["data"])
        return (200, "OK")
    except IOError as e:
        print(f"[ERROR] write failed for {event['id']}: {e}")
        return (200, "OK")    # avoid retry storm

Log dashboard (last 60 seconds)

09:14:23INFO[webhook]Received evt_DR7a (charge.succeeded, $42.00)
09:14:23ERROR[db]Write failed for evt_DR7a: connection timeout (retries=3 exhausted)
09:14:23INFO[webhook]Returned HTTP 200 to Stripe for evt_DR7a
09:14:25INFO[webhook]Received evt_XK9b (charge.succeeded, $89.00)
09:14:25INFO[db]Wrote evt_XK9b OK
09:14:25INFO[webhook]Returned HTTP 200 to Stripe for evt_XK9b
09:14:41INFO[webhook]Received evt_TM2c (charge.succeeded, $15.00)
09:14:41ERROR[db]Write failed for evt_TM2c: connection timeout (retries=3 exhausted)
09:14:41INFO[webhook]Returned HTTP 200 to Stripe for evt_TM2c
09:14:55WARN[stripe]Dashboard reports 10 events delivered in last 60s; our DB has 7

Pick the root cause

Four candidates. Three are plausible but mistake symptom for cause. Pick what you'd write in the postmortem timeline.