Skip to main content
IncidentPaged 16:42 UTC, Friday

Idempotency key returns wrong customer's response

Service: POST /charge API endpoint

Symptom: Support tickets from customers seeing a different name and charge id on their receipt. ~0.2% of API calls; only on retries.

The code that's running in prod

# payment_api.py
# POST /charge with an Idempotency-Key header. Duplicate keys return
# the cached response so a retry doesn't double-charge the customer.

cache = {}


def charge(customer_id, amount, idempotency_key):
    """Return either a fresh charge response or the cached one."""
    if idempotency_key in cache:
        return cache[idempotency_key]

    # ... process payment via the gateway here ...
    response = {
        "charge_id": f"ch_{customer_id}_{amount}",
        "customer": customer_id,
        "amount": amount,
    }
    cache[idempotency_key] = response
    return response

Log dashboard (last 60 seconds)

16:42:01INFO[api]POST /charge customer=cust_alice idempotency_key=retry-1 amount=$50
16:42:01INFO[api]→ 200 ch_cust_alice_50
16:42:03INFO[api]POST /charge customer=cust_bob idempotency_key=retry-1 amount=$25
16:42:03INFO[cache]HIT for idempotency_key=retry-1 (returning cached response)
16:42:03INFO[api]→ 200 ch_cust_alice_50 (customer=cust_alice, amount=50)
16:42:04WARN[billing]Mismatch: response customer=cust_alice but request authenticated as cust_bob
16:44:18INFO[api]POST /charge customer=cust_chen idempotency_key=retry-1 amount=$200
16:44:18INFO[cache]HIT for idempotency_key=retry-1 (returning cached response)
16:44:18INFO[api]→ 200 ch_cust_alice_50 (customer=cust_alice, amount=50)
16:45:02ERROR[support]User cust_bob ticket: 'My receipt shows someone else's name'

Pick the root cause

Four candidates. Three are plausible but mistake symptom for cause. Pick what you'd write in the postmortem timeline.