tenacity retry loop turns a 30-second upstream slowdown into a self-amplifying outage

The code that's running in prod

# orders/charge.py

from tenacity import retry, wait_fixed
from db import get_connection

@retry(wait=wait_fixed(1))  # retry forever every 1 second
async def charge_card(order_id: int, amount_cents: int):
    async with get_connection() as conn:                       # holds a pool slot
        order = await conn.fetchrow(
            "SELECT * FROM orders WHERE id = $1 FOR UPDATE",
            order_id,
        )
        # Slow path: Stripe API call. Default timeout = 30s.
        charge = await stripe.Charge.create(
            amount=amount_cents,
            currency="usd",
            source=order["card_token"],
        )
        await conn.execute(
            "UPDATE orders SET charge_id = $1 WHERE id = $2",
            charge.id,
            order_id,
        )
        return charge

Log dashboard (last 60 seconds)

Pick the root cause

Four candidates. Three are plausible but mistake symptom for cause. Pick what you'd write in the postmortem timeline.