Skip to main content
IncidentPaged 13:47 UTC, Wednesday

Connection pool exhausted during external HTTP call

Service: checkout-api (Flask + SQLAlchemy + external billing API)

Symptom: Under sustained 30+ RPS, p99 latency climbs from 200ms to 25s, then requests start returning 503 with `pool timeout`. DB itself shows <10% utilisation.

The code that's running in prod

# checkout_api.py
# POST /checkout opens a DB session, loads the cart, calls the
# external billing service, then commits the result.

import requests
from sqlalchemy.orm import sessionmaker

SessionLocal = sessionmaker(bind=engine, autocommit=False)


def checkout(cart_id):
    session = SessionLocal()
    try:
        cart = session.query(Cart).get(cart_id)
        if cart is None or cart.empty():
            return {"error": "empty cart"}, 400

        # External call to billing service. Typical: 2-5 seconds.
        resp = requests.post(
            "https://billing.example.com/charge",
            json={"cart_id": cart_id, "amount": cart.total},
            timeout=10,
        )

        cart.charge_id = resp.json()["charge_id"]
        cart.status = "paid"
        session.commit()
        return {"status": "ok", "charge_id": cart.charge_id}
    finally:
        session.close()

Log dashboard (last 60 seconds)

13:46:50INFO[api]POST /checkout cart=c_8821 → 200 (1.8s)
13:46:52INFO[api]POST /checkout cart=c_8822 → 200 (2.4s)
13:47:01WARN[pool]Pool 48/50 in use; queue depth 0
13:47:14WARN[pool]Pool 50/50 in use; queue depth 12
13:47:22WARN[pool]Pool 50/50 in use; queue depth 41 (wait time avg 11s)
13:47:30ERROR[api]POST /checkout cart=c_8901 → 503 (sqlalchemy.exc.TimeoutError: QueuePool limit reached)
13:47:33INFO[db]Active connections: 50/50. CPU 8%. Avg query time: 4ms.
13:47:33WARN[billing]External /charge p99 latency: 3.2s (normal: 2.1s)
13:47:42ERROR[api]POST /checkout cart=c_8907 → 503 (pool timeout after 30s)
13:48:01INFO[ops]Throughput dropped from 32 RPS to 4 RPS in 90s

Pick the root cause

Four candidates. Three are plausible but mistake symptom for cause. Pick what you'd write in the postmortem timeline.