IncidentPaged 13:47 UTC, Wednesday
Connection pool exhausted during external HTTP call
Service: checkout-api (Flask + SQLAlchemy + external billing API)
Symptom: Under sustained 30+ RPS, p99 latency climbs from 200ms to 25s, then requests start returning 503 with `pool timeout`. DB itself shows <10% utilisation.
The code that's running in prod
# checkout_api.py
# POST /checkout opens a DB session, loads the cart, calls the
# external billing service, then commits the result.
import requests
from sqlalchemy.orm import sessionmaker
SessionLocal = sessionmaker(bind=engine, autocommit=False)
def checkout(cart_id):
session = SessionLocal()
try:
cart = session.query(Cart).get(cart_id)
if cart is None or cart.empty():
return {"error": "empty cart"}, 400
# External call to billing service. Typical: 2-5 seconds.
resp = requests.post(
"https://billing.example.com/charge",
json={"cart_id": cart_id, "amount": cart.total},
timeout=10,
)
cart.charge_id = resp.json()["charge_id"]
cart.status = "paid"
session.commit()
return {"status": "ok", "charge_id": cart.charge_id}
finally:
session.close()
Log dashboard (last 60 seconds)
Pick the root cause
Four candidates. Three are plausible but mistake symptom for cause. Pick what you'd write in the postmortem timeline.