Skip to main content
IncidentPaged 16:30 UTC, Wednesday

asyncio service freezes for 600ms whenever a 5 MB upload arrives

Service: ingest-api (aiohttp on Python 3.12)

Symptom: P50 latency steady at 12ms; P99 randomly spikes to 600-800ms several times per minute. Investigation shows spikes correlate 1:1 with large multipart uploads.

The code that's running in prod

# ingest_api.py
# Receives JSON-LD documents, strips embedded scripts before parsing.

import json
import re
from aiohttp import web


async def parse_upload(request):
    body = await request.read()
    text = body.decode("utf-8")

    # Strip embedded <script> tags before JSON parsing.
    cleaned = re.sub(
        r"<script[^>]*>.*?</script>",
        "",
        text,
        flags=re.DOTALL | re.IGNORECASE,
    )
    parsed = json.loads(cleaned)

    return web.json_response({"ok": True, "fields": len(parsed)})

Log dashboard (last 60 seconds)

16:30:00INFO[api]POST /ingest (4.8 KB) → 200 in 11ms
16:30:00INFO[api]POST /ingest (5.1 KB) → 200 in 12ms
16:30:02INFO[api]POST /ingest (4.8 MB) — starting parse...
16:30:02WARN[loop]Event loop blocked for 612ms in parse_upload
16:30:02INFO[api]POST /ingest (4.8 MB) → 200 in 612ms (large)
16:30:02WARN[api]POST /ingest (4.3 KB) → 200 in 624ms (queued behind large upload)
16:30:02WARN[api]POST /ingest (3.7 KB) → 200 in 620ms (queued)
16:30:02WARN[api]POST /ingest (5.2 KB) → 200 in 615ms (queued)
16:30:02INFO[ops]Healthcheck timed out (5s limit); recovered next cycle. False alarm? Or symptom?
16:30:03INFO[api]P50 latency back to 12ms; queue drained

Pick the root cause

Four candidates. Three are plausible but mistake symptom for cause. Pick what you'd write in the postmortem timeline.