IncidentPaged 16:30 UTC, Wednesday
asyncio service freezes for 600ms whenever a 5 MB upload arrives
Service: ingest-api (aiohttp on Python 3.12)
Symptom: P50 latency steady at 12ms; P99 randomly spikes to 600-800ms several times per minute. Investigation shows spikes correlate 1:1 with large multipart uploads.
The code that's running in prod
# ingest_api.py
# Receives JSON-LD documents, strips embedded scripts before parsing.
import json
import re
from aiohttp import web
async def parse_upload(request):
body = await request.read()
text = body.decode("utf-8")
# Strip embedded <script> tags before JSON parsing.
cleaned = re.sub(
r"<script[^>]*>.*?</script>",
"",
text,
flags=re.DOTALL | re.IGNORECASE,
)
parsed = json.loads(cleaned)
return web.json_response({"ok": True, "fields": len(parsed)})
Log dashboard (last 60 seconds)
Pick the root cause
Four candidates. Three are plausible but mistake symptom for cause. Pick what you'd write in the postmortem timeline.