Skip to main content
IncidentPaged 02:18 UTC, Saturday (3rd time this week)

asyncio worker hits 'Too many open files' after 45 minutes

Service: background scraper-worker (asyncio)

Symptom: After 40-50 minutes of uptime, every request fails with `OSError: [Errno 24] Too many open files`. SIGTERM + restart clears it for another 45 minutes.

The code that's running in prod

# scraper_worker.py
# Polls a list of upstream feeds every 60 seconds, parses each
# response, persists deltas. Long-running asyncio worker.

import aiohttp


async def fetch_one(url):
    """Return the response body, or None on any error."""
    session = aiohttp.ClientSession()
    try:
        async with session.get(url) as resp:
            return await resp.text()
    except Exception as e:
        print(f"[ERROR] {url}: {e}")
        return None


async def poll_loop(feeds):
    while True:
        for url in feeds:
            body = await fetch_one(url)
            if body:
                process(body)
        await asyncio.sleep(60)

Log dashboard (last 60 seconds)

01:35:02INFO[worker]Polling 12 feeds (cycle 35)
01:35:08INFO[worker]Cycle 35 complete in 6.1s
01:36:02INFO[worker]Polling 12 feeds (cycle 36)
01:36:09INFO[worker]Cycle 36 complete in 6.4s
02:17:42WARN[system]Process FD count: 1019 / 1024 (ulimit -n)
02:18:02INFO[worker]Polling 12 feeds (cycle 78)
02:18:02ERROR[worker]https://api.x.com/feed1: [Errno 24] Too many open files
02:18:02ERROR[worker]https://api.x.com/feed2: [Errno 24] Too many open files
02:18:02ERROR[worker]https://api.x.com/feed3: [Errno 24] Too many open files
02:18:05WARN[system]All 12 feeds errored this cycle. Process FD count: 1024 / 1024

Pick the root cause

Four candidates. Three are plausible but mistake symptom for cause. Pick what you'd write in the postmortem timeline.