Skip to main content
IncidentPaged 09:14 UTC, Friday (security team escalation)

Attacker derives the API key from response time

Service: internal-admin-api (Flask)

Symptom: Security team flagged unusual traffic: 270,000 requests over 36 hours from one IP, all returning 401. Response times show a stair-step pattern — gradually longer per attempt.

The code that's running in prod

# auth.py
# Validates the X-API-Key header against the configured secret.

import os

API_KEY = os.environ["API_KEY"]   # 48 random chars, generated at deploy


def check_api_key(provided):
    """Return True if the header matches our secret."""
    if not provided:
        return False
    return provided == API_KEY

Log dashboard (last 60 seconds)

Day 1 03:14:02INFO[api]GET /admin/users → 401 (auth time 0.012 µs)
Day 1 03:14:02INFO[api]GET /admin/users → 401 (auth time 0.014 µs)
Day 1 03:14:02INFO[api]GET /admin/users → 401 (auth time 0.014 µs)
Day 1 03:14:03INFO[api]GET /admin/users → 401 (auth time 0.034 µs ← longer)
Day 1 03:15:00INFO[api]GET /admin/users → 401 (auth time 0.052 µs ← longer)
Day 1 05:30:00INFO[api]GET /admin/users → 401 (auth time 0.180 µs ← longer)
Day 2 14:00:00INFO[api]GET /admin/users → 401 (auth time 0.420 µs ← longer)
Day 2 18:42:00INFO[api]GET /admin/users → 200 (full key derived; this is the breach)
Day 2 18:42:01ERROR[security]ALERT: 270,000 401s preceded a single 200 from same IP. Pattern matches byte-by-byte key derivation.
Day 2 18:50:00INFO[ops]Rate limiter config: 100 req/min per IP. Attacker stayed under by spreading over 36h.

Pick the root cause

Four candidates. Three are plausible but mistake symptom for cause. Pick what you'd write in the postmortem timeline.