Skip to main content
IncidentPaged 10:21 UTC, Thursday

Logged-out at random — one pod's clock is 8 seconds behind

Service: auth-middleware (JWT validation, behind load balancer with 6 pods)

Symptom: ~1 in 6 authenticated requests returns 401 with 'token issued in the future'. Re-logging in works once. Users report the pattern feels random.

The code that's running in prod

# auth_middleware.py
# Validates JWTs at every authenticated endpoint. Rejects tokens
# whose 'iat' (issued at) timestamp is in the future to defeat
# replay-via-future-stamped attacks.

import time
import jwt


def validate_token(token, secret):
    payload = jwt.decode(token, secret, algorithms=["HS256"])
    now = time.time()

    if payload["iat"] > now:
        raise jwt.InvalidIssuedAtError("token issued in the future")

    if payload["exp"] < now:
        raise jwt.ExpiredSignatureError("token expired")

    return payload

Log dashboard (last 60 seconds)

10:21:14INFO[auth-svc]Issued token for user u_88 with iat=1719224474 (10:21:14 UTC)
10:21:14INFO[pod-A]GET /me u_88 → 200 (validate ok, now=10:21:14)
10:21:14INFO[pod-A]GET /me u_88 → 200 (validate ok, now=10:21:14)
10:21:14ERROR[pod-D]GET /me u_88 → 401 InvalidIssuedAtError: iat=1719224474 > now=1719224466 (10:21:06 UTC)
10:21:14INFO[pod-A]GET /me u_88 → 200 (validate ok, now=10:21:14)
10:21:14ERROR[pod-D]GET /me u_88 → 401 InvalidIssuedAtError: iat=1719224474 > now=1719224466 (10:21:06 UTC)
10:21:18WARN[ops]Pod clock drift report: pod-D is 8.2s behind cluster median (NTP last sync: 19h ago)
10:21:18INFO[pod-D]Local clock: 10:21:10. Cluster median: 10:21:18.
10:21:35INFO[support]User u_88 ticket: 'I keep getting logged out at random'
10:21:50WARN[ops]401-rate on pod-D = 16.4%. Other pods: 0.1%. Routing to pod-D is uniform.

Pick the root cause

Four candidates. Three are plausible but mistake symptom for cause. Pick what you'd write in the postmortem timeline.