Symptom: Roughly once a week, a consumer crashes on `JSONDecodeError`. The supervisor restarts it within seconds; the file then parses fine. No correlation with load, time of day, or specific config contents. Always within milliseconds of a rotation event.
The code that's running in prod
# config_rotator/writer.py
import json
import time
from pathlib import Path
CONFIG_PATH = Path("/etc/svc/config.json")
def rotate(new_config: dict):
# Re-emit the canonical config every 10 minutes (or on signal).
with open(CONFIG_PATH, "w") as f:
json.dump(new_config, f, indent=2)
f.write("\n")
print(f"[{time.strftime('%H:%M:%S')}] rotation done; wrote {CONFIG_PATH.stat().st_size} bytes")
Log dashboard (last 60 seconds)
03:00:00.012
INFO
[writer]
rotation started; writing /etc/svc/config.json
03:00:00.013
ERROR
[consumer]
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
03:00:00.013
ERROR
[consumer]
/etc/svc/config.json is 0 bytes; aborting
03:00:00.041
INFO
[writer]
rotation done; wrote 1,432 bytes
03:00:00.044
WARN
[supervisor]
consumer exited with code 1; restarting in 1s
03:00:01.052
INFO
[consumer]
started; config loaded OK (1,432 bytes)
09:15:00.008
INFO
[writer]
rotation started
09:15:00.022
ERROR
[consumer]
json.decoder.JSONDecodeError: Expecting ',' delimiter: line 12 column 8 (char 318)
09:15:00.041
INFO
[writer]
rotation done; wrote 1,432 bytes
09:15:00.045
WARN
[supervisor]
consumer exited; restarting
Pick the root cause
Four candidates. Three are plausible but mistake symptom for cause. Pick what you'd write in the postmortem timeline.