admin
8efd2d00df
gates / gates (push) Successful in 26s
Found because the operator heard the fans. romm 5.3.0 was promoted that morning; the update read `done` and the app ran clean for two hours, then OOM-crash-looped for six - 4530 worker SIGKILLs, ~500% CPU, host load 5.2 while otherwise idle, and nothing alarmed. Raising the limit to 768M was still a guess and fixed nothing (memory.peak hit exactly 768 MiB). Measured instead: ~216 MiB per warm uvicorn worker, so the image's default of 4 workers needs ~882 MiB. /init reads WEB_SERVER_CONCURRENCY; set to 2. Proven under load, not just at idle: 26,645 requests over 300 s, memory 416-614 MiB against 768, trending down, zero SIGKILLs, OOMKilled false. Idle CPU 500% -> 1.64%. The first soak measured nothing - it was pointed at the scratch-guest subdomain, every request 404'd at traefik in 9 ms, and the counter reported 14,026 successes. Positive and negative controls are now asserted before any load is driven. Carried into R-462: `proven` has meant "the update applied and the data survived", not "the new version runs". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
72 lines
3.1 KiB
Python
72 lines
3.1 KiB
Python
#!/usr/bin/env python3
|
|
"""romm_soak.py — drive real usage at romm on demo-hp and watch whether the fix holds.
|
|
|
|
Every request goes through the HOUSEHOLD'S OWN ROUTE (traefik, `Host: arcade.<domain>`), never at
|
|
the container directly, so what is measured is what a person's browser would actually cause.
|
|
|
|
The number that matters is not the peak alone but **peak against the limit with the OOM counter
|
|
still at zero**, because this failure announces itself as worker kills, not as a slow page.
|
|
"""
|
|
import json, subprocess, sys, threading, time
|
|
sys.path.insert(0, ".")
|
|
from demo import Box
|
|
|
|
# The subdomain romm was ACTUALLY deployed with on demo-hp. The first run of this script used
|
|
# "arcade" — the scratch-guest fixture's default — and every request 404'd at traefik while the
|
|
# counter cheerfully counted 14,026 "requests". An instrument that can drop results silently is
|
|
# not a measurement (R-96 rule 3), so both controls are now asserted before any load is driven.
|
|
SUB = "jatek"
|
|
PATHS = ["/", "/api/heartbeat", "/api/platforms", "/api/roms?limit=50", "/api/collections",
|
|
"/api/stats", "/api/users/me", "/api/firmware", "/api/saves", "/api/states", "/api/config"]
|
|
DUR = int(sys.argv[1]) if len(sys.argv) > 1 else 240
|
|
CONC = int(sys.argv[2]) if len(sys.argv) > 2 else 6
|
|
|
|
b = Box("demo-hp"); b.login()
|
|
stop = threading.Event()
|
|
hits = {"n": 0, "codes": {}}
|
|
lock = threading.Lock()
|
|
t0 = time.time()
|
|
|
|
|
|
def worker(i):
|
|
k = 0
|
|
while not stop.is_set():
|
|
p = PATHS[(i + k) % len(PATHS)]; k += 1
|
|
r = subprocess.run(["curl", "-sk", "--max-time", "20", "-o", "/dev/null",
|
|
"-w", "%{http_code}", "-H", f"Host: {SUB}.enkisfelhom.hu",
|
|
b.base + p], capture_output=True, text=True)
|
|
with lock:
|
|
hits["n"] += 1
|
|
c = (r.stdout or "?").strip()
|
|
hits["codes"][c] = hits["codes"].get(c, 0) + 1
|
|
|
|
|
|
samples = []
|
|
|
|
|
|
def sampler():
|
|
while not stop.is_set():
|
|
s = b.guest("docker stats --no-stream --format '{{.CPUPerc}}|{{.MemUsage}}' romm; "
|
|
"docker inspect romm --format '{{.State.OOMKilled}}|{{.RestartCount}}'; "
|
|
"docker logs --since 2026-09-22T15:55:34 romm 2>&1 | grep -c SIGKILL")
|
|
parts = [x.strip() for x in s.strip().split("\n") if x.strip()]
|
|
with lock:
|
|
n = hits["n"]
|
|
rec = {"t": round(time.time() - t0, 1), "stats": parts, "requests_so_far": n}
|
|
samples.append(rec)
|
|
print(f" +{rec['t']:5.0f}s {' '.join(parts)} reqs={n}", flush=True)
|
|
time.sleep(20)
|
|
|
|
|
|
print(f"driving {CONC} concurrent callers at romm for {DUR}s through the household's own route")
|
|
ts = [threading.Thread(target=worker, args=(i,), daemon=True) for i in range(CONC)]
|
|
ts.append(threading.Thread(target=sampler, daemon=True))
|
|
for t in ts:
|
|
t.start()
|
|
time.sleep(DUR)
|
|
stop.set(); time.sleep(3)
|
|
print(f"\nrequests: {hits['n']} codes: {json.dumps(hits['codes'])}")
|
|
json.dump({"requests": hits["n"], "codes": hits["codes"], "samples": samples,
|
|
"duration_s": DUR, "concurrency": CONC}, open("romm-soak.json", "w"), indent=2)
|
|
print("written romm-soak.json")
|