Found because the operator heard the fans. romm 5.3.0 was promoted that morning; the update read `done` and the app ran clean for two hours, then OOM-crash-looped for six - 4530 worker SIGKILLs, ~500% CPU, host load 5.2 while otherwise idle, and nothing alarmed. Raising the limit to 768M was still a guess and fixed nothing (memory.peak hit exactly 768 MiB). Measured instead: ~216 MiB per warm uvicorn worker, so the image's default of 4 workers needs ~882 MiB. /init reads WEB_SERVER_CONCURRENCY; set to 2. Proven under load, not just at idle: 26,645 requests over 300 s, memory 416-614 MiB against 768, trending down, zero SIGKILLs, OOMKilled false. Idle CPU 500% -> 1.64%. The first soak measured nothing - it was pointed at the scratch-guest subdomain, every request 404'd at traefik in 9 ms, and the counter reported 14,026 successes. Positive and negative controls are now asserted before any load is driven. Carried into R-462: `proven` has meant "the update applied and the data survived", not "the new version runs". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
1 parent
186546d562
commit
8efd2d00df
4 files changed
+223
No files matched your search
@@ -0,0 +1,137 @@
|
||||
{
|
||||
"requests": 26645,
|
||||
"codes": {
|
||||
"200": 9687,
|
||||
"401": 16958
|
||||
},
|
||||
"samples": [
|
||||
{
|
||||
"t": 2.4,
|
||||
"stats": [
|
||||
"102.83%|613.7MiB / 768MiB",
|
||||
"false|0",
|
||||
"0"
|
||||
],
|
||||
"requests_so_far": 422
|
||||
},
|
||||
{
|
||||
"t": 25.0,
|
||||
"stats": [
|
||||
"203.29%|531.9MiB / 768MiB",
|
||||
"false|0",
|
||||
"0"
|
||||
],
|
||||
"requests_so_far": 2240
|
||||
},
|
||||
{
|
||||
"t": 47.3,
|
||||
"stats": [
|
||||
"204.17%|422.1MiB / 768MiB",
|
||||
"false|0",
|
||||
"0"
|
||||
],
|
||||
"requests_so_far": 4586
|
||||
},
|
||||
{
|
||||
"t": 70.2,
|
||||
"stats": [
|
||||
"201.60%|517.4MiB / 768MiB",
|
||||
"false|0",
|
||||
"0"
|
||||
],
|
||||
"requests_so_far": 6934
|
||||
},
|
||||
{
|
||||
"t": 93.0,
|
||||
"stats": [
|
||||
"196.72%|477.7MiB / 768MiB",
|
||||
"false|0",
|
||||
"0"
|
||||
],
|
||||
"requests_so_far": 8879
|
||||
},
|
||||
{
|
||||
"t": 117.0,
|
||||
"stats": [
|
||||
"275.33%|574.4MiB / 768MiB",
|
||||
"false|0",
|
||||
"0"
|
||||
],
|
||||
"requests_so_far": 10682
|
||||
},
|
||||
{
|
||||
"t": 139.8,
|
||||
"stats": [
|
||||
"186.14%|594.1MiB / 768MiB",
|
||||
"false|0",
|
||||
"0"
|
||||
],
|
||||
"requests_so_far": 12591
|
||||
},
|
||||
{
|
||||
"t": 162.7,
|
||||
"stats": [
|
||||
"183.15%|567.8MiB / 768MiB",
|
||||
"false|0",
|
||||
"0"
|
||||
],
|
||||
"requests_so_far": 14456
|
||||
},
|
||||
{
|
||||
"t": 185.4,
|
||||
"stats": [
|
||||
"202.06%|525.9MiB / 768MiB",
|
||||
"false|0",
|
||||
"0"
|
||||
],
|
||||
"requests_so_far": 16279
|
||||
},
|
||||
{
|
||||
"t": 208.2,
|
||||
"stats": [
|
||||
"203.08%|442.4MiB / 768MiB",
|
||||
"false|0",
|
||||
"0"
|
||||
],
|
||||
"requests_so_far": 18535
|
||||
},
|
||||
{
|
||||
"t": 231.1,
|
||||
"stats": [
|
||||
"192.08%|546.9MiB / 768MiB",
|
||||
"false|0",
|
||||
"0"
|
||||
],
|
||||
"requests_so_far": 20929
|
||||
},
|
||||
{
|
||||
"t": 254.0,
|
||||
"stats": [
|
||||
"188.73%|504.5MiB / 768MiB",
|
||||
"false|0",
|
||||
"0"
|
||||
],
|
||||
"requests_so_far": 22759
|
||||
},
|
||||
{
|
||||
"t": 276.8,
|
||||
"stats": [
|
||||
"187.29%|484.4MiB / 768MiB",
|
||||
"false|0",
|
||||
"0"
|
||||
],
|
||||
"requests_so_far": 24490
|
||||
},
|
||||
{
|
||||
"t": 299.6,
|
||||
"stats": [
|
||||
"203.00%|416.4MiB / 768MiB",
|
||||
"false|0",
|
||||
"0"
|
||||
],
|
||||
"requests_so_far": 26639
|
||||
}
|
||||
],
|
||||
"duration_s": 300,
|
||||
"concurrency": 6
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
#!/usr/bin/env python3
|
||||
"""romm_soak.py — drive real usage at romm on demo-hp and watch whether the fix holds.
|
||||
|
||||
Every request goes through the HOUSEHOLD'S OWN ROUTE (traefik, `Host: arcade.<domain>`), never at
|
||||
the container directly, so what is measured is what a person's browser would actually cause.
|
||||
|
||||
The number that matters is not the peak alone but **peak against the limit with the OOM counter
|
||||
still at zero**, because this failure announces itself as worker kills, not as a slow page.
|
||||
"""
|
||||
import json, subprocess, sys, threading, time
|
||||
sys.path.insert(0, ".")
|
||||
from demo import Box
|
||||
|
||||
# The subdomain romm was ACTUALLY deployed with on demo-hp. The first run of this script used
|
||||
# "arcade" — the scratch-guest fixture's default — and every request 404'd at traefik while the
|
||||
# counter cheerfully counted 14,026 "requests". An instrument that can drop results silently is
|
||||
# not a measurement (R-96 rule 3), so both controls are now asserted before any load is driven.
|
||||
SUB = "jatek"
|
||||
PATHS = ["/", "/api/heartbeat", "/api/platforms", "/api/roms?limit=50", "/api/collections",
|
||||
"/api/stats", "/api/users/me", "/api/firmware", "/api/saves", "/api/states", "/api/config"]
|
||||
DUR = int(sys.argv[1]) if len(sys.argv) > 1 else 240
|
||||
CONC = int(sys.argv[2]) if len(sys.argv) > 2 else 6
|
||||
|
||||
b = Box("demo-hp"); b.login()
|
||||
stop = threading.Event()
|
||||
hits = {"n": 0, "codes": {}}
|
||||
lock = threading.Lock()
|
||||
t0 = time.time()
|
||||
|
||||
|
||||
def worker(i):
|
||||
k = 0
|
||||
while not stop.is_set():
|
||||
p = PATHS[(i + k) % len(PATHS)]; k += 1
|
||||
r = subprocess.run(["curl", "-sk", "--max-time", "20", "-o", "/dev/null",
|
||||
"-w", "%{http_code}", "-H", f"Host: {SUB}.enkisfelhom.hu",
|
||||
b.base + p], capture_output=True, text=True)
|
||||
with lock:
|
||||
hits["n"] += 1
|
||||
c = (r.stdout or "?").strip()
|
||||
hits["codes"][c] = hits["codes"].get(c, 0) + 1
|
||||
|
||||
|
||||
samples = []
|
||||
|
||||
|
||||
def sampler():
|
||||
while not stop.is_set():
|
||||
s = b.guest("docker stats --no-stream --format '{{.CPUPerc}}|{{.MemUsage}}' romm; "
|
||||
"docker inspect romm --format '{{.State.OOMKilled}}|{{.RestartCount}}'; "
|
||||
"docker logs --since 2026-09-22T15:55:34 romm 2>&1 | grep -c SIGKILL")
|
||||
parts = [x.strip() for x in s.strip().split("\n") if x.strip()]
|
||||
with lock:
|
||||
n = hits["n"]
|
||||
rec = {"t": round(time.time() - t0, 1), "stats": parts, "requests_so_far": n}
|
||||
samples.append(rec)
|
||||
print(f" +{rec['t']:5.0f}s {' '.join(parts)} reqs={n}", flush=True)
|
||||
time.sleep(20)
|
||||
|
||||
|
||||
print(f"driving {CONC} concurrent callers at romm for {DUR}s through the household's own route")
|
||||
ts = [threading.Thread(target=worker, args=(i,), daemon=True) for i in range(CONC)]
|
||||
ts.append(threading.Thread(target=sampler, daemon=True))
|
||||
for t in ts:
|
||||
t.start()
|
||||
time.sleep(DUR)
|
||||
stop.set(); time.sleep(3)
|
||||
print(f"\nrequests: {hits['n']} codes: {json.dumps(hits['codes'])}")
|
||||
json.dump({"requests": hits["n"], "codes": hits["codes"], "samples": samples,
|
||||
"duration_s": DUR, "concurrency": CONC}, open("romm-soak.json", "w"), indent=2)
|
||||
print("written romm-soak.json")
|
||||
@@ -804,6 +804,7 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
|
||||
| **R-632** | **[P3-LOW] Twenty-eight of the 53 templates have never been deployed by any update drill, so nothing is known about whether their updates work.** COUNTED 2026-09-22 against the 2026-09-21 sweep, which is the widest one ever run. **20 apps have a verdict record** (14 proven, 3 failed, 3 inconclusive, plus tandoor re-walked to proven on 2026-09-22); **4 more were deployed as props in the bad-days legs with no edge walked** (`bentopdf`, `glance`, `uptime-kuma`, `wishlist`); **1 was deployed only to measure its probe** (`wger`); and **28 have never been deployed at all**: `calcom`, `calibre-web`, `claper`, `code-server`, `crafty-controller`, `emby`, `ghost`, `gokapi`, `gramps-web`, `homebox`, `homepage`, `immich`, `jellyfin`, `kimai`, `komga`, `onlyoffice`, `outline`, `paperless-ngx`, `plant-it`, `plex`, `radarr`, `rallly`, `recipe-importer`, `seerr`, `sonarr`, `sparkyfitness`, `termix`, `wanderer`. **THIS IS NOT A COMPLAINT ABOUT THE SWEEP** — it took one app-catalog-wide night to go from 3 apps ever measured to 21, and a night is the unit available. It is a record of what the catalog's update promise currently rests on: **for 28 of 53 apps, nothing.** **The list is the nightly rotation's queue**, smallest and least stateful first; `paperless-ngx` should be early because R-630 needs a live reading from it anyway, and `crafty-controller`, `mealie` and `uptime-kuma` should be early because R-631 needs one from each. Machine-readable copy: `audits/probe-fix-2026-09-22/not-judged.json`. **WORKED IN ONE NIGHT, 2026-09-22 — all 28 walked, so this row CLOSES and hands its findings to others.** Every one was installed on guest 9202 against the private drill catalog and taken through the same walk: deploy at the live pin, seed through the app's own front door, read it back, „Mentés most”, the guarded Update where a real within-a-major edge exists upstream, **restore from that copy and read the seed back a second time** (the half the update night skipped), then remove and a 60-second check that nothing came back. **26 of 28 deployed; 6 proven; the rest inconclusive, no-edge or refused.** **What the night produced that this row could not have predicted:** R-630 raised to P1 by measurement (a stack with no probe container has its working app STOPPED by a successful update), R-633 (a remove during a restore leaves an orphan with a live public route), R-634 (an app running and healthy while recorded as not deployed, and then unremovable), and one real upstream edge that HELD honestly (`outline 1.9.1 → 1.10.1`). **Also settled:** `plant-it` is `lifecycle: abandoned` and the product refuses to install it — the only lifecycle-gated template in the catalog, and its gate is now proven live. Full record: `audits/DRILL-the-28-2026-09-22.md`. | **CLOSED 2026-09-22 — all 28 walked in one night; the findings live in R-630, R-633, R-634** |
|
||||
| **R-633** | **[P2-MEDIUM] A remove sent while a restore is still running reports success, deletes the app's record, and leaves a container restarting forever with a live public route.** MEASURED 2026-09-22 on guest 9202, controller v0.261.0, during the twenty-eight walk. `gokapi` was restored from its own local copy at 11:34:07 and removed at 11:34:22. `POST /backup/restore` answers **302 and works in the background**; the remove tore down what existed and the restore's own `compose up` then RE-CREATED the container at **11:34:24**. **Both calls returned success.** Twenty-five minutes later: `GET /api/stacks/gokapi` reads **`deployed: false`**, and `docker ps -a` shows `gokapi` **`Restarting (1)`** with `RestartCount` climbing, carrying its full traefik label set — including `traefik.http.routers.gokapi.rule: Host(`.enkisfelhom.hu`)`, **a rule with an empty subdomain**, because the deploy values that filled it were deleted with the app. Its own log loops *„Salt for admin password invalid, generating new salt… password does not appear to be a SHA-1 hash"* — the volume holding its config was removed correctly, so the binary can never start. **WHY THIS IS A ROW AND NOT A HARNESS ARTEFACT:** a household can press exactly these two buttons in exactly this order, the product accepted both, and **the remove reported success while leaving the orphan**. Presence of a success message is not evidence of a result. **WHAT IT COSTS:** an app the household believes is gone keeps a container in a restart loop, keeps a route registered on the public reverse proxy, and is invisible to every product surface because the controller no longer records the stack. Nothing in the alarm ladder fires: `08` §4 keys on stacks the controller KNOWS about. **This is R-626's class with the mechanism finally visible** — that row recorded a removed `navidrome` coming back and could not diagnose it because the controller had restarted; here the window is 17 seconds and both halves are in the evidence. **Needs:** the remove path to refuse, or to wait, while a restore for the same stack is in flight — and, either way, to verify the teardown rather than report success without looking. Evidence: `audits/the-28-2026-09-22/apps/gokapi/came-back-evidence.txt`, `apps/gokapi/log.txt`. | **OPEN — P2; owner: CC; product code, so not fixed in this unattended run** |
|
||||
| **R-634** | **[P1-HIGH] An app can be RUNNING, HEALTHY and serving while the controller records it as not deployed — and in that state the household cannot remove it through the product at all.** MEASURED 2026-09-22 on guest 9202, controller v0.261.0, on **two independent apps in one night**: `outline` and `sparkyfitness`. **The controller's own words, in order.** `outline` deploy accepted 11:54:47; **`11:55:51 StopStack outline: current state=deploying deployed=true containers=0`** — a stop while the stack is still deploying; `11:56:09 SaveAppConfig: saving /opt/docker/stacks/outline — 5 env vars, **0 encrypted**, 3 sensitive fields` (the two saves before it both read `3 encrypted`); then **`11:57:16` and `12:02:16 Health probe outline: API GET :3000/_health -> 200`** — the app is up and answering its own health endpoint; and **`12:02:19 StopStack outline: current state=running deployed=false containers=3`**. Three containers, state `running`, health 200, **`deployed=false`**. The remove then answers **`RemoveStack outline: state=not_deployed, deployed=false, orphaned=false, deploying=false` -> `[ERROR] Remove failed for outline: stack "outline" is not deployed`** for BOTH the remove-with-data and the remove-keeping-data call, while `ScanStacks` goes on finding the stack every ten seconds. `app.yaml` survives with `desired_state: stopped` and an **empty `installed_images`**. `sparkyfitness` produced the identical shape 13 minutes earlier. **WHAT THIS COSTS A HOUSEHOLD:** an app that works is invisible to the product as an installation — no badge, no update, no backup selection, and **no way to delete it**; the only exit is a shell. It is the mirror of R-633 (there the record is gone and the container remains; here the container is fine and the record is gone) and it is the **worse** of the two, because the app is serving customer traffic the whole time. **THE MECHANISM IS NOT DIAGNOSED, and this row says so rather than guessing.** What was tried: the controller's full container log for both apps (the sequence above), `app.yaml` on disk, `docker ps -a`, and `GET /api/stacks/<n>`. What was NOT done: reading `runComposeDeploy`'s pin-write path — this was an unattended run and the brief forbade product code. **The one discriminator worth running first:** both apps were walked while two other walks ran concurrently, and `POST /api/backup/run` is box-wide, so a backup or restore for a NEIGHBOURING app was in flight. A serial re-walk is queued tonight; if it reproduces alone, concurrency is not the cause. **A THIRD THING THE SAME LOG SHOWS, recorded here because it is one line away:** at `11:55:57` the health probe dialled **`http://outline-postgres:3000/_health`** — during startup, with the exactly-named container not yet running, `findProbeContainer`'s PREFIX fallback latched onto the POSTGRES sidecar and probed port 3000 on it. Transient, and it resolved once `outline` came up, but it is the same function R-630 is about. Evidence: `audits/the-28-2026-09-22/apps/half-state-outline-sparkyfitness.txt`. **THE SERIAL RE-WALK WAS RUN THE SAME NIGHT AND IT SPLITS THIS ROW IN TWO — recorded here rather than left as the first reading.** Walked again one at a time, with no other walk running: **`outline` deployed normally and removed clean**, and **`crafty-controller` deployed normally, updated `4.10.7 → 4.11.0` to `done`, restored and removed clean.** So for those two the half-state did NOT reproduce alone, and concurrency — a box-wide `POST /api/backup/run` or a restore in flight for a NEIGHBOURING app — is implicated rather than the deploy path itself. **`sparkyfitness` reproduced EXACTLY, alone, in 534 s**: deploy accepted, never reached `deployed` with a pin, `app.yaml` left with `desired_state: stopped` and an empty `installed_images`, and **both remove calls refused with `stack "sparkyfitness" is not deployed`.** **So the row stands, at one reproducible app instead of three, and the honest split is:** (a) `sparkyfitness` has a deploy that does not finish and leaves a record the product cannot clear — reproducible, P1; (b) under concurrent work the same unremovable half-state can be reached by apps that are otherwise fine, which is the more alarming half because those apps were **running, healthy and serving** while recorded as not deployed. **Neither half is diagnosed** — `runComposeDeploy`'s pin write was not read, because the brief forbade product code. | **OPEN — P1; owner: CC; reproducible alone on `sparkyfitness`, concurrency-linked on the other two; next step is `runComposeDeploy`'s pin write** |
|
||||
| **R-635** | **[P1-HIGH] `romm 5.3.0` does not fit the memory the template gives it, and the guarded Update called that a success — the app has been OOM-crash-looping on demo-hp for six hours at ~500% CPU.** FOUND 2026-09-22 17:37 **because the operator heard the fans**, which is the only reason it was found at all. `romm` was promoted `5.0.0 -> 5.3.0` on the live catalog that morning (`audits/PROBE-FIX-2026-09-22.md`) after the edge was PROVEN on scratch guest 9202, and the guarded Update was then pressed on demo-hp guest 9201 at **09:08:22Z**, reaching **`done` in 74.8 s** with the app `running`. **It ran clean for two hours.** The first worker kill is at **11:09:20Z**; by 15:38Z there had been **4,530** of them — `Worker (pid:…) was sent SIGKILL! Perhaps out of memory?` — with `docker inspect` reading **`OOMKilled: true`**, the container pinned at **457 MiB of its 512 MiB limit**, and `docker stats` showing **499.51% CPU**. The host's load average sat at **5.2 while otherwise idle**. `rq_cron` is killed and restarted every few seconds in a permanent storm. **THREE THINGS THIS ESTABLISHES, and the third is the one that changes how promotions are judged.** (1) The template's own comment says *`RAM: ~300MB (mem_limit: 1024M total — romm 512M + mariadb 384M + redis 128M)`*; **5.3.0 needs more than 512M and the template was not re-sized when the version moved.** A version move is not only an `image:` line. (2) **The update reported `done` and the app reads `running`**, because nginx answers `GET /` with 200 while the gunicorn workers behind it are being killed — a THIRD variant of the R-618/R-630 theme: the probe is right, the port is right, and the answer is still a false green. (3) **NOTHING ALARMED.** Docker OOM signals are silent in LXC guests (the `lxc-docker-oom-signals-unreliable` finding, R-528), `08` §4 does not treat a running-but-thrashing app as down, and the box is `hub.enabled: true` — so six hours of a machine at 500% CPU produced no event and no mail. **The detector that worked was a person in the room.** **AND THE LESSON FOR R-462's METHOD, which is the real cost:** every `proven` verdict in the update night and in the twenty-eight measures the app for the **minutes of the walk**, not for a day of running. `romm` passed its walk, was seeded, read back and restored — and broke two hours later. **`proven` currently means "the update applied and the data survived", NOT "the new version runs".** Needs: decide between rolling the catalog back to 5.0.0 and raising romm's `mem_limit` (measured, not guessed); and a soak longer than a walk before any future promotion. **FIXED AND MEASURED 2026-09-22, in two steps, and the FIRST step was still a guess.** *Step 1 (operator's choice):* the limit was raised 512M → **768M** (`app-catalog-felhom.eu@886956d`). It slowed the kills from ~12/min to ~7/min and **stopped nothing** — 37 SIGKILLs in five minutes, `OOMKilled` still true, and the cgroup's own `memory.peak` read **exactly 768 MiB**: it hit the new ceiling and died there. *Step 2, from a MEASUREMENT instead:* the per-process RSS inside the container reads **~216 MiB per warm uvicorn worker**, so the image's default of four workers plus the master needs **~882 MiB** before nginx and the job runner — more than any sensible limit for this box. **The lever was in the image all along:** `/init:143` runs `--workers "${WEB_SERVER_CONCURRENCY:-4}"`. **Four workers is a SERVER default on an appliance serving one household.** Setting **`WEB_SERVER_CONCURRENCY=2`** (`app-catalog-felhom.eu@f4eb94f`, limit left at 768M) and applying it through the product's own Update button fixed it. **PROVEN UNDER LOAD, not just at idle:** 6 concurrent callers driven at romm through the household's own route for 300 s — **26,645 requests** (9,687 × 200, 16,958 × 401 on the auth-gated endpoints), CPU a steady **~200%** (exactly two workers saturated, by design), memory oscillating **416–614 MiB against the 768 MiB limit and trending DOWN**, and **zero** SIGKILLs, `OOMKilled: false`, `RestartCount: 0` throughout. At idle afterwards: **1.64% CPU**, 610 MiB, host load falling from 5.2 to 2.4. **AND THE FIRST SOAK MEASURED NOTHING, which is worth more than the second one:** it was pointed at `arcade.enkisfelhom.hu` — the scratch-guest fixture's default subdomain — while this box deployed romm at `jatek`. Every request 404'd at traefik in 9 ms, romm sat idle at 0.64% CPU, and the counter cheerfully reported **14,026 successful requests**. It was caught only because 0.64% CPU under load is not believable. **A positive AND a negative control are now asserted before any load is driven** (`jatek` must not 404; a nonsense host must). R-96 rule 3, in a new surface. **WHAT STAYS OPEN, and it is the part that outlives romm:** 610 MiB of 768 MiB is **79%** — it works with ~158 MiB of headroom and the soak never exceeded 614 MiB, but it is not generous, and nothing watches it. **And the method lesson for R-4Line truncated
|
||||
<!-- DUE-CHECKS-BEGIN — machine-readable. Parsed by scripts/due_checks_gate.py.
|
||||
One row per dated check. The R-number must have a row above. Dates are UTC.
|
||||
Clearing a row means the check was DONE and its result recorded in that R-row —
|
||||
|
||||
Reference in new issue
Block a user