upgrade-test: watch memory after the readback (harness v2, R-635/R-462)
gates / gates (push) Successful in 2s

After an edge reads back, --soak seconds (default 600) of light load while
the kernel's own oom_kill counter is read host-side from the container's
cgroup. A kill or restart turns proven into failed; a peak over 80% of the
limit adds the memory_tight mark. New Romm fixture; edges M1 / M1old.

Red-proof on scratch 9202: M1old (template as promoted, 512M, 4 workers)
OOM-killed at +76 s -> failed. M1 (current, 768M, 2 workers) proven, 0
kills in 608.5 s, peak 81% -> memory_tight.

Test code only; no template changed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
2026-09-23 08:54:53 +02:00
parent 02844ae0a5
commit cfcfe52784
4 changed files with 320 additions and 41 deletions
+17 -32
View File
@@ -1,40 +1,25 @@
# REPORT — upgrade harness widened from the update night (2026-09-21)
# REPORT — the upgrade harness watches memory (2026-09-23)
**Test code only.** No template was changed; no `image:` line moved; `git diff` touches exactly
`scripts/upgrade_fixtures.py` and `scripts/upgrade-test.py`.
**Test code only.** No template was changed; no `image:` line moved; the diff touches exactly
`scripts/upgrade-test.py`, `scripts/upgrade_fixtures.py`, `CHANGELOG.md` and this file.
## What was done
The update night (`felhom.eu/documentation/audits/DRILL-update-night-2026-09-21.md`) walked real
within-a-major upstream edges on scratch guest 9202 **through the product's own guarded Update**,
each app seeded and read back through its own front door, against a **private drill catalog** — the
live catalog was never touched. This commit brings the expensive half of that work (the seed routes)
back into the harness, so the same edges can also be run here **with their ABORT step**, which the
box deliberately does not offer.
The RomM lesson (R-635): a walk proves "the update applied and the data survived", not "the new
version runs". `upgrade-test.py` v2 adds a **memory watch** after a successful readback — `--soak`
seconds (default 600) of light load, sampling the kernel's `oom_kill` counter host-side, the peak
against the compose limit, and restarts. Kill or restart → `failed`; peak > 80 % → mark
`memory_tight`. New `Romm` fixture and edges `M1` / `M1old`.
- Four new fixtures: `ActualBudget`, `Navidrome`, `AudiobookShelf`, `Vikunja`. Each seeds through the
app's OWN interface (R-156) and each carries a negative control that runs on every `verify()`.
- Seven new edges `U1`–`U7`, all real upstream moves that existed on 2026-09-21 and that this catalog
has NOT made. Every one holds its database engine constant.
## Red-proof (scratch guest 9202, `/opt/upg`, removed afterwards)
## What was proven, and where
| edge | template | verdict | memory |
|---|---|---|---|
| **M1old** | as promoted (`15f9ebf`): 512M, 4 workers | **failed** | first OOM kill at **+76 s**, peak 100 % of 512 MiB, restarts 0 |
| **M1** | current: 768M, 2 workers | **proven** + mark `memory_tight` | 608.5 s under 11 429 requests (5 712 × 200, 5 717 × 401): **0 kernel OOM kills, 0 restarts**, peak 621 MiB = **81 %** of 768 MiB — the watch passes the fix and still flags the thin headroom R-635 left open |
Box-side, through the guarded Update, with the data read back before and after — evidence per app in
`felhom.eu/documentation/audits/update-night-2026-09-21/apps/<app>/`. The harness-side runs of
U1–U7 are **owed**: the code is in, the runs are not.
`C3` (the standing negative control) was not run: its `container_name: privatebin` collides with the
privatebin the controller runs on 9202. The M1old/M1 pair is this step's own control.
## Gates
`python3 scripts/catalog_gates.py --fast` — image-pins OK, engine-major OK, catalog-since OK,
copy-i18n OK. `all catalog gates OK`, exit 0. The two slow gates need a container runtime and were
not run; no template changed, which is what they inspect.
## A catalog defect found by the night, filed and NOT fixed here
Two apps are presented to the household as UNHEALTHY while working perfectly: `tandoor`'s
`.felhom.yml` probe names port 8080 where the container listens only on 80, and `zipline`'s names
`/api/health` where that app answers 404 — **while the compose healthcheck in the same file uses
`/api/healthcheck` and is correct**. A static sweep of all 53 templates comparing the two health
checks against each other finds both, plus `wger` and `home-assistant` as unmeasured candidates and
`adventurelog` as a false positive. Filed as **R-618** with the proposed gate; deliberately not
fixed in the same commit as the harness change.
Gates: `python3 scripts/catalog_gates.py --fast` → all OK.
Full session report: `felhom.eu/REPORT.md`; evidence `felhom.eu/documentation/audits/update-rulings-2026-09-23/`.