upgrade-test: watch memory after the readback (harness v2, R-635/R-462)
gates / gates (push) Successful in 2s
gates / gates (push) Successful in 2s
After an edge reads back, --soak seconds (default 600) of light load while the kernel's own oom_kill counter is read host-side from the container's cgroup. A kill or restart turns proven into failed; a peak over 80% of the limit adds the memory_tight mark. New Romm fixture; edges M1 / M1old. Red-proof on scratch 9202: M1old (template as promoted, 512M, 4 workers) OOM-killed at +76 s -> failed. M1 (current, 768M, 2 workers) proven, 0 kills in 608.5 s, peak 81% -> memory_tight. Test code only; no template changed. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -1,40 +1,25 @@
|
||||
# REPORT — upgrade harness widened from the update night (2026-09-21)
|
||||
# REPORT — the upgrade harness watches memory (2026-09-23)
|
||||
|
||||
**Test code only.** No template was changed; no `image:` line moved; `git diff` touches exactly
|
||||
`scripts/upgrade_fixtures.py` and `scripts/upgrade-test.py`.
|
||||
**Test code only.** No template was changed; no `image:` line moved; the diff touches exactly
|
||||
`scripts/upgrade-test.py`, `scripts/upgrade_fixtures.py`, `CHANGELOG.md` and this file.
|
||||
|
||||
## What was done
|
||||
|
||||
The update night (`felhom.eu/documentation/audits/DRILL-update-night-2026-09-21.md`) walked real
|
||||
within-a-major upstream edges on scratch guest 9202 **through the product's own guarded Update**,
|
||||
each app seeded and read back through its own front door, against a **private drill catalog** — the
|
||||
live catalog was never touched. This commit brings the expensive half of that work (the seed routes)
|
||||
back into the harness, so the same edges can also be run here **with their ABORT step**, which the
|
||||
box deliberately does not offer.
|
||||
The RomM lesson (R-635): a walk proves "the update applied and the data survived", not "the new
|
||||
version runs". `upgrade-test.py` v2 adds a **memory watch** after a successful readback — `--soak`
|
||||
seconds (default 600) of light load, sampling the kernel's `oom_kill` counter host-side, the peak
|
||||
against the compose limit, and restarts. Kill or restart → `failed`; peak > 80 % → mark
|
||||
`memory_tight`. New `Romm` fixture and edges `M1` / `M1old`.
|
||||
|
||||
- Four new fixtures: `ActualBudget`, `Navidrome`, `AudiobookShelf`, `Vikunja`. Each seeds through the
|
||||
app's OWN interface (R-156) and each carries a negative control that runs on every `verify()`.
|
||||
- Seven new edges `U1`–`U7`, all real upstream moves that existed on 2026-09-21 and that this catalog
|
||||
has NOT made. Every one holds its database engine constant.
|
||||
## Red-proof (scratch guest 9202, `/opt/upg`, removed afterwards)
|
||||
|
||||
## What was proven, and where
|
||||
| edge | template | verdict | memory |
|
||||
|---|---|---|---|
|
||||
| **M1old** | as promoted (`15f9ebf`): 512M, 4 workers | **failed** | first OOM kill at **+76 s**, peak 100 % of 512 MiB, restarts 0 |
|
||||
| **M1** | current: 768M, 2 workers | **proven** + mark `memory_tight` | 608.5 s under 11 429 requests (5 712 × 200, 5 717 × 401): **0 kernel OOM kills, 0 restarts**, peak 621 MiB = **81 %** of 768 MiB — the watch passes the fix and still flags the thin headroom R-635 left open |
|
||||
|
||||
Box-side, through the guarded Update, with the data read back before and after — evidence per app in
|
||||
`felhom.eu/documentation/audits/update-night-2026-09-21/apps/<app>/`. The harness-side runs of
|
||||
U1–U7 are **owed**: the code is in, the runs are not.
|
||||
`C3` (the standing negative control) was not run: its `container_name: privatebin` collides with the
|
||||
privatebin the controller runs on 9202. The M1old/M1 pair is this step's own control.
|
||||
|
||||
## Gates
|
||||
|
||||
`python3 scripts/catalog_gates.py --fast` — image-pins OK, engine-major OK, catalog-since OK,
|
||||
copy-i18n OK. `all catalog gates OK`, exit 0. The two slow gates need a container runtime and were
|
||||
not run; no template changed, which is what they inspect.
|
||||
|
||||
## A catalog defect found by the night, filed and NOT fixed here
|
||||
|
||||
Two apps are presented to the household as UNHEALTHY while working perfectly: `tandoor`'s
|
||||
`.felhom.yml` probe names port 8080 where the container listens only on 80, and `zipline`'s names
|
||||
`/api/health` where that app answers 404 — **while the compose healthcheck in the same file uses
|
||||
`/api/healthcheck` and is correct**. A static sweep of all 53 templates comparing the two health
|
||||
checks against each other finds both, plus `wger` and `home-assistant` as unmeasured candidates and
|
||||
`adventurelog` as a false positive. Filed as **R-618** with the proposed gate; deliberately not
|
||||
fixed in the same commit as the harness change.
|
||||
Gates: `python3 scripts/catalog_gates.py --fast` → all OK.
|
||||
Full session report: `felhom.eu/REPORT.md`; evidence `felhom.eu/documentation/audits/update-rulings-2026-09-23/`.
|
||||
|
||||
Reference in New Issue
Block a user