felhom.eu/REPORT.md

REPORT — update arc: the operator's rulings recorded, the undo and ladder spiked, the memory watch, the build plan

2026-09-23. Repos touched: felhom.eu (docs, register, STATUS, CONTEXT, evidence), app-catalog-felhom.eu (scripts/ only — the memory watch), admin/app-catalog-drill (drill commits, reset to live main at the end). felhom-controller, felhom-agent, hub: read only. Architecture read first and named: documentation/architecture/09-update-architecture.md (all of it), 07-backup-architecture.md §6. Baselines verified live before starting: controller b9deec19077b, agent d9864a94bf62, felhom.eu 267dcad01bcf, catalog 02844ae0a579 — all equal to the brief.

---

1. Not done, or changed from the brief — first

item	state
Part 0 — rulings into 09	done, commit 805ad1e (documents only)
Part 1 — the undo, four cases + the wrong case	done, with three changes. (a) The fourth case, "files on disk", was measured on vikunja's attachment (a file in a volume), not on a bind-mounted drive folder: romm's two drive folders stayed EMPTY throughout — they measured nothing and are reported as unmeasured. (b) Starting the app by hand needed its decrypted secrets; the session's safety guard refused that, and it was not worked around. The product's own Start was used instead, after lifting the hold with the operator CLI + a controller restart. (c) The wrong case was run on BOTH engines, and on PostgreSQL with both the product's loader and the fixed one — the fixed one produced the session's most important finding (a truncated copy loads rc 0).
Part 2 — the ladder	done
Part 3 — the memory watch + red-proof	done; harness v2. C3, the harness's standing negative control, was NOT run: its template's container_name: privatebin collides with the privatebin the controller runs on 9202. The memory watch has its own pair instead — M1old must fail, M1 must pass.
Part 4 — the build plan	done, 09 §6.4 — with one open point for the operator the brief did not expect (§6)

Claims in the brief that turned out wrong:

1. "A safety dump exists for every app class" — false. An app with no database server gets none (update safety dump for vikunja: the app has no database — nothing to copy (no-op)), measured.
2. "The box keeps a git clone of the catalog" with history — false. Depth 1 on both demo guests, rev-list --count HEAD = 1; sync.go:283/:300 clone and fetch --depth 1.
3. "stacks.update_window is unread" — true, and the grep was widened from config.go + setup/handlers.go to the whole controller repo: the only other hits are configs/controller.yaml.example and the i18n base file. No Go code reads it.
4. The register held 329 row lines by grep -c '^| \*\*R-', not 326; the highest id was R-636 as stated.

2. Part 0 — the rulings

09 §3 gains decisions 11–18 in the existing shape; decision 3's second half and §6.1's abort paragraph are marked REPLACED with pointers; §4 says why the undo is not a rollback; §3b is kept, headed ANSWERED, each question pointing at its decision; §6.2 rewritten to the ruled shape; the slices table updated. Register: R-450, R-451, R-446, R-463 cite the decisions.

3. Part 1 — the undo, by hand

Full evidence and tables: documentation/audits/update-rulings-2026-09-23/README.md.

	docmost (PostgreSQL)	romm (MariaDB)	vikunja (volume, no DB server)
held after	95.3 s	102.6 s	93.3 s
old version on migrated data	refuses	refuses	starts
safety dump	135 816 B, DB only	62 943 B, DB only	none
product loader (ImportDump semantics)	FAILS, rc 3, foreign keys	rc 0, 12 tables left behind	—
fixed load (empty schema + copy, one transaction)	rc 0, 1.38 s	(product loader sufficed)	—
undo → healthy on the OLD probe	≈ 16 s	≈ 38 s	≈ 1 s
data before / after the backup	yes / yes	yes / yes	yes / yes (attachment too)

Does a product path load a safety dump back? Yes — rollbackSafetyDump, but only the off-site restore calls it; the update never reads its own dump, and failAndHold deletes the pre-update definition copies.

The wrong case: PostgreSQL + product loader → rc 3, nothing changed (honest hold). PostgreSQL + the fixed atomic loader + a half-length copy → rc 0 and an EMPTY database (0 users, 0 constraints, 0 indexes) that still shows 42 tables and 48 ledger rows — no hold, a dishonest success. MariaDB + half copy → rc 1, half the tables already replaced (not atomic). The whole copies carry an end marker the truncated ones lack; ValidateDump does not check it.

4. Part 2 — the ladder

One press on vikunja two steps behind: 2.3.0 → 2.5.0 in 9.5 s; 2.4.0 never ran. The box cannot see 2.4.0 (depth-1 clone). Recommended format: update_ladder: in .felhom.yml, intermediate steps with their own definition — not git history, because romm's image-moving commit is the definition that OOM-looped on demo-hp. Full comparison in the audit.

5. Part 3 — the memory watch

upgrade-test.py v2: after a successful readback, --soak seconds (default 600) of light load (4 callers), sampling every 15 s the kernel's oom_kill counter read host-side from the container's cgroup, the peak, the limit, the restarts and Docker's OOMKilled flag. Kill or restart → failed; peak > 80 % → mark memory_tight. New Romm fixture; edges M1 (current template) and M1old (the template as promoted, 15f9ebf).

• M1old (red-proof): failed — first OOM kill at +76 s, peak 512 MiB = 100 % of the limit, restarts 0 (the container kept running — the shape that hid it on demo-hp), abort refuses. Ten minutes is ample for this failure under load.
• M1 (positive control): proven + mark memory_tight — 608.5 s under 11 429 requests (5 712 × 200, 5 717 × 401): 0 kernel OOM kills, 0 restarts, peak 621 MiB = 81 % of 768 MiB — the watch passes the fix and still flags the thin headroom R-635 left open.

6. Part 4 — the build plan

09 §6.4: ten parts, ≈ 22 evenings, recommended order undo → sentences in the household's language

• notifier honesty → test record + gate + memory check → ladder → digests → the automatic leg → R-636 → R-625 → PostgreSQL conversion; fleet view deferred by ruling. One open point needs the operator (R-643): as ruled, the update leg sits between the off-site leg (W+105m) and the full-system gate (W+2h) — at most 15 minutes a night. Recommendation: the full-system backup waits for the leg inside its own four-hour window.

7. Rows

Opened (8): R-637 (build the undo), R-638 (the loader cannot replay over a newer schema — and the restore the hold names is UNMEASURED after a real schema migration), R-639 (pre-update copies deleted on hold), R-640 (a truncated PostgreSQL copy loads rc 0 into an empty database), R-641 (no-DB apps have no last-second copy), R-642 (Start returns 200 over a crash loop), R-643 (the ≤15-minute leg, operator), R-644 (gokapi crash-looping on 9202 at session start, not caused here). Updated: R-446, R-450, R-451, R-462, R-463. Closed: none. 329 → 337.

8. Teardown — three layers

layer	state
machine — guest 9202	controller.yaml restored from the saved copy and read back identical (live catalog, no update: block); catalog cache re-cloned from the live repo (02844ae); docmost, romm, vikunja removed through the product (no containers, no volumes); romm's drive folder (kept by the product, R-442) removed by name; /opt/upg and every temp file in /root removed; the eight test images removed by name — vikunja:2.4.0 was absent, i.e. never pulled, which is the jump seen from a second side; no prune. Containers afterwards: the same three apps as at the start (gokapi still crash-looping — R-644, pre-existing). 82-teardown-guest.txt
host — demo-hp	nothing provisioned; only transient /tmp files, removed
hub	nothing touched — no hub call was made
drill repo	reset to live main 02844ae0a579; has_actions: false; image lines identical to live. The local clone's push URL to the LIVE catalog was disabled at the start. 81-teardown-drill-repo.txt

One instrumentation slip, recorded: a background watcher and the first teardown both wrote the same temporary script file on demo-hp at the same moment, so the first teardown never ran (its output file held the watcher's lines). Caught by reading the file, re-run after the watcher ended; the second run is the one recorded.

Fences: DooPlex, Peti's box, ep0, the demo guests' apps, drill-r50, tester-1 and the hub were not touched. The live catalog's main was 02844ae0a579 before and after.

unproven.py --summary: walked 20 / partial 17 / built 14 / missing 4 — not walked 35 of 55, unchanged.