felhom.eu/documentation/audits/ladder-2026-09-24

2026-09-24 — the fleet to 0.267.0 and 0.268.0; the undo after a restore; option A; the ladder on the box

Architecture read first and named: architecture/09-update-architecture.md (§3 decisions 13–25, §6.1, §6.1a, §6.4 parts 4–5), 07-backup-architecture.md §6, 08-alarm-ladder.md §5. Method: endpoint-level — every product act is the endpoint the UI invokes; pages fetched as HTML in both languages (?lang=); no browser. Scratch guest 9202 on demo-hp (Tier 0), drill catalog for every failing edge; the live catalog carried only the step files and gates.

Not done, or changed

• Nothing not done. Parts 0, A, B, C, D and E all ran; the interim "one step per app per day" gate was not needed (Part D shipped).
• The operator's English sentence lost one word, „please" — the house rule (i18n_missing_gate.py, EN_FORBIDDEN) refuses it. The Hungarian is verbatim; its formal register raised the R-516 ceiling 18 → 19 with the reason written beside it.
• The hold names ONE copy (the newest whole one), not a list. A list would need new sentence copy; the existing sentence names one copy with what it holds.
• Part A's second failing update (g) did not fail: right after a restore the stack dir's .felhom.yml was the restored one (real probe port), so that press was a good update. The proof the brief asks for is (e) — an app restored under v0.267.0, then a failing update on v0.268.0 — and it passed. (g) still shows the v0.268.0 restore's volumes copied with no "unlabelled" warning. Filed as R-665 (not diagnosed).
• R-653 and R-656 are unit-tested and red-proofed, not run on a bench — no bench was provisioned this session.
• Two catalog test suites were already red at the start (test_gate_decoys.py, test_ladder_writer.py — last night's pin moves under literals). Fixed and filed as R-663 (closed).
• One of the first R-660 control's two alarm events is not attributable (the dropped-event line names no app); the repeat control produced exactly one. Likely gokapi (R-667), stated as inference.
• An older wiring test (TestR475_AdapterReadsEveryTier) asserted the adapter hands the hold the precondition copy — the thing R-659 changes on purpose. Its assertion was replaced, with the reason in the test.

Claims in the brief, checked

the brief's claim	what was found
compose resolves volume names as <project>_<volume> for every template (some set name:)	No template sets name:, top-level or per volume (all 53 read). Supported anyway, and tested.
the syncer can carry a steps/ folder	It does not need to. The syncer copies two files into the stack dir; the box reads steps/ from its catalog CLONE, which (depth 1) holds the whole tree. Measured on 9202: steps/spike-B.yml present in the clone after one sync (partD/01).
„saját meghajtó" never holds files for a file-leg app (from 07 §6)	Held, and it goes further: read from source, the SECOND-DRIVE unit restore refuses such an app too (the R-538 guard is in the shared function), and the second drive's file restore brings back no database. So only off-site brings a file app back whole (R-661). An app with only OPTIONAL legs (romm, komga, audiobookshelf) is accepted by the unit restore.
one press = one step is safe for an engine step whose datadir converts	Held for MariaDB, measured once: romm press 2 ran mariadb-upgrade 11.4.13 → 11.8.9 and the account read back. The undo copy covers the pre-conversion volume if the step fails (not exercised on an engine step this session). PostgreSQL majors stay refused (part 10).
today's press jumps A → C	Held on v0.267.0: vikunja 2.4.0 → 2.6.0 in one press, 2.5.0 never ran (partD/00-spike.json).

Part 0 / Part E — the floor

save (UTC)	floor / MinAgent	read back	demo-felhom	demo-hp
05:12:14	0.267.0 / 0.131.0	min_controller_version=0.267.0, min_agent=0.131.0; hub SERVED … from declared	0.267.0 healthy at +28 s	same
06:47:13	0.268.0 / 0.131.0	same shape	0.268.0 healthy at +30 s	same

Hub 0.122.0 rolled out first (ArgoCD, manifest 500488c, pod image read back). Stays below: drill-r50 (DOWN, agent 0.129.0 — held by MinAgent, a Tier-0 leftover). Evidence part0/.

Red-proofs (each seen failing, then restored)

row	mutation	failed at	file
R-658 undo	appVolumes returns the label selector	copied 0 of 2 declared volumes	redproofs/A-r658-undo.txt
R-658 restore	create without labels	created vikunja_files WITHOUT the project label	A-r658-restore.txt
R-658 remove report	appVolumeSet = labelled only	volumes_removed = []	C-r651-and-A-remove-report.txt
R-651	skip deleting the applied files	both files survive; undo would probe the REMOVED install's file	same
R-659 hold	WholeOnTier always true	round-11 case names „saját meghajtó"	B-r659-hold.txt
R-659 page	gate dropped from stacks.html	Mentések button beside a no-whole-copy hold	B-r659-page.txt
R-659 hub	dropped from operatorOnlyEvents	"must be operator-only"	B-hub-operator-only.txt
R-660	updateHeldSet returns nil / union line removed	"held app reported DOWN" / wiring	C-r660.txt
ladder (box)	nextLadderStep always the template	press 1 pinned C; C brought up after B failed; missing step file → done	D-ladder.txt
ladder (catalog gate)	rule 4 removed	3 decoys pass wrongly	D-catalog-gate.txt
ladder (writer)	STEP block off	"has no definition" / file absent	D-catalog-writer.txt
R-653 / R-656	load_verdict always reached / no removal	ghost's all-err watch reads reached; last run's config survives	C-r653-r656.txt

Green: controller go build/vet/test rc 0 (31 packages); controller_gates.py all OK; hub go test OK + repo_gates.py OK; catalog catalog_gates.py OK, decoys 84 OK, writer 6 OK, bench 5 OK.

Live proofs on 9202

• A (R-658) — v0.267.0: vikunja installed, a pull-failing update took its own backup, restore from „helyi" → both volumes' labels null (partA/01). v0.268.0: failing update (2.5.0 → 2.6.0, probe 8999) → log „carry no compose label … copied by name", „will hold 2 named volume(s)", undone in 7 s, seeds A (before) and B (after the restore) read back. v0.268.0 restore → labels project/volume/version, no compose warning (partA/02). Remove → volumes_removed names both.
• B (R-659) — nextcloud installed at ladder step 1, seeded; failing update with the undo copy's marker removed during verifying → HELD, hold_no_whole_copy: true; the sentence in hu and en exactly as ruled (ASCII fragments, positive and negative control); no Mentések button on the app page or the list, both languages; log „NO copy on this box brings it back whole (seen: tier 2 …, tier 1 …; drive files declared: true)" and „DROPPED event app_hold_no_whole_copy (severity critical)" (R-620's line). Steps line „… lépések: 1" (partB/01).
• C (R-660, R-651) — dead-app heartbeat 2 s after the hold: „0 currently down"; no app_start_failed for the held app in ~12 min; a throwaway stopped out of band raised one (partB/03, 06). Remove: applied-compose.yml and applied-meta/ present before, gone after; every undo copy gone (partB/07).
• D (the ladder) — romm installed at 5.3.0/11.4, seeded; the page read „Hátralévő frissítési lépések: 2" / "Update steps remaining: 2". Press 1 → „ladder — step 2 of 3 … from 90dd9d68258286ef.yml", pin 5.3.1/11.4, applied == that step file, MariaDB „upgrade not required", account read back, count 1. Press 2 → „the last step (3 of 3)", pin 5.3.1/11.8, mariadb-upgrade 11.4.13 → 11.8.9, account read back, count gone (partD/10). A second seed after press 1 was refused by RomM (403 for a non-admin) — the readback rests on seed A.

Teardown — three layers

• Machine (9202): vikunja, nextcloud, romm, actualbudget ×2 removed THROUGH THE PRODUCT; 0 volumes, 0 undo copies; drive folders nextcloud/romm removed by name (R-442 keeps them on 9202); 11 test images removed by name (/var/lib/docker 27 % → 18 %), no prune. controller.yaml restored from controller.yaml.pre-ladder0924, git URL read back as the live catalog, cache at live 5ed599c. 9202 stays on v0.268.0. Standing apps as found (gokapi was crash-looping before the session — R-644/R-667).
• Host (demo-hp): no guest created; nothing else touched.
• Hub: no customer or appliance record created. Floor 0.268.0 is the intended end state.
• Drill repo: reset to live main 5ed599c, has_actions: False, private — read back.

Rows

Closed: R-658, R-659, R-660, R-651, R-653, R-656, R-40 (superseded), R-663 (filed and fixed). Opened: R-661, R-662, R-664, R-665, R-666, R-667. Narrowed: R-450 (part 5 shipped). Register 336 → 335 rows, 683,233 → 679,393 bytes.