felhom.eu/documentation/audits/ladder-2026-09-24/README.md

# 2026-09-24 — the fleet to 0.267.0 and 0.268.0; the undo after a restore; option A; the ladder on the box

Architecture read first and named: `architecture/09-update-architecture.md` (§3 decisions 13–25, §6.1, §6.1a,
§6.4 parts 4–5), `07-backup-architecture.md` §6, `08-alarm-ladder.md` §5. **Method: endpoint-level** — every
product act is the endpoint the UI invokes; pages fetched as HTML in both languages (`?lang=`); no browser.
Scratch guest **9202** on demo-hp (Tier 0), drill catalog for every failing edge; the live catalog carried only
the step files and gates.

## Not done, or changed

- **Nothing not done.** Parts 0, A, B, C, D and E all ran; the interim "one step per app per day" gate was not
  needed (Part D shipped).
- **The operator's English sentence lost one word, „please"** — the house rule (`i18n_missing_gate.py`,
  `EN_FORBIDDEN`) refuses it. The Hungarian is verbatim; its formal register raised the R-516 ceiling 18 → 19 with
  the reason written beside it.
- **The hold names ONE copy (the newest whole one), not a list.** A list would need new sentence copy; the existing
  sentence names one copy with what it holds.
- **Part A's second failing update (g) did not fail:** right after a restore the stack dir's `.felhom.yml` was the
  restored one (real probe port), so that press was a good update. The proof the brief asks for is (e) — an app
  restored under v0.267.0, then a failing update on v0.268.0 — and it passed. (g) still shows the v0.268.0 restore's
  volumes copied with no "unlabelled" warning. Filed as R-665 (not diagnosed).
- **R-653 and R-656 are unit-tested and red-proofed, not run on a bench** — no bench was provisioned this session.
- **Two catalog test suites were already red at the start** (`test_gate_decoys.py`, `test_ladder_writer.py` — last
  night's pin moves under literals). Fixed and filed as R-663 (closed).
- **One of the first R-660 control's two alarm events is not attributable** (the dropped-event line names no app);
  the repeat control produced exactly one. Likely gokapi (R-667), stated as inference.
- An older wiring test (`TestR475_AdapterReadsEveryTier`) asserted the adapter hands the hold the precondition copy
  — the thing R-659 changes on purpose. Its assertion was replaced, with the reason in the test.

## Claims in the brief, checked

| the brief's claim | what was found |
|---|---|
| compose resolves volume names as `<project>_<volume>` for every template (some set `name:`) | **No template sets `name:`**, top-level or per volume (all 53 read). Supported anyway, and tested. |
| the syncer can carry a `steps/` folder | **It does not need to.** The syncer copies two files into the stack dir; the box reads `steps/` from its catalog CLONE, which (depth 1) holds the whole tree. Measured on 9202: `steps/spike-B.yml` present in the clone after one sync (`partD/01`). |
| „saját meghajtó" never holds files for a file-leg app (from `07` §6) | **Held, and it goes further:** read from source, the SECOND-DRIVE unit restore refuses such an app too (the R-538 guard is in the shared function), and the second drive's file restore brings back no database. So only off-site brings a file app back whole (R-661). An app with only OPTIONAL legs (romm, komga, audiobookshelf) is accepted by the unit restore. |
| one press = one step is safe for an engine step whose datadir converts | **Held for MariaDB, measured once:** romm press 2 ran `mariadb-upgrade` 11.4.13 → 11.8.9 and the account read back. The undo copy covers the pre-conversion volume if the step fails (not exercised on an engine step this session). PostgreSQL majors stay refused (part 10). |
| today's press jumps A → C | **Held** on v0.267.0: vikunja 2.4.0 → 2.6.0 in one press, 2.5.0 never ran (`partD/00-spike.json`). |

## Part 0 / Part E — the floor

| save (UTC) | floor / MinAgent | read back | demo-felhom | demo-hp |
|---|---|---|---|---|
| 05:12:14 | 0.267.0 / 0.131.0 | `min_controller_version=0.267.0`, `min_agent=0.131.0`; hub `SERVED … from declared` | 0.267.0 healthy at +28 s | same |
| 06:47:13 | 0.268.0 / 0.131.0 | same shape | 0.268.0 healthy at +30 s | same |

Hub 0.122.0 rolled out first (ArgoCD, manifest `500488c`, pod image read back). **Stays below:** `drill-r50`
(DOWN, agent 0.129.0 — held by MinAgent, a Tier-0 leftover). Evidence `part0/`.

## Red-proofs (each seen failing, then restored)

| row | mutation | failed at | file |
|---|---|---|---|
| R-658 undo | `appVolumes` returns the label selector | `copied 0 of 2 declared volumes` | `redproofs/A-r658-undo.txt` |
| R-658 restore | create without labels | `created vikunja_files WITHOUT the project label` | `A-r658-restore.txt` |
| R-658 remove report | `appVolumeSet` = labelled only | `volumes_removed = []` | `C-r651-and-A-remove-report.txt` |
| R-651 | skip deleting the applied files | both files survive; undo would probe the REMOVED install's file | same |
| R-659 hold | `WholeOnTier` always true | round-11 case names „saját meghajtó" | `B-r659-hold.txt` |
| R-659 page | gate dropped from `stacks.html` | Mentések button beside a no-whole-copy hold | `B-r659-page.txt` |
| R-659 hub | dropped from `operatorOnlyEvents` | "must be operator-only" | `B-hub-operator-only.txt` |
| R-660 | `updateHeldSet` returns nil / union line removed | "held app reported DOWN" / wiring | `C-r660.txt` |
| ladder (box) | `nextLadderStep` always the template | press 1 pinned C; C brought up after B failed; missing step file → `done` | `D-ladder.txt` |
| ladder (catalog gate) | rule 4 removed | 3 decoys pass wrongly | `D-catalog-gate.txt` |
| ladder (writer) | STEP block off | "has no definition" / file absent | `D-catalog-writer.txt` |
| R-653 / R-656 | `load_verdict` always reached / no removal | ghost's all-`err` watch reads reached; last run's config survives | `C-r653-r656.txt` |

Green: controller `go build/vet/test` rc 0 (31 packages); `controller_gates.py` all OK; hub `go test` OK +
`repo_gates.py` OK; catalog `catalog_gates.py` OK, decoys 84 OK, writer 6 OK, bench 5 OK.

## Live proofs on 9202

- **A (R-658)** — v0.267.0: vikunja installed, a pull-failing update took its own backup, restore from „helyi" →
  both volumes' labels `null` (`partA/01`). v0.268.0: failing update (2.5.0 → 2.6.0, probe 8999) → log
  *„carry no compose label … copied by name"*, *„will hold 2 named volume(s)"*, undone in 7 s, seeds A (before) and
  B (after the restore) read back. v0.268.0 restore → labels `project`/`volume`/`version`, no compose warning
  (`partA/02`). Remove → `volumes_removed` names both.
- **B (R-659)** — nextcloud installed at ladder step 1, seeded; failing update with the undo copy's marker removed
  during `verifying` → HELD, `hold_no_whole_copy: true`; the sentence in hu and en exactly as ruled (ASCII
  fragments, positive and negative control); no Mentések button on the app page or the list, both languages; log
  *„NO copy on this box brings it back whole (seen: tier 2 …, tier 1 …; drive files declared: true)"* and
  *„DROPPED event app_hold_no_whole_copy (severity critical)"* (R-620's line). Steps line „… lépések: 1"
  (`partB/01`).
- **C (R-660, R-651)** — dead-app heartbeat 2 s after the hold: *„0 currently down"*; no `app_start_failed` for
  the held app in ~12 min; a throwaway stopped out of band raised one (`partB/03`, `06`). Remove: `applied-compose.yml`
  and `applied-meta/` present before, gone after; every undo copy gone (`partB/07`).
- **D (the ladder)** — romm installed at 5.3.0/11.4, seeded; the page read „Hátralévő frissítési lépések: 2" /
  "Update steps remaining: 2". Press 1 → *„ladder — step 2 of 3 … from 90dd9d68258286ef.yml"*, pin 5.3.1/11.4,
  applied == that step file, MariaDB „upgrade not required", account read back, count 1. Press 2 → *„the last step
  (3 of 3)"*, pin 5.3.1/11.8, `mariadb-upgrade` 11.4.13 → 11.8.9, account read back, count gone (`partD/10`). A
  second seed after press 1 was refused by RomM (403 for a non-admin) — the readback rests on seed A.

## Teardown — three layers

- **Machine (9202):** vikunja, nextcloud, romm, actualbudget ×2 removed THROUGH THE PRODUCT; 0 volumes, 0 undo
  copies; drive folders nextcloud/romm removed by name (R-442 keeps them on 9202); 11 test images removed by name
  (`/var/lib/docker` 27 % → 18 %), no prune. `controller.yaml` restored from `controller.yaml.pre-ladder0924`, git
  URL read back as the live catalog, cache at live `5ed599c`. **9202 stays on v0.268.0.** Standing apps as found
  (gokapi was crash-looping before the session — R-644/R-667).
- **Host (demo-hp):** no guest created; nothing else touched.
- **Hub:** no customer or appliance record created. Floor 0.268.0 is the intended end state.
- **Drill repo:** reset to live `main` `5ed599c`, `has_actions: False`, private — read back.

## Rows

Closed: R-658, R-659, R-660, R-651, R-653, R-656, R-40 (superseded), R-663 (filed and fixed). Opened: R-661, R-662,
R-664, R-665, R-666, R-667. Narrowed: R-450 (part 5 shipped). Register **336 → 335 rows, 683,233 → 679,393 bytes**.