night 2026-09-25 (in progress): part 7 shipped as controller v0.271.0 — decisions 31-33, evidence A/B/C/F, register (R-685..R-687; closed R-672 R-673 R-684 R-680 R-678)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-24 23:04:43 +02:00
parent 75ff26408a
commit a7971c4822
126 changed files with 10324 additions and 9 deletions
@@ -370,6 +370,30 @@ R-636's louder repeated alarm.
    (a) broke the second. Shipped as controller **v0.269.1** — a second release in the session, instead of
    shipping (a) to the fleet with the floor.

+### 2026-09-25 (night) — three decisions taken by CC unattended, building §6.4 part 7
+
+31. **After W+5h the full-system backup waits only for an automatic step ALREADY in flight, and never past
+    W+5h30m** — *decided by CC unattended 2026-09-25 — operator may reverse.* *One sentence:* when the leg is
+    still running at W+5h, does the full-system backup start at once or wait for the step? **Options:** (a) start
+    at once — the backup's quiesce stops an app in the middle of its update's verify, which fails the step into
+    an undo during a backup; (b) wait for the step in flight, capped at W+5h30m. **Costs:** (a) a spurious undo
+    and a backup of an app mid-change; (b) the full-system backup keeps at least 30 minutes of its window instead
+    of 60 on the worst night; a stuck flag cannot hold it past the cap. **Why (b):** §6.4.2 point 3 already says
+    "a step already running finishes"; this makes the gate agree with it. `quiesce.updateLegDefers`,
+    `TestD20_UpdateLegDefers`.
+32. **The controller's own self-update waits for the whole automatic leg, not only for a step in flight** —
+    *decided by CC unattended 2026-09-25 — operator may reverse.* *One sentence:* may the controller swap itself
+    between two automatic steps? **Options:** (a) yes, as for manual updates (the lock covers only a step in
+    flight); (b) no, the lock covers the whole leg. **Costs:** (a) the swap restarts the controller and the rest
+    of the night's leg is lost (it is not resumed); (b) a floor-served controller release waits up to the leg's
+    length (at most until W+5h) on an update night. **Why (b):** the leg is the night's only update window
+    (decision 11); a release can wait an hour, a household's app cannot wait a day for nothing.
+33. **The automatic leg takes ONE step per app per night** — *follows the operator's own brief of 2026-09-24
+    ("one app at a time, one tested step per app"), recorded here because §6.4.2 point 6 (a) said "rescan between
+    steps", which reads as climbing several.* **Cost:** an app three steps behind takes three nights. **Why:**
+    each step then gets a night of the household using it before the next one, and a failure is one step wide.
+    Operator may widen it. `TestLeg_OneStepPerAppPerNight`.
+
 ---

 ## 3b. ANSWERED 2026-09-23 — the seven questions Slices 6 and 7 needed
@@ -1140,7 +1164,7 @@ what the part can do to a household's data if it is wrong, not how likely that i
 | **4** | **SHIPPED — catalog `6db08a5`, night 2026-09-23** (`audits/DRILL-night-2026-09-23.md`): `update_ladder:` in `.felhom.yml`, one JSON entry per line (spiked on controller v0.266.0 and v0.267.0 first — the controller ignores the key); `check-test-record.py` (static, CI) + `check-test-record-move.py` (history + the registry for moved refs only, decision 23); the only writer `upgrade-test.py --write-ladder`; the 21 moves of 2026-09-22 backfilled from their records (21 proven). Not built: `steps/<to>.yml` (part 5 needs it once an app has two steps); `CompareImageRefs` did NOT move to the gate — the gate asks for a proven test instead, which is decision 13's own test. **The test record + the catalog gate + the memory check.** The harness writes the ladder entry (below) from its verdict record, including the memory watch's peak and marks; the gate refuses an image move with no entry, an entry with a `failed` verdict, or one with no memory watch; `CompareImageRefs`' rule moves here as the push-time safety net. **Backfill:** one entry per current pin — the 21 proven moves from their records, every other pin `needs_person: "never tested"`, which is honest and keeps them manual. A version move re-checks `mem_limit` against the watch's peak (the RomM follow-up: gate, not checklist, because the watch now produces the number). | 13, R-635 follow-up | **2.5** | the memory watch (shipped 2026-09-23) | none on a box — catalog-side only |
 | **5** | **SHIPPED — controller v0.268.0 (`206b035`) + catalog `5ed599c`, proven live on 9202 2026-09-24** (`audits/ladder-2026-09-24/partD/`): romm 5.3.0/11.4 → 5.3.1/11.4 (its app step, from `steps/90dd9d68258286ef.yml`) → 5.3.1/11.8 (the engine step; `mariadb-upgrade` ran) in two presses, the seeded account read back after each, the page's „Hátralévő frissítési lépések" 2 → 1 → none. Step files: `templates/<app>/steps/<StepKey(to)>.yml` (sha256 of `to` as canonical JSON, 16 hex; `stacks.StepKey` = `ladder.step_key`), refused absent or wrong by `check-test-record.py` rule 4, written by `--write-ladder` when a step is superseded, 8 backfilled from the NEWEST commit naming each step's images (romm's from `f4eb94f`, not the OOM-looping `15f9ebf`). The box reads them from its `--depth 1` clone (the whole tree is there — measured). One press = one step; a missing step file refuses before anything moves; an installed version matching no entry jumps, logged by name (measured live: vikunja 2.5.0). Limitation: a step has no `.felhom.yml` of its own (R-664). **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/<to>.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape |
 | **6** | **BOTH HALVES SHIPPED. BOX HALF — controller v0.269.0 + v0.269.1, proven live on 9202 2026-09-24** (`audits/night-2026-09-24/B/`): the compose that runs pins `tag@sha256` from the ladder entry that tested those refs; pins and records stay digest-free; the badge reads behind for a newer TESTED digest; **an installed app keeps its digest until a guarded Update moves it** (v0.269.0 let the sync move it — found live, fixed in v0.269.1, `CarryDigests`); Compose accepts the form for all 25 ladder apps. **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. **The box half (compare, render `name:tag@sha256`) is not built.** **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
-| **7** | **SPIKED 2026-09-24 (night, Part C) — the measured build brief is §6.4.2 below.** **The update leg in the chain + the automatic caller + the switch.** A leg that starts when the off-site leg has ENDED (on every exit path, skips included), one app at a time, one step per press, `app_update.unattended` default ON (decision 12), `stacks.update_window` removed, reads `UpdateRefusal.Reason`, remembers a failed step so it never re-presses it, and the full-system backup's gate waits for it until W+5h (decision 20). | 11, 12, 14, 15, 20 | **3** | 1, 2, 5 | medium — the only part that acts with nobody watching; everything above is what makes it safe |
+| **7** | **SHIPPED — controller v0.271.0 (`9cf13a3`), proven on 9202 over four simulated nights 2026-09-24/25 and watched through the demo boxes' first real automatic night (`audits/DRILL-night-2026-09-25.md`).** Decisions 31–33 taken unattended. Gaps a scratch box cannot close: R-687; no resume after a restart: R-686. *(Before:)* **SPIKED 2026-09-24 (night, Part C) — the measured build brief is §6.4.2 below.** **The update leg in the chain + the automatic caller + the switch.** A leg that starts when the off-site leg has ENDED (on every exit path, skips included), one app at a time, one step per press, `app_update.unattended` default ON (decision 12), `stacks.update_window` removed, reads `UpdateRefusal.Reason`, remembers a failed step so it never re-presses it, and the full-system backup's gate waits for it until W+5h (decision 20). | 11, 12, 14, 15, 20 | **3** | 1, 2, 5 | medium — the only part that acts with nobody watching; everything above is what makes it safe |
 | **8** | **SHIPPED — controller v0.265.0 + hub v0.121.0, proven live on 9202 2026-09-23** (`audits/cleanup-2026-09-23/`; a kernel `oom_kill` counter, not the sticky flag — `08` §6.2). **R-636** — the same OOM key re-firing escalates instead of staying one `warning` for six hours. | R-636 | **1** | — | none |
 | **9** | **SHIPPED — controller v0.265.0, proven live on 9202 2026-09-23** (badge „Megállítva — visszaállítás szükséges" / "Stopped — restore needed", no Update button, 409 `held` unchanged). **R-625** — a held app stops offering an Update it will refuse. With the undo, holds become rarer; the lie on the page does not go away by itself. | R-625 | **0.5** | 1 | none |
 | **10** | **PostgreSQL majors converted by the box.** A guarded-update step: `pg_dumpall` from the old engine, a NEW datadir (the old one kept aside, never deleted, until the check passes), load, check; then each of the eleven apps proven on the bench before its catalog move. | 16, R-463 | **2 + 3** | 1 (the same load discipline), 4 | **HIGH** — it rebuilds the datadir; bounded by keeping the old datadir aside |
@@ -1225,6 +1249,17 @@ per-box switch `app_update.unattended`, **default ON (decision 12)**, and `stack
 removed; (e) the live proof: this night's four apps, with the window moved forward through the
 product's own form (`POST /backups/window` — measured: the three legs reschedule at once).

+**6.4.2 as BUILT (v0.271.0) — where the build differs from the brief above, named:** (a) the leg is
+`stacks.RunUpdateLeg`, chained by `chainUpdateLeg` inside the `offbox-backup` job (every path, a panic included);
+**one step per app per night** (decision 33 — not "rescan between steps" in one night); (b) the failed-step record is
+`app.yaml` `failed_update_step` tied to the ladder's print (R-680); R-679's `current` refusal shipped in v0.270.0;
+(c) the gate interlock is `quiesce.Options.UpdateLegFn` with decision 31's in-flight grace; (d) the switch is
+`settings.json` `app_update.unattended`, absent = ON, a card on the settings page; `stacks.update_window` removed;
+(e) the live proof: `audits/DRILL-night-2026-09-25.md` Part C (four nights on 9202) and Part D (the demo boxes).
+**Found live, not in the brief:** the leg presses the next app in the same second the previous step ends, so "a
+controller killed BETWEEN two apps" lands inside the next step's early phases — which the guarded update puts back
+(Scenario G); the rest of the night is lost (R-686).
+
 ### 6.4.1 (record) The update night — the drill brief that preceded the rulings, costed and re-costed

 **The ruling is decision 6: all 53 apps, through the nightly rotation.** This is an ORDER inside that