night shift 2026-09-23: the record, the register, the morning note

DRILL-night-2026-09-23.md: Parts A-E. 09 §3 decisions 21 (operator word),
22 and 23 (CC unattended, operator may reverse); §6.4 parts 4 and 6
(catalog half) shipped; §6.1a residuals R-658/R-659. Register 330 -> 336:
R-651..R-660 opened (R-658 and R-659 P1), R-650/R-640/R-499/R-626 closed.
Capability map, nightly rotation (opengist), STATUS (one question: the
floor), CONTEXT, REPORT. The floor stays 0.266.0.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

documentation/architecture/09-update-architecture.md

@@ -280,6 +280,37 @@ builds them (`audits/update-rulings-2026-09-23/`).
     full-system backup keeps at least one hour of its [W+2h, W+6h) window. **Built with §6.4 part 7,
     not before.**
 
+### 2026-09-23 (night) — one operator word, and two decisions taken by CC unattended
+
+21. **Tonight the catalog may move every app whose within-a-major upstream edge is proven on both
+    venues, each with its test record; nothing else moves** (operator word, 2026-09-23 evening). The
+    bar: `proven` on the test bench (seed through the front door, update, read back, ten-minute memory
+    watch) AND on a scratch box through the real guarded Update; no across-a-major edge, no PostgreSQL
+    engine major, no `inconclusive`, no app whose seed has no front-door route. **This is decision 13
+    applied, not a new rule.** Result: `audits/DRILL-night-2026-09-23.md`.
+
+22. **The memory watch's `memory_tight` mark reads the app's OWN memory (anon), not the cgroup peak**
+    — *decided by CC unattended 2026-09-23 night — operator may reverse.* *One sentence:* when the watch
+    says "tight", should the file cache count? **Options:** (a) the cgroup `memory.peak` as built
+    (R-635 follow-up); (b) the `anon` figure of `memory.stat`, sampled every 15 s, with the cgroup peak
+    kept beside it. **Costs:** (a) marks every app that reads files — measured tonight: nextcloud and
+    immich's PostgreSQL at **100 %** with **0** kernel kills, because the kernel fills the limit with
+    cache it drops before killing anything — and the gate then demands a raised limit, which inflates
+    `mem_limit` (a customer-box capacity figure) for no reason; (b) can miss a cache-heavy app whose own
+    memory is tight only if a kill never happens — and a kill or a restart still fails the edge under
+    both options. **Why (b):** decision 13's mark exists for "does not fit the memory" (RomM, R-635), and
+    only the app's own memory decides that. The cgroup peak stays in every record (`memory_cgroup_peak_pct`
+    in the ladder entry). R-652.
+
+23. **The push-time test-record gate asks the registry — for moved refs only** — *decided by CC
+    unattended 2026-09-23 night — operator may reverse.* *One sentence:* may a `--fast` (hook) gate use
+    the network? **Options:** (a) no — digests compared only in the slow periodic run; (b) yes, but only
+    for the refs a push MOVES, and an unreachable registry is INCONCLUSIVE (the push is refused until it
+    can ask). **Costs:** (a) a move whose digest changed between the test and the push is published;
+    (b) a push that moves an image needs the registry — zero requests for every other push. **Why (b):**
+    decision 17 says the box pulls the recorded digest; recording one the registry no longer serves makes
+    every box's pull of that step fail (Scenario E). `app-catalog-felhom.eu/scripts/check-test-record-move.py`.
+
 **RomM follow-ups, operator-agreed the same day:** the test bench watches memory after an update
 (`upgrade-test.py`, 2026-09-23); a version move checks the memory limit (gate or checklist — §6.4);
 R-636's louder repeated alarm.
@@ -807,7 +838,11 @@ only the live box could show, fixed the same day:** the undo's probe was never a
 current probe held the app `unhealthy` (v0.263.1), and the "old" `.felhom.yml` taken at update time
 was already the new one, because `.felhom.yml` flows in on every catalog sync (v0.263.2: the pinned
 version's file is now recorded in `applied-meta/` whenever a version is pinned). **Residual (R-646):**
-an app pinned before v0.263.2 has no such record until its next pin.
+an app pinned before v0.263.2 has no such record until its next pin. **Two more, found by the chaos hour of
+2026-09-23 night (`audits/DRILL-night-2026-09-23.md` Part D):** the undo finds an app's volumes by their compose
+label, and a restore recreates them WITHOUT it — so after any restore the undo copies nothing (R-658, P1); and a
+held file-leg app is pointed at a restore that refuses a database-only copy, with no route left on a box without
+an off-site tier (R-659, P1). A controller kill during `verifying` resumed and undid correctly (round 9).
 
 The spike, as it was run by hand before any build:
 
@@ -1045,9 +1080,9 @@ what the part can do to a household's data if it is wrong, not how likely that i
 | **1** | **SHIPPED — controller v0.263.2, proven live on 9202 2026-09-23** (`audits/undo-live-2026-09-23/`). **The undo.** Keep the pre-update copies (compose, applied, pin, **old `.felhom.yml`**) until the undo is over; in `failAndHold`: pin back → DB up alone → **validate the copy's completion marker** → **empty-then-load in one transaction** (PostgreSQL: the dump's schemas dropped and recreated inside the load's transaction; MariaDB: every table dropped first, and a failed load HOLDS with a sentence saying the database is in neither state) → full start → **health with the OLD probe** → `undone`, else HOLD. A volume tar at safety-dump time for apps with no database server. Household page + event; the mail rides part 2. | 15; the audit's 8-point list | **4** | — | **HIGH by nature** — it writes the customer's database. Bounded: it only ever loads the copy taken seconds before, validated first, atomically on PostgreSQL; every failure mode ends in today's hold. **It also makes the manual button safer on its own**, which is why it goes first. |
 | **2** | **SHIPPED — controller v0.264.0 + hub v0.120.0, proven live on 9202 and 9201 2026-09-23** (`audits/undo-fleet-2026-09-23/`). **The update sentences in the household's language** (R-606) and a mail when an automatic update is undone or held: events `app_update_undone` (warning) and `app_update_held` (error), one per app per outcome, on by default, mailed in the household's language with the app named in the subject; per-app cooldown on both legs. Leftovers: R-647. | R-606, 15 | **1** | — | none |
 | **3** | **SHIPPED — controller v0.264.0, proven live on 9202 2026-09-23.** **A disabled notifier says so** (R-620), so the mail of part 2 can be measured on a scratch box at all. | R-620 | **0.5** | — | none |
-| **4** | **The test record + the catalog gate + the memory check.** The harness writes the ladder entry (below) from its verdict record, including the memory watch's peak and marks; the gate refuses an image move with no entry, an entry with a `failed` verdict, or one with no memory watch; `CompareImageRefs`' rule moves here as the push-time safety net. **Backfill:** one entry per current pin — the 21 proven moves from their records, every other pin `needs_person: "never tested"`, which is honest and keeps them manual. A version move re-checks `mem_limit` against the watch's peak (the RomM follow-up: gate, not checklist, because the watch now produces the number). | 13, R-635 follow-up | **2.5** | the memory watch (shipped 2026-09-23) | none on a box — catalog-side only |
+| **4** | **SHIPPED — catalog `6db08a5`, night 2026-09-23** (`audits/DRILL-night-2026-09-23.md`): `update_ladder:` in `.felhom.yml`, one JSON entry per line (spiked on controller v0.266.0 and v0.267.0 first — the controller ignores the key); `check-test-record.py` (static, CI) + `check-test-record-move.py` (history + the registry for moved refs only, decision 23); the only writer `upgrade-test.py --write-ladder`; the 21 moves of 2026-09-22 backfilled from their records (21 proven). Not built: `steps/<to>.yml` (part 5 needs it once an app has two steps); `CompareImageRefs` did NOT move to the gate — the gate asks for a proven test instead, which is decision 13's own test. **The test record + the catalog gate + the memory check.** The harness writes the ladder entry (below) from its verdict record, including the memory watch's peak and marks; the gate refuses an image move with no entry, an entry with a `failed` verdict, or one with no memory watch; `CompareImageRefs`' rule moves here as the push-time safety net. **Backfill:** one entry per current pin — the 21 proven moves from their records, every other pin `needs_person: "never tested"`, which is honest and keeps them manual. A version move re-checks `mem_limit` against the watch's peak (the RomM follow-up: gate, not checklist, because the watch now produces the number). | 13, R-635 follow-up | **2.5** | the memory watch (shipped 2026-09-23) | none on a box — catalog-side only |
 | **5** | **The ladder on the box.** Read `update_ladder:` from the clone, find the installed step, apply ONE step with its OWN definition (`steps/<to>.yml`, the last step the current template), repeat next night; a failed step stops the ladder for that app. `CatalogOrder` compares refs with the digest stripped (see part 7). | 14 | **2.5** | 1, 4 | medium — each step is the guarded update + undo; the new risk is rendering the wrong step's definition, pinned by a test per step shape |
-| **6** | **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
+| **6** | **CATALOG HALF SHIPPED — night 2026-09-23:** every ladder entry carries the digest per `to` ref (`scripts/image_digest.py`, stdlib; equals Docker's `RepoDigests` on a box), and the move gate refuses a digest the registry no longer serves. **The box half (compare, render `name:tag@sha256`) is not built.** **Digests.** The catalog records `sha256` per pin at push time (`check-image-resolvable.py` already resolves it); the box compares it for the badge and renders `name:tag@sha256:…` when present. **Measured 2026-09-23 on 9202:** Docker and Compose both pull and run `redis:7-alpine@sha256:858f…`, and refuse a digest that does not exist (`audits/update-rulings-2026-09-23/70-…`). **Build trap, read from source:** `splitImageRef` returns "unorderable" for ANY ref containing `@` (`updateorder.go:134`), so the digest must be split off before ordering or every digest-pinned app reads Unknown. A digest gone upstream fails the PULL — Scenario E, pin back, nothing ran. | 17, R-446 | **2** | 4 (the entry carries the digest) | low |
 | **7** | **The update leg in the chain + the automatic caller + the switch.** A leg that starts when the off-site leg has FINISHED (legs are clock-scheduled today, not chained — a completion signal is new), one app at a time (there is no single-flight, §3b Q4), `app_update.unattended` default ON, `stacks.update_window` removed, reads `UpdateRefusal.Reason`, remembers a failed step so it never re-presses it. **See the one open point below.** | 11, 12 | **3** | 1, 2, 5 | medium — the only part that acts with nobody watching; everything above is what makes it safe |
 | **8** | **SHIPPED — controller v0.265.0 + hub v0.121.0, proven live on 9202 2026-09-23** (`audits/cleanup-2026-09-23/`; a kernel `oom_kill` counter, not the sticky flag — `08` §6.2). **R-636** — the same OOM key re-firing escalates instead of staying one `warning` for six hours. | R-636 | **1** | — | none |
 | **9** | **SHIPPED — controller v0.265.0, proven live on 9202 2026-09-23** (badge „Megállítva — visszaállítás szükséges" / "Stopped — restore needed", no Update button, 409 `held` unchanged). **R-625** — a held app stops offering an Update it will refuse. With the undo, holds become rarer; the lie on the page does not go away by itself. | R-625 | **0.5** | 1 | none |