More apps update themselves at night (30+2 -> 35+3); the same-name security-fix gap measured (R-740, decision for the operator)
gates / gates (push) Successful in 29s

- Twelve steps published on both venues (catalog): first ladders for calibre-web, gitea, wger,
  crafty-controller, uptime-kuma, zipline (two steps); within-major emby, ghost, home-assistant,
  outline, rallly. immich's step v3.0.3 -> v3.2.2 re-proven at 768M (Part D).
- Part C: the night leg skips a digest-only change AND the catalog never records a same-tag re-test,
  so a same-name upstream fix reaches no box. Row R-740; the decision in STATUS; `09` decision 30
  carries a dated note (the decision itself unchanged).
- Rows: 369 -> 377. Opened R-735..R-742; closed R-735, R-738, R-742; narrowed R-462, R-624, R-446,
  R-440, R-734, R-732. The currency audit gains §1b (and corrects its 31+1 to 30+2).

Evidence: documentation/audits/more-night-apps-2026-09-30/. Report: REPORT-more-night-apps-2026-09-30.md.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
adminandClaude Opus 5.5 committed 2026-09-30 20:27:28 +02:00
1 parent 2866f6a318
commit 42bf40bd2c
331 files changed
+25953 -25

No files matched your search

+14 -6
View File
@@ -661,17 +661,17 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
| **R-435** | **The snapshot-drop detector is blind to a single-app deletion — the exact shape the box can produce.** `snapshotDropFraction = 0.5` and `snapshotDropFloor = 5` (`hub/internal/monitor/offsite.go`) require a fall of MORE than half the previous count. demo-hp's baseline is **69** across 9 apps, so ~35 snapshots must go before it speaks; **one app's tag is ~9 and is invisible.** `offbox.go:1388` runs `forget --prune` **grouped by host,tags** — a per-tag wipe is precisely the shape a faulty retention or a targeted deletion produces. **This is deliberate, not accidental:** the constant's own comment argues the insensitivity, and *"a detector that cries wolf is switched off within a fortnight"* is a lesson this project paid for. **So this row is NOT a demand to lower the threshold.** It is a demand that the blind spot be written where the operator reads it, because "an unexplained fall is noticed within a day" (R-431, and `STATUS.md`) is true only of falls above half. **Discovered by arithmetic while planning the drill's Phase 2/3 pairing, which could not have worked:** Phase 2 deletes one app and Phase 3 expects the alarm to fire. **LIMITATION NOW WRITTEN INTO THE ALARM'S OWN DOCUMENTATION, hub v0.111.1** — the comment above `snapshotDropFraction` in `hub/internal/monitor/offsite.go` now states what the detector does NOT see, with the demo-hp arithmetic and the `forget --prune` grouping that makes the blind spot sit on the most likely single-app failure. **It also says explicitly that the numbers must NOT be lowered to "fix" this** and that per-app detection needs a SECOND signal keyed on the per-tag count. **The row stays OPEN because documenting a blind spot is not covering it** — and because `STATUS.md` and R-431 both still say "noticed within a day", which is true only of falls above half. | **OPEN — documented in code v0.111.1; the coverage gap itself is unclosed** |
| **R-436** | **LEAD, NOT A DEFECT — append-only may be reachable without a new machine, which would make R-95's real prevention far cheaper than the spike concluded.** Hetzner's port-23 restricted shell advertises, in its own `help`, these server-side backends: `borg`, `rsync`, `scp`, `sftp`, **`rclone serve restic --stdio`**. And restic 0.14.0 **recognises the `rclone:` backend** — MEASURED 2026-09-01 with a control: `banana:` → `Fatal: parsing repository location failed: invalid backend`, while `rclone:` → `exec: "rclone": executable file not found in $PATH` (i.e. the backend parsed and it tried to run the helper). rclone is **not** in the controller image today. `rclone serve restic` carries an **`--append-only`** flag. **Why this matters:** the R-95 spike's option 3 was priced at a new always-on service in the recovery path plus either a mount in the hot path or migrating every customer's history — and it was deferred on that price. This route needs neither: the server side already runs at the provider. **THE CAVEAT, STATED FIRST because it may kill the idea:** the **client** supplies the server command line, so a compromised guest could simply omit `--append-only` unless the provider pins it. **NOT ESTABLISHED:** whether Hetzner pins the flag or accepts client-supplied arguments. **That is a vendor question and it is cheap** — it should be asked before any code is written, because if the answer is "client-supplied" this lead is worth nothing. **STRENGTHENED 2026-09-01 (operator supplied the page): the backend is OFFICIALLY DOCUMENTED, not merely advertised in a shell banner.** `docs.hetzner.com/storage/storage-box/access/access-ssh-rsync-borg/#restic` reads: *"Restic is natively supported with the SFTP backend. As another option, we support the restic backend, which is provided by Rclone over SSH."* So the transport exists as a supported product feature and the client half is already proven (restic 0.14.0 parses `rclone:`, measured with a control). **AND THE SAME PAGE SETTLES THAT THE DOCS CANNOT ANSWER THE CAVEAT: neither its Rclone nor its Restic section mentions append-only at all.** That is worth stating because it closes the cheapest alternative to asking — nobody need re-read the documentation hoping for it. **Corroboration, unlooked for:** that page's table of port-23 commands matches, item for item, the `help` output measured live on our own sub-account — independent confirmation that the live measurement was reading the right product's surface. **-- ANSWERED BY HETZNER 2026-09-22 (ticket #2026090103040671), and the lead is GOOD.** Asked directly whether `--append-only` is enforced by Hetzner or taken from what the client sends, support replied: *"You can use --append-only and it would look like this: `command="rclone serve restic --stdio --append-only path/to/repo" <public key>`"*, pointing at `fluix.one/blog/hetzner-restic-append-only/`. **The mechanism is an SSH FORCED COMMAND pinned to the key in the Storage Box's `authorized_keys`** - so the flag is enforced on the server side, by a line WE write, and a client asking for a plain `rclone serve restic --stdio` on that key gets the forced one instead. **That is real prevention without a new machine, which is what this lead claimed and could not confirm.** **What it does NOT say, and must not be read as saying:** a key WITHOUT that prefix still gets a deleting server, so this protects a repository only if every key that can reach it carries the forced command - including any key an operator adds later. **NOTHING IS MEASURED YET.** Owed, and cheap: a forced-command key on `u629488`, a restic backup through it that SUCCEEDS, and a `forget --prune` through it that is REFUSED - with the refusal quoted. Until that runs this is a support answer, not a property of our backups. | **OPEN — ask the vendor before building anything** |
| **R-437** | **The register compression sweep is OWED, and it was deliberately NOT run inside the 2026-09-01 beta-line session — this row is the record of that choice, not a note.** **The ask:** compress what has closed in `OPEN-ITEMS.md`. **The measurement, taken before deciding:** 181 rows, 316 KB of row text, of which **12 rows / ~25 KB (about 7 %) carry a CLOSED/DECIDED/ANSWERED leading verdict.** So the sweep buys little and touches everything. **Why it was refused as a side-task, and the citation matters:** a compression sweep is the exact operation that has already gone wrong here. The 2026-08-22 sweep (`ef6ac6f`, R-376..R-378) matched a status word ANYWHERE in the row, moved rows that were not closed, and **R-378 caught six in the same session and missed a seventh** — R-87 sat in the wrong register for nine days while the ranking paragraph pointed at nothing (R-405). **That is a session-scale hazard, and running it as the tail end of a session about something else is how it happened the first time.** **WHAT IS OWED, scoped so it can be picked up cold:** (1) classify by the **LEADING VERDICT** of the state cell only — the rule `closed_register_gate.py` already implements and red-proofs, never a whole-row match; (2) move, never rewrite — a compressed row that loses its evidence is worse than a long one; (3) run `closed_register_gate.py` before and after and quote both; (4) re-read the ranking paragraph afterwards, because that is the surface that silently went stale last time. **Not urgent:** the file is 688 lines and every gate reads it in well under a second. | **OPEN — owed; needs its own session, not a tail end** |
| **R-440** | **[P2-MEDIUM] 23 catalog image pins float, so an update is not reproducible.** `compose pull` on a moving tag fetches whatever upstream published that day. **MEASURED 2026-09-01 over `app-catalog-felhom.eu` @ `29edad9c5bf4`: 79 `image:` lines across 53 apps, 66 distinct; 23 of those lines carry a tag with no patch version.** `postgres:16-alpine` (8 apps), `redis:7-alpine` (6), `mariadb:11.6` (2), plus one each of `postgres:15-alpine`, `postgis/postgis:16-3.5-alpine`, `mariadb:11.4`, `mariadb:12.3`, `ghcr.io/claperco/claper:2.5`, `ghcr.io/thomiceli/opengist:1.13`, `wger/server:2.6`. **A 24th is arguable and is recorded rather than rounded away:** `ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0` pins both extensions exactly but leaves the PostgreSQL patch floating. A customer pressing Frissites can therefore swap their DATABASE ENGINE build with no catalog change and no record; two boxes updated on two days end up different. **Severity MEDIUM on its own; it becomes BLOCKING the moment a pre-update copy exists**, because "what did we upgrade from and to" must be recordable and today it is not — which is also why R-440 must be read next to the digest discipline in Rule 10 of the spike. **MEASURED LIVE 2026-09-01 — the floating pins have ALREADY moved, with a passing control.** Running digests on demo-hp compared against what the registry serves for the same tag today: **`mariadb:11.4` MOVED** (`sha256:4f1d8d20...` -> `sha256:611a2fcc...`) and **`mariadb:12.3` MOVED** (`sha256:a02fe89c...` -> `sha256:dd9b303a...`), while `postgres:16-alpine`, `redis:7-alpine`, `mariadb:11.6` and `opengist:1.13` were SAME — **and both fully-pinned CONTROLS (`rommapp/romm:5.0.0`, `privatebin/pdo:2.0.5`) were SAME.** So on a box with ZERO visible drift by tag, pressing Frissites today silently swaps the DATABASE ENGINE build under `romm` and `bookstack`, with no catalog change and no record. **Compounding fact found while reading:** the recovery unit records `ImagePins` but the manifest comment says *"image NOT stored - re-pulled on restore"*, so a RESTORE of a floating-pinned app also re-pulls whatever is current — the same non-reproducibility on the recovery path. **HALF OF THE ANSWER SHIPPED 2026-09-02 (controller v0.233.0, slice 1): `app.yaml.installed_images` now records, per compose SERVICE, the reference AND the repo digest each container was actually created from — so "what did we upgrade FROM" is answerable on any box that has taken one lifecycle action since the upgrade.** What is still missing is the other half: comparing that digest against what the registry serves for the same tag TODAY, which needs a network call the render path deliberately does not make (see R-446). **The row therefore stays OPEN and its rank is unchanged** — recording a digest does not make a floating pin reproducible; it makes the drift measurable after the fact. `audits/SPIKE-app-update-2026-09-01.md` **— NIGHT 2026-09-23:** every image moved tonight (and the 21 backfilled moves) carries its resolved digest in the catalog's test record; the floating pins themselves still float until the box pulls by digest (`09` §6.4 part 6). **-- MEASURED 2026-09-30 on demo-hp 9201: a fresh install or a guarded Update renders `name:tag@sha256` from the ladder (`digest.go` `RenderWithLadderDigests`) — 9 of 20 services there run such a definition (adventurelog, docmost, paperless-ngx). The other 11 run TAG-ONLY definitions: bookstack, kimai, opengist, privatebin, romm were installed before v0.269.0 and not updated since, and `CarryDigests` (`digest.go:141`) keeps only a digest the running definition already has; bentopdf and calibre-web have no ladder at all. So a re-pull of those (a restore) takes whatever the tag serves that day. `audits/pg-last-six-2026-09-30/E/E2-digests-demo-hp.txt`.** | **NARROWED 2026-09-30 — floating only for (1) apps with no ladder entry and (2) apps installed before v0.269.0 until their next guarded Update; owner: CC.** |
| **R-440** | **[P2-MEDIUM] 23 catalog image pins float, so an update is not reproducible.** `compose pull` on a moving tag fetches whatever upstream published that day. **MEASURED 2026-09-01 over `app-catalog-felhom.eu` @ `29edad9c5bf4`: 79 `image:` lines across 53 apps, 66 distinct; 23 of those lines carry a tag with no patch version.** `postgres:16-alpine` (8 apps), `redis:7-alpine` (6), `mariadb:11.6` (2), plus one each of `postgres:15-alpine`, `postgis/postgis:16-3.5-alpine`, `mariadb:11.4`, `mariadb:12.3`, `ghcr.io/claperco/claper:2.5`, `ghcr.io/thomiceli/opengist:1.13`, `wger/server:2.6`. **A 24th is arguable and is recorded rather than rounded away:** `ghcr.io/immich-app/postgres:16-vectorchord0.4.3-pgvectors0.2.0` pins both extensions exactly but leaves the PostgreSQL patch floating. A customer pressing Frissites can therefore swap their DATABASE ENGINE build with no catalog change and no record; two boxes updated on two days end up different. **Severity MEDIUM on its own; it becomes BLOCKING the moment a pre-update copy exists**, because "what did we upgrade from and to" must be recordable and today it is not — which is also why R-440 must be read next to the digest discipline in Rule 10 of the spike. **MEASURED LIVE 2026-09-01 — the floating pins have ALREADY moved, with a passing control.** Running digests on demo-hp compared against what the registry serves for the same tag today: **`mariadb:11.4` MOVED** (`sha256:4f1d8d20...` -> `sha256:611a2fcc...`) and **`mariadb:12.3` MOVED** (`sha256:a02fe89c...` -> `sha256:dd9b303a...`), while `postgres:16-alpine`, `redis:7-alpine`, `mariadb:11.6` and `opengist:1.13` were SAME — **and both fully-pinned CONTROLS (`rommapp/romm:5.0.0`, `privatebin/pdo:2.0.5`) were SAME.** So on a box with ZERO visible drift by tag, pressing Frissites today silently swaps the DATABASE ENGINE build under `romm` and `bookstack`, with no catalog change and no record. **Compounding fact found while reading:** the recovery unit records `ImagePins` but the manifest comment says *"image NOT stored - re-pulled on restore"*, so a RESTORE of a floating-pinned app also re-pulls whatever is current — the same non-reproducibility on the recovery path. **HALF OF THE ANSWER SHIPPED 2026-09-02 (controller v0.233.0, slice 1): `app.yaml.installed_images` now records, per compose SERVICE, the reference AND the repo digest each container was actually created from — so "what did we upgrade FROM" is answerable on any box that has taken one lifecycle action since the upgrade.** What is still missing is the other half: comparing that digest against what the registry serves for the same tag TODAY, which needs a network call the render path deliberately does not make (see R-446). **The row therefore stays OPEN and its rank is unchanged** — recording a digest does not make a floating pin reproducible; it makes the drift measurable after the fact. `audits/SPIKE-app-update-2026-09-01.md` **— NIGHT 2026-09-23:** every image moved tonight (and the 21 backfilled moves) carries its resolved digest in the catalog's test record; the floating pins themselves still float until the box pulls by digest (`09` §6.4 part 6). **-- MEASURED 2026-09-30 on demo-hp 9201: a fresh install or a guarded Update renders `name:tag@sha256` from the ladder (`digest.go` `RenderWithLadderDigests`) — 9 of 20 services there run such a definition (adventurelog, docmost, paperless-ngx). The other 11 run TAG-ONLY definitions: bookstack, kimai, opengist, privatebin, romm were installed before v0.269.0 and not updated since, and `CarryDigests` (`digest.go:141`) keeps only a digest the running definition already has; bentopdf and calibre-web have no ladder at all. So a re-pull of those (a restore) takes whatever the tag serves that day. `audits/pg-last-six-2026-09-30/E/E2-digests-demo-hp.txt`.** **-- 2026-09-30 (evening):** calibre-web now has a ladder entry (`53a4a1d`), so a new install or a guarded Update renders its tested digest; bentopdf still has none. `audits/more-night-apps-2026-09-30/` | **NARROWED 2026-09-30 — floating only for (1) apps with no ladder entry and (2) apps installed before v0.269.0 until their next guarded Update; owner: CC.** |
| **R-444** | **[P3-LOW] Nothing runs `pct fstrim` on the fleet, and demo-hp's thin pool was carrying ~23.8 GB of blocks the guest had already freed.** MEASURED 2026-09-01 during this spike's teardown: the run itself added ~1.05 GiB that `local-lvm` did not reclaim on delete (68.97% -> 70.91%); `fstrim` INSIDE the unprivileged container is refused (`FITRIM ioctl failed: Operation not permitted`, all three mounts); `pct fstrim 9201` from the PVE host then trimmed **30.2 GiB + 57 GiB** and took `local-lvm` to **26.78%** — **23.8 GB BELOW this run's own starting point**, i.e. the surplus was long-standing, not ours. **Why it is not merely housekeeping:** a thin pool that only ever grows can reach 100% from DELETED data alone, and a full thin pool takes every guest on the host read-only. demo-hp had 16.4 GB free before the trim. **Not urgent, and the row says so** — but the appliance has no periodic trim and no operator surface reports the gap between guest-free and pool-used. Owner: **CC.** `audits/SPIKE-app-update-2026-09-01.md` | **OPEN — rank P3-LOW; owner: CC** |
| **R-445** | **[P3-LOW] Hub app telemetry survives the app's removal, so a 15-minute throwaway now sets a FLEET-WIDE memory recommendation.** MEASURED 2026-09-01: this spike's Phase 6 Nextcloud existed for ~15 minutes on demo-hp, spent part of it crash-looping, and was then removed with all volumes. The hub's `/apps/nextcloud` page still reports `Deployments`, `Avg Memory 208 MB`, `P95 Memory 280 MB` and **`Suggested Limit (P95x1.2) = 352 MB`**, plus three MariaDB `io_uring` rows under Known Issues attributed to demo-hp. **The suggested limit is an operator-facing recommendation derived from a sample that no longer exists anywhere** — and Nextcloud is a real catalog app whose limit someone may act on. **RETAINED DELIBERATELY BY THIS RUN, NOT CLEARED, and the reason is part of the row:** the hub offers `POST /apps/nextcloud/reset-telemetry` whose own confirm reads *"Delete all telemetry data for nextcloud? This cannot be undone."* — an irreversible write on the operator's surface, and the operator authorised Phase 6, not this. **The one-line command is recorded in the audit doc so it is a decision, not a task.** The general question is the row: should telemetry for an app with zero live deployments age out, or be excluded from the suggestion? Owner: **VIKTOR rules, CC implements.** `audits/SPIKE-app-update-2026-09-01.md` | **OPEN — rank P3-LOW; owner: VIKTOR rules, CC implements** |
| **R-446** | **[P2-MEDIUM] „Naprakész" can be FALSE, and the badge that says it cannot tell.** Slice 2 (controller v0.233.0, 2026-09-02) compares the RECORDED image reference per compose service against the reference the current template pins, and **queries no registry** — deliberately: a customer's box must not depend on reaching eight upstream registries to render a page (`felhom-controller/controller/internal/web/updatebadge.go`, `compareInstalledToTemplate`). **For the 23 floating pins that comparison is blind by construction:** `postgres:16-alpine`, `mariadb:11.6` and 21 others can carry an identical reference over an image that has moved. **MEASURED, not theorised — spike §5 found `mariadb:11.4` and `mariadb:12.3` had BOTH already moved upstream while two fully-pinned CONTROLS held.** So `romm` and `bookstack` on demo-hp would read „Naprakész" over a database engine build that is not the one the catalog now resolves to. **This is a KNOWN LIMITATION OF A SHIPPED FEATURE, filed the same session rather than left implicit**, and it is stated in the same words in `architecture/09-update-architecture.md` §8.1 and in the controller's `README.md`. The close is a digest comparison against the registry, which needs a network call, a cache and a failure posture — it is not a one-liner and it is not slice 2's job. **Depends on R-440**, whose fix (stop floating) would remove the problem instead of measuring it — take that route first if it is available. `architecture/09-update-architecture.md` **MEASURED 2026-09-21, and the blind spot is not one or two pins.** `audits/UPDATE-ARC-STATE-2026-09-21.md` §3.3: the catalog carries **10 floating pins of 66** (recounted — the old "23" was stale), and **6 of the 7 measurable engine pins have been repushed upstream since the catalog set them** — `postgres:16-alpine` (8 apps), `postgres:15-alpine`, `redis:7-alpine` (6 apps), `mariadb:11.4`, `mariadb:12.3`, `postgis:16-3.5-alpine`; only `mariadb:11.6` has not. The 8th (immich's own ghcr build) is UNMEASURED — ghcr exposes no anonymous last-modified timestamp. **So on demo-hp today four apps read „Naprakész" over a database engine image that has demonstrably moved.** The fix does NOT need the box to query a registry: the catalog can record each pin's digest at push time (`check-image-resolvable.py` already resolves it) and the box compares digests. Put to the operator as `09` §3b **Q6**, recommended YES — the cheapest real improvement on the arc's list. **— UPDATE NIGHT 2026-09-21:** **MEASURED ON A BOX 2026-09-21 (update night, leg B8), and it REFINES the row in two ways rather than merely confirming it.** §8.1's numbers came from a registry sweep on DooPlex; this is the same question asked of a customer-shaped box, where the badge actually renders. On guest 9202, `docmost`'s two floating pins were read as `installed_images` records them and compared against the upstream digests measured the same night: `postgres:16-alpine` → **`sha256:721873c34ceb9…` on the box and `sha256:721873c34ceb9…` upstream**, and `redis:7-alpine` → **`sha256:858f009f9709c…` both sides**. **Identical. So the badge „Naprakész" is TRUE for this box**, and the app reads correctly. **(1) The defect's size is set by INSTALL AGE, not by the catalog.** A floating pin is wrong only for a box that pulled BEFORE the tag moved; a box deployed after the repush holds the current image and its badge is right. R-446's "six repushed pins" measured the tag against the date the CATALOG set it, which is the right measure for *the catalog* and not for *a box*. **(2) The producer Q6 needs ALREADY EXISTS on the box.** `installed_images` records a real `digest` per service (`installed.go` §7.1) — the box knows exactly what it is running. What it cannot do is COMPARE, because the catalog carries no digest to compare against. That is Q6's proposal, and this is a concrete confirmation that only the catalog half is missing. Evidence: `audits/update-night-2026-09-21/23-B8-floating-pin.txt`. **-- RULED 2026-09-23 (`09` §3 decision 17):** YES — the catalog records the image digest of every pin at push time; the box compares against it and, where the catalog carries one, pulls **that exact image**, which makes a floating tag reproducible, not only the badge honest. *Pull-by-digest while the definition names a tag is a claim to verify in the build, not a ruling on mechanism.* **-- 2026-09-23:** pull-by-digest MEASURED on 9202 — Docker and Compose both pull and run `redis:7-alpine@sha256:858f…` and refuse a digest that does not exist. **Build trap, read from source:** `splitImageRef` returns "unorderable" for any ref containing `@` (`stacks/updateorder.go:134`), so a digest-carrying pin must have its digest split off before ordering or every such app reads Unknown. `09` §6.4 part 6. **— NIGHT 2026-09-23:** the CATALOG half of the close shipped: every ladder entry records the digest the registry served for each `to` ref (`scrLine truncated
| **R-446** | **[P2-MEDIUM] „Naprakész" can be FALSE, and the badge that says it cannot tell.** Slice 2 (controller v0.233.0, 2026-09-02) compares the RECORDED image reference per compose service against the reference the current template pins, and **queries no registry** — deliberately: a customer's box must not depend on reaching eight upstream registries to render a page (`felhom-controller/controller/internal/web/updatebadge.go`, `compareInstalledToTemplate`). **For the 23 floating pins that comparison is blind by construction:** `postgres:16-alpine`, `mariadb:11.6` and 21 others can carry an identical reference over an image that has moved. **MEASURED, not theorised — spike §5 found `mariadb:11.4` and `mariadb:12.3` had BOTH already moved upstream while two fully-pinned CONTROLS held.** So `romm` and `bookstack` on demo-hp would read „Naprakész" over a database engine build that is not the one the catalog now resolves to. **This is a KNOWN LIMITATION OF A SHIPPED FEATURE, filed the same session rather than left implicit**, and it is stated in the same words in `architecture/09-update-architecture.md` §8.1 and in the controller's `README.md`. The close is a digest comparison against the registry, which needs a network call, a cache and a failure posture — it is not a one-liner and it is not slice 2's job. **Depends on R-440**, whose fix (stop floating) would remove the problem instead of measuring it — take that route first if it is available. `architecture/09-update-architecture.md` **MEASURED 2026-09-21, and the blind spot is not one or two pins.** `audits/UPDATE-ARC-STATE-2026-09-21.md` §3.3: the catalog carries **10 floating pins of 66** (recounted — the old "23" was stale), and **6 of the 7 measurable engine pins have been repushed upstream since the catalog set them** — `postgres:16-alpine` (8 apps), `postgres:15-alpine`, `redis:7-alpine` (6 apps), `mariadb:11.4`, `mariadb:12.3`, `postgis:16-3.5-alpine`; only `mariadb:11.6` has not. The 8th (immich's own ghcr build) is UNMEASURED — ghcr exposes no anonymous last-modified timestamp. **So on demo-hp today four apps read „Naprakész" over a database engine image that has demonstrably moved.** The fix does NOT need the box to query a registry: the catalog can record each pin's digest at push time (`check-image-resolvable.py` already resolves it) and the box compares digests. Put to the operator as `09` §3b **Q6**, recommended YES — the cheapest real improvement on the arc's list. **— UPDATE NIGHT 2026-09-21:** **MEASURED ON A BOX 2026-09-21 (update night, leg B8), and it REFINES the row in two ways rather than merely confirming it.** §8.1's numbers came from a registry sweep on DooPlex; this is the same question asked of a customer-shaped box, where the badge actually renders. On guest 9202, `docmost`'s two floating pins were read as `installed_images` records them and compared against the upstream digests measured the same night: `postgres:16-alpine` → **`sha256:721873c34ceb9…` on the box and `sha256:721873c34ceb9…` upstream**, and `redis:7-alpine` → **`sha256:858f009f9709c…` both sides**. **Identical. So the badge „Naprakész" is TRUE for this box**, and the app reads correctly. **(1) The defect's size is set by INSTALL AGE, not by the catalog.** A floating pin is wrong only for a box that pulled BEFORE the tag moved; a box deployed after the repush holds the current image and its badge is right. R-446's "six repushed pins" measured the tag against the date the CATALOG set it, which is the right measure for *the catalog* and not for *a box*. **(2) The producer Q6 needs ALREADY EXISTS on the box.** `installed_images` records a real `digest` per service (`installed.go` §7.1) — the box knows exactly what it is running. What it cannot do is COMPARE, because the catalog carries no digest to compare against. That is Q6's proposal, and this is a concrete confirmation that only the catalog half is missing. Evidence: `audits/update-night-2026-09-21/23-B8-floating-pin.txt`. **-- RULED 2026-09-23 (`09` §3 decision 17):** YES — the catalog records the image digest of every pin at push time; the box compares against it and, where the catalog carries one, pulls **that exact image**, which makes a floating tag reproducible, not only the badge honest. *Pull-by-digest while the definition names a tag is a claim to verify in the build, not a ruling on mechanism.* **-- 2026-09-23:** pull-by-digest MEASURED on 9202 — Docker and Compose both pull and run `redis:7-alpine@sha256:858f…` and refuse a digest that does not exist. **Build trap, read from source:** `splitImageRef` returns "unorderable" for any ref containing `@` (`stacks/updateorder.go:134`), so a digest-carrying pin must have its digest split off before ordering or every such app reads Unknown. `09` §6.4 part 6. **— NIGHT 2026-09-23:** the CATALOG half of the close shipped: every ladder entry records the digest the registry served for each `to` ref (`scrLine truncated
| **R-450** | **[P2-MEDIUM] UPDATE ARC SLICE 6 — a version sequence: automatic WITHIN a major, never ACROSS one, and an engine change gets its OWN edge.** The first half is an operator ruling of 2026-09-02 and its justification is R-449's measurement: a cross-major jump can be refused by the app itself and cannot be undone. **The second half is a rule recorded now, while it is cheap:** an engine change must never be bundled with an app version bump. `bookstack`'s `0b73e5e` moved the application 25.02.2 → 26.05.2 **and** MariaDB 11.6 → 12.3 in one commit — **two migrations behind one edge**, and an unreadable failure when it breaks. Needs a catalog-side convention and, eventually, a gate. `architecture/09-update-architecture.md` §6 **HALF SHIPPED 2026-09-21 (catalog `5ff36d098cbc`): the second half — an engine change gets its OWN edge — is now ENFORCED** by `check-engine-major.py`, which refuses a commit moving a MariaDB major together with any other image move in that template, naming what it was bundled with. The FIRST half (automatic within a major) is Slice 6 and needs four operator answers — `09` §3b **Q1–Q4**, with the shape it would take in `09` §6.2. **The urgency is now measured:** 46 of the catalog's 58 exact pins are behind upstream and **39 of those are within a major** — the population the 2026-09-02 ruling already says may move without a human. **-- RULED 2026-09-23 (operator, `09` §3 decisions 11–15):** Q1–Q4 answered. The update is a leg of the backup chain after off-site and before the full-system backup (11); automatic with a per-box switch ON by default (12); **the TEST decides, not the tag** — the box applies every step the catalog holds because the catalog holds only tested steps, and `CompareImageRefs` moves to the catalog gate (13, REPLACES "never across a major"); a box behind climbs **one tested step at a time** (14); **the box UNDOES a failed update itself** — old definition + the pre-pin safety dump + health check again, HOLD only if the undo fails (15, REPLACES §6.1's no-auto-undo). The undo and the ladder were SPIKED the same day before any build (`audits/update-rulings-2026-09-23/`); build order and costs in `09` §6.4. **-- SPIKED 2026-09-23 (`audits/update-rulings-2026-09-23/`):** the undo works by hand on three real migrating edges and needs eight product additions (R-637..R-642); **the ladder is measured absent** — one press on a box two steps behind jumped vikunja 2.3.0 → 2.5.0 in 9.5 s and 2.4.0 never ran, and the box cannot see intermediate steps at all because its catalog clone is `--depth 1` (`sync.go:283`/`:300`, one commit visible on both demo guests). The ladder's recommended format is an `update_ladder:` list in `.felhom.yml` with each intermediate step's own definition, NOT the git history (romm's image-moving commit is the definition that OOM-looped). The chain's update leg has ≤15 min as ruled (R-643). Build order `09` §6.4. **— NIGHT 2026-09-23: `09` §6.4 part 4 SHIPPED** (catalog `6db08a5`): the test record `update_ladder:` + two gates + the only writer + the 21-move backfill; 12 more steps published with records. Parts 5 (the box climbs), 6-box-half and 7 remain; the romm press on demo-hp showed today's jump live — 5.3.0 → 5.3.1 AND mariadb 11.4 → 11.8 in one press (both tested steps; `done`). **— 2026-09-24: `09` §6.4 PART 5 SHIPPED** (controller v0.268.0 `206b035`, catalog `5ed599c`): one press = one tested step, each step's own definition at `templates/<app>/steps/<StepKey(to)>.yml`, proven live on 9202 (romm 5.3.0/11.4 → 5.3.1/11.4 → 5.3.1/11.8 in two presses, `audits/ladder-2026-09-24/partD/`). Left in this row: part 6's box half, part 7 (the automatic leg), part 10 (PostgreSQL majors). **— 2026-09-25 night: part 6's box half shipped in v0.269.0/v0.269.1; PART 7 SHIPPED as controller v0.271.0** (the automatic leg, proven over six simulated nights on 9202 and watched through the demo boxes' first real night, `audits/DRILL-night-2026-09-25.md`). **Left: part 10 only (PostgreSQL majors, R-463).** **-- 2026-09-25 (evening): part 10 SHIPPED for docmost** (controller v0.273.0, catalog `afd3a60`, decisions 35–39; `audits/night-2026-09-26/`). What stays open: each of the other ten PostgreSQL apps needs its own two-venue proof before the gate lets its major move. | **NARROWED — the other ten PostgreSQL apps; owner: CC** |
| **R-451** | **[P3-LOW] UPDATE ARC SLICE 7 — a fleet sweep: the operator can SEE, and MOVE, how far behind every box is.** Slices 1 and 2 make one box's state visible on that box's own pages. The operator has no fleet view, and **it is not derivable from what is already reported: the hub's report payload carries container name, state, CPU and memory, and NO image field at all** (spike §5, which is why Peti's box could only be recorded UNKNOWN). So this is a hub-side change as well as a controller one. Rank LOW today because the fleet is two enrolled boxes; it rises with the fleet. `architecture/09-update-architecture.md` §6, §8.4 **BOTH SIDES VERIFIED 2026-09-21, and it is cheaper than this row implies.** The controller's payload carries no image (`internal/report/types.go` L98–103) and the hub's `Store.SaveReport` (`hub/internal/store/store.go:965`) denormalises only container **counts** — but **the hub stores the raw report JSON whole**, so a new controller field lands there the day it is sent. What is missing is the denormalisation and the page, not the transport. Shape in `09` §6.2–6.3; the payload question is `09` §3b **Q7**. **-- RULED 2026-09-23 (`09` §3 decision 18):** the report carries, per compose service (database included), the installed reference, the catalog reference and the badge state. **Built later, when the fleet grows** — Q7's recommendation, confirmed. | **RULED — build deferred until the fleet grows; owner: CC** |
| **R-454** | **[P3-LOW] Five `internal/web` test files have been `gofmt`-unclean for an unknown length of time, and nothing notices.** MEASURED 2026-09-02: `gofmt -l controller/internal/web/` reports `backups_split_test.go`, `claim_code_naming_test.go`, `disk_health_test.go`, `r400_debug_routes_test.go`, `recovery_test.go` — at the **baseline** commit `960d29b0612c`, i.e. not introduced by v0.233.0 (both files added that day are clean). **`go vet` does not check formatting and `controller_gates.py` has no formatting gate**, so the only thing that would ever surface this is someone running `gofmt -l` by hand, which is how it was found. **Not reformatted in the same session, deliberately** — the minimal-changes rule, and a five-file whitespace commit inside a feature release makes that release's diff unreadable. **Small, and the cost of NOT having the instrument is the row:** the count can only grow, and every future `gofmt -l` run produces noise that hides a real one. Fix is two lines: a `gofmt -l` gate in `controller_gates.py` plus one formatting commit, in that order (the gate first, so the commit is provably complete). Owner: **CC.** | **READY — rank P3-LOW; owner: CC** |
| **R-457** | **[P3-LOW] A test that hardcodes a date AND asserts an age derived from it is green on the day it is written and red the next morning — one instance PROVEN, six candidate files named.** MEASURED 2026-09-03: `TestGroupD_BadgeRendersOnBothSurfaces` (shipped the previous day in v0.233.0) pinned a fixture `catalog_since: "2026-07-18"` and asserted the rendered string `"Frissítés elérhető — 46 napja"`. **The pure badge tests inject a clock; the RENDER test does not and cannot** — it goes through the production templates, which call the funcmap entry `updateBadge`, which reads `time.Now()`. The suite was green on 2026-09-02 and **FAILED on 2026-09-03** with *"the behind badge is missing"* on both surfaces, because the true answer had become 47. **Fixed by DERIVING the fixture** — `catalog_since` is computed as *today minus 46 days*, so the test asserts the real number through the real clock and cannot rot. **THE CLASS, which is why this is a row and not just a fix:** a clock-reading test that also carries a date LITERAL is a bomb with a fuse of unknown length, and the suite being green is not evidence it is defused — it is evidence the fuse has not burned down yet. **NAMED AS UNCHECKED CANDIDATES, NOT ACCUSED** — six other test files contain both a `20xx-xx-xx` literal and `time.Now()`: `internal/backup/offbox_test.go`, `internal/web/handler_export_upload_test.go`, `internal/web/r103_tier2_action_test.go`, `internal/web/dashboard_backup_card_test.go`, `internal/web/async_restore_test.go`, `internal/stacks/installed_test.go`. Mixing the two is not itself a defect — it is one only where a literal feeds an assertion evaluated against the real clock — so each needs reading, which is a sweep and not this session. **The instrument that would end the class:** run the suite once under a faked future date in CI and see what turns red. Owner: **CC.** `felhom-controller` v0.234.0 CHANGELOG | **READY — rank P3-LOW; owner: CC** |
| **R-458** | **[P3-LOW] `.felhom.yml` keeps flowing to an app whose compose file is FROZEN, so a frozen app can receive a health check written for a version it is not running.** The v0.235.0 render freezes `docker-compose.yml` for a pinned app once the catalog moves past its version, but copies `.felhom.yml` **verbatim in every case** (`Syncer.copyTemplates`). **The asymmetry is deliberate and both directions were considered:** `.felhom.yml` carries no image, and it carries `catalog_since` — the single input the update badge uses to say *„Frissítés elérhető — N napja"* — so freezing it would silently withhold the one number that tells a customer they are behind, i.e. it would break slice 2 to protect slice 3. **What it costs:** the file also carries the controller-side `healthcheck:` block and resource hints, so a template updated for a newer version can hand a frozen app a probe written for software it is not running. **THE FAILURE DIRECTION IS A FALSE ALARM, NEVER DATA LOSS** — the app keeps running; at worst it renders as degraded and, if it persisted, could reach the dead-app alarm path. That is the same class as R-330's false e-mails, which is why this is a row and not a footnote. **Not fixed now, and the reason is that the cheap fix is wrong:** freezing the whole file breaks the badge, and freezing only the `healthcheck:` key means the syncer would have to parse and re-assemble a customer-facing metadata file — new surface on the one path that touches every app on every box every 15 minutes. **What would settle it:** whether any catalog `healthcheck:` has ever been changed in the same commit as an `image:` line (measurable from the catalog's own history, no box needed). If the answer is "never", the exposure is theoretical and the row can be closed by measurement instead of by code. Owner: **CC.** `architecture/09-update-architecture.md` §5.4, §8.5 **— UPDATE NIGHT 2026-09-21:** **MEASURED 2026-09-21 (update night), leg B9, and the row's risk is NARROWER than it states.** A `.felhom.yml`-only change (a health check for a path only a newer version would serve) was pushed to a FROZEN `bentopdf` — installed `v2.8.6`, catalog ahead. §5.4's asymmetry is confirmed live: the new `.felhom.yml` reached the box while the compose `image:` line stayed `v2.8.6`. **But no false alarm was produced**: ten samples over two minutes all read `state=running` with the front door at `200`. The reason is the probe's own semantics, not luck — `healthprobe.go:258-261` treats **any response** as healthy for `type: http`, and the bogus path answers 404, which is a response. **So this row's false-alarm risk exists only for `type: api` probes carrying an `expect` block**, where the status is compared; for every `type: http` template and every `type: api` without `expect`, a newer version's path is invisible to the probe. The row's actual claim — the failure direction is a false alarm, never data loss — stands and is now measured. Evidence: `audits/update-night-2026-09-21/21-B9-frozen-app-newer-felhomyml.md`. | **READY — rank P3-LOW; owner: CC** |
| **R-460** | **[P3-LOW] BookStack's FILE half cannot be seeded or verified without a browser, so its upgrades can only ever be auto-proven for the DATABASE.** MEASURED 2026-09-06 while building the R-449 harness. BookStack's API needs a token that is only mintable through its web UI, and its HTTP login is unusable headlessly for a second, independent reason: `APP_URL` comes from the template as `https://${SUBDOMAIN}.${DOMAIN}`, so the app marks its session and XSRF cookies **`secure`**; curl over plain http stores neither and **every login POST returns 419 Page Expired**, which looks exactly like a wrong password. The container serves no TLS. **The database half IS provable** — the harness seeds with `php artisan bookstack:create-admin` and reads back with a DIFFERENT artisan command that must find the record, carrying its own negative control on every call. **What is unprovable is an uploaded image or attachment**, i.e. exactly the half a customer would notice. **THIS IS A FACT ABOUT THE APP, NOT A DEFECT IN THE HARNESS**, and it is recorded because Slice 6 needs to know which apps can be auto-verified and which can only be partly verified — nobody had that list before. **Deliberately NOT worked around:** planting a file in the volume would make the test pass while proving nothing, which is R-156's exact failure. **What would remove it:** a headless token route (upstream), or accepting a browser-driven step for this app alone, which DooPlex cannot run. Owner: **CC.** `audits/SPIKE-upgrade-test-2026-09-06.md` §6 **-- UPDATE NIGHT 2026-09-21:** bookstack's edge was walked again on 2026-09-21 and is again **half-proven**: the database half read back through `php artisan` with its own negative control, the file half untouched. The limitation is unchanged and is now measured on the box as well as on the harness. Two more apps joined the same class tonight for a different reason (R-624). | **READY — rank P3-LOW; owner: CC** |
| **R-462** | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.** **The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.** `audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. **— NIGHT 2026-09-23:** the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through `upgrade_boxport.py` (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (`/api/v2/users/me`) and nextcloud (wait for `occ status`). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (`audits/DRILL-night-2026-09-23.md` Part C). **-- 2026-09-30 (day brief): three new front-door fixtures (sparkyfitness, rallly, outline; catalog `e6f3ec2`) and five apps moved on both venues — the three PostgreSQL conversions (R-463) plus bookstack 26.05.5 → 26.09.1 and kimai apache-2.57.0 → 2.67.0 (upstream dropped the `apache-` prefix; the plain tag's digest equals `apache`'s, measured). The catLine truncated
| **R-462** | **[P2-MEDIUM] Widen the upgrade harness beyond three apps — and the cost is dominated by FIXTURES, not by machine time.** The R-449 harness works and is proven by a red negative control (`audits/SPIKE-upgrade-test-2026-09-06.md` §1). **Costed with this run's REAL numbers rather than an estimate:** a successful edge takes **6.4 s – 305.1 s, median 71.8 s**; a FAILING edge takes **556 s**, roughly **8×**, because a negative is only honest if it waits out the full settle window; 3 apps / 11 images cost **5.07 GB**, so 53 apps naively extrapolate to **~90 GB** and, at the median, about an hour of harness time for one edge each. **THAT EXTRAPOLATION UNDERSTATES THE REAL COST BY AN ORDER OF MAGNITUDE, and that is the point of this row.** Two of the three apps needed a bespoke non-browser seed route; one needed two attempts and a discarded approach; one (bookstack) can only ever be half-proven (R-460). **Fixture time scales with apps and does not amortise.** **The decision this row is really asking for is scope, not schedule:** all 53, or only the apps a customer would lose data from, or only apps whose catalog transition is a MAJOR. **Recommended shape, NOT a design — the operator picks:** start with the apps that carry a database, because §3 measured that the abort question only ever bites there. Owner: **VIKTOR rules on scope, CC implements.** `audits/SPIKE-upgrade-test-2026-09-06.md` §5 **ROW CORRECTED 2026-09-21: the scope is NOT open and this row said it was.** It read *"VIKTOR rules on scope"*; the operator ruled on 2026-09-13 (`09` §3 decision 6) that the upgrade test goes to **ALL** apps through the nightly rotation, explicitly *not* "database apps first". What is open is the WORK, not the scope. An ORDER inside that ruling — the 15 database services first, because that is where a wrong answer costs data rather than uptime — is costed as a drill brief in `09` §6.4: legs A–E ≈ **21–34 CC-hours** plus ~25–30 GB on a scratch host, with legs C (a PostgreSQL `pg_upgrade` rehearsal) and E (one automatic night on a throwaway) the two that unblock a decision. **-- UPDATE NIGHT 2026-09-21:** **The count moved from 3 apps to 21 EDGES ACROSS 19 APPS.** The update night walked real within-a-major upstream edges on scratch guest 9202 through the product's own guarded Update, each app seeded and read back through its OWN front door with a negative control on every readback: **14 proven, 3 failed, 4 inconclusive**. Proven: actualbudget, audiobookshelf, bookstack, docmost, grafana, home-assistant, mealie, n8n, navidrome, papra, privatebin, romm, vikunja, and nextcloud's MariaDB engine major. **Ten of the fourteen printed a verbatim migration line**, so the database really was rewritten and the data still read back. Box-side fixtures for 20 apps now exist at `audits/update-night-2026-09-21/fixtures.py`, and four (actualbudget, navidrome, audiobookshelf, vikunja) are ported into `app-catalog-felhom.eu/scripts/upgrade_fixtures.py` with seven new EDGES (U1-U7) so the same edges can be run on the harness venue **with their ABORT step**, which the box deliberately does not offer. **OWED, stated so it is not mistaken for done:** the U1-U7 harness RUNS (the code is in, the runs are not), and fixtures for the four inconclusive apps, of which two (vaultwarden, zipline) cannot be seeded at all while the catalog rightly closes their sign-up (see R-624). **-- 2026-09-23, the RomM lesson is IN THE HARNESS:** `upgrade-test.py` v2 runs every edge that read back under light load for `--soak` seconds (default 600) and reads the kernel's `oom_kill` counter host-side; a kill or restart turns `proven` into `failed`, a peak over 80 % of the limit adds `memory_tight`. **Red-proof:** romm 5.0.0 → 5.3.0 on the template AS PROMOTED (512M, four workers) — seeded, migrated, read back, and then **OOM-killed at +76 s** under four light callers, verdict `failed` (Docker's own OOMKilled read true here; restarts stayed 0, which is why the walk never saw it). Ten minutes is ample for this failure; demo-hp's first kill came at two hours only because nothing was loading it. Evidence `audits/update-rulings-2026-09-23/harness/`. **— NIGHT 2026-09-23:** the bench and the box now share ONE fixture set — the box walk's fixtures run on the bench through `upgrade_boxport.py` (ported verbatim into the catalog), plus a new wishlist fixture and fixes for opengist 1.15, komga (`/api/v2/users/me`) and nextcloud (wait for `occ status`). 14 apps / 15 edges tried; 12 proven on both venues and published with their test records (`audits/DRILL-night-2026-09-23.md` Part C). **-- 2026-09-30 (day brief): three new front-door fixtures (sparkyfitness, rallly, outline; catalog `e6f3ec2`) and five apps moved on both venues — the three PostgreSQL conversions (R-463) plus bookstack 26.05.5 → 26.09.1 and kimai apache-2.57.0 → 2.67.0 (upstream dropped the `apache-` prefix; the plain tag's digest equals `apache`'s, measured). The catLine truncated
| **R-463** | **[P2-MEDIUM] The day the catalog moves `postgres:16` to `17`, ELEVEN apps are affected and the container image will NOT perform the conversion — and nothing anywhere records that.** MEASURED 2026-09-06: 11 of the 53 templates carry PostgreSQL — **8 on `postgres:16-alpine`**, 1 on `postgres:15-alpine`, plus `postgis/postgis:16-3.5-alpine` and Immich's own `postgres:16-vectorchord…` build. **A grep of the whole register for `pg_upgrade`, "postgres major" or "postgresql major" returns ZERO** (confirmed this session, and confirmed again before filing). **WHY IT IS NOT THE SAME PROBLEM AS R-459, and this is the point of the row: the two engines fail in OPPOSITE directions.** MariaDB starts anyway and skips the conversion quietly, which is why R-459 went unnoticed until a harness looked. **PostgreSQL REFUSES TO START on a datadir from an older major** — the official image performs no `pg_upgrade` and exits with a message naming both versions. So the Postgres case cannot hide; it will present as eight apps down at once, on the sync after the catalog moves. **DELIBERATELY NOT MEASURED HERE, and saying so is the scope discipline:** R-459's task was scoped to MariaDB, and measuring the Postgres analogue is its own piece of work with its own venue. **This row exists so the gap is a record rather than a sentence in an audit nobody greps.** What would settle it: one edge on the existing harness (`postgres:16-alpine` → `17-alpine`) on a scratch host, which would also exercise the `engine_state_after` field's Postgres probe end to end — it is written but has never run against a real Postgres major. Owner: **CC.** `audits/SPIKE-r459-mariadb-upgrade-2026-09-06.md` §7 **-- UPDATE NIGHT 2026-09-21:** **Measured 2026-09-21, both halves.** (a) What a household sees TODAY: the guarded Update of `postgres:16-alpine` to `17-alpine` on docmost ended **`failed` in 5.1 s**, the app stopped and held, **the pin naming 17 while `installed_images` still said 16 and nothing ran**, the data intact, and the restore the hold sentence names back in **29.1 s**. The engine's refusal line had to be REPRODUCED independently because `failAndHold` destroyed it (R-621): *FATAL: database files are incompatible with server / DETAIL: The data directory was initialized by PostgreSQL version 16, which is not compatible with this version 17.11.* The datadir was still `16` afterwards, and the same copy started under 16 holding 48 tables as the positive control. (b) The conversion **COSTED** on a real seeded 49 MB / 48-table datadir: `pg_dumpall` **2.6 s / 132 201 B**, fresh 17 plus replay **6.5 s / 48 tables restored**, the app up on 17 saying *Database connection successful*, **the seeded account read back**, total **155.9 s of which ~9 s is engine work**. `pg_upgrade` was NOT run: it needs both majors' binaries in one image and no such image exists here. Full paragraph: `audits/update-night-2026-09-21/24-Q5-postgres-conversion-costed.md`. **-- RULED 2026-09-23 (`09` §3 decision 16):** PostgreSQL majors are converted BY THE BOX as a guarded-update step — save everything from the old engine, start the new one empty, load it back, check. Each of the eleven apps is proven on the test bench before the catalog may move it; the engine-major gate stays until then. **-- 2026-09-25 (evening): docmost CONVERTED by the box** (16 → 18, controller v0.273.0; catalog `afd3a60` moved it with the conversion mark; proven on the bench and on 9202 incl. three undo cases). Measured: `postgres:18` refuses even an EMPTY volume at `/var/lib/postgresql/data` — an 18 step moves the mount. Open: the other ten, one proof each (calcom, adventurelog/postgis, claper, rallly, immich's own image, zipline, outline, paperless-ngx, tandoor, sparkyfitness on 15). **-- 2026-09-27: paperless-ngx (to 18) and tandoor (to 17) moved, each proven on the bench and on 9202 through the product's conversion (`09` §3 decision 42, `audits/version-travel-2026-09-26/B/`); with docmost that is 3 of the 11. Not moved: calcom, claper, outline, rallly, zipline (no seed route checked, or none: R-624), immich and the PostGIS family (out of scope).** **-- 2026-09-28: claper → 17 (`4a249b9`) and calcom → 18 (`037f956`, after its memory fix R-703) moved, each proven on both venues (bench LXC 9401 + box 9202 through the guarded Update), `09` decision 43; fixtures `Claper`, `Calcom`. `audits/pg-calcom-claper-2026-09-28/`.** **-- 2026-09-30 (the last six, by day): rallly (`25ffd89`), outline (`aeb0cd6`) and sparkyfitness (`1666572`, from 15) → 18, each proven on both venues (bench LXC 9401: 31/42/94 tables equal; box 9202 through the guarded Update: CONVERTED in 7.1/7.9/9.8 s) with one undo case each on 9202 (the load failing on `adminpack` → undone in 57.5/57.5/74.9 s, back on the old major, the seed read back); none `memory_tight`. Each app's own upstream runs 18 (decision 42's rule). STAY, by the same rule, read from upstream 2026-09-30: zipline (upstream compose `postgrLine truncated
| **R-464** | **[P3-LOW] MariaDB's entrypoint prints `MariaDB upgrade not required` on an UNSUPPORTED DOWNGRADE, so that line cannot be used as a soundness signal.** MEASURED 2026-09-06. After converting a datadir to `12.3.3-MariaDB` and then starting **11.6** on it, the entrypoint logs, on every start: **`[Note] [Entrypoint]: MariaDB upgrade not required`**. Asked properly, the same engine answers **`FATAL ERROR: Version mismatch (12.3.3-MariaDB -> 11.6.2-MariaDB): Trying to downgrade from a higher to lower version is not supported!`** **The entrypoint compares the datadir's recorded version against its own and concludes there is nothing to DO. That is true, and it is not a statement that the state is sound.** **THIS IS THIS PROJECT'S MOST-REPEATED CLASS, in a new costume** — the same shape as `CLAUDE.md`'s "presence is not success" and as R-443's HTTP 200 over a crash-looping app: a reassuring sentence that answers a narrower question than the one a reader will take it for. **Why it is worth a row rather than a footnote: the obvious cheap instrument for R-459 is to grep container logs for that exact line**, and such an instrument would report "fine" for an unsupported downgrade. **The correct probe is `mariadb-upgrade --check-if-upgrade-is-needed`**, which is what `upgrade-test.py`'s `engine_state_after` now uses. **Also recorded, because it nearly produced a wrong answer here: run without credentials that command returns `ERROR 1045 … FATAL ERROR: Upgrade failed` with exit 1** — an authentication failure wearing the shape of a verdict. Owner: **CC.** `audits/SPIKE-r459-mariadb-upgrade-2026-09-06.md` §5.4 | **READY — rank P3-LOW; owner: CC** |
| **R-468** | **[P3-LOW] THE GOLDEN WAIVER — goldens on a cadence, not per release (operator ruling 2026-09-13).** 25 goldens in 26 days in August, almost one per release, because `golden_currency_gate.py` trips on every release by design and the only honest ways past it were a bake or a declared `--no-verify` (thirteen by 2026-09-01, R-404/R-417). **The ruling: bake WEEKLY, and always before any drill or fresh install.** Every release still raises the FLOOR, so both demo boxes keep getting each release in ~20 s; only the golden — which protects a fresh install and nothing else — moves to a cadence. **The mechanism (built 2026-09-13):** `documentation/tests/golden-waiver. **⚠ CORRECTED THE SAME DAY (R-472): between bakes the floor does NOT carry a release — the hub holds any floor above the vouched golden (publish-train rule 1), so releases between bakes reach the demo boxes only by hand-deploy.**yml`, four lines (`issued`, `expires`, `reason`, `register_row: R-468`), read by the gate. While valid, a golden BEHIND the record makes the gate print a loud ADVISORY and exit 0; when it expires the gate is red again until someone bakes or renews. **The 14-day cap is enforced by the gate, not the runbook** — a longer, undated, unparseable, reason-less or row-less waiver is INCONCLUSIVE (exit 2), never 0 and never silently ignored. **It never covers a golden that is UNRECORDED (R-385)** — that is not a cadence choice. **A dated waiver cannot be forgotten; it just expires** — the difference from R-242's original rule, which recurred the day after it was written. Tests: `scripts/test_golden_currency_gate.py` cases 5–15 (E/F/G/H, a 15-day, absent, unparseable, bad-row and empty-reason waiver each 2; the R-421 decoy — a file saying only `expires` — 2). **This is a PRE-CUSTOMER arrangement: the first external install retires it** (delete the file in that commit). Cadence written into `RUNBOOK-manual-build.md` §4.2 and the `felhom.eu` end-of-session checklist. **Does NOT touch R-242's open half (nothing gates the VOUCH).** | **WATCHING — rank P3-LOW; owner: CC (renew ≤ 14 days or bake); retire at the first external install** |
@@ -788,7 +788,7 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
| **R-621** | **[P2-MEDIUM] A held update DESTROYS the evidence of why it failed: `failAndHold` runs `compose down`, the failing containers are removed, and their output is gone before anyone — household, operator or the next session — can read it.** MEASURED 2026-09-21 on guest 9202 on a REAL upstream edge: `adventurelog v0.12.1 → v0.13.0`. The new backend applied **nine Django migrations successfully** and then never listened; the update held after the full 5-minute health wait. **`Manager.failAndHold` (`stacks/update.go:723`) calls `updateCompose(dir, env, "down")`**, which removes the containers rather than stopping them, and nothing captures their logs first. Within seconds the box's own log recorded `Logs result for adventurelog: 0 bytes returned (empty)` and `docker ps -a` held nothing at all. **What survives is the WHAT and not the WHY:** the controller line `update adventurelog FAILED after the new version was started: not healthy: not healthy within 5m0s (last: state unhealthy)` and the household's sentence, both of which say the app did not come up and neither of which says the migrations ran and the server then failed to bind. **This is R-320 ("evidence off the machine before the teardown") as a PRODUCT behaviour rather than a session habit** — the teardown here is the product's own, it is correct to perform (a half-started new version must not keep running), and it happens before anyone can look. **Why it matters beyond a drill:** the hold sentence sends the household to a restore, and after the restore the only remaining question is *should I press Update again?* — which nobody can answer, because the one artefact that would say so no longer exists. It also makes every future held update unreportable to an upstream project. **Fix shape:** capture `compose logs --no-color --tail N` into the stack directory (beside `applied-compose.yml`, which already travels with the stack) IMMEDIATELY before the `down`, and surface it on the app page's hold panel or at least through the existing `/api/stacks/<n>/logs` fallback. Bounded size, written once per hold. A test that holds an app and asserts the captured file is non-empty — it fails today. Evidence: `audits/update-night-2026-09-21/apps/adventurelog/why-it-failed.txt`, `state-after-hold.txt`. **FIXED in v0.262.0.** `failAndHold` now writes each service's log into `<stackdir>/hold-logs/<ts>/compose-logs.txt` (`compose logs --no-color --tail 400`) **before** the `down` that destroys them, best-effort by design: a hold must never fail because its evidence could not be written. This is what `adventurelog` cost — nine migrations ran, the app never bound its port, and the only log that could have said why was gone before anyone looked. | **CLOSED 2026-09-22 — v0.262.0: the hold keeps the app's own log before stopping it** |
| **R-622** | **[P2-MEDIUM] `adventurelog v0.13.0` migrates the customer's database and then does not serve — the edge must NOT be promoted, and it is the first real-catalog candidate this project has measured as unsafe.** MEASURED 2026-09-21 on guest 9202 through the product's own guarded Update. `v0.12.1 → v0.13.0` (backend AND frontend together, PostGIS held constant). The backend applied **nine migrations, every one `... OK`** — `adventures.0072_trail_wanderer_author_fields` through `integrations.0009_alter_endurainintegration_auth_method`, plus `billing.0001_initial` — and then the container's own healthcheck failed with `URLError: [Errno 111] Connection refused` on five consecutive checks. The app never bound its port. The update held honestly after the full 5-minute wait. **THE PRODUCT DID EVERYTHING RIGHT AND THAT IS HALF THE FINDING:** the precondition found a Tier-1 copy one minute old, the safety dump was written, the pin advanced BEFORE the pull, the health wait was not short-circuited, the app was stopped rather than left half-running, and the hold sentence named the tier, the date and what the copy holds — „saját meghajtó, 2026-09-21 20:47 — ez a másolat a beállításokat, az adatbázist és az adatköteteket tartalmazza." **This is exactly the case `09` §4 exists for:** the migration RAN, so there is no undo, only a restore — and the restore is the thing slice 4 made sure existed first. **What it needs:** adventurelog stays OFF the promotion list; the cause is not diagnosed here (R-621 is why); and before it is ever promoted the edge should be re-run on the harness with its ABORT step, since an app that migrates and then refuses is the shape most likely to refuse the old image too. Evidence: `audits/update-night-2026-09-21/apps/adventurelog/`. | **READY — rank P2-MEDIUM; owner: CC (catalog); NOT a Felhom defect — an upstream edge that fails** |
| **R-623** | **[P3-LOW] The unattended-update caller turned every SUCCESS into a `timeout`, and then refused to press that app again — the instrument, not the box.** FOUND 2026-09-21 (update night) by reading `unattended-caller.py` before relying on it for the Q4 hold measurement. Its `call()` returns the API **envelope** — `{"ok": true, "data": {…}}` — and `follow()` read `update_phase` and `updating` **off the envelope**, where neither exists. Both were therefore always `None`; the end test `not updating and phase in ("done","failed")` could never fire; every followed update ran the full **900-second** timeout and was recorded `timeout`, which the caller treats as terminal and adds to `never_again`. `main()` unwraps `data` for the stack LIST, which is exactly why the within-a-major half of that night worked and this half did not. **The 2026-09-21 run did not catch it because the only pass that reached `follow()` was Scenario F, whose log was lost to a buffering `tail`** — the run's own honestly-recorded instrumentation gap turns out to have hidden a second one underneath it. **This is the R-607 class in the evidence layer rather than the product layer: an instrument that can report a success as a timeout is not a measurement**, and worse, it is a measurement that says the box behaved badly when the box behaved well. **FIXED in the same file 2026-09-21** (unwrap `data`, with the reason written into the docstring so the next reader does not re-derive it), and the fixed caller is what produced tonight's unattended-hold leg. **What it does NOT invalidate:** the G-b no-retry proof, which is entirely in the refusal path and never reached `follow()`. **What it DOES qualify:** any future reading of that night's Scenario F timing — the "51 s – 1 m 26 s" figures come from the ATTENDED scenarios 04/05/07, not from the caller. | **CLOSED 2026-09-21 — fixed in `audits/update-arc-gaps-2026-09-21/unattended-caller.py`** |
| **R-624** | **[P3-LOW] Three of the catalog's apps cannot be seeded by ANY headless route, and for two of them that is a deliberate security decision — so the upgrade harness has a permanent ceiling nobody has written down.** FOUND 2026-09-21 while widening R-462 from 3 apps to <N>. **`vaultwarden` and `zipline` close self-registration ON PURPOSE** — vaultwarden by `SIGNUPS_ALLOWED=false` (R-512, *„a stranger who guesses vault.<domain> must not be able to register"*), zipline by answering `E1037: User registration is disabled` — and neither ships a CLI that could make an account instead. So there is no route to a first account without the admin secret, and **that is correct**: the harness must not be the reason a customer-facing app accepts strangers. **`gitea` is a different and fixable case:** the template sets no `INSTALL_LOCK`, so a fresh instance sits in its web-installer state and `gitea admin user create` refuses (`MustInstalled() [F] Unable to load config file for a installed Gitea instance`); POSTing the installer form first would work and was simply not written tonight. **Why this is a row rather than three notes:** `09` §3 decision 6 says the upgrade test goes to **all** apps, and R-462 is costed as if every app is reachable given enough fixture work. It is not. There is a class — *apps whose only account-creating route the catalog deliberately closes* — for which the honest maximum is `inconclusive` unless the harness is given the app's admin secret at deploy time, which is a decision nobody has taken. **Needs:** the class named in R-462's scope so the remaining count is honest; a decision on whether the harness may hold an app's admin secret (it already holds the ones IT generates — see R-619); and, separately and cheaply, a gitea installer-form fixture. Evidence: `audits/update-night-2026-09-21/apps/{vaultwarden,zipline,gitea}/verdict.json`. **— NIGHT 2026-09-23:** `gitea` stayed inconclusive on both venues (its installer); `vaultwarden`/`zipline` not attempted (closed sign-up, by design); `code-server`, `outline`, `rallly` have no front-door seed route (listed, not moved); `bentopdf`, `glance`, `crafty-controller`, `wger`, `wanderer` (meilisearch) and `uptime-kuma` have no fixture tonight (listed, not moved). **-- 2026-09-30: the ceiling was WRONG for three of the six named apps.** outline has a front-door first-run route (`POST /api/installation.create` — its self-hosted setup, refused once a team exists), rallly signs up through its own better-auth API with the e-mail code read from its own table in place of a mailbox, and zipline 4's first-run route is `POST /api/setup` (the fixture had tried `/api/auth/register` and `/api/auth/setup`, which are not it). Fixtures for outline and rallly are in `upgrade_fixtures_box.py` and both apps moved on both venues; zipline's fixture now tries `/api/setup` first (measured on the bench: a SUPERADMIN made, the login works). **What remains in the class:** vaultwarden (closed sign-up by design, no CLI), code-server (a browser IDE). `audits/pg-last-six-2026-09-30/`. |**READY — rank P3-LOW; owner: CC (catalog harness)** |
| **R-624** | **[P3-LOW] Three of the catalog's apps cannot be seeded by ANY headless route, and for two of them that is a deliberate security decision — so the upgrade harness has a permanent ceiling nobody has written down.** FOUND 2026-09-21 while widening R-462 from 3 apps to <N>. **`vaultwarden` and `zipline` close self-registration ON PURPOSE** — vaultwarden by `SIGNUPS_ALLOWED=false` (R-512, *„a stranger who guesses vault.<domain> must not be able to register"*), zipline by answering `E1037: User registration is disabled` — and neither ships a CLI that could make an account instead. So there is no route to a first account without the admin secret, and **that is correct**: the harness must not be the reason a customer-facing app accepts strangers. **`gitea` is a different and fixable case:** the template sets no `INSTALL_LOCK`, so a fresh instance sits in its web-installer state and `gitea admin user create` refuses (`MustInstalled() [F] Unable to load config file for a installed Gitea instance`); POSTing the installer form first would work and was simply not written tonight. **Why this is a row rather than three notes:** `09` §3 decision 6 says the upgrade test goes to **all** apps, and R-462 is costed as if every app is reachable given enough fixture work. It is not. There is a class — *apps whose only account-creating route the catalog deliberately closes* — for which the honest maximum is `inconclusive` unless the harness is given the app's admin secret at deploy time, which is a decision nobody has taken. **Needs:** the class named in R-462's scope so the remaining count is honest; a decision on whether the harness may hold an app's admin secret (it already holds the ones IT generates — see R-619); and, separately and cheaply, a gitea installer-form fixture. Evidence: `audits/update-night-2026-09-21/apps/{vaultwarden,zipline,gitea}/verdict.json`. **— NIGHT 2026-09-23:** `gitea` stayed inconclusive on both venues (its installer); `vaultwarden`/`zipline` not attempted (closed sign-up, by design); `code-server`, `outline`, `rallly` have no front-door seed route (listed, not moved); `bentopdf`, `glance`, `crafty-controller`, `wger`, `wanderer` (meilisearch) and `uptime-kuma` have no fixture tonight (listed, not moved). **-- 2026-09-30: the ceiling was WRONG for three of the six named apps.** outline has a front-door first-run route (`POST /api/installation.create` — its self-hosted setup, refused once a team exists), rallly signs up through its own better-auth API with the e-mail code read from its own table in place of a mailbox, and zipline 4's first-run route is `POST /api/setup` (the fixture had tried `/api/auth/register` and `/api/auth/setup`, which are not it). Fixtures for outline and rallly are in `upgrade_fixtures_box.py` and both apps moved on both venues; zipline's fixture now tries `/api/setup` first (measured on the bench: a SUPERADMIN made, the login works). **What remains in the class:** vaultwarden (closed sign-up by design, no CLI), code-server (a browser IDE). `audits/pg-last-six-2026-09-30/`. **-- 2026-09-30 (evening): gitea's fixable case is DONE** — the fixture posts gitea's own first-run installer form (the form's own defaults + an admin; no CSRF on that form), measured on the bench and through the setup gate on 9202; gitea 1.27.0 → 1.27.3 published `d7ba60c`. What remains in the class: vaultwarden, code-server. `audits/more-night-apps-2026-09-30/` |**READY — rank P3-LOW; owner: CC (catalog harness)** |
| **R-628** | **[P2-MEDIUM] An empty search of a mailbox I do not control was turned into a claim about what a THIRD PARTY had done, and it went into the register as fact.** FOUND 2026-09-22; the operator caught it within minutes by producing the thread. The `due_checks_gate` fired R-433 (*have Hetzner answered?*). Two searches of the felhom catch-all came back empty, and the emptiness was written into R-433 as **"Hetzner has not answered"** and **"there is no evidence the tickets were ever opened"**. Both were false: ticket **#2026090103040671** had been opened and answered. **THE PRECISE FAULT IS THE INFERENCE, NOT THE QUERY — and that distinction is the whole value of this row.** Re-run afterwards WITH a positive control: `from:monitoring@felhom.eu` returns **201 threads**, `in:anywhere … includeTrash` reaches SENT, TRASH and mail back to January — **the instrument works.** And the exact ticket number, the exact subject `Storage Box issue`, and `from:hetzner` each still return **nothing**. So the literal finding — *this correspondence is not in this mailbox* — was CORRECT. What was invented was the step from there to *Hetzner has not answered* and *the tickets were never opened*. **A mailbox I can read is not the only place a reply can be**, and the operator's own account is exactly where a support ticket he opened would land. An absent record in ONE place can never answer a question about what SOMEONE ELSE did. **This is R-96 rule 3 in a new surface, one step further out than R-607**: there the instrument reported a stale value as current; here a correct observation was promoted to a conclusion it could not carry. **It is sharper still because the same session, the night before, gave every fixture a negative control and every gate a red-proof — and then reached for a search with neither.** **THE RULE, in one line:** *an empty search may be reported as "absent from the place I looked", never as "it did not happen" — and only after a control query that MUST hit has been seen to hit.* **Done:** the rule is written into the Gmail-access memory, where the next session meets it before it searches rather than after. | **CLOSED 2026-09-22 — rule recorded; R-433 corrected with the real answers** |
| **R-627** | **[P2-MEDIUM] Nothing checked that the register is a well-formed table, so an append that ate two rows' state cells went unnoticed until a person read the file — and one row had been broken the same way for 45 days.** FOUND 2026-09-22. The 2026-09-21 update night appended measured results to eight rows with a regex that matched each row's trailing state cell; on **R-446** and **R-458** it consumed the cell and did not restore it, the cell reappeared as a stray FOURTH cell on a DUPLICATED copy of **R-626** and **R-625**, and a blank line was left between each pair. The register then reported **317 rows for 315 findings**, two rows carried no state at all, and two findings existed twice with contradictory state cells. **Nothing caught it:** `one_register_gate.py` compares this file against ROADMAP and `closed_register_gate.py` forbids an id in BOTH files — neither asks whether the file is a well-formed table, and neither notices an id duplicated WITHIN it. **THE RED-PROOF THEN FOUND AN OLDER INSTANCE NOBODY HAD SEEN: R-254 lost its state cell on 2026-08-08 (commit `59527d0`) and had rendered without a State column for 45 days.** **Closed the same day:** `scripts/register_shape_gate.py`, registered in `repo_gates.py` as gate 14 and reached by the pre-push hook, refusing a row that does not end with `|` (an eaten state cell), a duplicated id, or a blank line splitting the table; four decoys in `test_gate_decoys.py` — three convicting on the exact damage shapes and one asserting a healthy register still passes. **TWO THINGS THE RED-PROOF CORRECTED IN THE GATE ITSELF, kept because they are the finding's real content:** a first draft counted CELLS and convicted **125 innocent rows** — register cells carry literal `|` inside prose and shell snippets (`owner: CC | …`), so a row cannot be split on `|`, and a count that cannot be computed is not a check; and it skipped malformed rows before counting ids, reporting **5 duplicates where there were 2**. **Repaired:** both state cells restored from the stray cells that carried them, the two duplicate rows deleted, R-254's verdict sentence given its cell back, and **15 blank lines that split the register into 12 separate markdown tables** removed — every row's text byte-identical afterwards, proven by diff. **And the gate immediately earned itself:** the very next row inserted in this session (R-628) left a blank line behind and the gate refused it. | **CLOSED 2026-09-22 — gate 14, four decoys, register repaired 317→315** |
| **R-629** | **[P2-MEDIUM] The drill catalog sent the operator 47 CI-failure alarms in one night, and the drill method that created it did not mention CI at all.** FOUND 2026-09-22 while checking a different mailbox question — which is the only reason it was found. `admin/app-catalog-drill` was created on 2026-09-21 by `POST /api/v1/repos/migrate` from the live catalog, and a migrated repo inherits `has_actions: true`. Every drill push therefore ran the catalog's CI workflow, which failed immediately (the drill repo carries the workflow but the run has no meaningful gate context), and **each failure mailed `admin@felhom.eu`**: *"[felhom CI] gates FAILED in admin/app-catalog-drill"*. **47 runs, 47 alarms, all overnight, all unread and flagged IMPORTANT.** **WHY THIS MATTERS MORE THAN THE NOISE:** that mailbox is the operator's alarm channel, and R-168 made CI mail the thing that notices a bypassed gate. A night of throwaway failures from a repo nobody must ever act on trains the reader to skim exactly the sender that must never be skimmed — and it did it on the night the same mailbox was also carrying real `offsite_snapshots_dropped` and `offsite_proof_empty` alarms. **The drill method wrote down the fences it needed** — private repo, no customer box may follow it, reset at teardown — **and said nothing about CI, because nobody had run a drill repo through a CI-enabled Gitea before.** **FIXED 2026-09-22:** `has_actions` set to **false** on the drill repo (verified by re-reading the repo: `has_actions: False, private: True`), and `09` §6.5 now carries it as a step of creating a drill repo rather than as a thing to notice afterwards. **What is NOT done:** the 47 mails are still in the operator's inbox, unread — deleting another person's mail is not mine to do, and they are named here so they can be cleared in one search: `subject:"gates FAILED in admin/app-catalog-drill"`. | **CLOSED 2026-09-22 — actions disabled, method updated; the 47 mails are the operator's to clear** |
@@ -843,9 +843,17 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
| **R-729** | **[P3-LOW] An off-site target, once saved on the page, cannot be removed through the product.** MEASURED 2026-09-30 on 9202: `/backup/offbox/config` refuses an empty address and no route clears the target; the session removed its throwaway target from `settings.json` by hand, with the controller stopped (harness teardown on a scratch guest). A household that tries its own NAS and gives up keeps a disabled target forever. **Fix direction:** a „Távoli mentési cél törlése" press that clears the target (never the repository). | **READY — rank P3-LOW; owner: CC (controller)** |
| **R-730** | **[P3-LOW] The published ISO 1.29.0 was built from a tree that git cannot name, so it cannot be tagged.** MEASURED 2026-09-30 from the build record: the manifest says `repo-commit 8d539f97` (2026-09-18 17:48), built 20:41:45; the menu-width fix the published image carries (`grub-release.cfg.tmpl`, „… / text") was committed 35 minutes AFTER the build, in `31eeb36` (21:16). `scripts/iso/build-felhom-iso.sh:498` records `git rev-parse HEAD` with no dirty-tree check, so the manifest names a commit that is not the image. **And the brief's tag name would have been wrong:** the `installer-v*` tags are the host-install SCRIPT's versions (`SCRIPT_VERSION` 1.28.0 on main; the website serves `/scripts/` from `installer-v1.28.0`), a different number line from the ISO — an `installer-v1.29.0` tag would block the next script release. **Needs:** the ISO build refuses a dirty tree (the workspace clean-tree gate, applied to this script) and records `iso-v<version>`; optionally prove 1.29.0's source by content (the ISO's `grub.cfg` against `31eeb36`) and tag that. `audits/pg-last-six-2026-09-30/E/E4-installer-tag.txt` **-- 2026-09-30: BUILT.** `scripts/iso/build-felhom-iso.sh` now runs a clean-tree gate right after argument parsing: any uncommitted or untracked change, or HEAD ≠ `origin/main`, refuses the build (no bypass flag); the manifest records `repo-commit` from the gate and `iso-version-tag : iso-v<version>`. Test `scripts/iso/test/clean-tree.sh` (4 cases, throwaway repo via the `FELHOM_ISO_REPO` seam); **red-proof run:** the status check deleted → cases 2 and 4 FAIL → restored → all pass. `test/rootpw-emission.sh` now builds from the same kind of throwaway clean repo (it cannot run on DooPlex either way: the assistant image is absent, 9 FAILs before and after). ISO 1.29.0 stays untagged: its source cannot be proven. | **CLOSED 2026-09-30 — the gate + test; 1.29.0 untagged by decision (its build tree is not a commit)** |
| **R-731** | **[P3-LOW] The catalog-currency comparison cannot see an upstream that changes its TAG SHAPE, and two newest tags need a release check.** MEASURED 2026-09-30 (`audits/catalog-currency-2026-09-30.md` §2 item 5): the same-shape rule read three apps as up to date that were not — gramps-web (`v25.6.0` → upstream dropped the `v`, at `26.9.1`), jellyfin (`10.11.11` → two-part `12.1`), kimai (`apache-2.57.0` → plain `2.67.0`; the plain tag's digest equals `apache`'s, so kimai moved on 2026-09-30). A control pass over every shape caught them. Not checked: whether `mariadb:13.0` and `gitea/gitea:28.0.0` (pushed 2026-09-30 00:15 UTC) are general releases. **Needs:** the currency script's shape-switch control made standing (it is in the audit's tools today), and the two release checks before either is walked. **-- 2026-09-30: the two release checks answered** (`audits/immich-first-start-2026-09-30/D/D2-release-checks.txt`): **gitea v28.0.0** is a general release (GitHub: not prerelease, not draft, 2026-09-29) — upstream renumbered 1.27.x → 28; **mariadb 13.0** is `Stable` but a short-term `Rolling` line (no EOL date), while 12.3 and 11.8 are the LTS lines — so a move of any of the four MariaDB apps to 13.0 would leave LTS. Not done: the shape-switch control made standing. | **NARROWED 2026-09-30 — only the standing shape-switch control is left; owner: CC** |
| **R-732** | **[P2-MEDIUM] immich's FIRST start at today's catalog pin could not finish on the bench: its database container was OOM-killed at 512 MiB.** MEASURED 2026-09-30, twice (the second run alone on the bench): at FROM (v3.2.2, `…/postgres:16-vectorchord0.4.3-pgvectors0.2.0` at the ladder's own digest `1a078b23…`), the server's first-start reverse-geocoding import dropped its DB connection („terminating connection because of crash of another server process"), and the harness's 420 s settle never saw it healthy; a live read of the postgres cgroup showed `oom 413, oom_kill 36`, `OOMKilled true`. **The same template on 9202 the same hour** (a fresh install, kept data moved aside) seeded in 35 s and its v3.2.4 step ended `done` — but that first container was replaced by the update, so its kill counter is gone: whether a household's fresh install on a smaller box hits this is NOT measured. R-676's watch (a first start restarting ≥ 6 times in 10 min is stopped by decision 28) makes the consequence a stopped app. So immich's v3.2.4 step stays unpublished (box proven, bench inconclusive twice — Part F's stop rule). **Needs:** a sampled first-start memory watch of immich-postgres on a fresh install (bench and box, `anon` + `oom_kill` from the container's birth), then a measured `mem_limit`. `audits/pg-last-six-2026-09-30/F/immich-first-start-oom.txt`, `F/benchq-q6.txt`, `F/benchq-q10.txt` **-- 2026-09-30 (afternoon): CAUSE MEASURED and FIXED.** immich's first-start geodata import runs up to 9 concurrent 5000-row INSERTs; the database then needs ~400 MB anon + ~170 MB touched shared_buffers (the image's own `postgresql.conf` fixes 512MB — NOT sized from host RAM). At 512M without swap it is OOM-killed (bench: 61–104 kills); with 512 MiB swap it survives by swapping ~70–110 MB (why 9202 passed). Controls, one variable each: swap alone → 0 kills; limit 1024M alone → 0; `shared_buffers` 128MB alone → still 104 kills. **An update that ships a new geodata file re-runs the import** (v3.2.2 → v3.2.4: 228 294 → 228 571 places), so the fix had to ride the step. Catalog `56c4888`: immich v3.2.4 + `immich-postgres` 768M (`mem_limit` 4096M → 4480M; +256 MB per immich box). Proven with swap OFF: fresh install at the new definition on the bench ×2 (anon 409/412 MB = 53 %, 0 kills) and on 9202 (368 MB = 48 %, 0 kills); the step on both venues (bench proven, 0 kills in the 10-min watch; box done 58.5 s, album read back, running limit 768M). `audits/immich-first-start-2026-09-30/A-cause.md`. | **CLOSED 2026-09-30 — catalog `56c4888` (768M + v3.2.4); an installed immich gets it with its next guarded Update** |
| **R-732** | **[P2-MEDIUM] immich's FIRST start at today's catalog pin could not finish on the bench: its database container was OOM-killed at 512 MiB.** MEASURED 2026-09-30, twice (the second run alone on the bench): at FROM (v3.2.2, `…/postgres:16-vectorchord0.4.3-pgvectors0.2.0` at the ladder's own digest `1a078b23…`), the server's first-start reverse-geocoding import dropped its DB connection („terminating connection because of crash of another server process"), and the harness's 420 s settle never saw it healthy; a live read of the postgres cgroup showed `oom 413, oom_kill 36`, `OOMKilled true`. **The same template on 9202 the same hour** (a fresh install, kept data moved aside) seeded in 35 s and its v3.2.4 step ended `done` — but that first container was replaced by the update, so its kill counter is gone: whether a household's fresh install on a smaller box hits this is NOT measured. R-676's watch (a first start restarting ≥ 6 times in 10 min is stopped by decision 28) makes the consequence a stopped app. So immich's v3.2.4 step stays unpublished (box proven, bench inconclusive twice — Part F's stop rule). **Needs:** a sampled first-start memory watch of immich-postgres on a fresh install (bench and box, `anon` + `oom_kill` from the container's birth), then a measured `mem_limit`. `audits/pg-last-six-2026-09-30/F/immich-first-start-oom.txt`, `F/benchq-q6.txt`, `F/benchq-q10.txt` **-- 2026-09-30 (afternoon): CAUSE MEASURED and FIXED.** immich's first-start geodata import runs up to 9 concurrent 5000-row INSERTs; the database then needs ~400 MB anon + ~170 MB touched shared_buffers (the image's own `postgresql.conf` fixes 512MB — NOT sized from host RAM). At 512M without swap it is OOM-killed (bench: 61–104 kills); with 512 MiB swap it survives by swapping ~70–110 MB (why 9202 passed). Controls, one variable each: swap alone → 0 kills; limit 1024M alone → 0; `shared_buffers` 128MB alone → still 104 kills. **An update that ships a new geodata file re-runs the import** (v3.2.2 → v3.2.4: 228 294 → 228 571 places), so the fix had to ride the step. Catalog `56c4888`: immich v3.2.4 + `immich-postgres` 768M (`mem_limit` 4096M → 4480M; +256 MB per immich box). Proven with swap OFF: fresh install at the new definition on the bench ×2 (anon 409/412 MB = 53 %, 0 kills) and on 9202 (368 MB = 48 %, 0 kills); the step on both venues (bench proven, 0 kills in the 10-min watch; box done 58.5 s, album read back, running limit 768M). `audits/immich-first-start-2026-09-30/A-cause.md`. **-- 2026-09-30 (evening, Part D): the older step closed too.** immich's step `0b8272068aab36bf` (server v3.0.3 → v3.2.2) still pinned 512M and DOES re-run the import (geodata-date 2026-07-13 in v3.0.3, 2026-08-30 in v3.2.2 — read inside both images, `D/D2`). Re-proven on the bench, swap 0, on that step's own definition at 768M: the import ran in 17 s, kill counter 0 from the container's birth, album read back, 10-min watch 0 kills; the step's files rewritten by the new writer mode `upgrade-test.py --restep` (catalog `48440ce`, `63a96b0`). No box reporting to the hub runs immich at any version (hub `/apps`, positive control bookstack = 2). `audits/more-night-apps-2026-09-30/` | **CLOSED 2026-09-30 — catalog `56c4888` (768M + v3.2.4); an installed immich gets it with its next guarded Update** |
| **R-733** | **[P3-LOW] The test bench has NO swap and the boxes have 512 MiB — so a box proof can pass on swap where the bench fails, and nobody records whether a customer guest has swap.** MEASURED 2026-09-30 (R-732): immich's first start was OOM-killed 61–104 times on the bench (swap 0) and passed on 9202 by swapping ~108 MB; the bench given 512 MiB swap passed too. demo-hp 9201, 9202 and demo-felhom 9201 all read `swap: 512`; the golden's guest config is not recorded in its bake evidence, so a customer guest's swap is NOT measured. The harness's memory watch judges `anon` against the limit and never reads `memory.swap.current`. **Needs:** the golden's `swap` read and recorded; the box walk and the harness report `memory.swap.peak` beside `anon`; a decision whether proofs run with swap off (the stricter venue, as R-732's fix was proven). | **READY — rank P3-LOW; owner: CC (harness + golden evidence)** |
| **R-734** | **[P3-LOW] The harness marks immich `files_may_change` because immich rewrites six 13-byte `.immich` folder markers at every start.** MEASURED 2026-09-30 on the bench (v3.2.2 → v3.2.4): the bind-tree hash of `appdata/immich` changed; the only changed files were `{encoded-video,library,backups,profile,thumbs,upload}/.immich`, rewritten at each start — no household file. The mark is honest by the harness's rule and the ladder writer copies it (never edited by hand), so immich's v3.2.4 night step needs a fresh WHOLE copy (decision 13); on a box without one the night leg skips it and a person presses. The 2026-09-23 immich entry did not carry it (`files_changed []`). **Needs:** a decision whether app-owned marker files are excluded from the file hash (a per-template ignore list, or a size/name rule), or the mark stays. | **READY — rank P3-LOW; owner: CC (harness); the rule change needs a word** |
| **R-734** | **[P3-LOW] The harness marks immich `files_may_change` because immich rewrites six 13-byte `.immich` folder markers at every start.** MEASURED 2026-09-30 on the bench (v3.2.2 → v3.2.4): the bind-tree hash of `appdata/immich` changed; the only changed files were `{encoded-video,library,backups,profile,thumbs,upload}/.immich`, rewritten at each start — no household file. The mark is honest by the harness's rule and the ladder writer copies it (never edited by hand), so immich's v3.2.4 night step needs a fresh WHOLE copy (decision 13); on a box without one the night leg skips it and a person presses. The 2026-09-23 immich entry did not carry it (`files_changed []`). **Needs:** a decision whether app-owned marker files are excluded from the file hash (a per-template ignore list, or a size/name rule), or the mark stays. **-- 2026-09-30 (evening):** the harness now NAMES the files behind the mark (`files_changed_detail`, catalog `5b1972b`); on immich's step `0b82…` re-proof it named exactly the six `.immich` markers again. calibre-web's step (v4.0.6 → v4.0.8) carries the mark too, and the named files are its LIBRARY DATABASE: `media/books/metadata.db`, `metadata.db-shm`, `metadata.db-wal` changed; the book file did not (bench names re-run, `A/calibre-names/`). That is household data (the template's backup class `mandatory` holds DB and books as one unit), so the mark is right there and the night leg takes that step only with a fresh whole copy. On 9202 the same step changed no file in that folder (per-file hashes before/after) — not explained. `audits/more-night-apps-2026-09-30/` | **READY — rank P3-LOW; owner: CC (harness); the rule change needs a word** |
| **R-735** | **[P3-LOW] The test bench generates a `password:N:special` deploy field WITHOUT a special character, so calibre-web's own password rule refuses it and the bench cannot seed the app.** FOUND 2026-09-30 (more-night-apps brief, Part A) on the bench's second calibre-web run: the fixture ran the template's own `after_install` (`cps.py -s admin:<pw>`, the product's command) and the app answered „Password doesn't comply with password validation rules". Cause, read from source: `check-volume-persistence.py` `_gen` treats `password:24:special` as `password:24` (letters and digits); the controller's `generateValue` (`deploy.go` L1150–1164, `randomWithSpecial` L1335) adds one of `-_.!@#%+=` and at least one lower, upper and digit. The box walk was not affected (its own generator writes `Drill-<hex>`). **Instrument, not product.** **Needs:** `_gen` mirrors `randomWithSpecial`, with a test seen failing first. `audits/more-night-apps-2026-09-30/bench/apps/calibre-web/` **-- FIXED 2026-09-30 (catalog `5b1972b`):** `_gen` mirrors `randomWithSpecial`; test `test_password_special_carries_the_controllers_shape` seen failing first (length 32, no special), then 45/45 (`A/R735-red.txt`, `A/R735-green.txt`); calibre-web then proven on the bench. | **CLOSED 2026-09-30 — catalog `5b1972b`** |
| **R-736** | **[P2-MEDIUM] Removing an app or updating it never deletes the image it pulled, so a box's Docker disk fills with old images until an install or an update is refused.** MEASURED 2026-09-30 on scratch guest 9202: a wger install was refused „Docker volume below reserved buffer (4.3G free, reserve 6.8G of 68G)" — correctly; the disk held **84 images, 57 GB, 53 GB of them used by no container** (`docker system df`), left by earlier drills' installs, updates and removals through the product. Read from source: the remove runs `docker compose down --rmi local --volumes` (`felhom-controller` `stacks/delete.go:291`); `--rmi local` removes only images with no custom tag, i.e. never an image pulled by a registry tag. No image clean-up exists anywhere else in the controller (searched: `rmi`, `image prune`, `ImageRemove`). A guarded Update leaves the previous version's image too (it is also the undo's image — keeping the LAST one is deliberate; keeping every one is not decided anywhere). **A household box follows the same path more slowly:** every automatic step adds an image, and the refusal lands on the next install or update (the night leg's `disk` refusal). Not measured: how fast a real box fills. **Needs:** a decision on what the box keeps (the running image + the undo's previous one) and a clean-up after the undo window closes and at remove. `audits/more-night-apps-2026-09-30/box/T2-images-removed-by-name.txt` | **READY — rank P2-MEDIUM; owner: CC (controller); the retention rule needs a word** |
| **R-737** | **[P3-LOW] wger's app login API answers 500 on a CORRECT password: the template sets no `JWT_PRIVATE_KEY`.** MEASURED 2026-09-30 on the bench (wger 2.6, catalog template): `POST /allauth/app/v1/auth/login` with the right admin password → 500, `ValueError: Unable to load PEM file … MalformedFraming` (`allauth/core/internal/jwkkit.py`); a wrong password → 400 JSON (control). wger reads `JWT_PRIVATE_KEY` from the environment (`settings/main.py:109`) and the template sets none. The web login works. Not measured: whether wger's mobile app uses this route (likely — it is the headless API „consumed by the Flutter app", `wger/urls.py`). **Needs:** a generated key (an RSA PEM — the controller's generators do not make one today) or the page saying the mobile app cannot log in. `audits/more-night-apps-2026-09-30/B/wger-probe.txt` | **READY — rank P3-LOW; owner: CC (catalog)** |
| **R-738** | **[P2-MEDIUM] Every wger update that brings database migrations leaves wger broken, and the guarded Update reports `done`.** MEASURED 2026-09-30 on scratch guest 9202 (drill catalog): wger 2.6 → 2.7 through the product's guarded Update ended `done` in 72.8 s (health: the front page answers); the web login then answered **500** (`no such column: core_userprofile.time_zone`), still 500 a minute later, **12 migrations unapplied** (`manage.py showmigrations`). Cause, read from the image: `entrypoint.sh` runs `manage.py migrate` only when `DJANGO_PERFORM_MIGRATIONS=True`; the template does not set it (a fresh install works because `wger bootstrap` builds a new database). The harness caught it: the box verdict is `failed`, and no ladder entry was written. No box reporting to the hub runs wger (hub `/apps`, same day). **Also a product gap:** the guarded Update's health check cannot see an app whose front page serves while its data is unusable — the fixture's read-back is what saw it. **Fix:** `DJANGO_PERFORM_MIGRATIONS=True` in the template (the image's own switch for it), then the step again on both venues. `audits/more-night-apps-2026-09-30/box/wger/` **-- FIXED 2026-09-30:** catalog `7a4ff48` sets `DJANGO_PERFORM_MIGRATIONS=True`; the same 2.6 → 2.7 step then ran the migrations (`Applying …` in the log) and read back on 9202 (46.2 s) and on the bench (proven, 0 kills); published `4ad32aa`. The product-side gap (a health check that sees only the front page) remains, as a note: the fixture's read-back caught it, the box could not. | **CLOSED 2026-09-30 — catalog `7a4ff48` + `4ad32aa`** |
| **R-739** | **[P3-LOW] The test bench cannot run wanderer at all: its web server calls the database at the PUBLIC name `https://<SUBDOMAIN_DB>.<domain>`, which the bench has no name or TLS for.** MEASURED 2026-09-30 (more-night-apps brief, Part B): on bench 9401 the catalog template (v0.20.0) came up `wanderer-db` healthy, `wanderer-search` healthy, `wanderer` **unhealthy** for 12 min, every page 500 („Error 0: Something went wrong"); `PUBLIC_POCKETBASE_URL=https://hike-db.gate.invalid`. So the harness can only ever answer `inconclusive` at FROM for wanderer, never about an update. Its step `v0.20.0 → v0.21.0` (web + db) and meilisearch `v1.36 → v1.54` stay untested; **the brief's question — does the search index survive meilisearch's move or is it rebuilt — is NOT measured.** **Needs:** a bench venue that gives the stack the DB name (an `extra_hosts` + plain-http override in the bench's render only, stated in the verdict), or wanderer proven on the box venue alone with the operator's word. `audits/more-night-apps-2026-09-30/B/wanderer-probe.txt` | **READY — rank P3-LOW; owner: CC (catalog harness)** |
| **R-740** | **[P2-MEDIUM] A security fix that upstream ships under the SAME tag (`postgres:18-alpine`, `redis:7-alpine`, `mariadb:12.3` …) reaches NO box — not at night, and not by the household's Update button either, because the catalog never records a re-test of a tag at a new digest.** MEASURED 2026-09-30 (more-night-apps brief, Part C). **(1) The night leg, from source** (`felhom-controller` `d48da6c`): an app at the head tag whose running digest is older than the ladder's tested one reads Behind (`stacks/updateorder.go:86–111` `digestBehind`), but `legCandidate` finds no entry whose `from` is its pin and skips it — `unattended.go:433–435`, `return LegSkipNoTestRecord // at the head, behind only by something no step records`. A unit walk on a scratch copy of the controller confirms it: order Behind → the leg presses nothing, `skipped — no_test_record`; the household's Update button in the same state ends `done` and runs the tested digest (`audits/more-night-apps-2026-09-30/C/C2-*`). **(2) But the catalog never produces that state for a floating tag:** the only writer refuses a step that moves no image (`upgrade-test.py:919`, „--move: nothing moves"), and no ladder in the catalog has an entry with `from == to` or two entries with the same `to` (`C/C4-same-tag-retest.txt`). So the tested digest of `postgres:18-alpine` stays the one of the day the step was tested, a box installed since runs that digest, and the badge reads „Naprakész" while upstream has moved. **(3) How often it matters:** of the 11 distinct floating lines in the catalog today (26 services), the 8 on Docker Hub were pushed 9, 9, 9, 9, 12, 12, 30 and 105 days ago — **7 of 8 within 30 days** (`C/C3-floating-repush.txt`; a push is not proof that the image content changed; ghcr gives no date). Today every tested digest still equals what the registry serves (the tests came after the pushes). Only a box installed BEFORE a step's test can read Behind by digest; the demo boxes have none (`C/C1-digests-demo-hp-9201.txt`, 18 of 18 equal). **(4) Decision 30's cost line says the older image runs „until someone (or the automatic leg) presses Update"** — the automatic leg never does, and the manual press moves the digest only in the legacy case above. **(5) What the box would do if the catalog wrote a same-tag re-test** (unit walk): the leg PRESSES it with no controller change — the newest entry whose `from` is the pin is taken, the update renders the new tested digest, the next night reads Current. So option (a)'s cost is in the catalog. **Needs: a decision (STATUS, 2026-09-30).** | **WAITING-ON-OPERATOR — rank P2-MEDIUM; owner: VIKTOR decides, CC builds** |
| **R-741** | **[P3-LOW] For a few seconds after a fresh install, an `after_install` app answers its PUBLIC default login through the household's front door — the box changes the password only after the app already serves.** MEASURED 2026-09-30 on scratch guest 9202 (calibre-web, controller 0.283.1): polling `GET /opds` with `admin:admin123` (the image's README default) through traefik once a second from the deploy press: 404 until the route existed, **200 at 16:42:06**, 401 from 16:42:07 on (`after_install` done). The first install the same day logged the app started at 15:02:36 and `after_install … done` at 15:02:54 — a fixture probe at 15:02:52 still saw the new password refused, so that window was up to ~18 s. Decision 45 („no app is published with a login a stranger knows") holds after the window, not during it. Who can reach it: anyone who can resolve the app's name in those seconds; calibre-web is not behind the setup gate. Applies to every `after_install` app (calibre-web, mealie, wger, bookstack, claper, …) — not measured for the others. **Needs:** the route published only after `after_install` succeeds (the gate's own route hold, or traefik labels applied after), or the app behind the setup gate until then. `audits/more-night-apps-2026-09-30/A/A3-calibre-default-login-window.txt` | **READY — rank P3-LOW; owner: CC (controller)** |
| **R-742** | **[P3-LOW] zipline 4.8.0 cannot be reached from 4.6.1 in one step: it refuses to start until the database has run the release before it.** MEASURED 2026-09-30 on both venues, the same way: 4.6.1 → 4.8.0 — the new container restarts (exit 1) with `Error: cannot safely migrate from prisma to drizzle: expected migration 20260508022000_add_file_folder_created_at_index was not applied. To resolve this, repair the database with the previous (latest before this) Zipline release before upgrading` (bench `to-full.log`, 15×). **On box 9202 the product's guarded Update saw it unhealthy after 1 m 30 s and UNDID it by itself in 20 s — the previous version back on the data from before the update** (`box/zipline/step.txt`): the undo worked on a real upstream failure, not a drill. Bench verdict `failed`, box `undone`; nothing written. **Also found:** the zipline fixture had two faults, both fixed (`/` answers 301 → wait on `/api/healthcheck`; a wrong-password control first trips its login limit → 429 on the right one; the right password now goes first). **Needs:** the ladder climbs 4.6.1 → 4.7.x → 4.8.0 (two tested steps — the ladder exists for exactly this). `audits/more-night-apps-2026-09-30/bench/apps/zipline/`, `box/zipline/` **-- DONE 2026-09-30 (evening):** the two steps, each proven on both venues — 4.6.1 → 4.7.0 (catalog `a9700e2`; box 103.6 s, bench 0 kills) and 4.7.0 → 4.8.0 (`fb87030`; box 32.8 s, bench 0 kills). | **CLOSED 2026-09-30 — catalog `a9700e2` + `fb87030`** |
<!-- DUE-CHECKS-BEGIN — machine-readable. Parsed by scripts/due_checks_gate.py.
One row per dated check. The R-number must have a row above. Dates are UTC.