2026-09-24: 09 decisions 24-25, part 5 shipped; the whole-copy truth table; fourth suppression; rows; STATUS; evidence
gates / gates (push) Successful in 27s
gates / gates (push) Successful in 27s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
1 parent
500488cad6
commit
86c4b9a0d8
60 files changed
+4066
-57
No files matched your search
@@ -362,3 +362,17 @@ Compressed here to title, shipping version, evidence, and the sentences that sta
|
||||
| **R-640** | **A truncated PostgreSQL copy loaded with rc 0 into an empty database (P2, narrowed to the restore paths).** Closed in **v0.267.0**: `appbackup.CheckDumpComplete` (the engine's end marker); the unit restore and the off-site restore refuse before the first mutation, every replay checks again before any load. **Rule:** a dump is judged by its END — the header and a `CREATE TABLE` say nothing about whether it finished. | **CLOSED 2026-09-23 — three red-proofs + PROVEN-LIVE on 9202 (the household's restore button refused a half-length docmost copy; containers untouched; the whole copy then restored)** | `audits/night-2026-09-23/A2-*`, `E1-r640-live.*` |
|
||||
| **R-499** | **Every driveless app was told its data was „already in the full system backup (PBS)" (P2).** Closed in **v0.267.0**: the Tier-2 page's sentence has four branches from the box's own whole-system backup target (own drive / same disk / drive gone / cannot ask); „(PBS)" and „nincs külön teendő" only where true. | **CLOSED 2026-09-23 — two red-proofs; live on 9202 (the `unknown` branch, hu + en, matching `/api/storage/backup-target`)** | `audits/night-2026-09-23/A4-*`, `A7-*` |
|
||||
| **R-626** | **A removed app came back (P2).** Measured on v0.266.0, NOT reproduced: navidrome removed through the product, 390 s of `docker events` (the remove's destroy seen, no create), a controller restart at +150 s, a guest reboot after → no container, no volume. The two known creators (the restore/remove race R-633, the backup/deploy race R-634) are fixed. **Rule kept from the row:** a check that runs once, immediately, cannot see a thing created just after it — watch a window. Leftover found and filed: R-651. | **CLOSED 2026-09-23 — by measurement (a positive observable: the destroy events)** | `audits/night-2026-09-23/A3-*` |
|
||||
|
||||
## 2026-09-24 — the undo after a restore, the held app's page, the ladder (controller v0.268.0, hub v0.122.0, catalog `5ed599c`)
|
||||
|
||||
| Row | What | Closed | Full text |
|
||||
|---|---|---|---|
|
||||
| **R-658** | **After a restore, the undo copied NOTHING (P1).** v0.268.0 (`206b035`): the undo selects volumes from the rendered compose file (`DeclaredVolumeNames`: `name:` else `<project>_<key>`, each checked to exist); the label is a logged cross-check. The unit restore creates volumes WITH compose's project/volume/version labels (never a guessed config-hash); the remove counts unlabelled declared volumes. Live on 9202: restored under v0.267.0 → labels `null`; on v0.268.0 a failing update copied both by name and seeds A and B (B written after the restore) read back; a v0.268.0 restore → labels present, no compose warning. **Rule:** never select an app's volumes by the compose label. | v0.268.0, 2026-09-24 | `git show 500488cad673:documentation/backlog/OPEN-ITEMS.md` |
|
||||
| **R-659** | **A held app's page named a way back the restore refused (P1).** Operator ruling 2026-09-24 (`09` §3 decision 25, option A). v0.268.0 + hub v0.122.0: the hold names the newest WHOLE copy (`WholeOnTier` asks the refusal's own predicate); with none, `hold.update.no_whole_copy` in the household's language, no Mentések button, and `app_hold_no_whole_copy` (critical, operator-only). Live on 9202: round 11 reproduced (nextcloud, cut-off undo copy) — both languages, no button on either page, the event's R-620 line. **Rule:** a sentence may name a copy as a way back only when the restore for that copy would accept it. Left open beside it: R-661, R-666. | v0.268.0 / hub v0.122.0, 2026-09-24 | `git show 500488cad673:documentation/backlog/OPEN-ITEMS.md` |
|
||||
| **R-660** | **A held app also raised `app_start_failed` (P3).** v0.268.0: a fourth suppression set at `classifyRunStates` — the update-held apps (`backup.UpdateHeldStacks`; a RESTORE hold is not in it). Live: after the hold no `app_start_failed` in ~12 min of scans, while a throwaway stopped out of band raised exactly one. | v0.268.0, 2026-09-24 | `git show 500488cad673:documentation/backlog/OPEN-ITEMS.md` |
|
||||
| **R-651** | **A removed app left `applied-compose.yml` and `applied-meta/` (P3).** v0.268.0: remove deletes both (the catalog mirror and `hold-logs/` stay — the latter is evidence). Live: present before, gone after, on nextcloud and vikunja. | v0.268.0, 2026-09-24 | `git show 500488cad673:documentation/backlog/OPEN-ITEMS.md` |
|
||||
| **R-653** | **The memory watch wrote `proven` over a watch whose load never reached the app (P3).** Catalog `5ed599c`: `load_verdict` — `reached` only when at least half the requests got an HTTP answer, else the edge is `inconclusive`. Unit-tested and red-proofed; no bench run this session. | catalog, 2026-09-24 | `git show 500488cad673:documentation/backlog/OPEN-ITEMS.md` |
|
||||
| **R-656** | **The bench re-used an app's scratch drive folder (P3).** Catalog `5ed599c`: `clear_scratch_folders` removes the app's own `${HDD_PATH}`/`${USERDATA_PATH}`/`${IMPORT_PATH}` bind folders before FROM and says so; never a bare root, never outside. Unit-tested and red-proofed; no bench run this session. | catalog, 2026-09-24 | `git show 500488cad673:documentation/backlog/OPEN-ITEMS.md` |
|
||||
| **R-40** | **The update path could not express a multi-hop upgrade (P2).** Superseded by `09` §3 decisions 13–14 and shipped as §6.4 part 5 (v0.268.0): the catalog records every tested step with its own definition, and one press climbs one. **Rule kept:** a hop exists for a box only when the catalog holds it as a TESTED step — a >1-major catalog move without its intermediate steps is still a jump for a box below it. | v0.268.0, 2026-09-24 | `git show 500488cad673:documentation/backlog/OPEN-ITEMS.md` |
|
||||
| **R-663** | **Two catalog test suites had been red since the night of 2026-09-23 (P3).** Filed and fixed the same session: `test_gate_decoys.py` (kimai-db 11.6 → 11.8) and `test_ladder_writer.py` (navidrome 0.64.0 → 0.64.1) typed the live pins as literals; they now READ them. 84 decoy cases OK. **Rule:** a fixture that names a live pin reads it; the catalog moves under it every night. | catalog `5ed599c`, 2026-09-24 | this entry |
|
||||
|
||||
@@ -633,7 +633,6 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
|
||||
| **R-31** | **[P2-HIGH] Offsite provisioning is synchronous with no status affordance.** Save runs the Hetzner sync in-request, so the request can hit the nginx 504 **while succeeding server-side**: the operator cannot tell failed from slow, and a retry races the first attempt. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | Direction: make it async + a status card, reusing the proven **awaiting-card/poll idiom** (v0.138.0 escrow card). **Interim mitigation belongs in R-3 as an operator note: click once, wait, verify — do not re-click.** | CC |
|
||||
| **R-32** | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` | CC |
|
||||
| **R-35** | **Config-apply should not end the customer's session.** The offsite config push bumped `config_version` 10→11 at 16:54:58 and the controller self-restarted (container `StartedAt` 16:54:59Z, back up 16:55:02); in-memory sessions died with it and **customer zero was force-logged-out mid-flow**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size S, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | Direction: **hot-apply the offbox target** (no restart for a config the running process can adopt), or **persist sessions** across restart. The restart itself is by design — the collateral is not. Evidence `controller-log-full.txt` | CC |
|
||||
| **R-40** | **[P2-HIGH] The update path cannot express a MULTI-HOP major upgrade.** A template pin is a single value; the customer's update button pulls whatever the catalog now says. For apps whose upstream forbids version skipping this produces a broken upgrade. Nextcloud is explicit: *"You cannot skip major releases. Please re-run the upgrade until you have reached the highest available release."* Campaign 7 moved its template **31 → 34** (a fresh deploy validates fine — 302, 3/3 healthy), so an existing 31 customer pressing update would attempt a jump Nextcloud refuses. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-19, size M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | Origin: CAMPAIGN 7 (`audits/CAMPAIGN-7-catalog-sweep-2026-07-19.md` §7 F7). Not nextcloud-only — any app with sequential-major rules (gitea, tandoor, outline…) has the same shape. Directions: a per-app `upgrade_path:`/`max_hop:` in `.felhom.yml` that the update button walks in stages; or refuse-and-explain when the installed major is >1 behind; or pin an intermediate "stepping-stone" tag. **Until this exists, a >1-major catalog bump is safe for NEW deploys and unsafe for the update button** — which is exactly the asymmetry the campaign's MAJOR flag was meant to record but cannot enforce | CC |
|
||||
| **R-49** | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar` **823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar` **308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/` **18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-19, size S–M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` §4 (full byte breakdown).** This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. **Recorded, deliberately not changed** — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) `immich_ml_cache` — pure cache, strongest case; (b) the `postgres_data` volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) `upload/backups/`. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app | CC |
|
||||
| **R-76** | **FileBrowser-created folders break the setgid chain, and a drop-zone's mode is not stable** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-26, size S, roadmap state `idea (surfaced by the R-75 spike, 2026-07-26)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | Two related findings from `audits/SPIKE-catalog-data-paths-2026-07-26.md` P3/P5, both **pre-existing** and deliberately left alone by that spike. **(a)** FileBrowser Quantum 1.3.3 creates files `0644` and folders `0755` and does **not** propagate the setgid bit — even though the entrypoint wrapper's `umask 002` really is in effect (`/proc/1/status` `Umask: 0002`). Group inheritance itself works (a file uploaded into a 2775 group-100 dir landed group 100, not the process gid 1000), so the convention's *group* half holds and only its *mode* half is lost. The consequence is proven with a control: inside a UI-created `0755` folder a gid-1000 process's file landed group **1000**, while the identical write into the 2775 parent landed group **100**. So **any folder a customer creates through FileBrowser breaks the shared-group chain one level down.** Latent today — every userdata-touching catalog app that declares an identity declares uid/gid **1000**, the same uid FileBrowser runs as, so owner permissions mask it; it bites the day a content app runs as a different non-root uid with gid 1000. The comment at `infra/infra.go:156` is right that the image ignores `-e UMASK` but does not say t | CC |
|
||||
| **R-78** | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC |
|
||||
@@ -667,7 +666,7 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
|
||||
| **R-444** | **[P3-LOW] Nothing runs `pct fstrim` on the fleet, and demo-hp's thin pool was carrying ~23.8 GB of blocks the guest had already freed.** MEASURED 2026-09-01 during this spike's teardown: the run itself added ~1.05 GiB that `local-lvm` did not reclaim on delete (68.97% -> 70.91%); `fstrim` INSIDE the unprivileged container is refused (`FITRIM ioctl failed: Operation not permitted`, all three mounts); `pct fstrim 9201` from the PVE host then trimmed **30.2 GiB + 57 GiB** and took `local-lvm` to **26.78%** — **23.8 GB BELOW this run's own starting point**, i.e. the surplus was long-standing, not ours. **Why it is not merely housekeeping:** a thin pool that only ever grows can reach 100% from DELETED data alone, and a full thin pool takes every guest on the host read-only. demo-hp had 16.4 GB free before the trim. **Not urgent, and the row says so** — but the appliance has no periodic trim and no operator surface reports the gap between guest-free and pool-used. Owner: **CC.** `audits/SPIKE-app-update-2026-09-01.md` | **OPEN — rank P3-LOW; owner: CC** |
|
||||
| **R-445** | **[P3-LOW] Hub app telemetry survives the app's removal, so a 15-minute throwaway now sets a FLEET-WIDE memory recommendation.** MEASURED 2026-09-01: this spike's Phase 6 Nextcloud existed for ~15 minutes on demo-hp, spent part of it crash-looping, and was then removed with all volumes. The hub's `/apps/nextcloud` page still reports `Deployments`, `Avg Memory 208 MB`, `P95 Memory 280 MB` and **`Suggested Limit (P95x1.2) = 352 MB`**, plus three MariaDB `io_uring` rows under Known Issues attributed to demo-hp. **The suggested limit is an operator-facing recommendation derived from a sample that no longer exists anywhere** — and Nextcloud is a real catalog app whose limit someone may act on. **RETAINED DELIBERATELY BY THIS RUN, NOT CLEARED, and the reason is part of the row:** the hub offers `POST /apps/nextcloud/reset-telemetry` whose own confirm reads *"Delete all telemetry data for nextcloud? This cannot be undone."* — an irreversible write on the operator's surface, and the operator authorised Phase 6, not this. **The one-line command is recorded in the audit doc so it is a decision, not a task.** The general question is the row: should telemetry for an app with zero live deployments age out, or be excluded from the suggestion? Owner: **VIKTOR rules, CC implements.** `audits/SPIKE-app-update-2026-09-01.md` | **OPEN — rank P3-LOW; owner: VIKTOR rules, CC implements** |
|
||||
| **R-446** | **[P2-MEDIUM] „Naprakész" can be FALSE, and the badge that says it cannot tell.** Slice 2 (controller v0.233.0, 2026-09-02) compares the RECORDED image reference per compose service against the reference the current template pins, and **queries no registry** — deliberately: a customer's box must not depend on reaching eight upstream registries to render a page (`felhom-controller/controller/internal/web/updatebadge.go`, `compareInstalledToTemplate`). **For the 23 floating pins that comparison is blind by construction:** `postgres:16-alpine`, `mariadb:11.6` and 21 others can carry an identical reference over an image that has moved. **MEASURED, not theorised — spike §5 found `mariadb:11.4` and `mariadb:12.3` had BOTH already moved upstream while two fully-pinned CONTROLS held.** So `romm` and `bookstack` on demo-hp would read „Naprakész" over a database engine build that is not the one the catalog now resolves to. **This is a KNOWN LIMITATION OF A SHIPPED FEATURE, filed the same session rather than left implicit**, and it is stated in the same words in `architecture/09-update-architecture.md` §8.1 and in the controller's `README.md`. The close is a digest comparison against the registry, which needs a network call, a cache and a failure posture — it is not a one-liner and it is not slice 2's job. **Depends on R-440**, whose fix (stop floating) would remove the problem instead of measuring it — take that route first if it is available. `architecture/09-update-architecture.md` **MEASURED 2026-09-21, and the blind spot is not one or two pins.** `audits/UPDATE-ARC-STATE-2026-09-21.md` §3.3: the catalog carries **10 floating pins of 66** (recounted — the old "23" was stale), and **6 of the 7 measurable engine pins have been repushed upstream since the catalog set them** — `postgres:16-alpine` (8 apps), `postgres:15-alpine`, `redis:7-alpine` (6 apps), `mariadb:11.4`, `mariadb:12.3`, `postgis:16-3.5-alpine`; only `mariadb:11.6` has not. The 8th (immich's own ghcr build) is UNMEASURED — ghcr exposes no anonymous last-modified timestamp. **So on demo-hp today four apps read „Naprakész" over a database engine image that has demonstrably moved.** The fix does NOT need the box to query a registry: the catalog can record each pin's digest at push time (`check-image-resolvable.py` already resolves it) and the box compares digests. Put to the operator as `09` §3b **Q6**, recommended YES — the cheapest real improvement on the arc's list. **— UPDATE NIGHT 2026-09-21:** **MEASURED ON A BOX 2026-09-21 (update night, leg B8), and it REFINES the row in two ways rather than merely confirming it.** §8.1's numbers came from a registry sweep on DooPlex; this is the same question asked of a customer-shaped box, where the badge actually renders. On guest 9202, `docmost`'s two floating pins were read as `installed_images` records them and compared against the upstream digests measured the same night: `postgres:16-alpine` → **`sha256:721873c34ceb9…` on the box and `sha256:721873c34ceb9…` upstream**, and `redis:7-alpine` → **`sha256:858f009f9709c…` both sides**. **Identical. So the badge „Naprakész" is TRUE for this box**, and the app reads correctly. **(1) The defect's size is set by INSTALL AGE, not by the catalog.** A floating pin is wrong only for a box that pulled BEFORE the tag moved; a box deployed after the repush holds the current image and its badge is right. R-446's "six repushed pins" measured the tag against the date the CATALOG set it, which is the right measure for *the catalog* and not for *a box*. **(2) The producer Q6 needs ALREADY EXISTS on the box.** `installed_images` records a real `digest` per service (`installed.go` §7.1) — the box knows exactly what it is running. What it cannot do is COMPARE, because the catalog carries no digest to compare against. That is Q6's proposal, and this is a concrete confirmation that only the catalog half is missing. Evidence: `audits/update-night-2026-09-21/23-B8-floating-pin.txt`. **-- RULED 2026-09-23 (`09` §3 decision 17):** YES — the catalog records the image digest of every pin at push time; the box compares against it and, where the catalog carries one, pulls **that exact image**, which makes a floating tag reproducible, not only the badge honest. *Pull-by-digest while the definition names a tag is a claim to verify in the build, not a ruling on mechanism.* **-- 2026-09-23:** pull-by-digest MEASURED on 9202 — Docker and Compose both pull and run `redis:7-alpine@sha256:858f…` and refuse a digest that does not exist. **Build trap, read from source:** `splitImageRef` returns "unorderable" for any ref containing `@` (`stacks/updateorder.go:134`), so a digest-carrying pin must have its digest split off before ordering or every such app reads Unknown. `09` §6.4 part 6. **— NIGHT 2026-09-23:** the CATALOG half of the close shipped: every ladder entry records the digest the registry served for each `to` ref (`scrLine truncated
|
||||
| **R-450** | **[P2-MEDIUM] UPDATE ARC SLICE 6 — a version sequence: automatic WITHIN a major, never ACROSS one, and an engine change gets its OWN edge.** The first half is an operator ruling of 2026-09-02 and its justification is R-449's measurement: a cross-major jump can be refused by the app itself and cannot be undone. **The second half is a rule recorded now, while it is cheap:** an engine change must never be bundled with an app version bump. `bookstack`'s `0b73e5e` moved the application 25.02.2 → 26.05.2 **and** MariaDB 11.6 → 12.3 in one commit — **two migrations behind one edge**, and an unreadable failure when it breaks. Needs a catalog-side convention and, eventually, a gate. `architecture/09-update-architecture.md` §6 **HALF SHIPPED 2026-09-21 (catalog `5ff36d098cbc`): the second half — an engine change gets its OWN edge — is now ENFORCED** by `check-engine-major.py`, which refuses a commit moving a MariaDB major together with any other image move in that template, naming what it was bundled with. The FIRST half (automatic within a major) is Slice 6 and needs four operator answers — `09` §3b **Q1–Q4**, with the shape it would take in `09` §6.2. **The urgency is now measured:** 46 of the catalog's 58 exact pins are behind upstream and **39 of those are within a major** — the population the 2026-09-02 ruling already says may move without a human. **-- RULED 2026-09-23 (operator, `09` §3 decisions 11–15):** Q1–Q4 answered. The update is a leg of the backup chain after off-site and before the full-system backup (11); automatic with a per-box switch ON by default (12); **the TEST decides, not the tag** — the box applies every step the catalog holds because the catalog holds only tested steps, and `CompareImageRefs` moves to the catalog gate (13, REPLACES "never across a major"); a box behind climbs **one tested step at a time** (14); **the box UNDOES a failed update itself** — old definition + the pre-pin safety dump + health check again, HOLD only if the undo fails (15, REPLACES §6.1's no-auto-undo). The undo and the ladder were SPIKED the same day before any build (`audits/update-rulings-2026-09-23/`); build order and costs in `09` §6.4. **-- SPIKED 2026-09-23 (`audits/update-rulings-2026-09-23/`):** the undo works by hand on three real migrating edges and needs eight product additions (R-637..R-642); **the ladder is measured absent** — one press on a box two steps behind jumped vikunja 2.3.0 → 2.5.0 in 9.5 s and 2.4.0 never ran, and the box cannot see intermediate steps at all because its catalog clone is `--depth 1` (`sync.go:283`/`:300`, one commit visible on both demo guests). The ladder's recommended format is an `update_ladder:` list in `.felhom.yml` with each intermediate step's own definition, NOT the git history (romm's image-moving commit is the definition that OOM-looped). The chain's update leg has ≤15 min as ruled (R-643). Build order `09` §6.4. **— NIGHT 2026-09-23: `09` §6.4 part 4 SHIPPED** (catalog `6db08a5`): the test record `update_ladder:` + two gates + the only writer + the 21-move backfill; 12 more steps published with records. Parts 5 (the box climbs), 6-box-half and 7 remain; the romm press on demo-hp showed today's jump live — 5.3.0 → 5.3.1 AND mariadb 11.4 → 11.8 in one press (both tested steps; `done`). | **READY TO BUILD — owner: CC; `09` §6.4 part by part, each part returns to the operator for go/no-go** |
|
||||
| **R-450** | **[P2-MEDIUM] UPDATE ARC SLICE 6 — a version sequence: automatic WITHIN a major, never ACROSS one, and an engine change gets its OWN edge.** The first half is an operator ruling of 2026-09-02 and its justification is R-449's measurement: a cross-major jump can be refused by the app itself and cannot be undone. **The second half is a rule recorded now, while it is cheap:** an engine change must never be bundled with an app version bump. `bookstack`'s `0b73e5e` moved the application 25.02.2 → 26.05.2 **and** MariaDB 11.6 → 12.3 in one commit — **two migrations behind one edge**, and an unreadable failure when it breaks. Needs a catalog-side convention and, eventually, a gate. `architecture/09-update-architecture.md` §6 **HALF SHIPPED 2026-09-21 (catalog `5ff36d098cbc`): the second half — an engine change gets its OWN edge — is now ENFORCED** by `check-engine-major.py`, which refuses a commit moving a MariaDB major together with any other image move in that template, naming what it was bundled with. The FIRST half (automatic within a major) is Slice 6 and needs four operator answers — `09` §3b **Q1–Q4**, with the shape it would take in `09` §6.2. **The urgency is now measured:** 46 of the catalog's 58 exact pins are behind upstream and **39 of those are within a major** — the population the 2026-09-02 ruling already says may move without a human. **-- RULED 2026-09-23 (operator, `09` §3 decisions 11–15):** Q1–Q4 answered. The update is a leg of the backup chain after off-site and before the full-system backup (11); automatic with a per-box switch ON by default (12); **the TEST decides, not the tag** — the box applies every step the catalog holds because the catalog holds only tested steps, and `CompareImageRefs` moves to the catalog gate (13, REPLACES "never across a major"); a box behind climbs **one tested step at a time** (14); **the box UNDOES a failed update itself** — old definition + the pre-pin safety dump + health check again, HOLD only if the undo fails (15, REPLACES §6.1's no-auto-undo). The undo and the ladder were SPIKED the same day before any build (`audits/update-rulings-2026-09-23/`); build order and costs in `09` §6.4. **-- SPIKED 2026-09-23 (`audits/update-rulings-2026-09-23/`):** the undo works by hand on three real migrating edges and needs eight product additions (R-637..R-642); **the ladder is measured absent** — one press on a box two steps behind jumped vikunja 2.3.0 → 2.5.0 in 9.5 s and 2.4.0 never ran, and the box cannot see intermediate steps at all because its catalog clone is `--depth 1` (`sync.go:283`/`:300`, one commit visible on both demo guests). The ladder's recommended format is an `update_ladder:` list in `.felhom.yml` with each intermediate step's own definition, NOT the git history (romm's image-moving commit is the definition that OOM-looped). The chain's update leg has ≤15 min as ruled (R-643). Build order `09` §6.4. **— NIGHT 2026-09-23: `09` §6.4 part 4 SHIPPED** (catalog `6db08a5`): the test record `update_ladder:` + two gates + the only writer + the 21-move backfill; 12 more steps published with records. Parts 5 (the box climbs), 6-box-half and 7 remain; the romm press on demo-hp showed today's jump live — 5.3.0 → 5.3.1 AND mariadb 11.4 → 11.8 in one press (both tested steps; `done`). **— 2026-09-24: `09` §6.4 PART 5 SHIPPED** (controller v0.268.0 `206b035`, catalog `5ed599c`): one press = one tested step, each step's own definition at `templates/<app>/steps/<StepKey(to)>.yml`, proven live on 9202 (romm 5.3.0/11.4 → 5.3.1/11.4 → 5.3.1/11.8 in two presses, `audits/ladder-2026-09-24/partD/`). Left in this row: part 6's box half, part 7 (the automatic leg), part 10 (PostgreSQL majors). | **READY TO BUILD — owner: CC; `09` §6.4 part by part, each part returns to the operator for go/no-go** |
|
||||
| **R-451** | **[P3-LOW] UPDATE ARC SLICE 7 — a fleet sweep: the operator can SEE, and MOVE, how far behind every box is.** Slices 1 and 2 make one box's state visible on that box's own pages. The operator has no fleet view, and **it is not derivable from what is already reported: the hub's report payload carries container name, state, CPU and memory, and NO image field at all** (spike §5, which is why Peti's box could only be recorded UNKNOWN). So this is a hub-side change as well as a controller one. Rank LOW today because the fleet is two enrolled boxes; it rises with the fleet. `architecture/09-update-architecture.md` §6, §8.4 **BOTH SIDES VERIFIED 2026-09-21, and it is cheaper than this row implies.** The controller's payload carries no image (`internal/report/types.go` L98–103) and the hub's `Store.SaveReport` (`hub/internal/store/store.go:965`) denormalises only container **counts** — but **the hub stores the raw report JSON whole**, so a new controller field lands there the day it is sent. What is missing is the denormalisation and the page, not the transport. Shape in `09` §6.2–6.3; the payload question is `09` §3b **Q7**. **-- RULED 2026-09-23 (`09` §3 decision 18):** the report carries, per compose service (database included), the installed reference, the catalog reference and the badge state. **Built later, when the fleet grows** — Q7's recommendation, confirmed. | **RULED — build deferred until the fleet grows; owner: CC** |
|
||||
| **R-454** | **[P3-LOW] Five `internal/web` test files have been `gofmt`-unclean for an unknown length of time, and nothing notices.** MEASURED 2026-09-02: `gofmt -l controller/internal/web/` reports `backups_split_test.go`, `claim_code_naming_test.go`, `disk_health_test.go`, `r400_debug_routes_test.go`, `recovery_test.go` — at the **baseline** commit `960d29b0612c`, i.e. not introduced by v0.233.0 (both files added that day are clean). **`go vet` does not check formatting and `controller_gates.py` has no formatting gate**, so the only thing that would ever surface this is someone running `gofmt -l` by hand, which is how it was found. **Not reformatted in the same session, deliberately** — the minimal-changes rule, and a five-file whitespace commit inside a feature release makes that release's diff unreadable. **Small, and the cost of NOT having the instrument is the row:** the count can only grow, and every future `gofmt -l` run produces noise that hides a real one. Fix is two lines: a `gofmt -l` gate in `controller_gates.py` plus one formatting commit, in that order (the gate first, so the commit is provably complete). Owner: **CC.** | **READY — rank P3-LOW; owner: CC** |
|
||||
| **R-457** | **[P3-LOW] A test that hardcodes a date AND asserts an age derived from it is green on the day it is written and red the next morning — one instance PROVEN, six candidate files named.** MEASURED 2026-09-03: `TestGroupD_BadgeRendersOnBothSurfaces` (shipped the previous day in v0.233.0) pinned a fixture `catalog_since: "2026-07-18"` and asserted the rendered string `"Frissítés elérhető — 46 napja"`. **The pure badge tests inject a clock; the RENDER test does not and cannot** — it goes through the production templates, which call the funcmap entry `updateBadge`, which reads `time.Now()`. The suite was green on 2026-09-02 and **FAILED on 2026-09-03** with *"the behind badge is missing"* on both surfaces, because the true answer had become 47. **Fixed by DERIVING the fixture** — `catalog_since` is computed as *today minus 46 days*, so the test asserts the real number through the real clock and cannot rot. **THE CLASS, which is why this is a row and not just a fix:** a clock-reading test that also carries a date LITERAL is a bomb with a fuse of unknown length, and the suite being green is not evidence it is defused — it is evidence the fuse has not burned down yet. **NAMED AS UNCHECKED CANDIDATES, NOT ACCUSED** — six other test files contain both a `20xx-xx-xx` literal and `time.Now()`: `internal/backup/offbox_test.go`, `internal/web/handler_export_upload_test.go`, `internal/web/r103_tier2_action_test.go`, `internal/web/dashboard_backup_card_test.go`, `internal/web/async_restore_test.go`, `internal/stacks/installed_test.go`. Mixing the two is not itself a defect — it is one only where a literal feeds an assertion evaluated against the real clock — so each needs reading, which is a sweep and not this session. **The instrument that would end the class:** run the suite once under a faked future date in CI and see what turns red. Owner: **CC.** `felhom-controller` v0.234.0 CHANGELOG | **READY — rank P3-LOW; owner: CC** |
|
||||
@@ -803,16 +802,16 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
|
||||
| **R-643** | **[P2-MEDIUM] The ruled chain leaves the automatic update leg AT MOST 15 MINUTES a night.** FOUND 2026-09-23 while writing the build plan for `09` §3 decision 11 (*updates after the off-site copy, before the full-system backup*). The off-site leg starts at W+105m (`cmd/controller/main.go:961`) and the full-system backup's gate opens at W+2h (`quiesce/quiesce.go:656`, span to W+6h); the legs are clock-scheduled, not chained. One step takes ~1 min when it works and ~2–6 min when it fails and is undone. Options and the recommendation (the full-system backup waits for the leg inside its own window; the leg stops starting steps at W+5h) are in `09` §6.4. **-- RULED 2026-09-23 (`09` §3 decision 20):** the full-system backup waits for the update leg inside its own window; the leg stops starting new steps at W+5h. Built with `09` §6.4 part 7. | **RULED — build with §6.4 part 7; owner: CC** |
|
||||
| **R-644** | **[P3-LOW] `gokapi` on scratch guest 9202 is crash-looping — 329 restarts by 2026-09-23 07:51 UTC, *password does not appear to be a SHA-1 hash* — and the controller still lists it deployed.** OBSERVED at the start of the 2026-09-23 session, not caused by it. The twenty-eight walk's teardown (2026-09-22) removed a `gokapi` container left by R-633 by name; a `gokapi` is running again, recorded `deployed: true`. Not investigated (scope). Likely the R-633/R-634 shape — a restore-then-remove race leaving a record — and a scratch-box fact, not a customer one; filed so the next drill does not read it as its own. | **OPEN — P3; owner: CC; investigate before the next drill on 9202** |
|
||||
| **R-645** | **[P3-LOW] Lifting an update hold by the operator CLI lets the recovery unit be re-captured with the FAILED new definition within seconds — the copy the hold sentence names is overwritten.** MEASURED 2026-09-23 on 9202 during the undo bake-off: docmost was held at 07:59:16Z after a failed 0.95.0 → 0.96.0 update; `--clear-restore-hold docmost` + the controller restart it requires ran at ~07:59:23Z, and at **07:59:26Z** the controller logged *Recovery unit captured for docmost* — the unit's `compose/docker-compose.yml` now named `docmost/docmost:0.96.0`, the version that had just failed. The hold sentence had pointed the household at that unit („saját meghajtó, … 09:55"). The hold is what keeps the nightly legs off a held app (`isHeld`, v0.238.1); once it is lifted by hand, the checksum-gated refresh sees a changed definition and captures it. **Who it hits:** an operator who lifts a hold to inspect or repair, before restoring. With the undo (R-637) a failed update no longer holds unless the undo also fails, so the path is rarer — it does not go away. Candidate shapes, none chosen: the CLI refuses to lift an UPDATE hold (only a restore lifts it); or the lift also puts the pin back; or the capture skips an app whose pin is not what it is running. Evidence: `audits/undo-bakeoff-2026-09-23/docmost-40-undoF.txt` (the invalid run) and README §"Three things". **-- 2026-09-23 (controller v0.263.0):** the undo never reads the unit, so this no longer affects the automatic undo; it still affects an operator who lifts a hold by hand before restoring. | **OPEN — P3; owner: CC** |
|
||||
| **R-651** | **[P3-LOW] A removed app leaves `applied-compose.yml` and `applied-meta/` behind in its stack directory.** MEASURED 2026-09-23 night on 9202 (v0.266.0, R-626's run): after `POST /api/stacks/navidrome/remove` answered 200 with `verified: true`, `/opt/docker/stacks/navidrome/` still held `applied-compose.yml` and `applied-meta/` written by the deploy (the directory itself is the catalog mirror and belongs there). Seen again after every remove of the night (`after remove: leftovers='/opt/docker/stacks/<app>'`). **Consequence not measured:** whether a later reinstall of the same app reads a stale `applied-meta/` (the undo's OLD `.felhom.yml`, v0.263.2) before its own deploy overwrites it. **Needs:** the remove deletes both; a test that removes, reinstalls, fails an update and asserts the undo uses the reinstall's file. Evidence: `audits/night-2026-09-23/A3-r626-verdict.json`. | **READY — P3; owner: CC (controller)** |
|
||||
| **R-652** | **[P3-LOW] The memory watch counted the kernel's file cache as the app's memory.** MEASURED 2026-09-23 night on the bench: nextcloud 34.0.4 read **100 %** of its 1 GiB and immich's PostgreSQL **100 %**, each with **0** kernel `oom_kill`s — `memory.peak` includes page cache, which the kernel drops before it kills anything. Under the watch as built (R-635 follow-up) both would be marked `memory_tight`, and the gate would demand a raised `mem_limit` — a customer-box capacity figure — for cache. **Done the same night (09 §3 decision 22, CC — operator may reverse):** the watch samples the app's own memory (`anon` of `memory.stat`) every 15 s; the mark and the ladder's `memory_peak_pct` read it where measured; the cgroup peak stays beside it (`memory_cgroup_peak_pct`). **Open:** romm's backfilled entry carries M1's 80.9 % cgroup peak (measured before the anon sample existed) — re-measure it on its next step; and decide whether an app whose anon is low but whose cgroup stays pinned at its limit (cache thrash) should be marked at all. Evidence: `audits/night-2026-09-23/apps/nextcloud/bench-1024M/`, `apps/immich/bench-noanon/`. | **READY — P3; owner: CC (catalog harness)** |
|
||||
| **R-653** | **[P3-LOW] The memory watch's load did not reach box-fixture apps: every request errored.** MEASURED 2026-09-23 night: the load generator followed each app's redirect to `https://<sub>.gate.invalid`, which does not resolve, so ghost's and nextcloud's first ten-minute watches recorded **11 797 and 9 427 requests, every one `err`** — memory measured, but idle. Fixed the same night in `upgrade-test.py` (no redirects; the app's own Host header): ghost's re-run served **10 538 × 200**. **Why a row though fixed:** the harness wrote `proven` over a watch with zero effective load and nothing flagged it. **Needs:** the watch reports INCONCLUSIVE when fewer than half its requests reached the app. Evidence: `apps/ghost/bench-noload/`, `apps/ghost/bench/`. | **READY — P3; owner: CC (catalog harness)** |
|
||||
| **R-654** | **[P3-LOW] opengist 1.15 moved every page under `/-/` — a household's `/login` bookmark answers 404 after the update.** MEASURED 2026-09-23 night: 1.13 serves `/login`, `/register`, `/all`; 1.15.2 answers **404** on all three and serves `/-/login`, `/-/register`, `/-/all`; `/` redirects to `/-/all`. The app, its data and its probe (`/healthcheck`) are fine, and a household arriving at the root lands correctly — only a deep link breaks. 1.15 also marks its session cookie `Secure`. **Needs:** a line in opengist's `app_info` if the operator wants households told; nothing in the product. Evidence: `apps/opengist-oldfixture/`, `apps/opengist/`. | **READY — P3; owner: operator (copy decision) / CC (writes it)** |
|
||||
| **R-655** | **[P2-MEDIUM] adventurelog v0.13.0 cannot become healthy in the catalog's template — its update is undone on every box.** MEASURED 2026-09-23 night, both venues. v0.13.0's FRONTEND image adds its own `HEALTHCHECK` (`node -e fetch('http://127.0.0.1:3000/health')`), and `/health` answers **503 `{"ok":false,"backend":"unreachable"}`** unless the backend's `/health/` answers OK to the frontend's own request (read from the image's `django-proxy` chunk: `fetch(${getServerEndpoint()}/health/)`). On the bench the backend served `/api/` 200, the seed read back, the migration ran (6 lines) — and the frontend stayed `unhealthy` for 420 s; on 9202 the guarded Update went `verifying` → **`undoing` → `undone`** (under the drill's 90 s timeout). **MEASURED LATER THE SAME NIGHT — two causes, and the first hypothesis was wrong.** (1) The backend's `/health/` answers `200 {"ok": true}` to the frontend (read on a fresh v0.13.0 install); the unhealthy frontend is **our template's own healthcheck override**, `["CMD", "/nodejs/bin/node", …]` — v0.12.1's distroless image keeps node there, v0.13.0 moved it to `/usr/bin/node`, and Docker's health log reads `exec: "/nodejs/bin/node": no such file or directory` 19 times in a row. (2) With the override removed, a second bench run hit a harder wall: v0.13.0's backend runs `download-countries` at EVERY start, fetching world data from the internet, and a cut-short download (`ijson.common.IncompleteJSONError: Incomplete JSON content`) crash-loops the entrypoint — the backend never became healthy in 15 min. The first bench run's download had succeeded. **So v0.13.0's boot depends on an outside download, and whether an update succeeds depends on it too.** **The catalog did NOT move adventurelog.** **Needs:** the override dropped (or pointed at `/usr/bin/node`) in the SAME commit as the image move; and a measured answer on whether `download-countries` can be skipped or pre-seeded (an env switch, or the data in the volume) before the edge is re-proven on both venues. Evidence: `audits/night-2026-09-23/apps/adventurelog/`. | **READY — P2; owner: CC (catalog)** |
|
||||
| **R-656** | **[P3-LOW] The bench keeps an app's drive folder between runs, so a re-run of the same app starts on the last run's files.** MEASURED 2026-09-23 night: nextcloud's second bench run never installed (`occ status: installed: false` for 5 min) because `/srv/felhom-gate/hdd/appdata/nextcloud` still held the first run's `config/` and data — `compose down -v` removes named volumes, not bind-mounted host folders. Cleared by hand and re-run. Other re-runs tonight (romm's engine step after its app step, komga, immich) reused their drive folders too; their seeds live in databases in named volumes, so their verdicts stand, but the class is unguarded. **Needs:** the bench clears the app's scratch drive folder before FROM (and says so), or runs each edge under a fresh folder. | **READY — P3; owner: CC (catalog harness)** |
|
||||
| **R-657** | **[P2-MEDIUM] "Remove the app, keep my data", then install it again: nextcloud never installs, and the box only says „unhealthy".** MEASURED 2026-09-23 night on 9202 (v0.267.0): nextcloud was removed through the product keeping its drive folder (the remove with data was refused — R-442's fail-closed guard, as on every drill on 9202 — and the product's keep-data remove taken). An hour later a fresh install of nextcloud on the same box: the template binds `${HDD_PATH}/appdata/nextcloud` to `/var/www/html/data`, the kept folder still holds `admin/`, `appdata_*`, `.ncdata` and a 145 MB `nextcloud.log`, and the image's installer loops **„Login is invalid because files already exist for this user — Retrying install..."**; `occ status` reads `installed: false`. The controller records the deploy as done and the app as `unhealthy`; nothing tells the household that their kept files are what blocks the new install, or what to do. **Why it matters:** keep-data is the choice the product OFFERS a household at remove time — and for nextcloud the kept data makes the app uninstallable. **Needs:** decide the product's promise for a reinstall over kept data, per app class (adopt the data? refuse with a sentence? offer to move it aside?); at minimum a deploy-time refusal or warning when the app's drive folder is not empty. Evidence: `audits/night-2026-09-23/chaos/00-nextcloud-reinstall-over-kept-data.txt`. | **READY — P2; owner: operator (the promise) / CC (the build)** |
|
||||
| **R-658** | **[P1-HIGH] After a restore, an app's volumes lose their compose labels — so the next update's automatic undo copies NOTHING.** FOUND 2026-09-23 night by the chaos hour on 9202 (v0.267.0). The unit restore recreates each named volume with a bare `docker volume create <name>` (`internal/backup/restore.go:154`), which carries no `com.docker.compose.project` label; the undo selects the volumes it copies by exactly that label (`internal/stacks/undo.go:156`, `ProjectVolumes`). **Measured, with a control:** the three apps restored in the chaos hour (docmost, navidrome, vikunja) have **0** labeled volumes of 3 / 2 / 2; the three never restored (romm, nextcloud, adventurelog) have all of theirs labeled. **Consequence, seen live in round 9:** vikunja's broken update logged `the undo copy will hold 0 named volume(s), 0.0 MiB`, failed health, and was „UNDONE" — the OLD binary put back on the NEW version's migrated data, with no copy restored. vikunja happens to start on migrated data (the morning spike), so it read back; **an app whose old version refuses migrated data would HOLD, and one that starts but mis-reads it would run on a database the old code does not understand, reported as a successful undo.** The same label filter is used by the remove path (`delete.go:994`, `projectVolumes`) — **measured at teardown:** the remove of the three restored apps DID delete their volumes (compose's `down --volumes` finds them by name) but answered `volumes_removed: []` — the report is wrong, not the act. **Needs (not fixed tonight — one controller release per session):** the restore creates volumes WITH the project's labels (or `compose create` does it); the undo and the remove resolve an app's volumes from its compose file's declared names, not only the label; a test that restores, then fails an update, and asserts the undo copied every declared volume. Evidence: `audits/night-2026-09-23/chaos/round-09.*`, `chaos/09-volumes-unlabeled-after-restore.txt`. | **READY — P1; owner: CC (controller)** |
|
||||
| **R-659** | **[P1-HIGH] A held app's page names a way back that the restore then REFUSES — and no screen offers another.** MEASURED 2026-09-24 00:00 on 9202 (v0.267.0), chaos round 11: nextcloud's update failed, its undo copy was cut off (the drill removes the finished-marker, `09` §6.1a's measured case), and the product HELD it — honestly, in both languages: *„… Visszaállítható a Mentések oldalon ebből a biztonsági mentésből: saját meghajtó, 2026-09-23 22:55 — ez a másolat csak a beállításokat és az adatbázist tartalmazza, a fájlokat nem."* The household presses exactly that restore (`POST /backup/restore`, snapshot `helyi`, the ONE copy offered) and is refused: *„Ez a mentés nem tartalmazza az alkalmazás fájljait, ezért nem állítjuk vissza az adatbázist föléjük — a fájlok így a helyükön maradnak. A fájlok a távoli másolatból állíthatók vissza: … „Teljes visszaállítás (fájlok + adatbázis)"."* This box has no remote copy. The app stays stopped and held; Update and Start refuse a held app. **So the hold sentence and the restore disagree about the same copy, and on a box without an off-site tier nothing on any page brings the app back.** Both refusals are individually right (the restore will not pour a database over files it cannot also restore; the hold will not start a half-migrated app); together they strand a file-leg app. navidrome, held the same way in round 8, came back through the same button because its copy was accepted. **The brief's stop rule fired here** (a state neither the product nor its screens can recover): Part D had finished all twelve rounds; nothing further was injected. **Needs:** the hold must name only a route the restore will accept for that app (or say plainly that none exists on this box and the operator is needed); and a decision on what a household with no off-site tier may do for a file-leg app whose undo failed. Evidence: `audits/night-2026-09-23/chaos/round-11.*`, `chaos/round-11-recovery.*`. | **READY — P1; owner: operator (the promise) / CC (the build)** |
|
||||
| **R-660** | **[P3-LOW] A held app also raises `app_start_failed`, seconds after its own `app_update_held`.** MEASURED 2026-09-23 night on 9202 (chaos rounds 8 and 11, from the controller's full log): `app_update_held (error)` at 20:56:04 and 21:42:39, then `app_start_failed (warning)` at 20:56:15 and 21:42:52 — each at the next status refresh after the hold stopped the app. The `DROPPED event` line names no app, so the attribution is by timing, not by name; nothing else changed state at those moments. The hold is a deliberate stop BY THE PRODUCT, a fourth way to stop an app that `classifyRunStates`' three suppressions (`08` §5) do not know — the same class as the nightly volume-dump false alarm (fixed v0.224.0: "a third way to stop an app needs a third suppression set"). The operator gets two alarms for one fact. **Needs:** a held app is not "down" to the app-down ladder; a test that holds an app and asserts no `app_start_failed`. Evidence: `audits/night-2026-09-23/E8-events-*`. | **READY — P3; owner: CC (controller)** |
|
||||
| **R-661** | **[P2-MEDIUM] The second drive holds a file app WHOLE, and no single action brings the app back from it.** MEASURED FROM SOURCE 2026-09-24 (v0.267.0/v0.268.0): the Tier-2 mirror carries the unit AND the drive files, but „Teljes visszaállítás" there is `RestoreTier2Unit` → `RestoreFromRecoveryUnitAt`, whose R-538 guard refuses any app with declared drive files; the file restore (`RestoreTier2Files`) only ADDS missing files and replays no database. So for nextcloud/immich/paperless-ngx/calibre-web the second drive is not a way back after a failed update — which is why R-659's hold (v0.268.0) names only the off-site copy for those apps (`backup.WholeOnTier`). **Needs:** a decision on a Tier-2 "files + database" restore (the off-site full restore's shape), or a ruling that the second drive is files-only for these apps. Evidence: `audits/ladder-2026-09-24/README.md` (the truth table); `backup/update_guard.go` R-659 block. | **WAITING-ON-OPERATOR — P2; owner: operator (the promise) / CC (the build)** |
|
||||
| **R-662** | **[P3-LOW] The unit restore's documented second step („csak az adatbázist és a beállításokat", `accept_missing_files=1`) is dead.** MEASURED FROM SOURCE 2026-09-24: no template sends the field (`grep accept_missing_files internal/web/templates` → 0), and when a request does, the web handler skips its own refusal but calls `RestoreFromRecoveryUnit`, which passes `UnitRestoreOptions{}` — so the backup layer refuses anyway. Nothing ever calls `AcceptMissingFiles: true` outside a test. **Needs:** either wire it end to end (with its own confirm, R-48) or remove the flag and the comment that promises it. Evidence: `web/handlers.go` backupRestoreHandler; `backup/restore_unit.go` L226. | **READY — P3; owner: CC (controller)** |
|
||||
| **R-664** | **[P3-LOW] A ladder step has no `.felhom.yml` of its own.** v0.268.0 pins a step's own COMPOSE file (`steps/<key>.yml`); its health probe, memory check and the recorded `applied-meta/` still come from the catalog's CURRENT `.felhom.yml`. Harmless while a step's probe and limits equal the head's (true for all 8 step files today); wrong the day a step's probe differs. Also: the log line for a step pin says „pin advanced to the catalog's current definition" (it names the step file one line earlier). **Needs:** `steps/<key>.felhom.yml` (or the probe in the ladder entry) and the log line naming the source. Evidence: `audits/ladder-2026-09-24/partD/10-romm-two-steps.log`. | **READY — P3; owner: CC (controller + catalog)** |
|
||||
| **R-665** | **[P3-LOW] An update pressed soon after a restore was judged with the RESTORED `.felhom.yml`.** OBSERVED 2026-09-24 on 9202 (v0.268.0), not diagnosed: vikunja restored from its unit at 06:19; the drill catalog's `.felhom.yml` named probe port 8999 (a deliberately failing edge); the periodic probe used 8999 until 06:20:41, and the update pressed at 06:21:07 probed 3456 (the unit's own file) and ended `done`. The restore apparently writes the unit's `.felhom.yml` into the stack dir, and the next catalog sync has not yet put the catalog's back. Consequence bounded (the OLD probe, ≤ 15 min), but an update's verdict should not depend on how long ago a restore ran. **Needs:** a measurement of what the restore writes and when the sync overwrites it. Evidence: `audits/ladder-2026-09-24/partA/03-why-g-was-done.txt`. | **READY — P3; owner: CC (controller)** |
|
||||
| **R-666** | **[P3-LOW] A held app whose box has no whole copy is told „ne törölje az alkalmazást" — and its card still offers Eltávolítás.** v0.268.0 (R-659) hides the Mentések button beside the no-whole-copy sentence; the Remove button on the stacks card stays (the household may remove its own app — a promise, not CC's to take away). Measured live on 9202: the page and the sentence disagree. **Needs:** an operator word — hide Remove while support is informed, or soften the sentence. Evidence: `audits/ladder-2026-09-24/partB/01-round11-reproduced.json`. | **WAITING-ON-OPERATOR — P3; owner: operator** |
|
||||
| **R-667** | **[P2-MEDIUM] A crash-looping app whose container reads `running` for a moment between restarts never reaches the crash-loop alarm.** MEASURED 2026-09-24 on 9202 (v0.268.0): `gokapi` (R-644) at **385 restarts**, `restarting_since` = 06:46:30Z at 06:47 — the clock resets whenever a scan catches the container up — while the dead-app heartbeat said *„4 deployed app(s) evaluated, 0 currently down"*. `Stack.CrashLooping` needs `crashLoopAfter` (5 min) of CONTINUOUS `restarting`. Likely also the unattributed second `app_start_failed` at 06:31:41Z (a scan that caught gokapi `exited`): the dropped-event line names no app, so this is inference. **Needs:** crash-loop judged on the container's RestartCount growth over a window, not on an uninterrupted state; a test with a flapping fixture. Evidence: `audits/ladder-2026-09-24/partB/04-states-and-banner.txt`, `02-r660-positive-observables.txt`. | **READY — P2; owner: CC (controller)** |
|
||||
<!-- DUE-CHECKS-BEGIN — machine-readable. Parsed by scripts/due_checks_gate.py.
|
||||
One row per dated check. The R-number must have a row above. Dates are UTC.
|
||||
Clearing a row means the check was DONE and its result recorded in that R-row —
|
||||
|
||||
Reference in new issue
Block a user