Immich repair on every box: rollout read-back, R-1021 closed, R-1007 dated line, F12 fleet delivery, STATUS, report

documentation/backlog/CLOSED-ITEMS.md

@@ -26,11 +26,17 @@
 
 ---
 
+## 2026-10-14 — the Immich repair on every box; the shared checkout cleaned
+
+| Row | What | Closed | Evidence |
+|---|---|---|---|
+| **R-1021** | **`build-hub.sh` built STALE hub code when its `git pull` failed — it only warned** (P3) | CLOSED 2026-10-14 — FIXED: `build-hub.sh` v1.24.0 fetches, pulls `--ff-only`, and refuses (exit 1) unless the tree is clean and `HEAD == origin/main`; the commit label is read after the pull; no override (no runbook needs one); `FELHOM_HUB_REPO_DIR` builds from another clone under the same rule. `test_build_hub_tree_gate.sh` (stub docker): 6 cases right; the old script gets 5 wrong. The shared checkout's leftovers MOVED (operator ruling „move", nothing deleted) to `/mnt/5_hdd/felhom.eu/leftovers/2026-10-14/` (its README: all but one already in git; the Facebook `first-post/` folder is not in git and is the operator's to decide); the checkout is clean at `origin/main` | felhom.eu `621783b3`; `audits/immich-repair-2026-10-14/E/bh-new.txt`, `bh-old.txt` |
+
 ## 2026-10-14 — Immich repairs its broken photos after a power cut
 
 | Row | What | Closed | Evidence |
 |---|---|---|---|
-| **R-1007** | **After a power cut Immich shows photos as done that are broken, and nothing repairs them** (P3) | CLOSED 2026-10-14 — decision 236 (operator): the box asks Immich through Immich's OWN job queue (no key into the household's account; every user's photos), for the asset ids of the zero-byte thumbnail/preview files only, then reads Immich's own missing-files report with a read-only session. Controller v0.311.0 (`after_unclean_boot`, once per start after an UNCLEAN boot, after the 15-min crash grace), hub v0.150.0 (`app_repaired_after_crash`, operator-only, counts only), catalog Immich block (proven on `immich-server:v3.2.4` only). **Proven live on demo-hp 9201:** a real cut mid-upload (15:25:04 UTC) left 41 photos with empty files; 41 of 41 repaired in 10 s; 120 control files untouched; 0 originals lost, so the household's lost-photo note was not exercised live (unit- and 9202-tested). **Reasoning kept:** a household-made Immich API key reaches only the admin's photos — a family member's photo was refused (measured); Immich's nightly „missing thumbnails" reads only its database, never the disk | controller `8d862e4` + `3b85a0c`, hub `822d0393`, catalog `6a1bb04`; `audits/immich-repair-2026-10-14/` |
+| **R-1007** | **After a power cut Immich shows photos as done that are broken, and nothing repairs them** (P3) | CLOSED 2026-10-14 — decision 236 (operator): the box asks Immich through Immich's OWN job queue (no key into the household's account; every user's photos), for the asset ids of the zero-byte thumbnail/preview files only, then reads Immich's own missing-files report with a read-only session. Controller v0.311.0 (`after_unclean_boot`, once per start after an UNCLEAN boot, after the 15-min crash grace), hub v0.150.0 (`app_repaired_after_crash`, operator-only, counts only), catalog Immich block (proven on `immich-server:v3.2.4` only). **Proven live on demo-hp 9201:** a real cut mid-upload (15:25:04 UTC) left 41 photos with empty files; 41 of 41 repaired in 10 s; 120 control files untouched; 0 originals lost, so the household's lost-photo note was not exercised live (unit- and 9202-tested). **Reasoning kept:** a household-made Immich API key reaches only the admin's photos — a family member's photo was refused (measured); Immich's nightly „missing thumbnails" reads only its database, never the disk **2026-10-14 (later): rolled out to all demo boxes** — golden 0.311.0 vouched, global floor 0.311.0 / MinAgent 0.155.0; N100, demo-hp and Tester 1 run controller 0.311.0 + agent 0.156.1; **gate added:** catalog `unclean-boot-images` refuses an image bump the repair was not proven on (catalog `8f5ad8a`). | controller `8d862e4` + `3b85a0c`, hub `822d0393`, catalog `6a1bb04`; `audits/immich-repair-2026-10-14/` |
 
 ## 2026-10-14 — the estimator takes the second machine's figures, raise only
 

documentation/backlog/OPEN-ITEMS.md

@@ -300,7 +300,6 @@ stopping line that lies.
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
-| **R-1021** | Process & tooling | P3 | **`build-hub.sh` builds STALE hub code when its `git pull` fails — it only warns.** 2026-10-14: the shared checkout `git/felhom.eu` was behind `origin/main` (`52b6599d` vs `822d0393`) and its pull aborted on untracked leftovers (`audits/sizing-2026-10-13/`, `audits/usenet-2026-10-12/`, a Facebook folder, a modified `walk.py`) — the script would have shipped hub 0.149-era code labelled 0.150.0. This session built from its own clean clone instead (a copy of the script with only `REPO_DIR` changed). | **OPEN** — filed 2026-10-14 | operator: whose are the leftovers in the shared checkout | CC: make the build refuse unless the source tree is clean and `HEAD == origin/main` (the clean-tree gate, in the script); the operator or their owner cleans the shared checkout | CC |
 | **R-992** | Process & tooling | P3 | **The drill-catalog route that `09` §6.5 documents cannot be followed: `admin/app-catalog-drill` is PRIVATE and no box has catalog credentials.** Measured 2026-10-12 while publishing the Usenet pair: the box's clone fails with `fatal: could not read Username for 'https://gitea.dooplex.hu'` (exit 128), the Gitea API reports the drill repo `private=true` and the live catalog `private=false`, and NEITHER the current controller.yaml on 9202 NOR any of its 30 saved copies has ever carried `git.username` or `git.token` (the controller supports both — `internal/config/config.go` GitConfig). The route worked for the campaigns of 2026-10-10 and the torrent pair earlier the same day, so the repository's visibility changed between then and now. **Worked around, not fixed:** that campaign staged its two templates in the box's own catalog cache (`audits/usenet-2026-10-12/tools/stage_in_cache.py`), which is where the sync copies a catalog change anyway, with the sync interval raised to 12h for the duration and rescan used in place of sync. The real fix is an operator decision: either a read-only catalog credential on the scratch box (a secret on a Tier-0 guest) or making the drill repository readable. Until then every new-app campaign hits this. | WAITING-ON-OPERATOR | the operator's choice between a box credential and a readable drill repo | Decide; then either set `git.username`/`git.token` on 9202 from the credentials store, or flip the drill repo, and correct `09` §6.5 either way | operator |
 | **R-1017** | Process & tooling | P3 | **A comment names a guard test that does not exist.** `felhom-controller/controller/internal/appbackup/paths.go:58` says „`TestNoBareDrivePathToUserdataDir` is the guard that keeps the count from growing”; no test of that name (or `NoBareDrive*`) exists in the repo (grep 2026-10-14; positive control: `TestRemoveStack_R800_UserdataKeptIsNamed` is found). A comment asserting an invariant with no test pinning it — the class in the `felhom-testing` skill. | **READY** — filed 2026-10-14 | — | Write the guard (a test that fails when a bare drive path reaches `UserdataDir`), red-proof it, or correct the comment | CC |
 | **R-1022** | Process & tooling | P3 | **Tester 1 follows the drill catalog and carries two test drives — teardown owed (operator ruling (e), 2026-10-14).** Steps, only on the operator's word: (1) remove the test apps (Sonarr, Radarr, qBittorrent, Transmission, Plex, Jellyfin) through the dashboard WITHOUT „delete data” (R-1027); (2) decommission Media A and Media B through Tester 1's dashboard; (3) `repoint.py restore` — `controller.yaml` from `controller.yaml.pre-media-2026-10-14`, read back byte-identical, `repo_url` the live catalog; remove `drill-catalog.git` from the controller volume; (4) on demo-hp: `qm set 341 --delete scsi2,scsi3`, then delete the two volumes BY NAME (`nvme-scratch:341/vm-341-disk-3.raw`, `-disk-4.raw`) — never `qm destroy`; (5) VM memory unchanged (8192, not raised); (6) drill repo reset to live `main`. Tools: `audits/media-all-drives-2026-10-14/``tools/`. | **WAITING-ON-OPERATOR** — filed 2026-10-14 | the operator's teardown word | Run the six steps; quote each read-back | CC |