D63 live: R-1256 H4 + H5 failed stop proven on Tester 1 (test build); D62 proven live; R-1340 filed (hold mail tells the option-C story for any cause); 07 §6.3 [FACT]; evidence audits/d63-2026-10-29/

documentation/backlog/OPEN-ITEMS.md

@@ -154,6 +154,7 @@ stopping line that lies.
 | **R-1337** | Apps & catalog | P3 | **Calibre-Web (v4.0.8, the newest) loads a bad Brotli decoder: its `cps.py` maps Python Brotli 1.2.0 (fixed 1.2.1), Pillow's libbrotlidec 1.2.0 and Debian's libbrotlidec 1.1.0** — read from the process's maps on demo-hp 9201 (`audits/runtime-fixes-2026-10-28/calibre-web-brotli.txt`). Exposure: the Python decoder is urllib3's, for replies to the app's own outbound calls (book metadata), not for what a visitor sends — the Paperless shape (R-1331). No fixed release exists. `audits/runtime-fixes-2026-10-28/TABLE.md` | OPEN — filed 2026-10-28 (runtime fixes) | upstream: an image with Brotli ≥ 1.2.1 | take that release as a lane step; Debian's libbrotli1 fixed version is not known (R-1327's table has no Debian line) | CC |
 | **R-1338** | Apps & catalog | P3 | **Wanderer's Meilisearch fills whatever memory limit it gets: 92–100 % anon of 256 MB (bench and 9202), still 88.8 % of 384 MB on the bench (2026-10-28)** — it sizes its indexing memory from the memory it sees (its default: 2/3 of the available RAM), so a higher limit alone does not end the `memory_tight` mark of Wanderer's step. No kill in any run. `audits/dockerhub-bigsteps-2026-10-28/wanderer/` | OPEN — **2026-10-28 night: MEASURED — it is the measurement, not Wanderer, and the cap is not needed** (`audits/night-2026-10-28/r1338-meili/`; `09` §3 decision 22 refined). Meilisearch frees with MADV_FREE: the given-back pages stay in `anon` but sit on the inactive FILE list (`file` ~1 MB, `inactive_file` ~290 MB) and the kernel takes them first — 83–96 % `anon` on the bench (no swap) AND on 9202, with and without the cap, 0 kills; it uses ~100 MB. The evening's 17.7 % was the container after the swap, not the cap. **Fixed in the tools:** the mark reads the live anon (catalog `scripts/memfigure.py`, the larger of two estimates, errs high; `upgrade-test.py`, `box_walk.py` soak; `test_memory_live.py` red-proved). Re-run: bench Wanderer v0.20.0 → v0.21.2 `proven`, **no marks** (search 83.3 % anon, 27.7 % live); 9202 95.2 % anon, 25.1 % live. **Prepared, NOT pushed** (the night's catalog fence): the tool change + Wanderer's entry without `memory_tight` (`audits/night-2026-10-28/catalog-morning/`; the catalog gates passed in a scratch clone). **Earlier, 2026-10-28 evening: `MEILI_MAX_INDEXING_MEMORY` = 128 MiB (a third of 384 MB) measured, NOT pushed** (`audits/gitea-wanderer-2026-10-28/wanderer-meili/`): on the bench, under the harness's load, the search container peaked at 17.7 % (84 MB; 88.8 % without the cap, same load); on 9202 the cap was in the container (read back) and it still held 95.7 % anon (363 MB steady over 6 min, no kill). The box result decides: not shown to help, so not pushed. Also: the catalog's ladder has no entry type for a re-measure of a definition change at the SAME digests (`ladder.py`: a re-test whose digest equals its digest_from tests nothing new), so the v0.21.2 entry's `memory_tight` mark cannot be replaced without an image move. | the catalog push (the morning list) | push the prepared catalog change (the live figure + Wanderer's entry re-measured, `memory_tight` off — an existing entry's memory fields edited at the SAME digests, which the gates accept); then close | CC |
 | **R-1339** | Storage & devices | P4 | **At every start the controller logs „FileBrowser: drive …/userdata/<app> not mounted — skipping userdata skeleton" for per-app folders on a drive that IS mounted** (scratch 9202, 2026-10-29 03:35 UTC: nine lines, `scratch_hdd` a live mountpoint, mp8). `skipFileBrowserPath` (`controller/internal/web/handlers.go:3766`) asks whether the folder ITSELF is a mountpoint, and a folder under a mounted drive never is. No data is lost: the file browser shows the folders through the drive's own mount (`/srv/scratch_hdd`). The cost is a false „not mounted" in the log of a box whose drive works, which misleads a diagnosis. Fix: ask about the drive the folder lives on, with a test that a sub-folder of a mounted drive is not skipped. Evidence: `audits/night-2026-10-28/part5-claims/selfheal/controller-after-reboot.log`. | OPEN | — | the fix above, with its red-first test | CC |
+| **R-1340** | Backup & restore | P3 | **The operator's „HELD STOPPED" mail for a mixed restore says the DATABASE restore failed and was rolled back, and that the files and volumes are the snapshot's — whatever the cause.** Measured 2026-10-29 08:18 UTC on Tester 1 (D63's H5 live proof, test build): Radicale (no database) — the volume could not be removed (not written into), the put-back's stop failed, nothing moved; the mail read „its off-site database restore failed. The database was rolled back to the pre-restore copy and the live definition written back, but its files and named volumes are the snapshot's". The text (`controller/cmd/controller/main.go:1555`) predates the D20/D22 put-back, when the only mixed hold was R-893 option C. Support reads a wrong account of what is on the box. The household's sentence is conservative and stays. Fix: the operator text names the cause it was given (the replay error, the put-back step that failed, whether the copy is kept and where) instead of the option-C story; a test per cause. Evidence: `audits/d63-2026-10-29/run-h5.log`. | OPEN | — | the fix above, red first | CC |
 
 
 ## App updates — 10 rows (P2 1, P3 9)
@@ -187,7 +188,7 @@ stopping line that lies.
 | **R-548** | Backup & restore | P3 | **[P3-LOW] The whole-guest backup’s LOCAL tier cannot fit on a small-system-disk box, and will retry on that tier for ever.** MEASURED 2026-09-17 (chaos night) on `tester-1-022354`: a whole-guest backup wrote a **~29 GB source** (`mp0` = `local-lvm:vm-9201-disk-1`, 70 G provisioned, 40.58 % used, `backup=1`) into `pve-root`, which on a 32 GB system disk is **14 GB total with ~4.9 GB free**. Two samples thirty seconds apart showed the archive growing ~497 MB while free space fell ~475 MB — **~16 MB/s, i.e. under four minutes to a full `/`** on the nested PVE. **The product’s behaviour is correct and legible throughout:** it failed the tier and said which one — `whole_guest_backup_failed` (error, operator-only): „Whole-guest backup FAILED on the local tier — retrying with backoff (next attempt in 15m0s)” — its status surface agreed (`target_id:"local"`, `success:false`, `size_bytes:0`), and the **off-site tier then ran from the same snapshot and succeeded in ~8½ minutes, encrypted to ep0, consuming no local disk at all**. So the data still left the house. What is filed is the loop: on a box shaped like this the local tier can **never** succeed, and it keeps retrying on a backoff for ever, burning I/O and risking `/` each time. **Honest caveat:** the 32 GB system disk is this drill’s fixture choice, so the row is conditional on disk size — but nothing in the product checks whether the local target could ever hold the source before trying. **Fix shape:** compare the source size against the target’s free space before starting the local tier and skip it with a clear reason, rather than discovering it at ~16 MB/s. Evidence: `audits/evidence-chaos-night-2026-09-17/round-6.txt`. **-- 2026-09-30, the class on a real box: demo-hp's whole-guest LOCAL tier was refused by the space preflight (R-685) 10 times since 2026-09-27** („local has 12.1 GiB free; the last archive of guest 9201 was 8.9 GiB, so a new one needs about 12.1 GiB") — its newest local archive was 2026-09-29 04:42, 30 h old at the day's read (the weekly PBS tier, last 2026-09-24, is its whole copy meanwhile). The refusal is correct and named; what is missing is that nothing gives the box the room back (the host's `local` holds the 8.9 GiB archive of the only guest plus templates on a 39 GB root). `audits/pg-last-six-2026-09-30/C/`. | **READY — rank P3-LOW; owner: CC** **2026-10-23 night (demo-hp, read only):** the local tier was refused 4 times (00:59–02:39 UTC) by the space preflight — `local has 8.5 GiB free; the last archive of guest 9201 was 6.5 GiB, so a new one needs about 9.2 GiB` — and the controller stopped 11 apps for about a minute before each refused try (then backed off 30 m → 2 h). The 2026-10-22 run had fit; `local` now also holds 6.4 GB of installer ISOs (1.27.1–1.30.0, the newest uploaded for yesterday's proof installs). Two halves: the box (free space on demo-hp: remove the superseded ISOs — a demo-hp housekeeping step, not done tonight) and the product (ask the agent's preflight BEFORE stopping the apps — R-921's shape for the local tier). **2026-10-23 day — the box half DONE:** the three superseded installer ISOs (1.27.1, 1.28.0, 1.29.0; no guest referenced them) removed from demo-hp's `local` by name (`pvesm free`); `/` 8.5 GB → 14 GB free (`df` before/after, `audits/day-2026-10-23/09-demohp-space.txt`), above the 9.2 GiB the preflight asks. 1.30.0 stays. **The product half — the proposed fix (R-921's pattern):** the agent already computes the space verdict (R-685: free vs the last archive × the margin); it serves it per guest on `GET /backup/status` next to `host_busy` — e.g. `local_space: {fits, need_bytes, free_bytes}` — and the controller reads it before any stop: when the local tier cannot fit, it stops no app, records the tier as skipped for space (the same contention record as R-921, one alarm after the ceiling), and still runs the other due tiers. An agent without the field keeps the old order. Tests to write first and red-prove: agent `TestBackupStatus_R548_LocalSpace`, controller `TestR548_NoStopWhenTheLocalTierCannotFit`. Rides the next agent + controller pair. | — | — | CC |
 | **R-933** | Backup & restore | P3 | **Mirror the images of every catalog app we support into our own registry, before scale.** Operator, 2026-10-12 (S8, `09` §3 decision 203): *„Will need to mirror the apps we support later, just to be sure."* Today a restore re-pulls each image from its maker; a version the maker deleted cannot start (the accepted limit, `07` §6.6, R-698 closed). Size it first: the supported catalog's images (every ladder version a box can hold) × their size; then where the copy lives (the DooPlex registry is Tier 2, precious — the copy must not make DooPlex a single point of failure for every household's restore) and how a restore falls back to it. | **OPEN** — NEW 2026-10-12 | the size measurement | Measure the total size of the supported catalog's images; then a one-page design (where, how a restore falls back, cost) | CC (measure + design), operator (the go) |
 | **R-1255** | Backup & restore | P3 | **Escrow edge cases (review round 2, HUNT-3 H3/H4):** the R-1176 carry moves the newest kept package even when it holds no repository password (`hub/internal/store/escrow_carry.go:100-110` checks `identity_blob` only), while the right one stays under the deleted host (`store.go:3533-3534`); two set-asides on the same UTC day reuse a name and the purge can delete a still-stored copy's package (`hub/internal/offsitekeys/offsitekeys.go:378-385`; `store.go:4295-4300`). Static. | OPEN — **fixed on main 2026-10-28 night, not released** (felhom.eu `1ee6d5a5`, CI 2472; store tests red first; `03` §8 [DESIGN] 2026-10-28): the carry takes the newest retained row with an identity AND a repository-password hash (the newest row only when none has a hash — a legacy package, as before); a set-aside is `<repo>.orphaned-<YYYYMMDDTHHMMSSZ>`; a retained row is purged only when it was tagged no later than its path's deletion (`repo_tagged_at`, new column). No live proof: a hub deploy is not a night's. | the hub release (attended) | release it (the morning list), read the store back after one carry and one abandonment, then close | CC |
-| **R-1256** | Backup & restore | P3 | **Restore edge cases (review round 2, HUNT-1 H4/H5/H6/H8):** (H4) an off-site restore whose definition write fails AFTER the compose was written restarts the app on the snapshot's compose (`internal/backup/offbox_reconstitute.go:979-986`; `cmd/controller/main.go:3318-3331`); (H5) a put-back whose stop fails goes on and can merge (`restore_crash_putback.go:155-158`; `restore.go:218` ignores `volume rm -f`), then removes the copy; (H6) a second-drive mirror can start between the web refusal and the restore's own record (`web/restore_wizard.go:226`; `opstatus.go:56-66`), and the install-time kept-data load checks only RestoreStatus/IsRunning (`api/kept_install.go:89-92`); (H8) an unused restore copy is never cleaned up. Static. **2026-10-28 night:** all four fixed, each test red first (controller `342a394`, CI 2469 green; `07` §6.3 [DESIGN] 2026-10-28 — an unused copy goes after 14 days, decided by CC unattended). Live on scratch 9202 2026-10-29: H5's busy volume (not written into, the live data put back, no hold; `audits/night-2026-10-28/r1256-restore/h5-live/`) and H8 (the 03:29 sweep removed the two old copies and kept the young one; `…/h8-live/`). H4 and H5's failed stop: tests only; H6: tests only (a live press cannot tell old from new). | OPEN — fixed on main, not released | the controller release (morning list) | the release; then an attended live proof of H4 and H5's failed stop, or revert those two branches of `342a394` | CC |
+| **R-1256** | Backup & restore | P3 | **Restore edge cases (review round 2, HUNT-1 H4/H5/H6/H8):** (H4) an off-site restore whose definition write fails AFTER the compose was written restarts the app on the snapshot's compose (`internal/backup/offbox_reconstitute.go:979-986`; `cmd/controller/main.go:3318-3331`); (H5) a put-back whose stop fails goes on and can merge (`restore_crash_putback.go:155-158`; `restore.go:218` ignores `volume rm -f`), then removes the copy; (H6) a second-drive mirror can start between the web refusal and the restore's own record (`web/restore_wizard.go:226`; `opstatus.go:56-66`), and the install-time kept-data load checks only RestoreStatus/IsRunning (`api/kept_install.go:89-92`); (H8) an unused restore copy is never cleaned up. Static. **2026-10-28 night:** all four fixed, each test red first (controller `342a394`, CI 2469 green; `07` §6.3 [DESIGN] 2026-10-28 — an unused copy goes after 14 days, decided by CC unattended). Live on scratch 9202 2026-10-29: H5's busy volume (not written into, the live data put back, no hold; `audits/night-2026-10-28/r1256-restore/h5-live/`) and H8 (the 03:29 sweep removed the two old copies and kept the young one; `…/h8-live/`). H4 and H5's failed stop: tests only; H6: tests only (a live press cannot tell old from new). | OPEN — fixed on main (controller `342a394`), not released. **All four proven live:** H5 busy volume + H8 sweep (9202, night 2026-10-28); **H4 and H5's failed stop on Tester 1, 2026-10-29 08:10 / 08:18 UTC** (D63, a `-tags faultinject` test build; `audits/d63-2026-10-29/`). D62 set the sweep age to 30 days (`e3fce09`). Closes with the controller release | the controller release (morning list) | the release; then an attended live proof of H4 and H5's failed stop, or revert those two branches of `342a394` | CC |
 | **R-1259** | Backup & restore | P3 | **The DooPlex→ep0 copy jobs: three edges and tomorrow's clock change (review round 2, HUNT-2 H6–H8, HUNT-4 H1).** The release copy keeps the newest 5 agent versions whether vouched or not, so five unvouched releases drop the vouched one from ep0 (`scripts/release-copy/felhom-release-copy:199,218,237`); its `rsync --delete` removes old versions during the transfer and mirrors a shrinking registry as deletions (`:190,237`); the hub-DB push has no lower age bound (`scripts/hub-db-backup/felhom-hub-db-backup:41,49-50`). VERIFIED on DooPlex: on 2026-10-25 the 02:30 CEST push runs at 00:30Z (`systemd-analyze calendar`) and the hub's 02:00 slot at 01:00Z (Go takes the repeated hour's second pass), so the push re-sends the 20:00 snapshot and the off-site hub DB is ~13.5 h old until 08:30. Operator data only. | OPEN | — | always keep the vouched agent; prune after a verified copy; a future-age bound; the push slot after the hub's in UTC | CC |
 | **R-1274** | Backup & restore | P3 | **The restore refusal tells the household to „turn on remote backup, or connect a second drive” when the box has both and their first copy has not run yet.** Catalog sweep 2026-10-24, Grimmory and Calibre-Web: „Mentés most” makes the local copy only (`07` §8, by design), so the drive files had no copy yet; the refusal's sentence is `restore.refuse.files.none` (`felhom-controller/controller/internal/web/handlers.go:1835`) whatever the box has. The Hungarian sentence is frozen by `i18n_go_parity`, so the fix is a NEW key („the first copy is made in tonight's backup”) chosen when a tier is set up but holds nothing yet. | OPEN | — | The new key + the branch + a test | CC |
 | **R-91** | Backup & restore | P4 | Old 13 GB datastore copy at `/srv/pbs-felhom` on ep0's root disk **Checked from source 2026-10-05 (burn-down round 2):** Gate is long past (row waits on demo-felhom's first post-migration PBS backup, migration 2026-07-27). Last positive record of the copy: audits/CAMPAIGN-9-restore-proof-2026-07-28.md:759 'ep0 : /srv/pbs-felhom rollback copy intact (13G)'; CONTEXT.md:3666 still says it is 13 G of dead weight awaiting R-91. No later record of deletion found (grep srv/pbs-felhom across felhom.eu). Deleting is on ep0 (protected) and needs an operator word. | WATCHING | demo-felhom's first **post-migration** PBS backup | Delete once it lands; fix `CONTEXT.md:1018` same commit | CC |