Rehearsal: B8 all seven restore paths pass (NFC/NFD: both names return); B6 #2 power cut mid-restore → R-1195 (P2, proven live: the app runs on the backup's data, the page says only interrupted); B6 #5 slow line; register 173 -> 174

documentation/backlog/OPEN-ITEMS.md

@@ -151,7 +151,7 @@ stopping line that lies.
 | **R-683** | App updates | P3 | **[P3-LOW] Watch: after a power cut during an update's health check, the hold named an HOUR-OLD second-drive copy, not the one the update's own backup should have just made.** 2026-09-24 chaos round 3 (nextcloud, `backup_max_age: 1m`): no `backing-up` phase was seen and the hold named Tier 2 at 13:04 for an update pressed at 14:04; the pre-cut controller log was lost with the container (the runner now saves it at arm time — R-320). Round 11, the same action without a power cut, named a fresh 14:34 copy and logged the Tier-2 copy. The sentence was TRUE (it named the copy it offered); the question is why the update did not back up first. Not reproduced; watch the next power-cut drill. `audits/night-2026-09-24/E/round-03*.json`, `E/round-11-controller-pre.log` | **OPEN — P3; owner: CC (watch)** | — | — | CC |
 | **R-785** | App updates | P3 | **[P3-LOW] SparkyFitness is pinned 11 releases and a major behind upstream (v0.17.3; upstream v1.7.3, v1.6.0 dated 2026-07-24).** READ 2026-10-01 (`audits/visitors-2026-10-01/C/bench/C1-previous-tag.txt`). **Needs:** an update walk 0.17 → 1.x through the ladder (bench + box), after R-784 is decided. | **OPEN — rank P3-LOW; owner: CC (after R-784)** | — | — | CC |
 
-## Backup & restore — 41 rows (P1 2, P2 9, P3 18, P4 12)
+## Backup & restore — 42 rows (P1 2, P2 10, P3 18, P4 12)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -166,6 +166,7 @@ stopping line that lies.
 | **R-1180** | Backup & restore | P2 | **The nightly second-drive mirror (`rsync --delete`) can run DURING a restore from that same mirror and delete files not yet copied back.** `RunAllTier2` (`tier2.go:479-520`) takes no backup flag and does not ask about a running restore; the whole restore holds only the backup flag (`tier2_whole.go:301`), the night leg only `LegLock` (`cmd/controller/main.go:1348-1352`). A long restore of a photo library at 03:30, or 15 min after a missed night, overlaps. PROVEN against production code with the real `rsync -a --delete` mirror (`audits/dataloss-hunt-2026-10-20/C3-T1-mirror-during-restore/`): with the backup flag held by a restore, `RunAllTier2` still mirrors, and the mirror goes from [a,b,c,d] to [a,b] — the two files the restore was bringing back are gone from the only copy. Data-loss hunt 2026-10-20 (C1 reviewer 4, T1). | OPEN | — | Fix: the mirror leg skips (and says so) while any restore runs; test with the existing tier-2 seams | CC |
 | **R-1181** | Backup & restore | P2 | **A missing or emptied app folder deletes its second-drive copy within one night; for OPTIONAL folders (audiobooks, comics, ROMs, Immich's external library) that copy was the only one.** A missing declared leg is skipped (`tier2_capture.go:55-61`, silent for optional legs) and then reconciled away (`tier2.go:454`); an emptied one is mirrored with `rsync -a --delete` (`:744-751`). PROVEN for a missing folder against production code (`audits/dataloss-hunt-2026-10-20/C3-D2-missing-leg/`): the mirror's `photos` deleted, run „ok", warning only „nem kerültek a másodlagos mentésbe". 07 §8.2 chose „data legs are not guarded" for a leg that LEAVES the class set; a declared leg that is MISSING is not that case. Optional legs are in no off-site tier (07 §6.2). | WAITING-ON-OPERATOR | decision: keep a missing leg's mirror (and say so) until the household releases it? And: does an accidental folder deletion need a second-drive history for optional legs? | Operator: answer in STATUS | operator |
 | **R-1182** | Backup & restore | P2 | **Abandoning ONE set-aside off-site copy deletes EVERY retained key package of the customer, including ones that open other set-aside copies still stored.** hub `api/handler.go:626-627` → `PurgeSupersededEscrowForCustomer` (`store.go:4185-4190`) deletes customer-wide; the comment right above (`:4176-4179`) states the invariant this breaks. CONFIRMED in code (the SQL deletes every retained row of the customer, live and deleted hosts alike; nothing ties a row to the abandoned path). The R-1176 reinstall makes a second set-aside copy likely. Data-loss hunt 2026-10-20 (C1 reviewer 3, K2). | OPEN | — | Fix: purge only the package whose key opens the abandoned path (record the path or repo id with each retained row); hub store test as red-proof | CC |
+| **R-1195** | Backup & restore | P2 | **A power cut in the middle of a full off-site restore can leave the app running on the BACKUP's data while the page says only „the restore was interrupted, start it again" — the household's newer data is gone and nothing says so.** PROVEN LIVE in the rehearsal 2026-10-20 (`audits/rehearsal-2026-10-20/B6-failures/s2-power-cut-mid-restore/`): BookStack, a page added after the backup; „Teljes visszaállítás"; power cut at 14 s (the app stopped at 8 s, the database starting for the replay) → after power-on the crash recovery restarted the app (`appstop_marker.go`), recorded „A visszaállítás megszakadt (a doboz újraindult) — indítsd el újra." and pushed `restore_interrupted`; BookStack ran on the backup's 5 pages, the newer page gone; no hold; the undo copy written before the change (`offbox_restore_undo.go`) was not used. A cut at 6 s (before data moved) behaved correctly. This is R-1188 (a) proven; R-1188 keeps its other items. | OPEN | — | On start after an interrupted restore: put the undo copy back (D20's „exactly as it was") or hold the app and say „a visszaállítás félbeszakadt — az adatok a mentés állapotában lehetnek"; a test that kills the controller between the volume step and the replay | CC |
 | **R-49** | Backup & restore | P3 | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar` **823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar` **308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/` **18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-19, size S–M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: wasted space and transfer, no data risk; needs a capture-set ruling, not a sale blocker.** **READY 2026-10-20 — measured and planned, nothing changed; the decision is the operator's (STATUS D21).** Re-measured on 9202 (Immich v3.2.4): off-site ~1.33 GB = model cache 824 MB (62 %), raw database folder 283 MB (21 %), Immich's own backups 115 MB (9 %), logical dump 54 MB, originals 49 MB, thumbnails 2 MB — 102 MB must survive. A: leave out all three re-fetchable parts (−92 %; needs a never-run restore from the dump alone on the vectorchord image). B: only the model cache (−62 %; needs one restore with an empty cache). Pick: B now, A's other parts after their proof (`audits/night2-2026-10-20/P4/immich-offsite-plan.md`). | — | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` §4 (full byte breakdown).** This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. **Recorded, deliberately not changed** — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) `immich_ml_cache` — pure cache, strongest case; (b) the `postgres_data` volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) `upload/backups/`. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app | CC |
 | **R-314** | Backup & restore | P3 | **`StopAbandon` has no web route — a customer who telephones is served by a command line.** `--abandon-stop` exists on the controller binary (`cmd/controller/main.go:86`) and refuses rather than silently no-opping, which is right. But there is no handler: the only in-product way to cancel a countdown is the customer finding their recovery code (Scenario G cancels it at the moment the code proves they still have it). **An operator who is telephoned instead has to reach a shell on the customer's machine.** Used this session on the operator's ruling, container stopped first so the running controller could not overwrite `settings.json` from memory — a sequencing subtlety that is itself an argument for a route **-- 2026-10-08 design:** `audits/day-2026-10-08/design-R-314-279-177.md` — the box's own 14-day countdown has no operator door (the hub's 7-day phase does, decision 74); one door for R-314 + R-279: a fixed list of operator actions carried in the report reply. Question D1 on STATUS's decision sheet. **-- 2026-10-08 14:16 operator ruling D1 (`09` §3 decision 185):** yes — option A, a fixed list of operator actions carried in the box's report reply. | **VERIFY — 2026-10-09: DELIVERED (hub 0.144.0 + controller 0.304.0); `run_job fill-watch` proven live on the Tester 1 box (hub row „done — ran to its end"; control: the box log „checked 3 filesystem(s)"; an unknown action is refused 400). `abandon_stop` / `abandon_extend` NOT yet proven — no box has a deletion countdown; prove them on the first one.** **VERIFY — built on main 2026-10-08 (D1, decision 185), ships with tomorrow's controller + hub releases; closes after the live proof on 9202** (`abandon_stop` / `abandon_extend` through the hub's buttons). **READY (S) — NEW 2026-08-12, RANK 3** **2026-10-05 (burn-down night): NEEDS A DESIGN.** There is no operator door into a RUNNING controller (its HTTP surface is the household's session; the CLI runs in a separate process and cannot safely trigger a job in the running one). Same design as R-177 and R-279. | R-241, R-307 | An operator-authenticated POST that calls the same `StopAbandon`, so the telephone path and the code path converge | CC |
 | **R-401** | Backup & restore | P3 | **Revisit the off-site integrity depth when a real store is LARGE — the default rests on ONE measurement, on ONE 134 MB store.** Controller v0.228.0 (R-399) made `--read-data-subset=100%` the default for every box. The whole justification is a single data point: `demo-hp`, 2026-08-30, 140 829 678 B / 2 651 blobs / 67 snapshots, structure 35.0 s vs 100% **39.2 s** — four seconds. Re-proven live 2026-08-31 at 38.7 s. **It does not extrapolate, and the reason is structural: the structure check's cost tracks the INDEX, a read-data run's tracks the DATA.** A 50 GB store is ~370x the data and this curve says nothing about it. **Nothing was invented from that one point** — no rotation schedule, no size threshold, no bandwidth budget — because four production designs in this project were specced against unvalidated mechanisms and all four were wrong. **`readDataSubsetRe` already accepts `n/m`**, so a rotating schedule (`1/7` on a different seventh each week) needs no parser work when the time comes; the missing input is a measurement on a large store, not code. **THE TRIGGER IS AN EVENT, NOT A DATE:** the slow-check WARN from v0.228.0 firing on any box (`integritySlowNoticeThreshold`, 5 min) — that line names the duration, the depth and this row. **WHAT HAPPENS IF NOBODY ACTS:** every box re-reads its entire store every week, however large it grows, and the first person to notice is a customer whose upload is saturated. **Whoever acts must also revisit `integrityCheckTimeout` (30 min)**, which is now the number a large store meets first. | **OPEN — WATCHING** | — | When the WARN fires: measure the curve on that store, then choose between a rotation (`n/m`), a size-conditional default, or leaving it. Do NOT choose from this row's numbers — they are the small-store case. | CC |
@@ -193,7 +194,7 @@ stopping line that lies.
 | **R-1185** | Backup & restore | P3 | **A database dump that fails its own content check has already replaced last night's good dump, and the second drive copies it an hour later.** `appbackup/dbdump.go:388-422`: size check, rename over the good file (`:399`), then `ValidateDump` (`:419`). Volume tars are promoted with no content check either (`backup.go:902`). Tier 1 is one point per app by design (07 §6.1); the ORDER is the defect. PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-D5-dump-replaced/`, a docker stub prints a header-only dump): `ValidateDump` says invalid („dump file too small"), `DumpOneTo` returns no error, and last night's good dump (with its `CREATE TABLE`) is gone from disk. Data-loss hunt 2026-10-20 (D5). | OPEN | — | Fix: validate the temp file before the rename; keep the last good dump on a failed check | CC |
 | **R-1186** | Backup & restore | P3 | **The nightly off-site copy is skipped silently when another backup holds the flag, and the catch-up never re-runs it.** `offbox.go:981-996` returns nil with an INFO line on the nightly path; `withLeg` marks the leg ended anyway (`cmd/controller/main.go:1246`). The 05:30 proof and 06:00 integrity check take the same flag, so a backup time near them collides every night. Only the hub's 48 h `offsite_stale` notices. CONFIRMED in code. Data-loss hunt 2026-10-20 (T5). | OPEN | — | Fix: mark the leg NOT ended on a busy skip so the catch-up retries; say it on the backups page | CC |
 | **R-1187** | Backup & restore | P3 | **`.fab` export/import candidates from the data-loss hunt, not yet proven:** (a) an export silently drops a drive folder that holds a symlink (`appexport/fabplan.go:292`, `export.go:714`); (b) the import never sets file owners and drops symlinks (`restore.go:1157-1176`); (c) import into a STOPPED app merges into the existing volumes, and the default export (stop ticked) carries no SQL dump (`restore.go:361-374`, `export.go:262,366`, `appbackup/dbdump.go:142`); (d) no concurrency guard, no undo, no record (`restore.go:181-186`, `web/handler_export.go:312-353`); (e) PROVEN and split out as R-1191 (the import drops the family gate); (f) it trusts the bundle's HDD_PATH (`restore.go:589-631`); (g) the DB import runs psql without ON_ERROR_STOP and misses postgis (`restore.go:1052-1053, 928-937`). Read by reviewer 1; main session spot-checked (a); (e) proven live → R-1191. Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | VERIFY | — | Prove (c) first on 9202 with a throwaway app; split into rows only what reproduces | CC |
-| **R-1188** | Backup & restore | P3 | **Restore-path candidates from the data-loss hunt, not yet proven (one confirmed in code):** (a) a crash or power cut mid-restore restarts the app on a mixed state and never reads the undo copy (`appstop_marker.go:307`, `offbox_restore_undo.go:137`, `bootrecon/bootrecon.go:210-213`); (b) a failed `volume rm` is ignored before the untar (`restore.go:218`); (c) PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-R12-leftover-temp/`): an interrupted second-drive whole restore leaves `<file>.felhom-restoring`, and EVERY retry fails „fájlmásolás sikertelen: … file exists" with the file never back (`tier2_whole.go:86-87`, O_EXCL); (d) the full off-site download has a 30-min cap per press and the household is not told to retry (`web/offbox_handlers.go:455`); (e) place and full restore re-read the LATEST snapshot instead of the downloaded one (`offbox_restore.go:675-679`). Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | VERIFY | — | Fix (c) now (remove a stale temp of the same name before the copy); prove (a) with the B6 power-cut-mid-restore scenario | CC |
+| **R-1188** | Backup & restore | P3 | **Restore-path candidates from the data-loss hunt, not yet proven (one confirmed in code):** (a) PROVEN LIVE → R-1195 (a power cut mid-restore restarts the app on the backup's data; the undo copy is not used); (b) a failed `volume rm` is ignored before the untar (`restore.go:218`); (c) PROVEN against production code (`audits/dataloss-hunt-2026-10-20/C3-R12-leftover-temp/`): an interrupted second-drive whole restore leaves `<file>.felhom-restoring`, and EVERY retry fails „fájlmásolás sikertelen: … file exists" with the file never back (`tier2_whole.go:86-87`, O_EXCL); (d) the full off-site download has a 30-min cap per press and the household is not told to retry (`web/offbox_handlers.go:455`); (e) place and full restore re-read the LATEST snapshot instead of the downloaded one (`offbox_restore.go:675-679`). Evidence `audits/dataloss-hunt-2026-10-20/C1-restore-rollback-candidates.md`. | VERIFY | — | Fix (c) now (remove a stale temp of the same name before the copy); prove (a) with the B6 power-cut-mid-restore scenario | CC |
 | **R-1189** | Backup & restore | P3 | **Time and concurrency candidates from the data-loss hunt, not yet proven:** (a) a host clock ahead during one whole-guest backup silences that tier until real time passes it, and the hub reads the same clock (agent `localapi/server.go:1178-1192`, `backup/runner.go:607-629`, hub `monitor/deadline.go:243`; R-973 cap counts future runs `backup_cap.go:61`); (b) the whole-guest start creeps ~5 min/night until a night is skipped silently (`server.go:1184`, `quiesce.go:150, 807-818`); (c) the whole-guest quiesce stops apps mid-restore or mid-update (`quiesce.go:213-275, 565`); (d) CONFIRMED in code: the off-site proof and integrity check are plain daily jobs outside the catch-up (`nightchain.go` header: the catch-up runs only the three backup legs), so they never run on a box that is off at 05:30–06:00, while the comment at `cmd/controller/main.go:1575-1577` says the opposite; (e) CONFIRMED in code: the hub's 05:00 check wants a dump „since Budapest midnight" (`hub/internal/monitor/deadline.go:416-419, 556-565`), so an evening backup time (which the banner suggests) raises a false `expected_dbdump_missed` error every morning — and on 2026-10-25 the check also runs at 04:00 (R-1183); (f) the OS leg and kernel reboot follow a daytime valve backup (agent `server.go:982-986`, `kernel.go:101-114`) — design question; (g) spring forward 2027-03-28 fires dump and second copy together; (h) self-update busy check before the pull only (`updater.go:532-545, 764`); (i) a double-clicked restore shows failed while it runs (`web/handlers.go:1853-1874`). Evidence `audits/dataloss-hunt-2026-10-20/C1-time-concurrency-candidates.md`. | VERIFY | — | (a) is B6 #8's live test; (b) read the agent journal on demo-hp; the rest by unit tests | CC |
 | **R-1190** | Backup & restore | P3 | **Key and retention candidates from the data-loss hunt, not yet proven:** (a) a recovery ceremony while off-site is not configured seals a package with no repo password and supersedes the good one (`escrow_handlers.go:217`, agent `main.go:2975-2981`, hub `store.go:3392`); (b) the global „create your recovery code" bar links to the ceremony the backups page hides during a recovery offer (`escrow_banner.go:23-33` vs `backups_remote.html:144-160`); (c) after a reuse-path reinstall the new PBS key K is never escrowed and nothing compares fingerprints (`felhom-pbs-apply:99`, `escrow_confirm.go:182-205`); (d) whole-guest off-site copies run before any escrow exists (design question vs 07 §6.1's Tier-3 gate); (e) the escrow PUT is two statements; (f) the operator's off-site re-issue resets the provider password before saving it (hub `offsite.go:217-245`); (g) „Megőrzött adatok" can list a running app's live folder when its app.yaml is unreadable (`kept.go:220-235, 330-402`); (h) a unit restore can point an app back at an old drive and F5 then prunes the current drive's unit (`backup.go:1517-1574`). Evidence `audits/dataloss-hunt-2026-10-20/C1-keys-candidates.md`, `C1-delete-retention-candidates.md`. | VERIFY | — | Unit tests first for (a), (e), (g) | CC |