rehearsal 2026-10-20: B7 — re-linked after the operator's press; the household cannot restore from off-site (R-1176 proven live)

documentation/backlog/OPEN-ITEMS.md

@@ -158,7 +158,7 @@ stopping line that lies.
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
 | **R-1175** | Backup & restore | P1 | **Shared folders on two drives that use the same second drive: every night one drive's shares lose their second copy, and the night a drive dies its share copy is deleted.** The shares second-drive job runs one group per source drive; each group's clean-up walks EVERY drive key on the target and removes what is not its own (`internal/backup/tier2_shares.go:201-230`, `:266-267` → `tier2.go:299-329`). PROVEN against production code (fake copy seam): shares on hdd_2 and hdd_3 both target hdd_1 — run 1 logs `removed stale dest dir hdd_2/b_share`, so after EVERY run only the last group's shares have a copy while the status says ok; then hdd_3 goes away — run 2 logs `removed stale dest dir hdd_3/csaladi_fotok`: the dead drive's only second copy is deleted that night. Data-loss hunt 2026-10-20 (C1 reviewer 2, C2, C3: `audits/dataloss-hunt-2026-10-20/C3-D1-shares-tier2/`). | OPEN | — | Fix: reconcile only the group's OWN key dir (and keep a key whose drive is away/decommissioned until the household releases it); test with the C3 cases as red-proofs; release by the normal train | CC |
-| **R-1176** | Backup & restore | P1 | **A box rebuilt after „delete the old host first" cannot reach the household's off-site history: the recovery code is never asked for, a new key is minted, and the old history is set aside on night one.** D19 (`hub/internal/api/passphrase_scope.go:58`) makes the reinstall stop until the operator deletes the old host; the delete demotes the escrow under the OLD host id and deletes the host row (`hub/internal/store/store.go:3164-3229`); the new box's status query joins on live hosts only (`store.go:3499-3527`), so it reports no package and no retained package (both flags false) → the controller mints a fresh repo password (`offbox.go:542-555`), offers no recovery (`:1788`) and on its first night sets the old repository aside (`:905-906`); the retained-package API 404s for a deleted host (`handler.go:1691`). The reset and purge paths DO read `host_deletions` (`customer_reset.go:48,155`; `store.go:4189`) — the status path does not. PROVEN at the hub store against production code (`audits/dataloss-hunt-2026-10-20/C3-K1-escrow-after-host-delete/`: control = present; after delete + new host = nil). 03 §8a and 07 §7.0 promise this route; D19 + R-1154 removed it without a recorded choice. Data-loss hunt 2026-10-20 (C1 reviewer 3). | OPEN | the rehearsal's B7 is the live proof (needs Tester4 off-site ON) | Decide the shape (status query includes `host_deletions` for the same customer, and the retained GET answers for a deleted host of the caller's customer); red-proof with the C3 test; hub + controller release by the normal train | CC |
+| **R-1176** | Backup & restore | P1 | **A box rebuilt after „delete the old host first" cannot reach the household's off-site history: the recovery code is never asked for, a new key is minted, and the old history is set aside on night one.** D19 (`hub/internal/api/passphrase_scope.go:58`) makes the reinstall stop until the operator deletes the old host; the delete demotes the escrow under the OLD host id and deletes the host row (`hub/internal/store/store.go:3164-3229`); the new box's status query joins on live hosts only (`store.go:3499-3527`), so it reports no package and no retained package (both flags false) → the controller mints a fresh repo password (`offbox.go:542-555`), offers no recovery (`:1788`) and on its first night sets the old repository aside (`:905-906`); the retained-package API 404s for a deleted host (`handler.go:1691`). The reset and purge paths DO read `host_deletions` (`customer_reset.go:48,155`; `store.go:4189`) — the status path does not. PROVEN at the hub store against production code (`audits/dataloss-hunt-2026-10-20/C3-K1-escrow-after-host-delete/`: control = present; after delete + new host = nil). 03 §8a and 07 §7.0 promise this route; D19 + R-1154 removed it without a recorded choice. Data-loss hunt 2026-10-20 (C1 reviewer 3). | OPEN **PROVEN LIVE 2026-10-21 (rehearsal B7, `audits/rehearsal-2026-10-20/B7-died/README.md`):** Tester4 destroyed, reinstalled from the public ISO, the operator deleted the old host, bind + claim OK; the rebuilt dashboard shows only „A távoli mentés szünetel, amíg nem hozod létre a helyreállítási kódot. Helyreállítási kód létrehozása" (create, never enter), not the product's „A korábbi házon kívüli mentéseid visszaszerezhetők" sentence; the restore page: „Nem tudjuk elolvasni a távoli tárolót". The new code was not created and the box was torn down before its first night, so the history stays on the Storage Box. | the rehearsal's B7 is the live proof (needs Tester4 off-site ON) | Decide the shape (status query includes `host_deletions` for the same customer, and the retained GET answers for a deleted host of the caller's customer); red-proof with the C3 test; hub + controller release by the normal train | CC |
 | **R-232** | Backup & restore | P2 | **DooPlex's backup makes every copy inside the same box — and nothing tells anyone when it fails.** Surveyed read-only 2026-08-06 (`audits/RECON-dooplex-backup-2026-08-06.md`). **What works:** five sets, 14/14 successful runs in 14 days; a file was restored from the `data` repo and matched the live original **byte for byte**; every set except two is cross-disk; k3s is integrity-checked on every run. **What the matrix exposes, ranked:** (a) **`notify_failure` is a no-op** — `NOTIFY_ON_FAILURE=true` but `NOTIFY_WEBHOOK_URL` is commented out, so a failed backup notifies **nobody**; the project already has a working Resend path that CI uses. Cheapest item, and it makes every other failure visible. (b) **Nothing leaves the box** — no rclone, no remote repo, no off-site target anywhere; Longhorn's target is `nfs://192.168.0.180:` pointing at DooPlex itself, and the only outbound-looking cron pulls *inbound* from Hetzner for a different project. The machine that runs the hub managing the customers' off-site chain has no off-site copy of its own. (c) **The backup tree is a single writable path** and the restic repos are not append-only — one bad script or ransomware destroys every copy at once. (d) **Two same-disk sets**: `.claude-memory` and the PostgreSQL dumps, whose source directory sits *inside* the backup tree. (e) **Longhorn `retain=1`** — one generation per volume, so a corruption noticed a day late has no earlier copy. (f) **`/opt/backup/docs/BACKUP-RESTORE.md` does not exist** though the systemd unit advertises it. (g) **`secrets/restic-repo` has never held a snapshot** — `backup-secrets.sh` contains no `restic` call; the secrets are GPG files on `sda1` only. (h) **No restore has ever been run** beyond today's single-file probe — the matrix's "ever demonstrated?" column is otherwise entirely empty. **Not a finding:** the restic passphrase. The on-box copy is on `sdb1`, a different disk from the backups, and the **operator holds an offline copy out of band** — so a disk loss is recoverable. The narrow residual is that it is operator-held rather than system-held, unlike the customer case's hub-vaulted escrow, so it should be confirmed current and findable by someone else. **Nothing was changed by the recon.** | **NARROWED 2026-10-08 — owner Viktor.** (a) DONE with the operator's yes in chat: `notify_failure` now mails admin@felhom.eu through Resend; proven by one test mail that reached the inbox (`audits/day-2026-10-08/r232/`; no backup was started). (b) partly: the hub database leaves DooPlex nightly to ep0 (R-173); everything else stays on the box. (c)–(h) unchanged. **2026-10-09 (operator yes in chat): (b) and (h) DONE for Gitea and the secrets.** Nightly 00:20 `felhom-dooplex-offsite` pushes Gitea (614 MB of repositories, the `gitea` database dump taken first, `app.ini`) and the nightly GPG secrets export, encrypted with a NEW key, to ep0 `operator` (`host/dooplex-gitea`) on the hub-DB write-only token — no change on ep0; first push 57 s; restore test weekly (manifest, `git fsck` every repo, `pg_restore --list`), run once: 27 805 files, 10 repos OK; `DooplexGiteaOffsiteStale`/`DooplexGiteaRestoreTestStale` in force (`absent()` included, promtool + 2 red-proofs); every backup unit now mails on failure (`OnFailure=`, dry run reached the inbox). **First real Gitea restore:** into a throwaway on the bench, 10/10 repos, product `main` = live, a file byte for byte, a login (`runbooks/gitea-restore.md`). The hub came back too (R-173, closed). **The registry is left out on purpose:** its images rebuild from the code. **Correction to the recon §7:** the API mirror holds ONE repo (`homelab-manifests`), not all — Gitea had a single copy, on DooPlex. `audits/dooplex-survival-2026-10-09/` **LEFT:** (c) append-only for the on-box tree, (d)–(g) as written; the paper copy of the new key (operator). **READY** for the rest | — | — | operator |
 | **R-895** | Backup & restore | P2 | **The hub's clean-up-window check trusts the snapshot counts the box sends, so a broken-into box (or past-dated fakes added through the add-only key) can shrink the real off-site history without an alarm.** READ 2026-10-06 night in source (R-822's design): the before/after comparison uses counts the box itself reports (`hub/internal/offsitekeys/service.go:284`, `:343`); new fakes keep the count level. Decision 68 already accepts a box-trusted count. | **NARROWED 2026-10-18 — slice 2 built, not deployed** (hub v0.158.0): the 07:10 audit compares the `snapshots/` listing with yesterday's by name — files gone that no clean-up window explains → `offsite_snapshots_vanished` (error), files no reported box run explains → `offsite_snapshots_unexplained` (warning); operator-only, hub-origin (not in `allowedEventTypes`). First listing = baseline; a move-aside or RESET drops it; vanishing judged on a pinned tier only; files younger than 1 h judged the next day. Red-proofed twice (`snapshot_watch_test.go`). **Not built:** the window's before/after on the hub's own count (design §3.1) and refusing a window while unexplained files exist (§3.2). Unmeasured: the provider shell's `ls -l` time zone (± 3 h slack until then). **Deployed 2026-10-18 12:23 (hub v0.158.0).** The first listing (tomorrow 07:10) is a baseline; the first verdict is 2026-10-20 07:10. **Known limit (independent review 2026-10-18, F2):** `_unexplained` is explained by the box's OWN reported `offsite.last_run`, so whoever holds a box's key can silence it; `_vanished` is judged on the hub's own listing and is not affected. **2026-10-20 03:5x UTC: due date moved 10-20 → 10-21** — the first verdict lands at 07:10 hub time (05:10 UTC), after the night session that reached it ended; that brief did not include this read. Nothing is lost: the hub keeps its listing; read the 2026-10-20 07:10 lines on 10-21 (or later on 10-20). **FIRST VERDICT READ 2026-10-21 (rehearsal session, read-only):** the 07:10 CEST run judged three tiers against yesterday — demo-felhom 5, demo-hp 14, tester-1 4 new snapshot files, „each inside a reported box run"; nothing vanished, nothing unexplained; Tester4 stored its first baseline (44 files); Tester-2's own listing failed (no `snapshots/` dir — the box has never pushed). No `offsite_snapshots_vanished` / `_unexplained` mail in the operator mailbox (second channel). Evidence `audits/due-checks-2026-10-21/R-895-hub-0710.txt`. The positive alarm path is still proven only by the unit red-proofs. | the operator's yes to deploy hub v0.158.0 | Deploy; read the first 07:10 lines (newest upload vs the box's last run = the time-zone calibration), shrink the slack; then decide §3.1/§3.2 | CC |
 | **R-923** | Backup & restore | P2 | **The password manager (Vaultwarden) runs on DooPlex and holds the keys that open DooPlex's own off-site copies — a loss of DooPlex loses the keys with the copies.** Operator finding, 2026-10-09 11:19: "the password manager also runs on DooPlex (Vaultwarden), so I probably need to back up its DB somewhere too." Measured: `vaultwarden-system/vaultwarden` (1.37.3), PVC `vaultwarden-data` 7.4 MB (`db.sqlite3` + WAL, `rsa_key.pem`, no attachments); its only backups were Longhorn on DooPlex. **A key in a password manager that runs on DooPlex is not off DooPlex** — R-173's "both keys are off DooPlex" (2026-10-05) and the hub-DB runbook Step 0 were wrong in that sense. Also seen: Vaultwarden warns its `ADMIN_TOKEN` is plain text (an Argon2 hash is its guidance). | **NARROWED 2026-10-09 (operator yes in chat):** Vaultwarden's own `backup` copy + `rsa_key.pem` ride the nightly encrypted DooPlex→ep0 job (`scripts/dooplex-offsite/`), checked on every push and every Sunday (integrity, users, items — rows only). That copy opens only with the DooPlex off-site key, and the vault only with the master password — so the circle is broken only once both are on paper outside DooPlex: `runbooks/break-glass-sheet.md`, `runbooks/total-loss-of-dooplex.md`. **LEFT:** the operator prints the sheet and stores it away from home; the `ADMIN_TOKEN` hash (a DooPlex change, operator's word). | operator: print the sheet | Operator: print + store the sheet; then close | operator |