docs: offsite continuity Part B REPORT/CONTEXT; DIAGNOSE incident→fix pointer; capability map (offsite reinstall-continuity + escrow retention); ROADMAP R-25 (TOCTOU hardening) + R-26 (guided old-history recovery)

2026-07-17 10:55:17 +02:00
parent f7e3d50d5a
commit 106c3379b0
5 changed files with 83 additions and 37 deletions
@@ -0,0 +1,39 @@
+# DIAGNOSE — offsite restic repo orphaned by a controller-data volume recreate (2026-07-17)
+
+**Status: FIXED** — controller **v0.142.0** (Parts A + C) + hub **v0.60.0** (Part B), 2026-07-17.
+Original diagnosis: session `236a92d4`; this note is the incident→fix pointer.
+
+## Incident
+Demo 9201 offbox backup failed nightly: `[offbox] backup <app> failed: exit status 1: Fatal: wrong
+password or no key found`.
+
+**Root cause.** The `felhom-controller-data` Docker volume was recreated 2026-07-16 (fresh
+`encryption.key`, re-provisioned `settings.json`). The offsite repo password is **generate-once**
+(`offbox.go` writes it only on `os.IsNotExist`), so with the file gone the offsite re-apply minted a
+NEW random passphrase `P_new`. Two destructive facts then compounded:
+1. **Controller:** `ensureOffboxRepo` swallowed the wrong-password `cat config` failure (its `init`
+   fallback saw "config file already exists" → treated as success), so the failure surfaced only as a
+   raw nightly restic error at the backup step — no state, no operator-actionable signal.
+2. **Hub:** `SaveHostEscrow`'s `ON CONFLICT(host_id) DO UPDATE` OVERWROTE the `P_old` escrow blob with
+   the `P_new` one (`host_escrow.host_id` is the PRIMARY KEY, one row per host). `P_old` was then gone
+   from the box, the controller, AND escrow → 18 snapshots (Jul 9–16) cryptographically unrecoverable.
+
+**Manual fix (Viktor GO, 2026-07-17):** `mv felhom-repo felhom-repo.orphaned-20260717` on the box +
+`restic init` fresh under `P_new`. Local + Tier-2 backups were never affected (offsite = DR tier only).
+
+## The fix (this incident class is now closed)
+- **A (controller v0.142.0):** `ensureOffboxRepo` CLASSIFIES the `cat config` failure
+  (`wrong password or no key found` → ORPHANED). An orphaned repo → explicit ORPHANED state + a calm
+  Hungarian card + `offbox_repo_orphaned` event (once, not nightly spam); scheduled runs skip. Reset =
+  move-aside (`.orphaned-<date>`, never delete) + init: UNCLAIMED auto, CLAIMED reveal-then-confirm.
+- **B (hub v0.60.0):** `SaveHostEscrow` RETAINS a superseded (different-sha) blob into
+  `host_escrow_superseded` before overwriting — so `P_old` stays recoverable with its recovery code.
+  This turns the incident class from "history destroyed" into "history recoverable with the recovery
+  code" (the guided recovery flow itself is R-26).
+- **C (controller v0.142.0):** the remote-backup page auto-refreshes the run status.
+
+## Still open
+- **R-26** — guided old-history recovery via a retained superseded blob + the recovery code (Part B
+  makes it possible; the flow is not built).
+- Product note: fresh-volume provisioning against a pre-existing offsite repo still does not *recover*
+  the old password unattended (needs R) — it now cleanly SURFACES/RESETS instead of erroring nightly.