felhom.eu/documentation/audits/DIAGNOSE-offbox-repo-orphaned-2026-07-17.md

DIAGNOSE — offsite restic repo orphaned by a controller-data volume recreate (2026-07-17)

Status: FIXED — controller v0.142.0 (Parts A + C) + hub v0.60.0 (Part B), 2026-07-17. Original diagnosis: session 236a92d4; this note is the incident→fix pointer.

Incident

Demo 9201 offbox backup failed nightly: [offbox] backup <app> failed: exit status 1: Fatal: wrong password or no key found.

Root cause. The felhom-controller-data Docker volume was recreated 2026-07-16 (fresh encryption.key, re-provisioned settings.json). The offsite repo password is generate-once (offbox.go writes it only on os.IsNotExist), so with the file gone the offsite re-apply minted a NEW random passphrase P_new. Two destructive facts then compounded:

1. Controller: ensureOffboxRepo swallowed the wrong-password cat config failure (its init fallback saw "config file already exists" → treated as success), so the failure surfaced only as a raw nightly restic error at the backup step — no state, no operator-actionable signal.
2. Hub: SaveHostEscrow's ON CONFLICT(host_id) DO UPDATE OVERWROTE the P_old escrow blob with the P_new one (host_escrow.host_id is the PRIMARY KEY, one row per host). P_old was then gone from the box, the controller, AND escrow → 18 snapshots (Jul 9–16) cryptographically unrecoverable.

Manual fix (Viktor GO, 2026-07-17): mv felhom-repo felhom-repo.orphaned-20260717 on the box + restic init fresh under P_new. Local + Tier-2 backups were never affected (offsite = DR tier only).

The fix (this incident class is now closed)

• A (controller v0.142.0): ensureOffboxRepo CLASSIFIES the cat config failure (wrong password or no key found → ORPHANED). An orphaned repo → explicit ORPHANED state + a calm Hungarian card + offbox_repo_orphaned event (once, not nightly spam); scheduled runs skip. Reset = move-aside (.orphaned-<date>, never delete) + init: UNCLAIMED auto, CLAIMED reveal-then-confirm.
• B (hub v0.60.0): SaveHostEscrow RETAINS a superseded (different-sha) blob into host_escrow_superseded before overwriting — so P_old stays recoverable with its recovery code. This turns the incident class from "history destroyed" into "history recoverable with the recovery code" (the guided recovery flow itself is R-26).
• C (controller v0.142.0): the remote-backup page auto-refreshes the run status.

Still open

• R-26 — guided old-history recovery via a retained superseded blob + the recovery code (Part B makes it possible; the flow is not built).
• Product note: fresh-volume provisioning against a pre-existing offsite repo still does not recover the old password unattended (needs R) — it now cleanly SURFACES/RESETS instead of erroring nightly.