40 lines
2.8 KiB
Markdown
40 lines
2.8 KiB
Markdown
# DIAGNOSE — offsite restic repo orphaned by a controller-data volume recreate (2026-07-17)
|
||
|
||
**Status: FIXED** — controller **v0.142.0** (Parts A + C) + hub **v0.60.0** (Part B), 2026-07-17.
|
||
Original diagnosis: session `236a92d4`; this note is the incident→fix pointer.
|
||
|
||
## Incident
|
||
Demo 9201 offbox backup failed nightly: `[offbox] backup <app> failed: exit status 1: Fatal: wrong
|
||
password or no key found`.
|
||
|
||
**Root cause.** The `felhom-controller-data` Docker volume was recreated 2026-07-16 (fresh
|
||
`encryption.key`, re-provisioned `settings.json`). The offsite repo password is **generate-once**
|
||
(`offbox.go` writes it only on `os.IsNotExist`), so with the file gone the offsite re-apply minted a
|
||
NEW random passphrase `P_new`. Two destructive facts then compounded:
|
||
1. **Controller:** `ensureOffboxRepo` swallowed the wrong-password `cat config` failure (its `init`
|
||
fallback saw "config file already exists" → treated as success), so the failure surfaced only as a
|
||
raw nightly restic error at the backup step — no state, no operator-actionable signal.
|
||
2. **Hub:** `SaveHostEscrow`'s `ON CONFLICT(host_id) DO UPDATE` OVERWROTE the `P_old` escrow blob with
|
||
the `P_new` one (`host_escrow.host_id` is the PRIMARY KEY, one row per host). `P_old` was then gone
|
||
from the box, the controller, AND escrow → 18 snapshots (Jul 9–16) cryptographically unrecoverable.
|
||
|
||
**Manual fix (Viktor GO, 2026-07-17):** `mv felhom-repo felhom-repo.orphaned-20260717` on the box +
|
||
`restic init` fresh under `P_new`. Local + Tier-2 backups were never affected (offsite = DR tier only).
|
||
|
||
## The fix (this incident class is now closed)
|
||
- **A (controller v0.142.0):** `ensureOffboxRepo` CLASSIFIES the `cat config` failure
|
||
(`wrong password or no key found` → ORPHANED). An orphaned repo → explicit ORPHANED state + a calm
|
||
Hungarian card + `offbox_repo_orphaned` event (once, not nightly spam); scheduled runs skip. Reset =
|
||
move-aside (`.orphaned-<date>`, never delete) + init: UNCLAIMED auto, CLAIMED reveal-then-confirm.
|
||
- **B (hub v0.60.0):** `SaveHostEscrow` RETAINS a superseded (different-sha) blob into
|
||
`host_escrow_superseded` before overwriting — so `P_old` stays recoverable with its recovery code.
|
||
This turns the incident class from "history destroyed" into "history recoverable with the recovery
|
||
code" (the guided recovery flow itself is R-26).
|
||
- **C (controller v0.142.0):** the remote-backup page auto-refreshes the run status.
|
||
|
||
## Still open
|
||
- **R-26** — guided old-history recovery via a retained superseded blob + the recovery code (Part B
|
||
makes it possible; the flow is not built).
|
||
- Product note: fresh-volume provisioning against a pre-existing offsite repo still does not *recover*
|
||
the old password unattended (needs R) — it now cleanly SURFACES/RESETS instead of erroring nightly.
|