diff --git a/CONTEXT.md b/CONTEXT.md index 2b913be..c36a430 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,15 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-17 — OFFSITE CONTINUITY Part B SHIPPED: hub v0.60.0 (LIVE).** Superseded-escrow + retention (data-first, Viktor's ruling). `SaveHostEscrow`'s destructive `ON CONFLICT` overwrite + destroyed the old passphrase's only copy (the reinstall-orphan incident); now a new + different-sha blob RETAINS the old into the new `host_escrow_superseded` history table (tx, before + overwrite); same-sha = idempotent. ACK/restore read the current row unchanged. `escrow_superseded` + event + operator retained-count on host detail; registered `offbox_repo_orphaned`/`offbox_repo_reset`. + Red-proof `TestSaveHostEscrow_RetainsSuperseded`. Pairs with controller v0.142.0 (Part A + orphaned-repo guard). Guided old-history recovery = R-26 (not built). See REPORT.md. + - **2026-07-17 — N100 polish (F5 scripts + F6/F7 controller).** F5 (host-install v1.17.0): appliance mode auto-sizes the guest (RAM=clamp(host-4096,min 4096,max host-2048,ceil host-1024) + cores=host-1 min 2) when no explicit cap; explicit `--memory`/`--cores` win. Harness red-proof diff --git a/REPORT.md b/REPORT.md index ef6b68f..8520ba0 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,43 +2,39 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## N100 polish — F5 guest auto-sizing (host-install v1.17.0) + docs — 2026-07-17 +## Offsite continuity Part B — superseded-escrow retention (hub v0.60.0) — 2026-07-17 -Closes `documentation/tests/VALIDATION-n100-baremetal-2026-07-16.md` **F5 (MEDIUM)**. Companion to the -controller v0.141.0 F6/F7 fixes (see the felhom-controller REPORT). `bash -n` + the mode harness pass. +**Baseline:** hub v0.59.0 → **v0.60.0** (LIVE on prod, ArgoCD Synced/Healthy). Closes the data-loss +half of the reinstall-orphaned-repo incident +(`documentation/audits/DIAGNOSE-offbox-repo-orphaned-2026-07-17.md`): the destructive +`SaveHostEscrow` `ON CONFLICT` overwrite meant a new escrow blob DESTROYED the old passphrase's only +copy (18 snapshots unrecoverable). Viktor's ruling — data protection first — is now enforced. Pairs +with controller v0.142.0 (Part A). Green: `go build ./... && go vet ./... && go test ./...`. -### F5 — appliance guest auto-sizing (`scripts/felhom-host-install.sh` v1.17.0) -The golden default (2 GB RAM) reached the guest on a 16 GB host because appliance mode passed no -`--memory`/`--cores`. Now, in **appliance** mode with no explicit cap, the guest is auto-sized from -the host: `autosize_guest_caps` runs right after mode validation (before the plan summary + provision) -and fills `MEM_MIB`/`CPU_CORES`: -- RAM = `clamp(host-4096, min 4096, max host-2048)` then a hard ceiling `host-1024` (never - over-commit). Sizing: 8 GB→4096, 16 GB→12288, 32 GB→28672 MiB; a 4 GB host→3072 (min capped at the - ceiling — the edge rule). -- cores = `host-1, min 2`. -- An explicit `--memory`/`--cores` ALWAYS wins untouched; byo mode still requires explicit caps (never - auto-sized). The values flow into the existing `cap_args` → the agent's `-memory`/`-cores`. -- Host reads (`MemTotal`/`nproc`) are overridable via `FELHOM_FAKE_MEMTOTAL_MIB`/`FELHOM_FAKE_NPROC`. -- **Red-proof:** `hostinstall-mode-harness.sh` new F5 section (8/16/32 GB + small-host edge + - explicit-flag precedence) — the auto-size log line is emitted before any PVE contact, so it asserts - portably; pre-fix (auto-size disabled) FAILs, post-fix PASSes. +### Schema decision: history table (not a flag) +`host_escrow_superseded` — an append-only, one-row-per-superseded-blob table. A flag couldn't hold N +retained blobs, and Viktor's ruling is to keep ALL (no pruning); a history table is the only shape +that does. `SaveHostEscrow` now, in one tx: reads the current row; if it seals a DIFFERENT +`restic_pw_sha256`, COPIES it into the history table BEFORE the current row is overwritten; then upserts +the new row. Same-sha re-upload (idempotent re-ceremony) → refresh only, no supersede row. Returns +`superseded bool`. -**Same commit (R-16 doc-drift leftover, R-20 closure note):** the operator-signing-keys "EMPTY by -default" comment was stale (the keys are PINNED to the real `felhom-op-1`/`felhom-rec-1` ceremony -keypairs) — corrected, comment-only, no behavior. +### What ships (retention only — the guided-recovery flow is R-26, NOT built) +- `SaveHostEscrow` rewrite + `CountSupersededEscrow` / `ListSupersededEscrow` (the latter seeds R-26). + The hub still never decrypts; the ACK + restore-serving read the CURRENT row (`GetHostEscrow`) — + **unchanged**. `DeleteHost(deleteEscrow=true)` also drops the retained rows. +- Upload handler: `escrow_superseded` hub-internal audit event (tied to the owning customer) + a log + line with the retained count. Operator host-detail DR/Backup panel shows "N superseded escrow + blob(s) retained". Registered the controller-pushed `offbox_repo_orphaned` / `offbox_repo_reset` in + `allowedEventTypes` + `customerMessages`. +- **Red-proof** `TestSaveHostEscrow_RetainsSuperseded` (pre-fix destructive overwrite → old blob gone → + FAIL; fixed → retained + retrievable via `ListSupersededEscrow`; same-sha idempotent = no supersede). -### NOT live this task -F5 is validated by harness only — live at the next from-scratch rehearsal (Viktor's planned -delete/re-create run); stated in the ROADMAP and the VALIDATION disposition. +### Deploy +Built `felhom-hub:0.60.0`, bumped `manifests/hub.yaml`, ArgoCD Synced/Healthy, rollout complete +(`:0.60.0`). The `CREATE TABLE IF NOT EXISTS host_escrow_superseded` migration runs at startup +(idempotent). ACK semantics unaffected (current-row reads only). -### Docs updated -VALIDATION-n100 F5/F6/F7 dispositions → FIXED-in-vX (erratum style); capability map drive-wizard row → -F6 fresh-USB wizard enroll+format **PROVEN-LIVE**; ROADMAP R-16 (doc-drift bit closed), R-22 & R-13 -already PROVEN-LIVE, new **R-24** (guest resources as hub desired-state / live resize — the F5 -follow-on); operator-sidebar item #1 (BIOS "State After G3 → Power On") added to `scripts/iso/README.md`. - -### Operator sidebar (Viktor — CC does NOT do these; recorded verbatim) -1. **BIOS "State After G3" → "Power On"/"Last State"** before final sign-off — added to - `scripts/iso/README.md` hardware-prep notes. -2. **campaign6 autofs orphan** — clears at the next host reboot. -3. **tied-CreatedAt flash duplicates** (audiobookshelf/komga/romm) — standing action item. +### NOT done (registered as R-26) +The guided "recover old history with the old recovery code" flow — Part B ships retention + visibility +only; R-26 registers the recovery flow that consumes the retained blobs. diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index 2a2fab9..c7f6e6d 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -32,7 +32,7 @@ | Appliance day-0 install: golden image → first boot → auto-confirm (zero clicks) → claimable box | installer, agent, hub, golden | **PROVEN-LIVE** (nested VM) | `DRILL-day0-vm-2026-07-12`, `DRILL-day0-take2-2026-07-12` | First firing on real customer hardware pending → R-1 | | BYO install: `--mode byo`, mandatory caps, host-mutation disclosure, coexistence guards | installer v1.15+, agent | **PARTIAL** | `DRILL-GL6-2026-07-08` (demo box); GL-8 coexistence fixes | Peti clean-slate reinstall on proxmox2 is the first real BYO run of the current path → R-1 | | Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) | -| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | Endpoints driven on the drill VM; customer-facing **browser wizard** leg not yet live-validated. First supervised ceremony with a real customer pending → R-1. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart | +| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | Endpoints driven on the drill VM; customer-facing **browser wizard** leg not yet live-validated. First supervised ceremony with a real customer pending → R-1. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded` | | DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **IMPLEMENTED** | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected. Candidate upgrade to PROVEN-LIVE — see REPORT.) **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant | | Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 | @@ -54,7 +54,7 @@ |---|---|---|---|---| | Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | `CAMPAIGN-2` T-BAK-FULL (pg+mariadb autodiscovered); atomicity `CAMPAIGN-6B` P4 + `CAMPAIGN-6E` B1/B2 (SIGKILL mid-write → only `.tar.tmp` touched, last-good byte-unchanged); DB restore `CAMPAIGN-6D` P-FAB | (Cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes, and T-RST-DB was auth-hollow — corrected to the 6B/6E fix-proofs.) T-6E-1 dir-fsync asymmetry (LOW) → R-10 | | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C` | | -| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired | +| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs; live leg staged for the delete/re-create rehearsal (a scratch-target swap disturbs the live escrow state). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` | | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone) | | | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D | | Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill | diff --git a/documentation/audits/DIAGNOSE-offbox-repo-orphaned-2026-07-17.md b/documentation/audits/DIAGNOSE-offbox-repo-orphaned-2026-07-17.md new file mode 100644 index 0000000..5f544e7 --- /dev/null +++ b/documentation/audits/DIAGNOSE-offbox-repo-orphaned-2026-07-17.md @@ -0,0 +1,39 @@ +# DIAGNOSE — offsite restic repo orphaned by a controller-data volume recreate (2026-07-17) + +**Status: FIXED** — controller **v0.142.0** (Parts A + C) + hub **v0.60.0** (Part B), 2026-07-17. +Original diagnosis: session `236a92d4`; this note is the incident→fix pointer. + +## Incident +Demo 9201 offbox backup failed nightly: `[offbox] backup failed: exit status 1: Fatal: wrong +password or no key found`. + +**Root cause.** The `felhom-controller-data` Docker volume was recreated 2026-07-16 (fresh +`encryption.key`, re-provisioned `settings.json`). The offsite repo password is **generate-once** +(`offbox.go` writes it only on `os.IsNotExist`), so with the file gone the offsite re-apply minted a +NEW random passphrase `P_new`. Two destructive facts then compounded: +1. **Controller:** `ensureOffboxRepo` swallowed the wrong-password `cat config` failure (its `init` + fallback saw "config file already exists" → treated as success), so the failure surfaced only as a + raw nightly restic error at the backup step — no state, no operator-actionable signal. +2. **Hub:** `SaveHostEscrow`'s `ON CONFLICT(host_id) DO UPDATE` OVERWROTE the `P_old` escrow blob with + the `P_new` one (`host_escrow.host_id` is the PRIMARY KEY, one row per host). `P_old` was then gone + from the box, the controller, AND escrow → 18 snapshots (Jul 9–16) cryptographically unrecoverable. + +**Manual fix (Viktor GO, 2026-07-17):** `mv felhom-repo felhom-repo.orphaned-20260717` on the box + +`restic init` fresh under `P_new`. Local + Tier-2 backups were never affected (offsite = DR tier only). + +## The fix (this incident class is now closed) +- **A (controller v0.142.0):** `ensureOffboxRepo` CLASSIFIES the `cat config` failure + (`wrong password or no key found` → ORPHANED). An orphaned repo → explicit ORPHANED state + a calm + Hungarian card + `offbox_repo_orphaned` event (once, not nightly spam); scheduled runs skip. Reset = + move-aside (`.orphaned-`, never delete) + init: UNCLAIMED auto, CLAIMED reveal-then-confirm. +- **B (hub v0.60.0):** `SaveHostEscrow` RETAINS a superseded (different-sha) blob into + `host_escrow_superseded` before overwriting — so `P_old` stays recoverable with its recovery code. + This turns the incident class from "history destroyed" into "history recoverable with the recovery + code" (the guided recovery flow itself is R-26). +- **C (controller v0.142.0):** the remote-backup page auto-refreshes the run status. + +## Still open +- **R-26** — guided old-history recovery via a retained superseded blob + the recovery code (Part B + makes it possible; the flow is not built). +- Product note: fresh-volume provisioning against a pre-existing offsite repo still does not *recover* + the old password unattended (needs R) — it now cleanly SURFACES/RESETS instead of erroring nightly. diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 59dbc52..e5fd0a4 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -47,6 +47,8 @@ | ID | Item | Size | Status | Notes | |----|------|------|--------|-------| +| R-26 | **Guided old-history recovery via a retained superseded escrow + the recovery code.** Enabled by hub v0.60.0 (Part B) which now RETAINS superseded escrow blobs (`host_escrow_superseded`, `ListSupersededEscrow`). Build the flow that, given the customer's recovery code, unwraps a retained old blob → recovers the old repo passphrase → mounts/reads the moved-aside `.orphaned-` repo for restore. | M | idea (enabled by v0.60.0) | Turns "history recoverable in principle" into a real customer-drivable path; pairs with the controller v0.142.0 orphaned-repo move-aside. Origin `DIAGNOSE-offbox-repo-orphaned-2026-07-17` | +| R-25 | **Device-node TOCTOU hardening (drive init).** Graduate the controller v0.141.0 Observation: the `format → resolveEnrollUUID(path) → AssignDisk(uuid)` sequence has a narrow /dev-re-enumeration window (agent-guarded on the destructive format via anti-retarget durable-id; benign fs-UUID mount). Bind resolve+assign to the format's durable-id so the mount can't target a moved node. | S | idea | From the v0.141.0 F6 commit's security-review finding (`felhom-controller` REPORT). Low real risk (single-operator, agent-guarded), but cheap to close | | R-24 | **Guest resources as hub desired-state (live resize).** F5 (host-install v1.17.0) auto-sizes RAM/cores at INSTALL only. Make guest cores/RAM a per-host `pbs_dr`-sibling descriptor field the agent reconciles (`pct set -memory/-cores`), so the operator can right-size a running box from the hub — and land it in seconds via the agent-plane poke (R-13). | M | idea (F5 follow-on) | Follows F5 (`VALIDATION-n100` — appliance auto-size shipped); the live-resize path reuses the desired-state + poke machinery (agent v0.89 / hub v0.59). Would flip a new map row "operator right-sizes a running guest from the hub" | | R-12 | Cluster mode: agent-follows-guest, bind-mount reconciliation on HA migration | XL | idea | Scoped 07-15; interim = HA-group pin to one node. Driven by Peti's two-node cluster | | R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | **first slice PROVEN-LIVE (poke channel)** | **FIRST SLICE PROVEN-LIVE — the agent-plane poke channel (Direction-2a), agent v0.89.0 + hub v0.59.0 (2026-07-17):** the ep0-relayed contentless poke (hub→ep0 `felhom-poke` forced-cmd→UDP→box WG /32:51822, peer-confined, zero ep0/box infra change) reaches the **agent** and fires an immediate desired-state cycle. Full path live: real operator manifest save → `sync-poke delivered to 10.77.0.2`; box → `poke received → immediate desired-state cycle` (~31 ms ep0→box, save→tick ≈ ~0.45 s). This is ONLY the listener+sender; the rest of the mutual-repair arc (self-heal actions over the channel) stays open. Per `SPIKE-immediate-sync-transport-2026-07-16` P4. The controller-plane Direction-2 wait channel (hub v0.58 / controller v0.140) shipped the config-puller leg separately |