docs: offsite continuity Part B REPORT/CONTEXT; DIAGNOSE incident→fix pointer; capability map (offsite reinstall-continuity + escrow retention); ROADMAP R-25 (TOCTOU hardening) + R-26 (guided old-history recovery)
This commit is contained in:
@@ -3,6 +3,15 @@
|
||||
> Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub),
|
||||
> `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`.
|
||||
|
||||
- **2026-07-17 — OFFSITE CONTINUITY Part B SHIPPED: hub v0.60.0 (LIVE).** Superseded-escrow
|
||||
retention (data-first, Viktor's ruling). `SaveHostEscrow`'s destructive `ON CONFLICT` overwrite
|
||||
destroyed the old passphrase's only copy (the reinstall-orphan incident); now a new
|
||||
different-sha blob RETAINS the old into the new `host_escrow_superseded` history table (tx, before
|
||||
overwrite); same-sha = idempotent. ACK/restore read the current row unchanged. `escrow_superseded`
|
||||
event + operator retained-count on host detail; registered `offbox_repo_orphaned`/`offbox_repo_reset`.
|
||||
Red-proof `TestSaveHostEscrow_RetainsSuperseded`. Pairs with controller v0.142.0 (Part A
|
||||
orphaned-repo guard). Guided old-history recovery = R-26 (not built). See REPORT.md.
|
||||
|
||||
- **2026-07-17 — N100 polish (F5 scripts + F6/F7 controller).** F5 (host-install v1.17.0):
|
||||
appliance mode auto-sizes the guest (RAM=clamp(host-4096,min 4096,max host-2048,ceil host-1024) +
|
||||
cores=host-1 min 2) when no explicit cap; explicit `--memory`/`--cores` win. Harness red-proof
|
||||
|
||||
@@ -2,43 +2,39 @@
|
||||
|
||||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).
|
||||
|
||||
## N100 polish — F5 guest auto-sizing (host-install v1.17.0) + docs — 2026-07-17
|
||||
## Offsite continuity Part B — superseded-escrow retention (hub v0.60.0) — 2026-07-17
|
||||
|
||||
Closes `documentation/tests/VALIDATION-n100-baremetal-2026-07-16.md` **F5 (MEDIUM)**. Companion to the
|
||||
controller v0.141.0 F6/F7 fixes (see the felhom-controller REPORT). `bash -n` + the mode harness pass.
|
||||
**Baseline:** hub v0.59.0 → **v0.60.0** (LIVE on prod, ArgoCD Synced/Healthy). Closes the data-loss
|
||||
half of the reinstall-orphaned-repo incident
|
||||
(`documentation/audits/DIAGNOSE-offbox-repo-orphaned-2026-07-17.md`): the destructive
|
||||
`SaveHostEscrow` `ON CONFLICT` overwrite meant a new escrow blob DESTROYED the old passphrase's only
|
||||
copy (18 snapshots unrecoverable). Viktor's ruling — data protection first — is now enforced. Pairs
|
||||
with controller v0.142.0 (Part A). Green: `go build ./... && go vet ./... && go test ./...`.
|
||||
|
||||
### F5 — appliance guest auto-sizing (`scripts/felhom-host-install.sh` v1.17.0)
|
||||
The golden default (2 GB RAM) reached the guest on a 16 GB host because appliance mode passed no
|
||||
`--memory`/`--cores`. Now, in **appliance** mode with no explicit cap, the guest is auto-sized from
|
||||
the host: `autosize_guest_caps` runs right after mode validation (before the plan summary + provision)
|
||||
and fills `MEM_MIB`/`CPU_CORES`:
|
||||
- RAM = `clamp(host-4096, min 4096, max host-2048)` then a hard ceiling `host-1024` (never
|
||||
over-commit). Sizing: 8 GB→4096, 16 GB→12288, 32 GB→28672 MiB; a 4 GB host→3072 (min capped at the
|
||||
ceiling — the edge rule).
|
||||
- cores = `host-1, min 2`.
|
||||
- An explicit `--memory`/`--cores` ALWAYS wins untouched; byo mode still requires explicit caps (never
|
||||
auto-sized). The values flow into the existing `cap_args` → the agent's `-memory`/`-cores`.
|
||||
- Host reads (`MemTotal`/`nproc`) are overridable via `FELHOM_FAKE_MEMTOTAL_MIB`/`FELHOM_FAKE_NPROC`.
|
||||
- **Red-proof:** `hostinstall-mode-harness.sh` new F5 section (8/16/32 GB + small-host edge +
|
||||
explicit-flag precedence) — the auto-size log line is emitted before any PVE contact, so it asserts
|
||||
portably; pre-fix (auto-size disabled) FAILs, post-fix PASSes.
|
||||
### Schema decision: history table (not a flag)
|
||||
`host_escrow_superseded` — an append-only, one-row-per-superseded-blob table. A flag couldn't hold N
|
||||
retained blobs, and Viktor's ruling is to keep ALL (no pruning); a history table is the only shape
|
||||
that does. `SaveHostEscrow` now, in one tx: reads the current row; if it seals a DIFFERENT
|
||||
`restic_pw_sha256`, COPIES it into the history table BEFORE the current row is overwritten; then upserts
|
||||
the new row. Same-sha re-upload (idempotent re-ceremony) → refresh only, no supersede row. Returns
|
||||
`superseded bool`.
|
||||
|
||||
**Same commit (R-16 doc-drift leftover, R-20 closure note):** the operator-signing-keys "EMPTY by
|
||||
default" comment was stale (the keys are PINNED to the real `felhom-op-1`/`felhom-rec-1` ceremony
|
||||
keypairs) — corrected, comment-only, no behavior.
|
||||
### What ships (retention only — the guided-recovery flow is R-26, NOT built)
|
||||
- `SaveHostEscrow` rewrite + `CountSupersededEscrow` / `ListSupersededEscrow` (the latter seeds R-26).
|
||||
The hub still never decrypts; the ACK + restore-serving read the CURRENT row (`GetHostEscrow`) —
|
||||
**unchanged**. `DeleteHost(deleteEscrow=true)` also drops the retained rows.
|
||||
- Upload handler: `escrow_superseded` hub-internal audit event (tied to the owning customer) + a log
|
||||
line with the retained count. Operator host-detail DR/Backup panel shows "N superseded escrow
|
||||
blob(s) retained". Registered the controller-pushed `offbox_repo_orphaned` / `offbox_repo_reset` in
|
||||
`allowedEventTypes` + `customerMessages`.
|
||||
- **Red-proof** `TestSaveHostEscrow_RetainsSuperseded` (pre-fix destructive overwrite → old blob gone →
|
||||
FAIL; fixed → retained + retrievable via `ListSupersededEscrow`; same-sha idempotent = no supersede).
|
||||
|
||||
### NOT live this task
|
||||
F5 is validated by harness only — live at the next from-scratch rehearsal (Viktor's planned
|
||||
delete/re-create run); stated in the ROADMAP and the VALIDATION disposition.
|
||||
### Deploy
|
||||
Built `felhom-hub:0.60.0`, bumped `manifests/hub.yaml`, ArgoCD Synced/Healthy, rollout complete
|
||||
(`:0.60.0`). The `CREATE TABLE IF NOT EXISTS host_escrow_superseded` migration runs at startup
|
||||
(idempotent). ACK semantics unaffected (current-row reads only).
|
||||
|
||||
### Docs updated
|
||||
VALIDATION-n100 F5/F6/F7 dispositions → FIXED-in-vX (erratum style); capability map drive-wizard row →
|
||||
F6 fresh-USB wizard enroll+format **PROVEN-LIVE**; ROADMAP R-16 (doc-drift bit closed), R-22 & R-13
|
||||
already PROVEN-LIVE, new **R-24** (guest resources as hub desired-state / live resize — the F5
|
||||
follow-on); operator-sidebar item #1 (BIOS "State After G3 → Power On") added to `scripts/iso/README.md`.
|
||||
|
||||
### Operator sidebar (Viktor — CC does NOT do these; recorded verbatim)
|
||||
1. **BIOS "State After G3" → "Power On"/"Last State"** before final sign-off — added to
|
||||
`scripts/iso/README.md` hardware-prep notes.
|
||||
2. **campaign6 autofs orphan** — clears at the next host reboot.
|
||||
3. **tied-CreatedAt flash duplicates** (audiobookshelf/komga/romm) — standing action item.
|
||||
### NOT done (registered as R-26)
|
||||
The guided "recover old history with the old recovery code" flow — Part B ships retention + visibility
|
||||
only; R-26 registers the recovery flow that consumes the retained blobs.
|
||||
|
||||
@@ -32,7 +32,7 @@
|
||||
| Appliance day-0 install: golden image → first boot → auto-confirm (zero clicks) → claimable box | installer, agent, hub, golden | **PROVEN-LIVE** (nested VM) | `DRILL-day0-vm-2026-07-12`, `DRILL-day0-take2-2026-07-12` | First firing on real customer hardware pending → R-1 |
|
||||
| BYO install: `--mode byo`, mandatory caps, host-mutation disclosure, coexistence guards | installer v1.15+, agent | **PARTIAL** | `DRILL-GL6-2026-07-08` (demo box); GL-8 coexistence fixes | Peti clean-slate reinstall on proxmox2 is the first real BYO run of the current path → R-1 |
|
||||
| Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) |
|
||||
| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | Endpoints driven on the drill VM; customer-facing **browser wizard** leg not yet live-validated. First supervised ceremony with a real customer pending → R-1. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart |
|
||||
| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | Endpoints driven on the drill VM; customer-facing **browser wizard** leg not yet live-validated. First supervised ceremony with a real customer pending → R-1. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded` |
|
||||
| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **IMPLEMENTED** | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected. Candidate upgrade to PROVEN-LIVE — see REPORT.) **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant |
|
||||
| Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 |
|
||||
|
||||
@@ -54,7 +54,7 @@
|
||||
|---|---|---|---|---|
|
||||
| Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | `CAMPAIGN-2` T-BAK-FULL (pg+mariadb autodiscovered); atomicity `CAMPAIGN-6B` P4 + `CAMPAIGN-6E` B1/B2 (SIGKILL mid-write → only `.tar.tmp` touched, last-good byte-unchanged); DB restore `CAMPAIGN-6D` P-FAB | (Cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes, and T-RST-DB was auth-hollow — corrected to the 6B/6E fix-proofs.) T-6E-1 dir-fsync asymmetry (LOW) → R-10 |
|
||||
| Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C` | |
|
||||
| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired |
|
||||
| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs; live leg staged for the delete/re-create rehearsal (a scratch-target swap disturbs the live escrow state). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` |
|
||||
| Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone) | |
|
||||
| Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D |
|
||||
| Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill |
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
# DIAGNOSE — offsite restic repo orphaned by a controller-data volume recreate (2026-07-17)
|
||||
|
||||
**Status: FIXED** — controller **v0.142.0** (Parts A + C) + hub **v0.60.0** (Part B), 2026-07-17.
|
||||
Original diagnosis: session `236a92d4`; this note is the incident→fix pointer.
|
||||
|
||||
## Incident
|
||||
Demo 9201 offbox backup failed nightly: `[offbox] backup <app> failed: exit status 1: Fatal: wrong
|
||||
password or no key found`.
|
||||
|
||||
**Root cause.** The `felhom-controller-data` Docker volume was recreated 2026-07-16 (fresh
|
||||
`encryption.key`, re-provisioned `settings.json`). The offsite repo password is **generate-once**
|
||||
(`offbox.go` writes it only on `os.IsNotExist`), so with the file gone the offsite re-apply minted a
|
||||
NEW random passphrase `P_new`. Two destructive facts then compounded:
|
||||
1. **Controller:** `ensureOffboxRepo` swallowed the wrong-password `cat config` failure (its `init`
|
||||
fallback saw "config file already exists" → treated as success), so the failure surfaced only as a
|
||||
raw nightly restic error at the backup step — no state, no operator-actionable signal.
|
||||
2. **Hub:** `SaveHostEscrow`'s `ON CONFLICT(host_id) DO UPDATE` OVERWROTE the `P_old` escrow blob with
|
||||
the `P_new` one (`host_escrow.host_id` is the PRIMARY KEY, one row per host). `P_old` was then gone
|
||||
from the box, the controller, AND escrow → 18 snapshots (Jul 9–16) cryptographically unrecoverable.
|
||||
|
||||
**Manual fix (Viktor GO, 2026-07-17):** `mv felhom-repo felhom-repo.orphaned-20260717` on the box +
|
||||
`restic init` fresh under `P_new`. Local + Tier-2 backups were never affected (offsite = DR tier only).
|
||||
|
||||
## The fix (this incident class is now closed)
|
||||
- **A (controller v0.142.0):** `ensureOffboxRepo` CLASSIFIES the `cat config` failure
|
||||
(`wrong password or no key found` → ORPHANED). An orphaned repo → explicit ORPHANED state + a calm
|
||||
Hungarian card + `offbox_repo_orphaned` event (once, not nightly spam); scheduled runs skip. Reset =
|
||||
move-aside (`.orphaned-<date>`, never delete) + init: UNCLAIMED auto, CLAIMED reveal-then-confirm.
|
||||
- **B (hub v0.60.0):** `SaveHostEscrow` RETAINS a superseded (different-sha) blob into
|
||||
`host_escrow_superseded` before overwriting — so `P_old` stays recoverable with its recovery code.
|
||||
This turns the incident class from "history destroyed" into "history recoverable with the recovery
|
||||
code" (the guided recovery flow itself is R-26).
|
||||
- **C (controller v0.142.0):** the remote-backup page auto-refreshes the run status.
|
||||
|
||||
## Still open
|
||||
- **R-26** — guided old-history recovery via a retained superseded blob + the recovery code (Part B
|
||||
makes it possible; the flow is not built).
|
||||
- Product note: fresh-volume provisioning against a pre-existing offsite repo still does not *recover*
|
||||
the old password unattended (needs R) — it now cleanly SURFACES/RESETS instead of erroring nightly.
|
||||
@@ -47,6 +47,8 @@
|
||||
|
||||
| ID | Item | Size | Status | Notes |
|
||||
|----|------|------|--------|-------|
|
||||
| R-26 | **Guided old-history recovery via a retained superseded escrow + the recovery code.** Enabled by hub v0.60.0 (Part B) which now RETAINS superseded escrow blobs (`host_escrow_superseded`, `ListSupersededEscrow`). Build the flow that, given the customer's recovery code, unwraps a retained old blob → recovers the old repo passphrase → mounts/reads the moved-aside `.orphaned-<date>` repo for restore. | M | idea (enabled by v0.60.0) | Turns "history recoverable in principle" into a real customer-drivable path; pairs with the controller v0.142.0 orphaned-repo move-aside. Origin `DIAGNOSE-offbox-repo-orphaned-2026-07-17` |
|
||||
| R-25 | **Device-node TOCTOU hardening (drive init).** Graduate the controller v0.141.0 Observation: the `format → resolveEnrollUUID(path) → AssignDisk(uuid)` sequence has a narrow /dev-re-enumeration window (agent-guarded on the destructive format via anti-retarget durable-id; benign fs-UUID mount). Bind resolve+assign to the format's durable-id so the mount can't target a moved node. | S | idea | From the v0.141.0 F6 commit's security-review finding (`felhom-controller` REPORT). Low real risk (single-operator, agent-guarded), but cheap to close |
|
||||
| R-24 | **Guest resources as hub desired-state (live resize).** F5 (host-install v1.17.0) auto-sizes RAM/cores at INSTALL only. Make guest cores/RAM a per-host `pbs_dr`-sibling descriptor field the agent reconciles (`pct set -memory/-cores`), so the operator can right-size a running box from the hub — and land it in seconds via the agent-plane poke (R-13). | M | idea (F5 follow-on) | Follows F5 (`VALIDATION-n100` — appliance auto-size shipped); the live-resize path reuses the desired-state + poke machinery (agent v0.89 / hub v0.59). Would flip a new map row "operator right-sizes a running guest from the hub" |
|
||||
| R-12 | Cluster mode: agent-follows-guest, bind-mount reconciliation on HA migration | XL | idea | Scoped 07-15; interim = HA-group pin to one node. Driven by Peti's two-node cluster |
|
||||
| R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | **first slice PROVEN-LIVE (poke channel)** | **FIRST SLICE PROVEN-LIVE — the agent-plane poke channel (Direction-2a), agent v0.89.0 + hub v0.59.0 (2026-07-17):** the ep0-relayed contentless poke (hub→ep0 `felhom-poke` forced-cmd→UDP→box WG /32:51822, peer-confined, zero ep0/box infra change) reaches the **agent** and fires an immediate desired-state cycle. Full path live: real operator manifest save → `sync-poke delivered to 10.77.0.2`; box → `poke received → immediate desired-state cycle` (~31 ms ep0→box, save→tick ≈ ~0.45 s). This is ONLY the listener+sender; the rest of the mutual-repair arc (self-heal actions over the channel) stays open. Per `SPIKE-immediate-sync-transport-2026-07-16` P4. The controller-plane Direction-2 wait channel (hub v0.58 / controller v0.140) shipped the config-puller leg separately |
|
||||
|
||||
Reference in New Issue
Block a user