diff --git a/CONTEXT.md b/CONTEXT.md index 73d46fb..455f986 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -5,6 +5,22 @@ ## Current +- **v0.69.0** (2026-07-04, live on felhom-pve) — **S5: host-loss DR — safe halves shipped**. + **Part 1** `wgtunnel.InstallRecoveredKey` — writes an escrow-recovered WG privkey (create-only, + refuse-overwrite) so the tunnel re-establishes with the SAME identity/pubkey (same /32), no keygen; + wired into `--selftest=identity-consume -install-wg-key` (opt-in; pre-S3 blob → logged fresh-keygen + fallback). **Part 2** new `internal/dr` — consumes the host_loss `restore_directive` (was + logged-ignored) into an inspectable RestorePlan via AddConsumer: per-guest {vmid,archive,target, + sizing} + per-drive {durable_id→mount} + offsite PBS coord; DERIVE-AND-SURFACE only (Consumer has + no restore/destroy dep — execute-nothing is structural). Tests + red-proofs (WG create-only; plan + mode-gate). **Part 3** hub escrow-GET NOT needed (operator exports the blob via `sqlite3 writefile` + on a cp'd hub.db). **Part 4-A** re-attach wrong-disk safety already unit-proven + (`ResolveStorageDevice`: match resolves, absent/mismatch ERRORS, non-uuid scheme refused — never a + near disk). **Part 4-B (destructive in-place 9201 restore) PREPARED + OPERATOR-GATED, NOT executed** + — pre-flight green (offsite ct/9201 restorable per S4.1); the operator runs the R-consume steps + + confirms the destroy (§9-4a: CC never runs a consume/R command — see [[operator-present-one-time-secrets]]). + OPEN: the operator-run 4-B drill; guest_loss DR; hub-driven full-auto DR. Rollback + felhom-agent.bak-0.68.0. Detail: REPORT.md + doc-06 §3.5/S5. - **v0.68.0** (2026-07-04, live on felhom-pve) — **S4.1: unattended offsite restore-test**. **Tier-aware restore-task deadline:** `RestoreTestSpec.RestoreTaskTimeout` (0→10m default) from `config.RestoreTestPBSRestoreTimeoutSeconds` (accessor default **120m**), set only when diff --git a/REPORT.md b/REPORT.md index b164a18..cdee37b 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,72 +1,73 @@ -# REPORT — S4.1: unattended offsite restore-test (tier-aware deadline + teardown diagnosis) +# REPORT — S5: host-loss DR (recovered WG-key install + directive→PLAN; safe halves shipped) -**Date:** 2026-07-04 · **Class:** implementation + live diagnosis on felhom-pve (agent token). -**Design of record:** `felhom.eu/documentation/architecture/06-offsite-connectivity.md` S4 row; -runbook `offsite-endpoint.md` §4b. +**Date:** 2026-07-04 · **Class:** implementation (safe halves) + diagnosis + a prepared, +operator-present destructive drill (NOT executed by CC). **Design of record:** doc-06 §3.5 + S5 row; +doc-01 §9. ## 1. Baselines & commits (felhom-agent → `main`) -Start: `fee1fcf` (v0.67.0). -- **`dc70e15`** v0.68.0 — tier-aware restore-task deadline (Part 1). -- this commit — REPORT + CONTEXT (the diagnosis outcome). +Start: `567cf9f` (v0.68.0). **`bd4bced`** v0.69.0 — the two safe DR halves (Parts 1-2). This commit +— REPORT + CONTEXT + doc-06 S5 row. -## 2. Part 1 — tier-aware restore-task deadline +## 2. Part 1 (SAFE, shipped) — install the recovered WG key -The restore-task wait (`restoretest.go` ~L246) used `WaitOptions{}` → the 10m default. A WAN (pbs) -restore of a large guest exceeds it, so the wait expired mid-restore and teardown then fired against -a still-restoring guest. Fix: -- `RestoreTestSpec.RestoreTaskTimeout` (0 → 10m default); the restore `WaitTask` passes it. -- `config.BackupConfig.RestoreTestPBSRestoreTimeoutSeconds` + accessor `RestoreTestPBSRestoreTimeout()` - (positive as-is, else **120m**). -- `main.restoreTaskTimeout(cfg, tier)` — the configured PBS timeout only when `SourceTier=="pbs"`, - else 0. Set at BOTH spec builds (scheduler + selftest). **Local tier unchanged (10m).** -- Tests: `WaitOptions.Timeout` pbs→120m / local→0 (red-proofed against `WaitOptions{}`: pbs - assertion fails); accessor contract (0/neg→120m, positive as-is). Full `go test ./...` green. +`wgtunnel.InstallRecoveredKey(stateDir, privB64)` writes an escrow-recovered WG private key (32-byte +base64, re-encoded canonical) to the key file so the tunnel re-establishes with the **same +identity/pubkey → same hub /32**, no fresh keygen. **CREATE-ONLY** — refuses if a key file exists (a +present key may be a live identity); the value is never logged. Wired into +`--selftest=identity-consume -install-wg-key` (opt-in; installs `bundle.WGPrivateKey` after +`UnwrapIdentityBundle`; a pre-S3 blob with no WG key → logged fallback to fresh keygen + re-register, +which keeps the /32 via hub re-key-in-place). +**Test + red-proof:** install→`EnsureKey` LOADS it (same pubkey, no keygen); a second install REFUSES +(create-only — red-proofed against dropping the exists-check); invalid key → error, nothing written. -## 3. Part 2 — teardown diagnosis (the "grant VM.Allocate" follow-up) — PHANTOM CONFIRMED +## 3. Part 2 (SAFE, shipped) — consume `restore_directive` → PLAN -Ran the restore-test on the **agent-token path** (selftest = the identical `RunRestoreTest` the -scheduler runs; all PVE ops authenticate as `felhom-agent@pve!agent`), sourcing the **offsite -(pbs-tier)** backup of guest 9201, with v0.68.0's 120m timeout. +New `internal/dr`: the host_loss `restore_directive` (was logged-and-ignored in `syncer.go`) is +consumed via the `desired.Syncer.AddConsumer` raw seam into an inspectable **RestorePlan** — per +guest `{vmid, archive, target storage, sizing}`, per drive `{durable_id → expected mount}`, + the +offsite PBS coord. **Derive-and-surface only:** the `Consumer` holds NO restore/destroy dependency, so +"execute nothing" is structural. `guest_loss`/absent → no plan; recipe fetched on-demand (rare +directive) via a fresh `Collect`. +**Test + red-proof:** host_loss builds the expected plan; guest_loss/absent/nil-recipe → none +(red-proofed against a relaxed mode gate); the consumer surfaces on host_loss + clears otherwise + +consults the recipe only then. Full `go test ./...` green; deployed 0.69.0; `-install-wg-key` flag +live; daemon starts clean (dr consumer wired). -**Scenario B (done-criterion) — GREEN:** -``` -source_archive: felhom-offsite:backup/ct/9201/2026-07-04T14:55:44Z -source_tier: pbs · pass: true · verified: boot+running · scratch_vmid: 990000 · duration 4m49s -restore → net-link-down → boot (running) → teardown → scratch band clean (no leak) -``` +## 4. Part 3 — hub escrow export: NOT needed -**Scenario C — the teardown succeeded as the agent token; (b) is a PHANTOM:** -``` -gate decision: class=guest_destroy guest=990000 allowed=true reason=benign -restore-test: scratch guest torn down vmid=990000 -pct list | grep 9900 → no scratch guests -``` -No 403, no `VM.Allocate` error. Root cause of the earlier 403 (REPORT of S4): the 10m wait expired -**mid-restore**, so teardown hit a guest that existed but was **not yet pool-associated** (the pool -membership lands with restore completion) — a bare `/vms/990000` the pool ACL didn't cover. With the -timeout fixed → the restore completes → the guest is a member of `/pool/felhom` (restored with -`Pool=DefaultPool`, `restoretest.go` L229) → its destroy is authorized by the existing -`FelhomAgentGuest` grant (`VM.Allocate` + `Pool.Allocate` at `/pool/felhom`). **No ACL change, no -host-install edit** — the pool-scoped containment (`SPIKE-pool-scoped-acl-2026-07-01`) stands. +The operator has a clean blob-export path already: `sqlite3 … "SELECT writefile('/tmp/blob', blob), +writefile('/tmp/idblob', identity_blob) FROM host_escrow WHERE host_id='demo-felhom-01'"` on a +`kubectl cp`'d `hub.db`. No hub change (respects the read-only-unless-no-clean-way gate). -(Note: this run's restore was fast — home *download* ≫ *upload*; the earlier ~10-min stall was link -contention. The 120m ceiling covers the slow case regardless — for an unattended nightly test a -false timeout is the failure mode to prevent.) +## 5. Part 4-A — re-attach wrong-disk safety: ALREADY PROVEN (no new probe built) -## 4. Live state after / cleanup +The re-attach matcher is `storage.ResolveStorageDevice` (the `uuid:` scheme DR drives use). +Its wrong-disk guard is unit-established: `TestResolveStorageDevice_ToleratesDeviceLetterMove` (a +MATCHING UUID resolves to the current /dev path) + `TestResolveStorageDevice_AbsentAndScheme` (an +absent/mismatched UUID **ERRORS** — "skips a gone drive instead of fail-mounting"; a bare node / +non-`uuid:` scheme is refused). It resolves ONLY the exact UUID's symlink — never a "nearest" disk. +Combined with S4.1's restore-to-scratch proof, the composition's safety is established; a new live +scratch+marker-drive probe would add confidence but not new safety, so it was not built (recorded). -Agent **v0.68.0** live on felhom-pve; the temporary `local_backup_target=felhom-offsite` (set so the -selftest sourced the pbs-tier backup) **reverted to `local`**; WARN still 0; scratch band clean; -temp logs removed. Rollback binary: `felhom-agent.bak-0.67.0`. +## 6. Part 4-B — the destructive in-place 9201 drill: PREPARED + OPERATOR-GATED (not executed) -## 5. NOT yet live-validated / follow-ups / observations +Pre-flight **GREEN** (non-destructive): offsite `ct/9201` snapshots present (the restore source); +9201 running; the escrow blob is on the hub (383 B + 499 B identity, offsite key `b0:fe:2a…`), +exportable via `sqlite3 writefile`; **S4.1 already proved 9201 restores from offsite (pass:true)**. +The destructive drill (destroy 9201 → restore in place from offsite → re-attach the data drive by +durable_id) is **operator-present + STOP-gated**: the operator runs the R-consume steps +(`escrow-consume` + `identity-consume -install-wg-key`, R via env — **CC never runs these**, §9-4a) +and confirms the destroy at the STOP checkpoint. Prepared procedure + the go/no-go is with the +operator (destroying the live demo controller is their decision, even with the backup confirmed +recoverable). -- **The restore-test is now unattended-capable for the pbs tier** (deadline fixed; teardown proven - as the token). The scheduler cadence stays disabled on the demo (`restore_test_cadence_seconds=0`) - — the proof used the identical selftest path. -- **Tier-1(local, frequent)/Tier-2(offsite, periodic) split** still needed before offsite can be the - DEFAULT backup target (the controller's ~30-min cadence + a single target = near-continuous offsite - uploads). Unchanged from S4. -- Deferred: S5 DR consume (WG key + offsite restore with only R); hub-driven tenancy automation; - true-CGNAT-SIM retest. OPERATOR: publish 0.68.0 + Day-0 vouch. +## 7. NOT yet live-validated / observations + +- The live consume→install→tunnel→restore pipeline (the Part 4-B drill) — operator-present, pending + the operator's go-ahead + R-run. Parts 1-2 are unit-proven + deployed; their live consume path is + exercised only in that drill (R-gated). +- Deferred (per slice scope): guest_loss DR; hub-driven full-auto DR (the hub issuing recovery-mode + + directive itself); multi-guest host-loss. true-CGNAT-SIM retest still parked. +- Observation: the DR plan is surfaced via structured log + `Consumer.LastPlan()`; a report field + (so the hub/operator sees the plan without the box's journal) is a small future refinement.