docs: REPORT + CONTEXT for S5 safe halves (WG-key install + directive→plan)
Parts 1-2 shipped + deployed (v0.69.0); Part 3 not needed (sqlite3 export); Part 4-A re-attach safety already unit-proven; Part 4-B destructive 9201 drill prepared + operator-gated (pre-flight green), not executed by CC. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
This commit is contained in:
+16
@@ -5,6 +5,22 @@
|
|||||||
|
|
||||||
## Current
|
## Current
|
||||||
|
|
||||||
|
- **v0.69.0** (2026-07-04, live on felhom-pve) — **S5: host-loss DR — safe halves shipped**.
|
||||||
|
**Part 1** `wgtunnel.InstallRecoveredKey` — writes an escrow-recovered WG privkey (create-only,
|
||||||
|
refuse-overwrite) so the tunnel re-establishes with the SAME identity/pubkey (same /32), no keygen;
|
||||||
|
wired into `--selftest=identity-consume -install-wg-key` (opt-in; pre-S3 blob → logged fresh-keygen
|
||||||
|
fallback). **Part 2** new `internal/dr` — consumes the host_loss `restore_directive` (was
|
||||||
|
logged-ignored) into an inspectable RestorePlan via AddConsumer: per-guest {vmid,archive,target,
|
||||||
|
sizing} + per-drive {durable_id→mount} + offsite PBS coord; DERIVE-AND-SURFACE only (Consumer has
|
||||||
|
no restore/destroy dep — execute-nothing is structural). Tests + red-proofs (WG create-only; plan
|
||||||
|
mode-gate). **Part 3** hub escrow-GET NOT needed (operator exports the blob via `sqlite3 writefile`
|
||||||
|
on a cp'd hub.db). **Part 4-A** re-attach wrong-disk safety already unit-proven
|
||||||
|
(`ResolveStorageDevice`: match resolves, absent/mismatch ERRORS, non-uuid scheme refused — never a
|
||||||
|
near disk). **Part 4-B (destructive in-place 9201 restore) PREPARED + OPERATOR-GATED, NOT executed**
|
||||||
|
— pre-flight green (offsite ct/9201 restorable per S4.1); the operator runs the R-consume steps +
|
||||||
|
confirms the destroy (§9-4a: CC never runs a consume/R command — see [[operator-present-one-time-secrets]]).
|
||||||
|
OPEN: the operator-run 4-B drill; guest_loss DR; hub-driven full-auto DR. Rollback
|
||||||
|
felhom-agent.bak-0.68.0. Detail: REPORT.md + doc-06 §3.5/S5.
|
||||||
- **v0.68.0** (2026-07-04, live on felhom-pve) — **S4.1: unattended offsite restore-test**.
|
- **v0.68.0** (2026-07-04, live on felhom-pve) — **S4.1: unattended offsite restore-test**.
|
||||||
**Tier-aware restore-task deadline:** `RestoreTestSpec.RestoreTaskTimeout` (0→10m default) from
|
**Tier-aware restore-task deadline:** `RestoreTestSpec.RestoreTaskTimeout` (0→10m default) from
|
||||||
`config.RestoreTestPBSRestoreTimeoutSeconds` (accessor default **120m**), set only when
|
`config.RestoreTestPBSRestoreTimeoutSeconds` (accessor default **120m**), set only when
|
||||||
|
|||||||
@@ -1,72 +1,73 @@
|
|||||||
# REPORT — S4.1: unattended offsite restore-test (tier-aware deadline + teardown diagnosis)
|
# REPORT — S5: host-loss DR (recovered WG-key install + directive→PLAN; safe halves shipped)
|
||||||
|
|
||||||
**Date:** 2026-07-04 · **Class:** implementation + live diagnosis on felhom-pve (agent token).
|
**Date:** 2026-07-04 · **Class:** implementation (safe halves) + diagnosis + a prepared,
|
||||||
**Design of record:** `felhom.eu/documentation/architecture/06-offsite-connectivity.md` S4 row;
|
operator-present destructive drill (NOT executed by CC). **Design of record:** doc-06 §3.5 + S5 row;
|
||||||
runbook `offsite-endpoint.md` §4b.
|
doc-01 §9.
|
||||||
|
|
||||||
## 1. Baselines & commits (felhom-agent → `main`)
|
## 1. Baselines & commits (felhom-agent → `main`)
|
||||||
|
|
||||||
Start: `fee1fcf` (v0.67.0).
|
Start: `567cf9f` (v0.68.0). **`bd4bced`** v0.69.0 — the two safe DR halves (Parts 1-2). This commit
|
||||||
- **`dc70e15`** v0.68.0 — tier-aware restore-task deadline (Part 1).
|
— REPORT + CONTEXT + doc-06 S5 row.
|
||||||
- this commit — REPORT + CONTEXT (the diagnosis outcome).
|
|
||||||
|
|
||||||
## 2. Part 1 — tier-aware restore-task deadline
|
## 2. Part 1 (SAFE, shipped) — install the recovered WG key
|
||||||
|
|
||||||
The restore-task wait (`restoretest.go` ~L246) used `WaitOptions{}` → the 10m default. A WAN (pbs)
|
`wgtunnel.InstallRecoveredKey(stateDir, privB64)` writes an escrow-recovered WG private key (32-byte
|
||||||
restore of a large guest exceeds it, so the wait expired mid-restore and teardown then fired against
|
base64, re-encoded canonical) to the key file so the tunnel re-establishes with the **same
|
||||||
a still-restoring guest. Fix:
|
identity/pubkey → same hub /32**, no fresh keygen. **CREATE-ONLY** — refuses if a key file exists (a
|
||||||
- `RestoreTestSpec.RestoreTaskTimeout` (0 → 10m default); the restore `WaitTask` passes it.
|
present key may be a live identity); the value is never logged. Wired into
|
||||||
- `config.BackupConfig.RestoreTestPBSRestoreTimeoutSeconds` + accessor `RestoreTestPBSRestoreTimeout()`
|
`--selftest=identity-consume -install-wg-key` (opt-in; installs `bundle.WGPrivateKey` after
|
||||||
(positive as-is, else **120m**).
|
`UnwrapIdentityBundle`; a pre-S3 blob with no WG key → logged fallback to fresh keygen + re-register,
|
||||||
- `main.restoreTaskTimeout(cfg, tier)` — the configured PBS timeout only when `SourceTier=="pbs"`,
|
which keeps the /32 via hub re-key-in-place).
|
||||||
else 0. Set at BOTH spec builds (scheduler + selftest). **Local tier unchanged (10m).**
|
**Test + red-proof:** install→`EnsureKey` LOADS it (same pubkey, no keygen); a second install REFUSES
|
||||||
- Tests: `WaitOptions.Timeout` pbs→120m / local→0 (red-proofed against `WaitOptions{}`: pbs
|
(create-only — red-proofed against dropping the exists-check); invalid key → error, nothing written.
|
||||||
assertion fails); accessor contract (0/neg→120m, positive as-is). Full `go test ./...` green.
|
|
||||||
|
|
||||||
## 3. Part 2 — teardown diagnosis (the "grant VM.Allocate" follow-up) — PHANTOM CONFIRMED
|
## 3. Part 2 (SAFE, shipped) — consume `restore_directive` → PLAN
|
||||||
|
|
||||||
Ran the restore-test on the **agent-token path** (selftest = the identical `RunRestoreTest` the
|
New `internal/dr`: the host_loss `restore_directive` (was logged-and-ignored in `syncer.go`) is
|
||||||
scheduler runs; all PVE ops authenticate as `felhom-agent@pve!agent`), sourcing the **offsite
|
consumed via the `desired.Syncer.AddConsumer` raw seam into an inspectable **RestorePlan** — per
|
||||||
(pbs-tier)** backup of guest 9201, with v0.68.0's 120m timeout.
|
guest `{vmid, archive, target storage, sizing}`, per drive `{durable_id → expected mount}`, + the
|
||||||
|
offsite PBS coord. **Derive-and-surface only:** the `Consumer` holds NO restore/destroy dependency, so
|
||||||
|
"execute nothing" is structural. `guest_loss`/absent → no plan; recipe fetched on-demand (rare
|
||||||
|
directive) via a fresh `Collect`.
|
||||||
|
**Test + red-proof:** host_loss builds the expected plan; guest_loss/absent/nil-recipe → none
|
||||||
|
(red-proofed against a relaxed mode gate); the consumer surfaces on host_loss + clears otherwise +
|
||||||
|
consults the recipe only then. Full `go test ./...` green; deployed 0.69.0; `-install-wg-key` flag
|
||||||
|
live; daemon starts clean (dr consumer wired).
|
||||||
|
|
||||||
**Scenario B (done-criterion) — GREEN:**
|
## 4. Part 3 — hub escrow export: NOT needed
|
||||||
```
|
|
||||||
source_archive: felhom-offsite:backup/ct/9201/2026-07-04T14:55:44Z
|
|
||||||
source_tier: pbs · pass: true · verified: boot+running · scratch_vmid: 990000 · duration 4m49s
|
|
||||||
restore → net-link-down → boot (running) → teardown → scratch band clean (no leak)
|
|
||||||
```
|
|
||||||
|
|
||||||
**Scenario C — the teardown succeeded as the agent token; (b) is a PHANTOM:**
|
The operator has a clean blob-export path already: `sqlite3 … "SELECT writefile('/tmp/blob', blob),
|
||||||
```
|
writefile('/tmp/idblob', identity_blob) FROM host_escrow WHERE host_id='demo-felhom-01'"` on a
|
||||||
gate decision: class=guest_destroy guest=990000 allowed=true reason=benign
|
`kubectl cp`'d `hub.db`. No hub change (respects the read-only-unless-no-clean-way gate).
|
||||||
restore-test: scratch guest torn down vmid=990000
|
|
||||||
pct list | grep 9900 → no scratch guests
|
|
||||||
```
|
|
||||||
No 403, no `VM.Allocate` error. Root cause of the earlier 403 (REPORT of S4): the 10m wait expired
|
|
||||||
**mid-restore**, so teardown hit a guest that existed but was **not yet pool-associated** (the pool
|
|
||||||
membership lands with restore completion) — a bare `/vms/990000` the pool ACL didn't cover. With the
|
|
||||||
timeout fixed → the restore completes → the guest is a member of `/pool/felhom` (restored with
|
|
||||||
`Pool=DefaultPool`, `restoretest.go` L229) → its destroy is authorized by the existing
|
|
||||||
`FelhomAgentGuest` grant (`VM.Allocate` + `Pool.Allocate` at `/pool/felhom`). **No ACL change, no
|
|
||||||
host-install edit** — the pool-scoped containment (`SPIKE-pool-scoped-acl-2026-07-01`) stands.
|
|
||||||
|
|
||||||
(Note: this run's restore was fast — home *download* ≫ *upload*; the earlier ~10-min stall was link
|
## 5. Part 4-A — re-attach wrong-disk safety: ALREADY PROVEN (no new probe built)
|
||||||
contention. The 120m ceiling covers the slow case regardless — for an unattended nightly test a
|
|
||||||
false timeout is the failure mode to prevent.)
|
|
||||||
|
|
||||||
## 4. Live state after / cleanup
|
The re-attach matcher is `storage.ResolveStorageDevice` (the `uuid:<fs-uuid>` scheme DR drives use).
|
||||||
|
Its wrong-disk guard is unit-established: `TestResolveStorageDevice_ToleratesDeviceLetterMove` (a
|
||||||
|
MATCHING UUID resolves to the current /dev path) + `TestResolveStorageDevice_AbsentAndScheme` (an
|
||||||
|
absent/mismatched UUID **ERRORS** — "skips a gone drive instead of fail-mounting"; a bare node /
|
||||||
|
non-`uuid:` scheme is refused). It resolves ONLY the exact UUID's symlink — never a "nearest" disk.
|
||||||
|
Combined with S4.1's restore-to-scratch proof, the composition's safety is established; a new live
|
||||||
|
scratch+marker-drive probe would add confidence but not new safety, so it was not built (recorded).
|
||||||
|
|
||||||
Agent **v0.68.0** live on felhom-pve; the temporary `local_backup_target=felhom-offsite` (set so the
|
## 6. Part 4-B — the destructive in-place 9201 drill: PREPARED + OPERATOR-GATED (not executed)
|
||||||
selftest sourced the pbs-tier backup) **reverted to `local`**; WARN still 0; scratch band clean;
|
|
||||||
temp logs removed. Rollback binary: `felhom-agent.bak-0.67.0`.
|
|
||||||
|
|
||||||
## 5. NOT yet live-validated / follow-ups / observations
|
Pre-flight **GREEN** (non-destructive): offsite `ct/9201` snapshots present (the restore source);
|
||||||
|
9201 running; the escrow blob is on the hub (383 B + 499 B identity, offsite key `b0:fe:2a…`),
|
||||||
|
exportable via `sqlite3 writefile`; **S4.1 already proved 9201 restores from offsite (pass:true)**.
|
||||||
|
The destructive drill (destroy 9201 → restore in place from offsite → re-attach the data drive by
|
||||||
|
durable_id) is **operator-present + STOP-gated**: the operator runs the R-consume steps
|
||||||
|
(`escrow-consume` + `identity-consume -install-wg-key`, R via env — **CC never runs these**, §9-4a)
|
||||||
|
and confirms the destroy at the STOP checkpoint. Prepared procedure + the go/no-go is with the
|
||||||
|
operator (destroying the live demo controller is their decision, even with the backup confirmed
|
||||||
|
recoverable).
|
||||||
|
|
||||||
- **The restore-test is now unattended-capable for the pbs tier** (deadline fixed; teardown proven
|
## 7. NOT yet live-validated / observations
|
||||||
as the token). The scheduler cadence stays disabled on the demo (`restore_test_cadence_seconds=0`)
|
|
||||||
— the proof used the identical selftest path.
|
- The live consume→install→tunnel→restore pipeline (the Part 4-B drill) — operator-present, pending
|
||||||
- **Tier-1(local, frequent)/Tier-2(offsite, periodic) split** still needed before offsite can be the
|
the operator's go-ahead + R-run. Parts 1-2 are unit-proven + deployed; their live consume path is
|
||||||
DEFAULT backup target (the controller's ~30-min cadence + a single target = near-continuous offsite
|
exercised only in that drill (R-gated).
|
||||||
uploads). Unchanged from S4.
|
- Deferred (per slice scope): guest_loss DR; hub-driven full-auto DR (the hub issuing recovery-mode +
|
||||||
- Deferred: S5 DR consume (WG key + offsite restore with only R); hub-driven tenancy automation;
|
directive itself); multi-guest host-loss. true-CGNAT-SIM retest still parked.
|
||||||
true-CGNAT-SIM retest. OPERATOR: publish 0.68.0 + Day-0 vouch.
|
- Observation: the DR plan is surfaced via structured log + `Consumer.LastPlan()`; a report field
|
||||||
|
(so the hub/operator sees the plan without the box's journal) is a small future refinement.
|
||||||
|
|||||||
Reference in New Issue
Block a user