diff --git a/PLAN.md b/PLAN.md new file mode 100644 index 0000000..65042bf --- /dev/null +++ b/PLAN.md @@ -0,0 +1,136 @@ +# Slice 6 Phase A — backup + restore-test orchestration (felhom-agent) + +## Context + +The agent can observe/report storage (slice 5) and reconcile benign guest ops behind a +signed-op gate (slice 4), but it has no guest-level **backup/restore** layer and — the point +of this slice — no **self-restore-test**, which closes the "a backup you haven't restored +isn't a backup" theme (doc 03 §8). This adds: a vzdump backup primitive that resolves the +produced archive, a benign restore-to-a-**new** guest, and a journaled **self-restore-test** +(restore → boot → verify → teardown) that inherits the slice-4 journal/serialization/crash +recovery so a mid-test crash can't leak a scratch guest. Provisioning/identity-reset/golden +base (§9) are **slice 7**; PBS/offsite/zero-knowledge (§8 offsite tier) are **Phase B**. + +Everything here is **benign** (backup, restore-to-new, scratch teardown): reuse the slice-4 +classifier/gate/journal — **no new destructive class, no new crypto**. Version → **v0.6.0-rc1**, +stop at the checkpoint. + +**Locked decisions (from the task + clarifications):** +- Backups are **crash-consistent only** (marked so in the report); app-consistency needs the + controller quiesce (slice 8). +- Restore is **to a NEW guest only** (no overwrite anywhere this slice). +- Restore-test verify = **net link-down (benign SetConfig) → boot → reaches `running`** (no + in-guest probe — slice 8). Link-down avoids the cloned source MAC/IP conflicting on a live + host; it is test-safety, **not** slice-7 identity reset. +- Restore-test cadence **defaults ON at 24h**, configurable; runs only when a valid scratch + VMID band is set; also on-demand via the selftest harness. +- Scratch VMID band **990000–990009** (lowest-free, excludes 9999/real guests); refuse to run + if unset/invalid. +- **Local target only** this phase; `PBSSnapshot` stays a `struct{}` stub (Phase B). +- Bulk volumes: **report the gap only** (which `backup=0` mountpoints the guest vzdump omits). + +## The crash-safe core (load-bearing) + +The restore-test is journaled as a **single** entry whose `VMID` is the scratch id and +`Kind = "scratch_restore_test"` with a new `Scratch bool` flag. The entry is **terminal only +after teardown** — NOT when the restore sub-task's UPID completes. `Recover` must special-case +Scratch entries **before** the generic UPID-recheck path (else it would mark the entry +`succeeded` because the restore task ran OK, and drop it while the guest still exists → leak). + +`RunRestoreTest` (pseudocode) — `defer` teardown so it runs on every path incl. failed verify: +``` +vmid := pickScratchVMID(ListLXC, [min,max], exclude 9999) // refuse if band unset/invalid +opID := "scratch-restore-"+vmid+"-"+seq +append({OpID:opID, VMID:vmid, Kind:"scratch_restore_test", Scratch:true, State:OpStarted}) // BEFORE any mutation +defer teardownScratch(opID, vmid) // gate.Authorize(IntentForScratchDestroy)=benign -> DestroyLXC -> WaitTask -> terminal +upid := RestoreLXC{VMID:vmid, Archive, Storage:RestoreStorage}; waitOK(upid) // UPID = error-detection, NOT terminal +for net := range GuestConfig(vmid).Nets(): SetConfig(vmid, {net: existing+",link_down=1"}) // benign +Start(vmid); res.Reached = waitRunning(GuestStatus, vmid, BootTimeout) +res = {Archive, ScratchVMID:vmid, Pass:res.Reached, Verified:"boot+running", Duration, Err} +return res // defer tears down regardless +``` + +`Recover` scratch branch (added before the existing UPID logic): +``` +if entry.Scratch { + guests, err := ListLXC; if err { Unresolved++; continue } // can't decide -> leave in-flight + if !contains(guests, entry.VMID) { append(terminal(succeeded)); ScratchClean++; continue } // already gone + dec := gate.Authorize(IntentForScratchDestroy(hostID, entry.VMID), nil) + if !dec.Allowed { Unresolved++; continue } // fail-safe (should be benign) + upid, err := DestroyLXC(entry.VMID); if err { Unresolved++; continue } // retry next Recover + waitOK(upid); append(terminal(succeeded)); ScratchDestroyed++ +} +``` +Idempotent: crash mid-destroy → next `Recover` finds the guest gone → `ScratchClean`. Both +teardown paths (normal + recovery) go through `gate.Authorize` (benign `ClassGuestDestroy` + +`Provenance{AgentTaggedScratch:true}`) for the audit trail. + +## Files to create + +- **`internal/reconcile/restoretest.go`** (+ `restoretest_test.go`): `RunRestoreTest(ctx, RestoreTestSpec{Archive, RestoreStorage, BootTimeout}) RestoreTestResult` (engine method — it needs the journal/gate/queue internals), `RestoreTestResult` (reconcile-local data — must NOT return `hub.RestoreTest`, to avoid a reconcile→hub edge), `IntentForScratchDestroy(hostID, vmid)`, `pickScratchVMID`. Runs the sequence on the scratch VMID's `Queue` lane. +- **`internal/backup/{doc,runner,store,schedule}.go`** (+ tests): + - `runner.go`: `BackupRunner.Backup(ctx, vmid) (hub.Backup, error)` = `Vzdump`+`WaitTask`+volid-resolve + bulk-gap from `GuestConfig.MountPoints()` `backup=0`; `LatestBackup(ctx, target) (volid, error)`; the `RestoreTestRunner` seam (satisfied by `*reconcile.Engine`); `RestoreTestResult → hub.RestoreTest` mapping. + - `store.go`: mutex-guarded in-memory latest-`Backup`-per-target + latest-`RestoreTest`, implementing the hub `BackupReporter`/`RestoreTestReporter` seams. + - `schedule.go`: cadence goroutine (default 24h; 0=disabled) → `LatestBackup` → `engine.RunRestoreTest` → write `Store`. No-ops cleanly when no backup exists yet. + - Imports `reconcile`+`hub`+`proxmox` (acyclic; hub imports neither). +- **`configs/`**: example agent config with the `backup` block. + +## Files to modify + +- **`internal/proxmox/mutate.go`**: `DestroyLXC(ctx, vmid)` → `DELETE /nodes/{node}/lxc/{vmid}` with `purge=1&destroy-unreferenced-disks=1` via `dataString` (async→UPID); add `Notes string` to `VzdumpOptions` → `notes-template` param (probe `notes-template` vs `notes` on the demo PVE 9.2.2 first, read-only). +- **`internal/proxmox/query.go`**: `LatestBackupVolID(ctx, store, vmid) (string, error)` — `StorageContent` filtered `Content=="backup" && VMID==vmid`, max `CTime`. +- **`internal/reconcile/state.go`**: add `RestoreLXC`, `DestroyLXC`, `GuestStatus` to the `GuestAPI` interface (`*proxmox.Client` already satisfies all three). +- **`internal/reconcile/journal.go`**: add `Scratch bool \`json:"scratch,omitempty"\``; carry it (and `Kind`) through `terminal()`. +- **`internal/reconcile/recover.go`**: the scratch branch above; add `ScratchClean`/`ScratchDestroyed` to `RecoverResult`. +- **`internal/reconcile/recover_test.go`** + **`engine_test.go`**: `fakeAPI` gains `RestoreLXC`/`DestroyLXC`/`GuestStatus` recorders; new `TestRecover_LeakedScratchDestroyed` (in-flight Scratch + ListLXC returns the VMID → DestroyLXC called, gate benign, no longer in-flight), `…AlreadyGone` (ScratchClean), `…ListUnreadable` (Unresolved). +- **`internal/hub/report.go`**: fill `Backup` + `RestoreTest` structs (see wire shapes); `PBSSnapshot` stays `struct{}`. +- **`internal/hub/collect.go`**: add `BackupReporter { Backups(ctx) []Backup }` + `RestoreTestReporter { RestoreTests(ctx) []RestoreTest }` consumer seams (mirror `StorageObserver`); `collectBackups`/`collectRestoreTests` degrade nil/err → non-nil empty. +- **`internal/hub/contract_test.go`** + **`internal/hub/testdata/host-report.golden.json`**: populated `backups[0]`/`restore_tests[0]`; assert their key sets (bidirectional, slice-5 pattern). +- **`internal/config/config.go`**: `BackupConfig{RestoreTestCadenceSeconds, ScratchVMIDMin, ScratchVMIDMax, LocalBackupTarget, RestoreStorage}` + `RestoreTestCadence()` accessor (0→24h default) + env overlay; validate band (min>0, max>=min, 9999 excluded) only when cadence>0. `Config.Validate` stays proxmox-only. +- **`cmd/felhom-agent/main.go`**: wire `backup.Store` into `NewCollector`; add the cadence goroutine alongside `engine.Run`/`loop.Run`/`watchdog.Run`; add `backup` + `restore-test` to `selftestFlag.Set` + the switch + a `-archive` flag; `runSelftestBackup`/`runSelftestRestoreTest` reuse the `runSelftestStorage` wiring (`NewGate(nil, hostID, SlogAudit{}, logger)` + journal + engine). +- **felhom.eu/hub**: `hub/internal/api/handler.go` — add `hostBackup`/`hostRestoreTest` mirror structs to `hostReportPayload`, parse, persist via existing `report_json` (no new DB columns — slice-5 precedent), and **log a FAILED restore-test prominently** (`[WARN]`, the loudest DR signal). `hub/internal/api/testdata/host-report.golden.json` byte-identical with the agent golden; `host_test.go` adds `TestHostBackup_GoldenContract`/`TestHostRestoreTest_GoldenContract` (bidirectional key-set, slice-5 pattern). +- **`CHANGELOG.md`** (prepend v0.6.0-rc1), **`REPORT.md`** (overwrite), `CLAUDE.md` current-state line. + +## Proposed wire shapes (draft — must land byte-identical in both repos) + +```go +type Backup struct { + TargetID string `json:"target_id"` // backup storage name + VMID int `json:"vmid"` + Archive string `json:"archive"` // produced volid + Mode string `json:"mode"` // snapshot|stop + CrashConsistent bool `json:"crash_consistent"` // always true this slice + SizeBytes int64 `json:"size_bytes"` + Success bool `json:"success"` + Error string `json:"error,omitempty"` + StartedAt string `json:"started_at"` // RFC3339 + DurationSeconds float64 `json:"duration_seconds"` + UncoveredVolumes []string `json:"uncovered_volumes"` // backup=0 mountpoints (bulk gap) +} +type RestoreTest struct { + SourceArchive string `json:"source_archive"` + SourceTier string `json:"source_tier"` // "local" (pbs = Phase B) + ScratchVMID int `json:"scratch_vmid"` + Pass bool `json:"pass"` + Verified string `json:"verified"` // "boot+running" + Error string `json:"error,omitempty"` + TestedAt string `json:"tested_at"` // RFC3339 + DurationSeconds float64 `json:"duration_seconds"` +} +type PBSSnapshot struct{} // Phase B stub +``` + +## Verification + +- `go test ./...` (local, Windows) + `go test -race ./...` on the build server (192.168.0.180, cgo) — the cadence adds a goroutine; the Store is mutex-guarded. +- New unit tests: vzdump async UPID→WaitTask + volid-resolve (fake API); restore-to-new benign `ClassCreate` passes the gate; restore-test end-to-end against fakes incl. **(a) teardown-on-failed-verify** and **(b) journal-recovery cleanup** (extend the slice-4 recover test: in-flight Scratch → `Recover` destroys the leaked guest, idempotent when already gone); cadence fires on interval / no-ops when disabled; cross-repo golden + hub-ingest key-set tests. +- Build the linux binary on 192.168.0.180, relay to `felhom-pve` (see memory `demo-felhom-live-agent`: `MSYS_NO_PATHCONV=1`, config at `/root/.config/felhom-agent/agent.json`, mode=direct, served-cert pin `BA:7C:99…`). +- **Live on the demo** (the checkpoint validation): `--selftest=backup -vmid ` to a local target (e.g. `felhom-usb` or `local`, content=backup) → print the `Backup` record; then `--selftest=restore-test -archive ` → restore into a 990000-band scratch guest, net-link-down, boot, verify `running`, teardown → print the `RestoreTest` record. Confirm no leaked scratch guest remains (`pct list`), and that a simulated mid-test crash + restart triggers `Recover` teardown. + +## Push & checkpoint + +Push **v0.6.0-rc1** to felhom-agent (and the hub changes to felhom.eu, deploy per the GitOps +runbook if needed), update CHANGELOG/REPORT, then **stop and await validation** (restore-test +teardown + recovery, the benign classifications, reporting) + the live demo restore-test. +**Phase B (next): PBS** — datastore on the USB, zero-knowledge key custody, restore-from-PBS, +PBS integrity-verify as the lighter frequent check.