felhom-agent/PLAN.md

Slice 6 Phase A — backup + restore-test orchestration (felhom-agent)

Context

The agent can observe/report storage (slice 5) and reconcile benign guest ops behind a signed-op gate (slice 4), but it has no guest-level backup/restore layer and — the point of this slice — no self-restore-test, which closes the "a backup you haven't restored isn't a backup" theme (doc 03 §8). This adds: a vzdump backup primitive that resolves the produced archive, a benign restore-to-a-new guest, and a journaled self-restore-test (restore → boot → verify → teardown) that inherits the slice-4 journal/serialization/crash recovery so a mid-test crash can't leak a scratch guest. Provisioning/identity-reset/golden base (§9) are slice 7; PBS/offsite/zero-knowledge (§8 offsite tier) are Phase B.

Everything here is benign (backup, restore-to-new, scratch teardown): reuse the slice-4 classifier/gate/journal — no new destructive class, no new crypto. Version → v0.6.0-rc1, stop at the checkpoint.

Locked decisions (from the task + clarifications):

• Backups are crash-consistent only (marked so in the report); app-consistency needs the controller quiesce (slice 8).
• Restore is to a NEW guest only (no overwrite anywhere this slice).
• Restore-test verify = net link-down (benign SetConfig) → boot → reaches running (no in-guest probe — slice 8). Link-down avoids the cloned source MAC/IP conflicting on a live host; it is test-safety, not slice-7 identity reset.
• Restore-test cadence defaults ON at 24h, configurable; runs only when a valid scratch VMID band is set; also on-demand via the selftest harness.
• Scratch VMID band 990000–990009 (lowest-free, excludes 9999/real guests); refuse to run if unset/invalid.
• Local target only this phase; PBSSnapshot stays a struct{} stub (Phase B).
• Bulk volumes: report the gap only (which backup=0 mountpoints the guest vzdump omits).

The crash-safe core (load-bearing)

The restore-test is journaled as a single entry whose VMID is the scratch id and Kind = "scratch_restore_test" with a new Scratch bool flag. The entry is terminal only after teardown — NOT when the restore sub-task's UPID completes. Recover must special-case Scratch entries before the generic UPID-recheck path (else it would mark the entry succeeded because the restore task ran OK, and drop it while the guest still exists → leak).

RunRestoreTest (pseudocode) — defer teardown so it runs on every path incl. failed verify:

vmid := pickScratchVMID(ListLXC, [min,max], exclude 9999)        // refuse if band unset/invalid
opID := "scratch-restore-"+vmid+"-"+seq
append({OpID:opID, VMID:vmid, Kind:"scratch_restore_test", Scratch:true, State:OpStarted})  // BEFORE any mutation
defer teardownScratch(opID, vmid)   // gate.Authorize(IntentForScratchDestroy)=benign -> DestroyLXC -> WaitTask -> terminal
upid := RestoreLXC{VMID:vmid, Archive, Storage:RestoreStorage}; waitOK(upid)   // UPID = error-detection, NOT terminal
for net := range GuestConfig(vmid).Nets(): SetConfig(vmid, {net: existing+",link_down=1"})  // benign
Start(vmid); res.Reached = waitRunning(GuestStatus, vmid, BootTimeout)
res = {Archive, ScratchVMID:vmid, Pass:res.Reached, Verified:"boot+running", Duration, Err}
return res    // defer tears down regardless

Recover scratch branch (added before the existing UPID logic):

if entry.Scratch {
    guests, err := ListLXC; if err { Unresolved++; continue }              // can't decide -> leave in-flight
    if !contains(guests, entry.VMID) { append(terminal(succeeded)); ScratchClean++; continue }  // already gone
    dec := gate.Authorize(IntentForScratchDestroy(hostID, entry.VMID), nil)
    if !dec.Allowed { Unresolved++; continue }                             // fail-safe (should be benign)
    upid, err := DestroyLXC(entry.VMID); if err { Unresolved++; continue } // retry next Recover
    waitOK(upid); append(terminal(succeeded)); ScratchDestroyed++
}

Idempotent: crash mid-destroy → next Recover finds the guest gone → ScratchClean. Both teardown paths (normal + recovery) go through gate.Authorize (benign ClassGuestDestroy + Provenance{AgentTaggedScratch:true}) for the audit trail.

Files to create

• internal/reconcile/restoretest.go (+ restoretest_test.go): RunRestoreTest(ctx, RestoreTestSpec{Archive, RestoreStorage, BootTimeout}) RestoreTestResult (engine method — it needs the journal/gate/queue internals), RestoreTestResult (reconcile-local data — must NOT return hub.RestoreTest, to avoid a reconcile→hub edge), IntentForScratchDestroy(hostID, vmid), pickScratchVMID. Runs the sequence on the scratch VMID's Queue lane.
• internal/backup/{doc,runner,store,schedule}.go (+ tests):
  • runner.go: BackupRunner.Backup(ctx, vmid) (hub.Backup, error) = Vzdump+WaitTask+volid-resolve + bulk-gap from GuestConfig.MountPoints() backup=0; LatestBackup(ctx, target) (volid, error); the RestoreTestRunner seam (satisfied by *reconcile.Engine); RestoreTestResult → hub.RestoreTest mapping.
  • store.go: mutex-guarded in-memory latest-Backup-per-target + latest-RestoreTest, implementing the hub BackupReporter/RestoreTestReporter seams.
  • schedule.go: cadence goroutine (default 24h; 0=disabled) → LatestBackup → engine.RunRestoreTest → write Store. No-ops cleanly when no backup exists yet.
  • Imports reconcile+hub+proxmox (acyclic; hub imports neither).
• configs/: example agent config with the backup block.

Files to modify

• internal/proxmox/mutate.go: DestroyLXC(ctx, vmid) → DELETE /nodes/{node}/lxc/{vmid} with purge=1&destroy-unreferenced-disks=1 via dataString (async→UPID); add Notes string to VzdumpOptions → notes-template param (probe notes-template vs notes on the demo PVE 9.2.2 first, read-only).
• internal/proxmox/query.go: LatestBackupVolID(ctx, store, vmid) (string, error) — StorageContent filtered Content=="backup" && VMID==vmid, max CTime.
• internal/reconcile/state.go: add RestoreLXC, DestroyLXC, GuestStatus to the GuestAPI interface (*proxmox.Client already satisfies all three).
• internal/reconcile/journal.go: add Scratch bool \json:"scratch,omitempty"`; carry it (and Kind) through terminal()`.
• internal/reconcile/recover.go: the scratch branch above; add ScratchClean/ScratchDestroyed to RecoverResult.
• internal/reconcile/recover_test.go + engine_test.go: fakeAPI gains RestoreLXC/DestroyLXC/GuestStatus recorders; new TestRecover_LeakedScratchDestroyed (in-flight Scratch + ListLXC returns the VMID → DestroyLXC called, gate benign, no longer in-flight), …AlreadyGone (ScratchClean), …ListUnreadable (Unresolved).
• internal/hub/report.go: fill Backup + RestoreTest structs (see wire shapes); PBSSnapshot stays struct{}.
• internal/hub/collect.go: add BackupReporter { Backups(ctx) []Backup } + RestoreTestReporter { RestoreTests(ctx) []RestoreTest } consumer seams (mirror StorageObserver); collectBackups/collectRestoreTests degrade nil/err → non-nil empty.
• internal/hub/contract_test.go + internal/hub/testdata/host-report.golden.json: populated backups[0]/restore_tests[0]; assert their key sets (bidirectional, slice-5 pattern).
• internal/config/config.go: BackupConfig{RestoreTestCadenceSeconds, ScratchVMIDMin, ScratchVMIDMax, LocalBackupTarget, RestoreStorage} + RestoreTestCadence() accessor (0→24h default) + env overlay; validate band (min>0, max>=min, 9999 excluded) only when cadence>0. Config.Validate stays proxmox-only.
• cmd/felhom-agent/main.go: wire backup.Store into NewCollector; add the cadence goroutine alongside engine.Run/loop.Run/watchdog.Run; add backup + restore-test to selftestFlag.Set + the switch + a -archive flag; runSelftestBackup/runSelftestRestoreTest reuse the runSelftestStorage wiring (NewGate(nil, hostID, SlogAudit{}, logger) + journal + engine).
• felhom.eu/hub: hub/internal/api/handler.go — add hostBackup/hostRestoreTest mirror structs to hostReportPayload, parse, persist via existing report_json (no new DB columns — slice-5 precedent), and log a FAILED restore-test prominently ([WARN], the loudest DR signal). hub/internal/api/testdata/host-report.golden.json byte-identical with the agent golden; host_test.go adds TestHostBackup_GoldenContract/TestHostRestoreTest_GoldenContract (bidirectional key-set, slice-5 pattern).
• CHANGELOG.md (prepend v0.6.0-rc1), REPORT.md (overwrite), CLAUDE.md current-state line.

Proposed wire shapes (draft — must land byte-identical in both repos)

type Backup struct {
    TargetID         string   `json:"target_id"`          // backup storage name
    VMID             int      `json:"vmid"`
    Archive          string   `json:"archive"`            // produced volid
    Mode             string   `json:"mode"`               // snapshot|stop
    CrashConsistent  bool     `json:"crash_consistent"`   // always true this slice
    SizeBytes        int64    `json:"size_bytes"`
    Success          bool     `json:"success"`
    Error            string   `json:"error,omitempty"`
    StartedAt        string   `json:"started_at"`         // RFC3339
    DurationSeconds  float64  `json:"duration_seconds"`
    UncoveredVolumes []string `json:"uncovered_volumes"`  // backup=0 mountpoints (bulk gap)
}
type RestoreTest struct {
    SourceArchive   string  `json:"source_archive"`
    SourceTier      string  `json:"source_tier"`     // "local" (pbs = Phase B)
    ScratchVMID     int     `json:"scratch_vmid"`
    Pass            bool    `json:"pass"`
    Verified        string  `json:"verified"`        // "boot+running"
    Error           string  `json:"error,omitempty"`
    TestedAt        string  `json:"tested_at"`       // RFC3339
    DurationSeconds float64 `json:"duration_seconds"`
}
type PBSSnapshot struct{} // Phase B stub

Verification

• go test ./... (local, Windows) + go test -race ./... on the build server (192.168.0.180, cgo) — the cadence adds a goroutine; the Store is mutex-guarded.
• New unit tests: vzdump async UPID→WaitTask + volid-resolve (fake API); restore-to-new benign ClassCreate passes the gate; restore-test end-to-end against fakes incl. (a) teardown-on-failed-verify and (b) journal-recovery cleanup (extend the slice-4 recover test: in-flight Scratch → Recover destroys the leaked guest, idempotent when already gone); cadence fires on interval / no-ops when disabled; cross-repo golden + hub-ingest key-set tests.
• Build the linux binary on 192.168.0.180, relay to felhom-pve (see memory demo-felhom-live-agent: MSYS_NO_PATHCONV=1, config at /root/.config/felhom-agent/agent.json, mode=direct, served-cert pin BA:7C:99…).
• Live on the demo (the checkpoint validation): --selftest=backup -vmid <small stopped guest> to a local target (e.g. felhom-usb or local, content=backup) → print the Backup record; then --selftest=restore-test -archive <volid> → restore into a 990000-band scratch guest, net-link-down, boot, verify running, teardown → print the RestoreTest record. Confirm no leaked scratch guest remains (pct list), and that a simulated mid-test crash + restart triggers Recover teardown.

Push & checkpoint

Push v0.6.0-rc1 to felhom-agent (and the hub changes to felhom.eu, deploy per the GitOps runbook if needed), update CHANGELOG/REPORT, then stop and await validation (restore-test teardown + recovery, the benign classifications, reporting) + the live demo restore-test. Phase B (next): PBS — datastore on the USB, zero-knowledge key custody, restore-from-PBS, PBS integrity-verify as the lighter frequent check.