137 lines
12 KiB
Markdown
137 lines
12 KiB
Markdown
# Slice 6 Phase A — backup + restore-test orchestration (felhom-agent)
|
||
|
||
## Context
|
||
|
||
The agent can observe/report storage (slice 5) and reconcile benign guest ops behind a
|
||
signed-op gate (slice 4), but it has no guest-level **backup/restore** layer and — the point
|
||
of this slice — no **self-restore-test**, which closes the "a backup you haven't restored
|
||
isn't a backup" theme (doc 03 §8). This adds: a vzdump backup primitive that resolves the
|
||
produced archive, a benign restore-to-a-**new** guest, and a journaled **self-restore-test**
|
||
(restore → boot → verify → teardown) that inherits the slice-4 journal/serialization/crash
|
||
recovery so a mid-test crash can't leak a scratch guest. Provisioning/identity-reset/golden
|
||
base (§9) are **slice 7**; PBS/offsite/zero-knowledge (§8 offsite tier) are **Phase B**.
|
||
|
||
Everything here is **benign** (backup, restore-to-new, scratch teardown): reuse the slice-4
|
||
classifier/gate/journal — **no new destructive class, no new crypto**. Version → **v0.6.0-rc1**,
|
||
stop at the checkpoint.
|
||
|
||
**Locked decisions (from the task + clarifications):**
|
||
- Backups are **crash-consistent only** (marked so in the report); app-consistency needs the
|
||
controller quiesce (slice 8).
|
||
- Restore is **to a NEW guest only** (no overwrite anywhere this slice).
|
||
- Restore-test verify = **net link-down (benign SetConfig) → boot → reaches `running`** (no
|
||
in-guest probe — slice 8). Link-down avoids the cloned source MAC/IP conflicting on a live
|
||
host; it is test-safety, **not** slice-7 identity reset.
|
||
- Restore-test cadence **defaults ON at 24h**, configurable; runs only when a valid scratch
|
||
VMID band is set; also on-demand via the selftest harness.
|
||
- Scratch VMID band **990000–990009** (lowest-free, excludes 9999/real guests); refuse to run
|
||
if unset/invalid.
|
||
- **Local target only** this phase; `PBSSnapshot` stays a `struct{}` stub (Phase B).
|
||
- Bulk volumes: **report the gap only** (which `backup=0` mountpoints the guest vzdump omits).
|
||
|
||
## The crash-safe core (load-bearing)
|
||
|
||
The restore-test is journaled as a **single** entry whose `VMID` is the scratch id and
|
||
`Kind = "scratch_restore_test"` with a new `Scratch bool` flag. The entry is **terminal only
|
||
after teardown** — NOT when the restore sub-task's UPID completes. `Recover` must special-case
|
||
Scratch entries **before** the generic UPID-recheck path (else it would mark the entry
|
||
`succeeded` because the restore task ran OK, and drop it while the guest still exists → leak).
|
||
|
||
`RunRestoreTest` (pseudocode) — `defer` teardown so it runs on every path incl. failed verify:
|
||
```
|
||
vmid := pickScratchVMID(ListLXC, [min,max], exclude 9999) // refuse if band unset/invalid
|
||
opID := "scratch-restore-"+vmid+"-"+seq
|
||
append({OpID:opID, VMID:vmid, Kind:"scratch_restore_test", Scratch:true, State:OpStarted}) // BEFORE any mutation
|
||
defer teardownScratch(opID, vmid) // gate.Authorize(IntentForScratchDestroy)=benign -> DestroyLXC -> WaitTask -> terminal
|
||
upid := RestoreLXC{VMID:vmid, Archive, Storage:RestoreStorage}; waitOK(upid) // UPID = error-detection, NOT terminal
|
||
for net := range GuestConfig(vmid).Nets(): SetConfig(vmid, {net: existing+",link_down=1"}) // benign
|
||
Start(vmid); res.Reached = waitRunning(GuestStatus, vmid, BootTimeout)
|
||
res = {Archive, ScratchVMID:vmid, Pass:res.Reached, Verified:"boot+running", Duration, Err}
|
||
return res // defer tears down regardless
|
||
```
|
||
|
||
`Recover` scratch branch (added before the existing UPID logic):
|
||
```
|
||
if entry.Scratch {
|
||
guests, err := ListLXC; if err { Unresolved++; continue } // can't decide -> leave in-flight
|
||
if !contains(guests, entry.VMID) { append(terminal(succeeded)); ScratchClean++; continue } // already gone
|
||
dec := gate.Authorize(IntentForScratchDestroy(hostID, entry.VMID), nil)
|
||
if !dec.Allowed { Unresolved++; continue } // fail-safe (should be benign)
|
||
upid, err := DestroyLXC(entry.VMID); if err { Unresolved++; continue } // retry next Recover
|
||
waitOK(upid); append(terminal(succeeded)); ScratchDestroyed++
|
||
}
|
||
```
|
||
Idempotent: crash mid-destroy → next `Recover` finds the guest gone → `ScratchClean`. Both
|
||
teardown paths (normal + recovery) go through `gate.Authorize` (benign `ClassGuestDestroy` +
|
||
`Provenance{AgentTaggedScratch:true}`) for the audit trail.
|
||
|
||
## Files to create
|
||
|
||
- **`internal/reconcile/restoretest.go`** (+ `restoretest_test.go`): `RunRestoreTest(ctx, RestoreTestSpec{Archive, RestoreStorage, BootTimeout}) RestoreTestResult` (engine method — it needs the journal/gate/queue internals), `RestoreTestResult` (reconcile-local data — must NOT return `hub.RestoreTest`, to avoid a reconcile→hub edge), `IntentForScratchDestroy(hostID, vmid)`, `pickScratchVMID`. Runs the sequence on the scratch VMID's `Queue` lane.
|
||
- **`internal/backup/{doc,runner,store,schedule}.go`** (+ tests):
|
||
- `runner.go`: `BackupRunner.Backup(ctx, vmid) (hub.Backup, error)` = `Vzdump`+`WaitTask`+volid-resolve + bulk-gap from `GuestConfig.MountPoints()` `backup=0`; `LatestBackup(ctx, target) (volid, error)`; the `RestoreTestRunner` seam (satisfied by `*reconcile.Engine`); `RestoreTestResult → hub.RestoreTest` mapping.
|
||
- `store.go`: mutex-guarded in-memory latest-`Backup`-per-target + latest-`RestoreTest`, implementing the hub `BackupReporter`/`RestoreTestReporter` seams.
|
||
- `schedule.go`: cadence goroutine (default 24h; 0=disabled) → `LatestBackup` → `engine.RunRestoreTest` → write `Store`. No-ops cleanly when no backup exists yet.
|
||
- Imports `reconcile`+`hub`+`proxmox` (acyclic; hub imports neither).
|
||
- **`configs/`**: example agent config with the `backup` block.
|
||
|
||
## Files to modify
|
||
|
||
- **`internal/proxmox/mutate.go`**: `DestroyLXC(ctx, vmid)` → `DELETE /nodes/{node}/lxc/{vmid}` with `purge=1&destroy-unreferenced-disks=1` via `dataString` (async→UPID); add `Notes string` to `VzdumpOptions` → `notes-template` param (probe `notes-template` vs `notes` on the demo PVE 9.2.2 first, read-only).
|
||
- **`internal/proxmox/query.go`**: `LatestBackupVolID(ctx, store, vmid) (string, error)` — `StorageContent` filtered `Content=="backup" && VMID==vmid`, max `CTime`.
|
||
- **`internal/reconcile/state.go`**: add `RestoreLXC`, `DestroyLXC`, `GuestStatus` to the `GuestAPI` interface (`*proxmox.Client` already satisfies all three).
|
||
- **`internal/reconcile/journal.go`**: add `Scratch bool \`json:"scratch,omitempty"\``; carry it (and `Kind`) through `terminal()`.
|
||
- **`internal/reconcile/recover.go`**: the scratch branch above; add `ScratchClean`/`ScratchDestroyed` to `RecoverResult`.
|
||
- **`internal/reconcile/recover_test.go`** + **`engine_test.go`**: `fakeAPI` gains `RestoreLXC`/`DestroyLXC`/`GuestStatus` recorders; new `TestRecover_LeakedScratchDestroyed` (in-flight Scratch + ListLXC returns the VMID → DestroyLXC called, gate benign, no longer in-flight), `…AlreadyGone` (ScratchClean), `…ListUnreadable` (Unresolved).
|
||
- **`internal/hub/report.go`**: fill `Backup` + `RestoreTest` structs (see wire shapes); `PBSSnapshot` stays `struct{}`.
|
||
- **`internal/hub/collect.go`**: add `BackupReporter { Backups(ctx) []Backup }` + `RestoreTestReporter { RestoreTests(ctx) []RestoreTest }` consumer seams (mirror `StorageObserver`); `collectBackups`/`collectRestoreTests` degrade nil/err → non-nil empty.
|
||
- **`internal/hub/contract_test.go`** + **`internal/hub/testdata/host-report.golden.json`**: populated `backups[0]`/`restore_tests[0]`; assert their key sets (bidirectional, slice-5 pattern).
|
||
- **`internal/config/config.go`**: `BackupConfig{RestoreTestCadenceSeconds, ScratchVMIDMin, ScratchVMIDMax, LocalBackupTarget, RestoreStorage}` + `RestoreTestCadence()` accessor (0→24h default) + env overlay; validate band (min>0, max>=min, 9999 excluded) only when cadence>0. `Config.Validate` stays proxmox-only.
|
||
- **`cmd/felhom-agent/main.go`**: wire `backup.Store` into `NewCollector`; add the cadence goroutine alongside `engine.Run`/`loop.Run`/`watchdog.Run`; add `backup` + `restore-test` to `selftestFlag.Set` + the switch + a `-archive` flag; `runSelftestBackup`/`runSelftestRestoreTest` reuse the `runSelftestStorage` wiring (`NewGate(nil, hostID, SlogAudit{}, logger)` + journal + engine).
|
||
- **felhom.eu/hub**: `hub/internal/api/handler.go` — add `hostBackup`/`hostRestoreTest` mirror structs to `hostReportPayload`, parse, persist via existing `report_json` (no new DB columns — slice-5 precedent), and **log a FAILED restore-test prominently** (`[WARN]`, the loudest DR signal). `hub/internal/api/testdata/host-report.golden.json` byte-identical with the agent golden; `host_test.go` adds `TestHostBackup_GoldenContract`/`TestHostRestoreTest_GoldenContract` (bidirectional key-set, slice-5 pattern).
|
||
- **`CHANGELOG.md`** (prepend v0.6.0-rc1), **`REPORT.md`** (overwrite), `CLAUDE.md` current-state line.
|
||
|
||
## Proposed wire shapes (draft — must land byte-identical in both repos)
|
||
|
||
```go
|
||
type Backup struct {
|
||
TargetID string `json:"target_id"` // backup storage name
|
||
VMID int `json:"vmid"`
|
||
Archive string `json:"archive"` // produced volid
|
||
Mode string `json:"mode"` // snapshot|stop
|
||
CrashConsistent bool `json:"crash_consistent"` // always true this slice
|
||
SizeBytes int64 `json:"size_bytes"`
|
||
Success bool `json:"success"`
|
||
Error string `json:"error,omitempty"`
|
||
StartedAt string `json:"started_at"` // RFC3339
|
||
DurationSeconds float64 `json:"duration_seconds"`
|
||
UncoveredVolumes []string `json:"uncovered_volumes"` // backup=0 mountpoints (bulk gap)
|
||
}
|
||
type RestoreTest struct {
|
||
SourceArchive string `json:"source_archive"`
|
||
SourceTier string `json:"source_tier"` // "local" (pbs = Phase B)
|
||
ScratchVMID int `json:"scratch_vmid"`
|
||
Pass bool `json:"pass"`
|
||
Verified string `json:"verified"` // "boot+running"
|
||
Error string `json:"error,omitempty"`
|
||
TestedAt string `json:"tested_at"` // RFC3339
|
||
DurationSeconds float64 `json:"duration_seconds"`
|
||
}
|
||
type PBSSnapshot struct{} // Phase B stub
|
||
```
|
||
|
||
## Verification
|
||
|
||
- `go test ./...` (local, Windows) + `go test -race ./...` on the build server (192.168.0.180, cgo) — the cadence adds a goroutine; the Store is mutex-guarded.
|
||
- New unit tests: vzdump async UPID→WaitTask + volid-resolve (fake API); restore-to-new benign `ClassCreate` passes the gate; restore-test end-to-end against fakes incl. **(a) teardown-on-failed-verify** and **(b) journal-recovery cleanup** (extend the slice-4 recover test: in-flight Scratch → `Recover` destroys the leaked guest, idempotent when already gone); cadence fires on interval / no-ops when disabled; cross-repo golden + hub-ingest key-set tests.
|
||
- Build the linux binary on 192.168.0.180, relay to `felhom-pve` (see memory `demo-felhom-live-agent`: `MSYS_NO_PATHCONV=1`, config at `/root/.config/felhom-agent/agent.json`, mode=direct, served-cert pin `BA:7C:99…`).
|
||
- **Live on the demo** (the checkpoint validation): `--selftest=backup -vmid <small stopped guest>` to a local target (e.g. `felhom-usb` or `local`, content=backup) → print the `Backup` record; then `--selftest=restore-test -archive <volid>` → restore into a 990000-band scratch guest, net-link-down, boot, verify `running`, teardown → print the `RestoreTest` record. Confirm no leaked scratch guest remains (`pct list`), and that a simulated mid-test crash + restart triggers `Recover` teardown.
|
||
|
||
## Push & checkpoint
|
||
|
||
Push **v0.6.0-rc1** to felhom-agent (and the hub changes to felhom.eu, deploy per the GitOps
|
||
runbook if needed), update CHANGELOG/REPORT, then **stop and await validation** (restore-test
|
||
teardown + recovery, the benign classifications, reporting) + the live demo restore-test.
|
||
**Phase B (next): PBS** — datastore on the USB, zero-knowledge key custody, restore-from-PBS,
|
||
PBS integrity-verify as the lighter frequent check.
|