# Slice 6 Phase A — backup + restore-test orchestration (felhom-agent)

## Context

The agent can observe/report storage (slice 5) and reconcile benign guest ops behind a
signed-op gate (slice 4), but it has no guest-level **backup/restore** layer and — the point
of this slice — no **self-restore-test**, which closes the "a backup you haven't restored
isn't a backup" theme (doc 03 §8). This adds: a vzdump backup primitive that resolves the
produced archive, a benign restore-to-a-**new** guest, and a journaled **self-restore-test**
(restore → boot → verify → teardown) that inherits the slice-4 journal/serialization/crash
recovery so a mid-test crash can't leak a scratch guest. Provisioning/identity-reset/golden
base (§9) are **slice 7**; PBS/offsite/zero-knowledge (§8 offsite tier) are **Phase B**.

Everything here is **benign** (backup, restore-to-new, scratch teardown): reuse the slice-4
classifier/gate/journal — **no new destructive class, no new crypto**. Version → **v0.6.0-rc1**,
stop at the checkpoint.

**Locked decisions (from the task + clarifications):**
- Backups are **crash-consistent only** (marked so in the report); app-consistency needs the
  controller quiesce (slice 8).
- Restore is **to a NEW guest only** (no overwrite anywhere this slice).
- Restore-test verify = **net link-down (benign SetConfig) → boot → reaches `running`** (no
  in-guest probe — slice 8). Link-down avoids the cloned source MAC/IP conflicting on a live
  host; it is test-safety, **not** slice-7 identity reset.
- Restore-test cadence **defaults ON at 24h**, configurable; runs only when a valid scratch
  VMID band is set; also on-demand via the selftest harness.
- Scratch VMID band **990000–990009** (lowest-free, excludes 9999/real guests); refuse to run
  if unset/invalid.
- **Local target only** this phase; `PBSSnapshot` stays a `struct{}` stub (Phase B).
- Bulk volumes: **report the gap only** (which `backup=0` mountpoints the guest vzdump omits).

## The crash-safe core (load-bearing)

The restore-test is journaled as a **single** entry whose `VMID` is the scratch id and
`Kind = "scratch_restore_test"` with a new `Scratch bool` flag. The entry is **terminal only
after teardown** — NOT when the restore sub-task's UPID completes. `Recover` must special-case
Scratch entries **before** the generic UPID-recheck path (else it would mark the entry
`succeeded` because the restore task ran OK, and drop it while the guest still exists → leak).

`RunRestoreTest` (pseudocode) — `defer` teardown so it runs on every path incl. failed verify:
```
vmid := pickScratchVMID(ListLXC, [min,max], exclude 9999)        // refuse if band unset/invalid
opID := "scratch-restore-"+vmid+"-"+seq
append({OpID:opID, VMID:vmid, Kind:"scratch_restore_test", Scratch:true, State:OpStarted})  // BEFORE any mutation
defer teardownScratch(opID, vmid)   // gate.Authorize(IntentForScratchDestroy)=benign -> DestroyLXC -> WaitTask -> terminal
upid := RestoreLXC{VMID:vmid, Archive, Storage:RestoreStorage}; waitOK(upid)   // UPID = error-detection, NOT terminal
for net := range GuestConfig(vmid).Nets(): SetConfig(vmid, {net: existing+",link_down=1"})  // benign
Start(vmid); res.Reached = waitRunning(GuestStatus, vmid, BootTimeout)
res = {Archive, ScratchVMID:vmid, Pass:res.Reached, Verified:"boot+running", Duration, Err}
return res    // defer tears down regardless
```

`Recover` scratch branch (added before the existing UPID logic):
```
if entry.Scratch {
    guests, err := ListLXC; if err { Unresolved++; continue }              // can't decide -> leave in-flight
    if !contains(guests, entry.VMID) { append(terminal(succeeded)); ScratchClean++; continue }  // already gone
    dec := gate.Authorize(IntentForScratchDestroy(hostID, entry.VMID), nil)
    if !dec.Allowed { Unresolved++; continue }                             // fail-safe (should be benign)
    upid, err := DestroyLXC(entry.VMID); if err { Unresolved++; continue } // retry next Recover
    waitOK(upid); append(terminal(succeeded)); ScratchDestroyed++
}
```
Idempotent: crash mid-destroy → next `Recover` finds the guest gone → `ScratchClean`. Both
teardown paths (normal + recovery) go through `gate.Authorize` (benign `ClassGuestDestroy` +
`Provenance{AgentTaggedScratch:true}`) for the audit trail.

## Files to create

- **`internal/reconcile/restoretest.go`** (+ `restoretest_test.go`): `RunRestoreTest(ctx, RestoreTestSpec{Archive, RestoreStorage, BootTimeout}) RestoreTestResult` (engine method — it needs the journal/gate/queue internals), `RestoreTestResult` (reconcile-local data — must NOT return `hub.RestoreTest`, to avoid a reconcile→hub edge), `IntentForScratchDestroy(hostID, vmid)`, `pickScratchVMID`. Runs the sequence on the scratch VMID's `Queue` lane.
- **`internal/backup/{doc,runner,store,schedule}.go`** (+ tests):
  - `runner.go`: `BackupRunner.Backup(ctx, vmid) (hub.Backup, error)` = `Vzdump`+`WaitTask`+volid-resolve + bulk-gap from `GuestConfig.MountPoints()` `backup=0`; `LatestBackup(ctx, target) (volid, error)`; the `RestoreTestRunner` seam (satisfied by `*reconcile.Engine`); `RestoreTestResult → hub.RestoreTest` mapping.
  - `store.go`: mutex-guarded in-memory latest-`Backup`-per-target + latest-`RestoreTest`, implementing the hub `BackupReporter`/`RestoreTestReporter` seams.
  - `schedule.go`: cadence goroutine (default 24h; 0=disabled) → `LatestBackup` → `engine.RunRestoreTest` → write `Store`. No-ops cleanly when no backup exists yet.
  - Imports `reconcile`+`hub`+`proxmox` (acyclic; hub imports neither).
- **`configs/`**: example agent config with the `backup` block.

## Files to modify

- **`internal/proxmox/mutate.go`**: `DestroyLXC(ctx, vmid)` → `DELETE /nodes/{node}/lxc/{vmid}` with `purge=1&destroy-unreferenced-disks=1` via `dataString` (async→UPID); add `Notes string` to `VzdumpOptions` → `notes-template` param (probe `notes-template` vs `notes` on the demo PVE 9.2.2 first, read-only).
- **`internal/proxmox/query.go`**: `LatestBackupVolID(ctx, store, vmid) (string, error)` — `StorageContent` filtered `Content=="backup" && VMID==vmid`, max `CTime`.
- **`internal/reconcile/state.go`**: add `RestoreLXC`, `DestroyLXC`, `GuestStatus` to the `GuestAPI` interface (`*proxmox.Client` already satisfies all three).
- **`internal/reconcile/journal.go`**: add `Scratch bool \`json:"scratch,omitempty"\``; carry it (and `Kind`) through `terminal()`.
- **`internal/reconcile/recover.go`**: the scratch branch above; add `ScratchClean`/`ScratchDestroyed` to `RecoverResult`.
- **`internal/reconcile/recover_test.go`** + **`engine_test.go`**: `fakeAPI` gains `RestoreLXC`/`DestroyLXC`/`GuestStatus` recorders; new `TestRecover_LeakedScratchDestroyed` (in-flight Scratch + ListLXC returns the VMID → DestroyLXC called, gate benign, no longer in-flight), `…AlreadyGone` (ScratchClean), `…ListUnreadable` (Unresolved).
- **`internal/hub/report.go`**: fill `Backup` + `RestoreTest` structs (see wire shapes); `PBSSnapshot` stays `struct{}`.
- **`internal/hub/collect.go`**: add `BackupReporter { Backups(ctx) []Backup }` + `RestoreTestReporter { RestoreTests(ctx) []RestoreTest }` consumer seams (mirror `StorageObserver`); `collectBackups`/`collectRestoreTests` degrade nil/err → non-nil empty.
- **`internal/hub/contract_test.go`** + **`internal/hub/testdata/host-report.golden.json`**: populated `backups[0]`/`restore_tests[0]`; assert their key sets (bidirectional, slice-5 pattern).
- **`internal/config/config.go`**: `BackupConfig{RestoreTestCadenceSeconds, ScratchVMIDMin, ScratchVMIDMax, LocalBackupTarget, RestoreStorage}` + `RestoreTestCadence()` accessor (0→24h default) + env overlay; validate band (min>0, max>=min, 9999 excluded) only when cadence>0. `Config.Validate` stays proxmox-only.
- **`cmd/felhom-agent/main.go`**: wire `backup.Store` into `NewCollector`; add the cadence goroutine alongside `engine.Run`/`loop.Run`/`watchdog.Run`; add `backup` + `restore-test` to `selftestFlag.Set` + the switch + a `-archive` flag; `runSelftestBackup`/`runSelftestRestoreTest` reuse the `runSelftestStorage` wiring (`NewGate(nil, hostID, SlogAudit{}, logger)` + journal + engine).
- **felhom.eu/hub**: `hub/internal/api/handler.go` — add `hostBackup`/`hostRestoreTest` mirror structs to `hostReportPayload`, parse, persist via existing `report_json` (no new DB columns — slice-5 precedent), and **log a FAILED restore-test prominently** (`[WARN]`, the loudest DR signal). `hub/internal/api/testdata/host-report.golden.json` byte-identical with the agent golden; `host_test.go` adds `TestHostBackup_GoldenContract`/`TestHostRestoreTest_GoldenContract` (bidirectional key-set, slice-5 pattern).
- **`CHANGELOG.md`** (prepend v0.6.0-rc1), **`REPORT.md`** (overwrite), `CLAUDE.md` current-state line.

## Proposed wire shapes (draft — must land byte-identical in both repos)

```go
type Backup struct {
    TargetID         string   `json:"target_id"`          // backup storage name
    VMID             int      `json:"vmid"`
    Archive          string   `json:"archive"`            // produced volid
    Mode             string   `json:"mode"`               // snapshot|stop
    CrashConsistent  bool     `json:"crash_consistent"`   // always true this slice
    SizeBytes        int64    `json:"size_bytes"`
    Success          bool     `json:"success"`
    Error            string   `json:"error,omitempty"`
    StartedAt        string   `json:"started_at"`         // RFC3339
    DurationSeconds  float64  `json:"duration_seconds"`
    UncoveredVolumes []string `json:"uncovered_volumes"`  // backup=0 mountpoints (bulk gap)
}
type RestoreTest struct {
    SourceArchive   string  `json:"source_archive"`
    SourceTier      string  `json:"source_tier"`     // "local" (pbs = Phase B)
    ScratchVMID     int     `json:"scratch_vmid"`
    Pass            bool    `json:"pass"`
    Verified        string  `json:"verified"`        // "boot+running"
    Error           string  `json:"error,omitempty"`
    TestedAt        string  `json:"tested_at"`       // RFC3339
    DurationSeconds float64 `json:"duration_seconds"`
}
type PBSSnapshot struct{} // Phase B stub
```

## Verification

- `go test ./...` (local, Windows) + `go test -race ./...` on the build server (192.168.0.180, cgo) — the cadence adds a goroutine; the Store is mutex-guarded.
- New unit tests: vzdump async UPID→WaitTask + volid-resolve (fake API); restore-to-new benign `ClassCreate` passes the gate; restore-test end-to-end against fakes incl. **(a) teardown-on-failed-verify** and **(b) journal-recovery cleanup** (extend the slice-4 recover test: in-flight Scratch → `Recover` destroys the leaked guest, idempotent when already gone); cadence fires on interval / no-ops when disabled; cross-repo golden + hub-ingest key-set tests.
- Build the linux binary on 192.168.0.180, relay to `felhom-pve` (see memory `demo-felhom-live-agent`: `MSYS_NO_PATHCONV=1`, config at `/root/.config/felhom-agent/agent.json`, mode=direct, served-cert pin `BA:7C:99…`).
- **Live on the demo** (the checkpoint validation): `--selftest=backup -vmid <small stopped guest>` to a local target (e.g. `felhom-usb` or `local`, content=backup) → print the `Backup` record; then `--selftest=restore-test -archive <volid>` → restore into a 990000-band scratch guest, net-link-down, boot, verify `running`, teardown → print the `RestoreTest` record. Confirm no leaked scratch guest remains (`pct list`), and that a simulated mid-test crash + restart triggers `Recover` teardown.

## Push & checkpoint

Push **v0.6.0-rc1** to felhom-agent (and the hub changes to felhom.eu, deploy per the GitOps
runbook if needed), update CHANGELOG/REPORT, then **stop and await validation** (restore-test
teardown + recovery, the benign classifications, reporting) + the live demo restore-test.
**Phase B (next): PBS** — datastore on the USB, zero-knowledge key custody, restore-from-PBS,
PBS integrity-verify as the lighter frequent check.
