diff --git a/CHANGELOG.md b/CHANGELOG.md index 43cb758..1475633 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,6 +3,57 @@ All notable changes to **felhom-agent** are recorded here. Update on every code change that gets pushed. +## v0.6.0-rc1 — slice 6 Phase A: backup + the self-restore-test (local target) (2026-06-09) + +Phase A of the backup/restore slice (doc 03 §8) — the agent's guest-level backup layer and +the **self-restore-test**, which closes "a backup you haven't restored isn't a backup". +Everything here is BENIGN (backup, restore-to-NEW, scratch teardown): reuses the slice-4 +classifier/gate/journal — no new destructive class, no new crypto. Local target only; PBS is +Phase B. Restore is to a NEW guest only (no overwrite). Backups are crash-consistent only +(app-consistency needs the controller quiesce, slice 8) — marked so in the report. + +### Added +- **proxmox** (`mutate.go`/`query.go`): `DestroyLXC` (DELETE …/lxc/{vmid}?purge=1&destroy- + unreferenced-disks=1 → UPID; the scratch-teardown primitive); `VzdumpOptions.Notes` → + `notes-template` (verified on PVE 9.2.2); `LatestBackupVolID` (resolve a produced archive + from the backup-storage listing — the task status carries no result volid). +- **reconcile self-restore-test** (`restoretest.go`) — `Engine.RunRestoreTest`: pick a free + scratch VMID (configured band, excludes 9999; full band → skip, never out-of-band) → + **journal a Scratch-owned entry BEFORE any mutation** → restore-to-new → benign net + **link-down** SetConfig (so the clone can't conflict with a running source's MAC/IP; this + is test-safety, NOT slice-7 identity reset) → boot → verify **reaches `running`** → ALWAYS + teardown (defer; benign `ClassGuestDestroy` + agent-tagged-scratch provenance, gated). Runs + on the scratch VMID's queue lane. Reuses the journal/gate; result feeds the report. +- **Crash-safe recovery** (`recover.go`): a Scratch journal entry is resolved by TEARDOWN, + not by re-checking the restore sub-task's UPID — special-cased BEFORE the generic path + (else the restore task's OK would mark it succeeded while the guest leaks). `Recover` now + destroys a leaked scratch guest (idempotent: already-gone → clean; list-unreadable → left + in-flight for a later pass). `JournalEntry.Scratch` flag; `RecoverResult.ScratchClean/ + ScratchDestroyed`. GuestAPI gains `RestoreLXC`/`DestroyLXC`/`GuestStatus`. +- **`internal/backup` package**: `BackupRunner.Backup` (vzdump + archive/size resolve + + bulk-volume gap — a mountpoint is UNCOVERED unless it carries an explicit `backup=1`, so + an unset `backup=` is reported uncovered too, the safe DR direction); `PickRestoreCandidate` + (newest backup); an in-memory `Store` (latest-backup-per-target + latest-restore-test) + implementing the hub `BackupReporter`/`RestoreTestReporter` seams; a cadence `Scheduler` + (default 24h; the fourth daemon goroutine; disabled cleanly when off/misconfigured). +- **hub report** (`report.go`): filled the `Backup` + `RestoreTest` stubs (`PBSSnapshot` + stays a Phase-B stub); collector `BackupReporter`/`RestoreTestReporter` seams. Cross-repo + golden updated in BOTH repos (byte-identical) + bidirectional key-set tests for + `backups[0]`/`restore_tests[0]`. Hub `handler.go` parses + persists them (report_json; no + new columns) and logs a **FAILED restore-test prominently** (the loudest DR signal). +- **config** `BackupConfig` (local target, restore storage, restore-test cadence, scratch + VMID band 990000–990009 default) + accessors + env overlay + cadence-gated validation. +- **`--selftest=backup -vmid N`** (one-shot backup → print the Backup record) and + **`--selftest=restore-test [-archive volid]`** (Recover-then restore→boot→verify→teardown, + print the RestoreTest record). Standalone on the Proxmox host. + +### Notes +- The daemon runs cleanly with the cadence off or misconfigured (logs + disables, never + crashes); a leaked scratch guest from a mid-test crash is reaped by `engine.Recover` on + restart. `go test -race` covers the new scheduler goroutine. +- Slice-3/4/5 exported surfaces, goldens, and adversarial tests intact. Version bumps to + **v0.6.0** when Phase B (PBS) lands. + ## v0.5.1 — slice 5 live-validation prep: durable_id mis-id fix + re-mount UUID memory (2026-06-09) Two correctness fixes surfaced while preparing the live USB validation on `demo-felhom` diff --git a/CLAUDE.md b/CLAUDE.md index 5d9db5b..794249f 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -15,7 +15,7 @@ - Module `gitea.dooplex.hu/admin/felhom-agent`; binary `felhom-agent` (`cmd/felhom-agent/`). - **Pure Go stdlib + `golang.org/x/crypto` only** — no web frameworks. - `go.mod` directive **go 1.25.0**; dep `golang.org/x/crypto v0.52.0` (declares go 1.25, will NOT build on Go 1.24). The **build server (192.168.0.180) runs go1.26.0** (upstream Go on PATH, backward-compatible). Build/run the agent there for live tests (same LAN as the demo host). -- Version: `version` var in `cmd/felhom-agent/main.go`, overridable via `-ldflags "-X main.version="`; `--version` flag. **Current: v0.5.0** (slice 5 complete: storage observe/report + watchdog + the host-root surface — mounts/SMART/grow/destructive gate). Bump on meaningful changes + add a CHANGELOG entry. +- Version: `version` var in `cmd/felhom-agent/main.go`, overridable via `-ldflags "-X main.version="`; `--version` flag. **Current: v0.6.0-rc1** (slice 6 Phase A: backup + the self-restore-test, local target; PBS = Phase B). Bump on meaningful changes + add a CHANGELOG entry. ## Layout @@ -26,8 +26,9 @@ internal/log/ slog setup internal/proxmox/ API-first Client + fenced root-CLI Privileged + UPID WaitTask internal/authz/ operator signed-op verifier (SSHSIG); durable FileNonceStore internal/hub/ daemon: HostReport collector + Bearer client + resilient Loop -internal/reconcile/ reconcile engine + reversibility gate + op journal + crash recovery +internal/reconcile/ reconcile engine + reversibility gate + op journal + crash recovery + restore-test internal/storage/ storage-target observer + durable_id + fast-poll watchdog (slice 5) +internal/backup/ vzdump backup runner + restore-test scheduler + report store (slice 6) ``` ## Proxmox model (the load-bearing rules) @@ -54,7 +55,8 @@ Built in slices, all on `main`: - **v0.4.0** — slice-4 **Phase B** (security core): the benign/destructive **classifier** (provenance + data-bearing, not by verb; scratch/same-txn provenance is agent-internal, never hub-sourced), the **reversibility gate** (destructive → `pending_signature` unless a verified, role-scoped, action-bound operator signature), the **signed-op consuming layer** over `internal/authz` (role-scoping per doc 04 §4, op-to-action binding, idempotency-by-nonce, audit), and the **crash-recovery consumer** (`Recover` over `InFlight()`, resume-or-rollback). The gate fronts the queue's executor (every mutation passes it). **Inert this slice** — no destructive deltas served until slice 10; the destructive path is classified, gated, and adversarially tested but not wired to live execution. `authz` surface untouched. - **v0.5.0-rc1** — slice-5 **Phase A** (read-only, live): `internal/storage` — the `StorageTarget` wire contract (filled the slice-3 stub), `durable_id` derivation per type, the `Observer`, and the **storage watchdog** (third daemon goroutine; fast-poll → debounced out-of-band report on a known target's attach/disconnect). Hub ingest accepts/persists `storage_targets`; cross-repo golden byte-identical. - **v0.5.0** — slice-5 **Phase B** (the host-root surface): the `HostOps` seam + `SudoHostOps` (systemd `.mount` units by fs-UUID, detach, SMART, lvs) behind a **strict argument validator** (the adversarial matrix is the headline security test — hostile UUID/path/device refused with zero exec); SMART (SATA+NVMe) + thin-pool metadata enrichment; the watchdog's benign **re-mount response** (off the poll path); the **disk-grow executor** (`pct resize`, grow-only, benign) and **destructive storage ops** through the slice-4 gate (target-scoped; built + tested, inert live); `--selftest=storage [-watch]`; `configs/felhom-agent.sudoers`. -- **Next: slice 6 (backup/restore orchestration)** — vzdump/PBS + the self-restore-test; the restore-overwrite executor the gate already guards. +- **v0.6.0-rc1** — slice-6 **Phase A** (backup + self-restore-test, local target): proxmox `DestroyLXC`/`Vzdump`-notes/`LatestBackupVolID`; `Engine.RunRestoreTest` (journaled scratch lifecycle: restore-to-new → net link-down → boot → verify running → defer teardown, all benign); `Recover` extended to reap a leaked scratch guest (Scratch journal flag, special-cased before the UPID path); `internal/backup` (runner + bulk-gap + cadence scheduler + report store); hub `Backup`/`RestoreTest` filled (cross-repo golden + hub logs a failed restore-test); `--selftest=backup`/`--selftest=restore-test`. Live-validated on demo-felhom. +- **Next: slice 6 Phase B (PBS)** — datastore on the USB, zero-knowledge key custody, restore-from-PBS, PBS integrity-verify (the lighter frequent check). Then slice 7 (provisioning + identity-reset + golden base, §9). ## Demo host (for live tests) diff --git a/cmd/felhom-agent/main.go b/cmd/felhom-agent/main.go index 509da76..c47cbb8 100644 --- a/cmd/felhom-agent/main.go +++ b/cmd/felhom-agent/main.go @@ -21,6 +21,7 @@ import ( "time" "gitea.dooplex.hu/admin/felhom-agent/internal/authz" + "gitea.dooplex.hu/admin/felhom-agent/internal/backup" "gitea.dooplex.hu/admin/felhom-agent/internal/config" "gitea.dooplex.hu/admin/felhom-agent/internal/hub" applog "gitea.dooplex.hu/admin/felhom-agent/internal/log" @@ -31,7 +32,7 @@ import ( // version is the agent version. Overridable at build time with // -ldflags "-X main.version="; defaults to the in-repo CHANGELOG version. -var version = "0.5.1" +var version = "0.6.0-rc1" func main() { var ( @@ -39,12 +40,14 @@ func main() { selftest selftestFlag vmid int watch time.Duration + archive string showVersion bool ) flag.StringVar(&cfgPath, "config", envOr("FELHOM_AGENT_CONFIG", "/etc/felhom-agent/agent.json"), "path to the agent config file (JSON)") - flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report to the hub; `storage` = observe storage targets (+ -watch for the live watchdog)") - flag.IntVar(&vmid, "vmid", 0, "guest VMID for --selftest=task (the reversible snapshot/rollback exercise)") + flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup)") + flag.IntVar(&vmid, "vmid", 0, "guest VMID for --selftest=task|backup") flag.DurationVar(&watch, "watch", 0, "for --selftest=storage: run the watchdog verbose for this duration (e.g. 3m) with the re-mount response live; 0 = observe pass only") + flag.StringVar(&archive, "archive", "", "for --selftest=restore-test: the backup volid to restore (default: newest on the configured local target)") flag.BoolVar(&showVersion, "version", false, "print version and exit") flag.Parse() @@ -76,6 +79,10 @@ func main() { os.Exit(runSelftestHub(context.Background(), cfg, logger)) case "storage": os.Exit(runSelftestStorage(context.Background(), cfg, logger, watch)) + case "backup": + os.Exit(runSelftestBackup(context.Background(), cfg, logger, vmid)) + case "restore-test": + os.Exit(runSelftestRestoreTest(context.Background(), cfg, logger, archive)) } } @@ -178,7 +185,11 @@ func runDaemon(cfg config.Config, logger *slog.Logger) int { hostReader := storage.NewProcHostReader() hostOps := newHostOps(cfg, logger) observer := storage.NewObserver(px, hostReader, hostOps, logger) - collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, cfg.Hub.HostID, version, logger) + // Backup + restore-test state store (slice 6): holds the latest backup-per-target + + // latest restore-test result; the collector reads it via the BackupReporter / + // RestoreTestReporter seams; the cadence scheduler writes it. + backupStore := backup.NewStore() + collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, backupStore, backupStore, cfg.Hub.HostID, version, logger) loop := hub.NewLoop(collector, client, time.Duration(hcfg.PollSeconds)*time.Second, logger) interval := time.Duration(hcfg.PollSeconds) * time.Second @@ -256,20 +267,28 @@ func runDaemon(cfg config.Config, logger *slog.Logger) int { }) // Crash recovery (doc 03 §10): resolve any op that was in flight when the agent - // last died BEFORE issuing new mutations. With an empty journal this is a no-op. + // last died BEFORE issuing new mutations. For slice 6 this is load-bearing — a + // restore-test scratch guest leaked by a mid-test crash is torn down here. engine.Recover(ctx) - // Run reconcile, the hub loop, and the storage watchdog concurrently; any one - // returning ends the daemon (then ctx cancellation tears the others down). - errc := make(chan error, 3) + // Self-restore-test scheduler (slice 6): the fourth daemon goroutine. Runs the restore- + // test on the configured cadence (default 24h). Disabled cleanly when the cadence is off + // OR the scratch band / restore storage is misconfigured — the daemon still runs. + scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, logger) + + // Run reconcile, the hub loop, the storage watchdog, and the restore-test scheduler + // concurrently; any one returning ends the daemon (ctx cancellation tears down the rest). + errc := make(chan error, 4) go func() { errc <- engine.Run(ctx, interval) }() go func() { errc <- loop.Run(ctx) }() go func() { errc <- watchdog.Run(ctx) }() + go func() { errc <- scheduler.Run(ctx) }() err = <-errc stop() // tear down the siblings on the first exit <-errc // wait for the second <-errc // wait for the third + <-errc // wait for the fourth if err != nil && err != context.Canceled { logger.Error("daemon: exited with error", "err", err) return 1 @@ -277,6 +296,35 @@ func runDaemon(cfg config.Config, logger *slog.Logger) int { return 0 } +// buildRestoreTestScheduler constructs the restore-test cadence scheduler from config. It +// disables the cadence (returns a scheduler that just waits) when the cadence is off or the +// scratch band / restore storage is invalid — a misconfig must not crash the daemon, and the +// machinery still works on-demand via --selftest=restore-test. +func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *reconcile.Engine, store *backup.Store, logger *slog.Logger) *backup.Scheduler { + cadence := cfg.Backup.RestoreTestCadence() + if cadence > 0 { + if err := cfg.Backup.ValidateForRestoreTest(); err != nil { + logger.Warn("daemon: restore-test cadence disabled (config invalid)", "err", err) + cadence = 0 + } + } + min, max := cfg.Backup.ScratchBand() + runner := backup.NewBackupRunner(px, cfg.Backup.LocalBackupTarget, "", "felhom restore-test", logger) + return backup.NewScheduler(backup.SchedulerOptions{ + Runner: engine, + Pick: runner.PickRestoreCandidate, + Store: store, + Spec: reconcile.RestoreTestSpec{ + RestoreStorage: cfg.Backup.RestoreStorage, + ScratchMin: min, + ScratchMax: max, + SourceTier: "local", + }, + Cadence: cadence, + Logger: logger, + }) +} + // reconcileJournalPath chooses the op-journal path: a `journal.log` sibling of the // configured nonce store (both are durable agent state), falling back to the standard // host state dir when the nonce store is unset. @@ -338,7 +386,7 @@ func runSelftestHub(ctx context.Context, cfg config.Config, logger *slog.Logger) return 1 } observer := storage.NewObserver(px, storage.NewProcHostReader(), newHostOps(cfg, logger), logger) - collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, cfg.Hub.HostID, version, logger) + collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, nil, nil, cfg.Hub.HostID, version, logger) ctx, cancel := context.WithTimeout(ctx, 60*time.Second) defer cancel() @@ -460,6 +508,122 @@ func smartCounters(s hub.SmartSummary) string { return " " + strings.Join(parts, " ") } +// runSelftestBackup runs one vzdump of -vmid to the configured local backup target and +// prints the resulting Backup record. Standalone (no hub). Live + benign. +func runSelftestBackup(ctx context.Context, cfg config.Config, logger *slog.Logger, vmid int) int { + if err := cfg.Validate(); err != nil { + fmt.Fprintln(os.Stderr, "selftest: proxmox not configured:", err) + return 1 + } + if vmid == 0 { + fmt.Fprintln(os.Stderr, "selftest=backup requires -vmid N") + return 2 + } + if cfg.Backup.LocalBackupTarget == "" { + fmt.Fprintln(os.Stderr, "selftest=backup requires backup.local_backup_target in config (a content=backup storage)") + return 2 + } + px, err := newProxmoxClient(cfg) + if err != nil { + fmt.Fprintln(os.Stderr, "selftest: proxmox client:", err) + return 1 + } + ctx, cancel := context.WithTimeout(ctx, 30*time.Minute) + defer cancel() + + fmt.Printf("=== felhom-agent %s selftest=backup (vmid %d → %s) ===\n", version, vmid, cfg.Backup.LocalBackupTarget) + runner := backup.NewBackupRunner(px, cfg.Backup.LocalBackupTarget, "", "felhom selftest", logger) + rec, err := runner.Backup(ctx, vmid) + printJSON("backup record", rec) + if err != nil { + fmt.Fprintln(os.Stderr, " [FAIL] backup:", err) + return 1 + } + fmt.Printf("=== selftest=backup OK (crash-consistent=%v, archive=%s) ===\n", rec.CrashConsistent, rec.Archive) + return 0 +} + +// runSelftestRestoreTest runs one self-restore-test (restore → net-link-down → boot → verify +// running → teardown) of -archive (or the newest backup on the local target) into a scratch +// guest. Standalone (no hub). Runs engine.Recover first so a leaked scratch from a prior +// crashed test is reaped before this run. +func runSelftestRestoreTest(ctx context.Context, cfg config.Config, logger *slog.Logger, archive string) int { + if err := cfg.Validate(); err != nil { + fmt.Fprintln(os.Stderr, "selftest: proxmox not configured:", err) + return 1 + } + if err := cfg.Backup.ValidateForRestoreTest(); err != nil { + fmt.Fprintln(os.Stderr, "selftest=restore-test:", err) + return 2 + } + px, err := newProxmoxClient(cfg) + if err != nil { + fmt.Fprintln(os.Stderr, "selftest: proxmox client:", err) + return 1 + } + ctx, stop := signal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM) + defer stop() + + // Build a journaled engine so a leaked scratch from a prior crash is recoverable. + queue := reconcile.NewQueue() + defer queue.Close() + var journal *reconcile.Journal + if jp := reconcileJournalPath(cfg); jp != "" { + if err := os.MkdirAll(filepath.Dir(jp), 0o700); err == nil { + if j, err := reconcile.OpenJournal(jp); err == nil { + journal = j + defer journal.Close() + } + } + } + gate := reconcile.NewGate(nil, cfg.Hub.HostID, reconcile.SlogAudit{Logger: logger}, logger) + engine := reconcile.NewEngine(reconcile.EngineOptions{ + API: px, Queue: queue, Journal: journal, Gate: gate, HostID: cfg.Hub.HostID, Logger: logger, + }) + + fmt.Printf("=== felhom-agent %s selftest=restore-test ===\n", version) + fmt.Println(" --- recover: reaping any leaked scratch from a prior crashed test ---") + rec := engine.Recover(ctx) + fmt.Printf(" recover: examined=%d scratch_destroyed=%d scratch_clean=%d\n", rec.Examined, rec.ScratchDestroyed, rec.ScratchClean) + + if archive == "" { + runner := backup.NewBackupRunner(px, cfg.Backup.LocalBackupTarget, "", "", logger) + archive, err = runner.PickRestoreCandidate(ctx) + if err != nil { + fmt.Fprintln(os.Stderr, " [FAIL] pick backup:", err) + return 1 + } + if archive == "" { + fmt.Fprintln(os.Stderr, " [FAIL] no backup available on", cfg.Backup.LocalBackupTarget, "(run --selftest=backup first)") + return 1 + } + } + min, max := cfg.Backup.ScratchBand() + fmt.Printf(" restoring %s into scratch band [%d,%d] on %s …\n", archive, min, max, cfg.Backup.RestoreStorage) + res := engine.RunRestoreTest(ctx, reconcile.RestoreTestSpec{ + Archive: archive, RestoreStorage: cfg.Backup.RestoreStorage, + ScratchMin: min, ScratchMax: max, SourceTier: "local", + }) + printJSON("restore-test record", backup.ToHubRestoreTest(res, time.Now().UTC())) + if res.Skipped { + fmt.Println("=== selftest=restore-test SKIPPED (no free scratch VMID in band) ===") + return 0 + } + if res.Err != nil || !res.Pass { + fmt.Fprintf(os.Stderr, " [FAIL] restore-test (scratch %d): %v\n", res.ScratchVMID, res.Err) + return 1 + } + fmt.Printf("=== selftest=restore-test OK (scratch %d restored+booted+verified+torn-down in %s) ===\n", res.ScratchVMID, res.Duration.Round(time.Second)) + return 0 +} + +// printJSON prints a labelled, indented JSON dump (best-effort) to stdout. +func printJSON(label string, v any) { + if b, err := json.MarshalIndent(v, " ", " "); err == nil { + fmt.Printf(" --- %s ---\n %s\n", label, string(b)) + } +} + // runSelftestRead loads config, builds the API client, and runs the read-only // queries against the live host, printing a short health report. It mutates // nothing. Missing/invalid config is reported cleanly (no panic). @@ -748,8 +912,12 @@ func (f *selftestFlag) Set(v string) error { f.mode = "hub" case "storage": f.mode = "storage" + case "backup": + f.mode = "backup" + case "restore-test": + f.mode = "restore-test" default: - return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage)", v) + return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test)", v) } return nil } diff --git a/configs/agent.example.json b/configs/agent.example.json index 3e4d145..8d7a753 100644 --- a/configs/agent.example.json +++ b/configs/agent.example.json @@ -36,5 +36,12 @@ "timeout_seconds": 30, "ca_file": "" }, + "backup": { + "local_backup_target": "local", + "restore_storage": "local-lvm", + "restore_test_cadence_seconds": 0, + "scratch_vmid_min": 990000, + "scratch_vmid_max": 990009 + }, "log_level": "info" } diff --git a/internal/backup/backup_test.go b/internal/backup/backup_test.go new file mode 100644 index 0000000..9fe14e5 --- /dev/null +++ b/internal/backup/backup_test.go @@ -0,0 +1,199 @@ +package backup + +import ( + "context" + "encoding/json" + "errors" + "io" + "log/slog" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" + "gitea.dooplex.hu/admin/felhom-agent/internal/reconcile" +) + +func quiet() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) } + +// fakeBackupAPI is a synthetic BackupAPI. +type fakeBackupAPI struct { + vzdumpUPID string + vzdumpErr error + waitErr error + cfg proxmox.GuestConfig + cfgErr error + content []proxmox.StorageContent + contentErr error + vzdumps []proxmox.VzdumpOptions +} + +func (f *fakeBackupAPI) Vzdump(_ context.Context, o proxmox.VzdumpOptions) (string, error) { + f.vzdumps = append(f.vzdumps, o) + return f.vzdumpUPID, f.vzdumpErr +} +func (f *fakeBackupAPI) WaitTask(_ context.Context, _ string, _ proxmox.WaitOptions) (proxmox.TaskStatus, error) { + return proxmox.TaskStatus{Status: "stopped", ExitStatus: "OK"}, f.waitErr +} +func (f *fakeBackupAPI) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) { + return f.cfg, f.cfgErr +} +func (f *fakeBackupAPI) StorageContent(_ context.Context, _ string) ([]proxmox.StorageContent, error) { + return f.content, f.contentErr +} + +// guestCfgWithMounts builds a GuestConfig whose Extra carries the given mpN strings. +func guestCfgWithMounts(mps map[string]string) proxmox.GuestConfig { + extra := map[string]json.RawMessage{} + for k, v := range mps { + b, _ := json.Marshal(v) + extra[k] = b + } + return proxmox.GuestConfig{Extra: extra} +} + +func TestBackup_SuccessResolvesArchiveAndBulkGap(t *testing.T) { + api := &fakeBackupAPI{ + vzdumpUPID: "UPID:vzdump:1", + cfg: guestCfgWithMounts(map[string]string{ + "mp0": "local-lvm:8,mp=/mnt/bulk,backup=0", // explicit opt-out → uncovered + "mp1": "local-lvm:4,mp=/mnt/db,backup=1", // covered + "mp2": "local-lvm:2,mp=/mnt/scratch", // UNSET → uncovered (opt-in default) + }), + content: []proxmox.StorageContent{ + {VolID: "local:backup/old-9001.tar.zst", Content: "backup", VMID: 9001, Size: 100, CTime: 100}, + {VolID: "local:backup/new-9001.tar.zst", Content: "backup", VMID: 9001, Size: 524288000, CTime: 200}, + {VolID: "local:backup/other-9002.tar.zst", Content: "backup", VMID: 9002, Size: 7, CTime: 999}, + }, + } + r := NewBackupRunner(api, "local", "", "felhom test", quiet()) + rec, err := r.Backup(context.Background(), 9001) + if err != nil { + t.Fatalf("Backup: %v", err) + } + if !rec.Success || !rec.CrashConsistent { + t.Errorf("record = %+v, want success + crash_consistent", rec) + } + if rec.Archive != "local:backup/new-9001.tar.zst" || rec.SizeBytes != 524288000 { + t.Errorf("resolved wrong archive/size: %+v", rec) + } + if rec.Mode != string(proxmox.ModeSnapshot) { + t.Errorf("mode = %q, want snapshot (default)", rec.Mode) + } + // Bulk gap: mp0 (backup=0) AND mp2 (unset) are uncovered; mp1 (backup=1) is NOT. + if got := rec.UncoveredVolumes; len(got) != 2 || !has(got, "/mnt/bulk") || !has(got, "/mnt/scratch") { + t.Errorf("uncovered = %v, want [/mnt/bulk /mnt/scratch] (unset is uncovered too)", got) + } + if has(rec.UncoveredVolumes, "/mnt/db") { + t.Error("backup=1 mountpoint must NOT be reported uncovered") + } + // Notes-template threaded through. + if len(api.vzdumps) != 1 || api.vzdumps[0].Notes != "felhom test" { + t.Errorf("vzdump opts = %+v", api.vzdumps) + } +} + +func TestBackup_VzdumpFailureReturnsFailedRecord(t *testing.T) { + api := &fakeBackupAPI{vzdumpErr: errors.New("vzdump boom")} + r := NewBackupRunner(api, "local", "", "", quiet()) + rec, err := r.Backup(context.Background(), 9001) + if err == nil { + t.Fatal("expected error") + } + if rec.Success || rec.Error == "" { + t.Errorf("failed backup must produce a Success=false record with an Error: %+v", rec) + } +} + +func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) { + api := &fakeBackupAPI{content: []proxmox.StorageContent{ + {VolID: "a", Content: "backup", CTime: 10}, + {VolID: "b", Content: "backup", CTime: 99}, + {VolID: "iso", Content: "iso", CTime: 999}, // not a backup → ignored + }} + r := NewBackupRunner(api, "local", "", "", quiet()) + vol, err := r.PickRestoreCandidate(context.Background()) + if err != nil || vol != "b" { + t.Fatalf("pick = %q,%v want newest 'b'", vol, err) + } + // no backups → "". + api.content = []proxmox.StorageContent{{VolID: "iso", Content: "iso"}} + if vol, _ := r.PickRestoreCandidate(context.Background()); vol != "" { + t.Errorf("no backup → empty, got %q", vol) + } +} + +// --- scheduler --- + +type fakeRTRunner struct { + res reconcile.RestoreTestResult + runs int +} + +func (f *fakeRTRunner) RunRestoreTest(_ context.Context, _ reconcile.RestoreTestSpec) reconcile.RestoreTestResult { + f.runs++ + return f.res +} + +func TestScheduler_TickRunsAndRecords(t *testing.T) { + store := NewStore() + rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Archive: "vol", Pass: true, Verified: "boot+running", Duration: time.Second}} + s := NewScheduler(SchedulerOptions{ + Runner: rt, + Pick: func(context.Context) (string, error) { return "vol", nil }, + Store: store, + Cadence: time.Hour, + Logger: quiet(), + }) + s.tick(context.Background()) + if rt.runs != 1 { + t.Fatalf("tick should run the restore-test once, got %d", rt.runs) + } + got := store.RestoreTests(context.Background()) + if len(got) != 1 || !got[0].Pass || got[0].SourceArchive != "vol" { + t.Fatalf("store should have the recorded result: %+v", got) + } +} + +func TestScheduler_TickNoBackupNoOp(t *testing.T) { + rt := &fakeRTRunner{} + s := NewScheduler(SchedulerOptions{ + Runner: rt, Pick: func(context.Context) (string, error) { return "", nil }, + Store: NewStore(), Cadence: time.Hour, Logger: quiet(), + }) + s.tick(context.Background()) + if rt.runs != 0 { + t.Errorf("no backup available → no restore-test run, got %d", rt.runs) + } +} + +func TestScheduler_DisabledRunReturnsOnCancel(t *testing.T) { + s := NewScheduler(SchedulerOptions{Cadence: 0, Logger: quiet()}) + ctx, cancel := context.WithCancel(context.Background()) + done := make(chan error, 1) + go func() { done <- s.Run(ctx) }() + cancel() + select { + case err := <-done: + if err != nil { + t.Fatalf("disabled scheduler Run should return nil on cancel, got %v", err) + } + case <-time.After(time.Second): + t.Fatal("disabled scheduler did not return on cancel") + } +} + +func TestStore_Reporters(t *testing.T) { + s := NewStore() + if len(s.Backups(context.Background())) != 0 || len(s.RestoreTests(context.Background())) != 0 { + t.Fatal("empty store must report empty (non-nil) slices") + } +} + +func has(ss []string, want string) bool { + for _, s := range ss { + if s == want { + return true + } + } + return false +} diff --git a/internal/backup/doc.go b/internal/backup/doc.go new file mode 100644 index 0000000..0fbe2ba --- /dev/null +++ b/internal/backup/doc.go @@ -0,0 +1,23 @@ +// Package backup is the guest-level backup + self-restore-test layer (doc 03 §8, slice 6 +// Phase A). It orchestrates a crash-consistent vzdump to a LOCAL target, resolves the +// produced archive, and drives the self-restore-test (restore → boot → verify → teardown) +// through the reconcile engine so it inherits the journal / per-guest serialization / +// crash-safe recovery. +// +// Everything here is BENIGN (backup, restore-to-NEW, scratch teardown): it reuses the +// slice-4 classifier/gate/journal via reconcile — no new destructive class, no new crypto. +// Restore is to a NEW guest only (no overwrite this slice). PBS / offsite / zero-knowledge +// is Phase B. +// +// Layout: +// - runner.go — BackupRunner: vzdump + archive-volid/size resolve + the bulk-volume gap; +// restore-candidate picker. +// - store.go — in-memory latest-backup-per-target + latest-restore-test, implementing +// the hub BackupReporter / RestoreTestReporter seams (point-in-time state +// the collector reads; re-populated each cadence/selftest run). +// - schedule.go — the restore-test cadence goroutine (default 24h; disabled when 0). +// +// hub does NOT import this package (the report types live in hub; this package imports hub +// for them, mirroring the slice-5 storage seam). This package may import reconcile + hub + +// proxmox (acyclic). +package backup diff --git a/internal/backup/runner.go b/internal/backup/runner.go new file mode 100644 index 0000000..d4feb06 --- /dev/null +++ b/internal/backup/runner.go @@ -0,0 +1,198 @@ +package backup + +import ( + "context" + "fmt" + "log/slog" + "sort" + "strings" + "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/hub" + "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" + "gitea.dooplex.hu/admin/felhom-agent/internal/reconcile" +) + +// BackupAPI is the read+backup proxmox surface the runner needs. *proxmox.Client satisfies it. +type BackupAPI interface { + Vzdump(ctx context.Context, opts proxmox.VzdumpOptions) (string, error) + WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error) + GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig, error) + StorageContent(ctx context.Context, store string) ([]proxmox.StorageContent, error) +} + +// BackupRunner orchestrates a crash-consistent vzdump to a local target and reports the +// result (incl. the bulk-volume gap). An agent-initiated vzdump is crash-consistent only +// (no fsfreeze); the report says so. +type BackupRunner struct { + api BackupAPI + target string // backup storage (content=backup) + mode proxmox.BackupMode // default ModeSnapshot + notes string // optional notes-template + logger *slog.Logger + now func() time.Time +} + +// NewBackupRunner builds a runner. mode defaults to snapshot (works for a stopped guest and +// for lvm-thin); the caller may pass ModeStop for storages without snapshot support. +func NewBackupRunner(api BackupAPI, target string, mode proxmox.BackupMode, notes string, logger *slog.Logger) *BackupRunner { + if mode == "" { + mode = proxmox.ModeSnapshot + } + if logger == nil { + logger = slog.Default() + } + return &BackupRunner{api: api, target: target, mode: mode, notes: notes, logger: logger, now: func() time.Time { return time.Now().UTC() }} +} + +// Backup runs one vzdump of vmid to the local target and returns the report record. A +// failure is returned BOTH as an error and as a Backup{Success:false,...} so the caller can +// record the failed attempt. +func (r *BackupRunner) Backup(ctx context.Context, vmid int) (hub.Backup, error) { + start := r.now() + rec := hub.Backup{ + TargetID: r.target, + VMID: vmid, + Mode: string(r.mode), + CrashConsistent: true, // always, this slice (no controller quiesce) + StartedAt: start.Format(time.RFC3339), + } + if r.target == "" { + rec.Error = "no backup target configured" + return rec, fmt.Errorf("backup: %s", rec.Error) + } + + // Bulk-volume gap: which mountpoints the vzdump will EXCLUDE (best-effort; a config-read + // failure just leaves the gap unknown, never fails the backup). + if cfg, err := r.api.GuestConfig(ctx, vmid); err == nil { + rec.UncoveredVolumes = uncoveredMountpoints(cfg.MountPoints()) + } else { + r.logger.Warn("backup: could not read guest config for bulk-gap", "vmid", vmid, "err", err) + rec.UncoveredVolumes = []string{} + } + + upid, err := r.api.Vzdump(ctx, proxmox.VzdumpOptions{ + VMID: vmid, Storage: r.target, Mode: r.mode, Notes: r.notes, + }) + if err != nil { + rec.Error = err.Error() + rec.DurationSeconds = time.Since(start).Seconds() + return rec, fmt.Errorf("backup: vzdump vmid %d: %w", vmid, err) + } + if upid != "" { + if _, err := r.api.WaitTask(ctx, upid, proxmox.WaitOptions{Timeout: 30 * time.Minute}); err != nil { + rec.Error = err.Error() + rec.DurationSeconds = time.Since(start).Seconds() + return rec, fmt.Errorf("backup: vzdump task vmid %d: %w", vmid, err) + } + } + + // Resolve the produced archive (volid + size) — the task status carries no result volid. + vol, size, err := r.latestArchive(ctx, vmid) + if err != nil { + rec.Error = fmt.Sprintf("backup succeeded but archive not resolved: %v", err) + rec.DurationSeconds = time.Since(start).Seconds() + return rec, fmt.Errorf("backup: resolve archive vmid %d: %w", vmid, err) + } + rec.Archive = vol + rec.SizeBytes = size + rec.Success = true + rec.DurationSeconds = time.Since(start).Seconds() + r.logger.Info("backup: completed", "vmid", vmid, "target", r.target, "archive", vol, "size_bytes", size, + "uncovered_volumes", len(rec.UncoveredVolumes)) + return rec, nil +} + +// PickRestoreCandidate returns the newest backup archive on the target (any guest), or "" +// when there is none — the restore-test then no-ops cleanly. +func (r *BackupRunner) PickRestoreCandidate(ctx context.Context) (string, error) { + contents, err := r.api.StorageContent(ctx, r.target) + if err != nil { + return "", err + } + var best string + var bestCTime int64 = -1 + for _, e := range contents { + if e.Content == "backup" && e.CTime > bestCTime { + bestCTime, best = e.CTime, e.VolID + } + } + return best, nil +} + +// latestArchive finds the newest backup archive volid + size for vmid on the target. +func (r *BackupRunner) latestArchive(ctx context.Context, vmid int) (string, int64, error) { + contents, err := r.api.StorageContent(ctx, r.target) + if err != nil { + return "", 0, err + } + var vol string + var size, bestCTime int64 = 0, -1 + for _, e := range contents { + if e.Content == "backup" && e.VMID == vmid && e.CTime > bestCTime { + bestCTime, vol, size = e.CTime, e.VolID, e.Size + } + } + if vol == "" { + return "", 0, fmt.Errorf("no backup archive found for vmid %d on %s", vmid, r.target) + } + return vol, size, nil +} + +// uncoveredMountpoints returns the mountpoint paths the guest vzdump EXCLUDES. LXC mount +// points are OPT-IN to vzdump: a mpN with `backup=1` is covered; ANY other state — the +// `backup=` token absent OR `backup=0` — is excluded. We deliberately treat unset as +// uncovered (the safe DR direction: never imply an unprotected volume is backed up). +func uncoveredMountpoints(mps map[string]string) []string { + var out []string + for key, cfg := range mps { + if mountpointCovered(cfg) { + continue + } + out = append(out, mountpointLabel(key, cfg)) + } + sort.Strings(out) + if out == nil { + return []string{} + } + return out +} + +// mountpointCovered reports whether a mpN config string is included in the vzdump — true +// ONLY when it carries an explicit `backup=1`. +func mountpointCovered(cfg string) bool { + for _, tok := range strings.Split(cfg, ",") { + if v, ok := strings.CutPrefix(tok, "backup="); ok { + return v == "1" + } + } + return false // no backup= token → opt-out by default → not covered +} + +// mountpointLabel prefers the mp=/path token, falling back to the mpN key. +func mountpointLabel(key, cfg string) string { + for _, tok := range strings.Split(cfg, ",") { + if p, ok := strings.CutPrefix(tok, "mp="); ok && p != "" { + return p + } + } + return key +} + +// ToHubRestoreTest maps a reconcile restore-test result to the hub wire record (the backup +// package owns the reconcile→hub mapping so reconcile need not import hub for the result). +func ToHubRestoreTest(res reconcile.RestoreTestResult, testedAt time.Time) hub.RestoreTest { + rt := hub.RestoreTest{ + SourceArchive: res.Archive, + SourceTier: res.SourceTier, + ScratchVMID: res.ScratchVMID, + Pass: res.Pass, + Verified: res.Verified, + TestedAt: testedAt.Format(time.RFC3339), + DurationSeconds: res.Duration.Seconds(), + } + if res.Err != nil { + rt.Error = res.Err.Error() + } + return rt +} diff --git a/internal/backup/schedule.go b/internal/backup/schedule.go new file mode 100644 index 0000000..3ea4cea --- /dev/null +++ b/internal/backup/schedule.go @@ -0,0 +1,111 @@ +package backup + +import ( + "context" + "log/slog" + "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/reconcile" +) + +// RestoreTestRunner is the reconcile-engine seam the scheduler drives (*reconcile.Engine +// satisfies it). Kept narrow so the scheduler is unit-testable with a fake. +type RestoreTestRunner interface { + RunRestoreTest(ctx context.Context, spec reconcile.RestoreTestSpec) reconcile.RestoreTestResult +} + +// CandidatePicker resolves the archive volid to restore-test (newest backup), or "" when +// there is none yet (the tick then no-ops). +type CandidatePicker func(ctx context.Context) (string, error) + +// Scheduler runs the self-restore-test on an agent-internal cadence. It is the fourth daemon +// goroutine; it does real restore→boot→destroy, so it only runs when the cadence is enabled +// AND a valid scratch band is configured (validated by the caller before construction). +type Scheduler struct { + runner RestoreTestRunner + pick CandidatePicker + store *Store + spec reconcile.RestoreTestSpec // archive is filled per-tick + cadence time.Duration + logger *slog.Logger + now func() time.Time +} + +// SchedulerOptions configures a Scheduler. +type SchedulerOptions struct { + Runner RestoreTestRunner + Pick CandidatePicker + Store *Store + Spec reconcile.RestoreTestSpec // RestoreStorage, ScratchMin/Max, SourceTier, BootTimeout + Cadence time.Duration // 0 → disabled + Logger *slog.Logger +} + +// NewScheduler builds a Scheduler. +func NewScheduler(opts SchedulerOptions) *Scheduler { + logger := opts.Logger + if logger == nil { + logger = slog.Default() + } + return &Scheduler{ + runner: opts.Runner, + pick: opts.Pick, + store: opts.Store, + spec: opts.Spec, + cadence: opts.Cadence, + logger: logger, + now: func() time.Time { return time.Now().UTC() }, + } +} + +// Run fires a restore-test on the cadence until ctx is cancelled. A 0 cadence disables it +// (the goroutine just waits for shutdown). It does NOT fire immediately on start (a restore +// is heavy; the first runs one interval in) — on-demand runs use the selftest harness. +// Returns nil on ctx cancellation. +func (s *Scheduler) Run(ctx context.Context) error { + if s.cadence <= 0 || s.runner == nil || s.pick == nil { + s.logger.Info("backup: restore-test cadence disabled") + <-ctx.Done() + return nil + } + s.logger.Info("backup: restore-test scheduler starting", "cadence", s.cadence) + t := time.NewTicker(s.cadence) + defer t.Stop() + for { + select { + case <-ctx.Done(): + s.logger.Info("backup: restore-test scheduler shutting down", "reason", ctx.Err()) + return nil + case <-t.C: + s.tick(ctx) + } + } +} + +// tick runs one scheduled restore-test: pick a backup → run → record. No-ops cleanly when +// no backup exists yet. Deterministic given s.now — tests call it directly. +func (s *Scheduler) tick(ctx context.Context) { + archive, err := s.pick(ctx) + if err != nil { + s.logger.Warn("backup: restore-test could not pick a candidate; skipping", "err", err) + return + } + if archive == "" { + s.logger.Info("backup: restore-test skipped; no backup available yet") + return + } + spec := s.spec + spec.Archive = archive + res := s.runner.RunRestoreTest(ctx, spec) + if res.Skipped { + return // already logged by the engine (no free scratch VMID) + } + rt := ToHubRestoreTest(res, s.now()) + s.store.RecordRestoreTest(rt) + if rt.Pass { + s.logger.Info("backup: scheduled restore-test passed", "archive", rt.SourceArchive, "duration_s", rt.DurationSeconds) + } else { + // A failing restore-test is the loudest DR signal there is. + s.logger.Error("backup: scheduled restore-test FAILED", "archive", rt.SourceArchive, "err", rt.Error) + } +} diff --git a/internal/backup/store.go b/internal/backup/store.go new file mode 100644 index 0000000..6005059 --- /dev/null +++ b/internal/backup/store.go @@ -0,0 +1,61 @@ +package backup + +import ( + "context" + "sync" + + "gitea.dooplex.hu/admin/felhom-agent/internal/hub" +) + +// Store holds the agent's LATEST backup result per target and the latest restore-test +// result — the point-in-time state the host-report surfaces. It is updated by the backup +// runner + the restore-test scheduler/selftest and read by the collector via the hub +// BackupReporter / RestoreTestReporter seams. In-memory (lost on restart; the cadence +// re-populates) and mutex-guarded for the concurrent collector vs scheduler access. +type Store struct { + mu sync.Mutex + byTarget map[string]hub.Backup // latest backup per target id + lastTest *hub.RestoreTest +} + +// NewStore builds an empty Store. +func NewStore() *Store { + return &Store{byTarget: map[string]hub.Backup{}} +} + +// RecordBackup stores the latest backup for its target. +func (s *Store) RecordBackup(b hub.Backup) { + s.mu.Lock() + defer s.mu.Unlock() + s.byTarget[b.TargetID] = b +} + +// RecordRestoreTest stores the latest restore-test result. +func (s *Store) RecordRestoreTest(r hub.RestoreTest) { + s.mu.Lock() + defer s.mu.Unlock() + cp := r + s.lastTest = &cp +} + +// Backups implements hub.BackupReporter — the latest backup per target (stable order by +// target id is not guaranteed; the hub does not depend on order). +func (s *Store) Backups(context.Context) []hub.Backup { + s.mu.Lock() + defer s.mu.Unlock() + out := make([]hub.Backup, 0, len(s.byTarget)) + for _, b := range s.byTarget { + out = append(out, b) + } + return out +} + +// RestoreTests implements hub.RestoreTestReporter — the latest restore-test result (0 or 1). +func (s *Store) RestoreTests(context.Context) []hub.RestoreTest { + s.mu.Lock() + defer s.mu.Unlock() + if s.lastTest == nil { + return []hub.RestoreTest{} + } + return []hub.RestoreTest{*s.lastTest} +} diff --git a/internal/config/config.go b/internal/config/config.go index 5b9bf2c..93b10fd 100644 --- a/internal/config/config.go +++ b/internal/config/config.go @@ -27,9 +27,73 @@ type Config struct { Authz AuthzConfig `json:"authz"` Hub HubConfig `json:"hub"` Storage StorageConfig `json:"storage"` + Backup BackupConfig `json:"backup"` LogLevel string `json:"log_level"` // debug|info|warn|error (default info) } +// BackupConfig tunes the slice-6 backup + self-restore-test layer. The restore-test runs on +// an agent-internal cadence (no hub policy needed — it's self-validation); the backup +// schedule/retention/target-selection policy is hub-manifest-owned and unfed until slice 10. +type BackupConfig struct { + // LocalBackupTarget is the vzdump storage (content=backup) backups go to, e.g. "local". + LocalBackupTarget string `json:"local_backup_target"` + // RestoreStorage is where a restore-test's restored rootfs lands, e.g. "local-lvm". + RestoreStorage string `json:"restore_storage"` + // RestoreTestCadenceSeconds is the self-restore-test interval; 0 → default (24h). + // Set negative to DISABLE the automatic cadence (on-demand selftest still works). + RestoreTestCadenceSeconds int `json:"restore_test_cadence_seconds"` + // ScratchVMIDMin/Max bound the throwaway restore-test scratch-guest VMID band. The + // restore-test refuses to run unless this is a valid band (min>0, max>=min); 9999 is + // always excluded. Defaults to 990000–990009. + ScratchVMIDMin int `json:"scratch_vmid_min"` + ScratchVMIDMax int `json:"scratch_vmid_max"` +} + +// Default scratch VMID band + restore-test cadence. +const ( + defaultScratchVMIDMin = 990000 + defaultScratchVMIDMax = 990009 + defaultRestoreTestCadence = 24 * time.Hour +) + +// RestoreTestCadence returns the configured restore-test interval: a positive value as-is, +// 0 → 24h default, negative → 0 (disabled). +func (b BackupConfig) RestoreTestCadence() time.Duration { + switch { + case b.RestoreTestCadenceSeconds > 0: + return time.Duration(b.RestoreTestCadenceSeconds) * time.Second + case b.RestoreTestCadenceSeconds < 0: + return 0 // disabled + default: + return defaultRestoreTestCadence + } +} + +// ScratchBand returns the effective [min,max] scratch VMID band (defaults applied). +func (b BackupConfig) ScratchBand() (min, max int) { + min, max = b.ScratchVMIDMin, b.ScratchVMIDMax + if min == 0 && max == 0 { + return defaultScratchVMIDMin, defaultScratchVMIDMax + } + return min, max +} + +// ValidateForRestoreTest checks the scratch band is usable. Called only when the restore-test +// cadence is enabled (so a host that never restore-tests needn't configure a band). +func (b BackupConfig) ValidateForRestoreTest() error { + min, max := b.ScratchBand() + if min <= 0 || max < min { + return fmt.Errorf("config: backup.scratch_vmid_[min,max] is an invalid band [%d,%d]", min, max) + } + if 9999 >= min && 9999 <= max { + return fmt.Errorf("config: backup scratch band [%d,%d] must not include the standing scratch 9999", min, max) + } + if b.RestoreStorage == "" { + return fmt.Errorf("config: backup.restore_storage is required when the restore-test cadence is enabled") + } + return nil +} + // StorageConfig tunes the storage watchdog (slice 5). All optional — zero values fall back // to the storage package defaults via the accessor methods. The watchdog poll is FAST // (seconds) to catch a USB drop quickly; the debounce keeps a flapping drive from storming @@ -203,6 +267,14 @@ func applyEnv(cfg *Config) { } cfg.Hub.PollSeconds = envInt("FELHOM_AGENT_HUB_POLL_SECONDS", cfg.Hub.PollSeconds) cfg.Hub.TimeoutSeconds = envInt("FELHOM_AGENT_HUB_TIMEOUT_SECONDS", cfg.Hub.TimeoutSeconds) + // backup (slice 6) + if v := os.Getenv("FELHOM_AGENT_BACKUP_LOCAL_TARGET"); v != "" { + cfg.Backup.LocalBackupTarget = v + } + if v := os.Getenv("FELHOM_AGENT_BACKUP_RESTORE_STORAGE"); v != "" { + cfg.Backup.RestoreStorage = v + } + cfg.Backup.RestoreTestCadenceSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_CADENCE_SECONDS", cfg.Backup.RestoreTestCadenceSeconds) } // envInt overlays an int env var, keeping cur (with a stderr warning) on parse diff --git a/internal/hub/collect.go b/internal/hub/collect.go index c768c6d..d24313a 100644 --- a/internal/hub/collect.go +++ b/internal/hub/collect.go @@ -31,12 +31,25 @@ type StorageObserver interface { Observe(ctx context.Context) ([]StorageTarget, error) } +// BackupReporter / RestoreTestReporter are the slice-6 seams the backup layer plugs into +// (same consumer-side pattern as StorageObserver — hub does not import the backup package). +// They return the agent's LATEST-known backup-per-target / restore-test result (point-in-time +// state the backup layer accumulates), not a live scan. A nil reporter → empty slice. +type BackupReporter interface { + Backups(ctx context.Context) []Backup +} +type RestoreTestReporter interface { + RestoreTests(ctx context.Context) []RestoreTest +} + // Collector builds a HostReport from read-only sources. All deps are behind narrow // interfaces for unit testing. type Collector struct { px proxmoxReader cf CloudflaredProber storage StorageObserver + backups BackupReporter + restoreTests RestoreTestReporter hostID string agentVersion string logger *slog.Logger @@ -44,8 +57,8 @@ type Collector struct { } // NewCollector builds a collector. hostID echoes config.Hub.HostID; agentVersion is -// the binary version. storage may be nil (storage_targets emitted empty). -func NewCollector(px proxmoxReader, cf CloudflaredProber, storage StorageObserver, hostID, agentVersion string, logger *slog.Logger) *Collector { +// the binary version. storage/backups/restoreTests may be nil (their collections emit empty). +func NewCollector(px proxmoxReader, cf CloudflaredProber, storage StorageObserver, backups BackupReporter, restoreTests RestoreTestReporter, hostID, agentVersion string, logger *slog.Logger) *Collector { if logger == nil { logger = slog.Default() } @@ -53,6 +66,8 @@ func NewCollector(px proxmoxReader, cf CloudflaredProber, storage StorageObserve px: px, cf: cf, storage: storage, + backups: backups, + restoreTests: restoreTests, hostID: hostID, agentVersion: agentVersion, logger: logger, @@ -79,9 +94,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) { // storage_targets populated this slice (slice 5) via the observer; the rest stay // defined-but-empty (slice 6). Non-nil so they marshal as []. StorageTargets: c.collectStorage(ctx), - Backups: []Backup{}, - RestoreTests: []RestoreTest{}, - PBSSnapshots: []PBSSnapshot{}, + Backups: c.collectBackups(ctx), + RestoreTests: c.collectRestoreTests(ctx), + PBSSnapshots: []PBSSnapshot{}, // Phase B + AuditTail: []AuditEntry{}, Cloudflared: Cloudflared{Status: c.cloudflaredStatus(ctx)}, } @@ -159,6 +175,29 @@ func (c *Collector) collectStorage(ctx context.Context) []StorageTarget { return targets } +// collectBackups / collectRestoreTests read the agent's latest backup + restore-test state +// via the seams. Best-effort: a nil reporter or nil slice degrades to an empty (non-nil) +// list so the collection always marshals as []. +func (c *Collector) collectBackups(ctx context.Context) []Backup { + if c.backups == nil { + return []Backup{} + } + if b := c.backups.Backups(ctx); b != nil { + return b + } + return []Backup{} +} + +func (c *Collector) collectRestoreTests(ctx context.Context) []RestoreTest { + if c.restoreTests == nil { + return []RestoreTest{} + } + if r := c.restoreTests.RestoreTests(ctx); r != nil { + return r + } + return []RestoreTest{} +} + func (c *Collector) cloudflaredStatus(ctx context.Context) string { if c.cf == nil { return "unknown" diff --git a/internal/hub/collect_test.go b/internal/hub/collect_test.go index f0be57d..2ce750b 100644 --- a/internal/hub/collect_test.go +++ b/internal/hub/collect_test.go @@ -33,7 +33,7 @@ func TestCollect_StorageTargetsFromObserver(t *testing.T) { obs := fakeObserver{targets: []StorageTarget{ {Name: "local-lvm", Type: StorageTypeLVMThin, State: StorageStateAttached, Reachable: true}, }} - c := NewCollector(px, fakeProber{status: "active"}, obs, "h", "0.5.0", quietLogger()) + c := NewCollector(px, fakeProber{status: "active"}, obs, nil, nil, "h", "0.5.0", quietLogger()) r, err := c.Collect(context.Background()) if err != nil { t.Fatalf("Collect: %v", err) @@ -45,7 +45,7 @@ func TestCollect_StorageTargetsFromObserver(t *testing.T) { func TestCollect_StorageObserverErrorDegradesToEmpty(t *testing.T) { px := &fakePx{node: "n", ns: newTestNodeStatus()} - c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{err: errors.New("proxmox down")}, "h", "0.5.0", quietLogger()) + c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{err: errors.New("proxmox down")}, nil, nil, "h", "0.5.0", quietLogger()) r, err := c.Collect(context.Background()) if err != nil { t.Fatalf("a storage observe error must not sink the heartbeat: %v", err) @@ -64,7 +64,7 @@ func TestCollect_HostAndGuests(t *testing.T) { }, cfg: map[int]proxmox.GuestConfig{100: {Cores: 2, Memory: 2048}}, } - c := NewCollector(px, fakeProber{status: "active"}, nil, "demo-host-01", "0.3.0", quietLogger()) + c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, "demo-host-01", "0.3.0", quietLogger()) r, err := c.Collect(context.Background()) if err != nil { t.Fatalf("Collect: %v", err) @@ -104,7 +104,7 @@ func TestCollect_GuestConfigFailureKeepsStatusOmitsSpec(t *testing.T) { cfg: map[int]proxmox.GuestConfig{100: {Cores: 2}}, cfgErr: map[int]error{200: errors.New("config read failed")}, } - c := NewCollector(px, fakeProber{status: "active"}, nil, "h", "0.3.1", quietLogger()) + c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, "h", "0.3.1", quietLogger()) r, err := c.Collect(context.Background()) if err != nil { t.Fatalf("a per-guest failure must NOT fail the whole report: %v", err) @@ -125,7 +125,7 @@ func TestCollect_GuestConfigFailureKeepsStatusOmitsSpec(t *testing.T) { func TestCollect_NodeStatusFailureIsHardError(t *testing.T) { px := &fakePx{node: "n", nsErr: errors.New("proxmox down")} - c := NewCollector(px, fakeProber{status: "active"}, nil, "h", "0.3.0", quietLogger()) + c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, "h", "0.3.0", quietLogger()) if _, err := c.Collect(context.Background()); err == nil { t.Fatal("NodeStatus failure must be a hard error (no useful report)") } @@ -133,7 +133,7 @@ func TestCollect_NodeStatusFailureIsHardError(t *testing.T) { func TestCollect_CloudflaredProbeErrorIsUnknown(t *testing.T) { px := &fakePx{node: "n", ns: newTestNodeStatus()} - c := NewCollector(px, fakeProber{err: errors.New("no systemctl")}, nil, "h", "0.3.0", quietLogger()) + c := NewCollector(px, fakeProber{err: errors.New("no systemctl")}, nil, nil, nil, "h", "0.3.0", quietLogger()) r, err := c.Collect(context.Background()) if err != nil { t.Fatalf("cloudflared failure must not be fatal: %v", err) diff --git a/internal/hub/contract_test.go b/internal/hub/contract_test.go index bde5020..d992895 100644 --- a/internal/hub/contract_test.go +++ b/internal/hub/contract_test.go @@ -46,7 +46,19 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) { Smart: SmartSummary{Health: SmartUnknown}, }, }, - Backups: []Backup{}, RestoreTests: []RestoreTest{}, + Backups: []Backup{ + { + TargetID: "local", VMID: 9001, Archive: "local:backup/x.tar.zst", Mode: "snapshot", + CrashConsistent: true, SizeBytes: 1, Success: true, StartedAt: "2026-06-09T11:00:00Z", + DurationSeconds: 1, UncoveredVolumes: []string{"/mnt/bulk"}, + }, + }, + RestoreTests: []RestoreTest{ + { + SourceArchive: "local:backup/x.tar.zst", SourceTier: "local", ScratchVMID: 990000, + Pass: true, Verified: "boot+running", TestedAt: "2026-06-09T11:05:00Z", DurationSeconds: 1, + }, + }, PBSSnapshots: []PBSSnapshot{}, AuditTail: []AuditEntry{}, Cloudflared: Cloudflared{Status: "active"}, } @@ -66,6 +78,10 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) { assertSameKeys(t, "storage_targets[0]", gst, sst) assertSameKeys(t, "storage_targets[0].smart", field(gst, "smart"), field(sst, "smart")) assertSameKeys(t, "storage_targets[0].thin_pool", field(gst, "thin_pool"), field(sst, "thin_pool")) + + // slice-6 additions — backups[0] / restore_tests[0] key sets (the bidirectional guard). + assertSameKeys(t, "backups[0]", firstElem(golden["backups"]), firstElem(got["backups"])) + assertSameKeys(t, "restore_tests[0]", firstElem(golden["restore_tests"]), firstElem(got["restore_tests"])) } // field extracts a nested object value from a decoded JSON map (nil if absent/not a map). diff --git a/internal/hub/report.go b/internal/hub/report.go index 28acd57..5fc037b 100644 --- a/internal/hub/report.go +++ b/internal/hub/report.go @@ -165,10 +165,41 @@ const ( StorageStateDecommissioned = "decommissioned" ) -type Backup struct{} // slice 6: per-target backup status fields TBD -type RestoreTest struct{} // slice 6: self-restore-test result fields TBD -type PBSSnapshot struct{} // slice 6: PBS snapshot inventory fields TBD -type AuditEntry struct{} // audit-log tail entry fields TBD +// Backup is the latest guest-vzdump result per target (doc 03 §8, slice 6 Phase A). An +// agent-initiated vzdump is CRASH-CONSISTENT only (no fsfreeze; app-consistency needs the +// controller quiesce, slice 8) — marked so here. UncoveredVolumes lists the guest's +// backup=0 (or backup-unset) mountpoints excluded from the vzdump — the bulk-volume DR gap +// (the bulk-backup mechanism is slice 10). Cross-repo contract: keep byte-identical with +// felhom.eu/hub and the bidirectional golden key-set test. +type Backup struct { + TargetID string `json:"target_id"` // backup storage name (e.g. "local") + VMID int `json:"vmid"` // source guest + Archive string `json:"archive"` // produced vzdump volid + Mode string `json:"mode"` // snapshot | stop + CrashConsistent bool `json:"crash_consistent"` // always true this slice + SizeBytes int64 `json:"size_bytes"` + Success bool `json:"success"` + Error string `json:"error,omitempty"` + StartedAt string `json:"started_at"` // RFC3339 + DurationSeconds float64 `json:"duration_seconds"` + UncoveredVolumes []string `json:"uncovered_volumes"` // backup=0/unset mountpoints (bulk gap) +} + +// RestoreTest is the latest self-restore-test result (doc 03 §8). This slice verifies +// boot+running only (deep app-health is slice 8). SourceTier is "local" here; PBS is Phase B. +type RestoreTest struct { + SourceArchive string `json:"source_archive"` + SourceTier string `json:"source_tier"` // "local" (pbs = Phase B) + ScratchVMID int `json:"scratch_vmid"` + Pass bool `json:"pass"` + Verified string `json:"verified"` // "boot+running" this slice + Error string `json:"error,omitempty"` + TestedAt string `json:"tested_at"` // RFC3339 + DurationSeconds float64 `json:"duration_seconds"` +} + +type PBSSnapshot struct{} // slice 6 Phase B: PBS snapshot inventory fields TBD +type AuditEntry struct{} // audit-log tail entry fields TBD // ControlEnvelope is the hub's 200 response to a host-report. This slice the agent // adopts ONLY PollIntervalSeconds; the rest are reserved/forward-compat fields it diff --git a/internal/hub/testdata/host-report.golden.json b/internal/hub/testdata/host-report.golden.json index 73ff263..8778f0a 100644 --- a/internal/hub/testdata/host-report.golden.json +++ b/internal/hub/testdata/host-report.golden.json @@ -86,8 +86,31 @@ } } ], - "backups": [], - "restore_tests": [], + "backups": [ + { + "target_id": "local", + "vmid": 9001, + "archive": "local:backup/vzdump-lxc-9001-2026_06_09-11_00_00.tar.zst", + "mode": "snapshot", + "crash_consistent": true, + "size_bytes": 524288000, + "success": true, + "started_at": "2026-06-09T11:00:00Z", + "duration_seconds": 42.5, + "uncovered_volumes": ["/mnt/bulk"] + } + ], + "restore_tests": [ + { + "source_archive": "local:backup/vzdump-lxc-9001-2026_06_09-11_00_00.tar.zst", + "source_tier": "local", + "scratch_vmid": 990000, + "pass": true, + "verified": "boot+running", + "tested_at": "2026-06-09T11:05:00Z", + "duration_seconds": 38.2 + } + ], "pbs_snapshots": [], "cloudflared": { "status": "active" }, "audit_tail": [] diff --git a/internal/proxmox/mutate.go b/internal/proxmox/mutate.go index d0ca8c1..bbc83fc 100644 --- a/internal/proxmox/mutate.go +++ b/internal/proxmox/mutate.go @@ -61,10 +61,14 @@ type VzdumpOptions struct { Storage string // a storage whose content includes "backup" (e.g. "local") — NOT local-lvm Mode BackupMode // ModeStop | ModeSnapshot Compress string // "zstd" (default), "lzo", "gzip", or "" for none + // Notes is the PVE `notes-template` for the backup (a template string PVE expands, + // e.g. with {{guestname}}/{{node}}). Optional. + Notes string } // Vzdump starts a backup via POST /nodes/{node}/vzdump. Returns the UPID. An -// agent-initiated vzdump is crash-consistent only for an LXC (no fsfreeze). +// agent-initiated vzdump is crash-consistent only for an LXC (no fsfreeze) — +// app-consistency needs the controller to quiesce first (slice 8). func (c *Client) Vzdump(ctx context.Context, opts VzdumpOptions) (string, error) { if opts.VMID == 0 || opts.Storage == "" || opts.Mode == "" { return "", fmt.Errorf("proxmox: Vzdump needs vmid, storage and mode") @@ -77,9 +81,28 @@ func (c *Client) Vzdump(ctx context.Context, opts VzdumpOptions) (string, error) opts.Compress = "zstd" } v.Set("compress", opts.Compress) + if opts.Notes != "" { + v.Set("notes-template", opts.Notes) // PVE 9.x param name (verified on demo) + } return c.dataString(ctx, http.MethodPost, "/nodes/"+c.node+"/vzdump", v) } +// DestroyLXC destroys a guest via DELETE /nodes/{node}/lxc/{vmid}. Returns the UPID. +// `purge=1` also drops the guest from jobs/HA; `destroy-unreferenced-disks=1` reaps any +// orphaned volumes. This is the scratch-guest teardown primitive (slice 6); it is +// destructive-class and the caller MUST route it through the reversibility gate +// (benign only by agent-internal scratch/same-txn provenance — see reconcile.Classify). +func (c *Client) DestroyLXC(ctx context.Context, vmid int) (string, error) { + if vmid == 0 { + return "", fmt.Errorf("proxmox: DestroyLXC needs a vmid") + } + v := url.Values{} + v.Set("purge", "1") + v.Set("destroy-unreferenced-disks", "1") + path := fmt.Sprintf("/nodes/%s/lxc/%d", c.node, vmid) + return c.dataString(ctx, http.MethodDelete, path, v) +} + // Snapshot creates an LXC snapshot via POST /nodes/{node}/lxc/{vmid}/snapshot. // A running, unprivileged LXC can be snapshotted on LVM-thin with no stop // (phase1-2 §1.6) — this is the snapshot-before-change primitive. diff --git a/internal/proxmox/query.go b/internal/proxmox/query.go index 60aa082..62cf187 100644 --- a/internal/proxmox/query.go +++ b/internal/proxmox/query.go @@ -74,5 +74,28 @@ func (c *Client) StorageContent(ctx context.Context, store string) ([]StorageCon return cs, c.get(ctx, path, &cs) } +// LatestBackupVolID resolves the most recent vzdump archive for vmid on a backup storage. +// It lists the store's content, keeps only backup archives for that vmid, and returns the +// one with the greatest CTime. This is how a backup's produced archive is resolved after +// Vzdump+WaitTask (the task status carries no result volid), and how the restore-test picks +// a backup to restore. Returns ("", nil) when the guest has no archive on that store. +func (c *Client) LatestBackupVolID(ctx context.Context, store string, vmid int) (string, error) { + contents, err := c.StorageContent(ctx, store) + if err != nil { + return "", err + } + var bestVol string + var bestCTime int64 = -1 + for _, e := range contents { + if e.Content != "backup" || e.VMID != vmid { + continue + } + if e.CTime > bestCTime { + bestCTime, bestVol = e.CTime, e.VolID + } + } + return bestVol, nil +} + // urlEscape escapes a path segment (a UPID contains ':' and '@'). func urlEscape(s string) string { return url.PathEscape(s) } diff --git a/internal/reconcile/engine_test.go b/internal/reconcile/engine_test.go index b3250dd..b4d5fa3 100644 --- a/internal/reconcile/engine_test.go +++ b/internal/reconcile/engine_test.go @@ -20,18 +20,24 @@ type fakeAPI struct { startUPID, stopUPID, setUPID, resizeUPID string startErr, stopErr, setErr, resizeErr error + restoreUPID, destroyUPID string + restoreErr, destroyErr error + // status maps vmid -> the Guest returned by GuestStatus (default running if absent). + status map[int]proxmox.Guest // waitFunc maps a UPID to a (status, err); default = OK. Mirrors the real client, // which errors on a non-OK exitstatus. waitFunc func(upid string) (proxmox.TaskStatus, error) // statusFunc backs TaskStatusOnce (crash recovery); default = stopped/OK. statusFunc func(upid string) (proxmox.TaskStatus, error) - starts []int - stops []int - sets []setCall - resizes []resizeCall - waits []string - listErr error + starts []int + stops []int + sets []setCall + resizes []resizeCall + restores []proxmox.RestoreLXCOptions + destroys []int + waits []string + listErr error } type resizeCall struct { @@ -39,6 +45,29 @@ type resizeCall struct { disk, size string } +func (f *fakeAPI) RestoreLXC(_ context.Context, opts proxmox.RestoreLXCOptions) (string, error) { + f.mu.Lock() + f.restores = append(f.restores, opts) + f.mu.Unlock() + return f.restoreUPID, f.restoreErr +} + +func (f *fakeAPI) DestroyLXC(_ context.Context, vmid int) (string, error) { + f.mu.Lock() + f.destroys = append(f.destroys, vmid) + f.mu.Unlock() + return f.destroyUPID, f.destroyErr +} + +func (f *fakeAPI) GuestStatus(_ context.Context, vmid int) (proxmox.Guest, error) { + f.mu.Lock() + defer f.mu.Unlock() + if g, ok := f.status[vmid]; ok { + return g, nil + } + return proxmox.Guest{VMID: vmid, Status: "running"}, nil +} + func (f *fakeAPI) TaskStatusOnce(_ context.Context, upid string) (proxmox.TaskStatus, error) { if f.statusFunc != nil { return f.statusFunc(upid) diff --git a/internal/reconcile/journal.go b/internal/reconcile/journal.go index b548648..10bf472 100644 --- a/internal/reconcile/journal.go +++ b/internal/reconcile/journal.go @@ -45,7 +45,13 @@ type JournalEntry struct { UPID string `json:"upid,omitempty"` State OpState `json:"state"` IdempKey string `json:"idemp_key,omitempty"` - At time.Time `json:"at"` + // Scratch marks an entry that OWNS an agent-tagged scratch guest at VMID (slice 6 + // restore-test). While such an entry is in-flight, the load-bearing invariant is "the + // scratch guest may exist and MUST be destroyed" — so Recover resolves it by ensuring + // VMID is gone (a benign teardown), NOT by re-checking any sub-task UPID. The entry is + // terminal only after teardown. See recover.go. + Scratch bool `json:"scratch,omitempty"` + At time.Time `json:"at"` } // Journal is the durable operation log + idempotency store. It mirrors diff --git a/internal/reconcile/recover.go b/internal/reconcile/recover.go index 1e1e7f7..91bb97c 100644 --- a/internal/reconcile/recover.go +++ b/internal/reconcile/recover.go @@ -3,6 +3,8 @@ package reconcile import ( "context" "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" ) // Recover consumes the journal's in-flight set at startup: resume-or-rollback for any @@ -32,6 +34,16 @@ func (e *Engine) Recover(ctx context.Context) RecoverResult { } for _, entry := range e.journal.InFlight() { res.Examined++ + + // Scratch entries (slice-6 restore-test) are resolved by TEARDOWN, not by + // re-checking a sub-task UPID — a leaked scratch guest is the failure mode that + // matters. Handle them BEFORE the generic UPID path (else the restore sub-task's OK + // status would mark the entry succeeded while the guest still exists → leak). + if entry.Scratch { + e.recoverScratch(ctx, entry, &res) + continue + } + if entry.UPID == "" { // POST never confirmed → abandon (fail-safe). e.append(terminal(entry, OpFailed)) @@ -72,18 +84,80 @@ func (e *Engine) Recover(ctx context.Context) RecoverResult { return res } +// recoverScratch resolves a leaked restore-test scratch guest (slice 6, doc 03 §8/§10). +// The invariant: a Scratch entry in-flight at startup means "scratch guest VMID may exist +// and must be destroyed." It is idempotent — if the guest is already gone (crash after the +// destroy task but before the terminal record), it records terminal-clean. The teardown +// routes through the gate as a benign ClassGuestDestroy (agent-tagged scratch provenance) — +// the same audit-bearing path the normal teardown uses. +func (e *Engine) recoverScratch(ctx context.Context, entry JournalEntry, res *RecoverResult) { + lxc, err := e.api.ListLXC(ctx) + if err != nil { + // Can't tell whether the guest exists → leave in-flight; a later Recover retries. + res.Unresolved++ + e.logger.Warn("recover: cannot list guests to resolve leaked scratch; left in-flight", + "op_id", entry.OpID, "vmid", entry.VMID, "err", err) + return + } + exists := false + for _, g := range lxc { + if g.VMID == entry.VMID { + exists = true + break + } + } + if !exists { + // Already gone (idempotent) → the scratch left no leak. + e.append(terminal(entry, OpSucceeded)) + res.ScratchClean++ + e.logger.Info("recover: leaked-scratch entry resolved; guest already gone", + "op_id", entry.OpID, "vmid", entry.VMID) + return + } + + dec := e.gate.Authorize(IntentForScratchDestroy(e.hostID, entry.VMID), nil) + if !dec.Allowed { + // Should be benign; if not, fail-safe (leave in-flight, do NOT force a destroy). + res.Unresolved++ + e.logger.Error("recover: scratch teardown refused by gate (unexpected); left in-flight", + "op_id", entry.OpID, "vmid", entry.VMID, "reason", dec.Reason) + return + } + upid, err := e.api.DestroyLXC(ctx, entry.VMID) + if err != nil { + res.Unresolved++ + e.logger.Warn("recover: destroying leaked scratch failed; left in-flight (will retry)", + "op_id", entry.OpID, "vmid", entry.VMID, "err", err) + return + } + if upid != "" { + if _, err := e.api.WaitTask(ctx, upid, proxmox.WaitOptions{}); err != nil { + res.Unresolved++ + e.logger.Warn("recover: leaked-scratch destroy task failed; left in-flight (will retry)", + "op_id", entry.OpID, "vmid", entry.VMID, "err", err) + return + } + } + e.append(terminal(entry, OpSucceeded)) + res.ScratchDestroyed++ + e.logger.Warn("recover: destroyed leaked restore-test scratch guest", + "op_id", entry.OpID, "vmid", entry.VMID) +} + // RecoverResult summarizes a startup recovery pass. type RecoverResult struct { - Examined int - Resumed int // task found completed OK and recorded succeeded - Failed int // task found ended non-OK and recorded failed - RolledBack int // no task id → abandoned (fail-safe) - StillRunning int // task still executing → left in-flight - Unresolved int // task status unreadable → left in-flight + Examined int + Resumed int // task found completed OK and recorded succeeded + Failed int // task found ended non-OK and recorded failed + RolledBack int // no task id → abandoned (fail-safe) + StillRunning int // task still executing → left in-flight + Unresolved int // task status unreadable → left in-flight + ScratchClean int // scratch entry resolved: guest already gone (no leak) + ScratchDestroyed int // scratch entry resolved: leaked guest destroyed } // terminal builds a terminal journal record preserving the op's identity, with the -// idempotency key carried through so a SUCCEEDED one-shot op marks its key applied. +// idempotency key + scratch flag carried through. func terminal(e JournalEntry, state OpState) JournalEntry { return JournalEntry{ OpID: e.OpID, @@ -92,6 +166,7 @@ func terminal(e JournalEntry, state OpState) JournalEntry { UPID: e.UPID, State: state, IdempKey: e.IdempKey, + Scratch: e.Scratch, At: time.Now().UTC(), } } diff --git a/internal/reconcile/restoretest.go b/internal/reconcile/restoretest.go new file mode 100644 index 0000000..c78c444 --- /dev/null +++ b/internal/reconcile/restoretest.go @@ -0,0 +1,289 @@ +package reconcile + +import ( + "context" + "fmt" + "strconv" + "strings" + "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" +) + +// The self-restore-test (doc 03 §8) — the piece that closes "a backup you haven't restored +// isn't a backup". It is a JOURNALED reconcile job so it inherits the slice-4 journal, +// per-guest serialization, and crash-safe recovery: a mid-test crash can't leak a scratch +// guest (engine.Recover tears it down). Every step here is BENIGN — restore-to-new +// (ClassCreate), a benign net-link-down SetConfig, and a scratch teardown that is benign by +// agent-tagged-scratch provenance (no new destructive class, no new crypto). + +// scratchKind is the journal Kind for a restore-test scratch-guest-owning entry. Recover +// keys off JournalEntry.Scratch (not this string), but the Kind aids audit/debug. +const scratchKind = "scratch_restore_test" + +// DefaultBootTimeout bounds how long the restore-test waits for the scratch guest to reach +// running before declaring the verify failed. +const DefaultBootTimeout = 2 * time.Minute + +// RestoreTestSpec parameterizes one restore-test. +type RestoreTestSpec struct { + Archive string // source archive volid to restore (resolved by the caller) + SourceTier string // "local" this slice (pbs = Phase B) — for the report + RestoreStorage string // target storage for the restored rootfs (e.g. "local-lvm") + ScratchMin int // inclusive scratch VMID band (must be > 0) + ScratchMax int // inclusive + BootTimeout time.Duration // 0 → DefaultBootTimeout +} + +// RestoreTestResult is the reconcile-local outcome (the backup package maps it to the +// hub.RestoreTest wire record — reconcile must not import hub for this). +type RestoreTestResult struct { + Archive string + SourceTier string + ScratchVMID int + Pass bool + Verified string // "boot+running" this slice + Skipped bool // no free scratch VMID in band → test not run + Err error + StartedAt time.Time + Duration time.Duration +} + +// IntentForScratchDestroy builds the benign teardown intent for an agent-owned scratch +// guest: ClassGuestDestroy made benign by AgentTaggedScratch provenance (classify.go). The +// gate authorizes it unsigned but is genuinely in-path (wrong provenance → pending_signature). +func IntentForScratchDestroy(hostID string, vmid int) Intent { + return Intent{ + Class: ClassGuestDestroy, + HostID: hostID, + GuestID: strconv.Itoa(vmid), + VMID: vmid, + Provenance: Provenance{AgentTaggedScratch: true}, // agent-internal, never hub-sourced + Source: SourceOneShotJob, + } +} + +// RunRestoreTest runs one restore-test on the per-guest queue lane of a fresh scratch VMID. +// It journals a Scratch-owned entry BEFORE any mutation, so a crash anywhere after this +// point is recoverable (Recover destroys the scratch guest). Teardown runs on EVERY path +// (defer), including a failed verify. The returned Err is the TEST verdict's error (restore +// or boot failure), independent of teardown success. +func (e *Engine) RunRestoreTest(ctx context.Context, spec RestoreTestSpec) RestoreTestResult { + now := time.Now().UTC() + res := RestoreTestResult{Archive: spec.Archive, SourceTier: spec.SourceTier, StartedAt: now} + + if spec.Archive == "" || spec.RestoreStorage == "" { + res.Err = fmt.Errorf("reconcile: restore-test needs an archive and a restore storage") + return res + } + if spec.ScratchMin <= 0 || spec.ScratchMax < spec.ScratchMin { + res.Err = fmt.Errorf("reconcile: invalid scratch VMID band [%d,%d]", spec.ScratchMin, spec.ScratchMax) + return res + } + + lxc, err := e.api.ListLXC(ctx) + if err != nil { + res.Err = fmt.Errorf("reconcile: restore-test list guests: %w", err) + return res + } + vmid, ok := pickScratchVMID(lxc, spec.ScratchMin, spec.ScratchMax) + if !ok { + // Full band (e.g. an accumulation of un-torn-down scratch guests) → skip, never + // panic or pick out-of-band. Recover will reap any genuinely leaked ones. + e.logger.Warn("restore-test skipped: no free scratch VMID in band", + "min", spec.ScratchMin, "max", spec.ScratchMax) + res.Skipped = true + return res + } + res.ScratchVMID = vmid + + // Serialize on the scratch VMID's lane (inherits §10), and capture the result. + ch := e.queue.Submit(vmid, func() error { + e.runScratchTest(ctx, vmid, spec, &res) + return res.Err + }) + <-ch + res.Duration = time.Since(now) + return res +} + +// runScratchTest is the journaled body (runs on vmid's queue lane). +func (e *Engine) runScratchTest(ctx context.Context, vmid int, spec RestoreTestSpec, res *RestoreTestResult) { + base := JournalEntry{OpID: e.scratchOpID(vmid), VMID: vmid, Kind: scratchKind, Scratch: true} + + // OWN the scratch guest's cleanup BEFORE any mutation. From here, a crash is recoverable. + e.append(withState(base, OpStarted)) + + // Teardown ALWAYS runs (even on a failed verify). Uses a cancel-immune context so a + // daemon shutdown mid-test still tears down; if teardown fails, the entry stays + // in-flight and Recover reaps the guest on the next start. + defer e.teardownScratch(ctx, base) + + // 1. Restore into the fresh scratch VMID (benign create path). The UPID is for error + // detection only — it does NOT make the Scratch entry terminal (teardown does). + upid, err := e.api.RestoreLXC(ctx, proxmox.RestoreLXCOptions{ + VMID: vmid, Archive: spec.Archive, Storage: spec.RestoreStorage, + }) + if err != nil { + res.Err = fmt.Errorf("reconcile: restore-test restore: %w", err) + return + } + e.append(withUPID(base, upid, OpTaskRunning)) + if upid != "" { + if _, err := e.api.WaitTask(ctx, upid, proxmox.WaitOptions{}); err != nil { + res.Err = fmt.Errorf("reconcile: restore-test restore task: %w", err) + return + } + } + + // 2. Net link-down on every interface BEFORE boot — test-safety so the clone (which + // keeps the source MAC/hostname; identity-reset is slice 7) can't conflict with a + // running source on L2/IP. Benign SetConfig. + cfg, err := e.api.GuestConfig(ctx, vmid) + if err != nil { + res.Err = fmt.Errorf("reconcile: restore-test read scratch config: %w", err) + return + } + for key, val := range cfg.Nets() { + if _, err := e.api.SetConfig(ctx, vmid, map[string]string{key: withLinkDown(val)}); err != nil { + res.Err = fmt.Errorf("reconcile: restore-test net link-down %s: %w", key, err) + return + } + } + + // 3. Boot and verify it reaches running (basic liveness; deep app-health is slice 8). + startUPID, err := e.api.Start(ctx, vmid) + if err != nil { + res.Err = fmt.Errorf("reconcile: restore-test start: %w", err) + return + } + if startUPID != "" { + if _, err := e.api.WaitTask(ctx, startUPID, proxmox.WaitOptions{}); err != nil { + res.Err = fmt.Errorf("reconcile: restore-test start task: %w", err) + return + } + } + if err := e.waitRunning(ctx, vmid, bootTimeout(spec)); err != nil { + res.Err = err + return + } + res.Pass = true + res.Verified = "boot+running" +} + +// teardownScratch destroys the scratch guest (benign, gated) and records the entry terminal. +// On any teardown failure it leaves the entry in-flight so Recover reaps the guest later. +func (e *Engine) teardownScratch(ctx context.Context, base JournalEntry) { + // Cancel-immune + bounded, so a shutdown mid-test still tears down. + tctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Minute) + defer cancel() + + dec := e.gate.Authorize(IntentForScratchDestroy(e.hostID, base.VMID), nil) + if !dec.Allowed { + e.logger.Error("restore-test: scratch teardown refused by gate (unexpected); left for Recover", + "vmid", base.VMID, "reason", dec.Reason) + return + } + upid, err := e.api.DestroyLXC(tctx, base.VMID) + if err != nil { + e.logger.Error("restore-test: scratch teardown failed; left for Recover", "vmid", base.VMID, "err", err) + return + } + if upid != "" { + if _, err := e.api.WaitTask(tctx, upid, proxmox.WaitOptions{}); err != nil { + e.logger.Error("restore-test: scratch teardown task failed; left for Recover", "vmid", base.VMID, "err", err) + return + } + } + e.append(withState(base, OpSucceeded)) + e.logger.Info("restore-test: scratch guest torn down", "vmid", base.VMID) +} + +// waitRunning polls GuestStatus until the guest is running or the timeout elapses. The poll +// interval is 2s in production, but shrinks for short timeouts so it stays responsive. +func (e *Engine) waitRunning(ctx context.Context, vmid int, timeout time.Duration) error { + interval := 2 * time.Second + if timeout < 4*interval { + if interval = timeout / 4; interval < 10*time.Millisecond { + interval = 10 * time.Millisecond + } + } + deadline := time.Now().Add(timeout) + t := time.NewTicker(interval) + defer t.Stop() + for { + g, err := e.api.GuestStatus(ctx, vmid) + if err == nil && g.Status == "running" { + return nil + } + if time.Now().After(deadline) { + if err != nil { + return fmt.Errorf("reconcile: restore-test verify: guest %d not running within %s (last err: %w)", vmid, timeout, err) + } + return fmt.Errorf("reconcile: restore-test verify: guest %d not running within %s", vmid, timeout) + } + select { + case <-ctx.Done(): + return ctx.Err() + case <-t.C: + } + } +} + +// pickScratchVMID returns the lowest free VMID in [min,max], excluding the standing 9999 +// scratch and any in-use guest. ok=false when the band is fully occupied (the test is then +// skipped, never run out-of-band). +func pickScratchVMID(lxc []proxmox.Guest, min, max int) (int, bool) { + used := make(map[int]bool, len(lxc)) + for _, g := range lxc { + used[g.VMID] = true + } + for id := min; id <= max; id++ { + if id == 9999 || used[id] { + continue + } + return id, true + } + return 0, false +} + +// withLinkDown sets link_down=1 on a Proxmox netN config string, REPLACING any existing +// link_down token (never blind-concatenating, so a re-applied/pre-set value can't produce a +// malformed netN). +func withLinkDown(netN string) string { + parts := strings.Split(netN, ",") + out := parts[:0] + for _, p := range parts { + if p == "" || strings.HasPrefix(p, "link_down=") { + continue + } + out = append(out, p) + } + out = append(out, "link_down=1") + return strings.Join(out, ",") +} + +func bootTimeout(spec RestoreTestSpec) time.Duration { + if spec.BootTimeout > 0 { + return spec.BootTimeout + } + return DefaultBootTimeout +} + +func (e *Engine) scratchOpID(vmid int) string { + return "scratch-restore-" + strconv.Itoa(vmid) + "-" + nextSeq(&e.opSeq) +} + +// withState / withUPID build journal records from a base entry, preserving its identity + +// Scratch flag. +func withState(base JournalEntry, state OpState) JournalEntry { + base.State = state + base.At = time.Now().UTC() + return base +} +func withUPID(base JournalEntry, upid string, state OpState) JournalEntry { + base.UPID = upid + base.State = state + base.At = time.Now().UTC() + return base +} diff --git a/internal/reconcile/restoretest_test.go b/internal/reconcile/restoretest_test.go new file mode 100644 index 0000000..4d2e000 --- /dev/null +++ b/internal/reconcile/restoretest_test.go @@ -0,0 +1,227 @@ +package reconcile + +import ( + "context" + "encoding/json" + "errors" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/proxmox" +) + +// scratchCfg builds a fake GuestConfig with one net interface (so the link-down SetConfig +// step runs). +func scratchCfg() proxmox.GuestConfig { + return proxmox.GuestConfig{Extra: map[string]json.RawMessage{ + "net0": json.RawMessage(`"name=eth0,bridge=vmbr0,hwaddr=AA:BB:CC:DD:EE:FF,ip=dhcp"`), + }} +} + +func TestRunRestoreTest_PassAndTeardown(t *testing.T) { + api := &fakeAPI{cfg: map[int]proxmox.GuestConfig{990000: scratchCfg()}} // empty lxc → 990000 free; running default + e, j, q := newEngine(t, api, EmptyProvider{}) + defer q.Close() + + res := e.RunRestoreTest(context.Background(), RestoreTestSpec{ + Archive: "local:backup/x.tar.zst", RestoreStorage: "local-lvm", + ScratchMin: 990000, ScratchMax: 990009, SourceTier: "local", + }) + if res.Skipped || !res.Pass || res.Err != nil { + t.Fatalf("expected pass, got %+v", res) + } + if res.ScratchVMID != 990000 || res.Verified != "boot+running" { + t.Fatalf("result = %+v", res) + } + if len(api.restores) != 1 || api.restores[0].VMID != 990000 || api.restores[0].Archive != "local:backup/x.tar.zst" { + t.Fatalf("restore not issued correctly: %+v", api.restores) + } + // net link-down applied before boot. + foundLinkDown := false + for _, s := range api.sets { + if s.vmid == 990000 && s.params["net0"] != "" && contains2(s.params["net0"], "link_down=1") { + foundLinkDown = true + } + } + if !foundLinkDown { + t.Errorf("expected a net link-down SetConfig, got %+v", api.sets) + } + // teardown destroyed the scratch guest, and the journal entry is terminal (not in-flight). + if len(api.destroys) != 1 || api.destroys[0] != 990000 { + t.Fatalf("scratch not torn down: %+v", api.destroys) + } + if len(j.InFlight()) != 0 { + t.Errorf("scratch entry must be terminal after teardown: %+v", j.InFlight()) + } +} + +func TestRunRestoreTest_TeardownOnFailedVerify(t *testing.T) { + // Guest never reaches running → verify fails, but teardown MUST still run. + api := &fakeAPI{ + cfg: map[int]proxmox.GuestConfig{990000: scratchCfg()}, + status: map[int]proxmox.Guest{990000: {VMID: 990000, Status: "stopped"}}, + } + e, j, q := newEngine(t, api, EmptyProvider{}) + defer q.Close() + + res := e.RunRestoreTest(context.Background(), RestoreTestSpec{ + Archive: "vol", RestoreStorage: "local-lvm", + ScratchMin: 990000, ScratchMax: 990009, BootTimeout: 40 * time.Millisecond, + }) + if res.Pass || res.Err == nil { + t.Fatalf("expected a failed verify, got %+v", res) + } + if len(api.destroys) != 1 || api.destroys[0] != 990000 { + t.Fatalf("teardown MUST run even on a failed verify: destroys=%+v", api.destroys) + } + if len(j.InFlight()) != 0 { + t.Errorf("scratch entry must be terminal after teardown: %+v", j.InFlight()) + } +} + +func TestRunRestoreTest_RestoreFailureStillTearsDown(t *testing.T) { + api := &fakeAPI{restoreErr: errors.New("restore boom")} + e, j, q := newEngine(t, api, EmptyProvider{}) + defer q.Close() + + res := e.RunRestoreTest(context.Background(), RestoreTestSpec{ + Archive: "vol", RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009, + }) + if res.Pass || res.Err == nil { + t.Fatalf("expected restore failure, got %+v", res) + } + // Even though restore failed, the scratch entry was journaled BEFORE the restore, so + // teardown runs (idempotent — destroys the maybe-partial guest). + if len(api.destroys) != 1 { + t.Fatalf("teardown must run after a restore failure: %+v", api.destroys) + } + if len(j.InFlight()) != 0 { + t.Errorf("scratch entry must be terminal: %+v", j.InFlight()) + } +} + +func TestRunRestoreTest_FullBandSkips(t *testing.T) { + // Whole band occupied → skipped, never run / out-of-band. + var guests []proxmox.Guest + for id := 990000; id <= 990001; id++ { + guests = append(guests, proxmox.Guest{VMID: id}) + } + api := &fakeAPI{lxc: guests} + e, _, q := newEngine(t, api, EmptyProvider{}) + defer q.Close() + + res := e.RunRestoreTest(context.Background(), RestoreTestSpec{ + Archive: "vol", RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990001, + }) + if !res.Skipped { + t.Fatalf("full band must skip, got %+v", res) + } + if len(api.restores) != 0 || len(api.destroys) != 0 { + t.Errorf("a skipped test must not restore or destroy anything") + } +} + +func TestRunRestoreTest_InvalidBandErrors(t *testing.T) { + e, _, q := newEngine(t, &fakeAPI{}, EmptyProvider{}) + defer q.Close() + res := e.RunRestoreTest(context.Background(), RestoreTestSpec{Archive: "v", RestoreStorage: "s", ScratchMin: 0}) + if res.Err == nil { + t.Fatal("an invalid scratch band must error") + } +} + +func TestPickScratchVMID(t *testing.T) { + // excludes 9999 and in-use; lowest free. + got, ok := pickScratchVMID([]proxmox.Guest{{VMID: 990000}}, 990000, 990009) + if !ok || got != 990001 { + t.Errorf("pick = %d,%v want 990001,true", got, ok) + } + // full band. + full := []proxmox.Guest{{VMID: 990000}, {VMID: 990001}} + if _, ok := pickScratchVMID(full, 990000, 990001); ok { + t.Error("full band must return ok=false") + } +} + +func TestWithLinkDown(t *testing.T) { + got := withLinkDown("name=eth0,bridge=vmbr0,ip=dhcp") + if !contains2(got, "link_down=1") || !contains2(got, "name=eth0") { + t.Errorf("withLinkDown lost fields or didn't set link_down: %q", got) + } + // idempotent: an existing link_down is replaced, not duplicated. + got = withLinkDown("name=eth0,link_down=0,bridge=vmbr0") + if count(got, "link_down=") != 1 || !contains2(got, "link_down=1") { + t.Errorf("withLinkDown must replace an existing link_down (got %q)", got) + } +} + +// --- recover the leaked scratch guest (the headline crash-safety test) --- + +func TestRecover_LeakedScratchDestroyed(t *testing.T) { + // The scratch guest still exists at startup (agent crashed mid-test) → Recover destroys it. + api := &fakeAPI{lxc: []proxmox.Guest{{VMID: 990000, Status: "running"}}} + e, j, _ := newEngine(t, api, EmptyProvider{}) + if err := j.Append(JournalEntry{OpID: "scratch-990000-1", VMID: 990000, Kind: scratchKind, Scratch: true, State: OpTaskRunning, At: time.Now().UTC()}); err != nil { + t.Fatal(err) + } + res := e.Recover(context.Background()) + if res.ScratchDestroyed != 1 { + t.Fatalf("leaked scratch must be destroyed, got %+v", res) + } + if len(api.destroys) != 1 || api.destroys[0] != 990000 { + t.Fatalf("DestroyLXC not called for the leaked scratch: %+v", api.destroys) + } + if len(j.InFlight()) != 0 { + t.Errorf("resolved scratch entry must not be in-flight: %+v", j.InFlight()) + } +} + +func TestRecover_LeakedScratchAlreadyGone(t *testing.T) { + // Crash AFTER the destroy task but BEFORE the terminal record → guest already gone → + // idempotent clean (no destroy issued). + api := &fakeAPI{lxc: []proxmox.Guest{{VMID: 9001, Status: "stopped"}}} // 990000 absent + e, j, _ := newEngine(t, api, EmptyProvider{}) + j.Append(JournalEntry{OpID: "scratch-990000-1", VMID: 990000, Kind: scratchKind, Scratch: true, State: OpTaskRunning, At: time.Now().UTC()}) + res := e.Recover(context.Background()) + if res.ScratchClean != 1 || len(api.destroys) != 0 { + t.Fatalf("already-gone scratch must be clean with no destroy, got res=%+v destroys=%+v", res, api.destroys) + } + if len(j.InFlight()) != 0 { + t.Errorf("entry must be resolved: %+v", j.InFlight()) + } +} + +func TestRecover_LeakedScratchListUnreadable(t *testing.T) { + api := &fakeAPI{listErr: errors.New("api down")} + e, j, _ := newEngine(t, api, EmptyProvider{}) + j.Append(JournalEntry{OpID: "scratch-990000-1", VMID: 990000, Kind: scratchKind, Scratch: true, State: OpTaskRunning, At: time.Now().UTC()}) + res := e.Recover(context.Background()) + if res.Unresolved != 1 || len(j.InFlight()) != 1 { + t.Fatalf("unreadable list must leave the scratch in-flight for a later Recover, got res=%+v inflight=%d", res, len(j.InFlight())) + } + if len(api.destroys) != 0 { + t.Error("must not destroy when it can't confirm the guest exists") + } +} + +// small string helpers (avoid importing strings in the test for one call). +func contains2(s, sub string) bool { return indexOf(s, sub) >= 0 } +func count(s, sub string) int { + n, i := 0, 0 + for { + j := indexOf(s[i:], sub) + if j < 0 { + return n + } + n++ + i += j + len(sub) + } +} +func indexOf(s, sub string) int { + for i := 0; i+len(sub) <= len(s); i++ { + if s[i:i+len(sub)] == sub { + return i + } + } + return -1 +} diff --git a/internal/reconcile/state.go b/internal/reconcile/state.go index 3f9a128..9fd7b30 100644 --- a/internal/reconcile/state.go +++ b/internal/reconcile/state.go @@ -114,6 +114,14 @@ type GuestAPI interface { SetConfig(ctx context.Context, vmid int, params map[string]string) (string, error) // ResizeLXC grows a volume (grow-only; the planner never emits a shrink). Async → UPID. ResizeLXC(ctx context.Context, vmid int, disk, size string) (string, error) + // RestoreLXC restores an archive into a (fresh) vmid — the create path (slice 6). Async → UPID. + RestoreLXC(ctx context.Context, opts proxmox.RestoreLXCOptions) (string, error) + // DestroyLXC destroys a guest — the scratch-teardown primitive (slice 6). Async → UPID. + // Destructive-class; the engine only ever issues it for an agent-tagged scratch guest + // (benign by provenance) via the gate. + DestroyLXC(ctx context.Context, vmid int) (string, error) + // GuestStatus reads a single guest's current status (run-state poll during a restore-test). + GuestStatus(ctx context.Context, vmid int) (proxmox.Guest, error) WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error) // TaskStatusOnce is a single non-blocking task-status read — used by crash // recovery to learn the outcome of an op that was in flight when the agent died.