admin/felhom-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.6.0-rc1: slice 6 Phase A — backup + the self-restore-test (local target)

Browse Source 
The guest-level backup layer + the journaled self-restore-test (restore→boot→verify→
teardown) that closes "a backup you haven't restored isn't a backup". All benign
(reuses the slice-4 classifier/gate/journal; no new destructive class/crypto). Local
target only; PBS = Phase B. Restore to a NEW guest only. Backups crash-consistent.

- proxmox: DestroyLXC, VzdumpOptions.Notes (notes-template), LatestBackupVolID.
- reconcile: Engine.RunRestoreTest (journal Scratch entry BEFORE mutation; net link-down
  pre-boot; defer teardown always; benign gated destroy) + Recover extended to reap a
  leaked scratch guest (Scratch flag, special-cased before the UPID path; idempotent).
- internal/backup: runner (vzdump + archive resolve + bulk-gap = backup!=1) + cadence
  scheduler (4th daemon goroutine, default 24h) + in-memory report store.
- hub: Backup/RestoreTest filled; collector seams; cross-repo golden byte-identical +
  bidirectional key-set tests; hub handler logs a FAILED restore-test prominently.
- config BackupConfig (band 990000-990009 default); --selftest=backup / restore-test.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
authentik Default Admin
2026-06-09 13:49:39 +02:00
parent e548ab57fe
commit b527430ec7
23 changed files with 1727 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+51
View File
@@ -3,6 +3,57 @@
All notable changes to **felhom-agent** are recorded here. Update on every code
change that gets pushed.
## v0.6.0-rc1 — slice 6 Phase A: backup + the self-restore-test (local target) (2026-06-09)
Phase A of the backup/restore slice (doc 03 §8) — the agent's guest-level backup layer and
the **self-restore-test**, which closes "a backup you haven't restored isn't a backup".
Everything here is BENIGN (backup, restore-to-NEW, scratch teardown): reuses the slice-4
classifier/gate/journal — no new destructive class, no new crypto. Local target only; PBS is
Phase B. Restore is to a NEW guest only (no overwrite). Backups are crash-consistent only
(app-consistency needs the controller quiesce, slice 8) — marked so in the report.
### Added
- **proxmox** (`mutate.go`/`query.go`): `DestroyLXC` (DELETE …/lxc/{vmid}?purge=1&destroy-
unreferenced-disks=1 → UPID; the scratch-teardown primitive); `VzdumpOptions.Notes` →
`notes-template` (verified on PVE 9.2.2); `LatestBackupVolID` (resolve a produced archive
from the backup-storage listing — the task status carries no result volid).
- **reconcile self-restore-test** (`restoretest.go`) — `Engine.RunRestoreTest`: pick a free
scratch VMID (configured band, excludes 9999; full band → skip, never out-of-band) →
**journal a Scratch-owned entry BEFORE any mutation** → restore-to-new → benign net
**link-down** SetConfig (so the clone can't conflict with a running source's MAC/IP; this
is test-safety, NOT slice-7 identity reset) → boot → verify **reaches `running`** → ALWAYS
teardown (defer; benign `ClassGuestDestroy` + agent-tagged-scratch provenance, gated). Runs
on the scratch VMID's queue lane. Reuses the journal/gate; result feeds the report.
- **Crash-safe recovery** (`recover.go`): a Scratch journal entry is resolved by TEARDOWN,
not by re-checking the restore sub-task's UPID — special-cased BEFORE the generic path
(else the restore task's OK would mark it succeeded while the guest leaks). `Recover` now
destroys a leaked scratch guest (idempotent: already-gone → clean; list-unreadable → left
in-flight for a later pass). `JournalEntry.Scratch` flag; `RecoverResult.ScratchClean/
ScratchDestroyed`. GuestAPI gains `RestoreLXC`/`DestroyLXC`/`GuestStatus`.
- **`internal/backup` package**: `BackupRunner.Backup` (vzdump + archive/size resolve +
bulk-volume gap — a mountpoint is UNCOVERED unless it carries an explicit `backup=1`, so
an unset `backup=` is reported uncovered too, the safe DR direction); `PickRestoreCandidate`
(newest backup); an in-memory `Store` (latest-backup-per-target + latest-restore-test)
implementing the hub `BackupReporter`/`RestoreTestReporter` seams; a cadence `Scheduler`
(default 24h; the fourth daemon goroutine; disabled cleanly when off/misconfigured).
- **hub report** (`report.go`): filled the `Backup` + `RestoreTest` stubs (`PBSSnapshot`
stays a Phase-B stub); collector `BackupReporter`/`RestoreTestReporter` seams. Cross-repo
golden updated in BOTH repos (byte-identical) + bidirectional key-set tests for
`backups[0]`/`restore_tests[0]`. Hub `handler.go` parses + persists them (report_json; no
new columns) and logs a **FAILED restore-test prominently** (the loudest DR signal).
- **config** `BackupConfig` (local target, restore storage, restore-test cadence, scratch
VMID band 990000–990009 default) + accessors + env overlay + cadence-gated validation.
- **`--selftest=backup -vmid N`** (one-shot backup → print the Backup record) and
**`--selftest=restore-test [-archive volid]`** (Recover-then restore→boot→verify→teardown,
print the RestoreTest record). Standalone on the Proxmox host.
### Notes
- The daemon runs cleanly with the cadence off or misconfigured (logs + disables, never
crashes); a leaked scratch guest from a mid-test crash is reaped by `engine.Recover` on
restart. `go test -race` covers the new scheduler goroutine.
- Slice-3/4/5 exported surfaces, goldens, and adversarial tests intact. Version bumps to
**v0.6.0** when Phase B (PBS) lands.
## v0.5.1 — slice 5 live-validation prep: durable_id mis-id fix + re-mount UUID memory (2026-06-09)
Two correctness fixes surfaced while preparing the live USB validation on `demo-felhom`
CLAUDE.md
+5 -3
View File
@@ -15,7 +15,7 @@
- Module `gitea.dooplex.hu/admin/felhom-agent`; binary `felhom-agent` (`cmd/felhom-agent/`).
- **Pure Go stdlib + `golang.org/x/crypto` only** — no web frameworks.
- `go.mod` directive **go 1.25.0**; dep `golang.org/x/crypto v0.52.0` (declares go 1.25, will NOT build on Go 1.24). The **build server (192.168.0.180) runs go1.26.0** (upstream Go on PATH, backward-compatible). Build/run the agent there for live tests (same LAN as the demo host).
- Version: `version` var in `cmd/felhom-agent/main.go`, overridable via `-ldflags "-X main.version=<v>"`; `--version` flag. **Current: v0.5.0** (slice 5 complete: storage observe/report + watchdog + the host-root surface — mounts/SMART/grow/destructive gate). Bump on meaningful changes + add a CHANGELOG entry.
- Version: `version` var in `cmd/felhom-agent/main.go`, overridable via `-ldflags "-X main.version=<v>"`; `--version` flag. **Current: v0.6.0-rc1** (slice 6 Phase A: backup + the self-restore-test, local target; PBS = Phase B). Bump on meaningful changes + add a CHANGELOG entry.
## Layout
@@ -26,8 +26,9 @@ internal/log/ slog setup
internal/proxmox/ API-first Client + fenced root-CLI Privileged + UPID WaitTask
internal/authz/ operator signed-op verifier (SSHSIG); durable FileNonceStore
internal/hub/ daemon: HostReport collector + Bearer client + resilient Loop
internal/reconcile/ reconcile engine + reversibility gate + op journal + crash recovery
internal/reconcile/ reconcile engine + reversibility gate + op journal + crash recovery + restore-test
internal/storage/ storage-target observer + durable_id + fast-poll watchdog (slice 5)
internal/backup/ vzdump backup runner + restore-test scheduler + report store (slice 6)
```
## Proxmox model (the load-bearing rules)
@@ -54,7 +55,8 @@ Built in slices, all on `main`:
- **v0.4.0** — slice-4 **Phase B** (security core): the benign/destructive **classifier** (provenance + data-bearing, not by verb; scratch/same-txn provenance is agent-internal, never hub-sourced), the **reversibility gate** (destructive → `pending_signature` unless a verified, role-scoped, action-bound operator signature), the **signed-op consuming layer** over `internal/authz` (role-scoping per doc 04 §4, op-to-action binding, idempotency-by-nonce, audit), and the **crash-recovery consumer** (`Recover` over `InFlight()`, resume-or-rollback). The gate fronts the queue's executor (every mutation passes it). **Inert this slice** — no destructive deltas served until slice 10; the destructive path is classified, gated, and adversarially tested but not wired to live execution. `authz` surface untouched.
- **v0.5.0-rc1** — slice-5 **Phase A** (read-only, live): `internal/storage` — the `StorageTarget` wire contract (filled the slice-3 stub), `durable_id` derivation per type, the `Observer`, and the **storage watchdog** (third daemon goroutine; fast-poll → debounced out-of-band report on a known target's attach/disconnect). Hub ingest accepts/persists `storage_targets`; cross-repo golden byte-identical.
- **v0.5.0** — slice-5 **Phase B** (the host-root surface): the `HostOps` seam + `SudoHostOps` (systemd `.mount` units by fs-UUID, detach, SMART, lvs) behind a **strict argument validator** (the adversarial matrix is the headline security test — hostile UUID/path/device refused with zero exec); SMART (SATA+NVMe) + thin-pool metadata enrichment; the watchdog's benign **re-mount response** (off the poll path); the **disk-grow executor** (`pct resize`, grow-only, benign) and **destructive storage ops** through the slice-4 gate (target-scoped; built + tested, inert live); `--selftest=storage [-watch]`; `configs/felhom-agent.sudoers`.
- **Next: slice 6 (backup/restore orchestration)** — vzdump/PBS + the self-restore-test; the restore-overwrite executor the gate already guards.
- **v0.6.0-rc1** — slice-6 **Phase A** (backup + self-restore-test, local target): proxmox `DestroyLXC`/`Vzdump`-notes/`LatestBackupVolID`; `Engine.RunRestoreTest` (journaled scratch lifecycle: restore-to-new → net link-down → boot → verify running → defer teardown, all benign); `Recover` extended to reap a leaked scratch guest (Scratch journal flag, special-cased before the UPID path); `internal/backup` (runner + bulk-gap + cadence scheduler + report store); hub `Backup`/`RestoreTest` filled (cross-repo golden + hub logs a failed restore-test); `--selftest=backup`/`--selftest=restore-test`. Live-validated on demo-felhom.
- **Next: slice 6 Phase B (PBS)** — datastore on the USB, zero-knowledge key custody, restore-from-PBS, PBS integrity-verify (the lighter frequent check). Then slice 7 (provisioning + identity-reset + golden base, §9).
## Demo host (for live tests)
cmd/felhom-agent/main.go
+178 -10
View File
@@ -21,6 +21,7 @@ import (
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/authz"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/config"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
applog "gitea.dooplex.hu/admin/felhom-agent/internal/log"
@@ -31,7 +32,7 @@ import (
// version is the agent version. Overridable at build time with
// -ldflags "-X main.version=<v>"; defaults to the in-repo CHANGELOG version.
var version = "0.5.1"
var version = "0.6.0-rc1"
func main() {
var (
@@ -39,12 +40,14 @@ func main() {
selftest selftestFlag
vmid int
watch time.Duration
archive string
showVersion bool
)
flag.StringVar(&cfgPath, "config", envOr("FELHOM_AGENT_CONFIG", "/etc/felhom-agent/agent.json"), "path to the agent config file (JSON)")
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report to the hub; `storage` = observe storage targets (+ -watch for the live watchdog)")
flag.IntVar(&vmid, "vmid", 0, "guest VMID for --selftest=task (the reversible snapshot/rollback exercise)")
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup)")
flag.IntVar(&vmid, "vmid", 0, "guest VMID for --selftest=task|backup")
flag.DurationVar(&watch, "watch", 0, "for --selftest=storage: run the watchdog verbose for this duration (e.g. 3m) with the re-mount response live; 0 = observe pass only")
flag.StringVar(&archive, "archive", "", "for --selftest=restore-test: the backup volid to restore (default: newest on the configured local target)")
flag.BoolVar(&showVersion, "version", false, "print version and exit")
flag.Parse()
@@ -76,6 +79,10 @@ func main() {
os.Exit(runSelftestHub(context.Background(), cfg, logger))
case "storage":
os.Exit(runSelftestStorage(context.Background(), cfg, logger, watch))
case "backup":
os.Exit(runSelftestBackup(context.Background(), cfg, logger, vmid))
case "restore-test":
os.Exit(runSelftestRestoreTest(context.Background(), cfg, logger, archive))
}
}
@@ -178,7 +185,11 @@ func runDaemon(cfg config.Config, logger *slog.Logger) int {
hostReader := storage.NewProcHostReader()
hostOps := newHostOps(cfg, logger)
observer := storage.NewObserver(px, hostReader, hostOps, logger)
collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, cfg.Hub.HostID, version, logger)
// Backup + restore-test state store (slice 6): holds the latest backup-per-target +
// latest restore-test result; the collector reads it via the BackupReporter /
// RestoreTestReporter seams; the cadence scheduler writes it.
backupStore := backup.NewStore()
collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, backupStore, backupStore, cfg.Hub.HostID, version, logger)
loop := hub.NewLoop(collector, client, time.Duration(hcfg.PollSeconds)*time.Second, logger)
interval := time.Duration(hcfg.PollSeconds) * time.Second
@@ -256,20 +267,28 @@ func runDaemon(cfg config.Config, logger *slog.Logger) int {
})
// Crash recovery (doc 03 §10): resolve any op that was in flight when the agent
// last died BEFORE issuing new mutations. With an empty journal this is a no-op.
// last died BEFORE issuing new mutations. For slice 6 this is load-bearing — a
// restore-test scratch guest leaked by a mid-test crash is torn down here.
engine.Recover(ctx)
// Run reconcile, the hub loop, and the storage watchdog concurrently; any one
// returning ends the daemon (then ctx cancellation tears the others down).
errc := make(chan error, 3)
// Self-restore-test scheduler (slice 6): the fourth daemon goroutine. Runs the restore-
// test on the configured cadence (default 24h). Disabled cleanly when the cadence is off
// OR the scratch band / restore storage is misconfigured — the daemon still runs.
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, logger)
// Run reconcile, the hub loop, the storage watchdog, and the restore-test scheduler
// concurrently; any one returning ends the daemon (ctx cancellation tears down the rest).
errc := make(chan error, 4)
go func() { errc <- engine.Run(ctx, interval) }()
go func() { errc <- loop.Run(ctx) }()
go func() { errc <- watchdog.Run(ctx) }()
go func() { errc <- scheduler.Run(ctx) }()
err = <-errc
stop() // tear down the siblings on the first exit
<-errc // wait for the second
<-errc // wait for the third
<-errc // wait for the fourth
if err != nil && err != context.Canceled {
logger.Error("daemon: exited with error", "err", err)
return 1
@@ -277,6 +296,35 @@ func runDaemon(cfg config.Config, logger *slog.Logger) int {
return 0
}
// buildRestoreTestScheduler constructs the restore-test cadence scheduler from config. It
// disables the cadence (returns a scheduler that just waits) when the cadence is off or the
// scratch band / restore storage is invalid — a misconfig must not crash the daemon, and the
// machinery still works on-demand via --selftest=restore-test.
func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *reconcile.Engine, store *backup.Store, logger *slog.Logger) *backup.Scheduler {
cadence := cfg.Backup.RestoreTestCadence()
if cadence > 0 {
if err := cfg.Backup.ValidateForRestoreTest(); err != nil {
logger.Warn("daemon: restore-test cadence disabled (config invalid)", "err", err)
cadence = 0
}
}
min, max := cfg.Backup.ScratchBand()
runner := backup.NewBackupRunner(px, cfg.Backup.LocalBackupTarget, "", "felhom restore-test", logger)
return backup.NewScheduler(backup.SchedulerOptions{
Runner: engine,
Pick: runner.PickRestoreCandidate,
Store: store,
Spec: reconcile.RestoreTestSpec{
RestoreStorage: cfg.Backup.RestoreStorage,
ScratchMin: min,
ScratchMax: max,
SourceTier: "local",
},
Cadence: cadence,
Logger: logger,
})
}
// reconcileJournalPath chooses the op-journal path: a `journal.log` sibling of the
// configured nonce store (both are durable agent state), falling back to the standard
// host state dir when the nonce store is unset.
@@ -338,7 +386,7 @@ func runSelftestHub(ctx context.Context, cfg config.Config, logger *slog.Logger)
return 1
}
observer := storage.NewObserver(px, storage.NewProcHostReader(), newHostOps(cfg, logger), logger)
collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, cfg.Hub.HostID, version, logger)
collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, nil, nil, cfg.Hub.HostID, version, logger)
ctx, cancel := context.WithTimeout(ctx, 60*time.Second)
defer cancel()
@@ -460,6 +508,122 @@ func smartCounters(s hub.SmartSummary) string {
return " " + strings.Join(parts, " ")
}
// runSelftestBackup runs one vzdump of -vmid to the configured local backup target and
// prints the resulting Backup record. Standalone (no hub). Live + benign.
func runSelftestBackup(ctx context.Context, cfg config.Config, logger *slog.Logger, vmid int) int {
if err := cfg.Validate(); err != nil {
fmt.Fprintln(os.Stderr, "selftest: proxmox not configured:", err)
return 1
}
if vmid == 0 {
fmt.Fprintln(os.Stderr, "selftest=backup requires -vmid N")
return 2
}
if cfg.Backup.LocalBackupTarget == "" {
fmt.Fprintln(os.Stderr, "selftest=backup requires backup.local_backup_target in config (a content=backup storage)")
return 2
}
px, err := newProxmoxClient(cfg)
if err != nil {
fmt.Fprintln(os.Stderr, "selftest: proxmox client:", err)
return 1
}
ctx, cancel := context.WithTimeout(ctx, 30*time.Minute)
defer cancel()
fmt.Printf("=== felhom-agent %s selftest=backup (vmid %d → %s) ===\n", version, vmid, cfg.Backup.LocalBackupTarget)
runner := backup.NewBackupRunner(px, cfg.Backup.LocalBackupTarget, "", "felhom selftest", logger)
rec, err := runner.Backup(ctx, vmid)
printJSON("backup record", rec)
if err != nil {
fmt.Fprintln(os.Stderr, " [FAIL] backup:", err)
return 1
}
fmt.Printf("=== selftest=backup OK (crash-consistent=%v, archive=%s) ===\n", rec.CrashConsistent, rec.Archive)
return 0
}
// runSelftestRestoreTest runs one self-restore-test (restore → net-link-down → boot → verify
// running → teardown) of -archive (or the newest backup on the local target) into a scratch
// guest. Standalone (no hub). Runs engine.Recover first so a leaked scratch from a prior
// crashed test is reaped before this run.
func runSelftestRestoreTest(ctx context.Context, cfg config.Config, logger *slog.Logger, archive string) int {
if err := cfg.Validate(); err != nil {
fmt.Fprintln(os.Stderr, "selftest: proxmox not configured:", err)
return 1
}
if err := cfg.Backup.ValidateForRestoreTest(); err != nil {
fmt.Fprintln(os.Stderr, "selftest=restore-test:", err)
return 2
}
px, err := newProxmoxClient(cfg)
if err != nil {
fmt.Fprintln(os.Stderr, "selftest: proxmox client:", err)
return 1
}
ctx, stop := signal.NotifyContext(ctx, os.Interrupt, syscall.SIGTERM)
defer stop()
// Build a journaled engine so a leaked scratch from a prior crash is recoverable.
queue := reconcile.NewQueue()
defer queue.Close()
var journal *reconcile.Journal
if jp := reconcileJournalPath(cfg); jp != "" {
if err := os.MkdirAll(filepath.Dir(jp), 0o700); err == nil {
if j, err := reconcile.OpenJournal(jp); err == nil {
journal = j
defer journal.Close()
}
}
}
gate := reconcile.NewGate(nil, cfg.Hub.HostID, reconcile.SlogAudit{Logger: logger}, logger)
engine := reconcile.NewEngine(reconcile.EngineOptions{
API: px, Queue: queue, Journal: journal, Gate: gate, HostID: cfg.Hub.HostID, Logger: logger,
})
fmt.Printf("=== felhom-agent %s selftest=restore-test ===\n", version)
fmt.Println(" --- recover: reaping any leaked scratch from a prior crashed test ---")
rec := engine.Recover(ctx)
fmt.Printf(" recover: examined=%d scratch_destroyed=%d scratch_clean=%d\n", rec.Examined, rec.ScratchDestroyed, rec.ScratchClean)
if archive == "" {
runner := backup.NewBackupRunner(px, cfg.Backup.LocalBackupTarget, "", "", logger)
archive, err = runner.PickRestoreCandidate(ctx)
if err != nil {
fmt.Fprintln(os.Stderr, " [FAIL] pick backup:", err)
return 1
}
if archive == "" {
fmt.Fprintln(os.Stderr, " [FAIL] no backup available on", cfg.Backup.LocalBackupTarget, "(run --selftest=backup first)")
return 1
}
}
min, max := cfg.Backup.ScratchBand()
fmt.Printf(" restoring %s into scratch band [%d,%d] on %s …\n", archive, min, max, cfg.Backup.RestoreStorage)
res := engine.RunRestoreTest(ctx, reconcile.RestoreTestSpec{
Archive: archive, RestoreStorage: cfg.Backup.RestoreStorage,
ScratchMin: min, ScratchMax: max, SourceTier: "local",
})
printJSON("restore-test record", backup.ToHubRestoreTest(res, time.Now().UTC()))
if res.Skipped {
fmt.Println("=== selftest=restore-test SKIPPED (no free scratch VMID in band) ===")
return 0
}
if res.Err != nil || !res.Pass {
fmt.Fprintf(os.Stderr, " [FAIL] restore-test (scratch %d): %v\n", res.ScratchVMID, res.Err)
return 1
}
fmt.Printf("=== selftest=restore-test OK (scratch %d restored+booted+verified+torn-down in %s) ===\n", res.ScratchVMID, res.Duration.Round(time.Second))
return 0
}
// printJSON prints a labelled, indented JSON dump (best-effort) to stdout.
func printJSON(label string, v any) {
if b, err := json.MarshalIndent(v, " ", " "); err == nil {
fmt.Printf(" --- %s ---\n %s\n", label, string(b))
}
}
// runSelftestRead loads config, builds the API client, and runs the read-only
// queries against the live host, printing a short health report. It mutates
// nothing. Missing/invalid config is reported cleanly (no panic).
@@ -748,8 +912,12 @@ func (f *selftestFlag) Set(v string) error {
f.mode = "hub"
case "storage":
f.mode = "storage"
case "backup":
f.mode = "backup"
case "restore-test":
f.mode = "restore-test"
default:
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage)", v)
return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test)", v)
}
return nil
}
configs/agent.example.json
+7
View File
@@ -36,5 +36,12 @@
"timeout_seconds": 30,
"ca_file": ""
},
"backup": {
"local_backup_target": "local",
"restore_storage": "local-lvm",
"restore_test_cadence_seconds": 0,
"scratch_vmid_min": 990000,
"scratch_vmid_max": 990009
},
"log_level": "info"
}
internal/backup/backup_test.go
+199
View File
@@ -0,0 +1,199 @@
package backup
import (
"context"
"encoding/json"
"errors"
"io"
"log/slog"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
)
func quiet() *slog.Logger { return slog.New(slog.NewTextHandler(io.Discard, nil)) }
// fakeBackupAPI is a synthetic BackupAPI.
type fakeBackupAPI struct {
vzdumpUPID string
vzdumpErr error
waitErr error
cfg proxmox.GuestConfig
cfgErr error
content []proxmox.StorageContent
contentErr error
vzdumps []proxmox.VzdumpOptions
}
func (f *fakeBackupAPI) Vzdump(_ context.Context, o proxmox.VzdumpOptions) (string, error) {
f.vzdumps = append(f.vzdumps, o)
return f.vzdumpUPID, f.vzdumpErr
}
func (f *fakeBackupAPI) WaitTask(_ context.Context, _ string, _ proxmox.WaitOptions) (proxmox.TaskStatus, error) {
return proxmox.TaskStatus{Status: "stopped", ExitStatus: "OK"}, f.waitErr
}
func (f *fakeBackupAPI) GuestConfig(_ context.Context, _ int) (proxmox.GuestConfig, error) {
return f.cfg, f.cfgErr
}
func (f *fakeBackupAPI) StorageContent(_ context.Context, _ string) ([]proxmox.StorageContent, error) {
return f.content, f.contentErr
}
// guestCfgWithMounts builds a GuestConfig whose Extra carries the given mpN strings.
func guestCfgWithMounts(mps map[string]string) proxmox.GuestConfig {
extra := map[string]json.RawMessage{}
for k, v := range mps {
b, _ := json.Marshal(v)
extra[k] = b
}
return proxmox.GuestConfig{Extra: extra}
}
func TestBackup_SuccessResolvesArchiveAndBulkGap(t *testing.T) {
api := &fakeBackupAPI{
vzdumpUPID: "UPID:vzdump:1",
cfg: guestCfgWithMounts(map[string]string{
"mp0": "local-lvm:8,mp=/mnt/bulk,backup=0", // explicit opt-out → uncovered
"mp1": "local-lvm:4,mp=/mnt/db,backup=1", // covered
"mp2": "local-lvm:2,mp=/mnt/scratch", // UNSET → uncovered (opt-in default)
}),
content: []proxmox.StorageContent{
{VolID: "local:backup/old-9001.tar.zst", Content: "backup", VMID: 9001, Size: 100, CTime: 100},
{VolID: "local:backup/new-9001.tar.zst", Content: "backup", VMID: 9001, Size: 524288000, CTime: 200},
{VolID: "local:backup/other-9002.tar.zst", Content: "backup", VMID: 9002, Size: 7, CTime: 999},
},
}
r := NewBackupRunner(api, "local", "", "felhom test", quiet())
rec, err := r.Backup(context.Background(), 9001)
if err != nil {
t.Fatalf("Backup: %v", err)
}
if !rec.Success || !rec.CrashConsistent {
t.Errorf("record = %+v, want success + crash_consistent", rec)
}
if rec.Archive != "local:backup/new-9001.tar.zst" || rec.SizeBytes != 524288000 {
t.Errorf("resolved wrong archive/size: %+v", rec)
}
if rec.Mode != string(proxmox.ModeSnapshot) {
t.Errorf("mode = %q, want snapshot (default)", rec.Mode)
}
// Bulk gap: mp0 (backup=0) AND mp2 (unset) are uncovered; mp1 (backup=1) is NOT.
if got := rec.UncoveredVolumes; len(got) != 2 || !has(got, "/mnt/bulk") || !has(got, "/mnt/scratch") {
t.Errorf("uncovered = %v, want [/mnt/bulk /mnt/scratch] (unset is uncovered too)", got)
}
if has(rec.UncoveredVolumes, "/mnt/db") {
t.Error("backup=1 mountpoint must NOT be reported uncovered")
}
// Notes-template threaded through.
if len(api.vzdumps) != 1 || api.vzdumps[0].Notes != "felhom test" {
t.Errorf("vzdump opts = %+v", api.vzdumps)
}
}
func TestBackup_VzdumpFailureReturnsFailedRecord(t *testing.T) {
api := &fakeBackupAPI{vzdumpErr: errors.New("vzdump boom")}
r := NewBackupRunner(api, "local", "", "", quiet())
rec, err := r.Backup(context.Background(), 9001)
if err == nil {
t.Fatal("expected error")
}
if rec.Success || rec.Error == "" {
t.Errorf("failed backup must produce a Success=false record with an Error: %+v", rec)
}
}
func TestPickRestoreCandidate_NewestOrEmpty(t *testing.T) {
api := &fakeBackupAPI{content: []proxmox.StorageContent{
{VolID: "a", Content: "backup", CTime: 10},
{VolID: "b", Content: "backup", CTime: 99},
{VolID: "iso", Content: "iso", CTime: 999}, // not a backup → ignored
}}
r := NewBackupRunner(api, "local", "", "", quiet())
vol, err := r.PickRestoreCandidate(context.Background())
if err != nil || vol != "b" {
t.Fatalf("pick = %q,%v want newest 'b'", vol, err)
}
// no backups → "".
api.content = []proxmox.StorageContent{{VolID: "iso", Content: "iso"}}
if vol, _ := r.PickRestoreCandidate(context.Background()); vol != "" {
t.Errorf("no backup → empty, got %q", vol)
}
}
// --- scheduler ---
type fakeRTRunner struct {
res reconcile.RestoreTestResult
runs int
}
func (f *fakeRTRunner) RunRestoreTest(_ context.Context, _ reconcile.RestoreTestSpec) reconcile.RestoreTestResult {
f.runs++
return f.res
}
func TestScheduler_TickRunsAndRecords(t *testing.T) {
store := NewStore()
rt := &fakeRTRunner{res: reconcile.RestoreTestResult{Archive: "vol", Pass: true, Verified: "boot+running", Duration: time.Second}}
s := NewScheduler(SchedulerOptions{
Runner: rt,
Pick: func(context.Context) (string, error) { return "vol", nil },
Store: store,
Cadence: time.Hour,
Logger: quiet(),
})
s.tick(context.Background())
if rt.runs != 1 {
t.Fatalf("tick should run the restore-test once, got %d", rt.runs)
}
got := store.RestoreTests(context.Background())
if len(got) != 1 || !got[0].Pass || got[0].SourceArchive != "vol" {
t.Fatalf("store should have the recorded result: %+v", got)
}
}
func TestScheduler_TickNoBackupNoOp(t *testing.T) {
rt := &fakeRTRunner{}
s := NewScheduler(SchedulerOptions{
Runner: rt, Pick: func(context.Context) (string, error) { return "", nil },
Store: NewStore(), Cadence: time.Hour, Logger: quiet(),
})
s.tick(context.Background())
if rt.runs != 0 {
t.Errorf("no backup available → no restore-test run, got %d", rt.runs)
}
}
func TestScheduler_DisabledRunReturnsOnCancel(t *testing.T) {
s := NewScheduler(SchedulerOptions{Cadence: 0, Logger: quiet()})
ctx, cancel := context.WithCancel(context.Background())
done := make(chan error, 1)
go func() { done <- s.Run(ctx) }()
cancel()
select {
case err := <-done:
if err != nil {
t.Fatalf("disabled scheduler Run should return nil on cancel, got %v", err)
}
case <-time.After(time.Second):
t.Fatal("disabled scheduler did not return on cancel")
}
}
func TestStore_Reporters(t *testing.T) {
s := NewStore()
if len(s.Backups(context.Background())) != 0 || len(s.RestoreTests(context.Background())) != 0 {
t.Fatal("empty store must report empty (non-nil) slices")
}
}
func has(ss []string, want string) bool {
for _, s := range ss {
if s == want {
return true
}
}
return false
}
internal/backup/doc.go
+23
View File
@@ -0,0 +1,23 @@
// Package backup is the guest-level backup + self-restore-test layer (doc 03 §8, slice 6
// Phase A). It orchestrates a crash-consistent vzdump to a LOCAL target, resolves the
// produced archive, and drives the self-restore-test (restore → boot → verify → teardown)
// through the reconcile engine so it inherits the journal / per-guest serialization /
// crash-safe recovery.
//
// Everything here is BENIGN (backup, restore-to-NEW, scratch teardown): it reuses the
// slice-4 classifier/gate/journal via reconcile — no new destructive class, no new crypto.
// Restore is to a NEW guest only (no overwrite this slice). PBS / offsite / zero-knowledge
// is Phase B.
//
// Layout:
// - runner.go — BackupRunner: vzdump + archive-volid/size resolve + the bulk-volume gap;
// restore-candidate picker.
// - store.go — in-memory latest-backup-per-target + latest-restore-test, implementing
// the hub BackupReporter / RestoreTestReporter seams (point-in-time state
// the collector reads; re-populated each cadence/selftest run).
// - schedule.go — the restore-test cadence goroutine (default 24h; disabled when 0).
//
// hub does NOT import this package (the report types live in hub; this package imports hub
// for them, mirroring the slice-5 storage seam). This package may import reconcile + hub +
// proxmox (acyclic).
package backup
internal/backup/runner.go
+198
View File
@@ -0,0 +1,198 @@
package backup
import (
"context"
"fmt"
"log/slog"
"sort"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
)
// BackupAPI is the read+backup proxmox surface the runner needs. *proxmox.Client satisfies it.
type BackupAPI interface {
Vzdump(ctx context.Context, opts proxmox.VzdumpOptions) (string, error)
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig, error)
StorageContent(ctx context.Context, store string) ([]proxmox.StorageContent, error)
}
// BackupRunner orchestrates a crash-consistent vzdump to a local target and reports the
// result (incl. the bulk-volume gap). An agent-initiated vzdump is crash-consistent only
// (no fsfreeze); the report says so.
type BackupRunner struct {
api BackupAPI
target string // backup storage (content=backup)
mode proxmox.BackupMode // default ModeSnapshot
notes string // optional notes-template
logger *slog.Logger
now func() time.Time
}
// NewBackupRunner builds a runner. mode defaults to snapshot (works for a stopped guest and
// for lvm-thin); the caller may pass ModeStop for storages without snapshot support.
func NewBackupRunner(api BackupAPI, target string, mode proxmox.BackupMode, notes string, logger *slog.Logger) *BackupRunner {
if mode == "" {
mode = proxmox.ModeSnapshot
}
if logger == nil {
logger = slog.Default()
}
return &BackupRunner{api: api, target: target, mode: mode, notes: notes, logger: logger, now: func() time.Time { return time.Now().UTC() }}
}
// Backup runs one vzdump of vmid to the local target and returns the report record. A
// failure is returned BOTH as an error and as a Backup{Success:false,...} so the caller can
// record the failed attempt.
func (r *BackupRunner) Backup(ctx context.Context, vmid int) (hub.Backup, error) {
start := r.now()
rec := hub.Backup{
TargetID: r.target,
VMID: vmid,
Mode: string(r.mode),
CrashConsistent: true, // always, this slice (no controller quiesce)
StartedAt: start.Format(time.RFC3339),
}
if r.target == "" {
rec.Error = "no backup target configured"
return rec, fmt.Errorf("backup: %s", rec.Error)
}
// Bulk-volume gap: which mountpoints the vzdump will EXCLUDE (best-effort; a config-read
// failure just leaves the gap unknown, never fails the backup).
if cfg, err := r.api.GuestConfig(ctx, vmid); err == nil {
rec.UncoveredVolumes = uncoveredMountpoints(cfg.MountPoints())
} else {
r.logger.Warn("backup: could not read guest config for bulk-gap", "vmid", vmid, "err", err)
rec.UncoveredVolumes = []string{}
}
upid, err := r.api.Vzdump(ctx, proxmox.VzdumpOptions{
VMID: vmid, Storage: r.target, Mode: r.mode, Notes: r.notes,
})
if err != nil {
rec.Error = err.Error()
rec.DurationSeconds = time.Since(start).Seconds()
return rec, fmt.Errorf("backup: vzdump vmid %d: %w", vmid, err)
}
if upid != "" {
if _, err := r.api.WaitTask(ctx, upid, proxmox.WaitOptions{Timeout: 30 * time.Minute}); err != nil {
rec.Error = err.Error()
rec.DurationSeconds = time.Since(start).Seconds()
return rec, fmt.Errorf("backup: vzdump task vmid %d: %w", vmid, err)
}
}
// Resolve the produced archive (volid + size) — the task status carries no result volid.
vol, size, err := r.latestArchive(ctx, vmid)
if err != nil {
rec.Error = fmt.Sprintf("backup succeeded but archive not resolved: %v", err)
rec.DurationSeconds = time.Since(start).Seconds()
return rec, fmt.Errorf("backup: resolve archive vmid %d: %w", vmid, err)
}
rec.Archive = vol
rec.SizeBytes = size
rec.Success = true
rec.DurationSeconds = time.Since(start).Seconds()
r.logger.Info("backup: completed", "vmid", vmid, "target", r.target, "archive", vol, "size_bytes", size,
"uncovered_volumes", len(rec.UncoveredVolumes))
return rec, nil
}
// PickRestoreCandidate returns the newest backup archive on the target (any guest), or ""
// when there is none — the restore-test then no-ops cleanly.
func (r *BackupRunner) PickRestoreCandidate(ctx context.Context) (string, error) {
contents, err := r.api.StorageContent(ctx, r.target)
if err != nil {
return "", err
}
var best string
var bestCTime int64 = -1
for _, e := range contents {
if e.Content == "backup" && e.CTime > bestCTime {
bestCTime, best = e.CTime, e.VolID
}
}
return best, nil
}
// latestArchive finds the newest backup archive volid + size for vmid on the target.
func (r *BackupRunner) latestArchive(ctx context.Context, vmid int) (string, int64, error) {
contents, err := r.api.StorageContent(ctx, r.target)
if err != nil {
return "", 0, err
}
var vol string
var size, bestCTime int64 = 0, -1
for _, e := range contents {
if e.Content == "backup" && e.VMID == vmid && e.CTime > bestCTime {
bestCTime, vol, size = e.CTime, e.VolID, e.Size
}
}
if vol == "" {
return "", 0, fmt.Errorf("no backup archive found for vmid %d on %s", vmid, r.target)
}
return vol, size, nil
}
// uncoveredMountpoints returns the mountpoint paths the guest vzdump EXCLUDES. LXC mount
// points are OPT-IN to vzdump: a mpN with `backup=1` is covered; ANY other state — the
// `backup=` token absent OR `backup=0` — is excluded. We deliberately treat unset as
// uncovered (the safe DR direction: never imply an unprotected volume is backed up).
func uncoveredMountpoints(mps map[string]string) []string {
var out []string
for key, cfg := range mps {
if mountpointCovered(cfg) {
continue
}
out = append(out, mountpointLabel(key, cfg))
}
sort.Strings(out)
if out == nil {
return []string{}
}
return out
}
// mountpointCovered reports whether a mpN config string is included in the vzdump — true
// ONLY when it carries an explicit `backup=1`.
func mountpointCovered(cfg string) bool {
for _, tok := range strings.Split(cfg, ",") {
if v, ok := strings.CutPrefix(tok, "backup="); ok {
return v == "1"
}
}
return false // no backup= token → opt-out by default → not covered
}
// mountpointLabel prefers the mp=/path token, falling back to the mpN key.
func mountpointLabel(key, cfg string) string {
for _, tok := range strings.Split(cfg, ",") {
if p, ok := strings.CutPrefix(tok, "mp="); ok && p != "" {
return p
}
}
return key
}
// ToHubRestoreTest maps a reconcile restore-test result to the hub wire record (the backup
// package owns the reconcile→hub mapping so reconcile need not import hub for the result).
func ToHubRestoreTest(res reconcile.RestoreTestResult, testedAt time.Time) hub.RestoreTest {
rt := hub.RestoreTest{
SourceArchive: res.Archive,
SourceTier: res.SourceTier,
ScratchVMID: res.ScratchVMID,
Pass: res.Pass,
Verified: res.Verified,
TestedAt: testedAt.Format(time.RFC3339),
DurationSeconds: res.Duration.Seconds(),
}
if res.Err != nil {
rt.Error = res.Err.Error()
}
return rt
}
internal/backup/schedule.go
+111
View File
@@ -0,0 +1,111 @@
package backup
import (
"context"
"log/slog"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
)
// RestoreTestRunner is the reconcile-engine seam the scheduler drives (*reconcile.Engine
// satisfies it). Kept narrow so the scheduler is unit-testable with a fake.
type RestoreTestRunner interface {
RunRestoreTest(ctx context.Context, spec reconcile.RestoreTestSpec) reconcile.RestoreTestResult
}
// CandidatePicker resolves the archive volid to restore-test (newest backup), or "" when
// there is none yet (the tick then no-ops).
type CandidatePicker func(ctx context.Context) (string, error)
// Scheduler runs the self-restore-test on an agent-internal cadence. It is the fourth daemon
// goroutine; it does real restore→boot→destroy, so it only runs when the cadence is enabled
// AND a valid scratch band is configured (validated by the caller before construction).
type Scheduler struct {
runner RestoreTestRunner
pick CandidatePicker
store *Store
spec reconcile.RestoreTestSpec // archive is filled per-tick
cadence time.Duration
logger *slog.Logger
now func() time.Time
}
// SchedulerOptions configures a Scheduler.
type SchedulerOptions struct {
Runner RestoreTestRunner
Pick CandidatePicker
Store *Store
Spec reconcile.RestoreTestSpec // RestoreStorage, ScratchMin/Max, SourceTier, BootTimeout
Cadence time.Duration // 0 → disabled
Logger *slog.Logger
}
// NewScheduler builds a Scheduler.
func NewScheduler(opts SchedulerOptions) *Scheduler {
logger := opts.Logger
if logger == nil {
logger = slog.Default()
}
return &Scheduler{
runner: opts.Runner,
pick: opts.Pick,
store: opts.Store,
spec: opts.Spec,
cadence: opts.Cadence,
logger: logger,
now: func() time.Time { return time.Now().UTC() },
}
}
// Run fires a restore-test on the cadence until ctx is cancelled. A 0 cadence disables it
// (the goroutine just waits for shutdown). It does NOT fire immediately on start (a restore
// is heavy; the first runs one interval in) — on-demand runs use the selftest harness.
// Returns nil on ctx cancellation.
func (s *Scheduler) Run(ctx context.Context) error {
if s.cadence <= 0 || s.runner == nil || s.pick == nil {
s.logger.Info("backup: restore-test cadence disabled")
<-ctx.Done()
return nil
}
s.logger.Info("backup: restore-test scheduler starting", "cadence", s.cadence)
t := time.NewTicker(s.cadence)
defer t.Stop()
for {
select {
case <-ctx.Done():
s.logger.Info("backup: restore-test scheduler shutting down", "reason", ctx.Err())
return nil
case <-t.C:
s.tick(ctx)
}
}
}
// tick runs one scheduled restore-test: pick a backup → run → record. No-ops cleanly when
// no backup exists yet. Deterministic given s.now — tests call it directly.
func (s *Scheduler) tick(ctx context.Context) {
archive, err := s.pick(ctx)
if err != nil {
s.logger.Warn("backup: restore-test could not pick a candidate; skipping", "err", err)
return
}
if archive == "" {
s.logger.Info("backup: restore-test skipped; no backup available yet")
return
}
spec := s.spec
spec.Archive = archive
res := s.runner.RunRestoreTest(ctx, spec)
if res.Skipped {
return // already logged by the engine (no free scratch VMID)
}
rt := ToHubRestoreTest(res, s.now())
s.store.RecordRestoreTest(rt)
if rt.Pass {
s.logger.Info("backup: scheduled restore-test passed", "archive", rt.SourceArchive, "duration_s", rt.DurationSeconds)
} else {
// A failing restore-test is the loudest DR signal there is.
s.logger.Error("backup: scheduled restore-test FAILED", "archive", rt.SourceArchive, "err", rt.Error)
}
}
internal/backup/store.go
+61
View File
@@ -0,0 +1,61 @@
package backup
import (
"context"
"sync"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// Store holds the agent's LATEST backup result per target and the latest restore-test
// result — the point-in-time state the host-report surfaces. It is updated by the backup
// runner + the restore-test scheduler/selftest and read by the collector via the hub
// BackupReporter / RestoreTestReporter seams. In-memory (lost on restart; the cadence
// re-populates) and mutex-guarded for the concurrent collector vs scheduler access.
type Store struct {
mu sync.Mutex
byTarget map[string]hub.Backup // latest backup per target id
lastTest *hub.RestoreTest
}
// NewStore builds an empty Store.
func NewStore() *Store {
return &Store{byTarget: map[string]hub.Backup{}}
}
// RecordBackup stores the latest backup for its target.
func (s *Store) RecordBackup(b hub.Backup) {
s.mu.Lock()
defer s.mu.Unlock()
s.byTarget[b.TargetID] = b
}
// RecordRestoreTest stores the latest restore-test result.
func (s *Store) RecordRestoreTest(r hub.RestoreTest) {
s.mu.Lock()
defer s.mu.Unlock()
cp := r
s.lastTest = &cp
}
// Backups implements hub.BackupReporter — the latest backup per target (stable order by
// target id is not guaranteed; the hub does not depend on order).
func (s *Store) Backups(context.Context) []hub.Backup {
s.mu.Lock()
defer s.mu.Unlock()
out := make([]hub.Backup, 0, len(s.byTarget))
for _, b := range s.byTarget {
out = append(out, b)
}
return out
}
// RestoreTests implements hub.RestoreTestReporter — the latest restore-test result (0 or 1).
func (s *Store) RestoreTests(context.Context) []hub.RestoreTest {
s.mu.Lock()
defer s.mu.Unlock()
if s.lastTest == nil {
return []hub.RestoreTest{}
}
return []hub.RestoreTest{*s.lastTest}
}
internal/config/config.go
+72
View File
@@ -27,9 +27,73 @@ type Config struct {
Authz AuthzConfig `json:"authz"`
Hub HubConfig `json:"hub"`
Storage StorageConfig `json:"storage"`
Backup BackupConfig `json:"backup"`
LogLevel string `json:"log_level"` // debug|info|warn|error (default info)
}
// BackupConfig tunes the slice-6 backup + self-restore-test layer. The restore-test runs on
// an agent-internal cadence (no hub policy needed — it's self-validation); the backup
// schedule/retention/target-selection policy is hub-manifest-owned and unfed until slice 10.
type BackupConfig struct {
// LocalBackupTarget is the vzdump storage (content=backup) backups go to, e.g. "local".
LocalBackupTarget string `json:"local_backup_target"`
// RestoreStorage is where a restore-test's restored rootfs lands, e.g. "local-lvm".
RestoreStorage string `json:"restore_storage"`
// RestoreTestCadenceSeconds is the self-restore-test interval; 0 → default (24h).
// Set negative to DISABLE the automatic cadence (on-demand selftest still works).
RestoreTestCadenceSeconds int `json:"restore_test_cadence_seconds"`
// ScratchVMIDMin/Max bound the throwaway restore-test scratch-guest VMID band. The
// restore-test refuses to run unless this is a valid band (min>0, max>=min); 9999 is
// always excluded. Defaults to 990000–990009.
ScratchVMIDMin int `json:"scratch_vmid_min"`
ScratchVMIDMax int `json:"scratch_vmid_max"`
}
// Default scratch VMID band + restore-test cadence.
const (
defaultScratchVMIDMin = 990000
defaultScratchVMIDMax = 990009
defaultRestoreTestCadence = 24 * time.Hour
)
// RestoreTestCadence returns the configured restore-test interval: a positive value as-is,
// 0 → 24h default, negative → 0 (disabled).
func (b BackupConfig) RestoreTestCadence() time.Duration {
switch {
case b.RestoreTestCadenceSeconds > 0:
return time.Duration(b.RestoreTestCadenceSeconds) * time.Second
case b.RestoreTestCadenceSeconds < 0:
return 0 // disabled
default:
return defaultRestoreTestCadence
}
}
// ScratchBand returns the effective [min,max] scratch VMID band (defaults applied).
func (b BackupConfig) ScratchBand() (min, max int) {
min, max = b.ScratchVMIDMin, b.ScratchVMIDMax
if min == 0 && max == 0 {
return defaultScratchVMIDMin, defaultScratchVMIDMax
}
return min, max
}
// ValidateForRestoreTest checks the scratch band is usable. Called only when the restore-test
// cadence is enabled (so a host that never restore-tests needn't configure a band).
func (b BackupConfig) ValidateForRestoreTest() error {
min, max := b.ScratchBand()
if min <= 0 || max < min {
return fmt.Errorf("config: backup.scratch_vmid_[min,max] is an invalid band [%d,%d]", min, max)
}
if 9999 >= min && 9999 <= max {
return fmt.Errorf("config: backup scratch band [%d,%d] must not include the standing scratch 9999", min, max)
}
if b.RestoreStorage == "" {
return fmt.Errorf("config: backup.restore_storage is required when the restore-test cadence is enabled")
}
return nil
}
// StorageConfig tunes the storage watchdog (slice 5). All optional — zero values fall back
// to the storage package defaults via the accessor methods. The watchdog poll is FAST
// (seconds) to catch a USB drop quickly; the debounce keeps a flapping drive from storming
@@ -203,6 +267,14 @@ func applyEnv(cfg *Config) {
}
cfg.Hub.PollSeconds = envInt("FELHOM_AGENT_HUB_POLL_SECONDS", cfg.Hub.PollSeconds)
cfg.Hub.TimeoutSeconds = envInt("FELHOM_AGENT_HUB_TIMEOUT_SECONDS", cfg.Hub.TimeoutSeconds)
// backup (slice 6)
if v := os.Getenv("FELHOM_AGENT_BACKUP_LOCAL_TARGET"); v != "" {
cfg.Backup.LocalBackupTarget = v
}
if v := os.Getenv("FELHOM_AGENT_BACKUP_RESTORE_STORAGE"); v != "" {
cfg.Backup.RestoreStorage = v
}
cfg.Backup.RestoreTestCadenceSeconds = envInt("FELHOM_AGENT_BACKUP_RESTORE_TEST_CADENCE_SECONDS", cfg.Backup.RestoreTestCadenceSeconds)
}
// envInt overlays an int env var, keeping cur (with a stderr warning) on parse
internal/hub/collect.go
+44 -5
View File
@@ -31,12 +31,25 @@ type StorageObserver interface {
Observe(ctx context.Context) ([]StorageTarget, error)
}
// BackupReporter / RestoreTestReporter are the slice-6 seams the backup layer plugs into
// (same consumer-side pattern as StorageObserver — hub does not import the backup package).
// They return the agent's LATEST-known backup-per-target / restore-test result (point-in-time
// state the backup layer accumulates), not a live scan. A nil reporter → empty slice.
type BackupReporter interface {
Backups(ctx context.Context) []Backup
}
type RestoreTestReporter interface {
RestoreTests(ctx context.Context) []RestoreTest
}
// Collector builds a HostReport from read-only sources. All deps are behind narrow
// interfaces for unit testing.
type Collector struct {
px proxmoxReader
cf CloudflaredProber
storage StorageObserver
backups BackupReporter
restoreTests RestoreTestReporter
hostID string
agentVersion string
logger *slog.Logger
@@ -44,8 +57,8 @@ type Collector struct {
}
// NewCollector builds a collector. hostID echoes config.Hub.HostID; agentVersion is
// the binary version. storage may be nil (storage_targets emitted empty).
func NewCollector(px proxmoxReader, cf CloudflaredProber, storage StorageObserver, hostID, agentVersion string, logger *slog.Logger) *Collector {
// the binary version. storage/backups/restoreTests may be nil (their collections emit empty).
func NewCollector(px proxmoxReader, cf CloudflaredProber, storage StorageObserver, backups BackupReporter, restoreTests RestoreTestReporter, hostID, agentVersion string, logger *slog.Logger) *Collector {
if logger == nil {
logger = slog.Default()
}
@@ -53,6 +66,8 @@ func NewCollector(px proxmoxReader, cf CloudflaredProber, storage StorageObserve
px: px,
cf: cf,
storage: storage,
backups: backups,
restoreTests: restoreTests,
hostID: hostID,
agentVersion: agentVersion,
logger: logger,
@@ -79,9 +94,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
// storage_targets populated this slice (slice 5) via the observer; the rest stay
// defined-but-empty (slice 6). Non-nil so they marshal as [].
StorageTargets: c.collectStorage(ctx),
Backups: []Backup{},
RestoreTests: []RestoreTest{},
PBSSnapshots: []PBSSnapshot{},
Backups: c.collectBackups(ctx),
RestoreTests: c.collectRestoreTests(ctx),
PBSSnapshots: []PBSSnapshot{}, // Phase B
AuditTail: []AuditEntry{},
Cloudflared: Cloudflared{Status: c.cloudflaredStatus(ctx)},
}
@@ -159,6 +175,29 @@ func (c *Collector) collectStorage(ctx context.Context) []StorageTarget {
return targets
}
// collectBackups / collectRestoreTests read the agent's latest backup + restore-test state
// via the seams. Best-effort: a nil reporter or nil slice degrades to an empty (non-nil)
// list so the collection always marshals as [].
func (c *Collector) collectBackups(ctx context.Context) []Backup {
if c.backups == nil {
return []Backup{}
}
if b := c.backups.Backups(ctx); b != nil {
return b
}
return []Backup{}
}
func (c *Collector) collectRestoreTests(ctx context.Context) []RestoreTest {
if c.restoreTests == nil {
return []RestoreTest{}
}
if r := c.restoreTests.RestoreTests(ctx); r != nil {
return r
}
return []RestoreTest{}
}
func (c *Collector) cloudflaredStatus(ctx context.Context) string {
if c.cf == nil {
return "unknown"
internal/hub/collect_test.go
+6 -6
View File
@@ -33,7 +33,7 @@ func TestCollect_StorageTargetsFromObserver(t *testing.T) {
obs := fakeObserver{targets: []StorageTarget{
{Name: "local-lvm", Type: StorageTypeLVMThin, State: StorageStateAttached, Reachable: true},
}}
c := NewCollector(px, fakeProber{status: "active"}, obs, "h", "0.5.0", quietLogger())
c := NewCollector(px, fakeProber{status: "active"}, obs, nil, nil, "h", "0.5.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
@@ -45,7 +45,7 @@ func TestCollect_StorageTargetsFromObserver(t *testing.T) {
func TestCollect_StorageObserverErrorDegradesToEmpty(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{err: errors.New("proxmox down")}, "h", "0.5.0", quietLogger())
c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{err: errors.New("proxmox down")}, nil, nil, "h", "0.5.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("a storage observe error must not sink the heartbeat: %v", err)
@@ -64,7 +64,7 @@ func TestCollect_HostAndGuests(t *testing.T) {
},
cfg: map[int]proxmox.GuestConfig{100: {Cores: 2, Memory: 2048}},
}
c := NewCollector(px, fakeProber{status: "active"}, nil, "demo-host-01", "0.3.0", quietLogger())
c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, "demo-host-01", "0.3.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("Collect: %v", err)
@@ -104,7 +104,7 @@ func TestCollect_GuestConfigFailureKeepsStatusOmitsSpec(t *testing.T) {
cfg: map[int]proxmox.GuestConfig{100: {Cores: 2}},
cfgErr: map[int]error{200: errors.New("config read failed")},
}
c := NewCollector(px, fakeProber{status: "active"}, nil, "h", "0.3.1", quietLogger())
c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, "h", "0.3.1", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("a per-guest failure must NOT fail the whole report: %v", err)
@@ -125,7 +125,7 @@ func TestCollect_GuestConfigFailureKeepsStatusOmitsSpec(t *testing.T) {
func TestCollect_NodeStatusFailureIsHardError(t *testing.T) {
px := &fakePx{node: "n", nsErr: errors.New("proxmox down")}
c := NewCollector(px, fakeProber{status: "active"}, nil, "h", "0.3.0", quietLogger())
c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, "h", "0.3.0", quietLogger())
if _, err := c.Collect(context.Background()); err == nil {
t.Fatal("NodeStatus failure must be a hard error (no useful report)")
}
@@ -133,7 +133,7 @@ func TestCollect_NodeStatusFailureIsHardError(t *testing.T) {
func TestCollect_CloudflaredProbeErrorIsUnknown(t *testing.T) {
px := &fakePx{node: "n", ns: newTestNodeStatus()}
c := NewCollector(px, fakeProber{err: errors.New("no systemctl")}, nil, "h", "0.3.0", quietLogger())
c := NewCollector(px, fakeProber{err: errors.New("no systemctl")}, nil, nil, nil, "h", "0.3.0", quietLogger())
r, err := c.Collect(context.Background())
if err != nil {
t.Fatalf("cloudflared failure must not be fatal: %v", err)
internal/hub/contract_test.go
+17 -1
View File
@@ -46,7 +46,19 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
Smart: SmartSummary{Health: SmartUnknown},
},
},
Backups: []Backup{}, RestoreTests: []RestoreTest{},
Backups: []Backup{
{
TargetID: "local", VMID: 9001, Archive: "local:backup/x.tar.zst", Mode: "snapshot",
CrashConsistent: true, SizeBytes: 1, Success: true, StartedAt: "2026-06-09T11:00:00Z",
DurationSeconds: 1, UncoveredVolumes: []string{"/mnt/bulk"},
},
},
RestoreTests: []RestoreTest{
{
SourceArchive: "local:backup/x.tar.zst", SourceTier: "local", ScratchVMID: 990000,
Pass: true, Verified: "boot+running", TestedAt: "2026-06-09T11:05:00Z", DurationSeconds: 1,
},
},
PBSSnapshots: []PBSSnapshot{}, AuditTail: []AuditEntry{},
Cloudflared: Cloudflared{Status: "active"},
}
@@ -66,6 +78,10 @@ func TestHostReport_ContractMatchesGolden(t *testing.T) {
assertSameKeys(t, "storage_targets[0]", gst, sst)
assertSameKeys(t, "storage_targets[0].smart", field(gst, "smart"), field(sst, "smart"))
assertSameKeys(t, "storage_targets[0].thin_pool", field(gst, "thin_pool"), field(sst, "thin_pool"))
// slice-6 additions — backups[0] / restore_tests[0] key sets (the bidirectional guard).
assertSameKeys(t, "backups[0]", firstElem(golden["backups"]), firstElem(got["backups"]))
assertSameKeys(t, "restore_tests[0]", firstElem(golden["restore_tests"]), firstElem(got["restore_tests"]))
}
// field extracts a nested object value from a decoded JSON map (nil if absent/not a map).
internal/hub/report.go
+34 -3
View File
@@ -165,9 +165,40 @@ const (
StorageStateDecommissioned = "decommissioned"
)
type Backup struct{} // slice 6: per-target backup status fields TBD
type RestoreTest struct{} // slice 6: self-restore-test result fields TBD
type PBSSnapshot struct{} // slice 6: PBS snapshot inventory fields TBD
// Backup is the latest guest-vzdump result per target (doc 03 §8, slice 6 Phase A). An
// agent-initiated vzdump is CRASH-CONSISTENT only (no fsfreeze; app-consistency needs the
// controller quiesce, slice 8) — marked so here. UncoveredVolumes lists the guest's
// backup=0 (or backup-unset) mountpoints excluded from the vzdump — the bulk-volume DR gap
// (the bulk-backup mechanism is slice 10). Cross-repo contract: keep byte-identical with
// felhom.eu/hub and the bidirectional golden key-set test.
type Backup struct {
TargetID string `json:"target_id"` // backup storage name (e.g. "local")
VMID int `json:"vmid"` // source guest
Archive string `json:"archive"` // produced vzdump volid
Mode string `json:"mode"` // snapshot | stop
CrashConsistent bool `json:"crash_consistent"` // always true this slice
SizeBytes int64 `json:"size_bytes"`
Success bool `json:"success"`
Error string `json:"error,omitempty"`
StartedAt string `json:"started_at"` // RFC3339
DurationSeconds float64 `json:"duration_seconds"`
UncoveredVolumes []string `json:"uncovered_volumes"` // backup=0/unset mountpoints (bulk gap)
}
// RestoreTest is the latest self-restore-test result (doc 03 §8). This slice verifies
// boot+running only (deep app-health is slice 8). SourceTier is "local" here; PBS is Phase B.
type RestoreTest struct {
SourceArchive string `json:"source_archive"`
SourceTier string `json:"source_tier"` // "local" (pbs = Phase B)
ScratchVMID int `json:"scratch_vmid"`
Pass bool `json:"pass"`
Verified string `json:"verified"` // "boot+running" this slice
Error string `json:"error,omitempty"`
TestedAt string `json:"tested_at"` // RFC3339
DurationSeconds float64 `json:"duration_seconds"`
}
type PBSSnapshot struct{} // slice 6 Phase B: PBS snapshot inventory fields TBD
type AuditEntry struct{} // audit-log tail entry fields TBD
// ControlEnvelope is the hub's 200 response to a host-report. This slice the agent
internal/hub/testdata/host-report.golden.json
Vendored
+25 -2
View File
@@ -86,8 +86,31 @@
}
}
],
"backups": [],
"restore_tests": [],
"backups": [
{
"target_id": "local",
"vmid": 9001,
"archive": "local:backup/vzdump-lxc-9001-2026_06_09-11_00_00.tar.zst",
"mode": "snapshot",
"crash_consistent": true,
"size_bytes": 524288000,
"success": true,
"started_at": "2026-06-09T11:00:00Z",
"duration_seconds": 42.5,
"uncovered_volumes": ["/mnt/bulk"]
}
],
"restore_tests": [
{
"source_archive": "local:backup/vzdump-lxc-9001-2026_06_09-11_00_00.tar.zst",
"source_tier": "local",
"scratch_vmid": 990000,
"pass": true,
"verified": "boot+running",
"tested_at": "2026-06-09T11:05:00Z",
"duration_seconds": 38.2
}
],
"pbs_snapshots": [],
"cloudflared": { "status": "active" },
"audit_tail": []
internal/proxmox/mutate.go
+24 -1
View File
@@ -61,10 +61,14 @@ type VzdumpOptions struct {
Storage string // a storage whose content includes "backup" (e.g. "local") — NOT local-lvm
Mode BackupMode // ModeStop | ModeSnapshot
Compress string // "zstd" (default), "lzo", "gzip", or "" for none
// Notes is the PVE `notes-template` for the backup (a template string PVE expands,
// e.g. with {{guestname}}/{{node}}). Optional.
Notes string
}
// Vzdump starts a backup via POST /nodes/{node}/vzdump. Returns the UPID. An
// agent-initiated vzdump is crash-consistent only for an LXC (no fsfreeze).
// agent-initiated vzdump is crash-consistent only for an LXC (no fsfreeze) —
// app-consistency needs the controller to quiesce first (slice 8).
func (c *Client) Vzdump(ctx context.Context, opts VzdumpOptions) (string, error) {
if opts.VMID == 0 || opts.Storage == "" || opts.Mode == "" {
return "", fmt.Errorf("proxmox: Vzdump needs vmid, storage and mode")
@@ -77,9 +81,28 @@ func (c *Client) Vzdump(ctx context.Context, opts VzdumpOptions) (string, error)
opts.Compress = "zstd"
}
v.Set("compress", opts.Compress)
if opts.Notes != "" {
v.Set("notes-template", opts.Notes) // PVE 9.x param name (verified on demo)
}
return c.dataString(ctx, http.MethodPost, "/nodes/"+c.node+"/vzdump", v)
}
// DestroyLXC destroys a guest via DELETE /nodes/{node}/lxc/{vmid}. Returns the UPID.
// `purge=1` also drops the guest from jobs/HA; `destroy-unreferenced-disks=1` reaps any
// orphaned volumes. This is the scratch-guest teardown primitive (slice 6); it is
// destructive-class and the caller MUST route it through the reversibility gate
// (benign only by agent-internal scratch/same-txn provenance — see reconcile.Classify).
func (c *Client) DestroyLXC(ctx context.Context, vmid int) (string, error) {
if vmid == 0 {
return "", fmt.Errorf("proxmox: DestroyLXC needs a vmid")
}
v := url.Values{}
v.Set("purge", "1")
v.Set("destroy-unreferenced-disks", "1")
path := fmt.Sprintf("/nodes/%s/lxc/%d", c.node, vmid)
return c.dataString(ctx, http.MethodDelete, path, v)
}
// Snapshot creates an LXC snapshot via POST /nodes/{node}/lxc/{vmid}/snapshot.
// A running, unprivileged LXC can be snapshotted on LVM-thin with no stop
// (phase1-2 §1.6) — this is the snapshot-before-change primitive.
internal/proxmox/query.go
+23
View File
@@ -74,5 +74,28 @@ func (c *Client) StorageContent(ctx context.Context, store string) ([]StorageCon
return cs, c.get(ctx, path, &cs)
}
// LatestBackupVolID resolves the most recent vzdump archive for vmid on a backup storage.
// It lists the store's content, keeps only backup archives for that vmid, and returns the
// one with the greatest CTime. This is how a backup's produced archive is resolved after
// Vzdump+WaitTask (the task status carries no result volid), and how the restore-test picks
// a backup to restore. Returns ("", nil) when the guest has no archive on that store.
func (c *Client) LatestBackupVolID(ctx context.Context, store string, vmid int) (string, error) {
contents, err := c.StorageContent(ctx, store)
if err != nil {
return "", err
}
var bestVol string
var bestCTime int64 = -1
for _, e := range contents {
if e.Content != "backup" || e.VMID != vmid {
continue
}
if e.CTime > bestCTime {
bestCTime, bestVol = e.CTime, e.VolID
}
}
return bestVol, nil
}
// urlEscape escapes a path segment (a UPID contains ':' and '@').
func urlEscape(s string) string { return url.PathEscape(s) }
internal/reconcile/engine_test.go
+29
View File
@@ -20,6 +20,10 @@ type fakeAPI struct {
startUPID, stopUPID, setUPID, resizeUPID string
startErr, stopErr, setErr, resizeErr error
restoreUPID, destroyUPID string
restoreErr, destroyErr error
// status maps vmid -> the Guest returned by GuestStatus (default running if absent).
status map[int]proxmox.Guest
// waitFunc maps a UPID to a (status, err); default = OK. Mirrors the real client,
// which errors on a non-OK exitstatus.
waitFunc func(upid string) (proxmox.TaskStatus, error)
@@ -30,6 +34,8 @@ type fakeAPI struct {
stops []int
sets []setCall
resizes []resizeCall
restores []proxmox.RestoreLXCOptions
destroys []int
waits []string
listErr error
}
@@ -39,6 +45,29 @@ type resizeCall struct {
disk, size string
}
func (f *fakeAPI) RestoreLXC(_ context.Context, opts proxmox.RestoreLXCOptions) (string, error) {
f.mu.Lock()
f.restores = append(f.restores, opts)
f.mu.Unlock()
return f.restoreUPID, f.restoreErr
}
func (f *fakeAPI) DestroyLXC(_ context.Context, vmid int) (string, error) {
f.mu.Lock()
f.destroys = append(f.destroys, vmid)
f.mu.Unlock()
return f.destroyUPID, f.destroyErr
}
func (f *fakeAPI) GuestStatus(_ context.Context, vmid int) (proxmox.Guest, error) {
f.mu.Lock()
defer f.mu.Unlock()
if g, ok := f.status[vmid]; ok {
return g, nil
}
return proxmox.Guest{VMID: vmid, Status: "running"}, nil
}
func (f *fakeAPI) TaskStatusOnce(_ context.Context, upid string) (proxmox.TaskStatus, error) {
if f.statusFunc != nil {
return f.statusFunc(upid)
internal/reconcile/journal.go
+6
View File
@@ -45,6 +45,12 @@ type JournalEntry struct {
UPID string `json:"upid,omitempty"`
State OpState `json:"state"`
IdempKey string `json:"idemp_key,omitempty"`
// Scratch marks an entry that OWNS an agent-tagged scratch guest at VMID (slice 6
// restore-test). While such an entry is in-flight, the load-bearing invariant is "the
// scratch guest may exist and MUST be destroyed" — so Recover resolves it by ensuring
// VMID is gone (a benign teardown), NOT by re-checking any sub-task UPID. The entry is
// terminal only after teardown. See recover.go.
Scratch bool `json:"scratch,omitempty"`
At time.Time `json:"at"`
}
internal/reconcile/recover.go
+76 -1
View File
@@ -3,6 +3,8 @@ package reconcile
import (
"context"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// Recover consumes the journal's in-flight set at startup: resume-or-rollback for any
@@ -32,6 +34,16 @@ func (e *Engine) Recover(ctx context.Context) RecoverResult {
}
for _, entry := range e.journal.InFlight() {
res.Examined++
// Scratch entries (slice-6 restore-test) are resolved by TEARDOWN, not by
// re-checking a sub-task UPID — a leaked scratch guest is the failure mode that
// matters. Handle them BEFORE the generic UPID path (else the restore sub-task's OK
// status would mark the entry succeeded while the guest still exists → leak).
if entry.Scratch {
e.recoverScratch(ctx, entry, &res)
continue
}
if entry.UPID == "" {
// POST never confirmed → abandon (fail-safe).
e.append(terminal(entry, OpFailed))
@@ -72,6 +84,66 @@ func (e *Engine) Recover(ctx context.Context) RecoverResult {
return res
}
// recoverScratch resolves a leaked restore-test scratch guest (slice 6, doc 03 §8/§10).
// The invariant: a Scratch entry in-flight at startup means "scratch guest VMID may exist
// and must be destroyed." It is idempotent — if the guest is already gone (crash after the
// destroy task but before the terminal record), it records terminal-clean. The teardown
// routes through the gate as a benign ClassGuestDestroy (agent-tagged scratch provenance) —
// the same audit-bearing path the normal teardown uses.
func (e *Engine) recoverScratch(ctx context.Context, entry JournalEntry, res *RecoverResult) {
lxc, err := e.api.ListLXC(ctx)
if err != nil {
// Can't tell whether the guest exists → leave in-flight; a later Recover retries.
res.Unresolved++
e.logger.Warn("recover: cannot list guests to resolve leaked scratch; left in-flight",
"op_id", entry.OpID, "vmid", entry.VMID, "err", err)
return
}
exists := false
for _, g := range lxc {
if g.VMID == entry.VMID {
exists = true
break
}
}
if !exists {
// Already gone (idempotent) → the scratch left no leak.
e.append(terminal(entry, OpSucceeded))
res.ScratchClean++
e.logger.Info("recover: leaked-scratch entry resolved; guest already gone",
"op_id", entry.OpID, "vmid", entry.VMID)
return
}
dec := e.gate.Authorize(IntentForScratchDestroy(e.hostID, entry.VMID), nil)
if !dec.Allowed {
// Should be benign; if not, fail-safe (leave in-flight, do NOT force a destroy).
res.Unresolved++
e.logger.Error("recover: scratch teardown refused by gate (unexpected); left in-flight",
"op_id", entry.OpID, "vmid", entry.VMID, "reason", dec.Reason)
return
}
upid, err := e.api.DestroyLXC(ctx, entry.VMID)
if err != nil {
res.Unresolved++
e.logger.Warn("recover: destroying leaked scratch failed; left in-flight (will retry)",
"op_id", entry.OpID, "vmid", entry.VMID, "err", err)
return
}
if upid != "" {
if _, err := e.api.WaitTask(ctx, upid, proxmox.WaitOptions{}); err != nil {
res.Unresolved++
e.logger.Warn("recover: leaked-scratch destroy task failed; left in-flight (will retry)",
"op_id", entry.OpID, "vmid", entry.VMID, "err", err)
return
}
}
e.append(terminal(entry, OpSucceeded))
res.ScratchDestroyed++
e.logger.Warn("recover: destroyed leaked restore-test scratch guest",
"op_id", entry.OpID, "vmid", entry.VMID)
}
// RecoverResult summarizes a startup recovery pass.
type RecoverResult struct {
Examined int
@@ -80,10 +152,12 @@ type RecoverResult struct {
RolledBack int // no task id → abandoned (fail-safe)
StillRunning int // task still executing → left in-flight
Unresolved int // task status unreadable → left in-flight
ScratchClean int // scratch entry resolved: guest already gone (no leak)
ScratchDestroyed int // scratch entry resolved: leaked guest destroyed
}
// terminal builds a terminal journal record preserving the op's identity, with the
// idempotency key carried through so a SUCCEEDED one-shot op marks its key applied.
// idempotency key + scratch flag carried through.
func terminal(e JournalEntry, state OpState) JournalEntry {
return JournalEntry{
OpID: e.OpID,
@@ -92,6 +166,7 @@ func terminal(e JournalEntry, state OpState) JournalEntry {
UPID: e.UPID,
State: state,
IdempKey: e.IdempKey,
Scratch: e.Scratch,
At: time.Now().UTC(),
}
}
internal/reconcile/restoretest.go
+289
View File
@@ -0,0 +1,289 @@
package reconcile
import (
"context"
"fmt"
"strconv"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// The self-restore-test (doc 03 §8) — the piece that closes "a backup you haven't restored
// isn't a backup". It is a JOURNALED reconcile job so it inherits the slice-4 journal,
// per-guest serialization, and crash-safe recovery: a mid-test crash can't leak a scratch
// guest (engine.Recover tears it down). Every step here is BENIGN — restore-to-new
// (ClassCreate), a benign net-link-down SetConfig, and a scratch teardown that is benign by
// agent-tagged-scratch provenance (no new destructive class, no new crypto).
// scratchKind is the journal Kind for a restore-test scratch-guest-owning entry. Recover
// keys off JournalEntry.Scratch (not this string), but the Kind aids audit/debug.
const scratchKind = "scratch_restore_test"
// DefaultBootTimeout bounds how long the restore-test waits for the scratch guest to reach
// running before declaring the verify failed.
const DefaultBootTimeout = 2 * time.Minute
// RestoreTestSpec parameterizes one restore-test.
type RestoreTestSpec struct {
Archive string // source archive volid to restore (resolved by the caller)
SourceTier string // "local" this slice (pbs = Phase B) — for the report
RestoreStorage string // target storage for the restored rootfs (e.g. "local-lvm")
ScratchMin int // inclusive scratch VMID band (must be > 0)
ScratchMax int // inclusive
BootTimeout time.Duration // 0 → DefaultBootTimeout
}
// RestoreTestResult is the reconcile-local outcome (the backup package maps it to the
// hub.RestoreTest wire record — reconcile must not import hub for this).
type RestoreTestResult struct {
Archive string
SourceTier string
ScratchVMID int
Pass bool
Verified string // "boot+running" this slice
Skipped bool // no free scratch VMID in band → test not run
Err error
StartedAt time.Time
Duration time.Duration
}
// IntentForScratchDestroy builds the benign teardown intent for an agent-owned scratch
// guest: ClassGuestDestroy made benign by AgentTaggedScratch provenance (classify.go). The
// gate authorizes it unsigned but is genuinely in-path (wrong provenance → pending_signature).
func IntentForScratchDestroy(hostID string, vmid int) Intent {
return Intent{
Class: ClassGuestDestroy,
HostID: hostID,
GuestID: strconv.Itoa(vmid),
VMID: vmid,
Provenance: Provenance{AgentTaggedScratch: true}, // agent-internal, never hub-sourced
Source: SourceOneShotJob,
}
}
// RunRestoreTest runs one restore-test on the per-guest queue lane of a fresh scratch VMID.
// It journals a Scratch-owned entry BEFORE any mutation, so a crash anywhere after this
// point is recoverable (Recover destroys the scratch guest). Teardown runs on EVERY path
// (defer), including a failed verify. The returned Err is the TEST verdict's error (restore
// or boot failure), independent of teardown success.
func (e *Engine) RunRestoreTest(ctx context.Context, spec RestoreTestSpec) RestoreTestResult {
now := time.Now().UTC()
res := RestoreTestResult{Archive: spec.Archive, SourceTier: spec.SourceTier, StartedAt: now}
if spec.Archive == "" || spec.RestoreStorage == "" {
res.Err = fmt.Errorf("reconcile: restore-test needs an archive and a restore storage")
return res
}
if spec.ScratchMin <= 0 || spec.ScratchMax < spec.ScratchMin {
res.Err = fmt.Errorf("reconcile: invalid scratch VMID band [%d,%d]", spec.ScratchMin, spec.ScratchMax)
return res
}
lxc, err := e.api.ListLXC(ctx)
if err != nil {
res.Err = fmt.Errorf("reconcile: restore-test list guests: %w", err)
return res
}
vmid, ok := pickScratchVMID(lxc, spec.ScratchMin, spec.ScratchMax)
if !ok {
// Full band (e.g. an accumulation of un-torn-down scratch guests) → skip, never
// panic or pick out-of-band. Recover will reap any genuinely leaked ones.
e.logger.Warn("restore-test skipped: no free scratch VMID in band",
"min", spec.ScratchMin, "max", spec.ScratchMax)
res.Skipped = true
return res
}
res.ScratchVMID = vmid
// Serialize on the scratch VMID's lane (inherits §10), and capture the result.
ch := e.queue.Submit(vmid, func() error {
e.runScratchTest(ctx, vmid, spec, &res)
return res.Err
})
<-ch
res.Duration = time.Since(now)
return res
}
// runScratchTest is the journaled body (runs on vmid's queue lane).
func (e *Engine) runScratchTest(ctx context.Context, vmid int, spec RestoreTestSpec, res *RestoreTestResult) {
base := JournalEntry{OpID: e.scratchOpID(vmid), VMID: vmid, Kind: scratchKind, Scratch: true}
// OWN the scratch guest's cleanup BEFORE any mutation. From here, a crash is recoverable.
e.append(withState(base, OpStarted))
// Teardown ALWAYS runs (even on a failed verify). Uses a cancel-immune context so a
// daemon shutdown mid-test still tears down; if teardown fails, the entry stays
// in-flight and Recover reaps the guest on the next start.
defer e.teardownScratch(ctx, base)
// 1. Restore into the fresh scratch VMID (benign create path). The UPID is for error
// detection only — it does NOT make the Scratch entry terminal (teardown does).
upid, err := e.api.RestoreLXC(ctx, proxmox.RestoreLXCOptions{
VMID: vmid, Archive: spec.Archive, Storage: spec.RestoreStorage,
})
if err != nil {
res.Err = fmt.Errorf("reconcile: restore-test restore: %w", err)
return
}
e.append(withUPID(base, upid, OpTaskRunning))
if upid != "" {
if _, err := e.api.WaitTask(ctx, upid, proxmox.WaitOptions{}); err != nil {
res.Err = fmt.Errorf("reconcile: restore-test restore task: %w", err)
return
}
}
// 2. Net link-down on every interface BEFORE boot — test-safety so the clone (which
// keeps the source MAC/hostname; identity-reset is slice 7) can't conflict with a
// running source on L2/IP. Benign SetConfig.
cfg, err := e.api.GuestConfig(ctx, vmid)
if err != nil {
res.Err = fmt.Errorf("reconcile: restore-test read scratch config: %w", err)
return
}
for key, val := range cfg.Nets() {
if _, err := e.api.SetConfig(ctx, vmid, map[string]string{key: withLinkDown(val)}); err != nil {
res.Err = fmt.Errorf("reconcile: restore-test net link-down %s: %w", key, err)
return
}
}
// 3. Boot and verify it reaches running (basic liveness; deep app-health is slice 8).
startUPID, err := e.api.Start(ctx, vmid)
if err != nil {
res.Err = fmt.Errorf("reconcile: restore-test start: %w", err)
return
}
if startUPID != "" {
if _, err := e.api.WaitTask(ctx, startUPID, proxmox.WaitOptions{}); err != nil {
res.Err = fmt.Errorf("reconcile: restore-test start task: %w", err)
return
}
}
if err := e.waitRunning(ctx, vmid, bootTimeout(spec)); err != nil {
res.Err = err
return
}
res.Pass = true
res.Verified = "boot+running"
}
// teardownScratch destroys the scratch guest (benign, gated) and records the entry terminal.
// On any teardown failure it leaves the entry in-flight so Recover reaps the guest later.
func (e *Engine) teardownScratch(ctx context.Context, base JournalEntry) {
// Cancel-immune + bounded, so a shutdown mid-test still tears down.
tctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Minute)
defer cancel()
dec := e.gate.Authorize(IntentForScratchDestroy(e.hostID, base.VMID), nil)
if !dec.Allowed {
e.logger.Error("restore-test: scratch teardown refused by gate (unexpected); left for Recover",
"vmid", base.VMID, "reason", dec.Reason)
return
}
upid, err := e.api.DestroyLXC(tctx, base.VMID)
if err != nil {
e.logger.Error("restore-test: scratch teardown failed; left for Recover", "vmid", base.VMID, "err", err)
return
}
if upid != "" {
if _, err := e.api.WaitTask(tctx, upid, proxmox.WaitOptions{}); err != nil {
e.logger.Error("restore-test: scratch teardown task failed; left for Recover", "vmid", base.VMID, "err", err)
return
}
}
e.append(withState(base, OpSucceeded))
e.logger.Info("restore-test: scratch guest torn down", "vmid", base.VMID)
}
// waitRunning polls GuestStatus until the guest is running or the timeout elapses. The poll
// interval is 2s in production, but shrinks for short timeouts so it stays responsive.
func (e *Engine) waitRunning(ctx context.Context, vmid int, timeout time.Duration) error {
interval := 2 * time.Second
if timeout < 4*interval {
if interval = timeout / 4; interval < 10*time.Millisecond {
interval = 10 * time.Millisecond
}
}
deadline := time.Now().Add(timeout)
t := time.NewTicker(interval)
defer t.Stop()
for {
g, err := e.api.GuestStatus(ctx, vmid)
if err == nil && g.Status == "running" {
return nil
}
if time.Now().After(deadline) {
if err != nil {
return fmt.Errorf("reconcile: restore-test verify: guest %d not running within %s (last err: %w)", vmid, timeout, err)
}
return fmt.Errorf("reconcile: restore-test verify: guest %d not running within %s", vmid, timeout)
}
select {
case <-ctx.Done():
return ctx.Err()
case <-t.C:
}
}
}
// pickScratchVMID returns the lowest free VMID in [min,max], excluding the standing 9999
// scratch and any in-use guest. ok=false when the band is fully occupied (the test is then
// skipped, never run out-of-band).
func pickScratchVMID(lxc []proxmox.Guest, min, max int) (int, bool) {
used := make(map[int]bool, len(lxc))
for _, g := range lxc {
used[g.VMID] = true
}
for id := min; id <= max; id++ {
if id == 9999 || used[id] {
continue
}
return id, true
}
return 0, false
}
// withLinkDown sets link_down=1 on a Proxmox netN config string, REPLACING any existing
// link_down token (never blind-concatenating, so a re-applied/pre-set value can't produce a
// malformed netN).
func withLinkDown(netN string) string {
parts := strings.Split(netN, ",")
out := parts[:0]
for _, p := range parts {
if p == "" || strings.HasPrefix(p, "link_down=") {
continue
}
out = append(out, p)
}
out = append(out, "link_down=1")
return strings.Join(out, ",")
}
func bootTimeout(spec RestoreTestSpec) time.Duration {
if spec.BootTimeout > 0 {
return spec.BootTimeout
}
return DefaultBootTimeout
}
func (e *Engine) scratchOpID(vmid int) string {
return "scratch-restore-" + strconv.Itoa(vmid) + "-" + nextSeq(&e.opSeq)
}
// withState / withUPID build journal records from a base entry, preserving its identity +
// Scratch flag.
func withState(base JournalEntry, state OpState) JournalEntry {
base.State = state
base.At = time.Now().UTC()
return base
}
func withUPID(base JournalEntry, upid string, state OpState) JournalEntry {
base.UPID = upid
base.State = state
base.At = time.Now().UTC()
return base
}
internal/reconcile/restoretest_test.go
+227
View File
@@ -0,0 +1,227 @@
package reconcile
import (
"context"
"encoding/json"
"errors"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// scratchCfg builds a fake GuestConfig with one net interface (so the link-down SetConfig
// step runs).
func scratchCfg() proxmox.GuestConfig {
return proxmox.GuestConfig{Extra: map[string]json.RawMessage{
"net0": json.RawMessage(`"name=eth0,bridge=vmbr0,hwaddr=AA:BB:CC:DD:EE:FF,ip=dhcp"`),
}}
}
func TestRunRestoreTest_PassAndTeardown(t *testing.T) {
api := &fakeAPI{cfg: map[int]proxmox.GuestConfig{990000: scratchCfg()}} // empty lxc → 990000 free; running default
e, j, q := newEngine(t, api, EmptyProvider{})
defer q.Close()
res := e.RunRestoreTest(context.Background(), RestoreTestSpec{
Archive: "local:backup/x.tar.zst", RestoreStorage: "local-lvm",
ScratchMin: 990000, ScratchMax: 990009, SourceTier: "local",
})
if res.Skipped || !res.Pass || res.Err != nil {
t.Fatalf("expected pass, got %+v", res)
}
if res.ScratchVMID != 990000 || res.Verified != "boot+running" {
t.Fatalf("result = %+v", res)
}
if len(api.restores) != 1 || api.restores[0].VMID != 990000 || api.restores[0].Archive != "local:backup/x.tar.zst" {
t.Fatalf("restore not issued correctly: %+v", api.restores)
}
// net link-down applied before boot.
foundLinkDown := false
for _, s := range api.sets {
if s.vmid == 990000 && s.params["net0"] != "" && contains2(s.params["net0"], "link_down=1") {
foundLinkDown = true
}
}
if !foundLinkDown {
t.Errorf("expected a net link-down SetConfig, got %+v", api.sets)
}
// teardown destroyed the scratch guest, and the journal entry is terminal (not in-flight).
if len(api.destroys) != 1 || api.destroys[0] != 990000 {
t.Fatalf("scratch not torn down: %+v", api.destroys)
}
if len(j.InFlight()) != 0 {
t.Errorf("scratch entry must be terminal after teardown: %+v", j.InFlight())
}
}
func TestRunRestoreTest_TeardownOnFailedVerify(t *testing.T) {
// Guest never reaches running → verify fails, but teardown MUST still run.
api := &fakeAPI{
cfg: map[int]proxmox.GuestConfig{990000: scratchCfg()},
status: map[int]proxmox.Guest{990000: {VMID: 990000, Status: "stopped"}},
}
e, j, q := newEngine(t, api, EmptyProvider{})
defer q.Close()
res := e.RunRestoreTest(context.Background(), RestoreTestSpec{
Archive: "vol", RestoreStorage: "local-lvm",
ScratchMin: 990000, ScratchMax: 990009, BootTimeout: 40 * time.Millisecond,
})
if res.Pass || res.Err == nil {
t.Fatalf("expected a failed verify, got %+v", res)
}
if len(api.destroys) != 1 || api.destroys[0] != 990000 {
t.Fatalf("teardown MUST run even on a failed verify: destroys=%+v", api.destroys)
}
if len(j.InFlight()) != 0 {
t.Errorf("scratch entry must be terminal after teardown: %+v", j.InFlight())
}
}
func TestRunRestoreTest_RestoreFailureStillTearsDown(t *testing.T) {
api := &fakeAPI{restoreErr: errors.New("restore boom")}
e, j, q := newEngine(t, api, EmptyProvider{})
defer q.Close()
res := e.RunRestoreTest(context.Background(), RestoreTestSpec{
Archive: "vol", RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990009,
})
if res.Pass || res.Err == nil {
t.Fatalf("expected restore failure, got %+v", res)
}
// Even though restore failed, the scratch entry was journaled BEFORE the restore, so
// teardown runs (idempotent — destroys the maybe-partial guest).
if len(api.destroys) != 1 {
t.Fatalf("teardown must run after a restore failure: %+v", api.destroys)
}
if len(j.InFlight()) != 0 {
t.Errorf("scratch entry must be terminal: %+v", j.InFlight())
}
}
func TestRunRestoreTest_FullBandSkips(t *testing.T) {
// Whole band occupied → skipped, never run / out-of-band.
var guests []proxmox.Guest
for id := 990000; id <= 990001; id++ {
guests = append(guests, proxmox.Guest{VMID: id})
}
api := &fakeAPI{lxc: guests}
e, _, q := newEngine(t, api, EmptyProvider{})
defer q.Close()
res := e.RunRestoreTest(context.Background(), RestoreTestSpec{
Archive: "vol", RestoreStorage: "local-lvm", ScratchMin: 990000, ScratchMax: 990001,
})
if !res.Skipped {
t.Fatalf("full band must skip, got %+v", res)
}
if len(api.restores) != 0 || len(api.destroys) != 0 {
t.Errorf("a skipped test must not restore or destroy anything")
}
}
func TestRunRestoreTest_InvalidBandErrors(t *testing.T) {
e, _, q := newEngine(t, &fakeAPI{}, EmptyProvider{})
defer q.Close()
res := e.RunRestoreTest(context.Background(), RestoreTestSpec{Archive: "v", RestoreStorage: "s", ScratchMin: 0})
if res.Err == nil {
t.Fatal("an invalid scratch band must error")
}
}
func TestPickScratchVMID(t *testing.T) {
// excludes 9999 and in-use; lowest free.
got, ok := pickScratchVMID([]proxmox.Guest{{VMID: 990000}}, 990000, 990009)
if !ok || got != 990001 {
t.Errorf("pick = %d,%v want 990001,true", got, ok)
}
// full band.
full := []proxmox.Guest{{VMID: 990000}, {VMID: 990001}}
if _, ok := pickScratchVMID(full, 990000, 990001); ok {
t.Error("full band must return ok=false")
}
}
func TestWithLinkDown(t *testing.T) {
got := withLinkDown("name=eth0,bridge=vmbr0,ip=dhcp")
if !contains2(got, "link_down=1") || !contains2(got, "name=eth0") {
t.Errorf("withLinkDown lost fields or didn't set link_down: %q", got)
}
// idempotent: an existing link_down is replaced, not duplicated.
got = withLinkDown("name=eth0,link_down=0,bridge=vmbr0")
if count(got, "link_down=") != 1 || !contains2(got, "link_down=1") {
t.Errorf("withLinkDown must replace an existing link_down (got %q)", got)
}
}
// --- recover the leaked scratch guest (the headline crash-safety test) ---
func TestRecover_LeakedScratchDestroyed(t *testing.T) {
// The scratch guest still exists at startup (agent crashed mid-test) → Recover destroys it.
api := &fakeAPI{lxc: []proxmox.Guest{{VMID: 990000, Status: "running"}}}
e, j, _ := newEngine(t, api, EmptyProvider{})
if err := j.Append(JournalEntry{OpID: "scratch-990000-1", VMID: 990000, Kind: scratchKind, Scratch: true, State: OpTaskRunning, At: time.Now().UTC()}); err != nil {
t.Fatal(err)
}
res := e.Recover(context.Background())
if res.ScratchDestroyed != 1 {
t.Fatalf("leaked scratch must be destroyed, got %+v", res)
}
if len(api.destroys) != 1 || api.destroys[0] != 990000 {
t.Fatalf("DestroyLXC not called for the leaked scratch: %+v", api.destroys)
}
if len(j.InFlight()) != 0 {
t.Errorf("resolved scratch entry must not be in-flight: %+v", j.InFlight())
}
}
func TestRecover_LeakedScratchAlreadyGone(t *testing.T) {
// Crash AFTER the destroy task but BEFORE the terminal record → guest already gone →
// idempotent clean (no destroy issued).
api := &fakeAPI{lxc: []proxmox.Guest{{VMID: 9001, Status: "stopped"}}} // 990000 absent
e, j, _ := newEngine(t, api, EmptyProvider{})
j.Append(JournalEntry{OpID: "scratch-990000-1", VMID: 990000, Kind: scratchKind, Scratch: true, State: OpTaskRunning, At: time.Now().UTC()})
res := e.Recover(context.Background())
if res.ScratchClean != 1 || len(api.destroys) != 0 {
t.Fatalf("already-gone scratch must be clean with no destroy, got res=%+v destroys=%+v", res, api.destroys)
}
if len(j.InFlight()) != 0 {
t.Errorf("entry must be resolved: %+v", j.InFlight())
}
}
func TestRecover_LeakedScratchListUnreadable(t *testing.T) {
api := &fakeAPI{listErr: errors.New("api down")}
e, j, _ := newEngine(t, api, EmptyProvider{})
j.Append(JournalEntry{OpID: "scratch-990000-1", VMID: 990000, Kind: scratchKind, Scratch: true, State: OpTaskRunning, At: time.Now().UTC()})
res := e.Recover(context.Background())
if res.Unresolved != 1 || len(j.InFlight()) != 1 {
t.Fatalf("unreadable list must leave the scratch in-flight for a later Recover, got res=%+v inflight=%d", res, len(j.InFlight()))
}
if len(api.destroys) != 0 {
t.Error("must not destroy when it can't confirm the guest exists")
}
}
// small string helpers (avoid importing strings in the test for one call).
func contains2(s, sub string) bool { return indexOf(s, sub) >= 0 }
func count(s, sub string) int {
n, i := 0, 0
for {
j := indexOf(s[i:], sub)
if j < 0 {
return n
}
n++
i += j + len(sub)
}
}
func indexOf(s, sub string) int {
for i := 0; i+len(sub) <= len(s); i++ {
if s[i:i+len(sub)] == sub {
return i
}
}
return -1
}
internal/reconcile/state.go
+8
View File
@@ -114,6 +114,14 @@ type GuestAPI interface {
SetConfig(ctx context.Context, vmid int, params map[string]string) (string, error)
// ResizeLXC grows a volume (grow-only; the planner never emits a shrink). Async → UPID.
ResizeLXC(ctx context.Context, vmid int, disk, size string) (string, error)
// RestoreLXC restores an archive into a (fresh) vmid — the create path (slice 6). Async → UPID.
RestoreLXC(ctx context.Context, opts proxmox.RestoreLXCOptions) (string, error)
// DestroyLXC destroys a guest — the scratch-teardown primitive (slice 6). Async → UPID.
// Destructive-class; the engine only ever issues it for an agent-tagged scratch guest
// (benign by provenance) via the gate.
DestroyLXC(ctx context.Context, vmid int) (string, error)
// GuestStatus reads a single guest's current status (run-state poll during a restore-test).
GuestStatus(ctx context.Context, vmid int) (proxmox.Guest, error)
WaitTask(ctx context.Context, upid string, opts proxmox.WaitOptions) (proxmox.TaskStatus, error)
// TaskStatusOnce is a single non-blocking task-status read — used by crash
// recovery to learn the outcome of an op that was in flight when the agent died.