felhom-agent/REPORT.md

REPORT — felhom-agent v0.28.0

Backup re-target → offsite PBS (real DR) + operator-signed decommission + restore-test bind-mount fix. Implemented from the "whole-guest backup → felhom-pbs" CC SPEC (PHASE 0 gate → PHASE 1/3 + a validation-discovered fix). Live-validated on demo-felhom (PVE 9.2.2, guest 9201). Secrets (PBS token/fingerprint, encryption key) redacted / stored out-of-band.

PHASE 0 — GATE (proven live before any re-target code)

• Backup → PBS, snapshot mode fires the create storage snapshot 'vzdump' marker → 8B.2 early-resume/quiesce survives a PBS target (marker is mode-driven, not target-driven).
• Restore-test enumerates PBS through the SAME generic StorageContent (/nodes/<node>/storage/felhom-pbs/content returns content:"backup" + ctime/vmid/volid) — PickRestoreCandidate/latestArchive need NO PBS-client change.
• pct restore from a PBS volid round-trips (storage.cfg encryption key applied transparently). PBS gotchas (ignore-verified, node-from-UPID, privsep) touch only the verify-API path.

PHASE 1 — backup re-target → felhom-pbs

• config.BackupConfig.BackupTarget() returns backup.local_backup_target or, when empty, the new default felhom-pbs (separate hardware = real DR). Configurable; no call site hardcodes it. All NewBackupRunner sites + restore-test SourceTier route through it.
• Live: demo agent.json local_backup_target "local" → felhom-pbs; --selftest=backup -vmid 9201 → target=felhom-pbs, snapshot, crash-consistent ✓. Daemon /backup/status now reports target_id=felhom-pbs.
• PBS prune default set on the storage: prune-backups keep-daily=7,keep-weekly=4,keep-monthly=3 (was keep-all at 73% full).

Restore-test bind-mount fix (validation-discovered; scope-approved)

• A slice-10 enrolled guest's data drive is a host bind-mount mp0; vzrestore refuses it under the privsep token ("restoring 'mpN' to bind mount is only possible for root") — so the restore-test failed for EVERY enrolled guest, independent of backup tier.
• Fix: the restore-test reads the SOURCE guest config (vmid parsed from the archive volid) and passes RestoreLXCOptions.MountOverrides converting each bind-mount mpN → a throwaway 1G volume on the restore storage, plus a rootfs override sized from the source (PVE requires rootfs when mp params are present). Boot-verify doesn't need the data. Storage-backed mounts restore normally; best-effort if the source config is unreadable.
• Live: daemon scheduled restore-test from felhom-pbs → pass:true, verified:"boot+running", source_tier:"pbs" (44s, scratch torn down clean).

PHASE 3 — operator-signed decommission (reachable now)

• The previously-unreachable IntentDecommissioned is now reached ONLY via a gate-VERIFIED operator signature (classified destructive). New signedjobs.DecommissionExecutor (op decommission) → IntentStore.SetDecommissioned, keyed by the drive's storage durable-id (the watchdog's key, e.g. uuid:<fs-uuid> — NOT the device-level byid:/byuuid: of storage_wipe), so the intent actually gates remounts. New ExecutorChain serves both wipe + decommission; runner wiring moved below the intent-store open. felhom-opsign builds decommission params from -durable-id. No customer/controller UI — operator path is hub jobs-queue → signed-jobs runner. Distinct from a customer-confirmable safe eject.

Tests

• internal/signedjobs: decommission happy-path / foreign-op / unbound-refusal / no-store; ExecutorChain dispatch.
• internal/reconcile: archiveVMID (PBS ct/vm + vzdump forms), bindMountOverrides, rootfsSizeGB/sizeToGB.
• Full go build ./... + package tests green.

Live deploy

• felhom-agent 0.28.0 active on demo-felhom (prior binary kept as felhom-agent.bak-0.27.0); config local_backup_target=felhom-pbs, restore-test cadence restored to 86400s.
• Golden prune (housekeeping): kept the newest 2 golden (VMID 9100) archives on local, freed the older 11.
• Note (not actioned): the drive's legacy bare-metal felhom_data (underscore) dir is orphaned under the new felhom-data (hyphen) namespace on /mnt/felhom-usb; flagged for a real-customer migration path (out of scope here).