felhom-agent

The host agent for the Felhom platform — the operator-tier component that runs on each Proxmox host and owns all Proxmox interaction (provision/restore guests, host storage, backups, host+tunnel monitoring, hub control loop, per-guest local API). Design: felhom.eu/documentation/architecture/03-host-agent.md.

Status — slice 1 of N. This repo currently contains the project scaffold and the internal/proxmox interaction layer (the typed library every other module will call to talk to Proxmox), plus a runnable read-only --selftest. No reconcile loop, hub client, signing, or storage/backup orchestration yet — those are later slices.

Module: gitea.dooplex.hu/admin/felhom-agent · binary: felhom-agent · Go 1.24.

Layout

cmd/felhom-agent/      # entry point + --selftest modes + the daemon (poll loop + reconcile + watchdog)
internal/proxmox/      # the Proxmox interaction layer (API-first + fenced root-CLI)
internal/config/       # JSON config + env overrides (secrets never logged)
internal/log/          # slog setup
internal/authz/        # operator signed-op verifier (SSHSIG); durable nonce store
internal/hub/          # daemon: host-report collector + Bearer client + resilient poll loop
internal/reconcile/    # reconcile engine + reversibility gate + op journal + crash recovery
internal/storage/      # storage-target observer + durable_id + fast-poll watchdog (slice 5)
internal/capability/   # privileged-capability self-probe (manifest + sudo -n -l check, v0.44.0)
configs/agent.example.json

The capability package — privileged self-check (v0.44.0)

The non-root agent depends on a fixed set of sudo -n grants (configs/felhom-agent.sudoers). When the 2026-06-28 root→non-root cutover silently dropped some (lxc-info, make-private, …), features broke unnoticed until a user hit them. internal/capability makes that loud:

• Manifest() is the source of truth for the required (binary, representative-arg) vectors, seeded from the sudoers audit (the OK + CLOSED rows; the surfaced/deferred rows are excluded). When you add a privileged runner.Run call that needs a NEW grant, add it to BOTH the sudoers file AND the manifest — manifest_test.go asserts manifest ⊆ sudoers at build time (with a red-proof that dropping the lxc-info grant fails the gate), so a missing grant is caught in CI, not in prod.
• Prober.Probe lists each vector with sudo -n -l (a policy LIST — never executes, safe for mkfs/pct) + an os.Stat existence check → an ok/degraded snapshot. It runs at startup (logged loud) and on every collect, riding the hub report as HostReport.Capabilities; the hub (HostCapabilityChecker) alerts the operator on a Critical capability going degraded. Serve-degraded — the probe never blocks startup. (Next self-health slice: the controller↔agent channel check.)

Controller-swap under non-root (v0.45.0). The agent-owned controller image swap (internal/localapi/controllerswap.go) no longer shells out: writeImage pipes the image ref on stdin into an in-guest tee /etc/felhom-controller-image (via GuestExecStdin → Runner.RunStdin, the same fenced sudo -n runner) — no bash -c, no interpolation. Its 5 narrow grants live in the FELHOM_CONTROLLERSWAP sudoers alias (all read-only or fixed-target; the tee target is the FIXED image path, content stdin-fed) and in the capability manifest (Critical), so a dropped grant is a build failure + a live degraded signal. No general pct exec is granted.

The storage package — observe + watchdog (slice 5)

Read-only this slice (no hub desired-state until slice 10):

• Observer builds the host-report's storage_targets from ListStorage/NodeStorage joined with non-privileged host reads (/proc/mounts, /dev/disk/by-uuid, /sys/.../rotational). It reports each target's durable_id (the DR-load-bearing re-attach key: fs-UUID for usb/local-dir, server:export for nfs/cifs, repo+fingerprint for pbs, vg/pool for lvmthin), state, usage, a rotational class hint (never authoritative — class is hub-owned), and the lvmthin thin-pool data fill (a full pool corrupts every guest on it). SMART is a Phase-B privileged read.
• Watchdog is the third daemon goroutine: a fast poll (seconds) over the known target set that detects an attached↔disconnected transition and fires a debounced, out-of-band host-report so the hub learns of a USB drop in seconds rather than at the ~15-minute cycle. It mutates nothing (the benign re-mount-by-UUID response lands in Phase B). The HostReader seam keeps it root-free and unit-testable with no real devices.

The reported StorageTarget shape is a cross-repo contract duplicated in felhom.eu/hub; internal/hub/testdata/host-report.golden.json is byte-identical with the hub's copy and a bidirectional key-set test guards drift.

DR recipe — the storage/guest/PBS half (v0.38.0)

The host-report carries an additive dr_recipe section (internal/hub/dr_recipe.go) — the agent half of the secret-free reconstruction recipe (SPIKE-dr-recipe-2026-06-16.md). It is the non-secret re-provision scaffolding that complements escrow (keys) + PBS/restic (bytes): guests[] sizing, drives[] (user-data drives by durable_id → role → mount → intent), pve_storage[] (the storage.cfg defs), and pbs coordinates. Built by the pure BuildDRRecipeHostHalf from facts the report already collects — no new reads. Boundary: every field is an identifier/intent/size/ coordinate — never a key/password/token/hash/ENC:; the PBS key + restic password stay in escrow. recipe_version=1, ignore-unknown on read, pinned in the cross-repo golden. The hub assembles it with the controller's app half into one customer recipe.

The privileged HostOps surface (slice 5 Phase B)

The write side — the one place the agent steps outside its Proxmox API token into OS-root — is isolated behind the HostOps seam (hostops.go): production SudoHostOps shells out via a narrow sudoers allowlist (configs/felhom-agent.sudoers) with fixed argument vectors and no shell; tests use a fake (no real root in the suite).

• Persistent mounts are systemd .mount units keyed by fs-UUID (What=/dev/disk/by-uuid/<UUID>, enabled so they survive reboot) — not raw fstab or a transient mount. Benign re-mount is idempotent; detach (stop+disable) is destructive and routes through the gate.
• Host-reboot remount re-resolution (v0.37.0): at startup (before binding drives into the guest) and on the periodic tick, ReassertEnrolledMounts re-asserts every enrolled .mount unit that isn't currently mounted: it re-resolves the drive's uuid:<fs-uuid> durable id to its current /dev node by re-scanning /dev/disk/by-uuid (ResolveStorageDevice, never a cached node) and re-runs the idempotent enable --now. This re-enables a unit a prior detach left disabled AND tolerates kernel re-enumeration moving a drive's letter (/dev/sdb→sdc) — the reshuffle is a no-op. Already-mounted drives and genuinely-absent UUIDs are skipped.
• Every argument is validated before any command is constructed (validate.go): UUIDs against a strict hex regex, mount paths confined + traversal-checked, SMART devices whitelisted to raw disks, LVM names charset-checked. The adversarial matrix in validate_test.go proves a hostile UUID / path / device is refused with zero exec.
• SMART (smart.go) fills StorageTarget.smart via smartctl -a -j — SATA and NVMe attribute sets, degrading to UNKNOWN for devices that expose no SMART (e.g. a USB bridge). lvs fills the lvmthin thin-pool metadata fill (metadata exhaustion corrupts a pool like data exhaustion).
• The watchdog gains a benign re-mount response: when a known mount-backed target's device returns unmounted, it dispatches (off the poll path) a by-UUID re-mount, routed through the gate as benign. The disk-grow executor (pct resize, grow-only) lands in internal/reconcile as a benign action; destructive storage ops (detach/wipe/ data-losing-resize) construct a ClassStorageWipe/ClassDecommission intent bound to the storage target identity and go through the slice-4 gate (built + tested, inert live).

--selftest=storage (live storage harness)

Runs standalone on the Proxmox host (no hub needed):

• bare: an observe pass printing the full StorageTarget table incl. the SMART summary and thin-pool data+metadata fill.
• -watch <dur> (e.g. --selftest=storage -watch 3m): runs the watchdog verbose for the window with the re-mount response live, so an operator can physically cycle a drive and watch detect → report → re-mount in the logs.

The proxmox package — model

Two backends, one fixed routing policy (the fence is structural — Client never shells out, Privileged never makes an HTTP call; asserted in routing_test.go):

	Backend	Used for
API (default)	proxmox.Client	everything the scoped FelhomAgent token can do
root-CLI (fenced)	proxmox.Privileged	the three proven OS-root exceptions only

Grounded entirely in the spike findings (felhom.eu/documentation/proxmox-platform.md, tests/phase{0,1-2,3}-findings.md). Every mutating API op is async: it returns a UPID and the caller WaitTasks until the task stops, then asserts exitstatus == "OK" — authorization can surface at task execution, not the HTTP POST (phase1-2 §1.3).

Public surface

Client (API):

• Read: Version, Nodes, NodeStatus, ListLXC, GuestStatus, GuestConfig, ListStorage, NodeStorage, StorageContent.
• Async mutating (return UPID): RestoreLXC (primary create path), Vzdump, Snapshot, Rollback, DeleteSnapshot, SetConfig, Start, Stop.
• Tasks: WaitTask, TaskStatusOnce, TaskLogTail.
• Errors: *APIError (parses the offending privilege from a 403), *TaskError (parses it from a failed task exitstatus).

Privileged (fenced root-CLI) — each method documents why it can't be the API:

• CreateGoldenLXC — pct create with keyctl=1 (root@pam-only; the only root-fenced create — the per-customer path provisions by restore, which preserves keyctl).
• MountUSBByUUID — host mount-by-UUID (not a Proxmox API op).
• SMART, Sensors — hardware reads (not API-exposed).

API-vs-root routing table

See the table in internal/proxmox/doc.go. Summary: the entire guest lifecycle including restore is API-token-covered; OS-root is confined to golden-image keyctl create, host mounts, and SMART/sensors (phase3 §B3).

Controller swap (agentic controller update, Phase 1 — v0.42.0)

The local API (internal/localapi/) owns the in-guest controller image swap — the new-architecture replacement for the controller's dead in-container docker compose self-update. The in-guest controller pre-pulls the target image then calls the agent:

• POST /controller/swap {image} (withGuest-scoped) → 202, then async: record previous (/var/lib/felhom-agent/controller-swap-<vmid>.json) → confirm the target is present in the guest → write /etc/felhom-controller-image → systemctl restart felhom-controller-bootstrap.service → poll the new controller to healthy (docker inspect, ≤90s) → roll back to the previous image if not (the guest is never left without a controller). Strict image-ref gate; single-flight per guest (409).
  • (v0.47.0) Verify hardening (F1): the health poll reads {{.RestartCount}} (a 4th inspect field) — running && RestartCount>0 is not healthy (a process that already crash-restarted isn't stably up), and a no-healthcheck image must report ok on 3 consecutive polls (a stability dwell) before acceptance; a real healthy result is trusted immediately. This closes the hole where a no-HEALTHCHECK image that crash-loops could land one "Running" instant and false-pass the point-in-time check → no rollback. Verify predicate only — the rollback orchestration + the docker inspect -f * grant (the * spans the extended template) are unchanged.
• GET /controller/swap/status → {state, current, previous, target, error}.

The agent is external to the controller container, so it survives the controller being killed mid-swap (which the controller cannot do to itself). GuestBinder.GuestExec is the single pct exec seam. Exercise directly with --selftest=controller-swap -vmid <id> -image <ref>.

Agent self-update (operator-signed, A/B slots, crash-loop auto-rollback — v0.70.0, TASK D1)

The agent updates ITSELF the same way it swaps the controller: the thing that performs rollback is never the thing being updated. For the agent that means systemd + an ~80-line root shell wrapper (configs/felhom-selfupdate-guarded) that changes almost never; the Go binary is what flips.

Trust model. An update is an operator-signed agent_update op delivered through the existing signed-jobs pipeline (same LOCKED authz gate as storage_wipe/decommission). The signed params pin the exact version + sha256, so the pinned sha is the only integrity root — neither a compromised hub (dumb transport) nor a compromised Gitea (dumb storage) can substitute a binary. The operator signs offline with felhom-opsign -op agent_update -agent-version <v> -sha256 <hex>.

The flow (internal/selfupdate/ = the Go half; the wrapper = the root half):

1. The control loop sees a pending signed op → the gate verifies it (pinned-key SSHSIG → namespace → allow-list → crypto → host → time → durable nonce-burn) → the agent_update executor runs.
2. Executor downloads the binary for the signed version from the config'd artifact host (selfupdate.url_template, {version} interpolated) to /var/lib/felhom-agent/selfupdate/, verifies its sha256 against the signed value (mismatch → refuse, remove, agent untouched), and hands it to sudo -n felhom-selfupdate-guarded apply <staged> <sha>. The job is completed on the hub after verify+download, before apply (the nonce is already burned — a queued job would only re-fetch and no-op on the spent nonce; a failed/rolled-back update is visible via the report).
3. The wrapper (as root) re-verifies the sha, confines the staged path to the staging dir, asserts same-filesystem (the atomic-rename guarantee), snapshots the current binary to .prev, atomically mvs the new binary into place, writes a pending.json marker, reset-faileds, and schedules a detached restart (systemd-run --on-active=2s … systemctl restart felhom-agent, so the caller survives to log the handoff).
4. The new binary boots; after it has run cleanly for a dwell (selfupdate.dwell_seconds, default 60) and core init is done, internal/selfupdate.Manager calls the wrapper's commit (clears the marker; .prev retained as a manual net). A pending marker naming a different version than the running binary is not committed — loud WARN, marker left so the report shows why (a human decides).
5. Crash-loop auto-rollback (the safety property). If the new binary crashes, systemd's OnFailure=felhom-agent-rollback.service (the felhom-agent-limits.conf drop-in) runs the wrapper's rollback: pending marker present → restore .prev byte-identical → clear marker → restart → the old binary is back within seconds of the first crash. On systemd 257 OnFailure= fires on every crash, so rollback triggers at the first one; the marker-guard makes every later fire (and any crash with no update in flight) a harmless no-op. The tuned start-limit ([Unit] StartLimitIntervalSec=120 + StartLimitBurst=4) is the terminal backstop (e.g. an environmental crash loop of the known-good binary → terminal failed ≈20s → the hub's host_staleness dead-man's-switch alerts the operator).

Design provenance: every systemd behaviour above is empirically validated in felhom.eu/documentation/audits/SPIKE-agent-selfupdate-2026-07-05.md (the SF-findings). The host report carries selfupdate_pending (+ version) so a runs-but-never-commits binary is visible even though it never crashes. v1 scope: no hub-floor auto-update, no auto-retry of a failed update, no pending-timeout auto-rollback (a stuck-but-alive binary is caught by host_staleness).

TLS trust

The host serves a self-signed cert. Verification is not blanket-disabled. Pick one in config: ca_file (PEM, full verify), fingerprint (SHA-256 of the host leaf cert — pinned exact-cert match; the /nodes API returns each node's ssl_fingerprint to pin), or the explicitly-named insecure_skip_verify (off by default; selftest-against-127.0.0.1 only).

Provisioning the token (out-of-band, operator side)

The agent only consumes a privilege-separated API token; role setup is a provisioning step. The role must be granted on both the user AND the token for the same path, or the intersection is empty and every call 403s (phase1-2 §1.2):

pveum role add FelhomAgent -privs "VM.Allocate VM.Audit VM.Config.Disk VM.Config.CPU \
  VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Snapshot \
  VM.Snapshot.Rollback VM.Backup Datastore.Allocate Datastore.AllocateSpace \
  Datastore.Audit Sys.Audit SDN.Use"          # 16 privileges, validated Phase 3 B3
pveum user add felhom-agent@pve
pveum user token add felhom-agent@pve agent --privsep 1   # capture the secret (shown once)
pveum acl modify / -user  'felhom-agent@pve'       -role FelhomAgent
pveum acl modify / -token 'felhom-agent@pve!agent' -role FelhomAgent

(VM.Config.CPUMemory is not a real privilege; SDN.Use is required for bridge use.)

Run

go build ./...
# read-only health check against the host:
./felhom-agent --config configs/agent.example.json --selftest
# or via env (keeps the secret off disk):
FELHOM_AGENT_PROXMOX_TOKEN='felhom-agent@pve!agent=SECRET' \
FELHOM_AGENT_PROXMOX_NODE=demo-felhom \
FELHOM_AGENT_PROXMOX_ENDPOINT=https://192.168.0.162:8006 \
FELHOM_AGENT_PROXMOX_TLS_FINGERPRINT='BA:7C:...:CF' \
  ./felhom-agent --selftest

--selftest (read-only) loads config, builds the API client, and runs the read queries (version, nodes, node status, guests, storage), printing a short health report. It mutates nothing and says so cleanly if the token/endpoint isn't configured.

--selftest=task --vmid N (explicitly gated) exercises WaitTask on a reversible op (snapshot → rollback → delete-snapshot) against guest N. Default --selftest never mutates.

--selftest=bring-up|provision accept an optional operator CPU/RAM cap: -cores N and -memory M (MiB). Both default to 0 = keep the golden's baked sizes. When set, the cap is written into the SAME pre-start config PUT as the identity reset (via BringUpSpec.Cores/MemoryMB), so the guest never boots uncapped — useful when the appliance shares a host with other guests.

Pool-scoped restore (v0.53.0): every restore (provision bring-up, DR, and restore-test) allocates the guest INTO the felhom PVE pool (reconcile.DefaultPool; RestoreLXCOptions.Pool → pct restore --pool). This is what lets the agent token be scoped to /pool/felhom + /storage/<targets> instead of / (blast-radius containment on a shared host) — the restore is how a fresh vmid is allocated under that scoped token (VM.Allocate+Pool.Allocate at /pool/felhom). Layout + validation: felhom.eu/documentation/audits/SPIKE-pool-scoped-acl-2026-07-01.md.

Drive discovery + tracking (v0.55.0, Impl-2a): GET /disks/candidates lists host disks the Impl-1 filter proves are free to enroll (split initialize/attach). The watchdog's known-drive set is sourced from the intent registry + Felhom .mount units (RegistryKnownTargets), NOT PVE storages — so a drive enrolled with no PVE dir-storage is still health-tracked (Observe() stays for real PVE storages). The controller wizard consuming this is Impl-2b.

Format safety (v0.54.0, Impl-1; hardened v0.61.0, audit D1/D2/D3): Format (mkfs) is gated by a mandatory unclaimed-disk guard (internal/storage/claim.go) — it refuses any device not provably free for Felhom (OS disk, LVM PV, ZFS/mdraid member, foreign mount, read-only; fail-safe on any read error, and on an empty/target-absent lsblk topology — D2), independent of DataBearing. Below the agent, mkfs runs ONLY through configs/felhom-mkfs-guarded.sh (the sole mkfs the sudoers permits), which re-checks the catastrophic cases as root: system disk, LVM PV (absolute-path pvs), foreign mount, read-only device, and any LVM/ZFS/mdraid/LUKS/swap member signature (D1 — validated by scripts/mkfs-guarded-harness.sh, a loop-device + recorder harness). The blank-format local-API path binds to the device's durable id and anti-retarget re-resolves before mkfs, same as the confirmed wipe (D3, AGENT-001's benign-branch twin). The pool-scoped token does NOT touch mkfs (sudo op) — the filter + wrapper are the guard. See SPIKE-drive-enrollment-2026-07-01.md and felhom.eu/documentation/audits/AUDIT-blast-radius-hostroot-localapi-2026-07-02.md.

Process model

Native Go binary, systemd service, non-root felhom-agent service user holding the scoped token, with a narrow sudoers allowlist for the fenced host-root ops. privileged.mode: "sudo" matches this; "direct" is for dev/CI where the agent is already root.

The canonical artifacts (BUNDLE slice):

• configs/felhom-agent.service — the canonical unit (User=felhom-agent, ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json). It deliberately sets no NoNewPrivileges and no mount-namespacing hardening (ProtectHome/PrivateTmp/…): the first would block the sudo the agent needs, the second would put the agent in a private mount namespace so its mount --bind drive enrollments wouldn't propagate into the running guest. The security boundary is the sudoers allowlist, not systemd sandboxing.
• configs/felhom-agent.sudoers → /etc/sudoers.d/felhom-agent (0440, visudo -cf-validated).
• scripts/publish-agent.sh publishes the binary to Gitea as a generic package (/api/packages/admin/generic/felhom-agent/<ver>/felhom-agent), printing the sha256 the operator records in the hub artifact manifest.

Install is automated. The host-bootstrap script (felhom.eu/scripts/felhom-host-install.sh) fetches the binary from Gitea, verifies its sha256 against the hub-vouched manifest, then installs the user + binary + sudoers + unit + config — no manual agent install step.

Test

go vet ./... && go test ./...

Unit tests use a mock HTTP transport + mock runner (no live host): UPID parse, WaitTask (running→OK / running→failed-403 / timeout / ctx-cancel), 403→privilege-named error, response decoding against the captured live shapes, and the API-vs-root routing fence.