felhom-agent

The host agent for the Felhom platform — the operator-tier component that runs on each Proxmox host and owns all Proxmox interaction (provision/restore guests, host storage, backups, host+tunnel monitoring, hub control loop, per-guest local API). Design: felhom.eu/documentation/architecture/03-host-agent.md.

Status — slice 1 of N. This repo currently contains the project scaffold and the internal/proxmox interaction layer (the typed library every other module will call to talk to Proxmox), plus a runnable read-only --selftest. No reconcile loop, hub client, signing, or storage/backup orchestration yet — those are later slices.

Module: gitea.dooplex.hu/admin/felhom-agent · binary: felhom-agent · Go 1.24.

Layout

cmd/felhom-agent/      # entry point + --selftest modes + the daemon (poll loop + reconcile + watchdog)
internal/proxmox/      # the Proxmox interaction layer (API-first + fenced root-CLI)
internal/config/       # JSON config + env overrides (secrets never logged)
internal/log/          # slog setup
internal/authz/        # operator signed-op verifier (SSHSIG); durable nonce store
internal/hub/          # daemon: host-report collector + Bearer client + resilient poll loop
internal/reconcile/    # reconcile engine + reversibility gate + op journal + crash recovery
internal/storage/      # storage-target observer + durable_id + fast-poll watchdog (slice 5)
internal/capability/   # privileged-capability self-probe (manifest + sudo -n -l check, v0.44.0)
configs/agent.example.json

The capability package — privileged self-check (v0.44.0)

The non-root agent depends on a fixed set of sudo -n grants (configs/felhom-agent.sudoers). When the 2026-06-28 root→non-root cutover silently dropped some (lxc-info, make-private, …), features broke unnoticed until a user hit them. internal/capability makes that loud:

• Manifest() is the source of truth for the required (binary, representative-arg) vectors, seeded from the sudoers audit (the OK + CLOSED rows; the surfaced/deferred rows are excluded). When you add a privileged runner.Run call that needs a NEW grant, add it to BOTH the sudoers file AND the manifest — manifest_test.go asserts manifest ⊆ sudoers at build time (with a red-proof that dropping the lxc-info grant fails the gate), so a missing grant is caught in CI, not in prod.
• Prober.Probe lists each vector with sudo -n -l (a policy LIST — never executes, safe for mkfs/pct) + an os.Stat existence check → an ok/degraded snapshot. It runs at startup (logged loud) and on every collect, riding the hub report as HostReport.Capabilities; the hub (HostCapabilityChecker) alerts the operator on a Critical capability going degraded. Serve-degraded — the probe never blocks startup. (Next self-health slice: the controller↔agent channel check.)

Controller-swap under non-root (v0.45.0). The agent-owned controller image swap (internal/localapi/controllerswap.go) no longer shells out: writeImage pipes the image ref on stdin into an in-guest tee /etc/felhom-controller-image (via GuestExecStdin → Runner.RunStdin, the same fenced sudo -n runner) — no bash -c, no interpolation. Its 5 narrow grants live in the FELHOM_CONTROLLERSWAP sudoers alias (all read-only or fixed-target; the tee target is the FIXED image path, content stdin-fed) and in the capability manifest (Critical), so a dropped grant is a build failure + a live degraded signal. No general pct exec is granted.

The storage package — observe + watchdog (slice 5)

Read-only this slice (no hub desired-state until slice 10):

• Observer builds the host-report's storage_targets from ListStorage/NodeStorage joined with non-privileged host reads (/proc/mounts, /dev/disk/by-uuid, /sys/.../rotational). It reports each target's durable_id (the DR-load-bearing re-attach key: fs-UUID for usb/local-dir, server:export for nfs/cifs, repo+fingerprint for pbs, vg/pool for lvmthin), state, usage, a rotational class hint (never authoritative — class is hub-owned), and the lvmthin thin-pool data fill (a full pool corrupts every guest on it). SMART is a Phase-B privileged read.
• Watchdog is the third daemon goroutine: a fast poll (seconds) over the known target set that detects an attached↔disconnected transition and fires a debounced, out-of-band host-report so the hub learns of a USB drop in seconds rather than at the ~15-minute cycle. It mutates nothing (the benign re-mount-by-UUID response lands in Phase B). The HostReader seam keeps it root-free and unit-testable with no real devices.

The reported StorageTarget shape is a cross-repo contract duplicated in felhom.eu/hub; internal/hub/testdata/host-report.golden.json is byte-identical with the hub's copy and a bidirectional key-set test guards drift.

DR recipe — the storage/guest/PBS half (v0.38.0)

The host-report carries an additive dr_recipe section (internal/hub/dr_recipe.go) — the agent half of the secret-free reconstruction recipe (SPIKE-dr-recipe-2026-06-16.md). It is the non-secret re-provision scaffolding that complements escrow (keys) + PBS/restic (bytes): guests[] sizing, drives[] (user-data drives by durable_id → role → mount → intent), pve_storage[] (the storage.cfg defs), and pbs coordinates. Built by the pure BuildDRRecipeHostHalf from facts the report already collects — no new reads. Boundary: every field is an identifier/intent/size/ coordinate — never a key/password/token/hash/ENC:; the PBS key + restic password stay in escrow. recipe_version=1, ignore-unknown on read, pinned in the cross-repo golden. The hub assembles it with the controller's app half into one customer recipe.

The privileged HostOps surface (slice 5 Phase B)

The write side — the one place the agent steps outside its Proxmox API token into OS-root — is isolated behind the HostOps seam (hostops.go): production SudoHostOps shells out via a narrow sudoers allowlist (configs/felhom-agent.sudoers) with fixed argument vectors and no shell; tests use a fake (no real root in the suite).

• Persistent mounts are systemd .mount units keyed by fs-UUID (What=/dev/disk/by-uuid/<UUID>, enabled so they survive reboot) — not raw fstab or a transient mount. Benign re-mount is idempotent; detach (stop+disable) is destructive and routes through the gate.
• Host-reboot remount re-resolution (v0.37.0): at startup (before binding drives into the guest) and on the periodic tick, ReassertEnrolledMounts re-asserts every enrolled .mount unit that isn't currently mounted: it re-resolves the drive's uuid:<fs-uuid> durable id to its current /dev node by re-scanning /dev/disk/by-uuid (ResolveStorageDevice, never a cached node) and re-runs the idempotent enable --now. This re-enables a unit a prior detach left disabled AND tolerates kernel re-enumeration moving a drive's letter (/dev/sdb→sdc) — the reshuffle is a no-op. Already-mounted drives and genuinely-absent UUIDs are skipped.
• Every argument is validated before any command is constructed (validate.go): UUIDs against a strict hex regex, mount paths confined + traversal-checked, SMART devices whitelisted to raw disks, LVM names charset-checked. The adversarial matrix in validate_test.go proves a hostile UUID / path / device is refused with zero exec.
• SMART (smart.go) fills StorageTarget.smart via smartctl -a -j — SATA and NVMe attribute sets, degrading to UNKNOWN for devices that expose no SMART (e.g. a USB bridge). lvs fills the lvmthin thin-pool metadata fill (metadata exhaustion corrupts a pool like data exhaustion).
• The watchdog gains a benign re-mount response: when a known mount-backed target's device returns unmounted, it dispatches (off the poll path) a by-UUID re-mount, routed through the gate as benign. The disk-grow executor (pct resize, grow-only) lands in internal/reconcile as a benign action; destructive storage ops (detach/wipe/ data-losing-resize) construct a ClassStorageWipe/ClassDecommission intent bound to the storage target identity and go through the slice-4 gate (built + tested, inert live).

--selftest=storage (live storage harness)

Runs standalone on the Proxmox host (no hub needed):

• bare: an observe pass printing the full StorageTarget table incl. the SMART summary and thin-pool data+metadata fill.
• -watch <dur> (e.g. --selftest=storage -watch 3m): runs the watchdog verbose for the window with the re-mount response live, so an operator can physically cycle a drive and watch detect → report → re-mount in the logs.

The proxmox package — model

Two backends, one fixed routing policy (the fence is structural — Client never shells out, Privileged never makes an HTTP call; asserted in routing_test.go):

	Backend	Used for
API (default)	proxmox.Client	everything the scoped FelhomAgent token can do
root-CLI (fenced)	proxmox.Privileged	the three proven OS-root exceptions only

Grounded entirely in the spike findings (felhom.eu/documentation/proxmox-platform.md, tests/phase{0,1-2,3}-findings.md). Every mutating API op is async: it returns a UPID and the caller WaitTasks until the task stops, then asserts exitstatus == "OK" — authorization can surface at task execution, not the HTTP POST (phase1-2 §1.3).

Public surface

Client (API):

• Read: Version, Nodes, NodeStatus, ListLXC, GuestStatus, GuestConfig, ListStorage, NodeStorage, StorageContent.
• Async mutating (return UPID): RestoreLXC (primary create path), Vzdump, Snapshot, Rollback, DeleteSnapshot, SetConfig, Start, Stop.
• Tasks: WaitTask, TaskStatusOnce, TaskLogTail.
• Errors: *APIError (parses the offending privilege from a 403), *TaskError (parses it from a failed task exitstatus).

Privileged (fenced root-CLI) — each method documents why it can't be the API:

• CreateGoldenLXC — pct create with keyctl=1 (root@pam-only; the only root-fenced create — the per-customer path provisions by restore, which preserves keyctl).
• MountUSBByUUID — host mount-by-UUID (not a Proxmox API op).
• SMART, Sensors — hardware reads (not API-exposed).

API-vs-root routing table

See the table in internal/proxmox/doc.go. Summary: the entire guest lifecycle including restore is API-token-covered; OS-root is confined to golden-image keyctl create, host mounts, and SMART/sensors (phase3 §B3).

Controller swap (agentic controller update, Phase 1 — v0.42.0)

The local API (internal/localapi/) owns the in-guest controller image swap — the new-architecture replacement for the controller's dead in-container docker compose self-update. The in-guest controller pre-pulls the target image then calls the agent:

• POST /controller/swap {image} (withGuest-scoped) → 202, then async: record previous (/var/lib/felhom-agent/controller-swap-<vmid>.json) → confirm the target is present in the guest → write /etc/felhom-controller-image → systemctl restart felhom-controller-bootstrap.service → poll the new controller to healthy (docker inspect, ≤90s) → roll back to the previous image if not (the guest is never left without a controller). Strict image-ref gate; single-flight per guest (409).
  • (v0.47.0) Verify hardening (F1): the health poll reads {{.RestartCount}} (a 4th inspect field) — running && RestartCount>0 is not healthy (a process that already crash-restarted isn't stably up), and a no-healthcheck image must report ok on 3 consecutive polls (a stability dwell) before acceptance; a real healthy result is trusted immediately. This closes the hole where a no-HEALTHCHECK image that crash-loops could land one "Running" instant and false-pass the point-in-time check → no rollback. Verify predicate only — the rollback orchestration + the docker inspect -f * grant (the * spans the extended template) are unchanged.
• GET /controller/swap/status → {state, current, previous, target, error}.

The agent is external to the controller container, so it survives the controller being killed mid-swap (which the controller cannot do to itself). GuestBinder.GuestExec is the single pct exec seam. Exercise directly with --selftest=controller-swap -vmid <id> -image <ref>.

TLS trust

The host serves a self-signed cert. Verification is not blanket-disabled. Pick one in config: ca_file (PEM, full verify), fingerprint (SHA-256 of the host leaf cert — pinned exact-cert match; the /nodes API returns each node's ssl_fingerprint to pin), or the explicitly-named insecure_skip_verify (off by default; selftest-against-127.0.0.1 only).

Provisioning the token (out-of-band, operator side)

The agent only consumes a privilege-separated API token; role setup is a provisioning step. The role must be granted on both the user AND the token for the same path, or the intersection is empty and every call 403s (phase1-2 §1.2):

pveum role add FelhomAgent -privs "VM.Allocate VM.Audit VM.Config.Disk VM.Config.CPU \
  VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Snapshot \
  VM.Snapshot.Rollback VM.Backup Datastore.Allocate Datastore.AllocateSpace \
  Datastore.Audit Sys.Audit SDN.Use"          # 16 privileges, validated Phase 3 B3
pveum user add felhom-agent@pve
pveum user token add felhom-agent@pve agent --privsep 1   # capture the secret (shown once)
pveum acl modify / -user  'felhom-agent@pve'       -role FelhomAgent
pveum acl modify / -token 'felhom-agent@pve!agent' -role FelhomAgent

(VM.Config.CPUMemory is not a real privilege; SDN.Use is required for bridge use.)

Run

go build ./...
# read-only health check against the host:
./felhom-agent --config configs/agent.example.json --selftest
# or via env (keeps the secret off disk):
FELHOM_AGENT_PROXMOX_TOKEN='felhom-agent@pve!agent=SECRET' \
FELHOM_AGENT_PROXMOX_NODE=demo-felhom \
FELHOM_AGENT_PROXMOX_ENDPOINT=https://192.168.0.162:8006 \
FELHOM_AGENT_PROXMOX_TLS_FINGERPRINT='BA:7C:...:CF' \
  ./felhom-agent --selftest

--selftest (read-only) loads config, builds the API client, and runs the read queries (version, nodes, node status, guests, storage), printing a short health report. It mutates nothing and says so cleanly if the token/endpoint isn't configured.

--selftest=task --vmid N (explicitly gated) exercises WaitTask on a reversible op (snapshot → rollback → delete-snapshot) against guest N. Default --selftest never mutates.

--selftest=bring-up|provision accept an optional operator CPU/RAM cap: -cores N and -memory M (MiB). Both default to 0 = keep the golden's baked sizes. When set, the cap is written into the SAME pre-start config PUT as the identity reset (via BringUpSpec.Cores/MemoryMB), so the guest never boots uncapped — useful when the appliance shares a host with other guests.

Pool-scoped restore (v0.53.0): every restore (provision bring-up, DR, and restore-test) allocates the guest INTO the felhom PVE pool (reconcile.DefaultPool; RestoreLXCOptions.Pool → pct restore --pool). This is what lets the agent token be scoped to /pool/felhom + /storage/<targets> instead of / (blast-radius containment on a shared host) — the restore is how a fresh vmid is allocated under that scoped token (VM.Allocate+Pool.Allocate at /pool/felhom). Layout + validation: felhom.eu/documentation/audits/SPIKE-pool-scoped-acl-2026-07-01.md.

Drive discovery + tracking (v0.55.0, Impl-2a): GET /disks/candidates lists host disks the Impl-1 filter proves are free to enroll (split initialize/attach). The watchdog's known-drive set is sourced from the intent registry + Felhom .mount units (RegistryKnownTargets), NOT PVE storages — so a drive enrolled with no PVE dir-storage is still health-tracked (Observe() stays for real PVE storages). The controller wizard consuming this is Impl-2b.

Format safety (v0.54.0, Impl-1): Format (mkfs) is gated by a mandatory unclaimed-disk guard (internal/storage/claim.go) — it refuses any device not provably free for Felhom (OS disk, LVM PV, ZFS/mdraid member, foreign mount, read-only; fail-safe on any read error), independent of DataBearing. Below the agent, mkfs runs ONLY through configs/felhom-mkfs-guarded.sh (the sole mkfs the sudoers permits), which re-checks the catastrophic cases as root. The pool-scoped token does NOT touch mkfs (sudo op) — the filter + wrapper are the guard. See SPIKE-drive-enrollment-2026-07-01.md.

Process model

Native Go binary, systemd service, non-root felhom-agent service user holding the scoped token, with a narrow sudoers allowlist for the fenced host-root ops. privileged.mode: "sudo" matches this; "direct" is for dev/CI where the agent is already root.

The canonical artifacts (BUNDLE slice):

• configs/felhom-agent.service — the canonical unit (User=felhom-agent, ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json). It deliberately sets no NoNewPrivileges and no mount-namespacing hardening (ProtectHome/PrivateTmp/…): the first would block the sudo the agent needs, the second would put the agent in a private mount namespace so its mount --bind drive enrollments wouldn't propagate into the running guest. The security boundary is the sudoers allowlist, not systemd sandboxing.
• configs/felhom-agent.sudoers → /etc/sudoers.d/felhom-agent (0440, visudo -cf-validated).
• scripts/publish-agent.sh publishes the binary to Gitea as a generic package (/api/packages/admin/generic/felhom-agent/<ver>/felhom-agent), printing the sha256 the operator records in the hub artifact manifest.

Install is automated. The host-bootstrap script (felhom.eu/scripts/felhom-host-install.sh) fetches the binary from Gitea, verifies its sha256 against the hub-vouched manifest, then installs the user + binary + sudoers + unit + config — no manual agent install step.

Test

go vet ./... && go test ./...

Unit tests use a mock HTTP transport + mock runner (no live host): UPID parse, WaitTask (running→OK / running→failed-403 / timeout / ctx-cancel), 403→privilege-named error, response decoding against the captured live shapes, and the API-vs-root routing fence.