felhom-agent

The host agent for the Felhom platform — the operator-tier component that runs on each Proxmox host and owns all Proxmox interaction (provision/restore guests, host storage, backups, host+tunnel monitoring, hub control loop, per-guest local API). Design: felhom.eu/documentation/architecture/03-host-agent.md.

Status — slice 1 of N. This repo currently contains the project scaffold and the internal/proxmox interaction layer (the typed library every other module will call to talk to Proxmox), plus a runnable read-only --selftest. No reconcile loop, hub client, signing, or storage/backup orchestration yet — those are later slices.

Module: gitea.dooplex.hu/admin/felhom-agent · binary: felhom-agent · Go 1.24.

Layout

cmd/felhom-agent/      # entry point + --selftest modes + the daemon (poll loop + reconcile + watchdog)
internal/proxmox/      # the Proxmox interaction layer (API-first + fenced root-CLI)
internal/config/       # JSON config + env overrides (secrets never logged)
internal/log/          # slog setup
internal/authz/        # operator signed-op verifier (SSHSIG); durable nonce store
internal/hub/          # daemon: host-report collector + Bearer client + resilient poll loop
internal/reconcile/    # reconcile engine + reversibility gate + op journal + crash recovery
internal/storage/      # storage-target observer + durable_id + fast-poll watchdog (slice 5)
configs/agent.example.json

The storage package — observe + watchdog (slice 5)

Read-only this slice (no hub desired-state until slice 10):

• Observer builds the host-report's storage_targets from ListStorage/NodeStorage joined with non-privileged host reads (/proc/mounts, /dev/disk/by-uuid, /sys/.../rotational). It reports each target's durable_id (the DR-load-bearing re-attach key: fs-UUID for usb/local-dir, server:export for nfs/cifs, repo+fingerprint for pbs, vg/pool for lvmthin), state, usage, a rotational class hint (never authoritative — class is hub-owned), and the lvmthin thin-pool data fill (a full pool corrupts every guest on it). SMART is a Phase-B privileged read.
• Watchdog is the third daemon goroutine: a fast poll (seconds) over the known target set that detects an attached↔disconnected transition and fires a debounced, out-of-band host-report so the hub learns of a USB drop in seconds rather than at the ~15-minute cycle. It mutates nothing (the benign re-mount-by-UUID response lands in Phase B). The HostReader seam keeps it root-free and unit-testable with no real devices.

The reported StorageTarget shape is a cross-repo contract duplicated in felhom.eu/hub; internal/hub/testdata/host-report.golden.json is byte-identical with the hub's copy and a bidirectional key-set test guards drift.

The privileged HostOps surface (slice 5 Phase B)

The write side — the one place the agent steps outside its Proxmox API token into OS-root — is isolated behind the HostOps seam (hostops.go): production SudoHostOps shells out via a narrow sudoers allowlist (configs/felhom-agent.sudoers) with fixed argument vectors and no shell; tests use a fake (no real root in the suite).

• Persistent mounts are systemd .mount units keyed by fs-UUID (What=/dev/disk/by-uuid/<UUID>, enabled so they survive reboot) — not raw fstab or a transient mount. Benign re-mount is idempotent; detach (stop+disable) is destructive and routes through the gate.
• Every argument is validated before any command is constructed (validate.go): UUIDs against a strict hex regex, mount paths confined + traversal-checked, SMART devices whitelisted to raw disks, LVM names charset-checked. The adversarial matrix in validate_test.go proves a hostile UUID / path / device is refused with zero exec.
• SMART (smart.go) fills StorageTarget.smart via smartctl -a -j — SATA and NVMe attribute sets, degrading to UNKNOWN for devices that expose no SMART (e.g. a USB bridge). lvs fills the lvmthin thin-pool metadata fill (metadata exhaustion corrupts a pool like data exhaustion).
• The watchdog gains a benign re-mount response: when a known mount-backed target's device returns unmounted, it dispatches (off the poll path) a by-UUID re-mount, routed through the gate as benign. The disk-grow executor (pct resize, grow-only) lands in internal/reconcile as a benign action; destructive storage ops (detach/wipe/ data-losing-resize) construct a ClassStorageWipe/ClassDecommission intent bound to the storage target identity and go through the slice-4 gate (built + tested, inert live).

--selftest=storage (live storage harness)

Runs standalone on the Proxmox host (no hub needed):

• bare: an observe pass printing the full StorageTarget table incl. the SMART summary and thin-pool data+metadata fill.
• -watch <dur> (e.g. --selftest=storage -watch 3m): runs the watchdog verbose for the window with the re-mount response live, so an operator can physically cycle a drive and watch detect → report → re-mount in the logs.

The proxmox package — model

Two backends, one fixed routing policy (the fence is structural — Client never shells out, Privileged never makes an HTTP call; asserted in routing_test.go):

	Backend	Used for
API (default)	proxmox.Client	everything the scoped FelhomAgent token can do
root-CLI (fenced)	proxmox.Privileged	the three proven OS-root exceptions only

Grounded entirely in the spike findings (felhom.eu/documentation/proxmox-platform.md, tests/phase{0,1-2,3}-findings.md). Every mutating API op is async: it returns a UPID and the caller WaitTasks until the task stops, then asserts exitstatus == "OK" — authorization can surface at task execution, not the HTTP POST (phase1-2 §1.3).

Public surface

Client (API):

• Read: Version, Nodes, NodeStatus, ListLXC, GuestStatus, GuestConfig, ListStorage, NodeStorage, StorageContent.
• Async mutating (return UPID): RestoreLXC (primary create path), Vzdump, Snapshot, Rollback, DeleteSnapshot, SetConfig, Start, Stop.
• Tasks: WaitTask, TaskStatusOnce, TaskLogTail.
• Errors: *APIError (parses the offending privilege from a 403), *TaskError (parses it from a failed task exitstatus).

Privileged (fenced root-CLI) — each method documents why it can't be the API:

• CreateGoldenLXC — pct create with keyctl=1 (root@pam-only; the only root-fenced create — the per-customer path provisions by restore, which preserves keyctl).
• MountUSBByUUID — host mount-by-UUID (not a Proxmox API op).
• SMART, Sensors — hardware reads (not API-exposed).

API-vs-root routing table

See the table in internal/proxmox/doc.go. Summary: the entire guest lifecycle including restore is API-token-covered; OS-root is confined to golden-image keyctl create, host mounts, and SMART/sensors (phase3 §B3).

TLS trust

The host serves a self-signed cert. Verification is not blanket-disabled. Pick one in config: ca_file (PEM, full verify), fingerprint (SHA-256 of the host leaf cert — pinned exact-cert match; the /nodes API returns each node's ssl_fingerprint to pin), or the explicitly-named insecure_skip_verify (off by default; selftest-against-127.0.0.1 only).

Provisioning the token (out-of-band, operator side)

The agent only consumes a privilege-separated API token; role setup is a provisioning step. The role must be granted on both the user AND the token for the same path, or the intersection is empty and every call 403s (phase1-2 §1.2):

pveum role add FelhomAgent -privs "VM.Allocate VM.Audit VM.Config.Disk VM.Config.CPU \
  VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Snapshot \
  VM.Snapshot.Rollback VM.Backup Datastore.Allocate Datastore.AllocateSpace \
  Datastore.Audit Sys.Audit SDN.Use"          # 16 privileges, validated Phase 3 B3
pveum user add felhom-agent@pve
pveum user token add felhom-agent@pve agent --privsep 1   # capture the secret (shown once)
pveum acl modify / -user  'felhom-agent@pve'       -role FelhomAgent
pveum acl modify / -token 'felhom-agent@pve!agent' -role FelhomAgent

(VM.Config.CPUMemory is not a real privilege; SDN.Use is required for bridge use.)

Run

go build ./...
# read-only health check against the host:
./felhom-agent --config configs/agent.example.json --selftest
# or via env (keeps the secret off disk):
FELHOM_AGENT_PROXMOX_TOKEN='felhom-agent@pve!agent=SECRET' \
FELHOM_AGENT_PROXMOX_NODE=demo-felhom \
FELHOM_AGENT_PROXMOX_ENDPOINT=https://192.168.0.162:8006 \
FELHOM_AGENT_PROXMOX_TLS_FINGERPRINT='BA:7C:...:CF' \
  ./felhom-agent --selftest

--selftest (read-only) loads config, builds the API client, and runs the read queries (version, nodes, node status, guests, storage), printing a short health report. It mutates nothing and says so cleanly if the token/endpoint isn't configured.

--selftest=task --vmid N (explicitly gated) exercises WaitTask on a reversible op (snapshot → rollback → delete-snapshot) against guest N. Default --selftest never mutates.

Process model (proposed, not finalized — see 03 §3/§12)

Native Go binary, systemd service, non-root service user holding the scoped token, with a narrow sudoers allowlist for the three fenced ops. privileged.mode: "sudo" matches this; "direct" is for dev/CI where the agent is already root.

Test

go vet ./... && go test ./...

Unit tests use a mock HTTP transport + mock runner (no live host): UPID parse, WaitTask (running→OK / running→failed-403 / timeout / ctx-cancel), 403→privilege-named error, response decoding against the captured live shapes, and the API-vs-root routing fence.