# felhom-agent

The **host agent** for the Felhom platform — the operator-tier component that runs on each
Proxmox host and owns *all* Proxmox interaction (provision/restore guests, host storage,
backups, host+tunnel monitoring, hub control loop, per-guest local API). Design:
[`felhom.eu/documentation/architecture/03-host-agent.md`](https://gitea.dooplex.hu/admin/felhom.eu/raw/branch/main/documentation/architecture/03-host-agent.md).

> **Status — slice 1 of N.** This repo currently contains the project scaffold and the
> **`internal/proxmox`** interaction layer (the typed library every other module will call to
> talk to Proxmox), plus a runnable read-only `--selftest`. **No** reconcile loop, hub client,
> signing, or storage/backup orchestration yet — those are later slices.

Module: `gitea.dooplex.hu/admin/felhom-agent` · binary: `felhom-agent` · Go 1.24.

## Layout

```
cmd/felhom-agent/      # entry point + --selftest modes + the daemon (poll loop + reconcile + watchdog)
internal/proxmox/      # the Proxmox interaction layer (API-first + fenced root-CLI)
internal/config/       # JSON config + env overrides (secrets never logged)
internal/log/          # slog setup
internal/authz/        # operator signed-op verifier (SSHSIG); durable nonce store
internal/hub/          # daemon: host-report collector + Bearer client + resilient poll loop
internal/reconcile/    # reconcile engine + reversibility gate + op journal + crash recovery
internal/storage/      # storage-target observer + durable_id + fast-poll watchdog (slice 5)
internal/capability/   # privileged-capability self-probe (manifest + sudo -n -l check, v0.44.0)
configs/agent.example.json
```

## The `capability` package — privileged self-check (v0.44.0)

The non-root agent depends on a fixed set of `sudo -n` grants (`configs/felhom-agent.sudoers`). When
the 2026-06-28 root→non-root cutover silently dropped some (lxc-info, make-private, …), features broke
unnoticed until a user hit them. `internal/capability` makes that loud:

- **`Manifest()` is the source of truth** for the required `(binary, representative-arg)` vectors,
  seeded from the sudoers audit (the OK + CLOSED rows; the surfaced/deferred rows are excluded). When
  you add a privileged `runner.Run` call that needs a NEW grant, add it to BOTH the sudoers file AND
  the manifest — `manifest_test.go` asserts **manifest ⊆ sudoers** at build time (with a red-proof
  that dropping the lxc-info grant fails the gate), so a missing grant is caught in CI, not in prod.
- **`Prober.Probe`** lists each vector with `sudo -n -l` (a policy LIST — never executes, safe for
  mkfs/pct) + an `os.Stat` existence check → an `ok`/`degraded` snapshot. It runs at startup (logged
  loud) and on every collect, riding the hub report as `HostReport.Capabilities`; the hub
  (`HostCapabilityChecker`) alerts the operator on a Critical capability going degraded. Serve-degraded
  — the probe never blocks startup. (Next self-health slice: the controller↔agent channel check.)

**Controller-swap under non-root (v0.45.0).** The agent-owned controller image swap
(`internal/localapi/controllerswap.go`) no longer shells out: `writeImage` pipes the image ref on
**stdin** into an in-guest `tee /etc/felhom-controller-image` (via `GuestExecStdin` →
`Runner.RunStdin`, the same fenced `sudo -n` runner) — no `bash -c`, no interpolation. Its 5 narrow
grants live in the `FELHOM_CONTROLLERSWAP` sudoers alias (all read-only or fixed-target; the `tee`
target is the FIXED image path, content stdin-fed) and in the capability manifest (Critical), so a
dropped grant is a build failure + a live degraded signal. No general `pct exec` is granted.

## The `storage` package — observe + watchdog (slice 5)

Read-only this slice (no hub desired-state until slice 10):

- **Observer** builds the host-report's `storage_targets` from `ListStorage`/`NodeStorage`
  joined with non-privileged host reads (`/proc/mounts`, `/dev/disk/by-uuid`,
  `/sys/.../rotational`). It reports each target's `durable_id` (the DR-load-bearing
  re-attach key: fs-UUID for usb/local-dir, `server:export` for nfs/cifs,
  `repo+fingerprint` for pbs, `vg/pool` for lvmthin), state, usage, a rotational
  **class hint** (never authoritative — class is hub-owned), and the lvmthin **thin-pool
  data fill** (a full pool corrupts every guest on it). SMART is a Phase-B privileged read.
- **Watchdog** is the third daemon goroutine: a fast poll (seconds) over the *known*
  target set that detects an `attached↔disconnected` transition and fires a **debounced,
  out-of-band** host-report so the hub learns of a USB drop in seconds rather than at the
  ~15-minute cycle. It mutates nothing (the benign re-mount-by-UUID response lands in
  Phase B). The `HostReader` seam keeps it root-free and unit-testable with no real devices.

The reported `StorageTarget` shape is a cross-repo contract duplicated in `felhom.eu/hub`;
`internal/hub/testdata/host-report.golden.json` is byte-identical with the hub's copy and a
bidirectional key-set test guards drift.

### DR recipe — the storage/guest/PBS half (v0.38.0)

The host-report carries an additive `dr_recipe` section (`internal/hub/dr_recipe.go`) — the agent half
of the secret-free reconstruction recipe (`SPIKE-dr-recipe-2026-06-16.md`). It is the non-secret
**re-provision scaffolding** that complements escrow (keys) + PBS/restic (bytes): `guests[]` sizing,
`drives[]` (user-data drives by `durable_id` → role → mount → intent), `pve_storage[]` (the
`storage.cfg` defs), and `pbs` coordinates. Built by the pure `BuildDRRecipeHostHalf` from facts the
report already collects — **no new reads**. **Boundary:** every field is an identifier/intent/size/
coordinate — never a key/password/token/hash/`ENC:`; the PBS key + restic password stay in escrow.
`recipe_version=1`, ignore-unknown on read, pinned in the cross-repo golden. The hub assembles it with
the controller's app half into one customer recipe.

### The privileged `HostOps` surface (slice 5 Phase B)

The write side — the one place the agent steps outside its Proxmox API token into OS-root —
is isolated behind the `HostOps` seam (`hostops.go`): production `SudoHostOps` shells out via
a narrow **sudoers allowlist** (`configs/felhom-agent.sudoers`) with **fixed argument vectors
and no shell**; tests use a fake (no real root in the suite).

- **Persistent mounts** are **systemd `.mount` units keyed by fs-UUID**
  (`What=/dev/disk/by-uuid/<UUID>`, enabled so they survive reboot) — not raw fstab or a
  transient `mount`. Benign re-mount is idempotent; **detach** (stop+disable) is destructive
  and routes through the gate.
- **Host-reboot remount re-resolution (v0.37.0):** at startup (before binding drives into the
  guest) and on the periodic tick, `ReassertEnrolledMounts` re-asserts every enrolled `.mount`
  unit that isn't currently mounted: it re-resolves the drive's `uuid:<fs-uuid>` durable id to
  its **current** `/dev` node by re-scanning `/dev/disk/by-uuid` (`ResolveStorageDevice`, never a
  cached node) and re-runs the idempotent `enable --now`. This re-enables a unit a prior detach
  left `disabled` AND tolerates kernel re-enumeration moving a drive's letter (`/dev/sdb`→`sdc`)
  — the reshuffle is a no-op. Already-mounted drives and genuinely-absent UUIDs are skipped.
- **Every argument is validated before any command is constructed** (`validate.go`): UUIDs
  against a strict hex regex, mount paths confined + traversal-checked, SMART devices
  whitelisted to raw disks, LVM names charset-checked. The adversarial matrix in
  `validate_test.go` proves a hostile UUID / path / device is refused with **zero** exec.
- **SMART** (`smart.go`) fills `StorageTarget.smart` via `smartctl -a -j` — SATA *and* NVMe
  attribute sets, degrading to `UNKNOWN` for devices that expose no SMART (e.g. a USB bridge).
  **`lvs`** fills the lvmthin thin-pool **metadata** fill (metadata exhaustion corrupts a pool
  like data exhaustion).
- The **watchdog** gains a benign **re-mount response**: when a known mount-backed target's
  device returns unmounted, it dispatches (off the poll path) a by-UUID re-mount, routed
  through the gate as benign. The **disk-grow executor** (`pct resize`, **grow-only**) lands
  in `internal/reconcile` as a benign action; **destructive storage ops** (detach/wipe/
  data-losing-resize) construct a `ClassStorageWipe`/`ClassDecommission` intent bound to the
  storage **target identity** and go through the slice-4 gate (built + tested, inert live).

### `--selftest=storage` (live storage harness)

Runs standalone on the Proxmox host (no hub needed):
- bare: an **observe pass** printing the full `StorageTarget` table incl. the SMART summary
  and thin-pool data+metadata fill.
- `-watch <dur>` (e.g. `--selftest=storage -watch 3m`): runs the watchdog verbose for the
  window with the **re-mount response live**, so an operator can physically cycle a drive and
  watch detect → report → re-mount in the logs.

## The `proxmox` package — model

Two backends, one fixed routing policy (the fence is structural — `Client` never shells out,
`Privileged` never makes an HTTP call; asserted in `routing_test.go`):

| | Backend | Used for |
|---|---|---|
| **API (default)** | `proxmox.Client` | everything the scoped **FelhomAgent** token can do |
| **root-CLI (fenced)** | `proxmox.Privileged` | the **three** proven OS-root exceptions only |

Grounded entirely in the spike findings (`felhom.eu/documentation/proxmox-platform.md`,
`tests/phase{0,1-2,3}-findings.md`). Every mutating API op is **async**: it returns a UPID and
the caller `WaitTask`s until the task stops, then asserts `exitstatus == "OK"` — authorization
can surface at task execution, not the HTTP POST (phase1-2 §1.3).

### Public surface

`Client` (API):

- Read: `Version`, `Nodes`, `NodeStatus`, `ListLXC`, `GuestStatus`, `GuestConfig`,
  `ListStorage`, `NodeStorage`, `StorageContent`.
- Async mutating (return UPID): `RestoreLXC` (primary create path), `Vzdump`, `Snapshot`,
  `Rollback`, `DeleteSnapshot`, `SetConfig`, `Start`, `Stop`.
- Tasks: `WaitTask`, `TaskStatusOnce`, `TaskLogTail`.
- Errors: `*APIError` (parses the offending privilege from a 403), `*TaskError` (parses it from
  a failed task `exitstatus`).

`Privileged` (fenced root-CLI) — each method documents *why it can't be the API*:

- `CreateGoldenLXC` — `pct create` with `keyctl=1` (root@pam-only; the only root-fenced create —
  the per-customer path provisions by **restore**, which preserves keyctl).
- `MountUSBByUUID` — host mount-by-UUID (not a Proxmox API op).
- `SMART`, `Sensors` — hardware reads (not API-exposed).

### API-vs-root routing table

See the table in [`internal/proxmox/doc.go`](internal/proxmox/doc.go). Summary: the entire guest
lifecycle **including restore** is API-token-covered; OS-root is confined to golden-image
`keyctl` create, host mounts, and SMART/sensors (phase3 §B3).

### Controller swap (agentic controller update, Phase 1 — v0.42.0)

The local API (`internal/localapi/`) owns the in-guest controller image **swap** — the new-architecture
replacement for the controller's dead in-container `docker compose` self-update. The in-guest controller
pre-pulls the target image then calls the agent:

- `POST /controller/swap {image}` (`withGuest`-scoped) → **202**, then async: record previous
  (`/var/lib/felhom-agent/controller-swap-<vmid>.json`) → confirm the target is present in the guest →
  write `/etc/felhom-controller-image` → `systemctl restart felhom-controller-bootstrap.service` → poll
  the new controller to healthy (`docker inspect`, ≤90s) → **roll back** to the previous image if not (the
  guest is never left without a controller). Strict image-ref gate; single-flight per guest (409).
  - **(v0.47.0) Verify hardening (F1):** the health poll reads `{{.RestartCount}}` (a 4th inspect
    field) — `running && RestartCount>0` is **not** healthy (a process that already crash-restarted
    isn't stably up), and a **no-healthcheck** image must report ok on 3 **consecutive** polls (a
    stability dwell) before acceptance; a real `healthy` result is trusted immediately. This closes the
    hole where a no-HEALTHCHECK image that crash-loops could land one "Running" instant and false-pass
    the point-in-time check → no rollback. Verify predicate only — the rollback orchestration + the
    `docker inspect -f *` grant (the `*` spans the extended template) are unchanged.
- `GET /controller/swap/status` → `{state, current, previous, target, error}`.

The agent is **external** to the controller container, so it survives the controller being killed
mid-swap (which the controller cannot do to itself). `GuestBinder.GuestExec` is the single `pct exec`
seam. Exercise directly with `--selftest=controller-swap -vmid <id> -image <ref>`.

## Agent self-update (operator-signed, A/B slots, crash-loop auto-rollback — v0.70.0, TASK D1)

The agent updates ITSELF the same way it swaps the controller: **the thing that performs rollback is
never the thing being updated.** For the agent that means systemd + an ~80-line root shell wrapper
(`configs/felhom-selfupdate-guarded`) that changes almost never; the Go binary is what flips.

**Trust model.** An update is an **operator-signed `agent_update` op** delivered through the existing
signed-jobs pipeline (same LOCKED authz gate as `storage_wipe`/`decommission`). The signed params pin
the exact **version + sha256**, so the pinned sha is the *only* integrity root — **neither a
compromised hub (dumb transport) nor a compromised Gitea (dumb storage) can substitute a binary.**
The operator signs offline with `felhom-opsign -op agent_update -agent-version <v> -sha256 <hex>`.

**The flow** (`internal/selfupdate/` = the Go half; the wrapper = the root half):

1. The control loop sees a pending signed op → the gate verifies it (pinned-key SSHSIG → namespace →
   allow-list → crypto → host → time → **durable nonce-burn**) → the `agent_update` executor runs.
2. Executor downloads the binary for the signed version from the config'd artifact host
   (`selfupdate.url_template`, `{version}` interpolated) to `/var/lib/felhom-agent/selfupdate/`,
   verifies its sha256 against the **signed** value (mismatch → refuse, remove, agent untouched),
   and hands it to `sudo -n felhom-selfupdate-guarded apply <staged> <sha>`. The job is completed on
   the hub **after verify+download, before apply** (the nonce is already burned — a queued job would
   only re-fetch and no-op on the spent nonce; a failed/rolled-back update is visible via the report).
3. The wrapper (as root) **re-verifies** the sha, confines the staged path to the staging dir, asserts
   same-filesystem (the atomic-rename guarantee), snapshots the current binary to `.prev`, atomically
   `mv`s the new binary into place, writes a `pending.json` marker, `reset-failed`s, and schedules a
   **detached** restart (`systemd-run --on-active=2s … systemctl restart felhom-agent`, so the caller
   survives to log the handoff).
4. The **new** binary boots; after it has run cleanly for a dwell (`selfupdate.dwell_seconds`, default
   60) *and* core init is done, `internal/selfupdate.Manager` calls the wrapper's `commit` (clears the
   marker; `.prev` retained as a manual net). A pending marker naming a *different* version than the
   running binary is **not** committed — loud WARN, marker left so the report shows why (a human
   decides).
5. **Crash-loop auto-rollback (the safety property).** If the new binary crashes, systemd's
   `OnFailure=felhom-agent-rollback.service` (the `felhom-agent-limits.conf` drop-in) runs the
   wrapper's `rollback`: pending marker present → restore `.prev` byte-identical → clear marker →
   restart → the old binary is back **within seconds of the first crash**. On systemd 257 `OnFailure=`
   fires on *every* crash, so rollback triggers at the first one; the marker-guard makes every later
   fire (and any crash with no update in flight) a harmless no-op. The tuned start-limit
   (`[Unit] StartLimitIntervalSec=120 + StartLimitBurst=4`) is the terminal **backstop** (e.g. an
   environmental crash loop of the known-good binary → terminal `failed` ≈20s → the hub's
   `host_staleness` dead-man's-switch alerts the operator).

**Design provenance:** every systemd behaviour above is empirically validated in
`felhom.eu/documentation/audits/SPIKE-agent-selfupdate-2026-07-05.md` (the SF-findings). The host
report carries `selfupdate_pending` (+ version) so a runs-but-never-commits binary is visible even
though it never crashes. v1 scope: no hub-floor auto-update, no auto-retry of a failed update, no
pending-timeout auto-rollback (a stuck-but-alive binary is caught by `host_staleness`).

### TLS trust

The host serves a self-signed cert. Verification is **not** blanket-disabled. Pick one in
config: `ca_file` (PEM, full verify), `fingerprint` (SHA-256 of the host leaf cert — pinned
exact-cert match; the `/nodes` API returns each node's `ssl_fingerprint` to pin), or the
explicitly-named `insecure_skip_verify` (off by default; selftest-against-127.0.0.1 only).

## Provisioning the token (out-of-band, operator side)

The agent only **consumes** a privilege-separated API token; role setup is a provisioning step.
The role must be granted on **both the user AND the token** for the same path, or the
intersection is empty and every call 403s (phase1-2 §1.2):

```bash
pveum role add FelhomAgent -privs "VM.Allocate VM.Audit VM.Config.Disk VM.Config.CPU \
  VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Snapshot \
  VM.Snapshot.Rollback VM.Backup Datastore.Allocate Datastore.AllocateSpace \
  Datastore.Audit Sys.Audit SDN.Use"          # 16 privileges, validated Phase 3 B3
pveum user add felhom-agent@pve
pveum user token add felhom-agent@pve agent --privsep 1   # capture the secret (shown once)
pveum acl modify / -user  'felhom-agent@pve'       -role FelhomAgent
pveum acl modify / -token 'felhom-agent@pve!agent' -role FelhomAgent
```

(`VM.Config.CPUMemory` is **not** a real privilege; `SDN.Use` **is** required for bridge use.)

## Run

```bash
go build ./...
# read-only health check against the host:
./felhom-agent --config configs/agent.example.json --selftest
# or via env (keeps the secret off disk):
FELHOM_AGENT_PROXMOX_TOKEN='felhom-agent@pve!agent=SECRET' \
FELHOM_AGENT_PROXMOX_NODE=demo-felhom \
FELHOM_AGENT_PROXMOX_ENDPOINT=https://192.168.0.162:8006 \
FELHOM_AGENT_PROXMOX_TLS_FINGERPRINT='BA:7C:...:CF' \
  ./felhom-agent --selftest
```

`--selftest` (read-only) loads config, builds the API client, and runs the read queries (version,
nodes, node status, guests, storage), printing a short health report. It mutates nothing and says
so cleanly if the token/endpoint isn't configured.

`--selftest=task --vmid N` (explicitly gated) exercises `WaitTask` on a **reversible** op
(snapshot → rollback → delete-snapshot) against guest `N`. Default `--selftest` never mutates.

`--selftest=bring-up|provision` accept an optional operator **CPU/RAM cap**: `-cores N` and
`-memory M` (MiB). Both default to `0` = keep the golden's baked sizes. When set, the cap is written
into the SAME pre-start config PUT as the identity reset (via `BringUpSpec.Cores/MemoryMB`), so the
guest never boots uncapped — useful when the appliance shares a host with other guests.

**Pool-scoped restore (v0.53.0):** every restore (provision bring-up, DR, and restore-test) allocates
the guest INTO the `felhom` PVE pool (`reconcile.DefaultPool`; `RestoreLXCOptions.Pool` → `pct restore
--pool`). This is what lets the agent token be scoped to `/pool/felhom` + `/storage/<targets>` instead
of `/` (blast-radius containment on a shared host) — the restore is how a fresh vmid is allocated under
that scoped token (`VM.Allocate`+`Pool.Allocate` at `/pool/felhom`). Layout + validation:
`felhom.eu/documentation/audits/SPIKE-pool-scoped-acl-2026-07-01.md`.

**Drive discovery + tracking (v0.55.0, Impl-2a):** `GET /disks/candidates` lists host disks the Impl-1
filter proves are free to enroll (split initialize/attach). The watchdog's known-drive set is sourced
from the intent registry + Felhom `.mount` units (`RegistryKnownTargets`), NOT PVE storages — so a drive
enrolled with no PVE dir-storage is still health-tracked (`Observe()` stays for real PVE storages). The
controller wizard consuming this is Impl-2b.

**Format safety (v0.54.0, Impl-1; hardened v0.61.0, audit D1/D2/D3):** `Format` (mkfs) is gated by a
mandatory **unclaimed-disk guard** (`internal/storage/claim.go`) — it refuses any device not provably
free for Felhom (OS disk, LVM PV, ZFS/mdraid member, foreign mount, read-only; fail-safe on any read
error, and on an empty/target-absent lsblk topology — D2), independent of `DataBearing`. Below the
agent, mkfs runs ONLY through `configs/felhom-mkfs-guarded.sh` (the sole mkfs the sudoers permits),
which re-checks the catastrophic cases as root: system disk, LVM PV (absolute-path pvs), foreign
mount, read-only device, and any LVM/ZFS/mdraid/LUKS/swap member signature (D1 — validated by
`scripts/mkfs-guarded-harness.sh`, a loop-device + recorder harness). The blank-format local-API path
binds to the device's durable id and anti-retarget re-resolves before mkfs, same as the confirmed
wipe (D3, AGENT-001's benign-branch twin). The pool-scoped token does NOT touch mkfs (sudo op) — the
filter + wrapper are the guard. See `SPIKE-drive-enrollment-2026-07-01.md` and
`felhom.eu/documentation/audits/AUDIT-blast-radius-hostroot-localapi-2026-07-02.md`.

## Process model

Native Go binary, systemd service, **non-root** `felhom-agent` service user holding the scoped token,
with a **narrow sudoers allowlist** for the fenced host-root ops. `privileged.mode: "sudo"` matches
this; `"direct"` is for dev/CI where the agent is already root.

The canonical artifacts (BUNDLE slice):
- **`configs/felhom-agent.service`** — the canonical unit (`User=felhom-agent`,
  `ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json`). It deliberately sets
  **no** `NoNewPrivileges` and **no** mount-namespacing hardening (`ProtectHome`/`PrivateTmp`/…): the
  first would block the `sudo` the agent needs, the second would put the agent in a private mount
  namespace so its `mount --bind` drive enrollments wouldn't propagate into the running guest. The
  security boundary is the sudoers allowlist, not systemd sandboxing.
- **`configs/felhom-agent.sudoers`** → `/etc/sudoers.d/felhom-agent` (0440, `visudo -cf`-validated).
- **`scripts/publish-agent.sh`** publishes the binary to Gitea as a generic package
  (`/api/packages/admin/generic/felhom-agent/<ver>/felhom-agent`), printing the sha256 the operator
  records in the hub artifact manifest.

**Install is automated.** The host-bootstrap script (`felhom.eu/scripts/felhom-host-install.sh`) fetches
the binary from Gitea, verifies its sha256 against the hub-vouched manifest, then installs the user +
binary + sudoers + unit + config — no manual agent install step.

## Test

```bash
go vet ./... && go test ./...
```

Unit tests use a mock HTTP transport + mock runner (no live host): UPID parse, `WaitTask`
(running→OK / running→failed-403 / timeout / ctx-cancel), 403→privilege-named error, response
decoding against the captured live shapes, and the API-vs-root routing fence.