felhom-agent/README.md

# felhom-agent

The **host agent** for the Felhom platform — the operator-tier component that runs on each
Proxmox host and owns *all* Proxmox interaction (provision/restore guests, host storage,
backups, host+tunnel monitoring, hub control loop, per-guest local API). Design:
[`felhom.eu/documentation/architecture/03-host-agent.md`](https://gitea.dooplex.hu/admin/felhom.eu/raw/branch/main/documentation/architecture/03-host-agent.md).

> **Status — slice 1 of N.** This repo currently contains the project scaffold and the
> **`internal/proxmox`** interaction layer (the typed library every other module will call to
> talk to Proxmox), plus a runnable read-only `--selftest`. **No** reconcile loop, hub client,
> signing, or storage/backup orchestration yet — those are later slices.

Module: `gitea.dooplex.hu/admin/felhom-agent` · binary: `felhom-agent` · Go 1.24.

## Layout

```
cmd/felhom-agent/      # entry point + --selftest modes + the daemon (poll loop + reconcile + watchdog)
internal/proxmox/      # the Proxmox interaction layer (API-first + fenced root-CLI)
internal/config/       # JSON config + env overrides (secrets never logged)
internal/log/          # slog setup
internal/authz/        # operator signed-op verifier (SSHSIG); durable nonce store
internal/hub/          # daemon: host-report collector + Bearer client + resilient poll loop
internal/reconcile/    # reconcile engine + reversibility gate + op journal + crash recovery
internal/storage/      # storage-target observer + durable_id + fast-poll watchdog (slice 5)
internal/capability/   # privileged-capability self-probe (manifest + sudo -n -l check, v0.44.0)
configs/agent.example.json
```

## The `capability` package — privileged self-check (v0.44.0)

The non-root agent depends on a fixed set of `sudo -n` grants (`configs/felhom-agent.sudoers`). When
the 2026-06-28 root→non-root cutover silently dropped some (lxc-info, make-private, …), features broke
unnoticed until a user hit them. `internal/capability` makes that loud:

- **`Manifest()` is the source of truth** for the required `(binary, representative-arg)` vectors,
  seeded from the sudoers audit (the OK + CLOSED rows; the surfaced/deferred rows are excluded). When
  you add a privileged `runner.Run` call that needs a NEW grant, add it to BOTH the sudoers file AND
  the manifest — `manifest_test.go` asserts **manifest ⊆ sudoers** at build time (with a red-proof
  that dropping the lxc-info grant fails the gate), so a missing grant is caught in CI, not in prod.
- **`Prober.Probe`** lists each vector with `sudo -n -l` (a policy LIST — never executes, safe for
  mkfs/pct) + an `os.Stat` existence check → an `ok`/`degraded` snapshot. It runs at startup (logged
  loud) and on every collect, riding the hub report as `HostReport.Capabilities`; the hub
  (`HostCapabilityChecker`) alerts the operator on a Critical capability going degraded. Serve-degraded
  — the probe never blocks startup. (Next self-health slice: the controller↔agent channel check.)

## The `storage` package — observe + watchdog (slice 5)

Read-only this slice (no hub desired-state until slice 10):

- **Observer** builds the host-report's `storage_targets` from `ListStorage`/`NodeStorage`
  joined with non-privileged host reads (`/proc/mounts`, `/dev/disk/by-uuid`,
  `/sys/.../rotational`). It reports each target's `durable_id` (the DR-load-bearing
  re-attach key: fs-UUID for usb/local-dir, `server:export` for nfs/cifs,
  `repo+fingerprint` for pbs, `vg/pool` for lvmthin), state, usage, a rotational
  **class hint** (never authoritative — class is hub-owned), and the lvmthin **thin-pool
  data fill** (a full pool corrupts every guest on it). SMART is a Phase-B privileged read.
- **Watchdog** is the third daemon goroutine: a fast poll (seconds) over the *known*
  target set that detects an `attached↔disconnected` transition and fires a **debounced,
  out-of-band** host-report so the hub learns of a USB drop in seconds rather than at the
  ~15-minute cycle. It mutates nothing (the benign re-mount-by-UUID response lands in
  Phase B). The `HostReader` seam keeps it root-free and unit-testable with no real devices.

The reported `StorageTarget` shape is a cross-repo contract duplicated in `felhom.eu/hub`;
`internal/hub/testdata/host-report.golden.json` is byte-identical with the hub's copy and a
bidirectional key-set test guards drift.

### DR recipe — the storage/guest/PBS half (v0.38.0)

The host-report carries an additive `dr_recipe` section (`internal/hub/dr_recipe.go`) — the agent half
of the secret-free reconstruction recipe (`SPIKE-dr-recipe-2026-06-16.md`). It is the non-secret
**re-provision scaffolding** that complements escrow (keys) + PBS/restic (bytes): `guests[]` sizing,
`drives[]` (user-data drives by `durable_id` → role → mount → intent), `pve_storage[]` (the
`storage.cfg` defs), and `pbs` coordinates. Built by the pure `BuildDRRecipeHostHalf` from facts the
report already collects — **no new reads**. **Boundary:** every field is an identifier/intent/size/
coordinate — never a key/password/token/hash/`ENC:`; the PBS key + restic password stay in escrow.
`recipe_version=1`, ignore-unknown on read, pinned in the cross-repo golden. The hub assembles it with
the controller's app half into one customer recipe.

### The privileged `HostOps` surface (slice 5 Phase B)

The write side — the one place the agent steps outside its Proxmox API token into OS-root —
is isolated behind the `HostOps` seam (`hostops.go`): production `SudoHostOps` shells out via
a narrow **sudoers allowlist** (`configs/felhom-agent.sudoers`) with **fixed argument vectors
and no shell**; tests use a fake (no real root in the suite).

- **Persistent mounts** are **systemd `.mount` units keyed by fs-UUID**
  (`What=/dev/disk/by-uuid/<UUID>`, enabled so they survive reboot) — not raw fstab or a
  transient `mount`. Benign re-mount is idempotent; **detach** (stop+disable) is destructive
  and routes through the gate.
- **Host-reboot remount re-resolution (v0.37.0):** at startup (before binding drives into the
  guest) and on the periodic tick, `ReassertEnrolledMounts` re-asserts every enrolled `.mount`
  unit that isn't currently mounted: it re-resolves the drive's `uuid:<fs-uuid>` durable id to
  its **current** `/dev` node by re-scanning `/dev/disk/by-uuid` (`ResolveStorageDevice`, never a
  cached node) and re-runs the idempotent `enable --now`. This re-enables a unit a prior detach
  left `disabled` AND tolerates kernel re-enumeration moving a drive's letter (`/dev/sdb`→`sdc`)
  — the reshuffle is a no-op. Already-mounted drives and genuinely-absent UUIDs are skipped.
- **Every argument is validated before any command is constructed** (`validate.go`): UUIDs
  against a strict hex regex, mount paths confined + traversal-checked, SMART devices
  whitelisted to raw disks, LVM names charset-checked. The adversarial matrix in
  `validate_test.go` proves a hostile UUID / path / device is refused with **zero** exec.
- **SMART** (`smart.go`) fills `StorageTarget.smart` via `smartctl -a -j` — SATA *and* NVMe
  attribute sets, degrading to `UNKNOWN` for devices that expose no SMART (e.g. a USB bridge).
  **`lvs`** fills the lvmthin thin-pool **metadata** fill (metadata exhaustion corrupts a pool
  like data exhaustion).
- The **watchdog** gains a benign **re-mount response**: when a known mount-backed target's
  device returns unmounted, it dispatches (off the poll path) a by-UUID re-mount, routed
  through the gate as benign. The **disk-grow executor** (`pct resize`, **grow-only**) lands
  in `internal/reconcile` as a benign action; **destructive storage ops** (detach/wipe/
  data-losing-resize) construct a `ClassStorageWipe`/`ClassDecommission` intent bound to the
  storage **target identity** and go through the slice-4 gate (built + tested, inert live).

### `--selftest=storage` (live storage harness)

Runs standalone on the Proxmox host (no hub needed):
- bare: an **observe pass** printing the full `StorageTarget` table incl. the SMART summary
  and thin-pool data+metadata fill.
- `-watch <dur>` (e.g. `--selftest=storage -watch 3m`): runs the watchdog verbose for the
  window with the **re-mount response live**, so an operator can physically cycle a drive and
  watch detect → report → re-mount in the logs.

## The `proxmox` package — model

Two backends, one fixed routing policy (the fence is structural — `Client` never shells out,
`Privileged` never makes an HTTP call; asserted in `routing_test.go`):

| | Backend | Used for |
|---|---|---|
| **API (default)** | `proxmox.Client` | everything the scoped **FelhomAgent** token can do |
| **root-CLI (fenced)** | `proxmox.Privileged` | the **three** proven OS-root exceptions only |

Grounded entirely in the spike findings (`felhom.eu/documentation/proxmox-platform.md`,
`tests/phase{0,1-2,3}-findings.md`). Every mutating API op is **async**: it returns a UPID and
the caller `WaitTask`s until the task stops, then asserts `exitstatus == "OK"` — authorization
can surface at task execution, not the HTTP POST (phase1-2 §1.3).

### Public surface

`Client` (API):

- Read: `Version`, `Nodes`, `NodeStatus`, `ListLXC`, `GuestStatus`, `GuestConfig`,
  `ListStorage`, `NodeStorage`, `StorageContent`.
- Async mutating (return UPID): `RestoreLXC` (primary create path), `Vzdump`, `Snapshot`,
  `Rollback`, `DeleteSnapshot`, `SetConfig`, `Start`, `Stop`.
- Tasks: `WaitTask`, `TaskStatusOnce`, `TaskLogTail`.
- Errors: `*APIError` (parses the offending privilege from a 403), `*TaskError` (parses it from
  a failed task `exitstatus`).

`Privileged` (fenced root-CLI) — each method documents *why it can't be the API*:

- `CreateGoldenLXC` — `pct create` with `keyctl=1` (root@pam-only; the only root-fenced create —
  the per-customer path provisions by **restore**, which preserves keyctl).
- `MountUSBByUUID` — host mount-by-UUID (not a Proxmox API op).
- `SMART`, `Sensors` — hardware reads (not API-exposed).

### API-vs-root routing table

See the table in [`internal/proxmox/doc.go`](internal/proxmox/doc.go). Summary: the entire guest
lifecycle **including restore** is API-token-covered; OS-root is confined to golden-image
`keyctl` create, host mounts, and SMART/sensors (phase3 §B3).

### Controller swap (agentic controller update, Phase 1 — v0.42.0)

The local API (`internal/localapi/`) owns the in-guest controller image **swap** — the new-architecture
replacement for the controller's dead in-container `docker compose` self-update. The in-guest controller
pre-pulls the target image then calls the agent:

- `POST /controller/swap {image}` (`withGuest`-scoped) → **202**, then async: record previous
  (`/var/lib/felhom-agent/controller-swap-<vmid>.json`) → confirm the target is present in the guest →
  write `/etc/felhom-controller-image` → `systemctl restart felhom-controller-bootstrap.service` → poll
  the new controller to healthy (`docker inspect`, ≤90s) → **roll back** to the previous image if not (the
  guest is never left without a controller). Strict image-ref gate; single-flight per guest (409).
- `GET /controller/swap/status` → `{state, current, previous, target, error}`.

The agent is **external** to the controller container, so it survives the controller being killed
mid-swap (which the controller cannot do to itself). `GuestBinder.GuestExec` is the single `pct exec`
seam. Exercise directly with `--selftest=controller-swap -vmid <id> -image <ref>`.

### TLS trust

The host serves a self-signed cert. Verification is **not** blanket-disabled. Pick one in
config: `ca_file` (PEM, full verify), `fingerprint` (SHA-256 of the host leaf cert — pinned
exact-cert match; the `/nodes` API returns each node's `ssl_fingerprint` to pin), or the
explicitly-named `insecure_skip_verify` (off by default; selftest-against-127.0.0.1 only).

## Provisioning the token (out-of-band, operator side)

The agent only **consumes** a privilege-separated API token; role setup is a provisioning step.
The role must be granted on **both the user AND the token** for the same path, or the
intersection is empty and every call 403s (phase1-2 §1.2):

```bash
pveum role add FelhomAgent -privs "VM.Allocate VM.Audit VM.Config.Disk VM.Config.CPU \
  VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Snapshot \
  VM.Snapshot.Rollback VM.Backup Datastore.Allocate Datastore.AllocateSpace \
  Datastore.Audit Sys.Audit SDN.Use"          # 16 privileges, validated Phase 3 B3
pveum user add felhom-agent@pve
pveum user token add felhom-agent@pve agent --privsep 1   # capture the secret (shown once)
pveum acl modify / -user  'felhom-agent@pve'       -role FelhomAgent
pveum acl modify / -token 'felhom-agent@pve!agent' -role FelhomAgent
```

(`VM.Config.CPUMemory` is **not** a real privilege; `SDN.Use` **is** required for bridge use.)

## Run

```bash
go build ./...
# read-only health check against the host:
./felhom-agent --config configs/agent.example.json --selftest
# or via env (keeps the secret off disk):
FELHOM_AGENT_PROXMOX_TOKEN='felhom-agent@pve!agent=SECRET' \
FELHOM_AGENT_PROXMOX_NODE=demo-felhom \
FELHOM_AGENT_PROXMOX_ENDPOINT=https://192.168.0.162:8006 \
FELHOM_AGENT_PROXMOX_TLS_FINGERPRINT='BA:7C:...:CF' \
  ./felhom-agent --selftest
```

`--selftest` (read-only) loads config, builds the API client, and runs the read queries (version,
nodes, node status, guests, storage), printing a short health report. It mutates nothing and says
so cleanly if the token/endpoint isn't configured.

`--selftest=task --vmid N` (explicitly gated) exercises `WaitTask` on a **reversible** op
(snapshot → rollback → delete-snapshot) against guest `N`. Default `--selftest` never mutates.

## Process model

Native Go binary, systemd service, **non-root** `felhom-agent` service user holding the scoped token,
with a **narrow sudoers allowlist** for the fenced host-root ops. `privileged.mode: "sudo"` matches
this; `"direct"` is for dev/CI where the agent is already root.

The canonical artifacts (BUNDLE slice):
- **`configs/felhom-agent.service`** — the canonical unit (`User=felhom-agent`,
  `ExecStart=/usr/local/bin/felhom-agent --config /etc/felhom-agent/agent.json`). It deliberately sets
  **no** `NoNewPrivileges` and **no** mount-namespacing hardening (`ProtectHome`/`PrivateTmp`/…): the
  first would block the `sudo` the agent needs, the second would put the agent in a private mount
  namespace so its `mount --bind` drive enrollments wouldn't propagate into the running guest. The
  security boundary is the sudoers allowlist, not systemd sandboxing.
- **`configs/felhom-agent.sudoers`** → `/etc/sudoers.d/felhom-agent` (0440, `visudo -cf`-validated).
- **`scripts/publish-agent.sh`** publishes the binary to Gitea as a generic package
  (`/api/packages/admin/generic/felhom-agent/<ver>/felhom-agent`), printing the sha256 the operator
  records in the hub artifact manifest.

**Install is automated.** The host-bootstrap script (`felhom.eu/scripts/felhom-host-install.sh`) fetches
the binary from Gitea, verifies its sha256 against the hub-vouched manifest, then installs the user +
binary + sudoers + unit + config — no manual agent install step.

## Test

```bash
go vet ./... && go test ./...
```

Unit tests use a mock HTTP transport + mock runner (no live host): UPID parse, `WaitTask`
(running→OK / running→failed-403 / timeout / ctx-cancel), 403→privilege-named error, response
decoding against the captured live shapes, and the API-vs-root routing fence.