felhom-agent/CLAUDE.md

# CLAUDE.md — `felhom-agent`

> Place at the repo root (`felhom-agent/CLAUDE.md`). Loads when Claude Code touches this repo. Keep under ~200 lines. The cross-repo orientation lives in the workspace-root `e:\git\CLAUDE.md`; this file is `felhom-agent`-specific.

## What this repo is

`felhom-agent` is the operator-tier **host agent** that runs on each Proxmox host and owns **all** Proxmox interaction: provision/restore guests, host storage, backup/restore orchestration, the hub control loop, and a narrow per-guest local API. It is the **most privilege-sensitive** component.

- It is the renamed former `proxmox-controller` repo.
- **Distinct from `felhom-controller`** — that is the *in-guest* controller (Docker-only, no Proxmox creds). Do not confuse them.
- Control plane, not data plane: if the agent dies, apps keep serving; only management degrades.

## Build / run

- Module `gitea.dooplex.hu/admin/felhom-agent`; binary `felhom-agent` (`cmd/felhom-agent/`).
- **Pure Go stdlib + `golang.org/x/crypto` only** — no web frameworks.
- `go.mod` directive **go 1.25.0**; dep `golang.org/x/crypto v0.52.0` (declares go 1.25, will NOT build on Go 1.24). The **build server (192.168.0.180) runs go1.26.0** (upstream Go on PATH, backward-compatible). Build/run the agent there for live tests (same LAN as the demo host).
- Version: `version` var in `cmd/felhom-agent/main.go`, overridable via `-ldflags "-X main.version=<v>"`; `--version` flag. Bump on meaningful changes + add a CHANGELOG entry. **See `CHANGELOG.md` (top only) for the authoritative current state.**
- **Current: v0.31.0** (2026-06-14; deployed on demo host `felhom-pve`). Recent state (summarized from the CHANGELOG top — verify there, not here):
  - **v0.31.0** — live-drive disk fixes: **F9** guest data-drive bind survives a re-provision (`GuestBindStore` + `ReassertGuestBinds` on startup, durable-id-matched; `DiskInfo.GuestAttached` reporting); **F20-BUG2** one wipe durable-id scheme (`DiskInfo.WipeDurableID` via a shared `deviceDurableID` seam used by list + gate); **F20-BUG3** mkfs runs detached off `baseCtx` with a persisted `formatJob` + `GET /disks/format/status` + `RecoverFormatJob` startup recovery (survives a request deadline + an agent restart). Live-validated on 9201 + the 916 GB felhom-usb.
  - **v0.30.0** — **AGENT-001** security fix: the inline customer-confirmed wipe (`localapi handleDiskFormat`) re-resolves the confirmed durable id → current device, re-derives+matches, re-inspects, and formats the **re-resolved** device (never the mutable `req.Device`), closing a classify→mkfs TOCTOU. At parity with `signedjobs.WipeExecutor` (`internal/localapi/wipe_reresolve.go`).
  - **v0.29.x** — OS / Docker-data storage **split** (golden bakes split rootfs + Docker-data volume; provision) + `lanresolver` split-horizon DNS fix (RESTART, not reload, dnsmasq on a guest IP change).
  - **v0.28.0** — backup re-target to offsite **`felhom-pbs`** (DR) + operator-signed **decommission**.
  - **v0.23–0.27** — device-**ROLE** classification + tiered storage-wipe gate (system/backup operator-only, user-data customer-confirmable); eject role-gate; user-data drive **enroll/bind** into the guest (slice 10 P2/P3); self-heal watchdog + 4-state intent model.
  - **Foundations (slice 8–10):** per-guest **local-API server** (`internal/localapi`, self-scoped endpoints, hashed per-guest token store, pinned self-signed leaf); the `/disks` data-bearing classifier + signed-job destructive gate (`POST /disks/format` inspects the device itself; data-bearing → gate → `pending_signature` refused, caller's claim ignored; blank → benign `mkfs`); the provisioning back-half (`internal/provision`: mint token → `bootstrap.json` 0600 → `chown 100000:100000` → `pct set` bind, no registry cred in guest); host metrics; hub desired-state; operator-signed completion + offline signing CLI; PBS escrow + identity-restore (slice 10D). Runtime dep: `proxmox-backup-client`.

## Layout

```
cmd/felhom-agent/    main + flag handling + --selftest modes + the daemon entry
internal/config/     JSON config + FELHOM_AGENT_* env overlay; secrets redacted (Redacted())
internal/log/        slog setup
internal/proxmox/    API-first Client + fenced root-CLI Privileged + UPID WaitTask
internal/authz/      operator signed-op verifier (SSHSIG); durable FileNonceStore
internal/hub/        daemon: HostReport collector + Bearer client + resilient Loop
internal/reconcile/  reconcile engine + reversibility gate + op journal + crash recovery + restore-test
internal/storage/    storage-target observer + durable_id + fast-poll watchdog (slice 5)
internal/backup/     vzdump backup runner + restore-test scheduler + report store (slice 6)
internal/pbs/        PBS-API client (fingerprint-pinned) + verify maintenance loop (slice 6 Phase B)
```

## Proxmox model (the load-bearing rules)

- **API-first** via a scoped `FelhomAgent` token (16 privileges). Raw root-CLI is **fenced to exactly 3 exceptions**: keyctl `pct create` (golden image), USB mount/fstab, SMART/sensors. `Client` never shells out; `Privileged` never makes HTTP calls (asserted by tests). Keep that fence.
- **Every mutating op is async** → returns a UPID → `WaitTask` asserts `exitstatus == "OK"`. A 200 on the POST is **not** success; authorization can fail at task execution, not the POST.
- **TLS:** SHA-256 leaf-cert pinning (the host serves a self-signed cert). No insecure default.
- **Privsep token gotcha:** a `--privsep 1` token's rights = intersection of the backing user's perms AND the token's ACLs — so the role must be granted on **both** user and token, or every call 403s. (Token provisioning is out-of-band / human-run; the agent only consumes the token.)

## Design + platform facts (read before designing)

- Design doc: `felhom.eu/documentation/architecture/03-host-agent.md` (locked).
- Platform facts: `felhom.eu/documentation/proxmox-platform.md` + `tests/phase{0,1-2,3,4}-findings.md`.

## Current state

Built in slices, all on `main`:
- **v0.1.0** slice 1 — scaffold + `internal/proxmox` + `internal/config`/`log` + `--selftest`.
- **v0.2.0** slice 2 — `internal/authz` signed-op verifier.
- **v0.3.0** slice 3 — `internal/hub`: the first **daemon loop** (no-`--selftest` mode) posting a read-only `HostReport` to the hub (= the heartbeat). Report's storage/backup/restore/pbs/audit fields are **defined-but-empty** (slices 5/6); the envelope's desired-state/signed-ops fields are **parsed-but-ignored** (slice 4).
- **v0.3.1** — slice-3 validation follow-ups.
- **v0.3.2** — slice-4 pre-check: reversible `SetConfig` step added to `--selftest=task`; passed live on guest 9999. Findings: LXC `description` write is **synchronous** (empty UPID — dual-mode modeling confirmed); PVE appends a trailing `\n` to `description` on read (reconcile must normalize). First live `VM.Config.*` exercise.
- **v0.4.0-rc1** — slice-4 **Phase A** (structural): `internal/reconcile` — engine, per-guest serializer (§10), desired-state model + `DesiredProvider` seam, normalization layer (`NormDescription` promoted out of main.go), plan/diff engine (benign Start/Stop/SetConfig set), durable op journal + idempotency store. Wired into `runDaemon` sharing the queue. Runs **live but unfed** (EmptyProvider → zero mutations until slice 10).
- **v0.4.0** — slice-4 **Phase B** (security core): the benign/destructive **classifier** (provenance + data-bearing, not by verb; scratch/same-txn provenance is agent-internal, never hub-sourced), the **reversibility gate** (destructive → `pending_signature` unless a verified, role-scoped, action-bound operator signature), the **signed-op consuming layer** over `internal/authz` (role-scoping per doc 04 §4, op-to-action binding, idempotency-by-nonce, audit), and the **crash-recovery consumer** (`Recover` over `InFlight()`, resume-or-rollback). The gate fronts the queue's executor (every mutation passes it). **Inert this slice** — no destructive deltas served until slice 10; the destructive path is classified, gated, and adversarially tested but not wired to live execution. `authz` surface untouched.
- **v0.5.0-rc1** — slice-5 **Phase A** (read-only, live): `internal/storage` — the `StorageTarget` wire contract (filled the slice-3 stub), `durable_id` derivation per type, the `Observer`, and the **storage watchdog** (third daemon goroutine; fast-poll → debounced out-of-band report on a known target's attach/disconnect). Hub ingest accepts/persists `storage_targets`; cross-repo golden byte-identical.
- **v0.5.0** — slice-5 **Phase B** (the host-root surface): the `HostOps` seam + `SudoHostOps` (systemd `.mount` units by fs-UUID, detach, SMART, lvs) behind a **strict argument validator** (the adversarial matrix is the headline security test — hostile UUID/path/device refused with zero exec); SMART (SATA+NVMe) + thin-pool metadata enrichment; the watchdog's benign **re-mount response** (off the poll path); the **disk-grow executor** (`pct resize`, grow-only, benign) and **destructive storage ops** through the slice-4 gate (target-scoped; built + tested, inert live); `--selftest=storage [-watch]`; `configs/felhom-agent.sudoers`.
- **v0.6.0-rc1** — slice-6 **Phase A** (backup + self-restore-test, local target): proxmox `DestroyLXC`/`Vzdump`-notes/`LatestBackupVolID`; `Engine.RunRestoreTest` (journaled scratch lifecycle: restore-to-new → net link-down → boot → verify running → defer teardown, all benign); `Recover` extended to reap a leaked scratch guest (Scratch journal flag, special-cased before the UPID path); `internal/backup` (runner + bulk-gap + cadence scheduler + report store); hub `Backup`/`RestoreTest` filled (cross-repo golden + hub logs a failed restore-test); `--selftest=backup`/`--selftest=restore-test`. Live-validated on demo-felhom.
- **v0.6.0** — slice-6 **Phase B** (PBS offsite tier): `internal/pbs` — a fingerprint-pinned, token-authed PBS-API client (Verify/Snapshots/TaskStatus, node-from-UPID); the verify maintenance loop (own cadence, NOT gated/journaled — like the watchdog); `PBSSnapshot` reporting filled (cross-repo golden + hub failed-verify WARN); truthful vzdump mode from the task log; `--selftest=pbs-verify`. Backup/restore-to-PBS reuse Phase A unchanged. Live-validated against the spike's DooPlex PBS.
- **Next: slice 7 (provisioning + identity-reset + golden base, §9)** — the unified bring-up primitive; restore-overwrite + decommission executors the gate already guards; escrow + host-loss DR.

## Demo host (for live tests)

Node **`demo-felhom`**, API `https://192.168.0.162:8006`, PVE 9.2.2; leaf-cert SHA-256 fingerprint starts `BA:7C:99:7D:45:D0…` (verify it still matches before a live run — the agent pins it). `pveum`/`pct` ops need `root@pam` on the PVE (SSH alias `felhom-pve`) - available to Claude Code

Selftest modes (run from the build server, pointed at the demo API):
- `--selftest` / `--selftest=read` — read-only health checks.
- `--selftest=task -vmid N` — reversible snapshot→rollback→delete on guest N (gated; never under bare `--selftest`).
- `--selftest=hub` — one collect + report round-trip to the hub.
- No flag → the **daemon** (poll loop); requires `hub` config.

## Conventions

### Trunk-based — no branches

All shippable work commits **directly to `main`**; `main` is always equal to what is deployed. **Do NOT create feature/fix branches.**

- **Report-only artifacts** (audits, findings, fixspecs, reconciliations) → `felhom.eu/documentation/` (`audits/`, `backlog/`), committed to `main`. Never a branch, never left loose at the repo root.
- **Risky/supervised fixes** (agent / golden / provisioning / destructive) are spec'd, then implemented **during the supervised session itself, directly on `main`** — not prepared ahead on a branch. (This is the common case for *this* repo.)
- **Unattended escape hatch:** if a fix can't be cleanly verified/shipped, **revert it and report it** for a supervised redo (or paste the diff into the spec doc in `documentation/`) — do not park it on a branch.
- This supersedes any older "prepared on branch `fix/…`, pending review" pattern.

> **In every repository where you make a change, update both files in that repo:**
> - **`CHANGELOG.md`** — a cumulative log of **all** changes; newest entry on top.
> - **`REPORT.md`** — **overwrite** with a summary of the **most recent** implementation (or significant validation/operational run) only; not cumulative.
>
> **Never write secrets** — tokens, passwords, private keys, API keys — into `CHANGELOG.md`, `REPORT.md`, or any committed file. Reference them as "stored out-of-band" instead.

- Code quality: verify generated code for bugs/edge cases; add debug logging; **ask rather than guess** when you'd otherwise invent input/output.

### Live validation

Live validation of a user-facing feature must exercise the SERVER-SIDE PIPELINE a real user triggers, end-to-end (e.g. connect → enroll → deploy). The forbidden shortcut is BYPASSING that pipeline — e.g. raw agent guest-attach + hand-set state instead of the enrollment flow (the F9 episode) — which gives false confidence and leaves the system inconsistent. INVOKING THE EXACT ENDPOINT THE UI INVOKES — so the full server pipeline (gates, env injection, pre-create belts, compose generation) runs — is an ACCEPTABLE proxy when a browser-automation tool isn't available: it differs fundamentally from the F9 bypass because no server logic is skipped, only the browser rendering. The residual that proxy does NOT cover is purely client-side (progress panels, card/health rendering, client-side guards like checkBeforeDeploy); for strict end-to-end UI coverage use a real browser tool or a manual click-through — and SAY which was used. Low-level mechanism tests where the direct call IS the mechanism remain exempt.

## Workflow & artifacts

- Implement **`TASK.md` / `TASK-*.md`** specs (when placed as `TASK.md` or told to implement one), then push + CHANGELOG + REPORT.md.
- **`RUNBOOK-*.md`** — an operational procedure. CC executes the steps it has access and capability for, including live validation on the demo nodes and the demo Proxmox host (CC has root@felhom-pve SSH + the felhom-agent token). A step is human-only only when it genuinely needs physical presence, a real-world decision, or credentials CC truly lacks — mark those steps HUMAN. Do not decline a whole procedure because it touches a live host or a privileged token. (Judgment still applies: confirm before irreversible ops on real customer data — but demo scratch guests are fair game.)