felhom-agent/CLAUDE.md

# CLAUDE.md — `felhom-agent`

> Loads when Claude Code touches this repo. Stable orientation only — **current state lives in
> `CONTEXT.md` and the top of `CHANGELOG.md`**, never here. Cross-repo orientation: workspace-root
> `/mnt/5_hdd/felhom.eu/git/CLAUDE.md`.

## What this repo is

`felhom-agent` is the operator-tier **host agent** that runs on each Proxmox host and owns **all**
Proxmox interaction: provision/restore guests, host storage, backup/restore orchestration, the hub
control loop, and a narrow per-guest local API. It is the **most privilege-sensitive** component.

- Renamed former `proxmox-controller` repo.
- **Distinct from `felhom-controller`** — that is the *in-guest* controller (Docker-only, no Proxmox
  creds). Do not confuse them.
- Control plane, not data plane: if the agent dies, apps keep serving; only management degrades.

## Read before writing code

- **`REUSE.md`** — canonical helpers, format-safety guards, traps, seams. Check it first; update it
  in the same commit that changes a shared helper or pattern.
- `CONTEXT.md` (current state + open threads) and the top `CHANGELOG.md` entry (authoritative history).
- Design doc: `felhom.eu/documentation/architecture/03-host-agent.md` (locked). Platform facts:
  `felhom.eu/documentation/proxmox-platform.md` + `tests/phase{0,1-2,3,4}-findings.md`.

## Layout (verified against the tree)

```
cmd/felhom-agent/    main + flags + --selftest modes + the daemon entry
cmd/felhom-opsign/   offline operator signing CLI (SSHSIG)
internal/authz/      operator signed-op verifier (SSHSIG) + durable FileNonceStore
internal/backup/     vzdump backup runner + restore-test scheduler + report store
internal/capability/ live sudo-policy capability probe (degradation visibility)
internal/config/     JSON config + FELHOM_AGENT_* env overlay; secrets redacted (Redacted())
internal/desired/    hub desired-state syncer (envelope observer)
internal/escrow/     PBS-key escrow (zero-knowledge recovery code)
internal/guesthook/  pre-start self-heal hookscript install
internal/hub/        daemon: HostReport collector + Bearer client + resilient Loop
internal/lanresolver/ split-horizon DNS on guest IP change (dnsmasq RESTART, not reload)
internal/localapi/   per-guest local API: token store, disks/format, guest binds, controller swap,
                     stale-lock recovery, pinned self-signed leaf
internal/log/        slog setup
internal/pbs/        PBS-API client (fingerprint-pinned) + verify maintenance loop
internal/provision/  guest bootstrap back-half (token mint → bootstrap.json → pct bind)
internal/proxmox/    API-first Client + fenced root-CLI Privileged + UPID WaitTask
internal/reconcile/  reconcile engine + reversibility gate + op journal + crash recovery
internal/signedjobs/ operator-signed destructive executors (wipe, decommission)
internal/storage/    storage observer + durable ids + role/claim classifiers + SudoHostOps + watchdog
```

## Build / run

- Module `gitea.dooplex.hu/admin/felhom-agent`; binary `felhom-agent` (`cmd/felhom-agent/`).
- **Pure Go stdlib + `golang.org/x/crypto` only** — no web frameworks. The Go version is `go.mod`'s
  business, not this file's. DooPlex (where CC runs) has the toolchain — build and run live tests
  locally.
- Version via `-ldflags "-X main.version=<v>"`; `--version` flag. Bump on meaningful changes + CHANGELOG entry.
- **Full build/deploy/publish runbook: use the `felhom-build-deploy` skill.** Summary:

> **Clean-tree gate before any build:** `git status --porcelain` must be empty and
> `git rev-parse HEAD` must equal `git rev-parse origin/main` in the repo being built. An unpushed
> change does not exist — never build a dirty or unpushed tree. The `git pull` in the build step
> stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from
> elsewhere).

> **RELEASING IS ONE COMMAND, AND IT PUBLISHES (R-115).** Never hand-roll the build. The release
> script also creates the `v<version>` git TAG that `felhom-host-install.sh` fetches that version's
> sixteen config files from (R-183), and verifies by an **independent download** rather than trusting
> the publish step's own output. The order is **build → tag LOCALLY → publish → push tag**, and each
> step protects something (R-188, R-186). A released binary is independently verifiable: the build
> uses `-trimpath -buildvcs=false`, so the same source produces the same bytes.
>
> The ordering rationale, the verification recipe and the `publish-agent.sh` flag-parity trap are in
> the **`felhom-build-deploy`** skill.

<!--
RELEASE RATIONALE — history, not directives. Full procedure lives in the felhom-build-deploy skill.

R-115: there used to be a raw `go build` line here and a *separate* "Publish" row, so publishing was
a step someone had to remember — and it was forgotten three times in five days, the last leaving an
agent version deployed on both demo hosts and undownloadable, where a documented-path reinstall
would have silently downgraded them while reporting success. scripts/publish-agent.sh still exists
and is still correct — the release script CALLS it rather than reimplementing it.

R-188/R-186 ordering: the tag is created before the publish so the build and the tag describe the
same commit; it is *pushed* after, because the push is what wakes CI (on: [push]) and a tag visible
before its package makes the published-versions gate correctly fail a correct release — it did, on
roughly every second release, and R-168 sends that failure by mail. The invariant the old order
protected is asserted directly instead: the gate now also refuses a published version with no tag.
If the push fails after a successful publish the script says so loudly and prints the one-line
recovery; if the publish fails it removes the local-only tag so a retry is clean.

R-186 reproducibility: before -trimpath -buildvcs=false, a rebuild could not reproduce the sha you
were vouching. publish-agent.sh's fallback build uses the SAME flags — it used to force
CGO_ENABLED=0 and produce a 74 KB-smaller binary for the same version; if either build line ever
changes, change both, or one version name means two binaries again.
-->


| Step | Where | One-liner |
|---|---|---|
| **Release** (build + tag + publish + verify) | DooPlex (local) | `GITEA_USER=admin GITEA_TOKEN=<tok> scripts/release-agent.sh <ver>` — refuses a dirty/unpushed tree and refuses to re-release an existing version |
| Copy | local → felhom-pve | `scp /tmp/felhom-agent-<v> felhom-pve:/tmp/` (one hop) |
| Deploy | felhom-pve | backup `.bak-<old>` → `install -m0755` → `systemctl restart felhom-agent` (non-root `felhom-agent` user, config `/etc/felhom-agent/agent.json`) |
| Ship configs | felhom-pve | sudoers (`/etc/sudoers.d/felhom-agent`) + guarded-mkfs wrapper WITH the binary when `configs/` changed |
| **Verify** (anyone, any time) | anywhere with the repo + Go | `git checkout v<ver> && go build -trimpath -buildvcs=false -ldflags "-X main.version=<ver>" -o /tmp/a ./cmd/felhom-agent && sha256sum /tmp/a` — must equal `curl -fsSL <pkg-url> \| sha256sum` |
| **Vouch** | hub operator UI | Configs → Day-0 artifacts. **Deliberately NOT automated** — vouching is what points machines at a version, and it stays your act (prove-then-vouch) |
| Verify | felhom-pve | `felhom-agent --version` + journal (clean ReassertGuestBinds, no capability degradation) |

## Proxmox model (the load-bearing rules)

- **API-first** via a scoped `FelhomAgent` token. Raw root-CLI is **fenced to exactly 3 exceptions**:
  keyctl `pct create` (golden image), USB mount/fstab, SMART/sensors. `Client` never shells out;
  `Privileged` never makes HTTP calls (asserted by `routing_test.go`). Keep that fence.
- **Every mutating op is async** → returns a UPID → `WaitTask` asserts `exitstatus == "OK"`. A 200 on
  the POST is **not** success; authorization can fail at task execution.
- **TLS:** SHA-256 leaf-cert pinning (self-signed host cert). No insecure default.
- **Privsep token gotcha:** a `--privsep 1` token's rights = intersection of the backing user's perms
  AND the token's ACLs — the role must be granted on **both**, or every call 403s.
- Destructive ops go through the reconcile gate / signed-jobs path — never call `Client.DestroyLXC`/
  `Vzdump`/`SetConfig` ad-hoc (REUSE.md §3).

## Demo host (for live tests)

Two demo nodes: **`demo-felhom`** (N100, `ssh felhom-pve`) and **`demo-hp`** (HP t740, `ssh demo-hp`,
no baked key — break-glass via the hub). **Addresses, routes, node names, break-glass recipe and what
is provisioned on each: `felhom.eu/documentation/operations/nodes.md`** — the single home; do not
restate them here, and re-check an address rather than trusting one written down.

**Which box is safe to break, and what may be done to each:
`felhom.eu/documentation/runbooks/target-selection.md`** — read it before any destructive test.
Drill and build VMs belong on `demo-hp`, not DooPlex (operator ruling 2026-07-25).

The agent pins the served leaf cert — verify the fingerprint still matches before a live run.
Selftest modes (run locally on DooPlex, pointed at the demo API):
`--selftest[=read|task|hub|storage|backup|restore-test|pbs-verify]`; no flag = the daemon.

<!--
`localapi` does not bind a LAN literal: since the R-50 island migration (2026-07-25) it binds
169.254.253.1:8443 on vmbr9, which is location-independent by design, and proxmox.endpoint is
https://127.0.0.1:8006. That is why the N100 being travel-portable needs no config edit. The
2026-07-20..08-02 vacation window and its findings are recorded in
felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md.

LEGACY: WINDOWS WORKSTATION — until 2026-07-19 CC ran on Windows 11; pct commands over SSH needed
export MSYS_NO_PATHCONV=1, and every remote command used SSH=/c/Windows/System32/OpenSSH/ssh.exe.
Agent deploy was a two-hop copy via the Windows box (cygpath -w for the local scp path; CRLF hazard
on config files). The workspace-root CLAUDE.md carries the full version.
-->


## Conventions

### Trunk-based — no branches

All shippable work commits **directly to `main`**; `main` equals what is deployed.
- Report-only artifacts (audits, findings, fixspecs) → `felhom.eu/documentation/` (`audits/`, `backlog/`).
- Risky/supervised fixes are spec'd, then implemented **during the supervised session, on `main`**.
- Unattended escape hatch: if a fix can't be cleanly verified/shipped, revert + report — never park on a branch.

> **In every repository where you make a change, update both files in that repo:**
> - **`CHANGELOG.md`** — cumulative log, newest on top.
> - **`REPORT.md`** — **overwrite** with the most recent implementation/validation summary only.
>
> **Never write secrets** into any committed file — reference them as "stored out-of-band".

- Code quality: verify generated code for bugs/edge cases; add debug logging; **ask rather than
  guess** when you'd otherwise invent input/output.
- **A health check issues no block I/O** — full rule, measurement and scope:
  `.claude/rules/health-checks.md`, which loads when you touch the packages that write them.
- Update `REUSE.md` if you added/changed/deprecated a shared helper or pattern (same commit).
- **Run `python3 scripts/agent_gates.py` from the repo root after ANY change in this repo.** It is
  the ONE entry point for this repo's gates. Today it runs one — `reuse_refs_check` over this
  repo's `REUSE.md` — and it exists at one gate on purpose: a census on 2026-08-02 found that every
  check a `CLAUDE.md` names was passing and two of the four nobody is told to run were failing, and
  this repo was the extreme case, with nothing running against it at all and 90 cited paths checked
  by no one. It grows when the agent grows a second check. `--fast` selects the gates that touch no
  network and no container runtime; today that is all of them. A missing gate is a FAILURE, never a
  skip. **The shared `reuse_refs_check.py` lives in `felhom.eu/scripts/` and is never copied here**
  — a copy would recreate the drift it detects; an absent sibling clone FAILS the gate.
  **The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It
  is per-clone — switch it on once with `git config core.hooksPath .githooks`, and a manual run
  WARNS when this clone is unarmed. `git push --no-verify` bypasses it deliberately; **say so in the
  session report when you use it.** Both facts are why CI is still owed (`OPEN-ITEMS.md` R-168).
- Testing doctrine (non-hollow tests, red-proofs, seams): use the `felhom-testing` skill.
- **Logging**: the slog logger fans out to journald (configured level) + the always-DEBUG `applog.Ring`
  (remote pulls) — English, keys-never-values, durations on outcomes; full rules in
  `felhom.eu/documentation/runbooks/logging-conventions.md`.

### Live validation

Exercise the SERVER-SIDE PIPELINE a real user triggers, end-to-end. The forbidden shortcut is
BYPASSING it (the F9 episode: raw guest-attach + hand-set state). Invoking the exact endpoint the UI
invokes is an acceptable proxy when a browser isn't available — say which method was used. Low-level
mechanism tests where the direct call IS the mechanism are exempt.

## Workflow & artifacts

- Implement **`TASK.md` / `TASK-*.md`** specs (when placed as `TASK.md` or told to), then push +
  CHANGELOG + REPORT.md.
- **`RUNBOOK-*.md`** — an operational procedure. CC executes the steps it has access and capability
  for, including live validation on the demo Proxmox host (CC has root@felhom-pve SSH + the
  felhom-agent token). Mark a step HUMAN only when it genuinely needs physical presence, a real-world
  decision, or credentials CC truly lacks. Judgment still applies: confirm before irreversible ops on
  real customer data — demo scratch guests are fair game.