felhom-agent/CLAUDE.md

CLAUDE.md — felhom-agent

Loads when Claude Code touches this repo. Stable orientation only — current state lives in CONTEXT.md and the top of CHANGELOG.md, never here. Cross-repo orientation: workspace-root /mnt/5_hdd/felhom.eu/git/CLAUDE.md.

What this repo is

felhom-agent is the operator-tier host agent that runs on each Proxmox host and owns all Proxmox interaction: provision/restore guests, host storage, backup/restore orchestration, the hub control loop, and a narrow per-guest local API. It is the most privilege-sensitive component.

• Renamed former proxmox-controller repo.
• Distinct from felhom-controller — that is the in-guest controller (Docker-only, no Proxmox creds). Do not confuse them.
• Control plane, not data plane: if the agent dies, apps keep serving; only management degrades.

Read before writing code

• REUSE.md — canonical helpers, format-safety guards, traps, seams. Check it first; update it in the same commit that changes a shared helper or pattern.
• CONTEXT.md (current state + open threads) and the top CHANGELOG.md entry (authoritative history).
• Design doc: felhom.eu/documentation/architecture/03-host-agent.md (locked). Platform facts: felhom.eu/documentation/proxmox-platform.md + tests/phase{0,1-2,3,4}-findings.md.

Layout (verified against the tree)

cmd/felhom-agent/    main + flags + --selftest modes + the daemon entry
cmd/felhom-opsign/   offline operator signing CLI (SSHSIG)
internal/authz/      operator signed-op verifier (SSHSIG) + durable FileNonceStore
internal/backup/     vzdump backup runner + restore-test scheduler + report store
internal/capability/ live sudo-policy capability probe (degradation visibility)
internal/config/     JSON config + FELHOM_AGENT_* env overlay; secrets redacted (Redacted())
internal/desired/    hub desired-state syncer (envelope observer)
internal/escrow/     PBS-key escrow (zero-knowledge recovery code)
internal/guesthook/  pre-start self-heal hookscript install
internal/hub/        daemon: HostReport collector + Bearer client + resilient Loop
internal/lanresolver/ split-horizon DNS on guest IP change (dnsmasq RESTART, not reload)
internal/localapi/   per-guest local API: token store, disks/format, guest binds, controller swap,
                     stale-lock recovery, pinned self-signed leaf
internal/log/        slog setup
internal/pbs/        PBS-API client (fingerprint-pinned) + verify maintenance loop
internal/provision/  guest bootstrap back-half (token mint → bootstrap.json → pct bind)
internal/proxmox/    API-first Client + fenced root-CLI Privileged + UPID WaitTask
internal/reconcile/  reconcile engine + reversibility gate + op journal + crash recovery
internal/signedjobs/ operator-signed destructive executors (wipe, decommission)
internal/storage/    storage observer + durable ids + role/claim classifiers + SudoHostOps + watchdog

Build / run

• Module gitea.dooplex.hu/admin/felhom-agent; binary felhom-agent (cmd/felhom-agent/).
• Pure Go stdlib + golang.org/x/crypto only — no web frameworks. go.mod directive go 1.25.0; DooPlex (192.168.0.180, where CC runs) has the Go toolchain and is on the same LAN as the demo host — build and run live tests locally.
• Version via -ldflags "-X main.version=<v>"; --version flag. Bump on meaningful changes + CHANGELOG entry.
• Full build/deploy/publish runbook: use the felhom-build-deploy skill. Summary:

Clean-tree gate before any build: git status --porcelain must be empty and git rev-parse HEAD must equal git rev-parse origin/main in the repo being built. An unpushed change does not exist — never build a dirty or unpushed tree. The git pull in the build step stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from elsewhere).

RELEASING IS ONE COMMAND, AND IT PUBLISHES (R-115). There used to be a raw go build line here and a separate "Publish" row, so publishing was a step someone had to remember — and it was forgotten three times in five days, the last leaving agent v0.120.0 deployed on both demo hosts and undownloadable, where a documented-path reinstall would have silently downgraded them while reporting success. Do not hand-roll the build: the script also creates the v<version> git TAG that felhom-host-install.sh fetches this version's sixteen config files from (R-183), and it verifies by an independent download rather than trusting the publish step's own output. scripts/publish-agent.sh still exists and is still correct — the release script CALLS it rather than reimplementing it.

THE ORDER IS build → tag LOCALLY → publish → push tag, and each step protects something (R-188, R-186). The tag is created before the publish so the build and the tag describe the same commit; it is pushed after, because the push is what wakes CI (on: [push]) and a tag visible before its package makes the published-versions gate correctly fail a correct release — it did, on roughly every second release, and R-168 sends that failure to you by mail. The invariant the old order protected is asserted directly instead: the gate now also refuses a published version with no tag. If the push fails after a successful publish the script says so loudly and prints the one-line recovery; if the publish fails it removes the local-only tag so a retry is clean.

A RELEASED BINARY IS INDEPENDENTLY VERIFIABLE (R-186). The build uses -trimpath -buildvcs=false so the same source produces the same bytes whether or not the tag exists yet — before this, a rebuild could not reproduce the sha you were vouching. To check any published version yourself:

V=0.122.0
git checkout "v$V" && go build -trimpath -buildvcs=false -ldflags "-X main.version=$V" \
    -o /tmp/felhom-agent-check ./cmd/felhom-agent
sha256sum /tmp/felhom-agent-check
curl -fsSL "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/$V/felhom-agent" | sha256sum

The two hashes must match. publish-agent.sh's fallback build uses the same flags — it used to force CGO_ENABLED=0 and produce a 74 KB-smaller binary for the same version; if either build line ever changes, change both or one version name means two binaries again.

Step	Where	One-liner
Release (build + tag + publish + verify)	DooPlex (local)	GITEA_USER=admin GITEA_TOKEN=<tok> scripts/release-agent.sh <ver> — refuses a dirty/unpushed tree and refuses to re-release an existing version
Copy	local → felhom-pve	scp /tmp/felhom-agent-<v> felhom-pve:/tmp/ (one hop)
Deploy	felhom-pve	backup .bak-<old> → install -m0755 → systemctl restart felhom-agent (non-root felhom-agent user, config /etc/felhom-agent/agent.json)
Ship configs	felhom-pve	sudoers (/etc/sudoers.d/felhom-agent) + guarded-mkfs wrapper WITH the binary when configs/ changed
Verify (anyone, any time)	anywhere with the repo + Go	git checkout v<ver> && go build -trimpath -buildvcs=false -ldflags "-X main.version=<ver>" -o /tmp/a ./cmd/felhom-agent && sha256sum /tmp/a — must equal curl -fsSL <pkg-url> | sha256sum
Vouch	hub operator UI	Configs → Day-0 artifacts. Deliberately NOT automated — vouching is what points machines at a version, and it stays your act (prove-then-vouch)
Verify	felhom-pve	felhom-agent --version + journal (clean ReassertGuestBinds, no capability degradation)

Proxmox model (the load-bearing rules)

• API-first via a scoped FelhomAgent token. Raw root-CLI is fenced to exactly 3 exceptions: keyctl pct create (golden image), USB mount/fstab, SMART/sensors. Client never shells out; Privileged never makes HTTP calls (asserted by routing_test.go). Keep that fence.
• Every mutating op is async → returns a UPID → WaitTask asserts exitstatus == "OK". A 200 on the POST is not success; authorization can fail at task execution.
• TLS: SHA-256 leaf-cert pinning (self-signed host cert). No insecure default.
• Privsep token gotcha: a --privsep 1 token's rights = intersection of the backing user's perms AND the token's ACLs — the role must be granted on both, or every call 403s.
• Destructive ops go through the reconcile gate / signed-jobs path — never call Client.DestroyLXC/ Vzdump/SetConfig ad-hoc (REUSE.md §3).

Demo host (for live tests)

Node demo-felhom, API https://192.168.0.162:8006. SSH alias felhom-pve (root@pam) — available to CC as plain ssh felhom-pve. A second demo node demo-hp (HP t740, node name felhom-host, ssh demo-hp — no baked key; break-glass root via hub host_recovery/demo-hp-bb76ea + sshpass) is the designated drill+build VM host per the 2026-07-25 operator ruling, and that ruling is realized — it hosts drill VM 300 (drill-r50), so start there, not on DooPlex. (The historical golden-bake drill.qcow2 still lives on DooPlex and is a bake fixture, not a drill target.) Which box is safe to break, and what may be done to each: felhom.eu/documentation/runbooks/target-selection.md — read it before any destructive test. Both nodes + the break-glass recipe: felhom.eu/documentation/operations/nodes.md. The agent pins the served leaf cert — verify the fingerprint still matches before a live run. Selftest modes (run locally on DooPlex, pointed at the demo API): --selftest[=read|task|hub|storage|backup|restore-test|pbs-verify]; no flag = the daemon.

TEMPORARY — felhom-pve is at a remote site (until ~2026-08-02). The home-LAN literal 192.168.0.162 is NOT reachable from DooPlex for the duration. Access via Tailscale: felhom-pve = 100.70.170.35; the Host felhom-pve entry in ~/.ssh/config on DooPlex already points there (the direct-LAN path stays available as Host felhom-pve-lan). Delete this block on return. All documented ssh felhom-pve / pct exec workflows are unchanged. Path is direct (not DERP), ~37 ms rtt per hop. At the remote site the host is on DHCP; re-check its address rather than trusting one written here (ip -br addr show vmbr0 — it read 192.168.0.162/24 on 2026-07-30, and felhom-pve-lan from DooPlex is still No route to host). Details + findings: felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md

The "agent does not run at the remote site" warning this block used to carry is RETRACTED (2026-07-30) — it was true before R-50 and is false now. localapi no longer binds a LAN literal: since the R-50 island migration (2026-07-25) it binds 169.254.253.1:8443 on vmbr9, which is location-independent by design, and proxmox.endpoint is https://127.0.0.1:8006. Verified live: systemctl is-active felhom-agent → active, felhom-agent --version → 0.115.0, and the per-guest local API answered GET /disks over the island. No config edit and no Viktor GO are outstanding.

Legacy: Windows workstation. Until 2026-07-19 CC ran on Windows 11; pct commands over SSH needed export MSYS_NO_PATHCONV=1, and every remote command used SSH=/c/Windows/System32/OpenSSH/ssh.exe. Agent deploy was a two-hop copy via the Windows box (cygpath -w for the local scp path; CRLF hazard on config files).

Conventions

Trunk-based — no branches

All shippable work commits directly to main; main equals what is deployed.

• Report-only artifacts (audits, findings, fixspecs) → felhom.eu/documentation/ (audits/, backlog/).
• Risky/supervised fixes are spec'd, then implemented during the supervised session, on main.
• Unattended escape hatch: if a fix can't be cleanly verified/shipped, revert + report — never park on a branch.

In every repository where you make a change, update both files in that repo:

• CHANGELOG.md — cumulative log, newest on top.
• REPORT.md — overwrite with the most recent implementation/validation summary only.

Never write secrets into any committed file — reference them as "stored out-of-band".

• Code quality: verify generated code for bugs/edge cases; add debug logging; ask rather than guess when you'd otherwise invent input/output.
• A health check issues no block I/O — no statfs, no getdents, no read, write or fsync, not even behind a timeout. Liveness is decided from /proc and kernel state. The full rule + the measurement lives in felhom.eu/CLAUDE.md "Code quality rules"; it is repeated here because health checks are written in THIS repo and that file does not load in an agent-only session. R-117 spike §6.3.
• Update REUSE.md if you added/changed/deprecated a shared helper or pattern (same commit).
• Run python3 scripts/agent_gates.py from the repo root after ANY change in this repo. It is the ONE entry point for this repo's gates. Today it runs one — reuse_refs_check over this repo's REUSE.md — and it exists at one gate on purpose: a census on 2026-08-02 found that every check a CLAUDE.md names was passing and two of the four nobody is told to run were failing, and this repo was the extreme case, with nothing running against it at all and 90 cited paths checked by no one. It grows when the agent grows a second check. --fast selects the gates that touch no network and no container runtime; today that is all of them. A missing gate is a FAILURE, never a skip. The shared reuse_refs_check.py lives in felhom.eu/scripts/ and is never copied here — a copy would recreate the drift it detects; an absent sibling clone FAILS the gate. The pre-push hook (.githooks/pre-push) runs it with --fast and refuses a failing push. It is per-clone — switch it on once with git config core.hooksPath .githooks, and a manual run WARNS when this clone is unarmed. git push --no-verify bypasses it deliberately; say so in the session report when you use it. Both facts are why CI is still owed (OPEN-ITEMS.md R-168).
• Testing doctrine (non-hollow tests, red-proofs, seams): use the felhom-testing skill.
• Logging: the slog logger fans out to journald (configured level) + the always-DEBUG applog.Ring (remote pulls) — English, keys-never-values, durations on outcomes; full rules in felhom.eu/documentation/runbooks/logging-conventions.md.

Live validation

Exercise the SERVER-SIDE PIPELINE a real user triggers, end-to-end. The forbidden shortcut is BYPASSING it (the F9 episode: raw guest-attach + hand-set state). Invoking the exact endpoint the UI invokes is an acceptable proxy when a browser isn't available — say which method was used. Low-level mechanism tests where the direct call IS the mechanism are exempt.

Workflow & artifacts

• Implement TASK.md / TASK-*.md specs (when placed as TASK.md or told to), then push + CHANGELOG + REPORT.md.
• RUNBOOK-*.md — an operational procedure. CC executes the steps it has access and capability for, including live validation on the demo Proxmox host (CC has root@felhom-pve SSH + the felhom-agent token). Mark a step HUMAN only when it genuinely needs physical presence, a real-world decision, or credentials CC truly lacks. Judgment still applies: confirm before irreversible ops on real customer data — demo scratch guests are fair game.