admin
2f4ccab166
Both found while writing felhom.eu runbooks/target-selection.md. 1. The demo host block said the t740 is the designated drill+build VM host "but no drill VM is provisioned there yet". Stale since 2026-07-25 -- VM 300 (drill-r50) has been there since. The sentence read as discouragement from the very box the operator ruling designates, which is part of why a drill went to DooPlex instead. Now says the ruling is realized and to start there, points at the new target-selection runbook, and notes drill.qcow2 on DooPlex is a BAKE fixture, not a drill target. Agent version dropped (it changes several times a day; the hub host list and --version are the authorities) and the t740's PVE node name added. 2. RETRACTED the block's claim that the agent "does not run at all" at the remote site because localapi binds the LAN literal 192.168.0.162 and the service has "never started" -- with an outstanding config edit needing Viktor GO. That was true before R-50 and is false now: since the island migration (2026-07-25) localapi binds 169.254.253.1:8443 on vmbr9, which is location-independent by design, and proxmox.endpoint is https://127.0.0.1:8006. Verified live 2026-07-30: service active, version 0.115.0, and GET /disks answered over the island -- the whole R-116 payload capture went through it. A session trusting the old text would not have attempted the read that worked. The recorded remote-site address was also wrong (.162, not .147), so it now says re-check instead of asserting one. No code, no version bump, so no CHANGELOG entry (that file is version-keyed) and REPORT.md is left holding the v0.115.0 record rather than being overwritten by a docs fix.
163 lines
11 KiB
Markdown
163 lines
11 KiB
Markdown
# CLAUDE.md — `felhom-agent`
|
|
|
|
> Loads when Claude Code touches this repo. Stable orientation only — **current state lives in
|
|
> `CONTEXT.md` and the top of `CHANGELOG.md`**, never here. Cross-repo orientation: workspace-root
|
|
> `/mnt/5_hdd/felhom.eu/git/CLAUDE.md`.
|
|
|
|
## What this repo is
|
|
|
|
`felhom-agent` is the operator-tier **host agent** that runs on each Proxmox host and owns **all**
|
|
Proxmox interaction: provision/restore guests, host storage, backup/restore orchestration, the hub
|
|
control loop, and a narrow per-guest local API. It is the **most privilege-sensitive** component.
|
|
|
|
- Renamed former `proxmox-controller` repo.
|
|
- **Distinct from `felhom-controller`** — that is the *in-guest* controller (Docker-only, no Proxmox
|
|
creds). Do not confuse them.
|
|
- Control plane, not data plane: if the agent dies, apps keep serving; only management degrades.
|
|
|
|
## Read before writing code
|
|
|
|
- **`REUSE.md`** — canonical helpers, format-safety guards, traps, seams. Check it first; update it
|
|
in the same commit that changes a shared helper or pattern.
|
|
- `CONTEXT.md` (current state + open threads) and the top `CHANGELOG.md` entry (authoritative history).
|
|
- Design doc: `felhom.eu/documentation/architecture/03-host-agent.md` (locked). Platform facts:
|
|
`felhom.eu/documentation/proxmox-platform.md` + `tests/phase{0,1-2,3,4}-findings.md`.
|
|
|
|
## Layout (verified against the tree)
|
|
|
|
```
|
|
cmd/felhom-agent/ main + flags + --selftest modes + the daemon entry
|
|
cmd/felhom-opsign/ offline operator signing CLI (SSHSIG)
|
|
internal/authz/ operator signed-op verifier (SSHSIG) + durable FileNonceStore
|
|
internal/backup/ vzdump backup runner + restore-test scheduler + report store
|
|
internal/capability/ live sudo-policy capability probe (degradation visibility)
|
|
internal/config/ JSON config + FELHOM_AGENT_* env overlay; secrets redacted (Redacted())
|
|
internal/desired/ hub desired-state syncer (envelope observer)
|
|
internal/escrow/ PBS-key escrow (zero-knowledge recovery code)
|
|
internal/guesthook/ pre-start self-heal hookscript install
|
|
internal/hub/ daemon: HostReport collector + Bearer client + resilient Loop
|
|
internal/lanresolver/ split-horizon DNS on guest IP change (dnsmasq RESTART, not reload)
|
|
internal/localapi/ per-guest local API: token store, disks/format, guest binds, controller swap,
|
|
stale-lock recovery, pinned self-signed leaf
|
|
internal/log/ slog setup
|
|
internal/pbs/ PBS-API client (fingerprint-pinned) + verify maintenance loop
|
|
internal/provision/ guest bootstrap back-half (token mint → bootstrap.json → pct bind)
|
|
internal/proxmox/ API-first Client + fenced root-CLI Privileged + UPID WaitTask
|
|
internal/reconcile/ reconcile engine + reversibility gate + op journal + crash recovery
|
|
internal/signedjobs/ operator-signed destructive executors (wipe, decommission)
|
|
internal/storage/ storage observer + durable ids + role/claim classifiers + SudoHostOps + watchdog
|
|
```
|
|
|
|
## Build / run
|
|
|
|
- Module `gitea.dooplex.hu/admin/felhom-agent`; binary `felhom-agent` (`cmd/felhom-agent/`).
|
|
- **Pure Go stdlib + `golang.org/x/crypto` only** — no web frameworks. `go.mod` directive go 1.25.0;
|
|
DooPlex (192.168.0.180, where CC runs) has the Go toolchain and is on the same LAN as the demo
|
|
host — build and run live tests locally.
|
|
- Version via `-ldflags "-X main.version=<v>"`; `--version` flag. Bump on meaningful changes + CHANGELOG entry.
|
|
- **Full build/deploy/publish runbook: use the `felhom-build-deploy` skill.** Summary:
|
|
|
|
> **Clean-tree gate before any build:** `git status --porcelain` must be empty and
|
|
> `git rev-parse HEAD` must equal `git rev-parse origin/main` in the repo being built. An unpushed
|
|
> change does not exist — never build a dirty or unpushed tree. The `git pull` in the build step
|
|
> stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from
|
|
> elsewhere).
|
|
|
|
| Step | Where | One-liner |
|
|
|---|---|---|
|
|
| Build | DooPlex (local) | `cd /mnt/5_hdd/felhom.eu/git/felhom-agent && git pull && go build -ldflags '-X main.version=<v>' -o /tmp/felhom-agent-<v> ./cmd/felhom-agent` |
|
|
| Copy | local → felhom-pve | `scp /tmp/felhom-agent-<v> felhom-pve:/tmp/` (one hop) |
|
|
| Deploy | felhom-pve | backup `.bak-<old>` → `install -m0755` → `systemctl restart felhom-agent` (non-root `felhom-agent` user, config `/etc/felhom-agent/agent.json`) |
|
|
| Ship configs | felhom-pve | sudoers (`/etc/sudoers.d/felhom-agent`) + guarded-mkfs wrapper WITH the binary when `configs/` changed |
|
|
| Publish | DooPlex (local) | `scripts/publish-agent.sh <ver> <bin>` (REGISTRY_* creds); hub Day-0 manifest vouch = operator follow-up |
|
|
| Verify | felhom-pve | `felhom-agent --version` + journal (clean ReassertGuestBinds, no capability degradation) |
|
|
|
|
## Proxmox model (the load-bearing rules)
|
|
|
|
- **API-first** via a scoped `FelhomAgent` token. Raw root-CLI is **fenced to exactly 3 exceptions**:
|
|
keyctl `pct create` (golden image), USB mount/fstab, SMART/sensors. `Client` never shells out;
|
|
`Privileged` never makes HTTP calls (asserted by `routing_test.go`). Keep that fence.
|
|
- **Every mutating op is async** → returns a UPID → `WaitTask` asserts `exitstatus == "OK"`. A 200 on
|
|
the POST is **not** success; authorization can fail at task execution.
|
|
- **TLS:** SHA-256 leaf-cert pinning (self-signed host cert). No insecure default.
|
|
- **Privsep token gotcha:** a `--privsep 1` token's rights = intersection of the backing user's perms
|
|
AND the token's ACLs — the role must be granted on **both**, or every call 403s.
|
|
- Destructive ops go through the reconcile gate / signed-jobs path — never call `Client.DestroyLXC`/
|
|
`Vzdump`/`SetConfig` ad-hoc (REUSE.md §3).
|
|
|
|
## Demo host (for live tests)
|
|
|
|
Node **`demo-felhom`**, API `https://192.168.0.162:8006`. SSH alias `felhom-pve` (root@pam) —
|
|
available to CC as plain `ssh felhom-pve`. A **second demo node `demo-hp`** (HP t740, node name
|
|
`felhom-host`, `ssh demo-hp` — no baked key; break-glass root via hub `host_recovery/demo-hp-bb76ea` +
|
|
`sshpass`) is the **designated drill+build VM host** per the 2026-07-25 operator ruling, and that ruling
|
|
is **realized** — it hosts drill VM `300` (`drill-r50`), so **start there**, not on DooPlex. (The
|
|
historical golden-bake `drill.qcow2` still lives on DooPlex and is a bake fixture, not a drill target.)
|
|
**Which box is safe to break, and what may be done to each:
|
|
`felhom.eu/documentation/runbooks/target-selection.md`** — read it before any destructive test. Both
|
|
nodes + the break-glass recipe: `felhom.eu/documentation/operations/nodes.md`. The agent pins the served leaf cert — verify the
|
|
fingerprint still matches before a live run. Selftest modes (run locally on DooPlex, pointed at the
|
|
demo API): `--selftest[=read|task|hub|storage|backup|restore-test|pbs-verify]`; no flag = the daemon.
|
|
|
|
> **TEMPORARY — felhom-pve is at a remote site (until ~2026-08-02).** The home-LAN literal
|
|
> `192.168.0.162` is NOT reachable from DooPlex for the duration. Access via Tailscale:
|
|
> felhom-pve = 100.70.170.35; the `Host felhom-pve` entry in `~/.ssh/config` on DooPlex already
|
|
> points there (the direct-LAN path stays available as `Host felhom-pve-lan`). Delete this block on
|
|
> return. All documented `ssh felhom-pve` / `pct exec` workflows are unchanged. Path is **direct**
|
|
> (not DERP), ~37 ms rtt per hop. At the remote site the host is on **DHCP**; re-check its address
|
|
> rather than trusting one written here (`ip -br addr show vmbr0` — it read `192.168.0.162/24` on
|
|
> 2026-07-30, and `felhom-pve-lan` from DooPlex is still `No route to host`). Details + findings:
|
|
> `felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md`
|
|
>
|
|
> **The "agent does not run at the remote site" warning this block used to carry is RETRACTED
|
|
> (2026-07-30) — it was true before R-50 and is false now.** `localapi` no longer binds a LAN literal:
|
|
> since the R-50 island migration (2026-07-25) it binds `169.254.253.1:8443` on `vmbr9`, which is
|
|
> location-independent by design, and `proxmox.endpoint` is `https://127.0.0.1:8006`. Verified live:
|
|
> `systemctl is-active felhom-agent` → `active`, `felhom-agent --version` → 0.115.0, and the per-guest
|
|
> local API answered `GET /disks` over the island. No config edit and no Viktor GO are outstanding.
|
|
|
|
> **Legacy: Windows workstation.** Until 2026-07-19 CC ran on Windows 11; `pct` commands over SSH
|
|
> needed `export MSYS_NO_PATHCONV=1`, and every remote command used
|
|
> `SSH=/c/Windows/System32/OpenSSH/ssh.exe`. Agent deploy was a two-hop copy via the Windows box
|
|
> (`cygpath -w` for the local scp path; CRLF hazard on config files).
|
|
|
|
## Conventions
|
|
|
|
### Trunk-based — no branches
|
|
|
|
All shippable work commits **directly to `main`**; `main` equals what is deployed.
|
|
- Report-only artifacts (audits, findings, fixspecs) → `felhom.eu/documentation/` (`audits/`, `backlog/`).
|
|
- Risky/supervised fixes are spec'd, then implemented **during the supervised session, on `main`**.
|
|
- Unattended escape hatch: if a fix can't be cleanly verified/shipped, revert + report — never park on a branch.
|
|
|
|
> **In every repository where you make a change, update both files in that repo:**
|
|
> - **`CHANGELOG.md`** — cumulative log, newest on top.
|
|
> - **`REPORT.md`** — **overwrite** with the most recent implementation/validation summary only.
|
|
>
|
|
> **Never write secrets** into any committed file — reference them as "stored out-of-band".
|
|
|
|
- Code quality: verify generated code for bugs/edge cases; add debug logging; **ask rather than
|
|
guess** when you'd otherwise invent input/output.
|
|
- Update `REUSE.md` if you added/changed/deprecated a shared helper or pattern (same commit).
|
|
- Testing doctrine (non-hollow tests, red-proofs, seams): use the `felhom-testing` skill.
|
|
- **Logging**: the slog logger fans out to journald (configured level) + the always-DEBUG `applog.Ring`
|
|
(remote pulls) — English, keys-never-values, durations on outcomes; full rules in
|
|
`felhom.eu/documentation/runbooks/logging-conventions.md`.
|
|
|
|
### Live validation
|
|
|
|
Exercise the SERVER-SIDE PIPELINE a real user triggers, end-to-end. The forbidden shortcut is
|
|
BYPASSING it (the F9 episode: raw guest-attach + hand-set state). Invoking the exact endpoint the UI
|
|
invokes is an acceptable proxy when a browser isn't available — say which method was used. Low-level
|
|
mechanism tests where the direct call IS the mechanism are exempt.
|
|
|
|
## Workflow & artifacts
|
|
|
|
- Implement **`TASK.md` / `TASK-*.md`** specs (when placed as `TASK.md` or told to), then push +
|
|
CHANGELOG + REPORT.md.
|
|
- **`RUNBOOK-*.md`** — an operational procedure. CC executes the steps it has access and capability
|
|
for, including live validation on the demo Proxmox host (CC has root@felhom-pve SSH + the
|
|
felhom-agent token). Mark a step HUMAN only when it genuinely needs physical presence, a real-world
|
|
decision, or credentials CC truly lacks. Judgment still applies: confirm before irreversible ops on
|
|
real customer data — demo scratch guests are fair game.
|