admin
7581f8140a
gates / gates (push) Successful in 7s
All three are the reporting and release path misreporting its own work. No
customer machine, no backup, no restore, no data. The restore-test itself and
when it runs are unchanged.
R-189 — a passing restore-test no longer vanishes on a restart. restore_tests[]
came only from the in-memory store, whose comment ("lost on restart; the cadence
re-populates") was true under a timer and stopped being true when R-86 made the
agent refuse to re-test a proven archive: the proof is then not repeated for a
whole archive generation. Observed live — a 14.5 GB offsite PASS reached no
host-report because the agent was restarted 2m43s later. RestoreTestState now
carries tier + verified beside the archive and renders reportable entries; the
collector merges them, one per tier, newest by TestedAt. It refuses to lie: a
record missing archive-or-tier produces no entry, and run mechanics are not
re-invented. Only successes are persisted, and the asymmetry is now written where
it will be read.
R-188 — a correct release stops emailing a failure. Only the tag PUSH moved
(build -> tag locally -> publish -> push tag): the push wakes CI, and a tag
visible before its package made the gate correctly fail a correct release about
half the time. The old order's invariant is asserted directly instead — the gate
now refuses a published version with no tag, as a bounded probe that prints its
own coverage, because the package listing api is still 401 without a token.
R-186 — a released binary can be verified by rebuilding it. -trimpath
-buildvcs=false: same source, same bytes, tag or no tag. Measured. publish-agent's
fallback also forced CGO_ENABLED=0 and produced a 74 KB different binary for the
same version; both paths now build identically. CLAUDE.md records the command.
217 lines
15 KiB
Markdown
217 lines
15 KiB
Markdown
# CLAUDE.md — `felhom-agent`
|
|
|
|
> Loads when Claude Code touches this repo. Stable orientation only — **current state lives in
|
|
> `CONTEXT.md` and the top of `CHANGELOG.md`**, never here. Cross-repo orientation: workspace-root
|
|
> `/mnt/5_hdd/felhom.eu/git/CLAUDE.md`.
|
|
|
|
## What this repo is
|
|
|
|
`felhom-agent` is the operator-tier **host agent** that runs on each Proxmox host and owns **all**
|
|
Proxmox interaction: provision/restore guests, host storage, backup/restore orchestration, the hub
|
|
control loop, and a narrow per-guest local API. It is the **most privilege-sensitive** component.
|
|
|
|
- Renamed former `proxmox-controller` repo.
|
|
- **Distinct from `felhom-controller`** — that is the *in-guest* controller (Docker-only, no Proxmox
|
|
creds). Do not confuse them.
|
|
- Control plane, not data plane: if the agent dies, apps keep serving; only management degrades.
|
|
|
|
## Read before writing code
|
|
|
|
- **`REUSE.md`** — canonical helpers, format-safety guards, traps, seams. Check it first; update it
|
|
in the same commit that changes a shared helper or pattern.
|
|
- `CONTEXT.md` (current state + open threads) and the top `CHANGELOG.md` entry (authoritative history).
|
|
- Design doc: `felhom.eu/documentation/architecture/03-host-agent.md` (locked). Platform facts:
|
|
`felhom.eu/documentation/proxmox-platform.md` + `tests/phase{0,1-2,3,4}-findings.md`.
|
|
|
|
## Layout (verified against the tree)
|
|
|
|
```
|
|
cmd/felhom-agent/ main + flags + --selftest modes + the daemon entry
|
|
cmd/felhom-opsign/ offline operator signing CLI (SSHSIG)
|
|
internal/authz/ operator signed-op verifier (SSHSIG) + durable FileNonceStore
|
|
internal/backup/ vzdump backup runner + restore-test scheduler + report store
|
|
internal/capability/ live sudo-policy capability probe (degradation visibility)
|
|
internal/config/ JSON config + FELHOM_AGENT_* env overlay; secrets redacted (Redacted())
|
|
internal/desired/ hub desired-state syncer (envelope observer)
|
|
internal/escrow/ PBS-key escrow (zero-knowledge recovery code)
|
|
internal/guesthook/ pre-start self-heal hookscript install
|
|
internal/hub/ daemon: HostReport collector + Bearer client + resilient Loop
|
|
internal/lanresolver/ split-horizon DNS on guest IP change (dnsmasq RESTART, not reload)
|
|
internal/localapi/ per-guest local API: token store, disks/format, guest binds, controller swap,
|
|
stale-lock recovery, pinned self-signed leaf
|
|
internal/log/ slog setup
|
|
internal/pbs/ PBS-API client (fingerprint-pinned) + verify maintenance loop
|
|
internal/provision/ guest bootstrap back-half (token mint → bootstrap.json → pct bind)
|
|
internal/proxmox/ API-first Client + fenced root-CLI Privileged + UPID WaitTask
|
|
internal/reconcile/ reconcile engine + reversibility gate + op journal + crash recovery
|
|
internal/signedjobs/ operator-signed destructive executors (wipe, decommission)
|
|
internal/storage/ storage observer + durable ids + role/claim classifiers + SudoHostOps + watchdog
|
|
```
|
|
|
|
## Build / run
|
|
|
|
- Module `gitea.dooplex.hu/admin/felhom-agent`; binary `felhom-agent` (`cmd/felhom-agent/`).
|
|
- **Pure Go stdlib + `golang.org/x/crypto` only** — no web frameworks. `go.mod` directive go 1.25.0;
|
|
DooPlex (192.168.0.180, where CC runs) has the Go toolchain and is on the same LAN as the demo
|
|
host — build and run live tests locally.
|
|
- Version via `-ldflags "-X main.version=<v>"`; `--version` flag. Bump on meaningful changes + CHANGELOG entry.
|
|
- **Full build/deploy/publish runbook: use the `felhom-build-deploy` skill.** Summary:
|
|
|
|
> **Clean-tree gate before any build:** `git status --porcelain` must be empty and
|
|
> `git rev-parse HEAD` must equal `git rev-parse origin/main` in the repo being built. An unpushed
|
|
> change does not exist — never build a dirty or unpushed tree. The `git pull` in the build step
|
|
> stays (it is a no-op when you work in this tree, and load-bearing if anything was pushed from
|
|
> elsewhere).
|
|
|
|
> **RELEASING IS ONE COMMAND, AND IT PUBLISHES (R-115).** There used to be a raw `go build` line
|
|
> here and a *separate* "Publish" row, so publishing was a step someone had to remember — and it was
|
|
> **forgotten three times in five days**, the last leaving agent v0.120.0 deployed on both demo hosts
|
|
> and undownloadable, where a documented-path reinstall would have silently downgraded them while
|
|
> reporting success. Do not hand-roll the build: the script also creates the `v<version>` git TAG
|
|
> that `felhom-host-install.sh` fetches this version's sixteen config files from (R-183), and it
|
|
> verifies by an **independent download** rather than trusting the publish step's own output.
|
|
> `scripts/publish-agent.sh` still exists and is still correct — the release script CALLS it rather
|
|
> than reimplementing it.
|
|
>
|
|
> **THE ORDER IS build → tag LOCALLY → publish → push tag, and each step protects something (R-188,
|
|
> R-186).** The tag is created before the publish so the build and the tag describe the same commit;
|
|
> it is *pushed* after, because the push is what wakes CI (`on: [push]`) and a tag visible before its
|
|
> package makes the published-versions gate correctly fail a correct release — it did, on roughly
|
|
> every second release, and R-168 sends that failure to you by mail. The invariant the old order
|
|
> protected is asserted directly instead: the gate now also refuses a **published version with no
|
|
> tag**. If the push fails after a successful publish the script says so loudly and prints the
|
|
> one-line recovery; if the *publish* fails it removes the local-only tag so a retry is clean.
|
|
>
|
|
> **A RELEASED BINARY IS INDEPENDENTLY VERIFIABLE (R-186).** The build uses `-trimpath
|
|
> -buildvcs=false` so the same source produces the same bytes whether or not the tag exists yet —
|
|
> before this, a rebuild could not reproduce the sha you were vouching. To check any published
|
|
> version yourself:
|
|
>
|
|
> ```bash
|
|
> V=0.122.0
|
|
> git checkout "v$V" && go build -trimpath -buildvcs=false -ldflags "-X main.version=$V" \
|
|
> -o /tmp/felhom-agent-check ./cmd/felhom-agent
|
|
> sha256sum /tmp/felhom-agent-check
|
|
> curl -fsSL "https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/$V/felhom-agent" | sha256sum
|
|
> ```
|
|
>
|
|
> The two hashes must match. `publish-agent.sh`'s fallback build uses the **same** flags — it used to
|
|
> force `CGO_ENABLED=0` and produce a 74 KB-smaller binary for the same version; if either build line
|
|
> ever changes, change both or one version name means two binaries again.
|
|
|
|
| Step | Where | One-liner |
|
|
|---|---|---|
|
|
| **Release** (build + tag + publish + verify) | DooPlex (local) | `GITEA_USER=admin GITEA_TOKEN=<tok> scripts/release-agent.sh <ver>` — refuses a dirty/unpushed tree and refuses to re-release an existing version |
|
|
| Copy | local → felhom-pve | `scp /tmp/felhom-agent-<v> felhom-pve:/tmp/` (one hop) |
|
|
| Deploy | felhom-pve | backup `.bak-<old>` → `install -m0755` → `systemctl restart felhom-agent` (non-root `felhom-agent` user, config `/etc/felhom-agent/agent.json`) |
|
|
| Ship configs | felhom-pve | sudoers (`/etc/sudoers.d/felhom-agent`) + guarded-mkfs wrapper WITH the binary when `configs/` changed |
|
|
| **Verify** (anyone, any time) | anywhere with the repo + Go | `git checkout v<ver> && go build -trimpath -buildvcs=false -ldflags "-X main.version=<ver>" -o /tmp/a ./cmd/felhom-agent && sha256sum /tmp/a` — must equal `curl -fsSL <pkg-url> \| sha256sum` |
|
|
| **Vouch** | hub operator UI | Configs → Day-0 artifacts. **Deliberately NOT automated** — vouching is what points machines at a version, and it stays your act (prove-then-vouch) |
|
|
| Verify | felhom-pve | `felhom-agent --version` + journal (clean ReassertGuestBinds, no capability degradation) |
|
|
|
|
## Proxmox model (the load-bearing rules)
|
|
|
|
- **API-first** via a scoped `FelhomAgent` token. Raw root-CLI is **fenced to exactly 3 exceptions**:
|
|
keyctl `pct create` (golden image), USB mount/fstab, SMART/sensors. `Client` never shells out;
|
|
`Privileged` never makes HTTP calls (asserted by `routing_test.go`). Keep that fence.
|
|
- **Every mutating op is async** → returns a UPID → `WaitTask` asserts `exitstatus == "OK"`. A 200 on
|
|
the POST is **not** success; authorization can fail at task execution.
|
|
- **TLS:** SHA-256 leaf-cert pinning (self-signed host cert). No insecure default.
|
|
- **Privsep token gotcha:** a `--privsep 1` token's rights = intersection of the backing user's perms
|
|
AND the token's ACLs — the role must be granted on **both**, or every call 403s.
|
|
- Destructive ops go through the reconcile gate / signed-jobs path — never call `Client.DestroyLXC`/
|
|
`Vzdump`/`SetConfig` ad-hoc (REUSE.md §3).
|
|
|
|
## Demo host (for live tests)
|
|
|
|
Node **`demo-felhom`**, API `https://192.168.0.162:8006`. SSH alias `felhom-pve` (root@pam) —
|
|
available to CC as plain `ssh felhom-pve`. A **second demo node `demo-hp`** (HP t740, node name
|
|
`felhom-host`, `ssh demo-hp` — no baked key; break-glass root via hub `host_recovery/demo-hp-bb76ea` +
|
|
`sshpass`) is the **designated drill+build VM host** per the 2026-07-25 operator ruling, and that ruling
|
|
is **realized** — it hosts drill VM `300` (`drill-r50`), so **start there**, not on DooPlex. (The
|
|
historical golden-bake `drill.qcow2` still lives on DooPlex and is a bake fixture, not a drill target.)
|
|
**Which box is safe to break, and what may be done to each:
|
|
`felhom.eu/documentation/runbooks/target-selection.md`** — read it before any destructive test. Both
|
|
nodes + the break-glass recipe: `felhom.eu/documentation/operations/nodes.md`. The agent pins the served leaf cert — verify the
|
|
fingerprint still matches before a live run. Selftest modes (run locally on DooPlex, pointed at the
|
|
demo API): `--selftest[=read|task|hub|storage|backup|restore-test|pbs-verify]`; no flag = the daemon.
|
|
|
|
> **TEMPORARY — felhom-pve is at a remote site (until ~2026-08-02).** The home-LAN literal
|
|
> `192.168.0.162` is NOT reachable from DooPlex for the duration. Access via Tailscale:
|
|
> felhom-pve = 100.70.170.35; the `Host felhom-pve` entry in `~/.ssh/config` on DooPlex already
|
|
> points there (the direct-LAN path stays available as `Host felhom-pve-lan`). Delete this block on
|
|
> return. All documented `ssh felhom-pve` / `pct exec` workflows are unchanged. Path is **direct**
|
|
> (not DERP), ~37 ms rtt per hop. At the remote site the host is on **DHCP**; re-check its address
|
|
> rather than trusting one written here (`ip -br addr show vmbr0` — it read `192.168.0.162/24` on
|
|
> 2026-07-30, and `felhom-pve-lan` from DooPlex is still `No route to host`). Details + findings:
|
|
> `felhom.eu/documentation/audits/AUDIT-vacation-remote-ops-2026-07-20.md`
|
|
>
|
|
> **The "agent does not run at the remote site" warning this block used to carry is RETRACTED
|
|
> (2026-07-30) — it was true before R-50 and is false now.** `localapi` no longer binds a LAN literal:
|
|
> since the R-50 island migration (2026-07-25) it binds `169.254.253.1:8443` on `vmbr9`, which is
|
|
> location-independent by design, and `proxmox.endpoint` is `https://127.0.0.1:8006`. Verified live:
|
|
> `systemctl is-active felhom-agent` → `active`, `felhom-agent --version` → 0.115.0, and the per-guest
|
|
> local API answered `GET /disks` over the island. No config edit and no Viktor GO are outstanding.
|
|
|
|
> **Legacy: Windows workstation.** Until 2026-07-19 CC ran on Windows 11; `pct` commands over SSH
|
|
> needed `export MSYS_NO_PATHCONV=1`, and every remote command used
|
|
> `SSH=/c/Windows/System32/OpenSSH/ssh.exe`. Agent deploy was a two-hop copy via the Windows box
|
|
> (`cygpath -w` for the local scp path; CRLF hazard on config files).
|
|
|
|
## Conventions
|
|
|
|
### Trunk-based — no branches
|
|
|
|
All shippable work commits **directly to `main`**; `main` equals what is deployed.
|
|
- Report-only artifacts (audits, findings, fixspecs) → `felhom.eu/documentation/` (`audits/`, `backlog/`).
|
|
- Risky/supervised fixes are spec'd, then implemented **during the supervised session, on `main`**.
|
|
- Unattended escape hatch: if a fix can't be cleanly verified/shipped, revert + report — never park on a branch.
|
|
|
|
> **In every repository where you make a change, update both files in that repo:**
|
|
> - **`CHANGELOG.md`** — cumulative log, newest on top.
|
|
> - **`REPORT.md`** — **overwrite** with the most recent implementation/validation summary only.
|
|
>
|
|
> **Never write secrets** into any committed file — reference them as "stored out-of-band".
|
|
|
|
- Code quality: verify generated code for bugs/edge cases; add debug logging; **ask rather than
|
|
guess** when you'd otherwise invent input/output.
|
|
- **A health check issues no block I/O** — no `statfs`, no `getdents`, no read, write or `fsync`, **not
|
|
even behind a timeout**. Liveness is decided from `/proc` and kernel state. The full rule + the
|
|
measurement lives in `felhom.eu/CLAUDE.md` "Code quality rules"; it is repeated here because health
|
|
checks are written in THIS repo and that file does not load in an agent-only session. R-117 spike §6.3.
|
|
- Update `REUSE.md` if you added/changed/deprecated a shared helper or pattern (same commit).
|
|
- **Run `python3 scripts/agent_gates.py` from the repo root after ANY change in this repo.** It is
|
|
the ONE entry point for this repo's gates. Today it runs one — `reuse_refs_check` over this
|
|
repo's `REUSE.md` — and it exists at one gate on purpose: a census on 2026-08-02 found that every
|
|
check a `CLAUDE.md` names was passing and two of the four nobody is told to run were failing, and
|
|
this repo was the extreme case, with nothing running against it at all and 90 cited paths checked
|
|
by no one. It grows when the agent grows a second check. `--fast` selects the gates that touch no
|
|
network and no container runtime; today that is all of them. A missing gate is a FAILURE, never a
|
|
skip. **The shared `reuse_refs_check.py` lives in `felhom.eu/scripts/` and is never copied here**
|
|
— a copy would recreate the drift it detects; an absent sibling clone FAILS the gate.
|
|
**The pre-push hook** (`.githooks/pre-push`) runs it with `--fast` and refuses a failing push. It
|
|
is per-clone — switch it on once with `git config core.hooksPath .githooks`, and a manual run
|
|
WARNS when this clone is unarmed. `git push --no-verify` bypasses it deliberately; **say so in the
|
|
session report when you use it.** Both facts are why CI is still owed (`OPEN-ITEMS.md` R-168).
|
|
- Testing doctrine (non-hollow tests, red-proofs, seams): use the `felhom-testing` skill.
|
|
- **Logging**: the slog logger fans out to journald (configured level) + the always-DEBUG `applog.Ring`
|
|
(remote pulls) — English, keys-never-values, durations on outcomes; full rules in
|
|
`felhom.eu/documentation/runbooks/logging-conventions.md`.
|
|
|
|
### Live validation
|
|
|
|
Exercise the SERVER-SIDE PIPELINE a real user triggers, end-to-end. The forbidden shortcut is
|
|
BYPASSING it (the F9 episode: raw guest-attach + hand-set state). Invoking the exact endpoint the UI
|
|
invokes is an acceptable proxy when a browser isn't available — say which method was used. Low-level
|
|
mechanism tests where the direct call IS the mechanism are exempt.
|
|
|
|
## Workflow & artifacts
|
|
|
|
- Implement **`TASK.md` / `TASK-*.md`** specs (when placed as `TASK.md` or told to), then push +
|
|
CHANGELOG + REPORT.md.
|
|
- **`RUNBOOK-*.md`** — an operational procedure. CC executes the steps it has access and capability
|
|
for, including live validation on the demo Proxmox host (CC has root@felhom-pve SSH + the
|
|
felhom-agent token). Mark a step HUMAN only when it genuinely needs physical presence, a real-world
|
|
decision, or credentials CC truly lacks. Judgment still applies: confirm before irreversible ops on
|
|
real customer data — demo scratch guests are fair game.
|