docs: REPORT + README for v0.45.0 controller-swap (stdin tee + FELHOM_CONTROLLERSWAP)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
This commit is contained in:
2026-06-29 19:45:17 +02:00
parent 8a4ccab3e6
commit de6b72651b
2 changed files with 71 additions and 62 deletions
+8
View File
@@ -44,6 +44,14 @@ unnoticed until a user hit them. `internal/capability` makes that loud:
(`HostCapabilityChecker`) alerts the operator on a Critical capability going degraded. Serve-degraded
— the probe never blocks startup. (Next self-health slice: the controller↔agent channel check.)
**Controller-swap under non-root (v0.45.0).** The agent-owned controller image swap
(`internal/localapi/controllerswap.go`) no longer shells out: `writeImage` pipes the image ref on
**stdin** into an in-guest `tee /etc/felhom-controller-image` (via `GuestExecStdin`
`Runner.RunStdin`, the same fenced `sudo -n` runner) — no `bash -c`, no interpolation. Its 5 narrow
grants live in the `FELHOM_CONTROLLERSWAP` sudoers alias (all read-only or fixed-target; the `tee`
target is the FIXED image path, content stdin-fed) and in the capability manifest (Critical), so a
dropped grant is a build failure + a live degraded signal. No general `pct exec` is granted.
## The `storage` package — observe + watchdog (slice 5)
Read-only this slice (no hub desired-state until slice 10):
+63 -62
View File
@@ -1,79 +1,80 @@
# REPORT — felhom-agent v0.44.0: privileged-capability self-probe
# REPORT — felhom-agent v0.45.0: controller-swap under non-root (stdin `tee` + narrow grants)
**Date:** 2026-06-29 · **Class:** Implementation (agent self-health, slice 1) · **Baseline:**
`main` @ `1958204` (v0.43.0) → **v0.44.0** @ `25024d9`. Companion hub: felhom.eu **v0.19.0**.
**Date:** 2026-06-29 · **Class:** Risky/supervised · **Baseline:** `main` @ `8a4ccab` was v0.44.0 →
**v0.45.0** @ `8a4ccab` (this change). Build sha `a393c8bc…`. **No `felhom-controller` change.**
## What & why
A non-root agent that can't run a privileged command it depends on is **degraded** and must say so —
at cutover, not days later. Five casualties of the 2026-06-28 root→non-root migration (leaf/token/
lxc-info/make-private/controller-swap) went undetected until user-visible breakage. This adds a
self-check that catches the class at BUILD time (CI) and RUNTIME (startup + each hub-report cycle),
serve-degraded + loud. (The controller↔agent channel check is a separate later task.)
Restores fleet controller-swap / managed auto-update under the **non-root** agent — the capability the
2026-06-29 sudoers audit deliberately left broken (its only write vector, `bash -c "printf … >"`,
needed arbitrary in-guest execution). Option A, spike-proven GO
(`felhom.eu/documentation/audits/SPIKE-controllerswap-narrow-grants-2026-06-29.md`).
## `internal/capability` — manifest + probe
## The `writeImage` diff (the only orchestration change)
- **Manifest** (`Manifest()`): the required `(binary, representative-arg)` vectors, seeded from the
2026-06-29 sudoers audit (the OK + CLOSED rows). The SURFACED/DEFERRED rows are **excluded**:
general `pct exec * ` (controller-swap, arbitrary exec — an open operator decision), `pct create`
(golden build, no daemon caller), `mount UUID=…` (legacy), `sensors -j` (callerless + uninstalled).
- **Critical** (hub alerts on these): `guest-init-pid` (lxc-info), the intermediary mount surface
(`parent-self-bind`, `parent-make-shared`, `parent-make-private`, `drive-bind`, `drive-umount`),
and the disk format/inspect gate (`disk-blkid`, `disk-lsblk`, `disk-mkfs-ext4`). The rest
(provision/dnsmasq/guesthook/mount-units/smart/lvs) are non-critical (degrade reported, no alert).
- **Probe** (`Prober.Probe`): per entry, `sudo -n -l -- <binary> <reprArgs>` — a sudo policy LIST
(exit 0 ⇔ permitted), run via a **DIRECT** runner so it never executes the command (safe for
mkfs/pct entries) — plus an `os.Stat` existence check. Maps to `ok` / `degraded` ("sudo policy
denied" | "binary not found"). A total sudo failure (drop-in missing) collapses to ONE aggregate
signal. **Never** blocks startup, panics, or errors (serve-degraded).
```
- cmd := fmt.Sprintf("printf '%%s\\n' '%s' > %s", image, controllerImageFile)
- _, err := c.exec.GuestExec(ctx, vmid, "bash", "-c", cmd)
+ _, err := c.exec.GuestExecStdin(ctx, vmid, strings.NewReader(image+"\n"), "tee", controllerImageFile)
```
The image ref is piped on **stdin** into an in-guest `tee` — no shell, no interpolation. `image+"\n"`
is byte-identical to the golden's `printf '%s\n'`; the bootstrap reads `IMAGE=$(cat …)`
(newline-stripping), so the write is consumed identically. `ValidControllerImage` still gates upstream
in `Swap`. Swap orchestration / rollback / state-file logic are **unchanged**.
## Build-time gate (the headline) + red-proof
New stdin seam (routes through the SAME `sudo -n` fenced runner — no hand-rolled exec):
`proxmox.Runner.RunStdin` + `ExecRunner.RunStdin` (Run with `cmd.Stdin`), `GuestBinder.GuestExecStdin`,
`GuestExecutor.GuestExecStdin`. The 5 proxmox.Runner test mocks gained a delegating `RunStdin`.
`manifest_test.go` parses `configs/felhom-agent.sudoers`, glob→regex translates each grant, and
asserts **every manifest vector is covered** — exactly what would have caught the dropped
`lxc-info`/`make-private` lines in CI. **Red-proof** (`TestRedProof_DroppedGrantFailsCheck`): with
the `lxc-info` line removed from an in-memory copy, the check FAILS for `guest-init-pid`, and passes
on the real file. Demonstrated: ran with the line dropped → **FAIL**; restored → **PASS** (not
hollow).
## Sudoers — `FELHOM_CONTROLLERSWAP` (5 narrow grants)
## Runtime wiring + report
```
/usr/sbin/pct exec [0-9]* -- cat /etc/felhom-controller-image # read (fixed)
/usr/sbin/pct exec [0-9]* -- docker image inspect * # read (target present?)
/usr/sbin/pct exec [0-9]* -- docker inspect -f * # read (running/health/image)
/usr/sbin/pct exec [0-9]* -- systemctl restart felhom-controller-bootstrap.service # restart (fixed unit)
/usr/sbin/pct exec [0-9]* -- tee /etc/felhom-controller-image # write (FIXED path, stdin-fed)
```
No general `pct exec`, no `bash -c`. Added to the NOPASSWD line. Capability **manifest** gains the 5
(Critical: a silently-broken fleet auto-update is operator-alert-worthy) — so the v0.44.0 self-probe
watches them and the build-test asserts grant↔code coverage.
`Probe` runs once at startup (`logCapabilities`: INFO `capabilities self-check N/N ok` + an ERROR per
degraded naming the gated feature) and on every collect (`Collector.SetCapabilityProber`); the
snapshot rides the report as the new non-nil `HostReport.Capabilities []capability.Status`. Golden +
contract test updated; the cross-repo hub golden mirrors `capabilities: []`.
## Tests — all green (`go build/vet/test ./...`, 0 failures)
## Tests (all green: `go build/vet/test ./...`)
- `TestManifestCoveredBySudoers` (manifest ⊆ sudoers, 40 vectors) + `TestRedProof_...`.
- Probe: `AllOK`, `OneDenied` (critical degraded, sibling ok, serve-degraded), `BinaryMissing`,
`SudoUnavailableAggregates`, `NilRunnerNoPanic`. Asserted the probe never executes a non-`-l` call.
- New `TestControllerSwap_WriteViaStdinTee_NoShell`: the write is stdin-`tee` with exact `image\n`, the
`tee` target is the fixed path, and **no** shell vector (`bash`/`-c`/`printf`) appears — would FAIL on
the pre-change impl.
- Existing swap tests (happy / rollback-on-unhealthy / image-absent / no-healthcheck / bad-image /
single-flight) pass over the new write path (fake `GuestExecutor` now models `tee` via stdin).
- `RunStdin` forwards stdin; the 5 mock runners implement it.
- **Build-time coverage:** `TestManifestCoveredBySudoers` covers the 5 new vectors. **Two red-proofs:**
`TestRedProof_DroppedGrantFailsCheck` (lxc-info) + new `TestRedProof_DroppedControllerSwapTeeFailsCheck`.
**Demonstrated red→green on the REAL file:** dropping the `tee` grant →
`capability "controllerswap-write" … NOT covered by any sudoers grant` → FAIL; restored → PASS.
## Live validation (felhom-pve, guest 9201) — deployed + verified
- **Deploy:** built `0.44.0` on 180 (sha `675b711e…`), shipped 180→local→felhom-pve (sha verified
identical), backed up the prior binary (`felhom-agent.bak-0.43.0`), `install` + `systemctl restart`.
`systemctl is-active` = **active**.
- **OK-path (live):** startup log `capabilities self-check ok=40 total=40 degraded=0` — N/N ok, a
live proof the probe agrees with the post-audit-complete sudoers. Hub (v0.19.0) logged
`Host capability checker initialized: 1 ok, 0 degraded`.
- **Degraded detection (live, non-intrusive):** on the real host, a granted vector
`sudo -n -l -- /usr/bin/lxc-info -n 9201 -p -H` → exit 0 (ok); a bogus vector → exit 1 (degraded) —
proving the probe's grant discrimination without mutating the freshly-certified live sudoers.
- **Deploy:** built `0.45.0` on 180 (sha `a393c8bc…`), shipped 180→local→felhom-pve (sha verified),
backed up prior (`felhom-agent.bak-0.44.0`), `install` + restart → `is-active` = **active**. Sudoers
redeployed via the install mechanism (`visudo -cf` staged → `install -m 0440 -o root -g root`
re-validate `/etc/sudoers` OK). (Staged file needed a `\r` strip — Windows checkout CRLF — before
visudo; the installed drop-in is LF-clean.)
- **Self-probe (live):** `capabilities self-check ok=45 total=45 degraded=0` — the 5 swap vectors are
granted AND watched.
- **Grant matches (`sudo -n -l`, as felhom-agent):** all 5 → exit 0
(`cat` / `docker image inspect <ref>` / `docker inspect -f '<real template>' felhom-controller` /
`systemctl restart <unit>` / `tee <image file>`).
- **Negative controls → denied (exit 1):** `bash -c 'echo pwned'`, `tee /etc/passwd`,
`docker rm -f felhom-controller`, `rm -rf /etc`. The grants are scoped, not a blanket `pct exec`.
## NOT yet live-validated (deferred — see §16 forward note)
## NOT yet run — live swap (Scenario D) — **awaiting supervised**
- **Full live agent→hub→operator-email degraded transition.** The hub alert fires only on a
**Critical** capability degraded, which would require removing a Critical grant on the live host
(forbidden — would re-break drives) or a separate test host, and would send a real operator email.
It is **fully covered by unit tests** (`hub` `TestHostCapabilityChecker`: one event on ok→degraded,
none steady, recovered, non-critical ignored, old-agent ok). Deferred to a scratch-host exercise.
- **Build-test in CI:** runs in `go test ./...`; no CI runner is wired in this repo yet (the test is
the gate, run on every local/build-server `go test`).
The full end-to-end live swap (controller pre-pulls → swap endpoint → `tee` write → restart → healthy
on target; then a bad image → rollback) **restarts the live controller** and per §13.4 is
operator-supervised. It was NOT run (this session is unattended). When supervised, trigger it via the
controller's real update flow (not a hand-set state) and confirm the tee-write → restart → healthy and
the bad-image rollback over the new write path.
## Forward note (NOT this task)
The manifest is hand-curated; the build-test locks **manifest ⊆ sudoers** but **code → manifest**
drift (a new `runner.Run` call not added to the manifest) is not auto-caught — a future static check
of `runner.Run` call sites against the manifest would close that. The controller↔agent channel check
is the next self-health slice.
## Forward note
No code→manifest drift auto-check yet (a future static check of `runner.Run`/`GuestExec*` call sites
vs the manifest). The controller↔agent channel self-health check remains a separate slice.