docs: REPORT + README for v0.45.0 controller-swap (stdin tee + FELHOM_CONTROLLERSWAP)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
This commit is contained in:
@@ -44,6 +44,14 @@ unnoticed until a user hit them. `internal/capability` makes that loud:
|
||||
(`HostCapabilityChecker`) alerts the operator on a Critical capability going degraded. Serve-degraded
|
||||
— the probe never blocks startup. (Next self-health slice: the controller↔agent channel check.)
|
||||
|
||||
**Controller-swap under non-root (v0.45.0).** The agent-owned controller image swap
|
||||
(`internal/localapi/controllerswap.go`) no longer shells out: `writeImage` pipes the image ref on
|
||||
**stdin** into an in-guest `tee /etc/felhom-controller-image` (via `GuestExecStdin` →
|
||||
`Runner.RunStdin`, the same fenced `sudo -n` runner) — no `bash -c`, no interpolation. Its 5 narrow
|
||||
grants live in the `FELHOM_CONTROLLERSWAP` sudoers alias (all read-only or fixed-target; the `tee`
|
||||
target is the FIXED image path, content stdin-fed) and in the capability manifest (Critical), so a
|
||||
dropped grant is a build failure + a live degraded signal. No general `pct exec` is granted.
|
||||
|
||||
## The `storage` package — observe + watchdog (slice 5)
|
||||
|
||||
Read-only this slice (no hub desired-state until slice 10):
|
||||
|
||||
@@ -1,79 +1,80 @@
|
||||
# REPORT — felhom-agent v0.44.0: privileged-capability self-probe
|
||||
# REPORT — felhom-agent v0.45.0: controller-swap under non-root (stdin `tee` + narrow grants)
|
||||
|
||||
**Date:** 2026-06-29 · **Class:** Implementation (agent self-health, slice 1) · **Baseline:**
|
||||
`main` @ `1958204` (v0.43.0) → **v0.44.0** @ `25024d9`. Companion hub: felhom.eu **v0.19.0**.
|
||||
**Date:** 2026-06-29 · **Class:** Risky/supervised · **Baseline:** `main` @ `8a4ccab` was v0.44.0 →
|
||||
**v0.45.0** @ `8a4ccab` (this change). Build sha `a393c8bc…`. **No `felhom-controller` change.**
|
||||
|
||||
## What & why
|
||||
|
||||
A non-root agent that can't run a privileged command it depends on is **degraded** and must say so —
|
||||
at cutover, not days later. Five casualties of the 2026-06-28 root→non-root migration (leaf/token/
|
||||
lxc-info/make-private/controller-swap) went undetected until user-visible breakage. This adds a
|
||||
self-check that catches the class at BUILD time (CI) and RUNTIME (startup + each hub-report cycle),
|
||||
serve-degraded + loud. (The controller↔agent channel check is a separate later task.)
|
||||
Restores fleet controller-swap / managed auto-update under the **non-root** agent — the capability the
|
||||
2026-06-29 sudoers audit deliberately left broken (its only write vector, `bash -c "printf … >"`,
|
||||
needed arbitrary in-guest execution). Option A, spike-proven GO
|
||||
(`felhom.eu/documentation/audits/SPIKE-controllerswap-narrow-grants-2026-06-29.md`).
|
||||
|
||||
## `internal/capability` — manifest + probe
|
||||
## The `writeImage` diff (the only orchestration change)
|
||||
|
||||
- **Manifest** (`Manifest()`): the required `(binary, representative-arg)` vectors, seeded from the
|
||||
2026-06-29 sudoers audit (the OK + CLOSED rows). The SURFACED/DEFERRED rows are **excluded**:
|
||||
general `pct exec * ` (controller-swap, arbitrary exec — an open operator decision), `pct create`
|
||||
(golden build, no daemon caller), `mount UUID=…` (legacy), `sensors -j` (callerless + uninstalled).
|
||||
- **Critical** (hub alerts on these): `guest-init-pid` (lxc-info), the intermediary mount surface
|
||||
(`parent-self-bind`, `parent-make-shared`, `parent-make-private`, `drive-bind`, `drive-umount`),
|
||||
and the disk format/inspect gate (`disk-blkid`, `disk-lsblk`, `disk-mkfs-ext4`). The rest
|
||||
(provision/dnsmasq/guesthook/mount-units/smart/lvs) are non-critical (degrade reported, no alert).
|
||||
- **Probe** (`Prober.Probe`): per entry, `sudo -n -l -- <binary> <reprArgs>` — a sudo policy LIST
|
||||
(exit 0 ⇔ permitted), run via a **DIRECT** runner so it never executes the command (safe for
|
||||
mkfs/pct entries) — plus an `os.Stat` existence check. Maps to `ok` / `degraded` ("sudo policy
|
||||
denied" | "binary not found"). A total sudo failure (drop-in missing) collapses to ONE aggregate
|
||||
signal. **Never** blocks startup, panics, or errors (serve-degraded).
|
||||
```
|
||||
- cmd := fmt.Sprintf("printf '%%s\\n' '%s' > %s", image, controllerImageFile)
|
||||
- _, err := c.exec.GuestExec(ctx, vmid, "bash", "-c", cmd)
|
||||
+ _, err := c.exec.GuestExecStdin(ctx, vmid, strings.NewReader(image+"\n"), "tee", controllerImageFile)
|
||||
```
|
||||
The image ref is piped on **stdin** into an in-guest `tee` — no shell, no interpolation. `image+"\n"`
|
||||
is byte-identical to the golden's `printf '%s\n'`; the bootstrap reads `IMAGE=$(cat …)`
|
||||
(newline-stripping), so the write is consumed identically. `ValidControllerImage` still gates upstream
|
||||
in `Swap`. Swap orchestration / rollback / state-file logic are **unchanged**.
|
||||
|
||||
## Build-time gate (the headline) + red-proof
|
||||
New stdin seam (routes through the SAME `sudo -n` fenced runner — no hand-rolled exec):
|
||||
`proxmox.Runner.RunStdin` + `ExecRunner.RunStdin` (Run with `cmd.Stdin`), `GuestBinder.GuestExecStdin`,
|
||||
`GuestExecutor.GuestExecStdin`. The 5 proxmox.Runner test mocks gained a delegating `RunStdin`.
|
||||
|
||||
`manifest_test.go` parses `configs/felhom-agent.sudoers`, glob→regex translates each grant, and
|
||||
asserts **every manifest vector is covered** — exactly what would have caught the dropped
|
||||
`lxc-info`/`make-private` lines in CI. **Red-proof** (`TestRedProof_DroppedGrantFailsCheck`): with
|
||||
the `lxc-info` line removed from an in-memory copy, the check FAILS for `guest-init-pid`, and passes
|
||||
on the real file. Demonstrated: ran with the line dropped → **FAIL**; restored → **PASS** (not
|
||||
hollow).
|
||||
## Sudoers — `FELHOM_CONTROLLERSWAP` (5 narrow grants)
|
||||
|
||||
## Runtime wiring + report
|
||||
```
|
||||
/usr/sbin/pct exec [0-9]* -- cat /etc/felhom-controller-image # read (fixed)
|
||||
/usr/sbin/pct exec [0-9]* -- docker image inspect * # read (target present?)
|
||||
/usr/sbin/pct exec [0-9]* -- docker inspect -f * # read (running/health/image)
|
||||
/usr/sbin/pct exec [0-9]* -- systemctl restart felhom-controller-bootstrap.service # restart (fixed unit)
|
||||
/usr/sbin/pct exec [0-9]* -- tee /etc/felhom-controller-image # write (FIXED path, stdin-fed)
|
||||
```
|
||||
No general `pct exec`, no `bash -c`. Added to the NOPASSWD line. Capability **manifest** gains the 5
|
||||
(Critical: a silently-broken fleet auto-update is operator-alert-worthy) — so the v0.44.0 self-probe
|
||||
watches them and the build-test asserts grant↔code coverage.
|
||||
|
||||
`Probe` runs once at startup (`logCapabilities`: INFO `capabilities self-check N/N ok` + an ERROR per
|
||||
degraded naming the gated feature) and on every collect (`Collector.SetCapabilityProber`); the
|
||||
snapshot rides the report as the new non-nil `HostReport.Capabilities []capability.Status`. Golden +
|
||||
contract test updated; the cross-repo hub golden mirrors `capabilities: []`.
|
||||
## Tests — all green (`go build/vet/test ./...`, 0 failures)
|
||||
|
||||
## Tests (all green: `go build/vet/test ./...`)
|
||||
|
||||
- `TestManifestCoveredBySudoers` (manifest ⊆ sudoers, 40 vectors) + `TestRedProof_...`.
|
||||
- Probe: `AllOK`, `OneDenied` (critical degraded, sibling ok, serve-degraded), `BinaryMissing`,
|
||||
`SudoUnavailableAggregates`, `NilRunnerNoPanic`. Asserted the probe never executes a non-`-l` call.
|
||||
- New `TestControllerSwap_WriteViaStdinTee_NoShell`: the write is stdin-`tee` with exact `image\n`, the
|
||||
`tee` target is the fixed path, and **no** shell vector (`bash`/`-c`/`printf`) appears — would FAIL on
|
||||
the pre-change impl.
|
||||
- Existing swap tests (happy / rollback-on-unhealthy / image-absent / no-healthcheck / bad-image /
|
||||
single-flight) pass over the new write path (fake `GuestExecutor` now models `tee` via stdin).
|
||||
- `RunStdin` forwards stdin; the 5 mock runners implement it.
|
||||
- **Build-time coverage:** `TestManifestCoveredBySudoers` covers the 5 new vectors. **Two red-proofs:**
|
||||
`TestRedProof_DroppedGrantFailsCheck` (lxc-info) + new `TestRedProof_DroppedControllerSwapTeeFailsCheck`.
|
||||
**Demonstrated red→green on the REAL file:** dropping the `tee` grant →
|
||||
`capability "controllerswap-write" … NOT covered by any sudoers grant` → FAIL; restored → PASS.
|
||||
|
||||
## Live validation (felhom-pve, guest 9201) — deployed + verified
|
||||
|
||||
- **Deploy:** built `0.44.0` on 180 (sha `675b711e…`), shipped 180→local→felhom-pve (sha verified
|
||||
identical), backed up the prior binary (`felhom-agent.bak-0.43.0`), `install` + `systemctl restart`.
|
||||
`systemctl is-active` = **active**.
|
||||
- **OK-path (live):** startup log `capabilities self-check ok=40 total=40 degraded=0` — N/N ok, a
|
||||
live proof the probe agrees with the post-audit-complete sudoers. Hub (v0.19.0) logged
|
||||
`Host capability checker initialized: 1 ok, 0 degraded`.
|
||||
- **Degraded detection (live, non-intrusive):** on the real host, a granted vector
|
||||
`sudo -n -l -- /usr/bin/lxc-info -n 9201 -p -H` → exit 0 (ok); a bogus vector → exit 1 (degraded) —
|
||||
proving the probe's grant discrimination without mutating the freshly-certified live sudoers.
|
||||
- **Deploy:** built `0.45.0` on 180 (sha `a393c8bc…`), shipped 180→local→felhom-pve (sha verified),
|
||||
backed up prior (`felhom-agent.bak-0.44.0`), `install` + restart → `is-active` = **active**. Sudoers
|
||||
redeployed via the install mechanism (`visudo -cf` staged → `install -m 0440 -o root -g root` →
|
||||
re-validate `/etc/sudoers` OK). (Staged file needed a `\r` strip — Windows checkout CRLF — before
|
||||
visudo; the installed drop-in is LF-clean.)
|
||||
- **Self-probe (live):** `capabilities self-check ok=45 total=45 degraded=0` — the 5 swap vectors are
|
||||
granted AND watched.
|
||||
- **Grant matches (`sudo -n -l`, as felhom-agent):** all 5 → exit 0
|
||||
(`cat` / `docker image inspect <ref>` / `docker inspect -f '<real template>' felhom-controller` /
|
||||
`systemctl restart <unit>` / `tee <image file>`).
|
||||
- **Negative controls → denied (exit 1):** `bash -c 'echo pwned'`, `tee /etc/passwd`,
|
||||
`docker rm -f felhom-controller`, `rm -rf /etc`. The grants are scoped, not a blanket `pct exec`.
|
||||
|
||||
## NOT yet live-validated (deferred — see §16 forward note)
|
||||
## NOT yet run — live swap (Scenario D) — **awaiting supervised**
|
||||
|
||||
- **Full live agent→hub→operator-email degraded transition.** The hub alert fires only on a
|
||||
**Critical** capability degraded, which would require removing a Critical grant on the live host
|
||||
(forbidden — would re-break drives) or a separate test host, and would send a real operator email.
|
||||
It is **fully covered by unit tests** (`hub` `TestHostCapabilityChecker`: one event on ok→degraded,
|
||||
none steady, recovered, non-critical ignored, old-agent ok). Deferred to a scratch-host exercise.
|
||||
- **Build-test in CI:** runs in `go test ./...`; no CI runner is wired in this repo yet (the test is
|
||||
the gate, run on every local/build-server `go test`).
|
||||
The full end-to-end live swap (controller pre-pulls → swap endpoint → `tee` write → restart → healthy
|
||||
on target; then a bad image → rollback) **restarts the live controller** and per §13.4 is
|
||||
operator-supervised. It was NOT run (this session is unattended). When supervised, trigger it via the
|
||||
controller's real update flow (not a hand-set state) and confirm the tee-write → restart → healthy and
|
||||
the bad-image rollback over the new write path.
|
||||
|
||||
## Forward note (NOT this task)
|
||||
|
||||
The manifest is hand-curated; the build-test locks **manifest ⊆ sudoers** but **code → manifest**
|
||||
drift (a new `runner.Run` call not added to the manifest) is not auto-caught — a future static check
|
||||
of `runner.Run` call sites against the manifest would close that. The controller↔agent channel check
|
||||
is the next self-health slice.
|
||||
## Forward note
|
||||
No code→manifest drift auto-check yet (a future static check of `runner.Run`/`GuestExec*` call sites
|
||||
vs the manifest). The controller↔agent channel self-health check remains a separate slice.
|
||||
|
||||
Reference in New Issue
Block a user