From 8675bff3dacc1f94a623b116859fcc1737837024 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 5 Jul 2026 19:23:05 +0200 Subject: [PATCH] =?UTF-8?q?docs(G1):=20REPORT.md=20=E2=80=94=20break-glass?= =?UTF-8?q?=20agent=20half=20(v0.71.0)=20live-validated?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Auto-heal drill with the agent STOPPED healed /run/sshd in 30.0s (measured); mgmt_plane stanza + hub mgmt_plane_healed warning proven end-to-end. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6 --- REPORT.md | 160 +++++++++++++++++++++--------------------------------- 1 file changed, 63 insertions(+), 97 deletions(-) diff --git a/REPORT.md b/REPORT.md index a40824e..0cfb8cc 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,112 +1,78 @@ -# REPORT — TASK D1: agent self-update (operator-signed, A/B slots, crash-loop auto-rollback) +# REPORT — TASK G1: management-plane break-glass (felhom-agent half) — v0.71.0 -**Date:** 2026-07-05 · **Class:** implementation (code + host artifacts + tests + deploy + live drills) -**Baselines (reconfirmed):** felhom-agent `main` @ `72d7f05` (v0.69.0) → **v0.70.0**; felhom.eu @ `582917d`. -Not behind origin. Design provenance: `felhom.eu/documentation/audits/SPIKE-agent-selfupdate-2026-07-05.md`. -**Shipped + live-validated on felhom-pve (demo host only; Peti's box untouched).** +**Baseline:** felhom-agent `main` @ `1c75a45` (v0.70.0) → **v0.71.0** (commit `fd4e177`). felhom.eu @ +`2f97ce3`. Trunk-based, direct to `main`. -## Commits +## What shipped (this repo) -| Repo | Hash | What | -|---|---|---| -| felhom-agent | `b7cbded` | P1 host artifacts (wrapper, rollback unit, [Unit] limits drop-in, sudoers alias) | -| felhom-agent | `8033a52` | P2 Go plumbing (ClassAgentUpdate, opsign, executor, commit-manager, report field, wiring) | -| felhom-agent | `915642a` | P3 docs (README self-update, REUSE, CHANGELOG v0.70.0, CONTEXT) | -| felhom.eu | `b6bad95` | P3 host-install day-0 install of the artifacts + architecture §11 (implemented) | +The **prevent + auto-heal + observe** thirds of the break-glass system (the hub vault + surfacing are +the felhom.eu half, hub v0.34.1). Closes the lockout from +`felhom.eu/documentation/audits/SPIKE-felhom-sshd-2026-07-05.md` §8: a second sshd's +`RuntimeDirectory=sshd` removed the SHARED `/run/sshd` privsep dir → the stock sshd on :22 reset every +session after `SSH2_MSG_KEXINIT`. -## What shipped +- **Host artifacts** (`configs/`, installed by felhom-host-install): + - `felhom-privsep.tmpfiles` — `d /run/sshd 0755 root root -` (layer 1: boot-persistent, owned by no + unit's lifecycle). + - `felhom-mgmt-watchdog.sh` — the heal action: stat-first recreate `/run/sshd`, `reset-failed` the + stock sshd ONLY when `failed`, write an RFC3339 heal-marker; never restarts the stock sshd, never + touches a healthy dir. shellcheck-clean. + - `felhom-mgmt-watchdog.service` (oneshot) + `.timer` (OnUnitActiveSec=60s, Persistent). **No unit + declares `RuntimeDirectory=`** (the incident cause). +- **Go** (`internal/mgmtplane/`): a read-only `Reporter` (os.Stat `/run/sshd`, read the heal-marker, + TCP-dial sshd:22) → the additive `omitempty` `mgmt_plane` heartbeat stanza + (`privsep_dir_ok, sshd_reachable, healed_recently, privsep_healed_at`). Wired always-on via + `Collector.SetMgmtPlaneReporter` (report.go additive fields, the SelfUpdatePending precedent — no + hub-schema change; the golden contract test uses a nil reporter so the golden stays byte-identical). -- **Trust model:** an operator-signed `agent_update` op (`reconcile.ClassAgentUpdate`, always - Destructive) through the signed-jobs pipeline; params pin version + sha256 → the sha is the ONLY - integrity root (hub = dumb transport, Gitea = dumb storage). `felhom-opsign -op agent_update`. -- **Host artifacts** (`configs/`): `felhom-selfupdate-guarded` (apply/commit/rollback — sha re-verify - as root before `.prev`, path confinement, same-fs assert, atomic mv, pending marker, detached - restart; rollback pending-guarded); `felhom-agent-rollback.service`; `felhom-agent-limits.conf` - ([Unit]-only, `StartLimitIntervalSec=120`+`Burst=4`+`OnFailure=`); `FELHOM_SELFUPDATE` sudoers. -- **Go** (`internal/selfupdate/`): `Executor` (download→verify-vs-signed-sha→wrapper apply; job - completed after verify+download, before apply) + `Manager` (startup dwell→commit; version-mismatch - no-commit; report seam). Wired as the 3rd executor-chain element + a `MaybeCommit` goroutine after - core init. Config `SelfUpdateConfig`; additive report `selfupdate_pending`; 3 capability probes. -- **felhom.eu:** `felhom-host-install.sh` installs wrapper+rollback-unit+drop-in on day-0. +## Tests + red-proofs -## Tests & companion red-proofs +`go build ./... && go vet ./... && go test ./...` on the build server — **all green**. +- `internal/mgmtplane`: healthy (dir ok, no marker → healed_recently=false); missing dir detected + while sshd_reachable stays true (the "TCP up, sessions broken" trap); marker present → healed_recently + + timestamp; empty/whitespace marker treated as absent (red-proof for a hub warning with no + timestamp); sshd-unreachable reported. +- `internal/hub` collector: nil reporter → `mgmt_plane` omitted; wired → carried through. -Full `go build ./... && go vet ./... && go test ./...` green; `shellcheck` clean on the wrapper; -`visudo -cf` on the full sudoers. +## Live validation (felhom-pve) — the acceptance -- **Executor** (Group A): happy-path (staged sha asserted, apply args exact, job completed before - apply); **C2 sha-mismatch refused** + its **companion** (neutering the Go verify → the bad binary - reaches the apply call → `TestExecutor_ShaMismatchRefused` fails; verified, reverted); bad-params; - wrapper-failure surfacing. -- **Gate ride-along** (Group B): `agent_update` rides the REAL authz.Verifier + reconcile.Gate — - pinned-key executes, non-pinned + retarget rejected. Backing companion: - `TestClassify_AgentUpdateAlwaysDestructive` (if it flipped Benign the unsigned op would execute). -- **Commit** (Group C): dwell→commit; version-mismatch → no commit + WARN + marker left; no-pending - no-op; shutdown-before-dwell leaves pending. **Opsign** (Group D): `isHex64`. All packages green. +Deployed **manually** (agent self-update needs an operator-signed op + a pinned signer; the box has +**no signer pinned**, so D1's self-update path was not usable — stated honestly). Build on 180 +(`-X main.version=0.71.0`, sha `b3b616fe…`) → scp via local → `install -m0755` (backup `.bak-0.70.0`) +→ restart. Host artifacts installed + timer enabled; `systemd-analyze verify` + `systemd-tmpfiles +--create` + shellcheck all clean; RuntimeDirectory guard verified (no directive, only the warning +comment). -## Live validation (felhom-pve; the full agent-side pipeline ran for real) +**Auto-heal drill (agent STOPPED — the load-bearing proof):** `systemctl stop felhom-agent` → +`rm -rf /run/sshd` → a new `:22` session reproduced the incident (`Connection reset` right after the +banner) → the agent-independent watchdog **recreated `/run/sshd` and a new `:22` session succeeded +again in 30.0 s (measured), with the agent still `inactive`.** Marker written. This proves the heal is +login-free AND agent-independent. -Build v0.70.0 on 180 (sha `c5eb84cc…`), published to Gitea (anon-pullable, round-trip sha OK), and -manually deployed (the last manual agent deploy). Host artifacts installed; `systemctl show` confirmed -`StartLimitIntervalUSec=2min StartLimitBurst=4`; capabilities 56/56 degraded=0 (the 3 selfupdate -probes pass). A scratch operator key was generated + pinned and a `selfupdate` config block added for -the drill. +**End-to-end visibility chain (hub v0.34.1):** the v0.71.0 report carried +`mgmt_plane:{privsep_dir_ok:true, sshd_reachable:true, healed_recently:true, +privsep_healed_at:"2026-07-05T17:15:32Z"}`; the prior healthy report had `healed_recently:false`; the +old v0.70.0 report had no stanza (correctly absent). The hub monitor raised the +**`mgmt_plane_healed` warning** at 17:16:21 ("privsep dir … was missing and was AUTO-HEALED …"). -**Delivery method note (stated per the live-validation rule):** the hub's *enqueue* endpoint needs a -global operator key CC does not hold, so the operator-signed envelope (produced by the real -`felhom-opsign` over the pinned key) was injected directly into the hub's `signed_jobs` queue — -exactly what `Store.EnqueueSignedJob` does. Everything downstream is the **real, unmodified pipeline**: -the agent polls the hub, the real gate verifies the SSHSIG, the real executor downloads+verifies, the -real wrapper flips, the real commit-manager commits. Only the hub-side enqueue-auth (unit-tested -hub-side) was bypassed. Pickup was triggered by an agent restart (the loop reports immediately on -start → envelope `has_signed_ops` → RunOnce). +**KEXINIT-reset reproduction:** confirmed live in the drill (`Connection reset by 127.0.0.1 port 22` +immediately after the banner exchange, `/run/sshd` absent). -1. **Happy path (Scenario A) — PASS.** Signed `agent_update{0.70.1, sha 908700ab…}` → gate - `allowed=true reason=signed key_id=d1-drill-op` → executor downloaded + verified → wrapper - `applied 0.70.1 (prev 0.70.0)` + detached restart → **0.70.1 running**, `.prev`=0.70.0 (sha - `c5eb84cc`), pending marker written, job COMPLETED on the queue. After the 20s dwell: - `selfupdate: update committed`, marker cleared, `.prev` retained, hub queue empty. -2. **Crash-loop auto-rollback (Scenario B) — PASS (the safety property).** Signed - `agent_update{0.70.2-crash, sha c358d7bc…}` (a throwaway `os.Exit(1)` build, never committed) → - applied → **crash-to-recovered ≈ 2 s**: `15:49:50 apply` → `15:49:52 Main process exited - status=1/FAILURE` → `Triggering OnFailure=` → `rolled back to previous binary and restarted`. - Live binary **byte-identical** to published 0.70.1 (`908700ab…`); pending cleared; hub queue - cleared; **stable, no crash loop** (NRestarts=0, active). (Two near-simultaneous OnFailure fires - raced; both restored the same `.prev` bytes — idempotent, correct.) -3. **No-pending guard (C4) — PASS.** `systemctl start felhom-agent-rollback.service` with no pending - → wrapper `no pending update — no-op`; MainPID **unchanged**, binary **untouched**. -4. **Gate refusal (C1) — PASS.** `agent_update` signed by a NON-pinned key, injected + triggered → - `REJECTED signed op — executor not called … err="authz: signer not in allowed set"`; nothing - downloaded; version unchanged; job cleared. +## Cross-repo state -## Cleanup + final state +- Agent **v0.71.0** live on felhom-pve; three break-glass layers active + enabled (they STAY — this + is the shipped feature). Rollback: `.bak-0.70.0`. +- Hub **v0.34.1** live; break-glass vault + `mgmt_plane_healed` monitor. felhom.eu commits `05d8181` + (hub+installer), `28ac5f5` (runbook), `012e5f3` (monitor fix + v0.34.1). +- **H1 (felhom-sshd) can now assume `/run/sshd` is guaranteed present** (tmpfiles + watchdog) and must + never declare `RuntimeDirectory=` — the installer refuses any unit that does. -- **0.70.2-crash and 0.70.1 deleted from Gitea** (never leave a crashing artifact published); 0.70.0 - retained (the release). -- Box restored: **v0.70.0 active/running, NRestarts=0, capabilities 56/56 degraded=0**; original - `agent.json` restored (scratch operator key + drill selfupdate block removed — no scratch key left - pinned); staging dir empty; `.prev`/nonces cleared. **Host artifacts kept installed** (the shipped - feature): wrapper + rollback unit + [Unit] drop-in (3/3). Hub `signed_jobs` queue empty. -- Drill scratch removed from 180 and the hub pod. `felhom-agent.bak-0.69.0` retained on the box (the - rollback safety net, expected). -- **Self-inflicted incident during cleanup (recorded honestly):** restoring `agent.json` via `mv` of a - root-owned backup left it root-owned → the non-root agent got `permission denied` → 4 crashes → the - start-limit `failed` state. Fixed with `chown felhom-agent:felhom-agent` + `chmod 0600` + - `reset-failed` + start. Not a product defect (a cleanup ownership slip); it did incidentally - demonstrate the OnFailure path no-op'ing on a pending-less crash and the `reset-failed` requirement - ([SF-4]) live. +## Notes / observations -## Observations (not acted on — v1 scope-outs / follow-ups) - -1. **Operator-key provisioning is an operator follow-up** (parallel to the Day-0 vouch): the demo box - was returned to its pre-drill config with NO signer pinned, so self-update is installed-but-dormant - until an operator pins their real operational key + (optionally) a `selfupdate` config block. The - default URL template + anon Gitea pull mean no creds are needed for the download. -2. **Per-crash OnFailure can double-fire** (spike [SF-1], seen live): two rollback invocations raced, - both idempotent. Harmless, but a future hardening could serialize the rollback oneshot - (a flock in the wrapper, or `StartLimitIntervalSec` on the rollback unit) to make it single-shot. -3. **v1 scope-outs (locked, §8):** no hub-floor-driven auto-update, no failed-update auto-retry (the - operator re-signs with a fresh nonce), no pending-timeout auto-rollback (a runs-but-never-commits - binary is caught by the hub's `host_staleness` + the `selfupdate_pending` report flag). -4. **Hub enqueue-auth** was not exercised end-to-end from CC (no global operator key available); it is - covered by the hub-side unit tests (`desired_test.go` — enqueue requires the global key). +- **Secret discipline verified:** the root@pam recovery password appears in NO log/commit (the vault + handler logs username + length only; red-proofed hub-side). +- **Least-privilege console user** (vs root@pam): not probed live (PVE gates the node Shell to + root@pam historically); G1 ships the proven root@pam path — a non-blocking future item. +- **Auto-rotation** of the recovery credential is a noted future item; `--rotate-recovery` re-vaults on + demand. +- Heal cadence: OnUnitActiveSec=60s (measured 30 s this run, mid-cycle); worst case ~60 s.