From 61f9b4dcc3fc74db3aee9893a53ecb6f4aed00ca Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 29 Jun 2026 22:50:16 +0200 Subject: [PATCH] docs: REPORT + README for v0.47.0 F1 verify hardening (live rollback re-test PASS) Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ --- README.md | 7 ++++++ REPORT.md | 67 ++++++++++++++++++++++++++++++++++--------------------- 2 files changed, 48 insertions(+), 26 deletions(-) diff --git a/README.md b/README.md index dec8c8d..51dcd59 100644 --- a/README.md +++ b/README.md @@ -178,6 +178,13 @@ pre-pulls the target image then calls the agent: write `/etc/felhom-controller-image` → `systemctl restart felhom-controller-bootstrap.service` → poll the new controller to healthy (`docker inspect`, ≤90s) → **roll back** to the previous image if not (the guest is never left without a controller). Strict image-ref gate; single-flight per guest (409). + - **(v0.47.0) Verify hardening (F1):** the health poll reads `{{.RestartCount}}` (a 4th inspect + field) — `running && RestartCount>0` is **not** healthy (a process that already crash-restarted + isn't stably up), and a **no-healthcheck** image must report ok on 3 **consecutive** polls (a + stability dwell) before acceptance; a real `healthy` result is trusted immediately. This closes the + hole where a no-HEALTHCHECK image that crash-loops could land one "Running" instant and false-pass + the point-in-time check → no rollback. Verify predicate only — the rollback orchestration + the + `docker inspect -f *` grant (the `*` spans the extended template) are unchanged. - `GET /controller/swap/status` → `{state, current, previous, target, error}`. The agent is **external** to the controller container, so it survives the controller being killed diff --git a/REPORT.md b/REPORT.md index bdefd2e..4dc0b9c 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,35 +1,50 @@ -# REPORT — leaf lifecycle: signal + loud-log a regenerated leaf, v0.46.0 +# REPORT — F1: controller-swap verify hardening, v0.47.0 -**Task:** Part B.1 of the F2 + prevention slice — make an accidental local-API leaf **regeneration** -(the 2026-06-28 root→non-root migration class: a state-dir move silently minted a new leaf → every -controller's pin invalidated for days, R1) **visible immediately** instead of silent. +**Task:** close F1 from the no-mercy testrun — a controller image with **no HEALTHCHECK that +crash-loops** could land a single "Running" inspect poll → the swap marked it healthy → **no +rollback** (alpine tagged as the controller passed in ~4 s, then `Restarting (0)`). -**Baseline:** agent `main` @ `1958204` (live 0.45.0) → **v0.46.0** @ `9b0d6c2`, sha `03966da3…`. +**Baseline:** agent `main` @ `9b0d6c2` (live 0.46.0) → **v0.47.0** @ `3844df7`, sha `2d5a3ce4…`. -## Change +## The fix (`internal/localapi/controllerswap.go`) -- **`internal/localapi/cert.go` — `EnsureLeaf` now returns `generated bool`:** false = an existing - pair was LOADED (stable fingerprint); true = a fresh leaf was GENERATED. -- **`cmd/felhom-agent/main.go` — loud call-site:** a load logs `INFO local-api leaf LOADED`; a - regeneration logs **`WARN local-api leaf REGENERATED — any previously issued bootstrap pins are now - INVALID; controllers will fail the pin check until re-bootstrapped (or restore the prior leaf)`** - (with the new fingerprint). The `--selftest=provision` caller updated to the new signature. -- No new sudo/capability surface — pure return + log change. The companion install-script preservation - (`--preserve-state-from` + populated-host guard) is in `felhom.eu/scripts/felhom-host-install.sh`. +- **`controllerHealthy`** now reads a **4th** `docker inspect -f` field `{{.RestartCount}}`: + `running && RestartCount>0` → not-ok (a process that already crash-restarted isn't stably up, + regardless of healthcheck). It also returns a **`needsDwell`** signal for the no-healthcheck (`none`) + case. It stays a pure point-in-time predicate. +- **`verify`** adds a **stability dwell**: a no-healthcheck image must report ok on `verifyDwell` (=3) + **consecutive** polls before acceptance; a real `healthy` result is trusted immediately (Docker + already gated it). Any not-ok resets the dwell. Timeout → the existing rollback path runs. +- No change to `writeImage`, the sudoers grants, or the state-file/rollback orchestration. + +## §3 sudoers — no change (confirmed live) +The grant is `pct exec [0-9]* -- docker inspect -f *`; `*` spans the extended template. Live +`sudo -n -l` of the new `…|{{.RestartCount}}` template → **exit 0** (matches). No new grant. ## Tests (green: `go build/vet/test ./...`) -`TestEnsureLeaf_StableFingerprintAcrossReload`: first call `generated==true`, second `generated==false` -**AND same fingerprint** — persistence keeps the pin stable (a regression that regenerated would flip -gen2 true and change the fp, failing both asserts). +- **F1 red-proof** (`TestF1_Verify_CrashLoopRestartCountBlocks`): `RestartCount>0` → `verify` false + + `controllerHealthy` returns `(false, starting)`. **Companion:** the same image with `rc=0` + dwell=1 + **does** verify — proving the RestartCount check is what blocks the crasher (the old `running + none + → ok` shape had no guard). +- **Dwell** (`…DwellSingleOkThenCrash`): a single ok then a crash → `verify` false; **companion** + dwell=1 accepts the single ok. +- **Real healthcheck** (`…RealHealthcheckPassesPromptly`): `healthy` → `verify` true promptly, no false + rollback. Existing `RollbackOnUnhealthy` / `HealthyWithNoHealthcheck` stay green. -## Live validation (felhom-pve) — both paths PASS -- **Quiet load (the normal case):** deployed 0.46.0; the agent restart logged - `INFO local-api leaf LOADED` fp `60b5974d…` (the leaf existed → loaded, not regenerated). -- **Loud regenerate:** during the F2 born-down test, moving the leaf aside + restart produced - `WARN local-api leaf REGENERATED — any previously issued bootstrap pins are now INVALID …` fp - `751367a2…` — the loud signal fires exactly when a new leaf is minted. Restored cleanly (`leaf - LOADED` `60b5974d…` again). capabilities 45/45 ok throughout. +## Live re-test (felhom-pve guest 9201) — the previously-found defect is now FIXED +Re-ran the exact F1 scenario: tagged alpine as `felhom-controller:9.9.9` (no healthcheck, crash-loops), +swapped to it via the real agent `/controller/swap`: +``` +swap state: failed error: "new controller did not become healthy within timeout" +controller-swap: rolled back to previous, controller healthy previous=…:0.91.0 +running controller: …:0.91.0 Up (healthy) +``` +**Before the fix** this image was marked `done` in ~4 s with no rollback; **now** the verify (polling +the `…|{{.RestartCount}}` template) never accepts it → 90 s timeout → **rollback to 0.91.0**. The +rollback's verify of the real 0.91.0 (which has a healthcheck) passed promptly — the real-controller +path is unaffected. Bad tag removed; final state: agent 0.47.0, controller 0.91.0 healthy, capabilities +45/45, channel up. ## NOT changed -The pin/transport, the token store, the sudoers/capability surface, the drive gate. Cert lifecycle -signalling + logging only. +`writeImage`, the `FELHOM_CONTROLLERSWAP` grants, the state-file/rollback orchestration, the +capability/drive-gate/channel logic. Verify predicate + dwell only.