docs: REPORT + README for v0.47.0 F1 verify hardening (live rollback re-test PASS)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
This commit is contained in:
2026-06-29 22:50:16 +02:00
parent 3844df7c23
commit 61f9b4dcc3
2 changed files with 48 additions and 26 deletions
+41 -26
View File
@@ -1,35 +1,50 @@
# REPORT — leaf lifecycle: signal + loud-log a regenerated leaf, v0.46.0
# REPORT — F1: controller-swap verify hardening, v0.47.0
**Task:** Part B.1 of the F2 + prevention slice — make an accidental local-API leaf **regeneration**
(the 2026-06-28 root→non-root migration class: a state-dir move silently minted a new leaf → every
controller's pin invalidated for days, R1) **visible immediately** instead of silent.
**Task:** close F1 from the no-mercy testrun — a controller image with **no HEALTHCHECK that
crash-loops** could land a single "Running" inspect poll → the swap marked it healthy → **no
rollback** (alpine tagged as the controller passed in ~4 s, then `Restarting (0)`).
**Baseline:** agent `main` @ `1958204` (live 0.45.0) → **v0.46.0** @ `9b0d6c2`, sha `03966da3…`.
**Baseline:** agent `main` @ `9b0d6c2` (live 0.46.0) → **v0.47.0** @ `3844df7`, sha `2d5a3ce4…`.
## Change
## The fix (`internal/localapi/controllerswap.go`)
- **`internal/localapi/cert.go``EnsureLeaf` now returns `generated bool`:** false = an existing
pair was LOADED (stable fingerprint); true = a fresh leaf was GENERATED.
- **`cmd/felhom-agent/main.go` — loud call-site:** a load logs `INFO local-api leaf LOADED`; a
regeneration logs **`WARN local-api leaf REGENERATED — any previously issued bootstrap pins are now
INVALID; controllers will fail the pin check until re-bootstrapped (or restore the prior leaf)`**
(with the new fingerprint). The `--selftest=provision` caller updated to the new signature.
- No new sudo/capability surface — pure return + log change. The companion install-script preservation
(`--preserve-state-from` + populated-host guard) is in `felhom.eu/scripts/felhom-host-install.sh`.
- **`controllerHealthy`** now reads a **4th** `docker inspect -f` field `{{.RestartCount}}`:
`running && RestartCount>0` → not-ok (a process that already crash-restarted isn't stably up,
regardless of healthcheck). It also returns a **`needsDwell`** signal for the no-healthcheck (`none`)
case. It stays a pure point-in-time predicate.
- **`verify`** adds a **stability dwell**: a no-healthcheck image must report ok on `verifyDwell` (=3)
**consecutive** polls before acceptance; a real `healthy` result is trusted immediately (Docker
already gated it). Any not-ok resets the dwell. Timeout → the existing rollback path runs.
- No change to `writeImage`, the sudoers grants, or the state-file/rollback orchestration.
## §3 sudoers — no change (confirmed live)
The grant is `pct exec [0-9]* -- docker inspect -f *`; `*` spans the extended template. Live
`sudo -n -l` of the new `…|{{.RestartCount}}` template → **exit 0** (matches). No new grant.
## Tests (green: `go build/vet/test ./...`)
`TestEnsureLeaf_StableFingerprintAcrossReload`: first call `generated==true`, second `generated==false`
**AND same fingerprint** — persistence keeps the pin stable (a regression that regenerated would flip
gen2 true and change the fp, failing both asserts).
- **F1 red-proof** (`TestF1_Verify_CrashLoopRestartCountBlocks`): `RestartCount>0``verify` false +
`controllerHealthy` returns `(false, starting)`. **Companion:** the same image with `rc=0` + dwell=1
**does** verify — proving the RestartCount check is what blocks the crasher (the old `running + none
→ ok` shape had no guard).
- **Dwell** (`…DwellSingleOkThenCrash`): a single ok then a crash → `verify` false; **companion**
dwell=1 accepts the single ok.
- **Real healthcheck** (`…RealHealthcheckPassesPromptly`): `healthy``verify` true promptly, no false
rollback. Existing `RollbackOnUnhealthy` / `HealthyWithNoHealthcheck` stay green.
## Live validation (felhom-pve) — both paths PASS
- **Quiet load (the normal case):** deployed 0.46.0; the agent restart logged
`INFO local-api leaf LOADED` fp `60b5974d…` (the leaf existed → loaded, not regenerated).
- **Loud regenerate:** during the F2 born-down test, moving the leaf aside + restart produced
`WARN local-api leaf REGENERATED — any previously issued bootstrap pins are now INVALID …` fp
`751367a2…` — the loud signal fires exactly when a new leaf is minted. Restored cleanly (`leaf
LOADED` `60b5974d…` again). capabilities 45/45 ok throughout.
## Live re-test (felhom-pve guest 9201) — the previously-found defect is now FIXED
Re-ran the exact F1 scenario: tagged alpine as `felhom-controller:9.9.9` (no healthcheck, crash-loops),
swapped to it via the real agent `/controller/swap`:
```
swap state: failed error: "new controller did not become healthy within timeout"
controller-swap: rolled back to previous, controller healthy previous=…:0.91.0
running controller: …:0.91.0 Up (healthy)
```
**Before the fix** this image was marked `done` in ~4 s with no rollback; **now** the verify (polling
the `…|{{.RestartCount}}` template) never accepts it → 90 s timeout → **rollback to 0.91.0**. The
rollback's verify of the real 0.91.0 (which has a healthcheck) passed promptly — the real-controller
path is unaffected. Bad tag removed; final state: agent 0.47.0, controller 0.91.0 healthy, capabilities
45/45, channel up.
## NOT changed
The pin/transport, the token store, the sudoers/capability surface, the drive gate. Cert lifecycle
signalling + logging only.
`writeImage`, the `FELHOM_CONTROLLERSWAP` grants, the state-file/rollback orchestration, the
capability/drive-gate/channel logic. Verify predicate + dwell only.