docs: felhom.eu REPORT (hub F2 + install-script preservation) + TESTRUN F2-closed

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ

REPORT.md

@@ -2,43 +2,46 @@
 
 > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md).
 
-## Agent capability-degraded operator alert — hub leg (HostCapabilityChecker), hub v0.19.0
+## F2 (hub checkers) + prevention (host-install leaf preservation) — hub v0.21.0 + install script
 
-**Date:** 2026-06-29 · **Baseline:** hub `main` @ `1958204` (v0.18.0) → **v0.19.0** @ `b7b165b`;
-manifest synced @ `2312572`. Companion: felhom-agent **v0.44.0**.
+**Date:** 2026-06-29. Companions: felhom-controller **v0.91.0** (Part A controller), felhom-agent
+**v0.46.0** (Part B.1). This repo carries **Part A (hub checkers, F2)** + **Part B.2 (install script)**.
 
-### What & why
-felhom-agent v0.44.0 now rides a `capabilities` snapshot on its host report (each required `sudo -n`
-grant: ok/degraded). This is the hub leg: detect the ok→degraded transition and alert the **operator**
-— closing the loop that let the 2026-06-28 non-root-cutover regressions go undetected until breakage.
+### Part A — hub F2 (`internal/monitor/host_capability.go`, `host_staleness.go`), hub v0.21.0
+A host already **degraded** (capability) or **stale/down** (staleness) when the hub (re)started was
+seeded silently and **never alerted**. Fix: the constructors now seed only HEALTHY hosts; an unhealthy
+host is left **unseeded** so the first `Check()` emits once (`emitTransition(…, "unknown", newState,
+…)`). The dispatcher's 1 h operator cooldown dedups the re-emit across a hub bounce. Healthy first-obs
+still silent.
+- Tests: born-degraded **red-proof** (degraded at construction → unseeded → one
+  `agent_capability_degraded` on first Check, no dup next); the staleness test updated to F2 (born-stale
+  → unseeded → one `host_stale`). Build/vet/test green.
+- **Live:** deployed v0.21.0; startup logs confirm the change —
+  `Host capability checker initialized: 1 ok, 0 degraded (degraded left unseeded → first Check emits)`
+  and the staleness sibling likewise.
 
-### Implementation
-- **`monitor.HostCapabilityChecker` (NEW):** a deliberate sibling of `HostStalenessChecker` — same
-  per-host state map, seed-without-event, emit-only-on-transition. A host is `degraded` iff its latest
-  report has any **Critical** capability `status:"degraded"`; non-critical degradations never alert.
-  Runs on the existing 60s sweep alongside the staleness checkers.
-- **Events:** `agent_capability_degraded` (warning, ok→degraded; names the degraded caps + gated
-  features in message + details JSON), `agent_capability_recovered` (info, degraded→ok). Routed via
-  the existing `Dispatcher.ProcessEvent` — **operator-only** (not a customer notification toggle, same
-  mechanism as `host_stale`) with the standard 1 h operator cooldown.
-- **`store.GetHostCapabilities` (NEW):** reads the snapshot from the latest host-report's `report_json`
-  per host, keyed on `MAX(id)` (not `received_at` — within-second ties would return multiple rows).
-  **No schema migration.** A pre-v0.44.0 agent (no `capabilities`) reads as `ok` → no false alert.
-- Cross-repo `host-report.golden.json` mirrors `capabilities: []` (byte-identical with the agent copy).
+### Part B.2 — `scripts/felhom-host-install.sh` (prevention)
+- **`--preserve-state-from DIR`:** before the first agent start, copy `local-api.{crt,key}` +
+  `local-tokens.log` into `/var/lib/felhom-agent`, chown `felhom-agent`, modes 644/600/600, and
+  **validate the leaf parses** (`openssl x509 -noout`, else refuse). Keeps the pinned fingerprint
+  **stable** across a reinstall — no controller re-bootstrap. Distinct from `--preserve-from` (config
+  merge only).
+- **Populated-host guard:** if the state dir has **no** leaf AND the host already runs guests AND
+  neither `--preserve-state-from` nor `--allow-new-leaf` was given → **refuse** with a clear message
+  (converts the silent 2026-06-28 footgun into a hard stop). A plain re-run on an intact host (leaf
+  present) is unaffected. `bash -n` clean.
+- **Live logic validation (isolated, non-destructive — full installer not run):** on felhom-pve with
+  real openssl/pct — preserve copies with correct modes and the **fingerprint is preserved** across the
+  copy (`60b5974d…` source == dest); an unparseable leaf is rejected; the guard returns REFUSE for
+  leaf-absent + guests-present; the live agent state dir was untouched.
 
-### Tests (all green: `go build/vet/test ./...`)
-`TestHostCapabilityChecker` (seed no-event → ok→degraded ONE event → steady degraded none →
-degraded→ok recovered), `…_NonCriticalIgnored`, `…_OldAgentNoCaps`. The golden contract test +
-key-set tests pass with the mirrored field.
+### NOT yet run — Part B live reinstall (B.4) — **awaiting supervised**
+A full `felhom-host-install.sh` reinstall on felhom-pve (with `--preserve-state-from` → preserved fp,
+channel stays up; without it on the populated host → guard refuses) is the operator's supervised call.
+The agent B.1 loud-regenerate and the preserve/guard logic are proven; only the end-to-end reinstall
+is deferred.
 
-### Live validation (k3s felhom-system) — deployed + verified
-- Built + pushed `felhom-hub:0.19.0` on 180; bumped `manifests/hub.yaml`; ArgoCD hard-refresh → sync →
-  rollout. **Synced / Healthy**, running image `…felhom-hub:0.19.0`.
-- Startup: `felhom-hub 0.19.0 starting` + `Host capability checker initialized: 1 ok, 0 degraded` —
-  the checker is live and reading the demo host's snapshot (agent reports 40/40 ok).
-
-### NOT yet live-validated (deferred)
-The full live agent→hub→operator-email degraded transition: the hub alert fires only on a **Critical**
-capability degraded, which on the live host would require removing a Critical grant (forbidden — would
-re-break drives) or a separate test host, plus a real operator email. **Fully covered by the unit
-tests above.** Deferred to a scratch-host exercise.
+### Testrun residual fixed
+The full-stack testrun's guest reboot had left guest 9201's `bootstrap.json` a placeholder (redeploy
+blocked); restored the real file from the R1 aside copy (customer.id=demo-felhom) → the bootstrap
+redeploy path works again (controller 0.91.0 deployed through it).

documentation/audits/TESTRUN-fullstack-2026-06-29.md

@@ -156,6 +156,21 @@ next fresh boot).
 
 ---
 
+## F2 CLOSED (2026-06-29) + prevention shipped
+
+The testrun surfaced **F2**: a channel/host failure present at startup/reseed alerted only on a live
+transition, so a born-down (e.g. a controller booting into pin_mismatch) was dashboard-only, no
+operator email. **Closed** — controller **v0.91.0** (`alerted`-flag rework) + hub **v0.21.0** (checkers
+seed only healthy; born-degraded/stale emits on first Check, cooldown dedups). Live-validated: a
+controller restarted into a regenerated-leaf channel logged `[channel] agent channel DOWN
+(unseeded->down:pin_mismatch)` and **pushed `agent_channel_pin_mismatch`** (~60 s) — the capstone's
+missing half (detection at startup). **Prevention** for the original incident also shipped: agent
+**v0.46.0** loud-WARNs a regenerated leaf (`leaf REGENERATED — pins now INVALID`), and
+`felhom-host-install.sh` gained `--preserve-state-from` (keeps the leaf/pin stable across a reinstall)
++ a populated-host guard (refuse to regenerate the leaf on a host with guests). The §K prevention
+recommendation is now implemented; the full live reinstall is the only piece awaiting a supervised run.
+(F1 swap-rollback `RestartCount` hardening remains backlog.)
+
 ## Final state — known-good
 
 agent 0.45.0 / controller 0.90.0 / hub 0.20.0; leaf fp == pinned fp `60b5974d…`; capabilities 45/45