docs: felhom.eu REPORT (HostLeafChecker) + hub README + self-health arc complete (spike doc)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ

REPORT.md

@@ -2,46 +2,49 @@
 
 > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md).
 
-## F2 (hub checkers) + prevention (host-install leaf preservation) — hub v0.21.0 + install script
+## Hub-side agent leaf-fingerprint change detection (HostLeafChecker) — hub v0.22.1
 
-**Date:** 2026-06-29. Companions: felhom-controller **v0.91.0** (Part A controller), felhom-agent
+**Date:** 2026-06-29. Companion: felhom-agent **v0.48.0** (Part A — reports the served leaf fp). This
-**v0.46.0** (Part B.1). This repo carries **Part A (hub checkers, F2)** + **Part B.2 (install script)**.
+repo = **Part B (hub)**. The last self-health leg: a proactive, fleet-wide agent-re-key alert.
 
-### Part A — hub F2 (`internal/monitor/host_capability.go`, `host_staleness.go`), hub v0.21.0
+### Implementation
-A host already **degraded** (capability) or **stale/down** (staleness) when the hub (re)started was
+- **`monitor.HostLeafChecker` (NEW):** sibling of `HostCapabilityChecker`. **Trust-on-first-report** —
-seeded silently and **never alerted**. Fix: the constructors now seed only HEALTHY hosts; an unhealthy
+  the first leaf fp seen per host is the baseline; a later change emits **`host_leaf_changed`**
-host is left **unseeded** so the first `Check()` emits once (`emitTransition(…, "unknown", newState,
+  (operator-only, English; details carry old+new fp) and advances the baseline. First-obs seeds
-…)`). The dispatcher's 1 h operator cooldown dedups the re-emit across a hub bounce. Healthy first-obs
+  silently (a change needs a prior value → no F2 issue). An **empty** reported fp (pre-v0.48.0 /
-still silent.
+  local-API-disabled) is unknown — never seeds, never alerts, never overwrites a baseline.
-- Tests: born-degraded **red-proof** (degraded at construction → unseeded → one
+  Customer-blocked hosts dropped; unseen pruned. Same 60s sweep.
-  `agent_capability_degraded` on first Check, no dup next); the staleness test updated to F2 (born-stale
+- **`store.GetHostLeafFingerprints` (NEW):** latest reported fp per host, parsed from `report_json`
-  → unseeded → one `host_stale`). Build/vet/test green.
+  (mirrors `GetHostCapabilities` — `MAX(id)`, **no schema migration**; chose the report_json read over a
-- **Live:** deployed v0.21.0; startup logs confirm the change —
+  denorm column to avoid an `ALTER TABLE`).
-  `Host capability checker initialized: 1 ok, 0 degraded (degraded left unseeded → first Check emits)`
+- **No allowlist change:** `host_leaf_changed` is **hub-generated** (`SaveEvent` + `dispatcher.ProcessEvent`),
-  and the staleness sibling likewise.
+  not controller-pushed, so it bypasses the `/api/v1/event` `allowedEventTypes` gate (same as
+  `host_*`). The generic operator template relays it.
+- **Known limitation (documented):** trust-on-first-report can't detect a re-key that happened *before*
+  the hub's first report — but the controller channel-check catches the downstream pin mismatch, so
+  this is defense-in-depth, not the sole guard. (The served-fp-vs-pinned-fp authoritative cross-check is
+  a deliberate future enhancement.)
 
-### Part B.2 — `scripts/felhom-host-install.sh` (prevention)
+### Tests (green: `go build/vet/test ./...`)
-- **`--preserve-state-from DIR`:** before the first agent start, copy `local-api.{crt,key}` +
+Change **red-proof** (A→B → one `host_leaf_changed` + baseline advanced; companion: unchanged → none),
-  `local-tokens.log` into `/var/lib/felhom-agent`, chown `felhom-agent`, modes 644/600/600, and
+first-obs seeds silently, change-back re-alerts, empty fp skipped, customer-blocked dropped. Cross-repo
-  **validate the leaf parses** (`openssl x509 -noout`, else refuse). Keeps the pinned fingerprint
+golden mirrors `leaf_fingerprint`.
-  **stable** across a reinstall — no controller re-bootstrap. Distinct from `--preserve-from` (config
-  merge only).
-- **Populated-host guard:** if the state dir has **no** leaf AND the host already runs guests AND
-  neither `--preserve-state-from` nor `--allow-new-leaf` was given → **refuse** with a clear message
-  (converts the silent 2026-06-28 footgun into a hard stop). A plain re-run on an intact host (leaf
-  present) is unaffected. `bash -n` clean.
-- **Live logic validation (isolated, non-destructive — full installer not run):** on felhom-pve with
-  real openssl/pct — preserve copies with correct modes and the **fingerprint is preserved** across the
-  copy (`60b5974d…` source == dest); an unparseable leaf is rejected; the guard returns REFUSE for
-  leaf-absent + guests-present; the live agent state dir was untouched.
 
-### NOT yet run — Part B live reinstall (B.4) — **awaiting supervised**
+### Live validation — PASS (+ a wiring bug the live test caught)
-A full `felhom-host-install.sh` reinstall on felhom-pve (with `--preserve-state-from` → preserved fp,
+- **v0.22.0 shipped the checker but the `cmd/hub/main.go` goroutine wiring never applied** (a concurrent
-channel stays up; without it on the populated host → guard refuses) is the operator's supervised call.
+  file-touch made me wrongly assume the edit landed). The **live test caught it**: a leaf regen produced
-The agent B.1 loud-regenerate and the preserve/guard logic are proven; only the end-to-end reinstall
+  no `host_leaf_changed` — only the controller's complementary `agent_channel_pin_mismatch`. Fixed in
-is deferred.
+  **v0.22.1** (wired + redeployed). A good example of live validation catching what the build/test
+  couldn't.
+- After v0.22.1: `Host leaf checker initialized: 1 host fingerprint(s) seeded`; then restoring the leaf
+  (a change from the seeded baseline) →
+  `[WARN] Host leaf: demo-felhom-01 fp 82078fab77ea… → 60b5974d586f… (host_leaf_changed)` (~8 s, one
+  sweep), event saved (no error) + dispatched to the operator. **Independent of** the controller's
+  channel-check, which also fired `agent_channel_pin_mismatch` during the regen — the two are
+  complementary. Final state: leaf == pin `60b5974d…`, channel recovered, capabilities 45/45,
+  controller healthy.
 
-### Testrun residual fixed
+### Self-health arc — complete
-The full-stack testrun's guest reboot had left guest 9201's `bootstrap.json` a placeholder (redeploy
+Agent watches its own privileged capabilities (v0.44.0) + the controller watches its link to the agent
-blocked); restored the real file from the R1 aside copy (customer.id=demo-felhom) → the bootstrap
+(controller v0.90.0/F2) + the hub proactively watches every agent's leaf fp fleet-wide (this). The
-redeploy path works again (controller 0.91.0 deployed through it).
+original silent-multi-day-outage incident class is now caught from three independent angles.

documentation/audits/SPIKE-controller-agent-channel-health-2026-06-29.md

@@ -111,6 +111,22 @@ logged `transient down (unreachable, 1/2) — suppressed` then `agent channel DO
 `allowedEventTypes` allowlist rejected the controller-pushed `agent_channel_*` events (HTTP 400) —
 fixed in **hub v0.20.0**. See `felhom-controller/REPORT.md` + CHANGELOG v0.90.0.
 
+## Self-health arc COMPLETE (2026-06-29)
+
+This channel-check is one of three independent angles on the original silent-multi-day-outage incident
+class, all now shipped + live-validated:
+1. **agent** watches its own privileged capabilities (v0.44.0 + the build-time sudoers gate).
+2. **controller** watches its link to the agent — this channel-health check (v0.90.0) + F2 born-down
+   alerting (v0.91.0).
+3. **hub** proactively watches every agent's served local-API **leaf fingerprint** fleet-wide
+   (agent v0.48.0 reports it; hub v0.22.x `HostLeafChecker` → `host_leaf_changed`), independent of any
+   controller. Live: a leaf regen raised `host_leaf_changed` (`82078fab…→60b5974d…`) AND the controller
+   `agent_channel_pin_mismatch` — complementary detection.
+
+Prevention also shipped: agent v0.46.0 loud-WARNs a regenerated leaf + `felhom-host-install.sh`
+`--preserve-state-from`/populated-host guard. The original incident is now both prevented and caught
+from three angles. (Remaining future enhancement: the served-fp-vs-pinned-fp authoritative cross-check.)
+
 ## Teardown — confirmed
 
 Throwaway `channelspike_test.go` removed (build-server working tree clean — `git status` shows it

hub/README.md

@@ -109,6 +109,13 @@ Events are the primary monitoring mechanism. Each event has: customer_id, event_
 
 **Hub-generated events** (source="hub"):
 - `node_stale` / `node_down` / `node_recovered` — dead man's switch from staleness checker (every 60s)
+- `host_stale` / `host_down` / `host_recovered` — host-domain dead man's switch (agent host reports)
+- `agent_capability_degraded` / `agent_capability_recovered` — `HostCapabilityChecker`: an agent's
+  required `sudo -n` grant went missing (the non-root cutover class)
+- `host_leaf_changed` — `HostLeafChecker` (`monitor/host_leaf.go`, every 60s): an agent's reported
+  local-API **leaf fingerprint** changed (an agent re-key) — proactive, fleet-wide, independent of any
+  controller's channel-health check. Trust-on-first-report baseline; empty fp = unknown (never alerts).
+  Reads the fp from the latest host-report (`store.GetHostLeafFingerprints`, no schema migration).
 - `expected_backup_missed` / `expected_dbdump_missed` — backup deadline checker (daily at 05:00 Budapest)
 
 ### Notifications