diff --git a/REPORT.md b/REPORT.md index e644636..3a13318 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,46 +2,49 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md). -## F2 (hub checkers) + prevention (host-install leaf preservation) — hub v0.21.0 + install script +## Hub-side agent leaf-fingerprint change detection (HostLeafChecker) — hub v0.22.1 -**Date:** 2026-06-29. Companions: felhom-controller **v0.91.0** (Part A controller), felhom-agent -**v0.46.0** (Part B.1). This repo carries **Part A (hub checkers, F2)** + **Part B.2 (install script)**. +**Date:** 2026-06-29. Companion: felhom-agent **v0.48.0** (Part A — reports the served leaf fp). This +repo = **Part B (hub)**. The last self-health leg: a proactive, fleet-wide agent-re-key alert. -### Part A — hub F2 (`internal/monitor/host_capability.go`, `host_staleness.go`), hub v0.21.0 -A host already **degraded** (capability) or **stale/down** (staleness) when the hub (re)started was -seeded silently and **never alerted**. Fix: the constructors now seed only HEALTHY hosts; an unhealthy -host is left **unseeded** so the first `Check()` emits once (`emitTransition(…, "unknown", newState, -…)`). The dispatcher's 1 h operator cooldown dedups the re-emit across a hub bounce. Healthy first-obs -still silent. -- Tests: born-degraded **red-proof** (degraded at construction → unseeded → one - `agent_capability_degraded` on first Check, no dup next); the staleness test updated to F2 (born-stale - → unseeded → one `host_stale`). Build/vet/test green. -- **Live:** deployed v0.21.0; startup logs confirm the change — - `Host capability checker initialized: 1 ok, 0 degraded (degraded left unseeded → first Check emits)` - and the staleness sibling likewise. +### Implementation +- **`monitor.HostLeafChecker` (NEW):** sibling of `HostCapabilityChecker`. **Trust-on-first-report** — + the first leaf fp seen per host is the baseline; a later change emits **`host_leaf_changed`** + (operator-only, English; details carry old+new fp) and advances the baseline. First-obs seeds + silently (a change needs a prior value → no F2 issue). An **empty** reported fp (pre-v0.48.0 / + local-API-disabled) is unknown — never seeds, never alerts, never overwrites a baseline. + Customer-blocked hosts dropped; unseen pruned. Same 60s sweep. +- **`store.GetHostLeafFingerprints` (NEW):** latest reported fp per host, parsed from `report_json` + (mirrors `GetHostCapabilities` — `MAX(id)`, **no schema migration**; chose the report_json read over a + denorm column to avoid an `ALTER TABLE`). +- **No allowlist change:** `host_leaf_changed` is **hub-generated** (`SaveEvent` + `dispatcher.ProcessEvent`), + not controller-pushed, so it bypasses the `/api/v1/event` `allowedEventTypes` gate (same as + `host_*`). The generic operator template relays it. +- **Known limitation (documented):** trust-on-first-report can't detect a re-key that happened *before* + the hub's first report — but the controller channel-check catches the downstream pin mismatch, so + this is defense-in-depth, not the sole guard. (The served-fp-vs-pinned-fp authoritative cross-check is + a deliberate future enhancement.) -### Part B.2 — `scripts/felhom-host-install.sh` (prevention) -- **`--preserve-state-from DIR`:** before the first agent start, copy `local-api.{crt,key}` + - `local-tokens.log` into `/var/lib/felhom-agent`, chown `felhom-agent`, modes 644/600/600, and - **validate the leaf parses** (`openssl x509 -noout`, else refuse). Keeps the pinned fingerprint - **stable** across a reinstall — no controller re-bootstrap. Distinct from `--preserve-from` (config - merge only). -- **Populated-host guard:** if the state dir has **no** leaf AND the host already runs guests AND - neither `--preserve-state-from` nor `--allow-new-leaf` was given → **refuse** with a clear message - (converts the silent 2026-06-28 footgun into a hard stop). A plain re-run on an intact host (leaf - present) is unaffected. `bash -n` clean. -- **Live logic validation (isolated, non-destructive — full installer not run):** on felhom-pve with - real openssl/pct — preserve copies with correct modes and the **fingerprint is preserved** across the - copy (`60b5974d…` source == dest); an unparseable leaf is rejected; the guard returns REFUSE for - leaf-absent + guests-present; the live agent state dir was untouched. +### Tests (green: `go build/vet/test ./...`) +Change **red-proof** (A→B → one `host_leaf_changed` + baseline advanced; companion: unchanged → none), +first-obs seeds silently, change-back re-alerts, empty fp skipped, customer-blocked dropped. Cross-repo +golden mirrors `leaf_fingerprint`. -### NOT yet run — Part B live reinstall (B.4) — **awaiting supervised** -A full `felhom-host-install.sh` reinstall on felhom-pve (with `--preserve-state-from` → preserved fp, -channel stays up; without it on the populated host → guard refuses) is the operator's supervised call. -The agent B.1 loud-regenerate and the preserve/guard logic are proven; only the end-to-end reinstall -is deferred. +### Live validation — PASS (+ a wiring bug the live test caught) +- **v0.22.0 shipped the checker but the `cmd/hub/main.go` goroutine wiring never applied** (a concurrent + file-touch made me wrongly assume the edit landed). The **live test caught it**: a leaf regen produced + no `host_leaf_changed` — only the controller's complementary `agent_channel_pin_mismatch`. Fixed in + **v0.22.1** (wired + redeployed). A good example of live validation catching what the build/test + couldn't. +- After v0.22.1: `Host leaf checker initialized: 1 host fingerprint(s) seeded`; then restoring the leaf + (a change from the seeded baseline) → + `[WARN] Host leaf: demo-felhom-01 fp 82078fab77ea… → 60b5974d586f… (host_leaf_changed)` (~8 s, one + sweep), event saved (no error) + dispatched to the operator. **Independent of** the controller's + channel-check, which also fired `agent_channel_pin_mismatch` during the regen — the two are + complementary. Final state: leaf == pin `60b5974d…`, channel recovered, capabilities 45/45, + controller healthy. -### Testrun residual fixed -The full-stack testrun's guest reboot had left guest 9201's `bootstrap.json` a placeholder (redeploy -blocked); restored the real file from the R1 aside copy (customer.id=demo-felhom) → the bootstrap -redeploy path works again (controller 0.91.0 deployed through it). +### Self-health arc — complete +Agent watches its own privileged capabilities (v0.44.0) + the controller watches its link to the agent +(controller v0.90.0/F2) + the hub proactively watches every agent's leaf fp fleet-wide (this). The +original silent-multi-day-outage incident class is now caught from three independent angles. diff --git a/documentation/audits/SPIKE-controller-agent-channel-health-2026-06-29.md b/documentation/audits/SPIKE-controller-agent-channel-health-2026-06-29.md index f30acdf..fb609d5 100644 --- a/documentation/audits/SPIKE-controller-agent-channel-health-2026-06-29.md +++ b/documentation/audits/SPIKE-controller-agent-channel-health-2026-06-29.md @@ -111,6 +111,22 @@ logged `transient down (unreachable, 1/2) — suppressed` then `agent channel DO `allowedEventTypes` allowlist rejected the controller-pushed `agent_channel_*` events (HTTP 400) — fixed in **hub v0.20.0**. See `felhom-controller/REPORT.md` + CHANGELOG v0.90.0. +## Self-health arc COMPLETE (2026-06-29) + +This channel-check is one of three independent angles on the original silent-multi-day-outage incident +class, all now shipped + live-validated: +1. **agent** watches its own privileged capabilities (v0.44.0 + the build-time sudoers gate). +2. **controller** watches its link to the agent — this channel-health check (v0.90.0) + F2 born-down + alerting (v0.91.0). +3. **hub** proactively watches every agent's served local-API **leaf fingerprint** fleet-wide + (agent v0.48.0 reports it; hub v0.22.x `HostLeafChecker` → `host_leaf_changed`), independent of any + controller. Live: a leaf regen raised `host_leaf_changed` (`82078fab…→60b5974d…`) AND the controller + `agent_channel_pin_mismatch` — complementary detection. + +Prevention also shipped: agent v0.46.0 loud-WARNs a regenerated leaf + `felhom-host-install.sh` +`--preserve-state-from`/populated-host guard. The original incident is now both prevented and caught +from three angles. (Remaining future enhancement: the served-fp-vs-pinned-fp authoritative cross-check.) + ## Teardown — confirmed Throwaway `channelspike_test.go` removed (build-server working tree clean — `git status` shows it diff --git a/hub/README.md b/hub/README.md index 85d7c6d..76a95c9 100644 --- a/hub/README.md +++ b/hub/README.md @@ -109,6 +109,13 @@ Events are the primary monitoring mechanism. Each event has: customer_id, event_ **Hub-generated events** (source="hub"): - `node_stale` / `node_down` / `node_recovered` — dead man's switch from staleness checker (every 60s) +- `host_stale` / `host_down` / `host_recovered` — host-domain dead man's switch (agent host reports) +- `agent_capability_degraded` / `agent_capability_recovered` — `HostCapabilityChecker`: an agent's + required `sudo -n` grant went missing (the non-root cutover class) +- `host_leaf_changed` — `HostLeafChecker` (`monitor/host_leaf.go`, every 60s): an agent's reported + local-API **leaf fingerprint** changed (an agent re-key) — proactive, fleet-wide, independent of any + controller's channel-health check. Trust-on-first-report baseline; empty fp = unknown (never alerts). + Reads the fp from the latest host-report (`store.GetHostLeafFingerprints`, no schema migration). - `expected_backup_missed` / `expected_dbdump_missed` — backup deadline checker (daily at 05:00 Budapest) ### Notifications