docs: felhom.eu REPORT (HostLeafChecker) + hub README + self-health arc complete (spike doc)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
This commit is contained in:
@@ -2,46 +2,49 @@
|
||||
|
||||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md).
|
||||
|
||||
## F2 (hub checkers) + prevention (host-install leaf preservation) — hub v0.21.0 + install script
|
||||
## Hub-side agent leaf-fingerprint change detection (HostLeafChecker) — hub v0.22.1
|
||||
|
||||
**Date:** 2026-06-29. Companions: felhom-controller **v0.91.0** (Part A controller), felhom-agent
|
||||
**v0.46.0** (Part B.1). This repo carries **Part A (hub checkers, F2)** + **Part B.2 (install script)**.
|
||||
**Date:** 2026-06-29. Companion: felhom-agent **v0.48.0** (Part A — reports the served leaf fp). This
|
||||
repo = **Part B (hub)**. The last self-health leg: a proactive, fleet-wide agent-re-key alert.
|
||||
|
||||
### Part A — hub F2 (`internal/monitor/host_capability.go`, `host_staleness.go`), hub v0.21.0
|
||||
A host already **degraded** (capability) or **stale/down** (staleness) when the hub (re)started was
|
||||
seeded silently and **never alerted**. Fix: the constructors now seed only HEALTHY hosts; an unhealthy
|
||||
host is left **unseeded** so the first `Check()` emits once (`emitTransition(…, "unknown", newState,
|
||||
…)`). The dispatcher's 1 h operator cooldown dedups the re-emit across a hub bounce. Healthy first-obs
|
||||
still silent.
|
||||
- Tests: born-degraded **red-proof** (degraded at construction → unseeded → one
|
||||
`agent_capability_degraded` on first Check, no dup next); the staleness test updated to F2 (born-stale
|
||||
→ unseeded → one `host_stale`). Build/vet/test green.
|
||||
- **Live:** deployed v0.21.0; startup logs confirm the change —
|
||||
`Host capability checker initialized: 1 ok, 0 degraded (degraded left unseeded → first Check emits)`
|
||||
and the staleness sibling likewise.
|
||||
### Implementation
|
||||
- **`monitor.HostLeafChecker` (NEW):** sibling of `HostCapabilityChecker`. **Trust-on-first-report** —
|
||||
the first leaf fp seen per host is the baseline; a later change emits **`host_leaf_changed`**
|
||||
(operator-only, English; details carry old+new fp) and advances the baseline. First-obs seeds
|
||||
silently (a change needs a prior value → no F2 issue). An **empty** reported fp (pre-v0.48.0 /
|
||||
local-API-disabled) is unknown — never seeds, never alerts, never overwrites a baseline.
|
||||
Customer-blocked hosts dropped; unseen pruned. Same 60s sweep.
|
||||
- **`store.GetHostLeafFingerprints` (NEW):** latest reported fp per host, parsed from `report_json`
|
||||
(mirrors `GetHostCapabilities` — `MAX(id)`, **no schema migration**; chose the report_json read over a
|
||||
denorm column to avoid an `ALTER TABLE`).
|
||||
- **No allowlist change:** `host_leaf_changed` is **hub-generated** (`SaveEvent` + `dispatcher.ProcessEvent`),
|
||||
not controller-pushed, so it bypasses the `/api/v1/event` `allowedEventTypes` gate (same as
|
||||
`host_*`). The generic operator template relays it.
|
||||
- **Known limitation (documented):** trust-on-first-report can't detect a re-key that happened *before*
|
||||
the hub's first report — but the controller channel-check catches the downstream pin mismatch, so
|
||||
this is defense-in-depth, not the sole guard. (The served-fp-vs-pinned-fp authoritative cross-check is
|
||||
a deliberate future enhancement.)
|
||||
|
||||
### Part B.2 — `scripts/felhom-host-install.sh` (prevention)
|
||||
- **`--preserve-state-from DIR`:** before the first agent start, copy `local-api.{crt,key}` +
|
||||
`local-tokens.log` into `/var/lib/felhom-agent`, chown `felhom-agent`, modes 644/600/600, and
|
||||
**validate the leaf parses** (`openssl x509 -noout`, else refuse). Keeps the pinned fingerprint
|
||||
**stable** across a reinstall — no controller re-bootstrap. Distinct from `--preserve-from` (config
|
||||
merge only).
|
||||
- **Populated-host guard:** if the state dir has **no** leaf AND the host already runs guests AND
|
||||
neither `--preserve-state-from` nor `--allow-new-leaf` was given → **refuse** with a clear message
|
||||
(converts the silent 2026-06-28 footgun into a hard stop). A plain re-run on an intact host (leaf
|
||||
present) is unaffected. `bash -n` clean.
|
||||
- **Live logic validation (isolated, non-destructive — full installer not run):** on felhom-pve with
|
||||
real openssl/pct — preserve copies with correct modes and the **fingerprint is preserved** across the
|
||||
copy (`60b5974d…` source == dest); an unparseable leaf is rejected; the guard returns REFUSE for
|
||||
leaf-absent + guests-present; the live agent state dir was untouched.
|
||||
### Tests (green: `go build/vet/test ./...`)
|
||||
Change **red-proof** (A→B → one `host_leaf_changed` + baseline advanced; companion: unchanged → none),
|
||||
first-obs seeds silently, change-back re-alerts, empty fp skipped, customer-blocked dropped. Cross-repo
|
||||
golden mirrors `leaf_fingerprint`.
|
||||
|
||||
### NOT yet run — Part B live reinstall (B.4) — **awaiting supervised**
|
||||
A full `felhom-host-install.sh` reinstall on felhom-pve (with `--preserve-state-from` → preserved fp,
|
||||
channel stays up; without it on the populated host → guard refuses) is the operator's supervised call.
|
||||
The agent B.1 loud-regenerate and the preserve/guard logic are proven; only the end-to-end reinstall
|
||||
is deferred.
|
||||
### Live validation — PASS (+ a wiring bug the live test caught)
|
||||
- **v0.22.0 shipped the checker but the `cmd/hub/main.go` goroutine wiring never applied** (a concurrent
|
||||
file-touch made me wrongly assume the edit landed). The **live test caught it**: a leaf regen produced
|
||||
no `host_leaf_changed` — only the controller's complementary `agent_channel_pin_mismatch`. Fixed in
|
||||
**v0.22.1** (wired + redeployed). A good example of live validation catching what the build/test
|
||||
couldn't.
|
||||
- After v0.22.1: `Host leaf checker initialized: 1 host fingerprint(s) seeded`; then restoring the leaf
|
||||
(a change from the seeded baseline) →
|
||||
`[WARN] Host leaf: demo-felhom-01 fp 82078fab77ea… → 60b5974d586f… (host_leaf_changed)` (~8 s, one
|
||||
sweep), event saved (no error) + dispatched to the operator. **Independent of** the controller's
|
||||
channel-check, which also fired `agent_channel_pin_mismatch` during the regen — the two are
|
||||
complementary. Final state: leaf == pin `60b5974d…`, channel recovered, capabilities 45/45,
|
||||
controller healthy.
|
||||
|
||||
### Testrun residual fixed
|
||||
The full-stack testrun's guest reboot had left guest 9201's `bootstrap.json` a placeholder (redeploy
|
||||
blocked); restored the real file from the R1 aside copy (customer.id=demo-felhom) → the bootstrap
|
||||
redeploy path works again (controller 0.91.0 deployed through it).
|
||||
### Self-health arc — complete
|
||||
Agent watches its own privileged capabilities (v0.44.0) + the controller watches its link to the agent
|
||||
(controller v0.90.0/F2) + the hub proactively watches every agent's leaf fp fleet-wide (this). The
|
||||
original silent-multi-day-outage incident class is now caught from three independent angles.
|
||||
|
||||
@@ -111,6 +111,22 @@ logged `transient down (unreachable, 1/2) — suppressed` then `agent channel DO
|
||||
`allowedEventTypes` allowlist rejected the controller-pushed `agent_channel_*` events (HTTP 400) —
|
||||
fixed in **hub v0.20.0**. See `felhom-controller/REPORT.md` + CHANGELOG v0.90.0.
|
||||
|
||||
## Self-health arc COMPLETE (2026-06-29)
|
||||
|
||||
This channel-check is one of three independent angles on the original silent-multi-day-outage incident
|
||||
class, all now shipped + live-validated:
|
||||
1. **agent** watches its own privileged capabilities (v0.44.0 + the build-time sudoers gate).
|
||||
2. **controller** watches its link to the agent — this channel-health check (v0.90.0) + F2 born-down
|
||||
alerting (v0.91.0).
|
||||
3. **hub** proactively watches every agent's served local-API **leaf fingerprint** fleet-wide
|
||||
(agent v0.48.0 reports it; hub v0.22.x `HostLeafChecker` → `host_leaf_changed`), independent of any
|
||||
controller. Live: a leaf regen raised `host_leaf_changed` (`82078fab…→60b5974d…`) AND the controller
|
||||
`agent_channel_pin_mismatch` — complementary detection.
|
||||
|
||||
Prevention also shipped: agent v0.46.0 loud-WARNs a regenerated leaf + `felhom-host-install.sh`
|
||||
`--preserve-state-from`/populated-host guard. The original incident is now both prevented and caught
|
||||
from three angles. (Remaining future enhancement: the served-fp-vs-pinned-fp authoritative cross-check.)
|
||||
|
||||
## Teardown — confirmed
|
||||
|
||||
Throwaway `channelspike_test.go` removed (build-server working tree clean — `git status` shows it
|
||||
|
||||
@@ -109,6 +109,13 @@ Events are the primary monitoring mechanism. Each event has: customer_id, event_
|
||||
|
||||
**Hub-generated events** (source="hub"):
|
||||
- `node_stale` / `node_down` / `node_recovered` — dead man's switch from staleness checker (every 60s)
|
||||
- `host_stale` / `host_down` / `host_recovered` — host-domain dead man's switch (agent host reports)
|
||||
- `agent_capability_degraded` / `agent_capability_recovered` — `HostCapabilityChecker`: an agent's
|
||||
required `sudo -n` grant went missing (the non-root cutover class)
|
||||
- `host_leaf_changed` — `HostLeafChecker` (`monitor/host_leaf.go`, every 60s): an agent's reported
|
||||
local-API **leaf fingerprint** changed (an agent re-key) — proactive, fleet-wide, independent of any
|
||||
controller's channel-health check. Trust-on-first-report baseline; empty fp = unknown (never alerts).
|
||||
Reads the fp from the latest host-report (`store.GetHostLeafFingerprints`, no schema migration).
|
||||
- `expected_backup_missed` / `expected_dbdump_missed` — backup deadline checker (daily at 05:00 Budapest)
|
||||
|
||||
### Notifications
|
||||
|
||||
Reference in New Issue
Block a user