docs+config: operator email enabled (Part A, proven) + TESTRUN findings closed
Part A: hub-config operator_email/operator_enabled (committed earlier); operator + customer email paths proven end-to-end via real Resend sends. Append the "Findings closed" section to the TESTRUN audit (Part A proof, F2-a root cause + deploy fix, F2-b fix + live validation, pending §D supervised reboot) and remove the stale duplicate "Final state" block. REPORT overwritten. hub/CHANGELOG entry. No hub image change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
This commit is contained in:
@@ -2,49 +2,43 @@
|
||||
|
||||
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md).
|
||||
|
||||
## Hub-side agent leaf-fingerprint change detection (HostLeafChecker) — hub v0.22.1
|
||||
## Close the test-run findings — operator email (Part A) + the F2 closures (audit) — 2026-06-30
|
||||
|
||||
**Date:** 2026-06-29. Companion: felhom-agent **v0.48.0** (Part A — reports the served leaf fp). This
|
||||
repo = **Part B (hub)**. The last self-health leg: a proactive, fleet-wide agent-re-key alert.
|
||||
This repo's slice of the "close the TESTRUN findings" task. The agent-side fixes (F2-a, F2-b) ship in
|
||||
`felhom-agent` v0.49.0; here it is **Part A (hub config)** + the audit write-up. No hub image change.
|
||||
|
||||
### Implementation
|
||||
- **`monitor.HostLeafChecker` (NEW):** sibling of `HostCapabilityChecker`. **Trust-on-first-report** —
|
||||
the first leaf fp seen per host is the baseline; a later change emits **`host_leaf_changed`**
|
||||
(operator-only, English; details carry old+new fp) and advances the baseline. First-obs seeds
|
||||
silently (a change needs a prior value → no F2 issue). An **empty** reported fp (pre-v0.48.0 /
|
||||
local-API-disabled) is unknown — never seeds, never alerts, never overwrites a baseline.
|
||||
Customer-blocked hosts dropped; unseen pruned. Same 60s sweep.
|
||||
- **`store.GetHostLeafFingerprints` (NEW):** latest reported fp per host, parsed from `report_json`
|
||||
(mirrors `GetHostCapabilities` — `MAX(id)`, **no schema migration**; chose the report_json read over a
|
||||
denorm column to avoid an `ALTER TABLE`).
|
||||
- **No allowlist change:** `host_leaf_changed` is **hub-generated** (`SaveEvent` + `dispatcher.ProcessEvent`),
|
||||
not controller-pushed, so it bypasses the `/api/v1/event` `allowedEventTypes` gate (same as
|
||||
`host_*`). The generic operator template relays it.
|
||||
- **Known limitation (documented):** trust-on-first-report can't detect a re-key that happened *before*
|
||||
the hub's first report — but the controller channel-check catches the downstream pin mismatch, so
|
||||
this is defense-in-depth, not the sole guard. (The served-fp-vs-pinned-fp authoritative cross-check is
|
||||
a deliberate future enhancement.)
|
||||
### Part A — operator email delivery (the previously-unproven hop) — PROVEN end-to-end
|
||||
The hub dispatcher's operator path (`Dispatcher.processOperator`) sends only when
|
||||
`operatorOn && operatorEmail != ""`; the demo `hub-config` set neither, so the self-health pipeline
|
||||
(probe → report → checker → dispatch) stopped one hop short of the inbox. **Fix = config-only:**
|
||||
`manifests/hub.yaml` `hub-config` ConfigMap now sets `operator_email: admin@felhom.eu` +
|
||||
`operator_enabled: true`. Committed, ArgoCD-synced, hub pod restarted to reload; the mounted config shows
|
||||
both (Resend key NOT printed; it stays injected from `Secret/resend-api`). Live tag stays **v0.22.1**.
|
||||
|
||||
### Tests (green: `go build/vet/test ./...`)
|
||||
Change **red-proof** (A→B → one `host_leaf_changed` + baseline advanced; companion: unchanged → none),
|
||||
first-obs seeds silently, change-back re-alerts, empty fp skipped, customer-blocked dropped. Cross-repo
|
||||
golden mirrors `leaf_fingerprint`.
|
||||
**Proven (both channels):**
|
||||
- **Operator:** scratch-denied the new agent `stalelock-unlock` grant → `agent_capability_degraded` →
|
||||
`[INFO] Operator email sent for demo-felhom/agent_capability_degraded` (the send-success line that
|
||||
never fired while the path was gated off) → restored → `agent_capability_recovered`.
|
||||
- **Customer:** `POST /api/v1/notify` (`event_type:test`) → `{"sent":true}` →
|
||||
`Notification email sent to nagyfenyvesi.viktor@gmail.com for demo-felhom/test`. Same `sendEmail` →
|
||||
Resend the operator path uses.
|
||||
|
||||
### Live validation — PASS (+ a wiring bug the live test caught)
|
||||
- **v0.22.0 shipped the checker but the `cmd/hub/main.go` goroutine wiring never applied** (a concurrent
|
||||
file-touch made me wrongly assume the edit landed). The **live test caught it**: a leaf regen produced
|
||||
no `host_leaf_changed` — only the controller's complementary `agent_channel_pin_mismatch`. Fixed in
|
||||
**v0.22.1** (wired + redeployed). A good example of live validation catching what the build/test
|
||||
couldn't.
|
||||
- After v0.22.1: `Host leaf checker initialized: 1 host fingerprint(s) seeded`; then restoring the leaf
|
||||
(a change from the seeded baseline) →
|
||||
`[WARN] Host leaf: demo-felhom-01 fp 82078fab77ea… → 60b5974d586f… (host_leaf_changed)` (~8 s, one
|
||||
sweep), event saved (no error) + dispatched to the operator. **Independent of** the controller's
|
||||
channel-check, which also fired `agent_channel_pin_mismatch` during the regen — the two are
|
||||
complementary. Final state: leaf == pin `60b5974d…`, channel recovered, capabilities 45/45,
|
||||
controller healthy.
|
||||
### Audit
|
||||
`documentation/audits/TESTRUN-fullstack-2026-06-29.md`: appended the "Findings closed" section (Part A
|
||||
proof + F2-a root cause/fix + F2-b fix/live validation + the pending §D supervised reboot) and removed the
|
||||
stale duplicate "Final state" block (it showed old versions 0.45.0/0.90.0/0.20.0 and contradicted the real
|
||||
parity).
|
||||
|
||||
### Self-health arc — complete
|
||||
Agent watches its own privileged capabilities (v0.44.0) + the controller watches its link to the agent
|
||||
(controller v0.90.0/F2) + the hub proactively watches every agent's leaf fp fleet-wide (this). The
|
||||
original silent-multi-day-outage incident class is now caught from three independent angles.
|
||||
### Companion (felhom-agent v0.49.0, for cross-reference)
|
||||
- **F2-b:** startup stale-lock recovery (`pct unlock` → delete dangling `vzdump` snapshot → start iff
|
||||
onboot, guarded by a no-vzdump-running invariant). Live-validated on `felhom-pve` (recovery on scratch
|
||||
9999; invariant guard proven against a REAL in-flight 9201 backup). New narrow grant + Critical capability.
|
||||
- **F2-a:** the shared-parent boot script never redeployed because `EnsureSharedParent` gated on the unit
|
||||
only; now compares the script too. Root cause confirmed by reproducing the doubling in a `/mnt` scratch
|
||||
dir; corrected script is staged on disk (boot-time-only). The peer-group red-proof is the §D supervised
|
||||
reboot.
|
||||
|
||||
### Parity — known-good
|
||||
agent **v0.49.0** / controller v0.91.0 / hub v0.22.1 (+ operator-email hub-config); leaf==pin `60b5974d…`;
|
||||
caps **46/46**; both drives bound; channel up; apps healthy. Pending the §D supervised reboot: the F2-a
|
||||
host-side doubling (corrected script staged; live mount intentionally untouched). No secrets recorded.
|
||||
|
||||
Reference in New Issue
Block a user