docs+config: operator email enabled (Part A, proven) + TESTRUN findings closed

Part A: hub-config operator_email/operator_enabled (committed earlier);
operator + customer email paths proven end-to-end via real Resend sends.
Append the "Findings closed" section to the TESTRUN audit (Part A proof,
F2-a root cause + deploy fix, F2-b fix + live validation, pending §D
supervised reboot) and remove the stale duplicate "Final state" block.
REPORT overwritten. hub/CHANGELOG entry. No hub image change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
This commit is contained in:
2026-06-30 09:21:17 +02:00
parent 83ed127d53
commit 06727e3228
3 changed files with 110 additions and 46 deletions
+35 -41
View File
@@ -2,49 +2,43 @@
> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md).
## Hub-side agent leaf-fingerprint change detection (HostLeafChecker) — hub v0.22.1
## Close the test-run findings — operator email (Part A) + the F2 closures (audit) — 2026-06-30
**Date:** 2026-06-29. Companion: felhom-agent **v0.48.0** (Part A — reports the served leaf fp). This
repo = **Part B (hub)**. The last self-health leg: a proactive, fleet-wide agent-re-key alert.
This repo's slice of the "close the TESTRUN findings" task. The agent-side fixes (F2-a, F2-b) ship in
`felhom-agent` v0.49.0; here it is **Part A (hub config)** + the audit write-up. No hub image change.
### Implementation
- **`monitor.HostLeafChecker` (NEW):** sibling of `HostCapabilityChecker`. **Trust-on-first-report**
the first leaf fp seen per host is the baseline; a later change emits **`host_leaf_changed`**
(operator-only, English; details carry old+new fp) and advances the baseline. First-obs seeds
silently (a change needs a prior value → no F2 issue). An **empty** reported fp (pre-v0.48.0 /
local-API-disabled) is unknown — never seeds, never alerts, never overwrites a baseline.
Customer-blocked hosts dropped; unseen pruned. Same 60s sweep.
- **`store.GetHostLeafFingerprints` (NEW):** latest reported fp per host, parsed from `report_json`
(mirrors `GetHostCapabilities``MAX(id)`, **no schema migration**; chose the report_json read over a
denorm column to avoid an `ALTER TABLE`).
- **No allowlist change:** `host_leaf_changed` is **hub-generated** (`SaveEvent` + `dispatcher.ProcessEvent`),
not controller-pushed, so it bypasses the `/api/v1/event` `allowedEventTypes` gate (same as
`host_*`). The generic operator template relays it.
- **Known limitation (documented):** trust-on-first-report can't detect a re-key that happened *before*
the hub's first report — but the controller channel-check catches the downstream pin mismatch, so
this is defense-in-depth, not the sole guard. (The served-fp-vs-pinned-fp authoritative cross-check is
a deliberate future enhancement.)
### Part A — operator email delivery (the previously-unproven hop) — PROVEN end-to-end
The hub dispatcher's operator path (`Dispatcher.processOperator`) sends only when
`operatorOn && operatorEmail != ""`; the demo `hub-config` set neither, so the self-health pipeline
(probe → report → checker → dispatch) stopped one hop short of the inbox. **Fix = config-only:**
`manifests/hub.yaml` `hub-config` ConfigMap now sets `operator_email: admin@felhom.eu` +
`operator_enabled: true`. Committed, ArgoCD-synced, hub pod restarted to reload; the mounted config shows
both (Resend key NOT printed; it stays injected from `Secret/resend-api`). Live tag stays **v0.22.1**.
### Tests (green: `go build/vet/test ./...`)
Change **red-proof** (A→B → one `host_leaf_changed` + baseline advanced; companion: unchanged → none),
first-obs seeds silently, change-back re-alerts, empty fp skipped, customer-blocked dropped. Cross-repo
golden mirrors `leaf_fingerprint`.
**Proven (both channels):**
- **Operator:** scratch-denied the new agent `stalelock-unlock` grant → `agent_capability_degraded`
`[INFO] Operator email sent for demo-felhom/agent_capability_degraded` (the send-success line that
never fired while the path was gated off) → restored → `agent_capability_recovered`.
- **Customer:** `POST /api/v1/notify` (`event_type:test`) → `{"sent":true}`
`Notification email sent to nagyfenyvesi.viktor@gmail.com for demo-felhom/test`. Same `sendEmail`
Resend the operator path uses.
### Live validation — PASS (+ a wiring bug the live test caught)
- **v0.22.0 shipped the checker but the `cmd/hub/main.go` goroutine wiring never applied** (a concurrent
file-touch made me wrongly assume the edit landed). The **live test caught it**: a leaf regen produced
no `host_leaf_changed` — only the controller's complementary `agent_channel_pin_mismatch`. Fixed in
**v0.22.1** (wired + redeployed). A good example of live validation catching what the build/test
couldn't.
- After v0.22.1: `Host leaf checker initialized: 1 host fingerprint(s) seeded`; then restoring the leaf
(a change from the seeded baseline) →
`[WARN] Host leaf: demo-felhom-01 fp 82078fab77ea… → 60b5974d586f… (host_leaf_changed)` (~8 s, one
sweep), event saved (no error) + dispatched to the operator. **Independent of** the controller's
channel-check, which also fired `agent_channel_pin_mismatch` during the regen — the two are
complementary. Final state: leaf == pin `60b5974d…`, channel recovered, capabilities 45/45,
controller healthy.
### Audit
`documentation/audits/TESTRUN-fullstack-2026-06-29.md`: appended the "Findings closed" section (Part A
proof + F2-a root cause/fix + F2-b fix/live validation + the pending §D supervised reboot) and removed the
stale duplicate "Final state" block (it showed old versions 0.45.0/0.90.0/0.20.0 and contradicted the real
parity).
### Self-health arc — complete
Agent watches its own privileged capabilities (v0.44.0) + the controller watches its link to the agent
(controller v0.90.0/F2) + the hub proactively watches every agent's leaf fp fleet-wide (this). The
original silent-multi-day-outage incident class is now caught from three independent angles.
### Companion (felhom-agent v0.49.0, for cross-reference)
- **F2-b:** startup stale-lock recovery (`pct unlock` → delete dangling `vzdump` snapshot → start iff
onboot, guarded by a no-vzdump-running invariant). Live-validated on `felhom-pve` (recovery on scratch
9999; invariant guard proven against a REAL in-flight 9201 backup). New narrow grant + Critical capability.
- **F2-a:** the shared-parent boot script never redeployed because `EnsureSharedParent` gated on the unit
only; now compares the script too. Root cause confirmed by reproducing the doubling in a `/mnt` scratch
dir; corrected script is staged on disk (boot-time-only). The peer-group red-proof is the §D supervised
reboot.
### Parity — known-good
agent **v0.49.0** / controller v0.91.0 / hub v0.22.1 (+ operator-email hub-config); leaf==pin `60b5974d…`;
caps **46/46**; both drives bound; channel up; apps healthy. Pending the §D supervised reboot: the F2-a
host-side doubling (corrected script staged; live mount intentionally untouched). No secrets recorded.