diff --git a/REPORT.md b/REPORT.md index 3a13318..fdb7570 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,49 +2,43 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md). -## Hub-side agent leaf-fingerprint change detection (HostLeafChecker) — hub v0.22.1 +## Close the test-run findings — operator email (Part A) + the F2 closures (audit) — 2026-06-30 -**Date:** 2026-06-29. Companion: felhom-agent **v0.48.0** (Part A — reports the served leaf fp). This -repo = **Part B (hub)**. The last self-health leg: a proactive, fleet-wide agent-re-key alert. +This repo's slice of the "close the TESTRUN findings" task. The agent-side fixes (F2-a, F2-b) ship in +`felhom-agent` v0.49.0; here it is **Part A (hub config)** + the audit write-up. No hub image change. -### Implementation -- **`monitor.HostLeafChecker` (NEW):** sibling of `HostCapabilityChecker`. **Trust-on-first-report** — - the first leaf fp seen per host is the baseline; a later change emits **`host_leaf_changed`** - (operator-only, English; details carry old+new fp) and advances the baseline. First-obs seeds - silently (a change needs a prior value → no F2 issue). An **empty** reported fp (pre-v0.48.0 / - local-API-disabled) is unknown — never seeds, never alerts, never overwrites a baseline. - Customer-blocked hosts dropped; unseen pruned. Same 60s sweep. -- **`store.GetHostLeafFingerprints` (NEW):** latest reported fp per host, parsed from `report_json` - (mirrors `GetHostCapabilities` — `MAX(id)`, **no schema migration**; chose the report_json read over a - denorm column to avoid an `ALTER TABLE`). -- **No allowlist change:** `host_leaf_changed` is **hub-generated** (`SaveEvent` + `dispatcher.ProcessEvent`), - not controller-pushed, so it bypasses the `/api/v1/event` `allowedEventTypes` gate (same as - `host_*`). The generic operator template relays it. -- **Known limitation (documented):** trust-on-first-report can't detect a re-key that happened *before* - the hub's first report — but the controller channel-check catches the downstream pin mismatch, so - this is defense-in-depth, not the sole guard. (The served-fp-vs-pinned-fp authoritative cross-check is - a deliberate future enhancement.) +### Part A — operator email delivery (the previously-unproven hop) — PROVEN end-to-end +The hub dispatcher's operator path (`Dispatcher.processOperator`) sends only when +`operatorOn && operatorEmail != ""`; the demo `hub-config` set neither, so the self-health pipeline +(probe → report → checker → dispatch) stopped one hop short of the inbox. **Fix = config-only:** +`manifests/hub.yaml` `hub-config` ConfigMap now sets `operator_email: admin@felhom.eu` + +`operator_enabled: true`. Committed, ArgoCD-synced, hub pod restarted to reload; the mounted config shows +both (Resend key NOT printed; it stays injected from `Secret/resend-api`). Live tag stays **v0.22.1**. -### Tests (green: `go build/vet/test ./...`) -Change **red-proof** (A→B → one `host_leaf_changed` + baseline advanced; companion: unchanged → none), -first-obs seeds silently, change-back re-alerts, empty fp skipped, customer-blocked dropped. Cross-repo -golden mirrors `leaf_fingerprint`. +**Proven (both channels):** +- **Operator:** scratch-denied the new agent `stalelock-unlock` grant → `agent_capability_degraded` → + `[INFO] Operator email sent for demo-felhom/agent_capability_degraded` (the send-success line that + never fired while the path was gated off) → restored → `agent_capability_recovered`. +- **Customer:** `POST /api/v1/notify` (`event_type:test`) → `{"sent":true}` → + `Notification email sent to nagyfenyvesi.viktor@gmail.com for demo-felhom/test`. Same `sendEmail` → + Resend the operator path uses. -### Live validation — PASS (+ a wiring bug the live test caught) -- **v0.22.0 shipped the checker but the `cmd/hub/main.go` goroutine wiring never applied** (a concurrent - file-touch made me wrongly assume the edit landed). The **live test caught it**: a leaf regen produced - no `host_leaf_changed` — only the controller's complementary `agent_channel_pin_mismatch`. Fixed in - **v0.22.1** (wired + redeployed). A good example of live validation catching what the build/test - couldn't. -- After v0.22.1: `Host leaf checker initialized: 1 host fingerprint(s) seeded`; then restoring the leaf - (a change from the seeded baseline) → - `[WARN] Host leaf: demo-felhom-01 fp 82078fab77ea… → 60b5974d586f… (host_leaf_changed)` (~8 s, one - sweep), event saved (no error) + dispatched to the operator. **Independent of** the controller's - channel-check, which also fired `agent_channel_pin_mismatch` during the regen — the two are - complementary. Final state: leaf == pin `60b5974d…`, channel recovered, capabilities 45/45, - controller healthy. +### Audit +`documentation/audits/TESTRUN-fullstack-2026-06-29.md`: appended the "Findings closed" section (Part A +proof + F2-a root cause/fix + F2-b fix/live validation + the pending §D supervised reboot) and removed the +stale duplicate "Final state" block (it showed old versions 0.45.0/0.90.0/0.20.0 and contradicted the real +parity). -### Self-health arc — complete -Agent watches its own privileged capabilities (v0.44.0) + the controller watches its link to the agent -(controller v0.90.0/F2) + the hub proactively watches every agent's leaf fp fleet-wide (this). The -original silent-multi-day-outage incident class is now caught from three independent angles. +### Companion (felhom-agent v0.49.0, for cross-reference) +- **F2-b:** startup stale-lock recovery (`pct unlock` → delete dangling `vzdump` snapshot → start iff + onboot, guarded by a no-vzdump-running invariant). Live-validated on `felhom-pve` (recovery on scratch + 9999; invariant guard proven against a REAL in-flight 9201 backup). New narrow grant + Critical capability. +- **F2-a:** the shared-parent boot script never redeployed because `EnsureSharedParent` gated on the unit + only; now compares the script too. Root cause confirmed by reproducing the doubling in a `/mnt` scratch + dir; corrected script is staged on disk (boot-time-only). The peer-group red-proof is the §D supervised + reboot. + +### Parity — known-good +agent **v0.49.0** / controller v0.91.0 / hub v0.22.1 (+ operator-email hub-config); leaf==pin `60b5974d…`; +caps **46/46**; both drives bound; channel up; apps healthy. Pending the §D supervised reboot: the F2-a +host-side doubling (corrected script staged; live mount intentionally untouched). No secrets recorded. diff --git a/documentation/audits/TESTRUN-fullstack-2026-06-29.md b/documentation/audits/TESTRUN-fullstack-2026-06-29.md index 16dd43b..7237a09 100644 --- a/documentation/audits/TESTRUN-fullstack-2026-06-29.md +++ b/documentation/audits/TESTRUN-fullstack-2026-06-29.md @@ -246,9 +246,65 @@ agent v0.48.0 / controller v0.91.0 / hub v0.22.1; leaf==pin `60b5974d…`; caps `bound_under_parent=True`; channel up; nextcloud + controller healthy. (Persistent-but-harmless: the §4 host-side doubling — finding F2-a.) -## Final state — known-good +## Findings closed — operator email + F2-a + F2-b (2026-06-30, agent v0.49.0 + hub config) -agent 0.45.0 / controller 0.90.0 / hub 0.20.0; leaf fp == pinned fp `60b5974d…`; capabilities 45/45 -ok; both USB drives `bound_under_parent=True`, apps healthy; channel up, dashboard clean. All test -artifacts removed (`/root/testrun-leaf-backup`, the `9.9.9` tag). No secrets recorded (the per-guest -token was read out-of-band and never printed; leaf fingerprints are not secret). +The two Phase-2 findings (F2-a, F2-b) and the Phase-1 operator-email gap are now **fixed**. Diagnose- +before-fix on both risky parts; both root causes confirmed live on `felhom-pve` before any code change. + +### Part A — operator email delivery (the previously-unproven hop) — **PROVEN end-to-end** +The Phase-1 "Note (not a defect)" gap is closed. Root cause was `hub-config` lacking +`operator_email`/`operator_enabled` (so `Dispatcher.processOperator` returned before the send). Fix = +config-only (`manifests/hub.yaml` ConfigMap: `operator_email: admin@felhom.eu` + `operator_enabled: true`), +hub pod restarted to reload; the mounted config now shows both (Resend key NOT printed). +- **Operator path:** scratch-denied the new `stalelock-unlock` grant → agent `degraded=1` → + `[INFO] Host capability: demo-felhom-01 ok → degraded (agent_capability_degraded)` → + **`[INFO] Operator email sent for demo-felhom/agent_capability_degraded`** (this line prints ONLY after + Resend returns 2xx — it never appeared before, because the path was gated off). Restored the grant → + `agent_capability_recovered`. +- **Customer path:** `POST /api/v1/notify` (`event_type:test`) → `{"sent":true}` → + `[INFO] Notification email sent to nagyfenyvesi.viktor@gmail.com for demo-felhom/test` (same `sendEmail` + → Resend). Both channels of the dispatcher now proven to the Resend send. + +### Part B — reboot-during-backup stale-lock recovery (F2-b) — **FIXED + live-validated** +`agent v0.49.0`: at startup, `Server.RecoverStaleLockedGuests` clears a stale vzdump lock — `pct unlock` +→ delete the dangling `vzdump` snapshot → start iff `onboot` and not already running — **only when no +vzdump is genuinely in-flight** (the invariant; fail-safe otherwise). New narrow grant `FELHOM_STALELOCK += pct unlock [0-9]*` + Critical capability `stalelock-unlock` (caps 45→46, all green). Two real issues +surfaced by live validation and fixed: PVE 9.x rejects `?running=1` (→ `?source=active`); the LXC-start +`WARNINGS: 1` nesting advisory false-failed the start (→ `AllowWarnings`). +- **Live (B.3):** induced a `snapshot-delete` lock on scratch guest 9999 (onboot:1, stopped) → agent + restart → `stale-lock: clearing a stale backup lock` → `started CT after clearing the stale lock + (onboot)` → lock CLEARED, guest running. **Invariant guard proven against a REAL backup:** a genuine + agent-scheduled vzdump of 9201 was in-flight during the same restart → `stale-lock: a vzdump backup is + genuinely in-flight — leaving the lock (NOT stale)` → 9201's lock untouched; the real backup completed + and self-cleared. (Note: PVE reserves the snapshot name `vzdump`, so a dangling one can't be hand-made + — the delsnapshot-against-a-real-snapshot path is unit-tested and live-fires in the §D real reboot.) + +### Part C — host-side doubling (F2-a) — **root cause FOUND; deploy fix shipped; red-proof pending §D** +Reproduced the exact pathology in a scratch dir at `/mnt` (where the live parent lives): the **stale** +sequence (`bind + make-shared`, no `make-private`) leaves the self-bind in root's `shared:1` → sub-bind +count **2** (doubling); the **correct** sequence (`bind → make-private → make-shared`) gives an own group +(`shared:525`) → count **1**. The live boot script was the **pre-v0.36.6** body (no `make-private`). +**Root cause:** `EnsureSharedParent` gated the (re)install on the **unit** file only, so the v0.36.6 +*script-only* fix never deployed to a host whose unit was current. **Fix (`agent v0.49.0`):** the new +`sharedParentInstallStale` compares **both** the script and the unit. Boot-time-only — it rewrites the +on-disk script; it does **not** churn the live mount (still guarded on `!isHostMountpoint`). +- **Deploy fix proven:** on the v0.49.0 startup the corrected script (with `make-private`) was installed + to `/usr/local/sbin/felhom-shared-parent.sh` (sudo log shows the install). The live mount is left + untouched (`/mnt/felhom-drives` still `shared:1`, doubled — by design, until the next boot). +- **PENDING (§D, supervised):** the red-proof requires a reboot — the parent must come up as its **own** + peer group (≠ `shared:1`) AND the guest must still see **both** drives (`bound_under_parent=true`). Held + for a supervised window per the task's STOP rule (no unsupervised reboot / live mount-propagation change). + +### §D — supervised host reboot — PENDING (not run unsupervised) +Validates, in one window: **B** (start a vzdump of 9201 → reboot mid-backup → agent auto-recovers the +locked CT, the real delsnapshot path) and **C** (parent own-group + guest sees both drives). Per the task's +strict rule, NOT run autonomously. + +### Final parity (post-fix) — known-good +agent **v0.49.0** / controller v0.91.0 / hub v0.22.1 (+ operator-email hub-config); leaf==pin `60b5974d…`; +caps **46/46**; both drives bound + apps healthy; channel up; scratch guest 9999 back to baseline; the +scratch deny removed + `/etc/sudoers` valid; 9201 backup completed + lock self-cleared. **Persistent until +the §D reboot:** the F2-a host-side doubling (the corrected boot script is staged; live mount intentionally +untouched). No secrets recorded (tokens read out-of-band, never printed; leaf fps + the operator email are +not secret). diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 246328a..7895dcb 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,5 +1,19 @@ # Felhom Hub — Changelog +## hub-config — enable operator email alerts (config-only, no image change) (2026-06-30) + +`manifests/hub.yaml` `hub-config` ConfigMap: set `notifications.operator_email: admin@felhom.eu` + +`operator_enabled: true`. The dispatcher's operator path (`Dispatcher.processOperator`) sends only when +`operatorOn && operatorEmail != ""`; without these it returned early, so the self-health pipeline +(probe → report → checker → dispatch) stopped one hop short of the inbox (the TESTRUN's unproven hop). +**No hub image change** (live tag stays v0.22.1) — ConfigMap edit + pod restart to reload. +- **Proven end-to-end (2026-06-30):** a real capability-degrade alert produced + `[INFO] Operator email sent for demo-felhom/agent_capability_degraded` (the send-success line that + never fired while the path was gated); the customer path (`POST /api/v1/notify` `event_type:test`) sent + to the customer address via the same `sendEmail` → Resend. See + `documentation/audits/TESTRUN-fullstack-2026-06-29.md` ("Findings closed", Part A). Operator email is not + a secret; the Resend key stays injected from `Secret/resend-api`. + ## v0.22.1 — wire HostLeafChecker into the monitor loop (v0.22.0 missed the wiring) (2026-06-29) The v0.22.0 commit added `HostLeafChecker` but the `cmd/hub/main.go` goroutine edit never applied, so