docs+config: operator email enabled (Part A, proven) + TESTRUN findings closed

Part A: hub-config operator_email/operator_enabled (committed earlier);
operator + customer email paths proven end-to-end via real Resend sends.
Append the "Findings closed" section to the TESTRUN audit (Part A proof,
F2-a root cause + deploy fix, F2-b fix + live validation, pending §D
supervised reboot) and remove the stale duplicate "Final state" block.
REPORT overwritten. hub/CHANGELOG entry. No hub image change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K

REPORT.md

@@ -2,49 +2,43 @@
 
 > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md).
 
-## Hub-side agent leaf-fingerprint change detection (HostLeafChecker) — hub v0.22.1
+## Close the test-run findings — operator email (Part A) + the F2 closures (audit) — 2026-06-30
 
-**Date:** 2026-06-29. Companion: felhom-agent **v0.48.0** (Part A — reports the served leaf fp). This
-repo = **Part B (hub)**. The last self-health leg: a proactive, fleet-wide agent-re-key alert.
+This repo's slice of the "close the TESTRUN findings" task. The agent-side fixes (F2-a, F2-b) ship in
+`felhom-agent` v0.49.0; here it is **Part A (hub config)** + the audit write-up. No hub image change.
 
-### Implementation
-- **`monitor.HostLeafChecker` (NEW):** sibling of `HostCapabilityChecker`. **Trust-on-first-report** —
-  the first leaf fp seen per host is the baseline; a later change emits **`host_leaf_changed`**
-  (operator-only, English; details carry old+new fp) and advances the baseline. First-obs seeds
-  silently (a change needs a prior value → no F2 issue). An **empty** reported fp (pre-v0.48.0 /
-  local-API-disabled) is unknown — never seeds, never alerts, never overwrites a baseline.
-  Customer-blocked hosts dropped; unseen pruned. Same 60s sweep.
-- **`store.GetHostLeafFingerprints` (NEW):** latest reported fp per host, parsed from `report_json`
-  (mirrors `GetHostCapabilities` — `MAX(id)`, **no schema migration**; chose the report_json read over a
-  denorm column to avoid an `ALTER TABLE`).
-- **No allowlist change:** `host_leaf_changed` is **hub-generated** (`SaveEvent` + `dispatcher.ProcessEvent`),
-  not controller-pushed, so it bypasses the `/api/v1/event` `allowedEventTypes` gate (same as
-  `host_*`). The generic operator template relays it.
-- **Known limitation (documented):** trust-on-first-report can't detect a re-key that happened *before*
-  the hub's first report — but the controller channel-check catches the downstream pin mismatch, so
-  this is defense-in-depth, not the sole guard. (The served-fp-vs-pinned-fp authoritative cross-check is
-  a deliberate future enhancement.)
+### Part A — operator email delivery (the previously-unproven hop) — PROVEN end-to-end
+The hub dispatcher's operator path (`Dispatcher.processOperator`) sends only when
+`operatorOn && operatorEmail != ""`; the demo `hub-config` set neither, so the self-health pipeline
+(probe → report → checker → dispatch) stopped one hop short of the inbox. **Fix = config-only:**
+`manifests/hub.yaml` `hub-config` ConfigMap now sets `operator_email: admin@felhom.eu` +
+`operator_enabled: true`. Committed, ArgoCD-synced, hub pod restarted to reload; the mounted config shows
+both (Resend key NOT printed; it stays injected from `Secret/resend-api`). Live tag stays **v0.22.1**.
 
-### Tests (green: `go build/vet/test ./...`)
-Change **red-proof** (A→B → one `host_leaf_changed` + baseline advanced; companion: unchanged → none),
-first-obs seeds silently, change-back re-alerts, empty fp skipped, customer-blocked dropped. Cross-repo
-golden mirrors `leaf_fingerprint`.
+**Proven (both channels):**
+- **Operator:** scratch-denied the new agent `stalelock-unlock` grant → `agent_capability_degraded` →
+  `[INFO] Operator email sent for demo-felhom/agent_capability_degraded` (the send-success line that
+  never fired while the path was gated off) → restored → `agent_capability_recovered`.
+- **Customer:** `POST /api/v1/notify` (`event_type:test`) → `{"sent":true}` →
+  `Notification email sent to nagyfenyvesi.viktor@gmail.com for demo-felhom/test`. Same `sendEmail` →
+  Resend the operator path uses.
 
-### Live validation — PASS (+ a wiring bug the live test caught)
-- **v0.22.0 shipped the checker but the `cmd/hub/main.go` goroutine wiring never applied** (a concurrent
-  file-touch made me wrongly assume the edit landed). The **live test caught it**: a leaf regen produced
-  no `host_leaf_changed` — only the controller's complementary `agent_channel_pin_mismatch`. Fixed in
-  **v0.22.1** (wired + redeployed). A good example of live validation catching what the build/test
-  couldn't.
-- After v0.22.1: `Host leaf checker initialized: 1 host fingerprint(s) seeded`; then restoring the leaf
-  (a change from the seeded baseline) →
-  `[WARN] Host leaf: demo-felhom-01 fp 82078fab77ea… → 60b5974d586f… (host_leaf_changed)` (~8 s, one
-  sweep), event saved (no error) + dispatched to the operator. **Independent of** the controller's
-  channel-check, which also fired `agent_channel_pin_mismatch` during the regen — the two are
-  complementary. Final state: leaf == pin `60b5974d…`, channel recovered, capabilities 45/45,
-  controller healthy.
+### Audit
+`documentation/audits/TESTRUN-fullstack-2026-06-29.md`: appended the "Findings closed" section (Part A
+proof + F2-a root cause/fix + F2-b fix/live validation + the pending §D supervised reboot) and removed the
+stale duplicate "Final state" block (it showed old versions 0.45.0/0.90.0/0.20.0 and contradicted the real
+parity).
 
-### Self-health arc — complete
-Agent watches its own privileged capabilities (v0.44.0) + the controller watches its link to the agent
-(controller v0.90.0/F2) + the hub proactively watches every agent's leaf fp fleet-wide (this). The
-original silent-multi-day-outage incident class is now caught from three independent angles.
+### Companion (felhom-agent v0.49.0, for cross-reference)
+- **F2-b:** startup stale-lock recovery (`pct unlock` → delete dangling `vzdump` snapshot → start iff
+  onboot, guarded by a no-vzdump-running invariant). Live-validated on `felhom-pve` (recovery on scratch
+  9999; invariant guard proven against a REAL in-flight 9201 backup). New narrow grant + Critical capability.
+- **F2-a:** the shared-parent boot script never redeployed because `EnsureSharedParent` gated on the unit
+  only; now compares the script too. Root cause confirmed by reproducing the doubling in a `/mnt` scratch
+  dir; corrected script is staged on disk (boot-time-only). The peer-group red-proof is the §D supervised
+  reboot.
+
+### Parity — known-good
+agent **v0.49.0** / controller v0.91.0 / hub v0.22.1 (+ operator-email hub-config); leaf==pin `60b5974d…`;
+caps **46/46**; both drives bound; channel up; apps healthy. Pending the §D supervised reboot: the F2-a
+host-side doubling (corrected script staged; live mount intentionally untouched). No secrets recorded.

documentation/audits/TESTRUN-fullstack-2026-06-29.md

@@ -246,9 +246,65 @@ agent v0.48.0 / controller v0.91.0 / hub v0.22.1; leaf==pin `60b5974d…`; caps
 `bound_under_parent=True`; channel up; nextcloud + controller healthy. (Persistent-but-harmless: the
 §4 host-side doubling — finding F2-a.)
 
-## Final state — known-good
+## Findings closed — operator email + F2-a + F2-b (2026-06-30, agent v0.49.0 + hub config)
 
-agent 0.45.0 / controller 0.90.0 / hub 0.20.0; leaf fp == pinned fp `60b5974d…`; capabilities 45/45
-ok; both USB drives `bound_under_parent=True`, apps healthy; channel up, dashboard clean. All test
-artifacts removed (`/root/testrun-leaf-backup`, the `9.9.9` tag). No secrets recorded (the per-guest
-token was read out-of-band and never printed; leaf fingerprints are not secret).
+The two Phase-2 findings (F2-a, F2-b) and the Phase-1 operator-email gap are now **fixed**. Diagnose-
+before-fix on both risky parts; both root causes confirmed live on `felhom-pve` before any code change.
+
+### Part A — operator email delivery (the previously-unproven hop) — **PROVEN end-to-end**
+The Phase-1 "Note (not a defect)" gap is closed. Root cause was `hub-config` lacking
+`operator_email`/`operator_enabled` (so `Dispatcher.processOperator` returned before the send). Fix =
+config-only (`manifests/hub.yaml` ConfigMap: `operator_email: admin@felhom.eu` + `operator_enabled: true`),
+hub pod restarted to reload; the mounted config now shows both (Resend key NOT printed).
+- **Operator path:** scratch-denied the new `stalelock-unlock` grant → agent `degraded=1` →
+  `[INFO] Host capability: demo-felhom-01 ok → degraded (agent_capability_degraded)` →
+  **`[INFO] Operator email sent for demo-felhom/agent_capability_degraded`** (this line prints ONLY after
+  Resend returns 2xx — it never appeared before, because the path was gated off). Restored the grant →
+  `agent_capability_recovered`.
+- **Customer path:** `POST /api/v1/notify` (`event_type:test`) → `{"sent":true}` →
+  `[INFO] Notification email sent to nagyfenyvesi.viktor@gmail.com for demo-felhom/test` (same `sendEmail`
+  → Resend). Both channels of the dispatcher now proven to the Resend send.
+
+### Part B — reboot-during-backup stale-lock recovery (F2-b) — **FIXED + live-validated**
+`agent v0.49.0`: at startup, `Server.RecoverStaleLockedGuests` clears a stale vzdump lock — `pct unlock`
+→ delete the dangling `vzdump` snapshot → start iff `onboot` and not already running — **only when no
+vzdump is genuinely in-flight** (the invariant; fail-safe otherwise). New narrow grant `FELHOM_STALELOCK
+= pct unlock [0-9]*` + Critical capability `stalelock-unlock` (caps 45→46, all green). Two real issues
+surfaced by live validation and fixed: PVE 9.x rejects `?running=1` (→ `?source=active`); the LXC-start
+`WARNINGS: 1` nesting advisory false-failed the start (→ `AllowWarnings`).
+- **Live (B.3):** induced a `snapshot-delete` lock on scratch guest 9999 (onboot:1, stopped) → agent
+  restart → `stale-lock: clearing a stale backup lock` → `started CT after clearing the stale lock
+  (onboot)` → lock CLEARED, guest running. **Invariant guard proven against a REAL backup:** a genuine
+  agent-scheduled vzdump of 9201 was in-flight during the same restart → `stale-lock: a vzdump backup is
+  genuinely in-flight — leaving the lock (NOT stale)` → 9201's lock untouched; the real backup completed
+  and self-cleared. (Note: PVE reserves the snapshot name `vzdump`, so a dangling one can't be hand-made
+  — the delsnapshot-against-a-real-snapshot path is unit-tested and live-fires in the §D real reboot.)
+
+### Part C — host-side doubling (F2-a) — **root cause FOUND; deploy fix shipped; red-proof pending §D**
+Reproduced the exact pathology in a scratch dir at `/mnt` (where the live parent lives): the **stale**
+sequence (`bind + make-shared`, no `make-private`) leaves the self-bind in root's `shared:1` → sub-bind
+count **2** (doubling); the **correct** sequence (`bind → make-private → make-shared`) gives an own group
+(`shared:525`) → count **1**. The live boot script was the **pre-v0.36.6** body (no `make-private`).
+**Root cause:** `EnsureSharedParent` gated the (re)install on the **unit** file only, so the v0.36.6
+*script-only* fix never deployed to a host whose unit was current. **Fix (`agent v0.49.0`):** the new
+`sharedParentInstallStale` compares **both** the script and the unit. Boot-time-only — it rewrites the
+on-disk script; it does **not** churn the live mount (still guarded on `!isHostMountpoint`).
+- **Deploy fix proven:** on the v0.49.0 startup the corrected script (with `make-private`) was installed
+  to `/usr/local/sbin/felhom-shared-parent.sh` (sudo log shows the install). The live mount is left
+  untouched (`/mnt/felhom-drives` still `shared:1`, doubled — by design, until the next boot).
+- **PENDING (§D, supervised):** the red-proof requires a reboot — the parent must come up as its **own**
+  peer group (≠ `shared:1`) AND the guest must still see **both** drives (`bound_under_parent=true`). Held
+  for a supervised window per the task's STOP rule (no unsupervised reboot / live mount-propagation change).
+
+### §D — supervised host reboot — PENDING (not run unsupervised)
+Validates, in one window: **B** (start a vzdump of 9201 → reboot mid-backup → agent auto-recovers the
+locked CT, the real delsnapshot path) and **C** (parent own-group + guest sees both drives). Per the task's
+strict rule, NOT run autonomously.
+
+### Final parity (post-fix) — known-good
+agent **v0.49.0** / controller v0.91.0 / hub v0.22.1 (+ operator-email hub-config); leaf==pin `60b5974d…`;
+caps **46/46**; both drives bound + apps healthy; channel up; scratch guest 9999 back to baseline; the
+scratch deny removed + `/etc/sudoers` valid; 9201 backup completed + lock self-cleared. **Persistent until
+the §D reboot:** the F2-a host-side doubling (the corrected boot script is staged; live mount intentionally
+untouched). No secrets recorded (tokens read out-of-band, never printed; leaf fps + the operator email are
+not secret).

hub/CHANGELOG.md

@@ -1,5 +1,19 @@
 # Felhom Hub — Changelog
 
+## hub-config — enable operator email alerts (config-only, no image change) (2026-06-30)
+
+`manifests/hub.yaml` `hub-config` ConfigMap: set `notifications.operator_email: admin@felhom.eu` +
+`operator_enabled: true`. The dispatcher's operator path (`Dispatcher.processOperator`) sends only when
+`operatorOn && operatorEmail != ""`; without these it returned early, so the self-health pipeline
+(probe → report → checker → dispatch) stopped one hop short of the inbox (the TESTRUN's unproven hop).
+**No hub image change** (live tag stays v0.22.1) — ConfigMap edit + pod restart to reload.
+- **Proven end-to-end (2026-06-30):** a real capability-degrade alert produced
+  `[INFO] Operator email sent for demo-felhom/agent_capability_degraded` (the send-success line that
+  never fired while the path was gated); the customer path (`POST /api/v1/notify` `event_type:test`) sent
+  to the customer address via the same `sendEmail` → Resend. See
+  `documentation/audits/TESTRUN-fullstack-2026-06-29.md` ("Findings closed", Part A). Operator email is not
+  a secret; the Resend key stays injected from `Secret/resend-api`.
+
 ## v0.22.1 — wire HostLeafChecker into the monitor loop (v0.22.0 missed the wiring) (2026-06-29)
 
 The v0.22.0 commit added `HostLeafChecker` but the `cmd/hub/main.go` goroutine edit never applied, so