docs+config: operator email enabled (Part A, proven) + TESTRUN findings closed

Part A: hub-config operator_email/operator_enabled (committed earlier);
operator + customer email paths proven end-to-end via real Resend sends.
Append the "Findings closed" section to the TESTRUN audit (Part A proof,
F2-a root cause + deploy fix, F2-b fix + live validation, pending §D
supervised reboot) and remove the stale duplicate "Final state" block.
REPORT overwritten. hub/CHANGELOG entry. No hub image change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
This commit is contained in:
2026-06-30 09:21:17 +02:00
parent 83ed127d53
commit 06727e3228
3 changed files with 110 additions and 46 deletions
@@ -246,9 +246,65 @@ agent v0.48.0 / controller v0.91.0 / hub v0.22.1; leaf==pin `60b5974d…`; caps
`bound_under_parent=True`; channel up; nextcloud + controller healthy. (Persistent-but-harmless: the
§4 host-side doubling — finding F2-a.)
## Final state — known-good
## Findings closed — operator email + F2-a + F2-b (2026-06-30, agent v0.49.0 + hub config)
agent 0.45.0 / controller 0.90.0 / hub 0.20.0; leaf fp == pinned fp `60b5974d…`; capabilities 45/45
ok; both USB drives `bound_under_parent=True`, apps healthy; channel up, dashboard clean. All test
artifacts removed (`/root/testrun-leaf-backup`, the `9.9.9` tag). No secrets recorded (the per-guest
token was read out-of-band and never printed; leaf fingerprints are not secret).
The two Phase-2 findings (F2-a, F2-b) and the Phase-1 operator-email gap are now **fixed**. Diagnose-
before-fix on both risky parts; both root causes confirmed live on `felhom-pve` before any code change.
### Part A — operator email delivery (the previously-unproven hop) — **PROVEN end-to-end**
The Phase-1 "Note (not a defect)" gap is closed. Root cause was `hub-config` lacking
`operator_email`/`operator_enabled` (so `Dispatcher.processOperator` returned before the send). Fix =
config-only (`manifests/hub.yaml` ConfigMap: `operator_email: admin@felhom.eu` + `operator_enabled: true`),
hub pod restarted to reload; the mounted config now shows both (Resend key NOT printed).
- **Operator path:** scratch-denied the new `stalelock-unlock` grant → agent `degraded=1` →
`[INFO] Host capability: demo-felhom-01 ok → degraded (agent_capability_degraded)` →
**`[INFO] Operator email sent for demo-felhom/agent_capability_degraded`** (this line prints ONLY after
Resend returns 2xx — it never appeared before, because the path was gated off). Restored the grant →
`agent_capability_recovered`.
- **Customer path:** `POST /api/v1/notify` (`event_type:test`) → `{"sent":true}` →
`[INFO] Notification email sent to nagyfenyvesi.viktor@gmail.com for demo-felhom/test` (same `sendEmail`
→ Resend). Both channels of the dispatcher now proven to the Resend send.
### Part B — reboot-during-backup stale-lock recovery (F2-b) — **FIXED + live-validated**
`agent v0.49.0`: at startup, `Server.RecoverStaleLockedGuests` clears a stale vzdump lock — `pct unlock`
→ delete the dangling `vzdump` snapshot → start iff `onboot` and not already running — **only when no
vzdump is genuinely in-flight** (the invariant; fail-safe otherwise). New narrow grant `FELHOM_STALELOCK
= pct unlock [0-9]*` + Critical capability `stalelock-unlock` (caps 45→46, all green). Two real issues
surfaced by live validation and fixed: PVE 9.x rejects `?running=1` (→ `?source=active`); the LXC-start
`WARNINGS: 1` nesting advisory false-failed the start (→ `AllowWarnings`).
- **Live (B.3):** induced a `snapshot-delete` lock on scratch guest 9999 (onboot:1, stopped) → agent
restart → `stale-lock: clearing a stale backup lock` → `started CT after clearing the stale lock
(onboot)` → lock CLEARED, guest running. **Invariant guard proven against a REAL backup:** a genuine
agent-scheduled vzdump of 9201 was in-flight during the same restart → `stale-lock: a vzdump backup is
genuinely in-flight — leaving the lock (NOT stale)` → 9201's lock untouched; the real backup completed
and self-cleared. (Note: PVE reserves the snapshot name `vzdump`, so a dangling one can't be hand-made
— the delsnapshot-against-a-real-snapshot path is unit-tested and live-fires in the §D real reboot.)
### Part C — host-side doubling (F2-a) — **root cause FOUND; deploy fix shipped; red-proof pending §D**
Reproduced the exact pathology in a scratch dir at `/mnt` (where the live parent lives): the **stale**
sequence (`bind + make-shared`, no `make-private`) leaves the self-bind in root's `shared:1` → sub-bind
count **2** (doubling); the **correct** sequence (`bind → make-private → make-shared`) gives an own group
(`shared:525`) → count **1**. The live boot script was the **pre-v0.36.6** body (no `make-private`).
**Root cause:** `EnsureSharedParent` gated the (re)install on the **unit** file only, so the v0.36.6
*script-only* fix never deployed to a host whose unit was current. **Fix (`agent v0.49.0`):** the new
`sharedParentInstallStale` compares **both** the script and the unit. Boot-time-only — it rewrites the
on-disk script; it does **not** churn the live mount (still guarded on `!isHostMountpoint`).
- **Deploy fix proven:** on the v0.49.0 startup the corrected script (with `make-private`) was installed
to `/usr/local/sbin/felhom-shared-parent.sh` (sudo log shows the install). The live mount is left
untouched (`/mnt/felhom-drives` still `shared:1`, doubled — by design, until the next boot).
- **PENDING (§D, supervised):** the red-proof requires a reboot — the parent must come up as its **own**
peer group (≠ `shared:1`) AND the guest must still see **both** drives (`bound_under_parent=true`). Held
for a supervised window per the task's STOP rule (no unsupervised reboot / live mount-propagation change).
### §D — supervised host reboot — PENDING (not run unsupervised)
Validates, in one window: **B** (start a vzdump of 9201 → reboot mid-backup → agent auto-recovers the
locked CT, the real delsnapshot path) and **C** (parent own-group + guest sees both drives). Per the task's
strict rule, NOT run autonomously.
### Final parity (post-fix) — known-good
agent **v0.49.0** / controller v0.91.0 / hub v0.22.1 (+ operator-email hub-config); leaf==pin `60b5974d…`;
caps **46/46**; both drives bound + apps healthy; channel up; scratch guest 9999 back to baseline; the
scratch deny removed + `/etc/sudoers` valid; 9201 backup completed + lock self-cleared. **Persistent until
the §D reboot:** the F2-a host-side doubling (the corrected boot script is staged; live mount intentionally
untouched). No secrets recorded (tokens read out-of-band, never printed; leaf fps + the operator email are
not secret).