docs+config: operator email enabled (Part A, proven) + TESTRUN findings closed
Part A: hub-config operator_email/operator_enabled (committed earlier); operator + customer email paths proven end-to-end via real Resend sends. Append the "Findings closed" section to the TESTRUN audit (Part A proof, F2-a root cause + deploy fix, F2-b fix + live validation, pending §D supervised reboot) and remove the stale duplicate "Final state" block. REPORT overwritten. hub/CHANGELOG entry. No hub image change. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
This commit is contained in:
@@ -246,9 +246,65 @@ agent v0.48.0 / controller v0.91.0 / hub v0.22.1; leaf==pin `60b5974d…`; caps
|
||||
`bound_under_parent=True`; channel up; nextcloud + controller healthy. (Persistent-but-harmless: the
|
||||
§4 host-side doubling — finding F2-a.)
|
||||
|
||||
## Final state — known-good
|
||||
## Findings closed — operator email + F2-a + F2-b (2026-06-30, agent v0.49.0 + hub config)
|
||||
|
||||
agent 0.45.0 / controller 0.90.0 / hub 0.20.0; leaf fp == pinned fp `60b5974d…`; capabilities 45/45
|
||||
ok; both USB drives `bound_under_parent=True`, apps healthy; channel up, dashboard clean. All test
|
||||
artifacts removed (`/root/testrun-leaf-backup`, the `9.9.9` tag). No secrets recorded (the per-guest
|
||||
token was read out-of-band and never printed; leaf fingerprints are not secret).
|
||||
The two Phase-2 findings (F2-a, F2-b) and the Phase-1 operator-email gap are now **fixed**. Diagnose-
|
||||
before-fix on both risky parts; both root causes confirmed live on `felhom-pve` before any code change.
|
||||
|
||||
### Part A — operator email delivery (the previously-unproven hop) — **PROVEN end-to-end**
|
||||
The Phase-1 "Note (not a defect)" gap is closed. Root cause was `hub-config` lacking
|
||||
`operator_email`/`operator_enabled` (so `Dispatcher.processOperator` returned before the send). Fix =
|
||||
config-only (`manifests/hub.yaml` ConfigMap: `operator_email: admin@felhom.eu` + `operator_enabled: true`),
|
||||
hub pod restarted to reload; the mounted config now shows both (Resend key NOT printed).
|
||||
- **Operator path:** scratch-denied the new `stalelock-unlock` grant → agent `degraded=1` →
|
||||
`[INFO] Host capability: demo-felhom-01 ok → degraded (agent_capability_degraded)` →
|
||||
**`[INFO] Operator email sent for demo-felhom/agent_capability_degraded`** (this line prints ONLY after
|
||||
Resend returns 2xx — it never appeared before, because the path was gated off). Restored the grant →
|
||||
`agent_capability_recovered`.
|
||||
- **Customer path:** `POST /api/v1/notify` (`event_type:test`) → `{"sent":true}` →
|
||||
`[INFO] Notification email sent to nagyfenyvesi.viktor@gmail.com for demo-felhom/test` (same `sendEmail`
|
||||
→ Resend). Both channels of the dispatcher now proven to the Resend send.
|
||||
|
||||
### Part B — reboot-during-backup stale-lock recovery (F2-b) — **FIXED + live-validated**
|
||||
`agent v0.49.0`: at startup, `Server.RecoverStaleLockedGuests` clears a stale vzdump lock — `pct unlock`
|
||||
→ delete the dangling `vzdump` snapshot → start iff `onboot` and not already running — **only when no
|
||||
vzdump is genuinely in-flight** (the invariant; fail-safe otherwise). New narrow grant `FELHOM_STALELOCK
|
||||
= pct unlock [0-9]*` + Critical capability `stalelock-unlock` (caps 45→46, all green). Two real issues
|
||||
surfaced by live validation and fixed: PVE 9.x rejects `?running=1` (→ `?source=active`); the LXC-start
|
||||
`WARNINGS: 1` nesting advisory false-failed the start (→ `AllowWarnings`).
|
||||
- **Live (B.3):** induced a `snapshot-delete` lock on scratch guest 9999 (onboot:1, stopped) → agent
|
||||
restart → `stale-lock: clearing a stale backup lock` → `started CT after clearing the stale lock
|
||||
(onboot)` → lock CLEARED, guest running. **Invariant guard proven against a REAL backup:** a genuine
|
||||
agent-scheduled vzdump of 9201 was in-flight during the same restart → `stale-lock: a vzdump backup is
|
||||
genuinely in-flight — leaving the lock (NOT stale)` → 9201's lock untouched; the real backup completed
|
||||
and self-cleared. (Note: PVE reserves the snapshot name `vzdump`, so a dangling one can't be hand-made
|
||||
— the delsnapshot-against-a-real-snapshot path is unit-tested and live-fires in the §D real reboot.)
|
||||
|
||||
### Part C — host-side doubling (F2-a) — **root cause FOUND; deploy fix shipped; red-proof pending §D**
|
||||
Reproduced the exact pathology in a scratch dir at `/mnt` (where the live parent lives): the **stale**
|
||||
sequence (`bind + make-shared`, no `make-private`) leaves the self-bind in root's `shared:1` → sub-bind
|
||||
count **2** (doubling); the **correct** sequence (`bind → make-private → make-shared`) gives an own group
|
||||
(`shared:525`) → count **1**. The live boot script was the **pre-v0.36.6** body (no `make-private`).
|
||||
**Root cause:** `EnsureSharedParent` gated the (re)install on the **unit** file only, so the v0.36.6
|
||||
*script-only* fix never deployed to a host whose unit was current. **Fix (`agent v0.49.0`):** the new
|
||||
`sharedParentInstallStale` compares **both** the script and the unit. Boot-time-only — it rewrites the
|
||||
on-disk script; it does **not** churn the live mount (still guarded on `!isHostMountpoint`).
|
||||
- **Deploy fix proven:** on the v0.49.0 startup the corrected script (with `make-private`) was installed
|
||||
to `/usr/local/sbin/felhom-shared-parent.sh` (sudo log shows the install). The live mount is left
|
||||
untouched (`/mnt/felhom-drives` still `shared:1`, doubled — by design, until the next boot).
|
||||
- **PENDING (§D, supervised):** the red-proof requires a reboot — the parent must come up as its **own**
|
||||
peer group (≠ `shared:1`) AND the guest must still see **both** drives (`bound_under_parent=true`). Held
|
||||
for a supervised window per the task's STOP rule (no unsupervised reboot / live mount-propagation change).
|
||||
|
||||
### §D — supervised host reboot — PENDING (not run unsupervised)
|
||||
Validates, in one window: **B** (start a vzdump of 9201 → reboot mid-backup → agent auto-recovers the
|
||||
locked CT, the real delsnapshot path) and **C** (parent own-group + guest sees both drives). Per the task's
|
||||
strict rule, NOT run autonomously.
|
||||
|
||||
### Final parity (post-fix) — known-good
|
||||
agent **v0.49.0** / controller v0.91.0 / hub v0.22.1 (+ operator-email hub-config); leaf==pin `60b5974d…`;
|
||||
caps **46/46**; both drives bound + apps healthy; channel up; scratch guest 9999 back to baseline; the
|
||||
scratch deny removed + `/etc/sudoers` valid; 9201 backup completed + lock self-cleared. **Persistent until
|
||||
the §D reboot:** the F2-a host-side doubling (the corrected boot script is staged; live mount intentionally
|
||||
untouched). No secrets recorded (tokens read out-of-band, never printed; leaf fps + the operator email are
|
||||
not secret).
|
||||
|
||||
Reference in New Issue
Block a user