From c978ea1951e99b2ff4fac842938927f93e2ff16c Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 29 Jun 2026 23:56:54 +0200 Subject: [PATCH] =?UTF-8?q?testrun:=20coverage=20gaps=20RESOLVED=20?= =?UTF-8?q?=E2=80=94=20capability->hub=20alert=20fired=20live;=202=20reboo?= =?UTF-8?q?t=20findings=20(doubling=20NOT=20self-corrected,=20backup-lock)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Phase 1: capability->hub operator alert fired live (event captured, English, names cap+feature); email is operator-config-gated in demo (not a defect). Phase 2: drive re-propagation + zero false pages PASS, but TWO findings — F2-a the §4 doubling did NOT self-correct (boot script runs as root, never needed the grant → hypothesis falsified), F2-b a vzdump backup in flight at reboot left a stale snapshot-delete lock blocking guest autostart. Phase 3 referenced (host_leaf_changed proven on v0.22.1). Final parity known-good. Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ --- .../audits/TESTRUN-fullstack-2026-06-29.md | 68 +++++++++++++++++++ 1 file changed, 68 insertions(+) diff --git a/documentation/audits/TESTRUN-fullstack-2026-06-29.md b/documentation/audits/TESTRUN-fullstack-2026-06-29.md index 2f4d85c..16dd43b 100644 --- a/documentation/audits/TESTRUN-fullstack-2026-06-29.md +++ b/documentation/audits/TESTRUN-fullstack-2026-06-29.md @@ -178,6 +178,74 @@ missing half (detection at startup). **Prevention** for the original incident al recommendation is now implemented; the full live reinstall is the only piece awaiting a supervised run. (F1 swap-rollback `RestartCount` hardening remains backlog.) +## Coverage gaps — NOW RESOLVED (supervised follow-up, 2026-06-29) + +Baselines: agent v0.48.0, controller v0.91.0, hub v0.22.1; leaf==pin `60b5974d…`; caps 45/45; both +drives bound; channel up. + +### Phase 1 — capability→hub operator alert, LIVE — **PASS** (the path that never fired in the field) +Induced a minimal-blast-radius degrade: a scratch deny drop-in (`/etc/sudoers.d/zz-felhom-testdeny`, +`!…pct exec [0-9]* -- tee /etc/felhom-controller-image`, `visudo`-clean) + agent restart. +- Agent stayed **up** (serve-degraded): `capabilities self-check ok=44 total=45 degraded=1` + + `capability DEGRADED … controllerswap-write … critical=true reason="sudo policy denied"`. +- The box kept working — both drives `bound_under_parent=True` throughout (lxc-info still granted). +- **Hub fired the alert (~24s):** `Host capability: demo-felhom-01 ok → degraded + (agent_capability_degraded)`; the saved event message is **English** and names the cap+feature: + *"agent privileged capability degraded — controllerswap-write (impairs: controller-swap / managed + auto-update)"* (severity warning). +- Recover: removed the scratch deny + `visudo -cf /etc/sudoers` OK + restart → `ok=45/45` → + `Host capability: demo-felhom-01 degraded → ok (agent_capability_recovered)`. +- **Note (not a defect):** a literal operator *email* was **not** dispatched — the demo hub has no + operator email address configured (no "Operator email sent/failed" log → `processOperator` returns + before the send). In prod with the operator address set, the same path sends via FormatOperatorEmail + + Resend (RESEND key IS injected). The alert path itself (probe→report→checker→event+dispatch) is proven. + +### Phase 2 — host-reboot "doubling" self-correction — **2 FINDINGS** (drive recovery PASS) +Rebooted felhom-pve (uptime was 5d, no boot since the `make-private` grant). After boot: +- **PASS — drive recovery + no false page:** the boot ran `felhom-shared-parent.service`; both drives + re-propagated into the guest (`bound_under_parent=True`, `master:` tags alive), nextcloud + controller + recovered healthy, leaf==pin MATCH `60b5974d…`, caps 45/45. **Zero** spurious alerts on the clean boot + (drive-ABSENT 0, channel DOWN 0, hub capability/leaf/stale 0) — confirms F2 first-obs-seed doesn't + false-page a normal reboot. +- **FINDING F2-a (doubling did NOT self-correct):** the host parent `/mnt/felhom-drives` is **still + `shared:1`** (== root's peer group) and the drives are **still doubled** (each under parent 149 AND + root 33) — identical to the pre-reboot state. **The §4 hypothesis is falsified:** the boot script + `felhom-shared-parent.sh` runs as **root** (systemd), so it never needed the sudoers `make-private` + grant — yet its `make-private`+`make-shared` sequence does not isolate the self-bind from root's + `shared:1`. Harmless functionally (drives work), but the doubling persists across a clean boot. + Severity: low (cosmetic/correctness; the agent's `AttachDrive` already converges binds). **Repro:** + reboot the host, then `grep ' /mnt/felhom-drives ' /proc/self/mountinfo` → `shared:1`. Root-cause not + pursued (won't churn the live parent — would orphan the guest slave). A fix should assert the parent's + group ≠ root's after the boot script and investigate the make-private timing/semantics for a + subdir-of-root self-bind. +- **FINDING F2-b (reboot during backup → stuck guest):** a **vzdump backup was in flight** at reboot + (snapshot @ 23:43:45); the reboot interrupted the `snapshot-delete` cleanup, leaving a **stale + `lock: snapshot-delete`** + a dangling `vzdump` snapshot. Guest 9201 therefore **failed to autostart** + (`onboot:1` can't start a locked CT: *"Starting CT 9201 failed: CT is locked (snapshot-delete)"*). + Recovered with `pct unlock 9201` + `pct start` + `pct delsnapshot 9201 vzdump`. The "power-loss/reboot + during a backup" edge — a host reboot should check for an in-flight backup, or boot-time recovery + should clear a stale vzdump snapshot lock + auto-start. Severity: medium (an unattended host reboot + during the daily backup window would leave the customer box down until manual unlock). + +### Phase 3 — `host_leaf_changed` LIVE on the wired hub (v0.22.1) — **PASS (referenced)** +Already live-proven against the wired v0.22.1 in the leaf-fp task: a leaf regen produced the controller +`agent_channel_pin_mismatch` AND (on restore) the hub `host_leaf_changed` +(`82078fab…→60b5974d…`) — two **independent** detections of the same re-key (controller-local + +hub-proactive). Not re-run here (would require another regen + R1). See +`felhom.eu/REPORT.md` (HostLeafChecker) + the §"Self-health arc COMPLETE" note in the channel-health +spike doc. + +### Coverage-gaps verdict +All three deferred gaps now exercised: Phase 1 fired live (event captured; email is operator-config- +gated, not a code defect); Phase 2 drive-recovery PASS but surfaced **two findings** (doubling NOT +self-corrected — §4 hypothesis falsified; reboot-during-backup stale lock); Phase 3 referenced. Scratch +artifacts removed (`zz-felhom-testdeny`; the dangling vzdump snapshot); `/etc/sudoers` re-validated. + +### Final parity (post follow-up) — known-good +agent v0.48.0 / controller v0.91.0 / hub v0.22.1; leaf==pin `60b5974d…`; caps 45/45; both drives +`bound_under_parent=True`; channel up; nextcloud + controller healthy. (Persistent-but-harmless: the +§4 host-side doubling — finding F2-a.) + ## Final state — known-good agent 0.45.0 / controller 0.90.0 / hub 0.20.0; leaf fp == pinned fp `60b5974d…`; capabilities 45/45