felhom.eu/documentation/audits/TESTRUN-fullstack-2026-06-29.md

TESTRUN — full-stack no-mercy campaign (2026-06-29)

Adversarial integration/chaos validation of everything shipped this session, on the live demo stack (felhom-pve, guest 9201, USB drives felhom-usb/felhom-flash — expendable). No production code changed. Destructive steps were always recovered; the stack ended at known-good parity (§B re-run).

Scope note (honest coverage): this run executed the headline + highest-value NEW tests with full evidence and recovery — the controller-swap happy/rollback (the deferred Scenario D), and the capstone (leaf-regeneration → channel-health detection → recovery) — plus baseline, multi-drive re-propagation, and the swap negative controls. Behaviors already live-proven during this session's implementation tasks (multi-drive concurrency + other-drive-survival; capability degraded detection; channel-health unreachable/transient/recovered) are referenced, not re-ground exhaustively. Deferred items (host reboot, full felhom-host-install.sh reinstall, exhaustive flap loops, DNS/construction live-alert) are listed in Coverage gaps with rationale.

---

§B Baseline (oracle) — captured

Item	Value
Versions	agent 0.45.0, controller 0.90.0, hub 0.20.0
Leaf fp (agent) == pinned fp (controller.yaml)	60b5974d…279d8245 == 60b5974d…279d8245 ✓
Capabilities self-check	45/45 ok, 0 degraded
Drives	felhom-usb + felhom-flash both state=attached, bound_under_parent=True
Host parent peer group	shared:1 (== root's — the §4 "doubling" still present; host not rebooted since make-private granted)
Guest propagation	guest sees both drives, master:51/master:49 (propagation alive)
Apps	nextcloud (on felhom-usb) + controller healthy
Channel-health	steady up (no transitions)

Final parity re-run (end of campaign): identical — agent 0.45.0 / controller 0.90.0, leaf==pin 60b5974d…, capabilities 45/45, both drives bound, nextcloud+controller healthy, channel up (0 down events in 90 s), dashboard banner clear. Stack restored to known-good.

---

§C Multi-drive — PASS

• Concurrency: both drives bound_under_parent=True + state=attached concurrently throughout (baseline + every post-churn check). Guest mountinfo shows both with live master: propagation tags.
• Guest reboot resilience (NEW): pct reboot 9201 → controller healthy, both drives re-propagated (bound_under_parent=True, guest sees both), nextcloud Up (healthy), and 0 drive ABSENT gate events (no flapping). The C1 pre-start hook fired (placeholder for the momentarily-absent bootstrap mount) as designed.
• Other-drive-survival (the original headline bug — detach one, the other survives) was live-proven during the sudoers task (both drives Aktív concurrently; the gate RETURNED both without cross-impact) — referenced, not re-run here.

Deferred: host reboot (would prove the "doubling" self-corrects to its own peer group now that make-private is granted) — needs a maintenance window on the non-expendable host; the §4 doubling is latent/harmless meanwhile. Mid-write detach flap loops — referenced via the prior chaos campaigns.

---

§E Controller-swap — happy PASS; 1 FINDING on rollback

• Happy swap (real agent POST /controller/swap pipeline): 0.90.0 → 0.89.0. Evidence: the tee-written /etc/felhom-controller-image is byte-exact (od -c: …controller:0.89.0\n, trailing newline); agent logged controller swap requested → controller-swap: new controller healthy; container came up 0.89.0 Up (healthy). Swapped back 0.89.0 → 0.90.0 cleanly. The stdin-tee write + the 5 narrow grants + bootstrap restart all work live. PASS.
• Negative controls (scoping): as felhom-agent, sudo -n -l of pct exec 9201 -- {bash -c id, rm -rf /tmp/x, docker rm -f felhom-controller, tee /etc/shadow} → all DENIED (exit 1); the real tee /etc/felhom-controller-image → permitted (0). PASS.
• Rollback — FINDING (recorded, not fixed): swapped to a deliberately-bad image (alpine tagged felhom-controller:9.9.9). Expected health-fail → rollback; instead the swap marked it healthy in ~4 s and did NOT roll back. Root cause: alpine has no HEALTHCHECK, so the swap's "running + no-healthcheck = healthy" rule (a deliberate feature for images that legitimately lack a healthcheck) passed on a transient "running" instant — then the container crash-loops (Restarting (0)). Severity: low for the real controller (its image has a healthcheck, so a broken real build would report unhealthy/never-running → rollback; the deterministic rollback path is unit-proven by TestControllerSwap_RollbackOnUnhealthy). Gap: a controller image that has no healthcheck AND crash-loops would be marked "done" by the point-in-time verify and not roll back. Suggested hardening (operator decision): the verify should re-check health over a few seconds / reject RestartCount>0, so a fast start-then-exit can't pass. Recovered cleanly by swapping back to 0.90.0 (has a healthcheck → healthy); bad tag removed.

Deferred: a clean live rollback needs a present-but-healthcheck-failing controller image, which can't be manufactured without building one (out of scope — "no new builds"). Rollback logic is unit-proven; the live finding above is the more interesting result.

---

§F + §K Channel-health & CAPSTONE — PASS (the headline)

Capstone (§K): original-incident class → DETECTED in ~25 s

Reproduced the exact failure that started this work: with the controller running (channel seeded up), moved the agent leaf+key aside and restarted the agent → it regenerated a new leaf (60b5974d… → 911d703c…), so the agent serves a cert the controller doesn't pin. Agent stayed active, capabilities 45/45 ok (leaf change doesn't affect capabilities).

Detection (the whole point):

[channel] agent channel DOWN (up→down:pin_mismatch): agentapi: GET /storage: … TLS pin mismatch …
Event pushed: agent_channel_pin_mismatch (error) — … the agent's leaf cert no longer matches … (HTTP 200)
dashboard banner: "A tárolókezelő ügynök tanúsítványa megváltozott"

DETECTED ~25 s after the regeneration (one channel cycle) — vs the original silent multi-day disk-UI outage. The operator alert fired with the correct event type + the dashboard pin-mismatch banner appeared.

Recovery (R1): restored the backed-up leaf+key → restart agent → serves 60b5974d… again → [channel] agent channel recovered (was down:pin_mismatch) + Event pushed: agent_channel_recovered

• both drives bound_under_parent=True + banner cleared. ~one cycle after the restart.

§F other modes

• Unreachable / transient / recovered were live-proven during the v0.90.0 task (and the recovered path again here): a clean systemctl restart felhom-agent → no alert (debounce); systemctl stop → transient down (unreachable, 1/2) — suppressed then DOWN (up→down:unreachable)
  • Event pushed: agent_channel_unreachable (HTTP 200) + Hungarian banner; start → recovered. Referenced.
• 401 / DNS-misconfigured / construction — classification is unit-proven (channelhealth tests). Live alert for these is harder to induce without a controller restart, which reseeds state (first-obs = seed, no alert) — see Coverage gaps.

§K verdict: the original-incident class is now DETECTED within a cycle by the channel-health check (PASS). Prevention recommendation (still worth doing, operator decision): ship --preserve-state-from (carry local-api.{crt,key} + local-tokens.log across a reinstall) + loud-on-regenerate, to turn the now-detected-but-disruptive reinstall into a non-event rather than a detect-then-R1-recover. Detection is the safety net; preservation is the fix.

---

Defects / findings (separated from EXPECTED outcomes)

#	Finding	Severity	Repro
F1	Swap verify passes a no-healthcheck image that crash-loops (transient "running" + no-healthcheck rule → false "done", no rollback)	low (real controller has a healthcheck)	tag alpine as felhom-controller:X.Y.Z, swap to it → marked healthy ~4 s, then Restarting (0)

F1 RESOLVED 2026-06-29 (agent v0.47.0): the swap verify now reads {{.RestartCount}} (running && rc>0 → not healthy) and requires a 3-poll stability dwell for a no-healthcheck image (a real healthcheck is trusted immediately). Live re-test: the same alpine :9.9.9 swap now reaches state: failed ("did not become healthy within timeout") and rolls back to 0.91.0 — no false "done". The docker inspect -f * grant is unchanged (* spans the extended template, exit-0 confirmed). See felhom-agent/REPORT.md + CHANGELOG v0.47.0.

EXPECTED (not defects): the capstone leaf regeneration → pin mismatch (the prevention fix was never shipped — regeneration is expected; detection is the pass criterion, and it passed). The §4 host-side "doubling" persisting until a clean host reboot (expected; make-private is granted, applies next fresh boot).

---

Coverage gaps (couldn't test safely / deferred — with rationale)

• Host reboot (§C) — would confirm the parent self-corrects to its own peer group; needs a maintenance window on the non-expendable host. Deferred.
• Full felhom-host-install.sh reinstall (§K) — reinstalls the agent on felhom-pve (non- expendable); the equivalent leaf-regeneration was exercised instead (same incident class, same detection, recoverable). The full reinstall is the operator's call in a window.
• 401 / DNS / construction live alert (§F) — inducing these needs a controller-config change + restart, which reseeds the channel state (first-obs seeds silently, no alert). Classification is unit-proven; the dashboard would show the state, but the up→down notification for these isn't cleanly inducible live without an agent-side trick (pin/unreachable have one; 401 would need a token-store edit). Deferred as lower-value (unit-covered).
• Capability live scratch-drop → hub alert (§D) — the lxc-info-degraded path + drive flap was the live incident the capability task already diagnosed/fixed; the hub HostCapabilityChecker transition is unit-proven. A live drop requires editing the certified sudoers (sudoers can't "override-to-deny" via a drop-in cleanly) — deferred to avoid mutating the freshly-certified allowlist for a unit-covered path.
• Exhaustive flap/mid-write/3rd-drive loops (§C/§G adversarial) — referenced via the prior chaos campaigns (#1–#3, all FIXED); not re-run.

---

F2 CLOSED (2026-06-29) + prevention shipped

The testrun surfaced F2: a channel/host failure present at startup/reseed alerted only on a live transition, so a born-down (e.g. a controller booting into pin_mismatch) was dashboard-only, no operator email. Closed — controller v0.91.0 (alerted-flag rework) + hub v0.21.0 (checkers seed only healthy; born-degraded/stale emits on first Check, cooldown dedups). Live-validated: a controller restarted into a regenerated-leaf channel logged [channel] agent channel DOWN (unseeded->down:pin_mismatch) and pushed agent_channel_pin_mismatch (~60 s) — the capstone's missing half (detection at startup). Prevention for the original incident also shipped: agent v0.46.0 loud-WARNs a regenerated leaf (leaf REGENERATED — pins now INVALID), and felhom-host-install.sh gained --preserve-state-from (keeps the leaf/pin stable across a reinstall)

• a populated-host guard (refuse to regenerate the leaf on a host with guests). The §K prevention recommendation is now implemented; the full live reinstall is the only piece awaiting a supervised run. (F1 swap-rollback RestartCount hardening remains backlog.)

Coverage gaps — NOW RESOLVED (supervised follow-up, 2026-06-29)

Baselines: agent v0.48.0, controller v0.91.0, hub v0.22.1; leaf==pin 60b5974d…; caps 45/45; both drives bound; channel up.

Phase 1 — capability→hub operator alert, LIVE — PASS (the path that never fired in the field)

Induced a minimal-blast-radius degrade: a scratch deny drop-in (/etc/sudoers.d/zz-felhom-testdeny, !…pct exec [0-9]* -- tee /etc/felhom-controller-image, visudo-clean) + agent restart.

• Agent stayed up (serve-degraded): capabilities self-check ok=44 total=45 degraded=1 + capability DEGRADED … controllerswap-write … critical=true reason="sudo policy denied".
• The box kept working — both drives bound_under_parent=True throughout (lxc-info still granted).
• Hub fired the alert (~24s): Host capability: demo-felhom-01 ok → degraded (agent_capability_degraded); the saved event message is English and names the cap+feature: "agent privileged capability degraded — controllerswap-write (impairs: controller-swap / managed auto-update)" (severity warning).
• Recover: removed the scratch deny + visudo -cf /etc/sudoers OK + restart → ok=45/45 → Host capability: demo-felhom-01 degraded → ok (agent_capability_recovered).
• Note (not a defect): a literal operator email was not dispatched — the demo hub has no operator email address configured (no "Operator email sent/failed" log → processOperator returns before the send). In prod with the operator address set, the same path sends via FormatOperatorEmail + Resend (RESEND key IS injected). The alert path itself (probe→report→checker→event+dispatch) is proven.

Phase 2 — host-reboot "doubling" self-correction — 2 FINDINGS (drive recovery PASS)

Rebooted felhom-pve (uptime was 5d, no boot since the make-private grant). After boot:

• PASS — drive recovery + no false page: the boot ran felhom-shared-parent.service; both drives re-propagated into the guest (bound_under_parent=True, master: tags alive), nextcloud + controller recovered healthy, leaf==pin MATCH 60b5974d…, caps 45/45. Zero spurious alerts on the clean boot (drive-ABSENT 0, channel DOWN 0, hub capability/leaf/stale 0) — confirms F2 first-obs-seed doesn't false-page a normal reboot.
• FINDING F2-a (doubling did NOT self-correct): the host parent /mnt/felhom-drives is still shared:1 (== root's peer group) and the drives are still doubled (each under parent 149 AND root 33) — identical to the pre-reboot state. The §4 hypothesis is falsified: the boot script felhom-shared-parent.sh runs as root (systemd), so it never needed the sudoers make-private grant — yet its make-private+make-shared sequence does not isolate the self-bind from root's shared:1. Harmless functionally (drives work), but the doubling persists across a clean boot. Severity: low (cosmetic/correctness; the agent's AttachDrive already converges binds). Repro: reboot the host, then grep ' /mnt/felhom-drives ' /proc/self/mountinfo → shared:1. Root-cause not pursued (won't churn the live parent — would orphan the guest slave). A fix should assert the parent's group ≠ root's after the boot script and investigate the make-private timing/semantics for a subdir-of-root self-bind.
• FINDING F2-b (reboot during backup → stuck guest): a vzdump backup was in flight at reboot (snapshot @ 23:43:45); the reboot interrupted the snapshot-delete cleanup, leaving a stale lock: snapshot-delete + a dangling vzdump snapshot. Guest 9201 therefore failed to autostart (onboot:1 can't start a locked CT: "Starting CT 9201 failed: CT is locked (snapshot-delete)"). Recovered with pct unlock 9201 + pct start + pct delsnapshot 9201 vzdump. The "power-loss/reboot during a backup" edge — a host reboot should check for an in-flight backup, or boot-time recovery should clear a stale vzdump snapshot lock + auto-start. Severity: medium (an unattended host reboot during the daily backup window would leave the customer box down until manual unlock).

Phase 3 — host_leaf_changed LIVE on the wired hub (v0.22.1) — PASS (referenced)

Already live-proven against the wired v0.22.1 in the leaf-fp task: a leaf regen produced the controller agent_channel_pin_mismatch AND (on restore) the hub host_leaf_changed (82078fab…→60b5974d…) — two independent detections of the same re-key (controller-local + hub-proactive). Not re-run here (would require another regen + R1). See felhom.eu/REPORT.md (HostLeafChecker) + the §"Self-health arc COMPLETE" note in the channel-health spike doc.

Coverage-gaps verdict

All three deferred gaps now exercised: Phase 1 fired live (event captured; email is operator-config- gated, not a code defect); Phase 2 drive-recovery PASS but surfaced two findings (doubling NOT self-corrected — §4 hypothesis falsified; reboot-during-backup stale lock); Phase 3 referenced. Scratch artifacts removed (zz-felhom-testdeny; the dangling vzdump snapshot); /etc/sudoers re-validated.

Final parity (post follow-up) — known-good

agent v0.48.0 / controller v0.91.0 / hub v0.22.1; leaf==pin 60b5974d…; caps 45/45; both drives bound_under_parent=True; channel up; nextcloud + controller healthy. (Persistent-but-harmless: the §4 host-side doubling — finding F2-a.)

Findings closed — operator email + F2-a + F2-b (2026-06-30, agent v0.49.0 + hub config)

The two Phase-2 findings (F2-a, F2-b) and the Phase-1 operator-email gap are now fixed. Diagnose- before-fix on both risky parts; both root causes confirmed live on felhom-pve before any code change.

Part A — operator email delivery (the previously-unproven hop) — PROVEN end-to-end

The Phase-1 "Note (not a defect)" gap is closed. Root cause was hub-config lacking operator_email/operator_enabled (so Dispatcher.processOperator returned before the send). Fix = config-only (manifests/hub.yaml ConfigMap: operator_email: admin@felhom.eu + operator_enabled: true), hub pod restarted to reload; the mounted config now shows both (Resend key NOT printed).

• Operator path: scratch-denied the new stalelock-unlock grant → agent degraded=1 → [INFO] Host capability: demo-felhom-01 ok → degraded (agent_capability_degraded) → [INFO] Operator email sent for demo-felhom/agent_capability_degraded (this line prints ONLY after Resend returns 2xx — it never appeared before, because the path was gated off). Restored the grant → agent_capability_recovered.
• Customer path: POST /api/v1/notify (event_type:test) → {"sent":true} → [INFO] Notification email sent to nagyfenyvesi.viktor@gmail.com for demo-felhom/test (same sendEmail → Resend). Both channels of the dispatcher now proven to the Resend send.

Part B — reboot-during-backup stale-lock recovery (F2-b) — FIXED + live-validated

agent v0.49.0: at startup, Server.RecoverStaleLockedGuests clears a stale vzdump lock — pct unlock → delete the dangling vzdump snapshot → start iff onboot and not already running — only when no vzdump is genuinely in-flight (the invariant; fail-safe otherwise). New narrow grant FELHOM_STALELOCK = pct unlock [0-9]* + Critical capability stalelock-unlock (caps 45→46, all green). Two real issues surfaced by live validation and fixed: PVE 9.x rejects ?running=1 (→ ?source=active); the LXC-start WARNINGS: 1 nesting advisory false-failed the start (→ AllowWarnings).

• Live (B.3): induced a snapshot-delete lock on scratch guest 9999 (onboot:1, stopped) → agent restart → stale-lock: clearing a stale backup lock → started CT after clearing the stale lock (onboot) → lock CLEARED, guest running. Invariant guard proven against a REAL backup: a genuine agent-scheduled vzdump of 9201 was in-flight during the same restart → stale-lock: a vzdump backup is genuinely in-flight — leaving the lock (NOT stale) → 9201's lock untouched; the real backup completed and self-cleared. (Note: PVE reserves the snapshot name vzdump, so a dangling one can't be hand-made — the delsnapshot-against-a-real-snapshot path is unit-tested and live-fires in the §D real reboot.)

Part C — host-side doubling (F2-a) — root cause FOUND; fix shipped; red-proof PASSED (§D)

Reproduced the exact pathology in a scratch dir at /mnt (where the live parent lives): the stale sequence (bind + make-shared, no make-private) leaves the self-bind in root's shared:1 → sub-bind count 2 (doubling); the correct sequence (bind → make-private → make-shared) gives an own group (shared:525) → count 1. The live boot script was the pre-v0.36.6 body (no make-private). Root cause: EnsureSharedParent gated the (re)install on the unit file only, so the v0.36.6 script-only fix never deployed to a host whose unit was current. Fix (agent v0.49.0): the new sharedParentInstallStale compares both the script and the unit. Boot-time-only — it rewrites the on-disk script; it does not churn the live mount (still guarded on !isHostMountpoint).

• Deploy fix proven: on the v0.49.0 startup the corrected script (with make-private) was installed to /usr/local/sbin/felhom-shared-parent.sh (sudo log shows the install). The live mount was left untouched until the reboot (/mnt/felhom-drives still shared:1, doubled — by design).
• RED-PROOF PASSED (§D, two reboots): /mnt/felhom-drives came up as shared:94 — its OWN peer group, NOT root's shared:1 — and each drive has exactly 1 host bind (doubling GONE; was 2 each). The guest still sees both drives (bound_under_parent=true; apps all healthy). Consistent across both reboots. (Separate, pre-existing, harmless observation: the GUEST shows 2 mounts per drive — two parent mounts in the guest namespace, same /dev/sdX1 /felhom-data source, master: slaved correctly; NOT a regression of this fix, which only corrected the host parent's peer group. Worth a future look.)

§D — supervised host reboot — DONE (PASS)

Run in a supervised window (user-authorized). Two reboots:

1. Reboot mid-backup (caught the backup/tar phase): PVE auto-cleared the backup lock and started 9201 normally (no stuck symptom in that phase); the agent recovery saw no lock → correctly skipped. The C red-proof passed (parent shared:94, no doubling). A real vzdump snapshot remained.
2. Reboot with the EXACT F2-b state (induced lock: snapshot-delete + the real dangling vzdump snapshot from reboot 1): pve-guests FAILED to start the locked CT — "starting CT 9201: CT is locked (snapshot-delete)" → "Starting CT 9201 failed" — the genuine stuck symptom. The agent then auto-recovered: clearing a stale backup lock (lock=snapshot-delete) → removed dangling vzdump snapshot (the real delsnapshot path) → started CT after clearing the stale lock (onboot). 9201 running, lock cleared, snapshot deleted, no error (the ?source=active + AllowWarnings fixes held). Zero spurious operator pages across both reboots (no host_down/stale/capability/leaf events — only the controller's benign notification-prefs re-register). The interrupted-backup temp .dat was cleaned.

Final parity (post-fix, post-§D) — known-good

agent v0.49.0 / controller v0.91.0 / hub v0.22.1 (+ operator-email hub-config); leaf==pin 60b5974d…; caps 46/46; /mnt/felhom-drives is its own peer group shared:94 with no doubling (1 bind each); both drives bound + 12 healthy containers; scratch guest 9999 back to baseline; the scratch deny removed + /etc/sudoers valid; 9201 lock cleared + dangling snapshot deleted. Both findings F2-a and F2-b are now CLOSED and live-validated. No secrets recorded (tokens read out-of-band, never printed; leaf fps + the operator email are not secret).