felhom.eu/documentation/audits/TESTRUN-fullstack-2026-06-29.md

# TESTRUN — full-stack no-mercy campaign (2026-06-29)

Adversarial integration/chaos validation of everything shipped this session, on the live demo stack
(felhom-pve, guest 9201, USB drives `felhom-usb`/`felhom-flash` — expendable). No production code
changed. Destructive steps were always recovered; the stack ended at known-good parity (§B re-run).

**Scope note (honest coverage):** this run executed the **headline + highest-value NEW** tests with
full evidence and recovery — the controller-swap happy/rollback (the deferred Scenario D), and the
**capstone** (leaf-regeneration → channel-health detection → recovery) — plus baseline, multi-drive
re-propagation, and the swap negative controls. Behaviors already **live-proven during this session's
implementation tasks** (multi-drive concurrency + other-drive-survival; capability degraded detection;
channel-health unreachable/transient/recovered) are **referenced**, not re-ground exhaustively. Deferred
items (host reboot, full `felhom-host-install.sh` reinstall, exhaustive flap loops, DNS/construction
live-alert) are listed in **Coverage gaps** with rationale.

---

## §B Baseline (oracle) — captured

| Item | Value |
|---|---|
| Versions | agent **0.45.0**, controller **0.90.0**, hub **0.20.0** |
| Leaf fp (agent) == pinned fp (controller.yaml) | `60b5974d…279d8245` == `60b5974d…279d8245` ✓ |
| Capabilities self-check | **45/45 ok, 0 degraded** |
| Drives | felhom-usb + felhom-flash both `state=attached`, `bound_under_parent=True` |
| Host parent peer group | `shared:1` (== root's — the §4 "doubling" still present; host not rebooted since `make-private` granted) |
| Guest propagation | guest sees both drives, `master:51`/`master:49` (propagation alive) |
| Apps | nextcloud (on felhom-usb) + controller healthy |
| Channel-health | steady up (no transitions) |

**Final parity re-run (end of campaign):** identical — agent 0.45.0 / controller 0.90.0, leaf==pin
`60b5974d…`, capabilities 45/45, both drives bound, nextcloud+controller healthy, channel up (0 down
events in 90 s), dashboard banner clear. **Stack restored to known-good.**

---

## §C Multi-drive — PASS

- **Concurrency:** both drives `bound_under_parent=True` + `state=attached` concurrently throughout
  (baseline + every post-churn check). Guest mountinfo shows both with live `master:` propagation tags.
- **Guest reboot resilience (NEW):** `pct reboot 9201` → controller healthy, **both** drives
  re-propagated (`bound_under_parent=True`, guest sees both), nextcloud `Up (healthy)`, and **0**
  `drive ABSENT` gate events (no flapping). The C1 pre-start hook fired (placeholder for the
  momentarily-absent bootstrap mount) as designed.
- **Other-drive-survival** (the original headline bug — detach one, the other survives) was
  **live-proven during the sudoers task** (both drives Aktív concurrently; the gate RETURNED both
  without cross-impact) — referenced, not re-run here.

**Deferred:** host reboot (would prove the "doubling" self-corrects to its own peer group now that
`make-private` is granted) — needs a maintenance window on the non-expendable host; the §4 doubling is
latent/harmless meanwhile. Mid-write detach flap loops — referenced via the prior chaos campaigns.

---

## §E Controller-swap — happy PASS; **1 FINDING** on rollback

- **Happy swap (real agent `POST /controller/swap` pipeline):** 0.90.0 → 0.89.0. Evidence: the
  `tee`-written `/etc/felhom-controller-image` is **byte-exact** (`od -c`: `…controller:0.89.0\n`,
  trailing newline); agent logged `controller swap requested` → `controller-swap: new controller
  healthy`; container came up **0.89.0 Up (healthy)**. Swapped back 0.89.0 → 0.90.0 cleanly. The
  stdin-`tee` write + the 5 narrow grants + bootstrap restart all work live. **PASS.**
- **Negative controls (scoping):** as `felhom-agent`, `sudo -n -l` of
  `pct exec 9201 -- {bash -c id, rm -rf /tmp/x, docker rm -f felhom-controller, tee /etc/shadow}` →
  **all DENIED (exit 1)**; the real `tee /etc/felhom-controller-image` → **permitted (0)**. **PASS.**
- **Rollback — FINDING (recorded, not fixed):** swapped to a deliberately-bad image (alpine tagged
  `felhom-controller:9.9.9`). Expected health-fail → rollback; instead the swap marked it
  **healthy in ~4 s** and did **NOT** roll back. Root cause: alpine has **no `HEALTHCHECK`**, so the
  swap's "running + no-healthcheck = healthy" rule (a deliberate feature for images that legitimately
  lack a healthcheck) passed on a transient "running" instant — then the container crash-loops
  (`Restarting (0)`). **Severity: low** for the real controller (its image *has* a healthcheck, so a
  broken real build would report unhealthy/never-running → rollback; the deterministic rollback path is
  unit-proven by `TestControllerSwap_RollbackOnUnhealthy`). **Gap:** a controller image that has **no
  healthcheck AND crash-loops** would be marked "done" by the point-in-time verify and not roll back.
  **Suggested hardening (operator decision):** the verify should re-check health over a few seconds /
  reject `RestartCount>0`, so a fast start-then-exit can't pass. Recovered cleanly by swapping back to
  0.90.0 (has a healthcheck → healthy); bad tag removed.

**Deferred:** a *clean* live rollback needs a present-but-healthcheck-failing controller image, which
can't be manufactured without building one (out of scope — "no new builds"). Rollback logic is
unit-proven; the live finding above is the more interesting result.

---

## §F + §K Channel-health & CAPSTONE — PASS (the headline)

### Capstone (§K): original-incident class → DETECTED in ~25 s
Reproduced the exact failure that started this work: with the controller running (channel seeded up),
moved the agent leaf+key aside and restarted the agent → it **regenerated a new leaf**
(`60b5974d…` → `911d703c…`), so the agent serves a cert the controller doesn't pin. Agent stayed
**active**, capabilities **45/45 ok** (leaf change doesn't affect capabilities).

**Detection (the whole point):**
```
[channel] agent channel DOWN (up→down:pin_mismatch): agentapi: GET /storage: … TLS pin mismatch …
Event pushed: agent_channel_pin_mismatch (error) — … the agent's leaf cert no longer matches … (HTTP 200)
dashboard banner: "A tárolókezelő ügynök tanúsítványa megváltozott"
```
**DETECTED ~25 s** after the regeneration (one channel cycle) — vs the original **silent multi-day**
disk-UI outage. The operator alert fired with the correct event type + the dashboard pin-mismatch
banner appeared.

**Recovery (R1):** restored the backed-up leaf+key → restart agent → serves `60b5974d…` again →
`[channel] agent channel recovered (was down:pin_mismatch)` + `Event pushed: agent_channel_recovered`
+ both drives `bound_under_parent=True` + banner cleared. ~one cycle after the restart.

### §F other modes
- **Unreachable / transient / recovered** were **live-proven during the v0.90.0 task** (and the
  recovered path again here): a clean `systemctl restart felhom-agent` → no alert (debounce);
  `systemctl stop` → `transient down (unreachable, 1/2) — suppressed` then `DOWN (up→down:unreachable)`
  + `Event pushed: agent_channel_unreachable` (HTTP 200) + Hungarian banner; `start` → recovered.
  Referenced.
- **401 / DNS-misconfigured / construction** — classification is unit-proven (`channelhealth` tests).
  Live **alert** for these is harder to induce without a controller restart, which **reseeds** state
  (first-obs = seed, no alert) — see Coverage gaps.

**§K verdict:** the original-incident class is now **DETECTED within a cycle** by the channel-health
check (PASS). **Prevention recommendation (still worth doing, operator decision):** ship
`--preserve-state-from` (carry `local-api.{crt,key}` + `local-tokens.log` across a reinstall) +
loud-on-regenerate, to turn the now-*detected*-but-disruptive reinstall into a **non-event** rather
than a detect-then-R1-recover. Detection is the safety net; preservation is the fix.

---

## Defects / findings (separated from EXPECTED outcomes)

| # | Finding | Severity | Repro |
|---|---|---|---|
| F1 | Swap verify passes a **no-healthcheck image that crash-loops** (transient "running" + no-healthcheck rule → false "done", no rollback) | low (real controller has a healthcheck) | tag alpine as `felhom-controller:X.Y.Z`, swap to it → marked healthy ~4 s, then `Restarting (0)` |

**F1 RESOLVED 2026-06-29 (agent v0.47.0):** the swap verify now reads `{{.RestartCount}}`
(`running && rc>0` → not healthy) and requires a 3-poll **stability dwell** for a no-healthcheck image
(a real healthcheck is trusted immediately). Live re-test: the same alpine `:9.9.9` swap now reaches
`state: failed` ("did not become healthy within timeout") and **rolls back** to 0.91.0 — no false
"done". The `docker inspect -f *` grant is unchanged (`*` spans the extended template, exit-0 confirmed).
See `felhom-agent/REPORT.md` + CHANGELOG v0.47.0.

**EXPECTED (not defects):** the capstone leaf regeneration → pin mismatch (the prevention fix was
never shipped — regeneration is expected; **detection** is the pass criterion, and it passed). The §4
host-side "doubling" persisting until a clean host reboot (expected; `make-private` is granted, applies
next fresh boot).

---

## Coverage gaps (couldn't test safely / deferred — with rationale)

- **Host reboot** (§C) — would confirm the parent self-corrects to its own peer group; needs a
  maintenance window on the **non-expendable** host. Deferred.
- **Full `felhom-host-install.sh` reinstall** (§K) — reinstalls the agent **on felhom-pve** (non-
  expendable); the **equivalent** leaf-regeneration was exercised instead (same incident class, same
  detection, recoverable). The full reinstall is the operator's call in a window.
- **401 / DNS / construction live alert** (§F) — inducing these needs a controller-config change +
  restart, which **reseeds** the channel state (first-obs seeds silently, no alert). Classification is
  unit-proven; the dashboard would show the state, but the up→down **notification** for these isn't
  cleanly inducible live without an agent-side trick (pin/unreachable have one; 401 would need a
  token-store edit). Deferred as lower-value (unit-covered).
- **Capability live scratch-drop → hub alert** (§D) — the lxc-info-degraded path + drive flap was the
  **live incident** the capability task already diagnosed/fixed; the hub `HostCapabilityChecker`
  transition is unit-proven. A live drop requires editing the certified sudoers (sudoers can't
  "override-to-deny" via a drop-in cleanly) — deferred to avoid mutating the freshly-certified
  allowlist for a unit-covered path.
- **Exhaustive flap/mid-write/3rd-drive loops** (§C/§G adversarial) — referenced via the prior chaos
  campaigns (#1–#3, all FIXED); not re-run.

---

## F2 CLOSED (2026-06-29) + prevention shipped

The testrun surfaced **F2**: a channel/host failure present at startup/reseed alerted only on a live
transition, so a born-down (e.g. a controller booting into pin_mismatch) was dashboard-only, no
operator email. **Closed** — controller **v0.91.0** (`alerted`-flag rework) + hub **v0.21.0** (checkers
seed only healthy; born-degraded/stale emits on first Check, cooldown dedups). Live-validated: a
controller restarted into a regenerated-leaf channel logged `[channel] agent channel DOWN
(unseeded->down:pin_mismatch)` and **pushed `agent_channel_pin_mismatch`** (~60 s) — the capstone's
missing half (detection at startup). **Prevention** for the original incident also shipped: agent
**v0.46.0** loud-WARNs a regenerated leaf (`leaf REGENERATED — pins now INVALID`), and
`felhom-host-install.sh` gained `--preserve-state-from` (keeps the leaf/pin stable across a reinstall)
+ a populated-host guard (refuse to regenerate the leaf on a host with guests). The §K prevention
recommendation is now implemented; the full live reinstall is the only piece awaiting a supervised run.
(F1 swap-rollback `RestartCount` hardening remains backlog.)

## Coverage gaps — NOW RESOLVED (supervised follow-up, 2026-06-29)

Baselines: agent v0.48.0, controller v0.91.0, hub v0.22.1; leaf==pin `60b5974d…`; caps 45/45; both
drives bound; channel up.

### Phase 1 — capability→hub operator alert, LIVE — **PASS** (the path that never fired in the field)
Induced a minimal-blast-radius degrade: a scratch deny drop-in (`/etc/sudoers.d/zz-felhom-testdeny`,
`!…pct exec [0-9]* -- tee /etc/felhom-controller-image`, `visudo`-clean) + agent restart.
- Agent stayed **up** (serve-degraded): `capabilities self-check ok=44 total=45 degraded=1` +
  `capability DEGRADED … controllerswap-write … critical=true reason="sudo policy denied"`.
- The box kept working — both drives `bound_under_parent=True` throughout (lxc-info still granted).
- **Hub fired the alert (~24s):** `Host capability: demo-felhom-01 ok → degraded
  (agent_capability_degraded)`; the saved event message is **English** and names the cap+feature:
  *"agent privileged capability degraded — controllerswap-write (impairs: controller-swap / managed
  auto-update)"* (severity warning).
- Recover: removed the scratch deny + `visudo -cf /etc/sudoers` OK + restart → `ok=45/45` →
  `Host capability: demo-felhom-01 degraded → ok (agent_capability_recovered)`.
- **Note (not a defect):** a literal operator *email* was **not** dispatched — the demo hub has no
  operator email address configured (no "Operator email sent/failed" log → `processOperator` returns
  before the send). In prod with the operator address set, the same path sends via FormatOperatorEmail +
  Resend (RESEND key IS injected). The alert path itself (probe→report→checker→event+dispatch) is proven.

### Phase 2 — host-reboot "doubling" self-correction — **2 FINDINGS** (drive recovery PASS)
Rebooted felhom-pve (uptime was 5d, no boot since the `make-private` grant). After boot:
- **PASS — drive recovery + no false page:** the boot ran `felhom-shared-parent.service`; both drives
  re-propagated into the guest (`bound_under_parent=True`, `master:` tags alive), nextcloud + controller
  recovered healthy, leaf==pin MATCH `60b5974d…`, caps 45/45. **Zero** spurious alerts on the clean boot
  (drive-ABSENT 0, channel DOWN 0, hub capability/leaf/stale 0) — confirms F2 first-obs-seed doesn't
  false-page a normal reboot.
- **FINDING F2-a (doubling did NOT self-correct):** the host parent `/mnt/felhom-drives` is **still
  `shared:1`** (== root's peer group) and the drives are **still doubled** (each under parent 149 AND
  root 33) — identical to the pre-reboot state. **The §4 hypothesis is falsified:** the boot script
  `felhom-shared-parent.sh` runs as **root** (systemd), so it never needed the sudoers `make-private`
  grant — yet its `make-private`+`make-shared` sequence does not isolate the self-bind from root's
  `shared:1`. Harmless functionally (drives work), but the doubling persists across a clean boot.
  Severity: low (cosmetic/correctness; the agent's `AttachDrive` already converges binds). **Repro:**
  reboot the host, then `grep ' /mnt/felhom-drives ' /proc/self/mountinfo` → `shared:1`. Root-cause not
  pursued (won't churn the live parent — would orphan the guest slave). A fix should assert the parent's
  group ≠ root's after the boot script and investigate the make-private timing/semantics for a
  subdir-of-root self-bind.
- **FINDING F2-b (reboot during backup → stuck guest):** a **vzdump backup was in flight** at reboot
  (snapshot @ 23:43:45); the reboot interrupted the `snapshot-delete` cleanup, leaving a **stale
  `lock: snapshot-delete`** + a dangling `vzdump` snapshot. Guest 9201 therefore **failed to autostart**
  (`onboot:1` can't start a locked CT: *"Starting CT 9201 failed: CT is locked (snapshot-delete)"*).
  Recovered with `pct unlock 9201` + `pct start` + `pct delsnapshot 9201 vzdump`. The "power-loss/reboot
  during a backup" edge — a host reboot should check for an in-flight backup, or boot-time recovery
  should clear a stale vzdump snapshot lock + auto-start. Severity: medium (an unattended host reboot
  during the daily backup window would leave the customer box down until manual unlock).

### Phase 3 — `host_leaf_changed` LIVE on the wired hub (v0.22.1) — **PASS (referenced)**
Already live-proven against the wired v0.22.1 in the leaf-fp task: a leaf regen produced the controller
`agent_channel_pin_mismatch` AND (on restore) the hub `host_leaf_changed`
(`82078fab…→60b5974d…`) — two **independent** detections of the same re-key (controller-local +
hub-proactive). Not re-run here (would require another regen + R1). See
`felhom.eu/REPORT.md` (HostLeafChecker) + the §"Self-health arc COMPLETE" note in the channel-health
spike doc.

### Coverage-gaps verdict
All three deferred gaps now exercised: Phase 1 fired live (event captured; email is operator-config-
gated, not a code defect); Phase 2 drive-recovery PASS but surfaced **two findings** (doubling NOT
self-corrected — §4 hypothesis falsified; reboot-during-backup stale lock); Phase 3 referenced. Scratch
artifacts removed (`zz-felhom-testdeny`; the dangling vzdump snapshot); `/etc/sudoers` re-validated.

### Final parity (post follow-up) — known-good
agent v0.48.0 / controller v0.91.0 / hub v0.22.1; leaf==pin `60b5974d…`; caps 45/45; both drives
`bound_under_parent=True`; channel up; nextcloud + controller healthy. (Persistent-but-harmless: the
§4 host-side doubling — finding F2-a.)

## Findings closed — operator email + F2-a + F2-b (2026-06-30, agent v0.49.0 + hub config)

The two Phase-2 findings (F2-a, F2-b) and the Phase-1 operator-email gap are now **fixed**. Diagnose-
before-fix on both risky parts; both root causes confirmed live on `felhom-pve` before any code change.

### Part A — operator email delivery (the previously-unproven hop) — **PROVEN end-to-end**
The Phase-1 "Note (not a defect)" gap is closed. Root cause was `hub-config` lacking
`operator_email`/`operator_enabled` (so `Dispatcher.processOperator` returned before the send). Fix =
config-only (`manifests/hub.yaml` ConfigMap: `operator_email: admin@felhom.eu` + `operator_enabled: true`),
hub pod restarted to reload; the mounted config now shows both (Resend key NOT printed).
- **Operator path:** scratch-denied the new `stalelock-unlock` grant → agent `degraded=1` →
  `[INFO] Host capability: demo-felhom-01 ok → degraded (agent_capability_degraded)` →
  **`[INFO] Operator email sent for demo-felhom/agent_capability_degraded`** (this line prints ONLY after
  Resend returns 2xx — it never appeared before, because the path was gated off). Restored the grant →
  `agent_capability_recovered`.
- **Customer path:** `POST /api/v1/notify` (`event_type:test`) → `{"sent":true}` →
  `[INFO] Notification email sent to nagyfenyvesi.viktor@gmail.com for demo-felhom/test` (same `sendEmail`
  → Resend). Both channels of the dispatcher now proven to the Resend send.

### Part B — reboot-during-backup stale-lock recovery (F2-b) — **FIXED + live-validated**
`agent v0.49.0`: at startup, `Server.RecoverStaleLockedGuests` clears a stale vzdump lock — `pct unlock`
→ delete the dangling `vzdump` snapshot → start iff `onboot` and not already running — **only when no
vzdump is genuinely in-flight** (the invariant; fail-safe otherwise). New narrow grant `FELHOM_STALELOCK
= pct unlock [0-9]*` + Critical capability `stalelock-unlock` (caps 45→46, all green). Two real issues
surfaced by live validation and fixed: PVE 9.x rejects `?running=1` (→ `?source=active`); the LXC-start
`WARNINGS: 1` nesting advisory false-failed the start (→ `AllowWarnings`).
- **Live (B.3):** induced a `snapshot-delete` lock on scratch guest 9999 (onboot:1, stopped) → agent
  restart → `stale-lock: clearing a stale backup lock` → `started CT after clearing the stale lock
  (onboot)` → lock CLEARED, guest running. **Invariant guard proven against a REAL backup:** a genuine
  agent-scheduled vzdump of 9201 was in-flight during the same restart → `stale-lock: a vzdump backup is
  genuinely in-flight — leaving the lock (NOT stale)` → 9201's lock untouched; the real backup completed
  and self-cleared. (Note: PVE reserves the snapshot name `vzdump`, so a dangling one can't be hand-made
  — the delsnapshot-against-a-real-snapshot path is unit-tested and live-fires in the §D real reboot.)

### Part C — host-side doubling (F2-a) — **root cause FOUND; fix shipped; red-proof PASSED (§D)**
Reproduced the exact pathology in a scratch dir at `/mnt` (where the live parent lives): the **stale**
sequence (`bind + make-shared`, no `make-private`) leaves the self-bind in root's `shared:1` → sub-bind
count **2** (doubling); the **correct** sequence (`bind → make-private → make-shared`) gives an own group
(`shared:525`) → count **1**. The live boot script was the **pre-v0.36.6** body (no `make-private`).
**Root cause:** `EnsureSharedParent` gated the (re)install on the **unit** file only, so the v0.36.6
*script-only* fix never deployed to a host whose unit was current. **Fix (`agent v0.49.0`):** the new
`sharedParentInstallStale` compares **both** the script and the unit. Boot-time-only — it rewrites the
on-disk script; it does **not** churn the live mount (still guarded on `!isHostMountpoint`).
- **Deploy fix proven:** on the v0.49.0 startup the corrected script (with `make-private`) was installed
  to `/usr/local/sbin/felhom-shared-parent.sh` (sudo log shows the install). The live mount was left
  untouched until the reboot (`/mnt/felhom-drives` still `shared:1`, doubled — by design).
- **RED-PROOF PASSED (§D, two reboots):** `/mnt/felhom-drives` came up as **`shared:94`** — its OWN peer
  group, NOT root's `shared:1` — and each drive has **exactly 1 host bind** (doubling GONE; was 2 each).
  The guest still sees both drives (`bound_under_parent=true`; apps all healthy). Consistent across both
  reboots. (Separate, pre-existing, harmless observation: the GUEST shows 2 mounts per drive — two parent
  mounts in the guest namespace, same `/dev/sdX1 /felhom-data` source, `master:` slaved correctly; NOT a
  regression of this fix, which only corrected the host parent's peer group. Worth a future look.)

### §D — supervised host reboot — DONE (PASS)
Run in a supervised window (user-authorized). Two reboots:
1. **Reboot mid-backup (caught the `backup`/tar phase):** PVE auto-cleared the `backup` lock and started
   9201 normally (no stuck symptom in that phase); the agent recovery saw no lock → correctly skipped.
   The C red-proof passed (parent `shared:94`, no doubling). A real `vzdump` snapshot remained.
2. **Reboot with the EXACT F2-b state** (induced `lock: snapshot-delete` + the real dangling `vzdump`
   snapshot from reboot 1): pve-guests **FAILED** to start the locked CT — *"starting CT 9201: CT is
   locked (snapshot-delete)"* → *"Starting CT 9201 failed"* — the genuine stuck symptom. The agent then
   auto-recovered: `clearing a stale backup lock` (lock=snapshot-delete) → **`removed dangling vzdump
   snapshot`** (the real delsnapshot path) → `started CT after clearing the stale lock (onboot)`. 9201
   running, lock cleared, snapshot deleted, no error (the `?source=active` + `AllowWarnings` fixes held).
**Zero spurious operator pages** across both reboots (no host_down/stale/capability/leaf events — only the
controller's benign notification-prefs re-register). The interrupted-backup temp `.dat` was cleaned.

### Final parity (post-fix, post-§D) — known-good
agent **v0.49.0** / controller v0.91.0 / hub v0.22.1 (+ operator-email hub-config); leaf==pin `60b5974d…`;
caps **46/46**; `/mnt/felhom-drives` is its **own peer group `shared:94`** with **no doubling** (1 bind
each); both drives bound + 12 healthy containers; scratch guest 9999 back to baseline; the scratch deny
removed + `/etc/sudoers` valid; 9201 lock cleared + dangling snapshot deleted. **Both findings F2-a and
F2-b are now CLOSED and live-validated.** No secrets recorded (tokens read out-of-band, never printed;
leaf fps + the operator email are not secret).