test campaign #2: Phase 3 C7-C8 (quiesce crash-safety + reboot-during-op PASS); phase complete

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

TEST-REPORT.md

@@ -43,7 +43,7 @@ Agent path works (v0.74.0), so these ran through the real controller→agent pip
 **Findings:**
 1. **Agent device whitelist** refuses non-whitelisted block devices (loopback) — good security, but means a loopback can't serve as the destructive-disk scratch target the runbook assumed. Destructive format/migrate now need either a real spare disk or a supervised run.
 2. **FileBrowser sync** logs `mkdir …/userdata: permission denied` when ensuring the skeleton on a just-unmounted/decommissioned path (recurs on disconnect + decommission). Cosmetic (the drive is intentionally unmounted), but noisy — could gate the skeleton-ensure on "is the path a live mountpoint."
-## Phase 3 — Break-and-recover chaos (C1–C8) — _in progress_
+## Phase 3 — Break-and-recover chaos (C1–C8) — **PASS (7/8 run, C4 skipped; 2 findings)**
 
 | # | Break → Detect → Recover → Verify | Result | Evidence |
 |---|---|---|---|
@@ -55,6 +55,9 @@ Agent path works (v0.74.0), so these ran through the real controller→agent pip
 | C5 | rename `navidrome/app.yaml` away → rescan → restore | **PASS** | rescan after removal: controller stays `running/healthy` (no crash), container keeps running (data plane), card → `state=running deployed=false` (graceful orphan). Restored + rescan → `state=running deployed=true` |
 | C6 | disconnect felhom-flash (9 apps) → recover | **PASS (recovered) + finding** | disconnect stopped all 9 apps + unmounted (`UNMOUNTED`, komga `stopped`); reconnect → felhom-flash remounted `/dev/sdc1`, all 9 restarted → **26 running, 0 unhealthy** after start_period. **Finding:** during the drive-absent window the controller's userdata-belt/recreate tried to bring drive-backed apps up while the mount was gone → `mkdir …/userdata: permission denied` + transient `Created` states (rather than a clean held "missing-storage" stop). Reconnect cleared it; but the disconnect→recreate interaction is noisy — worth a guard ("don't recreate/ensure-userdata when the drive is absent"). |
 
+| C7 | kill controller MID-quiesce → Recover resumes | **PASS** | triggered whole-guest backup → quiesce marker written **upfront** (`active:true`, 15 stopped_stacks) → `docker kill felhom-controller` mid-quiesce → **marker persisted across the crash**; controller restart logged `[quiesce] crash recovery: a quiesce was in progress (15 stack(s) stopped) — restarting them` → **26 running, 0 unhealthy, marker cleared**, no stack stranded. **Crash-safety contract proven.** Bonus: a concurrent trigger was rejected `mentés már folyamatban van` (single-flight works). |
+| C8 | guest reboot during a backup op | **PASS** | triggered DB-dump + `pct reboot 9201` mid-op → recovered to 26 containers/0 unhealthy; **drives remounted by UUID** (felhom-flash /dev/sdc1, felhom-usb /dev/sdb1, ext4); cloudflared+controller healthy; **romm DB intact (17 tables — no corruption)**; UI 200 |
+
 **🔴 Finding (C1/C2) — docker restart policy not honored:** containers with `restart: unless-stopped` do **not** auto-restart after `docker kill` (navidrome `RestartCount=0` over two trials; dockerd logged `stopping restart-manager`; Docker 29.5.3, Live Restore off). The controller is not stopping them (no reconcile-stop in logs/source). Impact: **no docker-level self-heal for crashed app or controller containers** — the controller detects the stopped state and a UI/operator restart recovers apps, but a crashed *controller* itself has no automatic recovery (oneshot bootstrap + dead restart policy). **Needs supervised investigation** (docker daemon restart-manager behaviour) + a controller-recovery watchdog (e.g. bootstrap as a `Restart=always` monitor, or a systemd unit watching the container).
 ## Phase 4 — Backup/restore edge cases — _pending_
 ## Phase 5 — Host reboot (optional) — _pending_