docs(tests): test campaign #2 findings (chaos/edge-cases, demo 9201)

Storage lifecycle (unblocked by v0.74.0), break-and-recover C1-C8, backup edge cases.
Headline finding: docker restart-policy not honored -> a crashed felhom-controller has
no automatic recovery (oneshot bootstrap + dead restart policy). Plus disconnect-recreate
race, agent loopback whitelist, carried-forward #1 gaps. Demo restored to baseline.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

documentation/tests/test-campaign-2-2026-06-22-findings.md

@@ -0,0 +1,88 @@
+# Test campaign #2 — broader apps, edge cases, break-and-recover — 2026-06-22 (demo 9201)
+
+> **Status: empirical findings from an unattended chaos/edge-case campaign (2026-06-22).** Follow-on to
+> `unattended-test-campaign-2026-06-22-findings.md`. The campaign-#1 blocker (controller→agent leak) is
+> fixed (ctrl **v0.74.0**), so the **storage lifecycle that was BLOCKED in #1 is now tested**, plus
+> deliberate break-and-recover chaos against the architectural contracts. Demo restored to baseline
+> (25 containers, 0 unhealthy) and healthy at the end. Per-check raw evidence: `felhom-controller/TEST-REPORT.md`.
+
+## Topology
+ctrl **v0.74.0** (guest 9201), agent v0.39.0 (felhom-pve), hub on DooPlex; PBS `felhom-pbs`. Drives
+felhom-flash (uuid:81a26531) + felhom-usb (uuid:da9e7089, empty). DR source / master net:
+PBS `felhom-pbs:backup/ct/9201/2026-06-22T16:13:44Z` (verified). UI driven via an in-guest forwarder
+(on-LAN traefik :443 still 404s — campaign-#1 finding #5).
+
+## What passed (high-value)
+- **Leak fix holds (Phase 0):** `/api/disks` + `/api/host-metrics` work; controller→agent idle sockets
+  to `:8443` **flat at 2** across a 60-call burst (v0.74.0).
+- **Storage lifecycle (was BLOCKED in #1):** scan/health/durable-id; label edit→revert; **disconnect↔
+  reconnect** (4-state enrolled→disconnected→enrolled); **decommission↔re-enroll** (non-destructive —
+  fs uuid preserved, never formatted). All through the real controller→agent pipeline.
+- **Chaos crash-safety:**
+  - **C7 (the headline):** killed the controller **mid-quiesce** — the quiesce marker (written upfront,
+    15 stopped_stacks) **persisted across the crash**; controller restart ran `Recover()`
+    (`[quiesce] crash recovery … restarting them`) → all stacks resumed, marker cleared, none stranded.
+  - **C3:** stopped the agent → controller degraded **gracefully** (fast ~98ms clean "connection
+    refused", no hang) → agent restart → calls succeeded on the **first** try (the v0.74.0 client
+    drains the dead pooled conn), sockets bounded.
+  - **C8:** reboot during a backup op → 26 containers back, drives remounted **by UUID**, tunnel
+    reconnected, romm DB intact (no corruption).
+  - C1 (kill app→restart), C5 (remove app.yaml→graceful orphan→restore), C6 (eject drive-with-apps→
+    reconnect→recover) all recovered + verified.
+- **Single-flight** is robust — rejected concurrent restore (mutex), concurrent whole-guest backup, and
+  Tier-2-vs-DB-dump.
+- **Backup edge cases:** concurrent restore blocked ("backup or restore already in progress");
+  backup with a stopped stack handled (no error).
+
+## Findings (new)
+
+### 🔴 1. docker restart policy not honored → no auto-recovery for a crashed controller
+`restart: unless-stopped` containers do **not** auto-restart after `docker kill` (navidrome
+`RestartCount=0` over two trials; dockerd logged `stopping restart-manager`; Docker 29.5.3, Live
+Restore off). The controller does not reconcile-stop them (verified in logs/source). Consequences:
+- **Apps:** a crashed app stays down; the controller **detects** `stopped` and a UI/operator restart
+  recovers it (C1) — but there is **no docker-level self-heal**.
+- **🔴 Controller itself (C2):** killed → stayed down 48 s, **no automatic recovery** — docker policy
+  dead + `felhom-controller-bootstrap.service` is **oneshot** (already exited). Recovery required a
+  manual `systemctl restart felhom-controller-bootstrap.service`. Data plane (apps + tunnel) kept
+  running throughout (control/data separation confirmed), but the **management plane has no
+  auto-recovery** if the controller crashes (OOM/panic/kill) — it would stay down until a guest reboot
+  or manual intervention.
+- **Fix direction:** add a controller watchdog — e.g. make the bootstrap a `Restart=always` monitor, or
+  a small systemd unit/timer that recreates `felhom-controller` if absent; and investigate why the
+  docker daemon's restart-manager is being cancelled (Docker 29.5.3 behaviour / cgroup).
+
+### 2. Disconnect→recreate race on a drive-with-apps (C6)
+Disconnecting felhom-flash (9 apps) recovered cleanly on reconnect, but during the drive-absent window
+the controller's **userdata-belt/recreate** tried to bring drive-backed apps up while the mount was
+gone → `mkdir …/userdata: permission denied` + transient `Created` states, instead of a clean held
+"missing-storage" stop (B2b). *Fix:* gate ensure-userdata/recreate on "the drive is a live mountpoint."
+
+### 3. FileBrowser skeleton-ensure on an unmounted path (cosmetic)
+disconnect/decommission log `mkdir …/userdata: permission denied` from the FileBrowser sync against the
+just-unmounted path. Harmless but noisy; gate on live-mountpoint.
+
+### 4. Agent device whitelist refuses loopback (good security)
+`/api/storage/init` on `/dev/loop0` → agent `refusing to operate on non-whitelisted block device` — a
+deliberate safety control. So a loopback can't be the destructive-disk scratch target the runbook
+assumed; destructive format/migrate need a real spare disk or a supervised run.
+
+### Carried from campaign #1 (still open)
+- `GET /api/backup/snapshots` → 404 (no handler) — restore-UI snapshot dropdown can't populate.
+- Restore DB import is **additive** — a planted marker table survived the restore (no `--clean`).
+
+## Deferred / needs supervised or implementation
+- **Controller-recovery watchdog** + docker restart-policy investigation (finding #1) — highest priority.
+- Destructive storage **format/migrate** on a real spare disk (loopback blocked).
+- **Host reboot** resilience (Phase 5 — skipped unattended: no physical recovery if the N100 doesn't return).
+- Disconnect-recreate gate (#2); FileBrowser unmounted-skeleton gate (#3).
+- `/api/backup/snapshots` handler + UI snapshot selection; restore `--clean` (campaign-#1 gaps).
+
+## Code changes
+**None this campaign** — no broken apps surfaced that needed a catalog fix (navidrome deployed clean).
+All findings are controller/agent-side behaviours logged for supervised follow-up, not changed here.
+
+## Cleanup / after-state
+9201 running, **25 containers, 0 unhealthy**; agent active; controller + cloudflared healthy; both
+drives re-enrolled + mounted. Scratch loopback torn down; navidrome test app removed; all timed guards
+killed; forwarder + temp files removed. No throwaway guest created this campaign.