test campaign #2: Phase 5 skip (host reboot) + final matrix/findings/cleanup

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

TEST-REPORT.md

@@ -70,9 +70,70 @@ Agent path works (v0.74.0), so these ran through the real controller→agent pip
 | Low-space backup behavior | SKIPPED | tied to C4 (filling 100+ GB impractical/unsafe unattended) |
 
 Bonus across the phase: **single-flight** is robust — observed rejecting concurrent restore (mutex), concurrent whole-guest backup (C7), and Tier-2 vs DB-dump.
-## Phase 5 — Host reboot (optional) — _pending_
+## Phase 5 — Host reboot (optional) — **SKIPPED (unattended risk)**
+A host reboot of the N100 is the highest-risk action unattended — if it doesn't POST/return (BIOS
+AC-recovery etc.) the demo is down until physical access, which CC doesn't have. Deferred to supervised
+(consistent with campaign #1). Guest-reboot recovery is proven (C8 + campaign #1).
 
 ---
-## Feature matrix — _filled at end_
-## Cleanup confirmation — _filled at end_
-## Deferred / needs supervised — _filled at end_
+## Feature matrix
+| Area | Verdict |
+|---|---|
+| PBS backup + verify + leak-fix gate | PASS |
+| App deploy (HDD-bound navidrome) + redeploy race | PASS |
+| Storage: scan/health/durable-id | PASS |
+| Storage: label edit/revert | PASS |
+| Storage: disconnect↔reconnect (4-state) | PASS |
+| Storage: decommission↔re-enroll (non-destructive) | PASS |
+| Storage: destructive format on loopback | BLOCKED (agent device-whitelist) |
+| Chaos C1 kill app → restart | PASS |
+| Chaos C2 kill controller | PASS recovery / 🔴 no auto-heal |
+| Chaos C3 stop agent (graceful + drain) | PASS |
+| Chaos C5 remove app.yaml | PASS |
+| Chaos C6 eject drive w/ apps | PASS recovery / finding (recreate noise) |
+| Chaos C7 kill mid-quiesce → Recover | PASS (crash-safety proven) |
+| Chaos C8 reboot during op | PASS |
+| Backup: concurrent mutex | PASS |
+| Backup: stopped-stack | PASS |
+| Single-flight (backup/restore/migrate) | PASS |
+| `/api/backup/snapshots` | 404 (gap, carried from #1) |
+| restore DB import additive | confirmed (gap, carried from #1) |
+| C4 disk-full / low-space | SKIPPED (impractical) |
+| Phase 5 host reboot | SKIPPED (unattended risk) |
+
+## Findings (new this campaign)
+1. **🔴 docker restart policy not honored** — `unless-stopped` containers don't auto-restart after a
+   crash (`docker kill` → `RestartCount=0`, dockerd `stopping restart-manager`; Docker 29.5.3). **No
+   docker-level self-heal for crashed apps OR the controller.** The controller detects + a UI/operator
+   restart recovers apps; but a crashed **controller** has **no automatic recovery** (oneshot bootstrap
+   + dead restart policy) — manual `systemctl restart felhom-controller-bootstrap.service` needed.
+   *Recommend:* a controller watchdog (e.g. bootstrap `Restart=always` monitor or a systemd unit
+   watching the container) + investigate why the daemon's restart-manager is cancelled.
+2. **Disconnect-recreate race (C6)** — disconnecting a drive-with-apps triggers the controller's
+   userdata-belt/recreate to try bringing drive-backed apps up *while the drive is absent* →
+   `mkdir …/userdata: permission denied` + transient `Created` states, instead of a clean held
+   missing-storage stop. Reconnect recovers it. *Recommend:* gate ensure-userdata/recreate on
+   "drive is a live mountpoint."
+3. **FileBrowser skeleton on unmounted path** — disconnect/decommission log
+   `mkdir …/userdata: permission denied` from the FileBrowser sync on the just-unmounted path. Cosmetic
+   but noisy; gate on live-mountpoint.
+4. **Agent device whitelist** refuses loopback (`/dev/loop0`) — good security, but means a loopback
+   can't be the destructive-disk scratch target; destructive format/migrate need a real spare disk or
+   a supervised run.
+
+Gaps carried from campaign #1 (still open): `/api/backup/snapshots` 404; restore DB import additive.
+
+## Cleanup confirmation
+- No throwaway guest created this campaign (DR was campaign #1); scratch loopback torn down (`losetup -d` + rm).
+- navidrome (test app) removed → back to **25 containers** baseline; uptime-kuma restarted healthy.
+- All chaos breaks recovered + verified (C1–C8); agent **active**; felhom-flash + felhom-usb **re-enrolled** + mounted.
+- All timed guards killed (0 `sleep` guards); test forwarder + host temp files removed; **0 loopbacks**.
+- Notifications untouched this campaign (none configured); no password changes.
+- **9201 running, 25 containers, 0 unhealthy; controller + cloudflared healthy; agent active.**
+
+## Deferred / needs supervised
+- **Controller-recovery watchdog** + docker restart-policy investigation (finding #1) — highest priority.
+- Destructive storage **format/migrate** on a real spare disk (loopback blocked) — supervised.
+- **Host reboot** resilience (Phase 5).
+- Disconnect-recreate gate (finding #2); FileBrowser unmounted-skeleton gate (finding #3).
+- `/api/backup/snapshots` handler + UI snapshot selection; restore `--clean` option (campaign-#1 gaps).