test campaign #2: Phase 3 C4-C6 (C5 app.yaml graceful, C6 drive-eject recover + recreate-noise finding, C4 skip)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

TEST-REPORT.md

@@ -51,6 +51,10 @@ Agent path works (v0.74.0), so these ran through the real controller→agent pip
 | C2 | `docker kill felhom-controller` → (expected bootstrap self-heal) → recover → verify | **PASS (w/ 🔴 finding)** | killed → **stayed Exited 48s, NO auto-recovery** (docker `unless-stopped` didn't fire; bootstrap.service is oneshot/exited). Data plane unaffected (apps+cloudflared+traefik up). Recovered via `systemctl restart felhom-controller-bootstrap.service` → running/healthy; UI 200; 26 containers/0 unhealthy. **Finding: a crashed controller has NO automatic recovery.** |
 | C3 | stop `felhom-agent` → controller degrades gracefully → restart agent → immediate success | **PASS** | agent down → `/api/disks` **fast ~98ms clean error** ("connection refused"), no hang/crash; agent restarted → `/api/disks` OK on the **first** try (4/4), no stale-conn errors; idle sockets bounded at 1 — the v0.74.0 client's restarted-agent drain works |
 
+| C4 | fill a drive near-full → disk warning → clear | **SKIPPED(impractical)** | drives are 118 GB / 916 GB and `/var/lib/docker` 252 GB; reaching the 90% warning / the deploy gate's `max(5GB,10%)`≈25 GB-free needs 100+ GB of fill — unsafe/slow unattended. Gate logic `GetDockerVolumeHeadroom` (hard-refuse deploy below the reserve) is present; monitoring reports per-drive usage. |
+| C5 | rename `navidrome/app.yaml` away → rescan → restore | **PASS** | rescan after removal: controller stays `running/healthy` (no crash), container keeps running (data plane), card → `state=running deployed=false` (graceful orphan). Restored + rescan → `state=running deployed=true` |
+| C6 | disconnect felhom-flash (9 apps) → recover | **PASS (recovered) + finding** | disconnect stopped all 9 apps + unmounted (`UNMOUNTED`, komga `stopped`); reconnect → felhom-flash remounted `/dev/sdc1`, all 9 restarted → **26 running, 0 unhealthy** after start_period. **Finding:** during the drive-absent window the controller's userdata-belt/recreate tried to bring drive-backed apps up while the mount was gone → `mkdir …/userdata: permission denied` + transient `Created` states (rather than a clean held "missing-storage" stop). Reconnect cleared it; but the disconnect→recreate interaction is noisy — worth a guard ("don't recreate/ensure-userdata when the drive is absent"). |
+
 **🔴 Finding (C1/C2) — docker restart policy not honored:** containers with `restart: unless-stopped` do **not** auto-restart after `docker kill` (navidrome `RestartCount=0` over two trials; dockerd logged `stopping restart-manager`; Docker 29.5.3, Live Restore off). The controller is not stopping them (no reconcile-stop in logs/source). Impact: **no docker-level self-heal for crashed app or controller containers** — the controller detects the stopped state and a UI/operator restart recovers apps, but a crashed *controller* itself has no automatic recovery (oneshot bootstrap + dead restart policy). **Needs supervised investigation** (docker daemon restart-manager behaviour) + a controller-recovery watchdog (e.g. bootstrap as a `Restart=always` monitor, or a systemd unit watching the container).
 ## Phase 4 — Backup/restore edge cases — _pending_
 ## Phase 5 — Host reboot (optional) — _pending_