diff --git a/TEST-REPORT.md b/TEST-REPORT.md index d9667e2..b2589d6 100644 --- a/TEST-REPORT.md +++ b/TEST-REPORT.md @@ -51,6 +51,10 @@ Agent path works (v0.74.0), so these ran through the real controllerβ†’agent pip | C2 | `docker kill felhom-controller` β†’ (expected bootstrap self-heal) β†’ recover β†’ verify | **PASS (w/ πŸ”΄ finding)** | killed β†’ **stayed Exited 48s, NO auto-recovery** (docker `unless-stopped` didn't fire; bootstrap.service is oneshot/exited). Data plane unaffected (apps+cloudflared+traefik up). Recovered via `systemctl restart felhom-controller-bootstrap.service` β†’ running/healthy; UI 200; 26 containers/0 unhealthy. **Finding: a crashed controller has NO automatic recovery.** | | C3 | stop `felhom-agent` β†’ controller degrades gracefully β†’ restart agent β†’ immediate success | **PASS** | agent down β†’ `/api/disks` **fast ~98ms clean error** ("connection refused"), no hang/crash; agent restarted β†’ `/api/disks` OK on the **first** try (4/4), no stale-conn errors; idle sockets bounded at 1 β€” the v0.74.0 client's restarted-agent drain works | +| C4 | fill a drive near-full β†’ disk warning β†’ clear | **SKIPPED(impractical)** | drives are 118 GB / 916 GB and `/var/lib/docker` 252 GB; reaching the 90% warning / the deploy gate's `max(5GB,10%)`β‰ˆ25 GB-free needs 100+ GB of fill β€” unsafe/slow unattended. Gate logic `GetDockerVolumeHeadroom` (hard-refuse deploy below the reserve) is present; monitoring reports per-drive usage. | +| C5 | rename `navidrome/app.yaml` away β†’ rescan β†’ restore | **PASS** | rescan after removal: controller stays `running/healthy` (no crash), container keeps running (data plane), card β†’ `state=running deployed=false` (graceful orphan). Restored + rescan β†’ `state=running deployed=true` | +| C6 | disconnect felhom-flash (9 apps) β†’ recover | **PASS (recovered) + finding** | disconnect stopped all 9 apps + unmounted (`UNMOUNTED`, komga `stopped`); reconnect β†’ felhom-flash remounted `/dev/sdc1`, all 9 restarted β†’ **26 running, 0 unhealthy** after start_period. **Finding:** during the drive-absent window the controller's userdata-belt/recreate tried to bring drive-backed apps up while the mount was gone β†’ `mkdir …/userdata: permission denied` + transient `Created` states (rather than a clean held "missing-storage" stop). Reconnect cleared it; but the disconnectβ†’recreate interaction is noisy β€” worth a guard ("don't recreate/ensure-userdata when the drive is absent"). | + **πŸ”΄ Finding (C1/C2) β€” docker restart policy not honored:** containers with `restart: unless-stopped` do **not** auto-restart after `docker kill` (navidrome `RestartCount=0` over two trials; dockerd logged `stopping restart-manager`; Docker 29.5.3, Live Restore off). The controller is not stopping them (no reconcile-stop in logs/source). Impact: **no docker-level self-heal for crashed app or controller containers** β€” the controller detects the stopped state and a UI/operator restart recovers apps, but a crashed *controller* itself has no automatic recovery (oneshot bootstrap + dead restart policy). **Needs supervised investigation** (docker daemon restart-manager behaviour) + a controller-recovery watchdog (e.g. bootstrap as a `Restart=always` monitor, or a systemd unit watching the container). ## Phase 4 β€” Backup/restore edge cases β€” _pending_ ## Phase 5 β€” Host reboot (optional) β€” _pending_