diff --git a/TEST-REPORT.md b/TEST-REPORT.md index ea6691e..d9667e2 100644 --- a/TEST-REPORT.md +++ b/TEST-REPORT.md @@ -43,7 +43,15 @@ Agent path works (v0.74.0), so these ran through the real controller→agent pip **Findings:** 1. **Agent device whitelist** refuses non-whitelisted block devices (loopback) — good security, but means a loopback can't serve as the destructive-disk scratch target the runbook assumed. Destructive format/migrate now need either a real spare disk or a supervised run. 2. **FileBrowser sync** logs `mkdir …/userdata: permission denied` when ensuring the skeleton on a just-unmounted/decommissioned path (recurs on disconnect + decommission). Cosmetic (the drive is intentionally unmounted), but noisy — could gate the skeleton-ensure on "is the path a live mountpoint." -## Phase 3 — Break-and-recover chaos (C1–C8) — _pending_ +## Phase 3 — Break-and-recover chaos (C1–C8) — _in progress_ + +| # | Break → Detect → Recover → Verify | Result | Evidence | +|---|---|---|---| +| C1 | `docker kill navidrome` → controller detects → restart via UI → healthy | **PASS** | killed → `Exited(137)`; controller state `stopped`; `POST /api/stacks/navidrome/restart` → `running/healthy`, state `running` | +| C2 | `docker kill felhom-controller` → (expected bootstrap self-heal) → recover → verify | **PASS (w/ 🔴 finding)** | killed → **stayed Exited 48s, NO auto-recovery** (docker `unless-stopped` didn't fire; bootstrap.service is oneshot/exited). Data plane unaffected (apps+cloudflared+traefik up). Recovered via `systemctl restart felhom-controller-bootstrap.service` → running/healthy; UI 200; 26 containers/0 unhealthy. **Finding: a crashed controller has NO automatic recovery.** | +| C3 | stop `felhom-agent` → controller degrades gracefully → restart agent → immediate success | **PASS** | agent down → `/api/disks` **fast ~98ms clean error** ("connection refused"), no hang/crash; agent restarted → `/api/disks` OK on the **first** try (4/4), no stale-conn errors; idle sockets bounded at 1 — the v0.74.0 client's restarted-agent drain works | + +**🔴 Finding (C1/C2) — docker restart policy not honored:** containers with `restart: unless-stopped` do **not** auto-restart after `docker kill` (navidrome `RestartCount=0` over two trials; dockerd logged `stopping restart-manager`; Docker 29.5.3, Live Restore off). The controller is not stopping them (no reconcile-stop in logs/source). Impact: **no docker-level self-heal for crashed app or controller containers** — the controller detects the stopped state and a UI/operator restart recovers apps, but a crashed *controller* itself has no automatic recovery (oneshot bootstrap + dead restart policy). **Needs supervised investigation** (docker daemon restart-manager behaviour) + a controller-recovery watchdog (e.g. bootstrap as a `Restart=always` monitor, or a systemd unit watching the container). ## Phase 4 — Backup/restore edge cases — _pending_ ## Phase 5 — Host reboot (optional) — _pending_