From 3ee26c575fd65375a747bb04874092e02f25e3d6 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 22 Jun 2026 18:44:04 +0200 Subject: [PATCH] =?UTF-8?q?test=20campaign=20#2:=20Phase=203=20C1-C3=20(ki?= =?UTF-8?q?ll=20app/controller/agent=20=E2=80=94=20chaos;=20docker=20resta?= =?UTF-8?q?rt-policy=20not=20honored=20finding)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Opus 4.8 (1M context) --- TEST-REPORT.md | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/TEST-REPORT.md b/TEST-REPORT.md index ea6691e..d9667e2 100644 --- a/TEST-REPORT.md +++ b/TEST-REPORT.md @@ -43,7 +43,15 @@ Agent path works (v0.74.0), so these ran through the real controller→agent pip **Findings:** 1. **Agent device whitelist** refuses non-whitelisted block devices (loopback) — good security, but means a loopback can't serve as the destructive-disk scratch target the runbook assumed. Destructive format/migrate now need either a real spare disk or a supervised run. 2. **FileBrowser sync** logs `mkdir …/userdata: permission denied` when ensuring the skeleton on a just-unmounted/decommissioned path (recurs on disconnect + decommission). Cosmetic (the drive is intentionally unmounted), but noisy — could gate the skeleton-ensure on "is the path a live mountpoint." -## Phase 3 — Break-and-recover chaos (C1–C8) — _pending_ +## Phase 3 — Break-and-recover chaos (C1–C8) — _in progress_ + +| # | Break → Detect → Recover → Verify | Result | Evidence | +|---|---|---|---| +| C1 | `docker kill navidrome` → controller detects → restart via UI → healthy | **PASS** | killed → `Exited(137)`; controller state `stopped`; `POST /api/stacks/navidrome/restart` → `running/healthy`, state `running` | +| C2 | `docker kill felhom-controller` → (expected bootstrap self-heal) → recover → verify | **PASS (w/ 🔴 finding)** | killed → **stayed Exited 48s, NO auto-recovery** (docker `unless-stopped` didn't fire; bootstrap.service is oneshot/exited). Data plane unaffected (apps+cloudflared+traefik up). Recovered via `systemctl restart felhom-controller-bootstrap.service` → running/healthy; UI 200; 26 containers/0 unhealthy. **Finding: a crashed controller has NO automatic recovery.** | +| C3 | stop `felhom-agent` → controller degrades gracefully → restart agent → immediate success | **PASS** | agent down → `/api/disks` **fast ~98ms clean error** ("connection refused"), no hang/crash; agent restarted → `/api/disks` OK on the **first** try (4/4), no stale-conn errors; idle sockets bounded at 1 — the v0.74.0 client's restarted-agent drain works | + +**🔴 Finding (C1/C2) — docker restart policy not honored:** containers with `restart: unless-stopped` do **not** auto-restart after `docker kill` (navidrome `RestartCount=0` over two trials; dockerd logged `stopping restart-manager`; Docker 29.5.3, Live Restore off). The controller is not stopping them (no reconcile-stop in logs/source). Impact: **no docker-level self-heal for crashed app or controller containers** — the controller detects the stopped state and a UI/operator restart recovers apps, but a crashed *controller* itself has no automatic recovery (oneshot bootstrap + dead restart policy). **Needs supervised investigation** (docker daemon restart-manager behaviour) + a controller-recovery watchdog (e.g. bootstrap as a `Restart=always` monitor, or a systemd unit watching the container). ## Phase 4 — Backup/restore edge cases — _pending_ ## Phase 5 — Host reboot (optional) — _pending_