admin/felhom-controller
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test campaign #2: Phase 3 C1-C3 (kill app/controller/agent — chaos; docker restart-policy not honored finding)

Browse Source 
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
authentik Default Admin
2026-06-22 18:44:04 +02:00
parent d07848a786
commit 3ee26c575f
1 changed files with 9 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
TEST-REPORT.md
+9 -1
View File
@@ -43,7 +43,15 @@ Agent path works (v0.74.0), so these ran through the real controller→agent pip
**Findings:** **Findings:**
1. **Agent device whitelist** refuses non-whitelisted block devices (loopback) — good security, but means a loopback can't serve as the destructive-disk scratch target the runbook assumed. Destructive format/migrate now need either a real spare disk or a supervised run. 1. **Agent device whitelist** refuses non-whitelisted block devices (loopback) — good security, but means a loopback can't serve as the destructive-disk scratch target the runbook assumed. Destructive format/migrate now need either a real spare disk or a supervised run.
2. **FileBrowser sync** logs `mkdir …/userdata: permission denied` when ensuring the skeleton on a just-unmounted/decommissioned path (recurs on disconnect + decommission). Cosmetic (the drive is intentionally unmounted), but noisy — could gate the skeleton-ensure on "is the path a live mountpoint." 2. **FileBrowser sync** logs `mkdir …/userdata: permission denied` when ensuring the skeleton on a just-unmounted/decommissioned path (recurs on disconnect + decommission). Cosmetic (the drive is intentionally unmounted), but noisy — could gate the skeleton-ensure on "is the path a live mountpoint."
## Phase 3 — Break-and-recover chaos (C1–C8) — _pending_ ## Phase 3 — Break-and-recover chaos (C1–C8) — _in progress_
| # | Break → Detect → Recover → Verify | Result | Evidence |
|---|---|---|---|
| C1 | `docker kill navidrome` → controller detects → restart via UI → healthy | **PASS** | killed → `Exited(137)`; controller state `stopped`; `POST /api/stacks/navidrome/restart` → `running/healthy`, state `running` |
| C2 | `docker kill felhom-controller` → (expected bootstrap self-heal) → recover → verify | **PASS (w/ 🔴 finding)** | killed → **stayed Exited 48s, NO auto-recovery** (docker `unless-stopped` didn't fire; bootstrap.service is oneshot/exited). Data plane unaffected (apps+cloudflared+traefik up). Recovered via `systemctl restart felhom-controller-bootstrap.service` → running/healthy; UI 200; 26 containers/0 unhealthy. **Finding: a crashed controller has NO automatic recovery.** |
| C3 | stop `felhom-agent` → controller degrades gracefully → restart agent → immediate success | **PASS** | agent down → `/api/disks` **fast ~98ms clean error** ("connection refused"), no hang/crash; agent restarted → `/api/disks` OK on the **first** try (4/4), no stale-conn errors; idle sockets bounded at 1 — the v0.74.0 client's restarted-agent drain works |
**🔴 Finding (C1/C2) — docker restart policy not honored:** containers with `restart: unless-stopped` do **not** auto-restart after `docker kill` (navidrome `RestartCount=0` over two trials; dockerd logged `stopping restart-manager`; Docker 29.5.3, Live Restore off). The controller is not stopping them (no reconcile-stop in logs/source). Impact: **no docker-level self-heal for crashed app or controller containers** — the controller detects the stopped state and a UI/operator restart recovers apps, but a crashed *controller* itself has no automatic recovery (oneshot bootstrap + dead restart policy). **Needs supervised investigation** (docker daemon restart-manager behaviour) + a controller-recovery watchdog (e.g. bootstrap as a `Restart=always` monitor, or a systemd unit watching the container).
## Phase 4 — Backup/restore edge cases — _pending_ ## Phase 4 — Backup/restore edge cases — _pending_
## Phase 5 — Host reboot (optional) — _pending_ ## Phase 5 — Host reboot (optional) — _pending_