From 58fe93034f50cd58340798f2ddebe361d56d1765 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 22 Jun 2026 19:10:51 +0200 Subject: [PATCH] docs(tests): test campaign #2 findings (chaos/edge-cases, demo 9201) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Storage lifecycle (unblocked by v0.74.0), break-and-recover C1-C8, backup edge cases. Headline finding: docker restart-policy not honored -> a crashed felhom-controller has no automatic recovery (oneshot bootstrap + dead restart policy). Plus disconnect-recreate race, agent loopback whitelist, carried-forward #1 gaps. Demo restored to baseline. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.8 (1M context) --- .../test-campaign-2-2026-06-22-findings.md | 88 +++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100644 documentation/tests/test-campaign-2-2026-06-22-findings.md diff --git a/documentation/tests/test-campaign-2-2026-06-22-findings.md b/documentation/tests/test-campaign-2-2026-06-22-findings.md new file mode 100644 index 0000000..548d1fa --- /dev/null +++ b/documentation/tests/test-campaign-2-2026-06-22-findings.md @@ -0,0 +1,88 @@ +# Test campaign #2 — broader apps, edge cases, break-and-recover — 2026-06-22 (demo 9201) + +> **Status: empirical findings from an unattended chaos/edge-case campaign (2026-06-22).** Follow-on to +> `unattended-test-campaign-2026-06-22-findings.md`. The campaign-#1 blocker (controller→agent leak) is +> fixed (ctrl **v0.74.0**), so the **storage lifecycle that was BLOCKED in #1 is now tested**, plus +> deliberate break-and-recover chaos against the architectural contracts. Demo restored to baseline +> (25 containers, 0 unhealthy) and healthy at the end. Per-check raw evidence: `felhom-controller/TEST-REPORT.md`. + +## Topology +ctrl **v0.74.0** (guest 9201), agent v0.39.0 (felhom-pve), hub on DooPlex; PBS `felhom-pbs`. Drives +felhom-flash (uuid:81a26531) + felhom-usb (uuid:da9e7089, empty). DR source / master net: +PBS `felhom-pbs:backup/ct/9201/2026-06-22T16:13:44Z` (verified). UI driven via an in-guest forwarder +(on-LAN traefik :443 still 404s — campaign-#1 finding #5). + +## What passed (high-value) +- **Leak fix holds (Phase 0):** `/api/disks` + `/api/host-metrics` work; controller→agent idle sockets + to `:8443` **flat at 2** across a 60-call burst (v0.74.0). +- **Storage lifecycle (was BLOCKED in #1):** scan/health/durable-id; label edit→revert; **disconnect↔ + reconnect** (4-state enrolled→disconnected→enrolled); **decommission↔re-enroll** (non-destructive — + fs uuid preserved, never formatted). All through the real controller→agent pipeline. +- **Chaos crash-safety:** + - **C7 (the headline):** killed the controller **mid-quiesce** — the quiesce marker (written upfront, + 15 stopped_stacks) **persisted across the crash**; controller restart ran `Recover()` + (`[quiesce] crash recovery … restarting them`) → all stacks resumed, marker cleared, none stranded. + - **C3:** stopped the agent → controller degraded **gracefully** (fast ~98ms clean "connection + refused", no hang) → agent restart → calls succeeded on the **first** try (the v0.74.0 client + drains the dead pooled conn), sockets bounded. + - **C8:** reboot during a backup op → 26 containers back, drives remounted **by UUID**, tunnel + reconnected, romm DB intact (no corruption). + - C1 (kill app→restart), C5 (remove app.yaml→graceful orphan→restore), C6 (eject drive-with-apps→ + reconnect→recover) all recovered + verified. +- **Single-flight** is robust — rejected concurrent restore (mutex), concurrent whole-guest backup, and + Tier-2-vs-DB-dump. +- **Backup edge cases:** concurrent restore blocked ("backup or restore already in progress"); + backup with a stopped stack handled (no error). + +## Findings (new) + +### 🔴 1. docker restart policy not honored → no auto-recovery for a crashed controller +`restart: unless-stopped` containers do **not** auto-restart after `docker kill` (navidrome +`RestartCount=0` over two trials; dockerd logged `stopping restart-manager`; Docker 29.5.3, Live +Restore off). The controller does not reconcile-stop them (verified in logs/source). Consequences: +- **Apps:** a crashed app stays down; the controller **detects** `stopped` and a UI/operator restart + recovers it (C1) — but there is **no docker-level self-heal**. +- **🔴 Controller itself (C2):** killed → stayed down 48 s, **no automatic recovery** — docker policy + dead + `felhom-controller-bootstrap.service` is **oneshot** (already exited). Recovery required a + manual `systemctl restart felhom-controller-bootstrap.service`. Data plane (apps + tunnel) kept + running throughout (control/data separation confirmed), but the **management plane has no + auto-recovery** if the controller crashes (OOM/panic/kill) — it would stay down until a guest reboot + or manual intervention. +- **Fix direction:** add a controller watchdog — e.g. make the bootstrap a `Restart=always` monitor, or + a small systemd unit/timer that recreates `felhom-controller` if absent; and investigate why the + docker daemon's restart-manager is being cancelled (Docker 29.5.3 behaviour / cgroup). + +### 2. Disconnect→recreate race on a drive-with-apps (C6) +Disconnecting felhom-flash (9 apps) recovered cleanly on reconnect, but during the drive-absent window +the controller's **userdata-belt/recreate** tried to bring drive-backed apps up while the mount was +gone → `mkdir …/userdata: permission denied` + transient `Created` states, instead of a clean held +"missing-storage" stop (B2b). *Fix:* gate ensure-userdata/recreate on "the drive is a live mountpoint." + +### 3. FileBrowser skeleton-ensure on an unmounted path (cosmetic) +disconnect/decommission log `mkdir …/userdata: permission denied` from the FileBrowser sync against the +just-unmounted path. Harmless but noisy; gate on live-mountpoint. + +### 4. Agent device whitelist refuses loopback (good security) +`/api/storage/init` on `/dev/loop0` → agent `refusing to operate on non-whitelisted block device` — a +deliberate safety control. So a loopback can't be the destructive-disk scratch target the runbook +assumed; destructive format/migrate need a real spare disk or a supervised run. + +### Carried from campaign #1 (still open) +- `GET /api/backup/snapshots` → 404 (no handler) — restore-UI snapshot dropdown can't populate. +- Restore DB import is **additive** — a planted marker table survived the restore (no `--clean`). + +## Deferred / needs supervised or implementation +- **Controller-recovery watchdog** + docker restart-policy investigation (finding #1) — highest priority. +- Destructive storage **format/migrate** on a real spare disk (loopback blocked). +- **Host reboot** resilience (Phase 5 — skipped unattended: no physical recovery if the N100 doesn't return). +- Disconnect-recreate gate (#2); FileBrowser unmounted-skeleton gate (#3). +- `/api/backup/snapshots` handler + UI snapshot selection; restore `--clean` (campaign-#1 gaps). + +## Code changes +**None this campaign** — no broken apps surfaced that needed a catalog fix (navidrome deployed clean). +All findings are controller/agent-side behaviours logged for supervised follow-up, not changed here. + +## Cleanup / after-state +9201 running, **25 containers, 0 unhealthy**; agent active; controller + cloudflared healthy; both +drives re-enrolled + mounted. Scratch loopback torn down; navidrome test app removed; all timed guards +killed; forwarder + temp files removed. No throwaway guest created this campaign.