felhom.eu/documentation/tests/test-campaign-2-2026-06-22-findings.md

Test campaign #2 — broader apps, edge cases, break-and-recover — 2026-06-22 (demo 9201)

Status: empirical findings from an unattended chaos/edge-case campaign (2026-06-22). Follow-on to unattended-test-campaign-2026-06-22-findings.md. The campaign-#1 blocker (controller→agent leak) is fixed (ctrl v0.74.0), so the storage lifecycle that was BLOCKED in #1 is now tested, plus deliberate break-and-recover chaos against the architectural contracts. Demo restored to baseline (25 containers, 0 unhealthy) and healthy at the end. Per-check raw evidence: felhom-controller/TEST-REPORT.md.

Topology

ctrl v0.74.0 (guest 9201), agent v0.39.0 (felhom-pve), hub on DooPlex; PBS felhom-pbs. Drives felhom-flash (uuid:81a26531) + felhom-usb (uuid:da9e7089, empty). DR source / master net: PBS felhom-pbs:backup/ct/9201/2026-06-22T16:13:44Z (verified). UI driven via an in-guest forwarder (on-LAN traefik :443 still 404s — campaign-#1 finding #5).

What passed (high-value)

• Leak fix holds (Phase 0): /api/disks + /api/host-metrics work; controller→agent idle sockets to :8443 flat at 2 across a 60-call burst (v0.74.0).
• Storage lifecycle (was BLOCKED in #1): scan/health/durable-id; label edit→revert; disconnect↔ reconnect (4-state enrolled→disconnected→enrolled); decommission↔re-enroll (non-destructive — fs uuid preserved, never formatted). All through the real controller→agent pipeline.
• Chaos crash-safety:
  • C7 (the headline): killed the controller mid-quiesce — the quiesce marker (written upfront, 15 stopped_stacks) persisted across the crash; controller restart ran Recover() ([quiesce] crash recovery … restarting them) → all stacks resumed, marker cleared, none stranded.
  • C3: stopped the agent → controller degraded gracefully (fast ~98ms clean "connection refused", no hang) → agent restart → calls succeeded on the first try (the v0.74.0 client drains the dead pooled conn), sockets bounded.
  • C8: reboot during a backup op → 26 containers back, drives remounted by UUID, tunnel reconnected, romm DB intact (no corruption).
  • C1 (kill app→restart), C5 (remove app.yaml→graceful orphan→restore), C6 (eject drive-with-apps→ reconnect→recover) all recovered + verified.
• Single-flight is robust — rejected concurrent restore (mutex), concurrent whole-guest backup, and Tier-2-vs-DB-dump.
• Backup edge cases: concurrent restore blocked ("backup or restore already in progress"); backup with a stopped stack handled (no error).

Findings (new)

✅ 1. [DOWNGRADED — TEST ARTIFACT] "docker restart policy not honored / no auto-recovery"

Correction (2026-06-22, read-only re-diagnosis — see test-campaign-2-finding1-recovery-diagnosis.md): Finding #1 below was a test artifact. The controller does auto-recover from a genuine crash: kill -9 of the controller's container PID (a real process death) → docker unless-stopped auto-restarted it (RestartCount=1, healthy in ~5 s). The campaign tested with docker kill, which Docker treats as a manual stop (restart policy intentionally suppressed) — not a crash. A natural-exit probe confirmed the restart-manager works (restartCount climbs, manualRestart=false). Daemon is stable (NRestarts=0, no OOM). No watchdog needed; no real recovery gap. The original (incorrect) text is kept below for the record.

(original, superseded) docker restart policy not honored → no auto-recovery for a crashed controller

restart: unless-stopped containers do not auto-restart after docker kill (navidrome RestartCount=0 over two trials; dockerd logged stopping restart-manager; Docker 29.5.3, Live Restore off). The controller does not reconcile-stop them (verified in logs/source). Consequences:

• Apps: a crashed app stays down; the controller detects stopped and a UI/operator restart recovers it (C1) — but there is no docker-level self-heal.
• 🔴 Controller itself (C2): killed → stayed down 48 s, no automatic recovery — docker policy dead + felhom-controller-bootstrap.service is oneshot (already exited). Recovery required a manual systemctl restart felhom-controller-bootstrap.service. Data plane (apps + tunnel) kept running throughout (control/data separation confirmed), but the management plane has no auto-recovery if the controller crashes (OOM/panic/kill) — it would stay down until a guest reboot or manual intervention.
• Fix direction: add a controller watchdog — e.g. make the bootstrap a Restart=always monitor, or a small systemd unit/timer that recreates felhom-controller if absent; and investigate why the docker daemon's restart-manager is being cancelled (Docker 29.5.3 behaviour / cgroup).

2. Disconnect→recreate race on a drive-with-apps (C6)

Disconnecting felhom-flash (9 apps) recovered cleanly on reconnect, but during the drive-absent window the controller's userdata-belt/recreate tried to bring drive-backed apps up while the mount was gone → mkdir …/userdata: permission denied + transient Created states, instead of a clean held "missing-storage" stop (B2b). Fix: gate ensure-userdata/recreate on "the drive is a live mountpoint."

3. FileBrowser skeleton-ensure on an unmounted path (cosmetic)

disconnect/decommission log mkdir …/userdata: permission denied from the FileBrowser sync against the just-unmounted path. Harmless but noisy; gate on live-mountpoint.

4. Agent device whitelist refuses loopback (good security)

/api/storage/init on /dev/loop0 → agent refusing to operate on non-whitelisted block device — a deliberate safety control. So a loopback can't be the destructive-disk scratch target the runbook assumed; destructive format/migrate need a real spare disk or a supervised run.

Carried from campaign #1 (still open)

• GET /api/backup/snapshots → 404 (no handler) — restore-UI snapshot dropdown can't populate.
• Restore DB import is additive — a planted marker table survived the restore (no --clean).

Deferred / needs supervised or implementation

• Controller-recovery watchdog + docker restart-policy investigation (finding #1) — highest priority.
• Destructive storage format/migrate on a real spare disk (loopback blocked).
• Host reboot resilience (Phase 5 — skipped unattended: no physical recovery if the N100 doesn't return).
• Disconnect-recreate gate (#2); FileBrowser unmounted-skeleton gate (#3).
• /api/backup/snapshots handler + UI snapshot selection; restore --clean (campaign-#1 gaps).

Code changes

None this campaign — no broken apps surfaced that needed a catalog fix (navidrome deployed clean). All findings are controller/agent-side behaviours logged for supervised follow-up, not changed here.

Cleanup / after-state

9201 running, 25 containers, 0 unhealthy; agent active; controller + cloudflared healthy; both drives re-enrolled + mounted. Scratch loopback torn down; navidrome test app removed; all timed guards killed; forwarder + temp files removed. No throwaway guest created this campaign.