# Test campaign #2 — broader apps, edge cases, break-and-recover — 2026-06-22 (demo 9201)

> **Status: empirical findings from an unattended chaos/edge-case campaign (2026-06-22).** Follow-on to
> `unattended-test-campaign-2026-06-22-findings.md`. The campaign-#1 blocker (controller→agent leak) is
> fixed (ctrl **v0.74.0**), so the **storage lifecycle that was BLOCKED in #1 is now tested**, plus
> deliberate break-and-recover chaos against the architectural contracts. Demo restored to baseline
> (25 containers, 0 unhealthy) and healthy at the end. Per-check raw evidence: `felhom-controller/TEST-REPORT.md`.

## Topology
ctrl **v0.74.0** (guest 9201), agent v0.39.0 (felhom-pve), hub on DooPlex; PBS `felhom-pbs`. Drives
felhom-flash (uuid:81a26531) + felhom-usb (uuid:da9e7089, empty). DR source / master net:
PBS `felhom-pbs:backup/ct/9201/2026-06-22T16:13:44Z` (verified). UI driven via an in-guest forwarder
(on-LAN traefik :443 still 404s — campaign-#1 finding #5).

## What passed (high-value)
- **Leak fix holds (Phase 0):** `/api/disks` + `/api/host-metrics` work; controller→agent idle sockets
  to `:8443` **flat at 2** across a 60-call burst (v0.74.0).
- **Storage lifecycle (was BLOCKED in #1):** scan/health/durable-id; label edit→revert; **disconnect↔
  reconnect** (4-state enrolled→disconnected→enrolled); **decommission↔re-enroll** (non-destructive —
  fs uuid preserved, never formatted). All through the real controller→agent pipeline.
- **Chaos crash-safety:**
  - **C7 (the headline):** killed the controller **mid-quiesce** — the quiesce marker (written upfront,
    15 stopped_stacks) **persisted across the crash**; controller restart ran `Recover()`
    (`[quiesce] crash recovery … restarting them`) → all stacks resumed, marker cleared, none stranded.
  - **C3:** stopped the agent → controller degraded **gracefully** (fast ~98ms clean "connection
    refused", no hang) → agent restart → calls succeeded on the **first** try (the v0.74.0 client
    drains the dead pooled conn), sockets bounded.
  - **C8:** reboot during a backup op → 26 containers back, drives remounted **by UUID**, tunnel
    reconnected, romm DB intact (no corruption).
  - C1 (kill app→restart), C5 (remove app.yaml→graceful orphan→restore), C6 (eject drive-with-apps→
    reconnect→recover) all recovered + verified.
- **Single-flight** is robust — rejected concurrent restore (mutex), concurrent whole-guest backup, and
  Tier-2-vs-DB-dump.
- **Backup edge cases:** concurrent restore blocked ("backup or restore already in progress");
  backup with a stopped stack handled (no error).

## Findings (new)

### ✅ 1. [DOWNGRADED — TEST ARTIFACT] "docker restart policy not honored / no auto-recovery"
> **Correction (2026-06-22, read-only re-diagnosis — see `test-campaign-2-finding1-recovery-diagnosis.md`):**
> Finding #1 below was a **test artifact**. The controller **does** auto-recover from a genuine crash:
> `kill -9` of the controller's container PID (a real process death) → docker `unless-stopped`
> auto-restarted it (**RestartCount=1, healthy in ~5 s**). The campaign tested with `docker kill`, which
> Docker treats as a **manual stop** (restart policy intentionally suppressed) — not a crash. A
> natural-exit probe confirmed the restart-manager works (`restartCount` climbs, `manualRestart=false`).
> Daemon is stable (NRestarts=0, no OOM). **No watchdog needed; no real recovery gap.** The original
> (incorrect) text is kept below for the record.

#### (original, superseded) docker restart policy not honored → no auto-recovery for a crashed controller
`restart: unless-stopped` containers do **not** auto-restart after `docker kill` (navidrome
`RestartCount=0` over two trials; dockerd logged `stopping restart-manager`; Docker 29.5.3, Live
Restore off). The controller does not reconcile-stop them (verified in logs/source). Consequences:
- **Apps:** a crashed app stays down; the controller **detects** `stopped` and a UI/operator restart
  recovers it (C1) — but there is **no docker-level self-heal**.
- **🔴 Controller itself (C2):** killed → stayed down 48 s, **no automatic recovery** — docker policy
  dead + `felhom-controller-bootstrap.service` is **oneshot** (already exited). Recovery required a
  manual `systemctl restart felhom-controller-bootstrap.service`. Data plane (apps + tunnel) kept
  running throughout (control/data separation confirmed), but the **management plane has no
  auto-recovery** if the controller crashes (OOM/panic/kill) — it would stay down until a guest reboot
  or manual intervention.
- **Fix direction:** add a controller watchdog — e.g. make the bootstrap a `Restart=always` monitor, or
  a small systemd unit/timer that recreates `felhom-controller` if absent; and investigate why the
  docker daemon's restart-manager is being cancelled (Docker 29.5.3 behaviour / cgroup).

### 2. Disconnect→recreate race on a drive-with-apps (C6)
Disconnecting felhom-flash (9 apps) recovered cleanly on reconnect, but during the drive-absent window
the controller's **userdata-belt/recreate** tried to bring drive-backed apps up while the mount was
gone → `mkdir …/userdata: permission denied` + transient `Created` states, instead of a clean held
"missing-storage" stop (B2b). *Fix:* gate ensure-userdata/recreate on "the drive is a live mountpoint."

### 3. FileBrowser skeleton-ensure on an unmounted path (cosmetic)
disconnect/decommission log `mkdir …/userdata: permission denied` from the FileBrowser sync against the
just-unmounted path. Harmless but noisy; gate on live-mountpoint.

### 4. Agent device whitelist refuses loopback (good security)
`/api/storage/init` on `/dev/loop0` → agent `refusing to operate on non-whitelisted block device` — a
deliberate safety control. So a loopback can't be the destructive-disk scratch target the runbook
assumed; destructive format/migrate need a real spare disk or a supervised run.

### Carried from campaign #1 (still open)
- `GET /api/backup/snapshots` → 404 (no handler) — restore-UI snapshot dropdown can't populate.
- Restore DB import is **additive** — a planted marker table survived the restore (no `--clean`).

## Deferred / needs supervised or implementation
- **Controller-recovery watchdog** + docker restart-policy investigation (finding #1) — highest priority.
- Destructive storage **format/migrate** on a real spare disk (loopback blocked).
- **Host reboot** resilience (Phase 5 — skipped unattended: no physical recovery if the N100 doesn't return).
- Disconnect-recreate gate (#2); FileBrowser unmounted-skeleton gate (#3).
- `/api/backup/snapshots` handler + UI snapshot selection; restore `--clean` (campaign-#1 gaps).

## Code changes
**None this campaign** — no broken apps surfaced that needed a catalog fix (navidrome deployed clean).
All findings are controller/agent-side behaviours logged for supervised follow-up, not changed here.

## Cleanup / after-state
9201 running, **25 containers, 0 unhealthy**; agent active; controller + cloudflared healthy; both
drives re-enrolled + mounted. Scratch loopback torn down; navidrome test app removed; all timed guards
killed; forwarder + temp files removed. No throwaway guest created this campaign.