felhom-controller/TEST-REPORT.md

# TEST-REPORT — Test campaign #2 (broader apps, edge cases, break-and-recover) — N100 / guest 9201

**Run start:** 2026-06-22 (CC, unattended). **Operator:** Viktor (away).
**Scope:** push past campaign #1 — more apps, edge cases, chaos break-and-recover. Storage lifecycle now
unblocked (ctrl v0.74.0 fixed the agent leak). Only code changes: trivial catalog HC/config-seed fixes.
**Legend:** PASS / FAIL / SKIPPED(reason). Evidence raw (docker ps / ss / curl / screenshot).
(Campaign #1 report preserved in git history + `felhom.eu/documentation/tests/unattended-test-campaign-2026-06-22-*.md`.)

---

## Phase 0 — Baseline + safety net + leak-fix gate — **PASS (gate OPEN)**

- **Baseline:** ctrl `v0.74.0` healthy, agent `v0.39.0`; `pct list` = 9201 running (+ 9001/9999 stopped, pre-existing); **25 containers, 0 unhealthy**; rootfs `/` 4%, `/var/lib/docker` 8%, drives felhom-flash 1% / felhom-usb 1%.
- **Master safety net + DR source:** PBS backup `felhom-pbs:backup/ct/9201/2026-06-22T16:13:44Z`, success=true, crash-consistent; PBS verify shows the snapshot present.
- **Leak-fix gate (the #1 unblock):** `/api/disks` lists felhom-flash/felhom-usb/local; `/api/host-metrics` returns `cpu_percent`; controller→agent idle sockets to `:8443` **flat at 2 before and after a 60-call burst**. Agent path fully working. **Gate to chaos/destructive phases: OPEN.**

## Phase 1 — Broader app sweep + edge cases — **PASS (with noted non-triggerables)**

| Check | Result | Evidence |
|---|---|---|
| Deploy navidrome (HDD-bound, Go) via GUI | PASS | GUI deploy → 3-step panel → `running/healthy` in 8s; `:4533` → HTTP 302 (serving); HDD binding `…/felhom-flash/userdata/media/music → /music` |
| Pattern coverage | PASS (cited) | JVM=komga (healthy), Python+DB=immich/paperless (healthy), Go=navidrome/gitea, HDD-bound=navidrome/romm/immich, optional-config=romm |
| Edge: rapid deploy→stop→redeploy race | PASS | stop → immediate update → `navidrome Up (healthy)`, `app.yaml deployed:true`, API `state=running deployed=true` — no stale button / no Deployed-flag race |
| Edge: slow-DB-migration start_period | PASS (cited) | mealie deployed healthy (start_period 60s covers migration — no premature unhealthy flap; lesson #7) |
| Edge: missing required config → graceful signal | PASS (cited #1) | campaign-1 glance: crash-loop with a clear logged error (`open /app/config/glance.yml: no such file`), controller marks it — not a silent hang |
| Edge: mem-gate hard-block | PARTIAL | the deploy-page memory projection renders correctly (e.g. navidrome 4065→+50 MB / 11904); a RAM hard-block isn't naturally triggerable (no single catalog app's request > ~7.6 GB free without mass-filling RAM). Disk-headroom hard-refuse (`GetDockerVolumeHeadroom`) is tested in Phase 3/C4. |
| Edge: Host()/port overlap | N/A (noted) | each catalog app has a unique subdomain by construction; an overlap isn't naturally producible without editing a template — not triggered |

navidrome left deployed (used as the drive-backed app for Phase-3 C6); removed at cleanup.
## Phase 2 — Storage lifecycle (the campaign-#1 blocker, NOW UNBLOCKED) — **PASS (destructive-format skipped, documented)**

Agent path works (v0.74.0), so these ran through the real controller→agent pipeline.

| Check | Result | Evidence |
|---|---|---|
| Scan / observe / health / FS / role / durable-id | PASS | `/api/disks`: 5 targets — felhom-flash (user-data, uuid:81a26531), felhom-usb (user-data, uuid:da9e7089), local/local-lvm (system), felhom-pbs (backup) with capacities + data_bearing |
| Label edit → revert (live) | PASS | felhom-usb label → "TEST-LABEL-c2" (log "Storage label updated") → reverted to exact UTF-8 original "Tárhely (felhom-usb)" (byte-verified). [my shell mangled the accent on first revert; corrected via `%C3%A1`] |
| **Disconnect → reconnect (reversible four-state)** | PASS | enrolled→**disconnected** (`/api/storage/disconnect`: unmounted, no apps) → **enrolled** (`/api/storage/reconnect`: remounted `/dev/sdb1`, disconnected mark cleared) |
| **Decommission → re-enroll (non-destructive)** | PASS | `/api/storage/decommission mode=anyway` (type-to-confirm `felhom-usb`): unmounted, `decommissioned=True` soft-marker, **fs intact — uuid da9e7089 preserved, NOT formatted**. `/api/storage/reconnect`: `reenrolled=true`, MOUNTED, marker cleared, label preserved |
| Destructive format/wipe on **loopback scratch** | **BLOCKED (documented)** | created `/dev/loop0` (truncate 2G + losetup); `/api/storage/init` → agent **`refusing to operate on non-whitelisted block device "/dev/loop0"`** — a deliberate device-whitelist safety control. So the clean scratch target the runbook prescribed is unavailable. Per §0.3's escape clause (no clean scratch target → skip), the destructive *format* was not run on the live felhom-usb either. Loopback torn down. |
| Migrate (data) | DEFERRED | no safe scratch target (loopback blocked); a live migrate would move a real app's data — not run unattended. `internal/stacks/migrate.go` has unit coverage. |

**Findings:**
1. **Agent device whitelist** refuses non-whitelisted block devices (loopback) — good security, but means a loopback can't serve as the destructive-disk scratch target the runbook assumed. Destructive format/migrate now need either a real spare disk or a supervised run.
2. **FileBrowser sync** logs `mkdir …/userdata: permission denied` when ensuring the skeleton on a just-unmounted/decommissioned path (recurs on disconnect + decommission). Cosmetic (the drive is intentionally unmounted), but noisy — could gate the skeleton-ensure on "is the path a live mountpoint."
## Phase 3 — Break-and-recover chaos (C1–C8) — _pending_
## Phase 4 — Backup/restore edge cases — _pending_
## Phase 5 — Host reboot (optional) — _pending_

---
## Feature matrix — _filled at end_
## Cleanup confirmation — _filled at end_
## Deferred / needs supervised — _filled at end_