felhom-controller/TEST-REPORT.md

# TEST-REPORT — Unattended test campaign (N100 / guest 9201)

**Run start:** 2026-06-22 (CC, unattended)
**Operator:** Viktor (away, glancing remotely)
**Scope:** RUNBOOK phased test matrix. Only code change permitted = komga catalog HC fix.
**Legend:** PASS / FAIL / SKIPPED(reason). Evidence is raw (command output / volid / screenshot path).

---

## Baseline state (Phase 0 "before")

- **Host:** felhom-pve (`demo-felhom`), agent **v0.39.0** (service active).
- **Controller:** image `gitea.dooplex.hu/admin/felhom-controller:0.73.0`, container `Up 5 days (healthy)`.
- **`pct list`:** 9201 running (`demo-felhom`); 9001 stopped (`spike-lxc`); 9999 stopped (`felhom-selftest-scratch`).
- **Infra containers (9201):** felhom-controller, cloudflared, traefik, filebrowser — all Up 5 days, healthy.
- **App containers (9201):** 24 containers Up 18h. All healthy **except `komga` = unhealthy** (Phase-1 target).
- **Disk (9201):** rootfs `/` 32G (4% used); `/var/lib/docker` 252G (8%, 19G); `/mnt/felhom-drives` (pve-root) 94G (31%).
- **Drives:** `felhom-flash` sdc1 118G (~159M used) `/mnt/felhom-drives/felhom-flash`; `felhom-usb` sdb1 916G (~2M used) `/mnt/felhom-drives/felhom-usb`.
- **PBS storage:** `felhom-pbs` active, 9.6 TB total, 21% used. Nightly 9201 backups present (latest `2026-06-21T18:55:12Z`, ~19.5 GB).
- **pvesm:** felhom-flash, felhom-usb, felhom-pbs, local, local-lvm — all active.

---

## Phase 0 — Baseline + safety net

State: 9201 healthy, controller up, tunnel up (cloudflared Up 5 days).

| Check | Result | Evidence |
|---|---|---|
| Baseline capture | PASS | see above |
| GUI screenshots (before) | _pending_ | Chrome bridge check next |
| Fresh PBS whole-CT backup of 9201 | **PASS** | volid `felhom-pbs:backup/ct/9201/2026-06-22T12:45:37Z`, 19.62 GB, crash-consistent=true, 161.5s, success=true. uncovered_volumes (by design): /etc/felhom-bootstrap, /mnt/felhom-drives (drives backed up separately) |
| PBS verify of new snapshot | **PASS** | `verify_state: ok` for backup_id 9201 backup_time 2026-06-22T12:45:37Z (datastore felhom-spike, verify UPID present) |

**DR source for Phase 4:** `felhom-pbs:backup/ct/9201/2026-06-22T12:45:37Z` (verified ok). Gate to destructive phases: **OPEN**.

**GUI render smoke-test (before):** dashboard PASS (17 running / 0 stopped / 55 total; mem 5.8/12 GB; CPU 5%; 58°C; v0.73.0; app-backup "Még nem futott"), `/monitoring` PASS, `/stacks` PASS.

### Access path note (important finding)
- On-LAN, split-horizon DNS (resolver 192.168.0.250) maps `*.demo-felhom.eu` → guest 9201 (192.168.0.141), bypassing Cloudflare (no `cf-ray` on responses).
- **The guest's traefik `:443` (websecure) returns Go `404 page not found` for ALL hosts** (controller AND apps, e.g. romm), while serving a valid LE `*.demo-felhom.eu` cert. `:80` 301-redirects to https → so the browser dead-ends at the 404. → on-LAN HTTPS to the dashboard/apps is non-functional. (The supported external path is Cloudflare; couldn't validate it from the LAN due to DNS hijack.) **FINDING — flag for supervised review.**
- **Workaround used for UI testing (reversible):** a python TCP forwarder inside guest 9201 (`/tmp/felhom-test-fwd.py`, listens `0.0.0.0:18080` → controller `172.18.0.10:8080`). Browser → `http://felhom.demo-felhom.eu:18080/` (LAN DNS → guest:18080 → controller). `CatchAllMiddleware` strips the port so Host matches `felhom.<domain>` → dashboard serves over plain HTTP. **To be killed at cleanup.**
- The dashboard currently serves with **no auth wall** (`/` → 200, no login redirect, no session cookie) — no password configured/enforced. (Bears on Phase 6 "password change" step.)

### Scope reality vs runbook
- Runbook assumed "14 enabled apps"; catalog is now **55 total, 38 not-deployed**. Deploying all 38 unattended would exhaust the 12 GiB guest. **Phase 2 will deploy a representative sample** (a few light apps) + cite the already-deployed heavy multi-container case (immich, 4 containers healthy). Documented, not silently truncated.

**Phase 0 verdict: PASS — gate OPEN.**

---

## Phase 1 — komga HC fix + git-sync + health detection — **PASS**

State: 9201 healthy, controller up, tunnel up.

**Diagnosis (live, guest 9201):** docker HC `curl -f .../api/v1/actuator/health` → exit 22 (HTTP 401, auth-gated) → `unhealthy` for 18h while serving fine on :25600. Endpoint probe matrix (wget in image): `/`, `/actuator/health`, `/api/v1/oauth2/providers`, `/login` all 200; only `/api/v1/actuator/health` → 401. Image ships `curl` (`/usr/bin/curl`). **Found a second probe:** the controller's own `.felhom.yml` `healthcheck.checks[].path` (drives the UI badge + route publishing) also pointed at the 401 path.

| Check | Result | Evidence |
|---|---|---|
| Catalog HC fix (docker-compose.yml) | PASS | `/api/v1/actuator/health` → `/actuator/health`; commit `3faa5ae` pushed to `main` |
| Catalog probe fix (.felhom.yml) | PASS | controller-side probe path fixed; commit `9b066de` pushed to `main` |
| CHANGELOG + REPORT (app-catalog) | PASS | created `CHANGELOG.md` (newest-on-top) + overwrote `REPORT.md` (`f51f794`) |
| GUI git-sync ("Sablonok frissítése") | PASS | logs: `[sync] Updated komga/docker-compose.yml` + `komga/.felhom.yml`; `Sablonok frissítve — frissítve: komga`. Live stack `/opt/docker/stacks/komga/docker-compose.yml` updated to `/actuator/health` (content-hash copy verified) |
| GUI redeploy ("Frissítés") | PASS | `POST /api/stacks/komga/update` → 200; `docker compose up -d --remove-orphans`; "Stack komga updated successfully (4.2s)"; container recreated (StartedAt 2026-06-22T13:09:50Z) |
| komga reaches healthy (docker) | PASS | `docker ps`: `komga Up (healthy)`; HC now `http://localhost:25600/actuator/health` |
| Controller health-state machine | PASS | controller log `Health probes: N ok (of N probed)` (no more komga WARN); card badge `Nem egészséges` → **Fut** |
| Route publishing | PASS | "⚠ URL nem elérhető – útvonal nincs publikálva" warning cleared; `komga.demo-felhom.eu ↗` link shown |

Minor cosmetic note (not fixed): a few catalog logo SVGs 404 (`adventurelog-logo.svg`, `cloudflared-logo.svg`, `crafty-controller-logo.svg`).
## Phase 2 — App deployment sweep — **PASS (with 1 logged catalog defect)**

State: 9201 healthy, controller up, tunnel up. Scope: representative sample (see Phase-0 scope note) — catalog has 38 not-deployed apps; deploying all on a 12 GiB guest is unsafe.

| App | Result | Evidence |
|---|---|---|
| **gitea** (deploy via GUI) | PASS (after HC fix) | GUI deploy page → memory-gate bar (`6043/11904 MB 50%`, two-segment current+new) → 3-step progress panel (✅ Konfiguráció mentve → ✅ Konténerek elindultak → ⏳ inicializálás) → `POST /api/stacks/gitea/update` 200. Initially **unhealthy**: HC `curl -f /api/v1/version` → 404 (404 until install-lock) while `/api/healthz` → 200. **Fixed catalog** (commit `17e00b7` + CHANGELOG `1a44c14`), re-synced, redeployed → `docker inspect`: `healthy`, HC now `/api/healthz`; endpoint `http://127.0.0.1:3000/` → `HTTP/1.1 200 OK`. |
| **glance** (deploy via GUI) | **FAIL — logged, not fixed** | GUI deploy → 3-step panel OK, but container `Restarting (1)` crash-loop. Logs: `parsing config: reading main YAML file: open /app/config/glance.yml: no such file or directory`. **Root cause:** catalog template mounts an empty `glance_config:/app/config` named volume but never seeds the required `glance.yml`; glance refuses to start without it. **Not a trivial HC/start_period fix** (needs a seeded default config asset + wiring) → logged per runbook, not fixed unattended. Cleaned up: `POST .../stop` + `POST .../remove` → container gone. |
| **immich** (heavy multi-container, pre-deployed) | PASS (cited) | 4 containers (server/ML/postgres/redis) all `Up (healthy)` — the heavy-app coverage case. |

**Bonus coverage:** stop + remove endpoints exercised (glance teardown). **Catalog defects found this phase:** gitea HC (fixed), glance missing-config (logged for supervised fix).
## Phase 3 — Backup + restore — **PASS (2 findings)**

State: 9201 healthy, controller up, tunnel up.

| Check | Result | Evidence |
|---|---|---|
| Manual DB-dump backup ("Mentés most") | PASS | `POST /api/backup/run` → 3 DBs dumped: romm-mariadb 38.5 KB/14 tbl, paperless-postgres 360 KB/67 tbl, immich-postgres 43.2 MB/60 tbl; 43.6 MB total in 2.18s; `status.success=true`, `last_run 2026-06-22T13:27:26Z`. Files on disk fresh **13:28**: `…/primary/immich/db-dumps/immich-postgres.sql` (45.3 MB), `…/primary/romm/db-dumps/romm-mariadb.sql` (39 KB) |
| Cross-drive copy (Tier-2) | PASS | `POST /api/backup/tier2` → 8 HDD apps copied to `/mnt/sys_drive/felhom-data/backups/secondary/<app>` (immich 155.3 MB verified 156 MB on disk; romm has appdata + recovery-unit{db-dump,manifest,compose}). `crossdrive_completed` events pushed |
| Backup page reflects status | PASS | page shows "Adatmentés aktív", 3 DBs, schedule 02:30, last DB backup timestamp; status API success=true |
| Restore flow (per-app) | PASS | `POST /backup/restore stack_name=romm` → `RestoreFromRecoveryUnit`: compose down → redeploy from recovery unit (6 env, 3 encrypted) → "Imported DB dump romm-mariadb.sql into romm-db" → containers up → "Restore-from-unit completed: romm" |
| Restore is non-hollow | PASS | post-restore romm DB has real `platforms` table + 17 tables; all 3 romm containers `Up (healthy)`. Marker-table test: planted `restore_test_marker`(id=42) pre-restore → **survived** restore (see finding) but real snapshot tables were reimported; marker dropped afterward (DB back to 17-table baseline) |

**Findings:**
1. **`GET /api/backup/snapshots` has no server handler** (api-router returns "endpoint not found"). The restore UI's snapshot dropdown (`backups.html:682`) can't populate → a pure-UI restore is blocked at snapshot selection. Restore itself works via `POST /backup/restore` (the form's action). **Flag for fix.**
2. **Restore DB import is additive** — `RestoreFromRecoveryUnit` replays the dump but does not drop tables absent from it (my marker table persisted). Real snapshot data is restored; stale/extra tables are not pruned. Minor behavioral note.
3. Whole-server ("Rendszermentés") backup shows **"A host-ügynök jelenleg nem elérhető"** (agent local API unreachable from controller) — same root as the monitoring host-metrics gap; tracked in Phase 6.
## Phase 4 — DR rebuild (9300) — _pending_
## Phase 5 — Storage lifecycle — _pending_
## Phase 6 — Monitoring/alerts/hub — _pending_
## Phase 7 — Resilience (optional) — _pending_

---

## Feature matrix (filled at end)
## Cleanup confirmation (filled at end)
## Deferred — needs supervised run (filled at end)