felhom-controller/TEST-REPORT.md

# TEST-REPORT — Unattended test campaign (N100 / guest 9201)

**Run start:** 2026-06-22 (CC, unattended)
**Operator:** Viktor (away, glancing remotely)
**Scope:** RUNBOOK phased test matrix. Only code change permitted = komga catalog HC fix.
**Legend:** PASS / FAIL / SKIPPED(reason). Evidence is raw (command output / volid / screenshot path).

---

## Baseline state (Phase 0 "before")

- **Host:** felhom-pve (`demo-felhom`), agent **v0.39.0** (service active).
- **Controller:** image `gitea.dooplex.hu/admin/felhom-controller:0.73.0`, container `Up 5 days (healthy)`.
- **`pct list`:** 9201 running (`demo-felhom`); 9001 stopped (`spike-lxc`); 9999 stopped (`felhom-selftest-scratch`).
- **Infra containers (9201):** felhom-controller, cloudflared, traefik, filebrowser — all Up 5 days, healthy.
- **App containers (9201):** 24 containers Up 18h. All healthy **except `komga` = unhealthy** (Phase-1 target).
- **Disk (9201):** rootfs `/` 32G (4% used); `/var/lib/docker` 252G (8%, 19G); `/mnt/felhom-drives` (pve-root) 94G (31%).
- **Drives:** `felhom-flash` sdc1 118G (~159M used) `/mnt/felhom-drives/felhom-flash`; `felhom-usb` sdb1 916G (~2M used) `/mnt/felhom-drives/felhom-usb`.
- **PBS storage:** `felhom-pbs` active, 9.6 TB total, 21% used. Nightly 9201 backups present (latest `2026-06-21T18:55:12Z`, ~19.5 GB).
- **pvesm:** felhom-flash, felhom-usb, felhom-pbs, local, local-lvm — all active.

---

## Phase 0 — Baseline + safety net

State: 9201 healthy, controller up, tunnel up (cloudflared Up 5 days).

| Check | Result | Evidence |
|---|---|---|
| Baseline capture | PASS | see above |
| GUI screenshots (before) | _pending_ | Chrome bridge check next |
| Fresh PBS whole-CT backup of 9201 | **PASS** | volid `felhom-pbs:backup/ct/9201/2026-06-22T12:45:37Z`, 19.62 GB, crash-consistent=true, 161.5s, success=true. uncovered_volumes (by design): /etc/felhom-bootstrap, /mnt/felhom-drives (drives backed up separately) |
| PBS verify of new snapshot | **PASS** | `verify_state: ok` for backup_id 9201 backup_time 2026-06-22T12:45:37Z (datastore felhom-spike, verify UPID present) |

**DR source for Phase 4:** `felhom-pbs:backup/ct/9201/2026-06-22T12:45:37Z` (verified ok). Gate to destructive phases: **OPEN**.

**GUI render smoke-test (before):** dashboard PASS (17 running / 0 stopped / 55 total; mem 5.8/12 GB; CPU 5%; 58°C; v0.73.0; app-backup "Még nem futott"), `/monitoring` PASS, `/stacks` PASS.

### Access path note (important finding)
- On-LAN, split-horizon DNS (resolver 192.168.0.250) maps `*.demo-felhom.eu` → guest 9201 (192.168.0.141), bypassing Cloudflare (no `cf-ray` on responses).
- **The guest's traefik `:443` (websecure) returns Go `404 page not found` for ALL hosts** (controller AND apps, e.g. romm), while serving a valid LE `*.demo-felhom.eu` cert. `:80` 301-redirects to https → so the browser dead-ends at the 404. → on-LAN HTTPS to the dashboard/apps is non-functional. (The supported external path is Cloudflare; couldn't validate it from the LAN due to DNS hijack.) **FINDING — flag for supervised review.**
- **Workaround used for UI testing (reversible):** a python TCP forwarder inside guest 9201 (`/tmp/felhom-test-fwd.py`, listens `0.0.0.0:18080` → controller `172.18.0.10:8080`). Browser → `http://felhom.demo-felhom.eu:18080/` (LAN DNS → guest:18080 → controller). `CatchAllMiddleware` strips the port so Host matches `felhom.<domain>` → dashboard serves over plain HTTP. **To be killed at cleanup.**
- The dashboard currently serves with **no auth wall** (`/` → 200, no login redirect, no session cookie) — no password configured/enforced. (Bears on Phase 6 "password change" step.)

### Scope reality vs runbook
- Runbook assumed "14 enabled apps"; catalog is now **55 total, 38 not-deployed**. Deploying all 38 unattended would exhaust the 12 GiB guest. **Phase 2 will deploy a representative sample** (a few light apps) + cite the already-deployed heavy multi-container case (immich, 4 containers healthy). Documented, not silently truncated.

**Phase 0 verdict: PASS — gate OPEN.**

---

## Phase 1 — komga HC fix + git-sync + health detection — **PASS**

State: 9201 healthy, controller up, tunnel up.

**Diagnosis (live, guest 9201):** docker HC `curl -f .../api/v1/actuator/health` → exit 22 (HTTP 401, auth-gated) → `unhealthy` for 18h while serving fine on :25600. Endpoint probe matrix (wget in image): `/`, `/actuator/health`, `/api/v1/oauth2/providers`, `/login` all 200; only `/api/v1/actuator/health` → 401. Image ships `curl` (`/usr/bin/curl`). **Found a second probe:** the controller's own `.felhom.yml` `healthcheck.checks[].path` (drives the UI badge + route publishing) also pointed at the 401 path.

| Check | Result | Evidence |
|---|---|---|
| Catalog HC fix (docker-compose.yml) | PASS | `/api/v1/actuator/health` → `/actuator/health`; commit `3faa5ae` pushed to `main` |
| Catalog probe fix (.felhom.yml) | PASS | controller-side probe path fixed; commit `9b066de` pushed to `main` |
| CHANGELOG + REPORT (app-catalog) | PASS | created `CHANGELOG.md` (newest-on-top) + overwrote `REPORT.md` (`f51f794`) |
| GUI git-sync ("Sablonok frissítése") | PASS | logs: `[sync] Updated komga/docker-compose.yml` + `komga/.felhom.yml`; `Sablonok frissítve — frissítve: komga`. Live stack `/opt/docker/stacks/komga/docker-compose.yml` updated to `/actuator/health` (content-hash copy verified) |
| GUI redeploy ("Frissítés") | PASS | `POST /api/stacks/komga/update` → 200; `docker compose up -d --remove-orphans`; "Stack komga updated successfully (4.2s)"; container recreated (StartedAt 2026-06-22T13:09:50Z) |
| komga reaches healthy (docker) | PASS | `docker ps`: `komga Up (healthy)`; HC now `http://localhost:25600/actuator/health` |
| Controller health-state machine | PASS | controller log `Health probes: N ok (of N probed)` (no more komga WARN); card badge `Nem egészséges` → **Fut** |
| Route publishing | PASS | "⚠ URL nem elérhető – útvonal nincs publikálva" warning cleared; `komga.demo-felhom.eu ↗` link shown |

Minor cosmetic note (not fixed): a few catalog logo SVGs 404 (`adventurelog-logo.svg`, `cloudflared-logo.svg`, `crafty-controller-logo.svg`).
## Phase 2 — App deployment sweep — **PASS (with 1 logged catalog defect)**

State: 9201 healthy, controller up, tunnel up. Scope: representative sample (see Phase-0 scope note) — catalog has 38 not-deployed apps; deploying all on a 12 GiB guest is unsafe.

| App | Result | Evidence |
|---|---|---|
| **gitea** (deploy via GUI) | PASS (after HC fix) | GUI deploy page → memory-gate bar (`6043/11904 MB 50%`, two-segment current+new) → 3-step progress panel (✅ Konfiguráció mentve → ✅ Konténerek elindultak → ⏳ inicializálás) → `POST /api/stacks/gitea/update` 200. Initially **unhealthy**: HC `curl -f /api/v1/version` → 404 (404 until install-lock) while `/api/healthz` → 200. **Fixed catalog** (commit `17e00b7` + CHANGELOG `1a44c14`), re-synced, redeployed → `docker inspect`: `healthy`, HC now `/api/healthz`; endpoint `http://127.0.0.1:3000/` → `HTTP/1.1 200 OK`. |
| **glance** (deploy via GUI) | **FAIL — logged, not fixed** | GUI deploy → 3-step panel OK, but container `Restarting (1)` crash-loop. Logs: `parsing config: reading main YAML file: open /app/config/glance.yml: no such file or directory`. **Root cause:** catalog template mounts an empty `glance_config:/app/config` named volume but never seeds the required `glance.yml`; glance refuses to start without it. **Not a trivial HC/start_period fix** (needs a seeded default config asset + wiring) → logged per runbook, not fixed unattended. Cleaned up: `POST .../stop` + `POST .../remove` → container gone. |
| **immich** (heavy multi-container, pre-deployed) | PASS (cited) | 4 containers (server/ML/postgres/redis) all `Up (healthy)` — the heavy-app coverage case. |

**Bonus coverage:** stop + remove endpoints exercised (glance teardown). **Catalog defects found this phase:** gitea HC (fixed), glance missing-config (logged for supervised fix).
## Phase 3 — Backup + restore — **PASS (2 findings)**

State: 9201 healthy, controller up, tunnel up.

| Check | Result | Evidence |
|---|---|---|
| Manual DB-dump backup ("Mentés most") | PASS | `POST /api/backup/run` → 3 DBs dumped: romm-mariadb 38.5 KB/14 tbl, paperless-postgres 360 KB/67 tbl, immich-postgres 43.2 MB/60 tbl; 43.6 MB total in 2.18s; `status.success=true`, `last_run 2026-06-22T13:27:26Z`. Files on disk fresh **13:28**: `…/primary/immich/db-dumps/immich-postgres.sql` (45.3 MB), `…/primary/romm/db-dumps/romm-mariadb.sql` (39 KB) |
| Cross-drive copy (Tier-2) | PASS | `POST /api/backup/tier2` → 8 HDD apps copied to `/mnt/sys_drive/felhom-data/backups/secondary/<app>` (immich 155.3 MB verified 156 MB on disk; romm has appdata + recovery-unit{db-dump,manifest,compose}). `crossdrive_completed` events pushed |
| Backup page reflects status | PASS | page shows "Adatmentés aktív", 3 DBs, schedule 02:30, last DB backup timestamp; status API success=true |
| Restore flow (per-app) | PASS | `POST /backup/restore stack_name=romm` → `RestoreFromRecoveryUnit`: compose down → redeploy from recovery unit (6 env, 3 encrypted) → "Imported DB dump romm-mariadb.sql into romm-db" → containers up → "Restore-from-unit completed: romm" |
| Restore is non-hollow | PASS | post-restore romm DB has real `platforms` table + 17 tables; all 3 romm containers `Up (healthy)`. Marker-table test: planted `restore_test_marker`(id=42) pre-restore → **survived** restore (see finding) but real snapshot tables were reimported; marker dropped afterward (DB back to 17-table baseline) |

**Findings:**
1. **`GET /api/backup/snapshots` has no server handler** (api-router returns "endpoint not found"). The restore UI's snapshot dropdown (`backups.html:682`) can't populate → a pure-UI restore is blocked at snapshot selection. Restore itself works via `POST /backup/restore` (the form's action). **Flag for fix.**
2. **Restore DB import is additive** — `RestoreFromRecoveryUnit` replays the dump but does not drop tables absent from it (my marker table persisted). Real snapshot data is restored; stale/extra tables are not pruned. Minor behavioral note.
3. Whole-server ("Rendszermentés") backup shows **"A host-ügynök jelenleg nem elérhető"** (agent local API unreachable from controller) — same root as the monitoring host-metrics gap; tracked in Phase 6.
## Phase 4 — DR rebuild from backup — **PASS** (non-destructive to 9201)

State: 9201 healthy throughout, controller up, tunnel up (verified before + after).

| Check | Result | Evidence |
|---|---|---|
| Agent DR primitive (`--selftest=restore-test`) | PASS | restored `felhom-pbs:backup/ct/9201/2026-06-22T12:45:37Z` into scratch **990000** (band 990000-990009, no collision) → bind-mounts neutralized → **net link-down** → boot → `verified: boot+running` → torn down. `pass:true`, 2m56s, self-cleaning |
| Deep DR restore into throwaway **9300** | PASS | `pct restore 9300 <archive> --storage local-lvm`. **Safety:** before start, removed shared binds `mp3:/mnt/felhom-drives` + `mp9:/etc/felhom-bootstrap`, removed `net0`, set `onboot 0` — verified config clean (only restored `mp0` docker-data + rootfs remain). |
| 9300 boots + controller process up | PASS | `pct status 9300: running`; `felhom-controller Up (healthy)` inside |
| Restored volumes + app.yaml + stacks intact | PASS | `/opt/docker/stacks/` has all 57 stack dirs; romm `app.yaml` restored (`deployed:true`, env, locked_fields) |
| `docker ps` inside 9300 | PASS | **25 containers up** from restored volumes — identical set to 9201 (filebrowser, vikunja, uptime-kuma, romm×3, radarr, privatebin, paperless×3, mealie, komga, jellyfin, immich×4, gokapi, calibre-web, audiobookshelf, actualbudget, felhom-controller healthy, cloudflared, traefik) |
| Restored data non-hollow | PASS | 9300 `romm-db`: 17 tables (the real restored DB) |
| No collision with live 9201 | PASS | 9300 has **no eth0** (net removed) → cloudflared cannot reach Cloudflare/hub; live 9201 `cloudflared Up 5 days` unchanged throughout |
| Cleanup | PASS | `pct stop 9300 && pct destroy 9300` → both LVs removed; `pct list` shows only 9001/9201/9999; local-lvm freed to 9.71% |

**Core DR claim proven end-to-end:** PBS whole-CT restore → guest boots → controller + all app stacks + DB data recovered from restored volumes — twice (agent self-cleaning primitive + manual deep inspection), both collision-safe per §0.6.
## Phase 5 — Storage lifecycle — **PARTIAL (blocked by a controller→agent connectivity defect)**

State: 9201 healthy, controller up, tunnel up. Agent (`felhom-agent v0.39.0`) healthy on the host.

### 🔴 HEADLINE FINDING — controller cannot reach the agent local API (8443)
The controller container's calls to the agent local API fail: `agentapi: GET /disks: dial tcp 192.168.0.162:8443: connect: cannot assign requested address` (and same for `/host/metrics`). Reachability matrix **from the controller container**:
- `192.168.0.162:8006` (Proxmox pveproxy, binds `0.0.0.0`) → **200** (reachable)
- `192.168.0.162:8443` (felhom-agent local API, binds **`192.168.0.162`** specifically) → **refused/EADDRNOTAVAIL**
- `192.168.0.141:443` (guest's own traefik) → 404 (reachable)
From the guest **host namespace**, `192.168.0.162:8443` is OPEN. So the agent is up and listening, but its 8443 socket is unreachable specifically from the **docker container** path while 8006 on the same host is fine — consistent with the agent binding only to the host LAN IP vs a NAT/interface mismatch on the container→host path. **Confirmed:** `ss` shows agent `LISTEN 192.168.0.162:8443` (bound to the specific LAN IP, not `0.0.0.0` like pveproxy:8006); **no** iptables rule references 8443; pve-firewall disabled. So it is not a firewall — likely a container→host-IP routing/source-bind mismatch (the docker-NAT'd path to the host's own `.162` fails for the agent socket while pveproxy's `0.0.0.0` socket is reachable). Suggested fix direction (supervised): bind the agent local API to `0.0.0.0` (or the vmbr0 bridge address the guest routes through), or point the controller's `local_api.endpoint` at a container-reachable address. **Impact: the whole agent-backed feature set is down in this demo** — storage management UI (`/api/disks`, `/api/storage`), host metrics (monitoring "gazdagép metrikái nem elérhetők"), and whole-guest backup ("A host-ügynök jelenleg nem elérhető"). **Needs supervised network/bind diagnosis** (likely agent listen-address or PVE firewall on 8443). Not fixed unattended (network reconfig is out of scope per §0.1/§0.8).

| Check | Result | Evidence |
|---|---|---|
| Scan / observe / health / FS / model / durable_id (agent tier) | PASS | `--selftest=storage`: 5 targets — felhom-flash (usb, attached, class=slow, `uuid:81a26531…`, /dev/sdc1, /mnt/felhom-flash); **felhom-usb (local-dir, `uuid:da9e7089…`, /dev/sdb1, SMART=PASSED temp=35 poh=2736 realloc=0)**; local-lvm (thin data=9.7% meta=0.8%); felhom-pbs (21%); local. All `reach=true` |
| Label edit + revert (via UI) | SKIPPED | blocked — controller→agent path down (headline finding); UI storage mgmt non-functional |
| Eject → re-enroll cycle (via UI) | SKIPPED | blocked — same; not driven directly at agent-tier (would mutate live-drive state via the broken path; unattended caution per §0.7) |
| Destructive loopback (scan→format→mount→migrate→decommission) | SKIPPED | the init wizard drives the agent via the broken path; doing it purely agent-side needs the per-guest local-API token + TLS-pin reconstruction — high-effort/low-confidence unattended. Deferred to supervised. No scratch loopback was created (nothing to clean up). |
## Phase 6 — Monitoring / alerts / notifications / settings / hub — **PASS**

State: 9201 healthy, controller up, tunnel up.

| Check | Result | Evidence |
|---|---|---|
| Monitoring — host/server metrics | PASS | `/monitoring`: CPU 12.4%, load 0.35/0.76/1.35, mem 7.6/15.4 GB (host's 16 GB → **/proc/LXC kernel-shared view**, not the agent API), CPU temp 43 °C, uptime 5d21h, "Frissítve 16:03:12". (The Phase-0 momentary "gazdagép metrikái nem elérhetők" did not persist.) |
| Monitoring — container/app metrics | PASS | "Memória eloszlás" per-container breakdown (felhom-controller 1.1 GB, paperless 594 MB, immich-server 414 MB, komga 387 MB, gitea 85 MB, …26 containers); "Alkalmazás erőforrások" per-app CPU/MEM charts; time-range charts 1h/6h/24h/7d/30d |
| Monitoring — storage capacity | PASS | all 5 stores with capacities + felhom-usb SMART temp 35 °C (cached from a prior agent poll) |
| Alerts — state-based (komga) | PASS | controller logs: komga `Health probes: 0 ok, 1 unhealthy` before fix → `Health probes: N ok` after (Phase 1); UI badge red→green; route-warning cleared |
| Notification path (exercised ONCE) | PASS | set email + 2 events + cooldown=2h → `POST /settings/notifications` 200 → hub log `Notification preferences updated for demo-felhom`. Test: `SendTest()` → hub log **`Event from demo-felhom: test (info)` → `Test email sent to nagyfenyvesi.viktor@gmail.com for demo-felhom`** = controller→hub→Resend→email end-to-end. Only one email sent (§0.5). |
| Cooldown / dup-suppression | PARTIAL | cooldown_hours=2 persisted + synced to hub (mechanism present); not stress-tested (would require generating duplicate alerts — avoided per §0.5) |
| Settings persistence | PASS | notification prefs set → persisted to `settings.json` (email/cooldown/events) → reverted (email cleared, events None) — both directions verified |
| Settings — password change | N/A | password protection is **operator-only** and **not configured** ("A jelszavas védelem nincs beállítva. Kérd az üzemeltetőt"); no customer-UI password change to test. Dashboard currently serves with no auth wall. |
| Hub — report received | PASS | hub logs: `Received report from demo-felhom (11983 bytes)` every 15 min; `host-report from demo-felhom-01 (3 guests, 5 storage targets, 1 backups, 1 restore-tests, 9 pbs-snapshots)`; UI "Hub kapcsolat: Kapcsolódva, utolsó jelentés 8 perce" |
| Hub — DR recipe carries PBS coord | PASS | hub DB `dr_recipe` (demo-felhom): host-half `pbs={"repo_id":"felhom-pbs","namespace":"root","latest_snapshot_id":"9201"}`; app-half customer demo-felhom / 15 apps. No secrets in either half. |
| Hub — update panel | PASS | settings "Verzió és frissítés": current 0.73.0, latest "0.73.0 — naprakész", last check 2026-06-22T10:54:46Z |
| Hub GUI tiles (analytics/healthchecks) | SKIPPED(no operator creds) | verified hub-side state via logs + DB instead of the auth-gated operator GUI |

**Note (reconciles Phase 5):** the agent local-API path stays **persistently** down (6/6 `EADDRNOTAVAIL` on `/api/disks`). Monitoring host metrics work anyway because they are `/proc`-derived (LXC shares the host kernel); the agent gap only blocks features needing a *live* agent call (raw-disk view, storage mgmt, whole-guest backup trigger).
## Phase 7 — Resilience (optional) — _pending_

---

## Feature matrix (filled at end)
## Cleanup confirmation (filled at end)
## Deferred — needs supervised run (filled at end)