Files
felhom-controller/TEST-REPORT.md
T
2026-06-22 15:25:51 +02:00

8.2 KiB
Raw Blame History

TEST-REPORT — Unattended test campaign (N100 / guest 9201)

Run start: 2026-06-22 (CC, unattended) Operator: Viktor (away, glancing remotely) Scope: RUNBOOK phased test matrix. Only code change permitted = komga catalog HC fix. Legend: PASS / FAIL / SKIPPED(reason). Evidence is raw (command output / volid / screenshot path).


Baseline state (Phase 0 "before")

  • Host: felhom-pve (demo-felhom), agent v0.39.0 (service active).
  • Controller: image gitea.dooplex.hu/admin/felhom-controller:0.73.0, container Up 5 days (healthy).
  • pct list: 9201 running (demo-felhom); 9001 stopped (spike-lxc); 9999 stopped (felhom-selftest-scratch).
  • Infra containers (9201): felhom-controller, cloudflared, traefik, filebrowser — all Up 5 days, healthy.
  • App containers (9201): 24 containers Up 18h. All healthy except komga = unhealthy (Phase-1 target).
  • Disk (9201): rootfs / 32G (4% used); /var/lib/docker 252G (8%, 19G); /mnt/felhom-drives (pve-root) 94G (31%).
  • Drives: felhom-flash sdc1 118G (~159M used) /mnt/felhom-drives/felhom-flash; felhom-usb sdb1 916G (~2M used) /mnt/felhom-drives/felhom-usb.
  • PBS storage: felhom-pbs active, 9.6 TB total, 21% used. Nightly 9201 backups present (latest 2026-06-21T18:55:12Z, ~19.5 GB).
  • pvesm: felhom-flash, felhom-usb, felhom-pbs, local, local-lvm — all active.

Phase 0 — Baseline + safety net

State: 9201 healthy, controller up, tunnel up (cloudflared Up 5 days).

Check Result Evidence
Baseline capture PASS see above
GUI screenshots (before) pending Chrome bridge check next
Fresh PBS whole-CT backup of 9201 PASS volid felhom-pbs:backup/ct/9201/2026-06-22T12:45:37Z, 19.62 GB, crash-consistent=true, 161.5s, success=true. uncovered_volumes (by design): /etc/felhom-bootstrap, /mnt/felhom-drives (drives backed up separately)
PBS verify of new snapshot PASS verify_state: ok for backup_id 9201 backup_time 2026-06-22T12:45:37Z (datastore felhom-spike, verify UPID present)

DR source for Phase 4: felhom-pbs:backup/ct/9201/2026-06-22T12:45:37Z (verified ok). Gate to destructive phases: OPEN.

GUI render smoke-test (before): dashboard PASS (17 running / 0 stopped / 55 total; mem 5.8/12 GB; CPU 5%; 58°C; v0.73.0; app-backup "Még nem futott"), /monitoring PASS, /stacks PASS.

Access path note (important finding)

  • On-LAN, split-horizon DNS (resolver 192.168.0.250) maps *.demo-felhom.eu → guest 9201 (192.168.0.141), bypassing Cloudflare (no cf-ray on responses).
  • The guest's traefik :443 (websecure) returns Go 404 page not found for ALL hosts (controller AND apps, e.g. romm), while serving a valid LE *.demo-felhom.eu cert. :80 301-redirects to https → so the browser dead-ends at the 404. → on-LAN HTTPS to the dashboard/apps is non-functional. (The supported external path is Cloudflare; couldn't validate it from the LAN due to DNS hijack.) FINDING — flag for supervised review.
  • Workaround used for UI testing (reversible): a python TCP forwarder inside guest 9201 (/tmp/felhom-test-fwd.py, listens 0.0.0.0:18080 → controller 172.18.0.10:8080). Browser → http://felhom.demo-felhom.eu:18080/ (LAN DNS → guest:18080 → controller). CatchAllMiddleware strips the port so Host matches felhom.<domain> → dashboard serves over plain HTTP. To be killed at cleanup.
  • The dashboard currently serves with no auth wall (/ → 200, no login redirect, no session cookie) — no password configured/enforced. (Bears on Phase 6 "password change" step.)

Scope reality vs runbook

  • Runbook assumed "14 enabled apps"; catalog is now 55 total, 38 not-deployed. Deploying all 38 unattended would exhaust the 12 GiB guest. Phase 2 will deploy a representative sample (a few light apps) + cite the already-deployed heavy multi-container case (immich, 4 containers healthy). Documented, not silently truncated.

Phase 0 verdict: PASS — gate OPEN.


Phase 1 — komga HC fix + git-sync + health detection — PASS

State: 9201 healthy, controller up, tunnel up.

Diagnosis (live, guest 9201): docker HC curl -f .../api/v1/actuator/health → exit 22 (HTTP 401, auth-gated) → unhealthy for 18h while serving fine on :25600. Endpoint probe matrix (wget in image): /, /actuator/health, /api/v1/oauth2/providers, /login all 200; only /api/v1/actuator/health → 401. Image ships curl (/usr/bin/curl). Found a second probe: the controller's own .felhom.yml healthcheck.checks[].path (drives the UI badge + route publishing) also pointed at the 401 path.

Check Result Evidence
Catalog HC fix (docker-compose.yml) PASS /api/v1/actuator/health/actuator/health; commit 3faa5ae pushed to main
Catalog probe fix (.felhom.yml) PASS controller-side probe path fixed; commit 9b066de pushed to main
CHANGELOG + REPORT (app-catalog) PASS created CHANGELOG.md (newest-on-top) + overwrote REPORT.md (f51f794)
GUI git-sync ("Sablonok frissítése") PASS logs: [sync] Updated komga/docker-compose.yml + komga/.felhom.yml; Sablonok frissítve — frissítve: komga. Live stack /opt/docker/stacks/komga/docker-compose.yml updated to /actuator/health (content-hash copy verified)
GUI redeploy ("Frissítés") PASS POST /api/stacks/komga/update → 200; docker compose up -d --remove-orphans; "Stack komga updated successfully (4.2s)"; container recreated (StartedAt 2026-06-22T13:09:50Z)
komga reaches healthy (docker) PASS docker ps: komga Up (healthy); HC now http://localhost:25600/actuator/health
Controller health-state machine PASS controller log Health probes: N ok (of N probed) (no more komga WARN); card badge Nem egészségesFut
Route publishing PASS "⚠ URL nem elérhető útvonal nincs publikálva" warning cleared; komga.demo-felhom.eu ↗ link shown

Minor cosmetic note (not fixed): a few catalog logo SVGs 404 (adventurelog-logo.svg, cloudflared-logo.svg, crafty-controller-logo.svg).

Phase 2 — App deployment sweep — PASS (with 1 logged catalog defect)

State: 9201 healthy, controller up, tunnel up. Scope: representative sample (see Phase-0 scope note) — catalog has 38 not-deployed apps; deploying all on a 12 GiB guest is unsafe.

App Result Evidence
gitea (deploy via GUI) PASS (after HC fix) GUI deploy page → memory-gate bar (6043/11904 MB 50%, two-segment current+new) → 3-step progress panel ( Konfiguráció mentve → Konténerek elindultak → inicializálás) → POST /api/stacks/gitea/update 200. Initially unhealthy: HC curl -f /api/v1/version → 404 (404 until install-lock) while /api/healthz → 200. Fixed catalog (commit 17e00b7 + CHANGELOG 1a44c14), re-synced, redeployed → docker inspect: healthy, HC now /api/healthz; endpoint http://127.0.0.1:3000/HTTP/1.1 200 OK.
glance (deploy via GUI) FAIL — logged, not fixed GUI deploy → 3-step panel OK, but container Restarting (1) crash-loop. Logs: parsing config: reading main YAML file: open /app/config/glance.yml: no such file or directory. Root cause: catalog template mounts an empty glance_config:/app/config named volume but never seeds the required glance.yml; glance refuses to start without it. Not a trivial HC/start_period fix (needs a seeded default config asset + wiring) → logged per runbook, not fixed unattended. Cleaned up: POST .../stop + POST .../remove → container gone.
immich (heavy multi-container, pre-deployed) PASS (cited) 4 containers (server/ML/postgres/redis) all Up (healthy) — the heavy-app coverage case.

Bonus coverage: stop + remove endpoints exercised (glance teardown). Catalog defects found this phase: gitea HC (fixed), glance missing-config (logged for supervised fix).

Phase 3 — Backup + restore — pending

Phase 4 — DR rebuild (9300) — pending

Phase 5 — Storage lifecycle — pending

Phase 6 — Monitoring/alerts/hub — pending

Phase 7 — Resilience (optional) — pending


Feature matrix (filled at end)

Cleanup confirmation (filled at end)

Deferred — needs supervised run (filled at end)