19 KiB
19 KiB
TEST-REPORT — Unattended test campaign (N100 / guest 9201)
Run start: 2026-06-22 (CC, unattended) Operator: Viktor (away, glancing remotely) Scope: RUNBOOK phased test matrix. Only code change permitted = komga catalog HC fix. Legend: PASS / FAIL / SKIPPED(reason). Evidence is raw (command output / volid / screenshot path).
Baseline state (Phase 0 "before")
- Host: felhom-pve (
demo-felhom), agent v0.39.0 (service active). - Controller: image
gitea.dooplex.hu/admin/felhom-controller:0.73.0, containerUp 5 days (healthy). pct list: 9201 running (demo-felhom); 9001 stopped (spike-lxc); 9999 stopped (felhom-selftest-scratch).- Infra containers (9201): felhom-controller, cloudflared, traefik, filebrowser — all Up 5 days, healthy.
- App containers (9201): 24 containers Up 18h. All healthy except
komga= unhealthy (Phase-1 target). - Disk (9201): rootfs
/32G (4% used);/var/lib/docker252G (8%, 19G);/mnt/felhom-drives(pve-root) 94G (31%). - Drives:
felhom-flashsdc1 118G (~159M used)/mnt/felhom-drives/felhom-flash;felhom-usbsdb1 916G (~2M used)/mnt/felhom-drives/felhom-usb. - PBS storage:
felhom-pbsactive, 9.6 TB total, 21% used. Nightly 9201 backups present (latest2026-06-21T18:55:12Z, ~19.5 GB). - pvesm: felhom-flash, felhom-usb, felhom-pbs, local, local-lvm — all active.
Phase 0 — Baseline + safety net
State: 9201 healthy, controller up, tunnel up (cloudflared Up 5 days).
| Check | Result | Evidence |
|---|---|---|
| Baseline capture | PASS | see above |
| GUI screenshots (before) | pending | Chrome bridge check next |
| Fresh PBS whole-CT backup of 9201 | PASS | volid felhom-pbs:backup/ct/9201/2026-06-22T12:45:37Z, 19.62 GB, crash-consistent=true, 161.5s, success=true. uncovered_volumes (by design): /etc/felhom-bootstrap, /mnt/felhom-drives (drives backed up separately) |
| PBS verify of new snapshot | PASS | verify_state: ok for backup_id 9201 backup_time 2026-06-22T12:45:37Z (datastore felhom-spike, verify UPID present) |
DR source for Phase 4: felhom-pbs:backup/ct/9201/2026-06-22T12:45:37Z (verified ok). Gate to destructive phases: OPEN.
GUI render smoke-test (before): dashboard PASS (17 running / 0 stopped / 55 total; mem 5.8/12 GB; CPU 5%; 58°C; v0.73.0; app-backup "Még nem futott"), /monitoring PASS, /stacks PASS.
Access path note (important finding)
- On-LAN, split-horizon DNS (resolver 192.168.0.250) maps
*.demo-felhom.eu→ guest 9201 (192.168.0.141), bypassing Cloudflare (nocf-rayon responses). - The guest's traefik
:443(websecure) returns Go404 page not foundfor ALL hosts (controller AND apps, e.g. romm), while serving a valid LE*.demo-felhom.eucert.:80301-redirects to https → so the browser dead-ends at the 404. → on-LAN HTTPS to the dashboard/apps is non-functional. (The supported external path is Cloudflare; couldn't validate it from the LAN due to DNS hijack.) FINDING — flag for supervised review. - Workaround used for UI testing (reversible): a python TCP forwarder inside guest 9201 (
/tmp/felhom-test-fwd.py, listens0.0.0.0:18080→ controller172.18.0.10:8080). Browser →http://felhom.demo-felhom.eu:18080/(LAN DNS → guest:18080 → controller).CatchAllMiddlewarestrips the port so Host matchesfelhom.<domain>→ dashboard serves over plain HTTP. To be killed at cleanup. - The dashboard currently serves with no auth wall (
/→ 200, no login redirect, no session cookie) — no password configured/enforced. (Bears on Phase 6 "password change" step.)
Scope reality vs runbook
- Runbook assumed "14 enabled apps"; catalog is now 55 total, 38 not-deployed. Deploying all 38 unattended would exhaust the 12 GiB guest. Phase 2 will deploy a representative sample (a few light apps) + cite the already-deployed heavy multi-container case (immich, 4 containers healthy). Documented, not silently truncated.
Phase 0 verdict: PASS — gate OPEN.
Phase 1 — komga HC fix + git-sync + health detection — PASS
State: 9201 healthy, controller up, tunnel up.
Diagnosis (live, guest 9201): docker HC curl -f .../api/v1/actuator/health → exit 22 (HTTP 401, auth-gated) → unhealthy for 18h while serving fine on :25600. Endpoint probe matrix (wget in image): /, /actuator/health, /api/v1/oauth2/providers, /login all 200; only /api/v1/actuator/health → 401. Image ships curl (/usr/bin/curl). Found a second probe: the controller's own .felhom.yml healthcheck.checks[].path (drives the UI badge + route publishing) also pointed at the 401 path.
| Check | Result | Evidence |
|---|---|---|
| Catalog HC fix (docker-compose.yml) | PASS | /api/v1/actuator/health → /actuator/health; commit 3faa5ae pushed to main |
| Catalog probe fix (.felhom.yml) | PASS | controller-side probe path fixed; commit 9b066de pushed to main |
| CHANGELOG + REPORT (app-catalog) | PASS | created CHANGELOG.md (newest-on-top) + overwrote REPORT.md (f51f794) |
| GUI git-sync ("Sablonok frissítése") | PASS | logs: [sync] Updated komga/docker-compose.yml + komga/.felhom.yml; Sablonok frissítve — frissítve: komga. Live stack /opt/docker/stacks/komga/docker-compose.yml updated to /actuator/health (content-hash copy verified) |
| GUI redeploy ("Frissítés") | PASS | POST /api/stacks/komga/update → 200; docker compose up -d --remove-orphans; "Stack komga updated successfully (4.2s)"; container recreated (StartedAt 2026-06-22T13:09:50Z) |
| komga reaches healthy (docker) | PASS | docker ps: komga Up (healthy); HC now http://localhost:25600/actuator/health |
| Controller health-state machine | PASS | controller log Health probes: N ok (of N probed) (no more komga WARN); card badge Nem egészséges → Fut |
| Route publishing | PASS | "⚠ URL nem elérhető – útvonal nincs publikálva" warning cleared; komga.demo-felhom.eu ↗ link shown |
Minor cosmetic note (not fixed): a few catalog logo SVGs 404 (adventurelog-logo.svg, cloudflared-logo.svg, crafty-controller-logo.svg).
Phase 2 — App deployment sweep — PASS (with 1 logged catalog defect)
State: 9201 healthy, controller up, tunnel up. Scope: representative sample (see Phase-0 scope note) — catalog has 38 not-deployed apps; deploying all on a 12 GiB guest is unsafe.
| App | Result | Evidence |
|---|---|---|
| gitea (deploy via GUI) | PASS (after HC fix) | GUI deploy page → memory-gate bar (6043/11904 MB 50%, two-segment current+new) → 3-step progress panel (✅ Konfiguráció mentve → ✅ Konténerek elindultak → ⏳ inicializálás) → POST /api/stacks/gitea/update 200. Initially unhealthy: HC curl -f /api/v1/version → 404 (404 until install-lock) while /api/healthz → 200. Fixed catalog (commit 17e00b7 + CHANGELOG 1a44c14), re-synced, redeployed → docker inspect: healthy, HC now /api/healthz; endpoint http://127.0.0.1:3000/ → HTTP/1.1 200 OK. |
| glance (deploy via GUI) | FAIL — logged, not fixed | GUI deploy → 3-step panel OK, but container Restarting (1) crash-loop. Logs: parsing config: reading main YAML file: open /app/config/glance.yml: no such file or directory. Root cause: catalog template mounts an empty glance_config:/app/config named volume but never seeds the required glance.yml; glance refuses to start without it. Not a trivial HC/start_period fix (needs a seeded default config asset + wiring) → logged per runbook, not fixed unattended. Cleaned up: POST .../stop + POST .../remove → container gone. |
| immich (heavy multi-container, pre-deployed) | PASS (cited) | 4 containers (server/ML/postgres/redis) all Up (healthy) — the heavy-app coverage case. |
Bonus coverage: stop + remove endpoints exercised (glance teardown). Catalog defects found this phase: gitea HC (fixed), glance missing-config (logged for supervised fix).
Phase 3 — Backup + restore — PASS (2 findings)
State: 9201 healthy, controller up, tunnel up.
| Check | Result | Evidence |
|---|---|---|
| Manual DB-dump backup ("Mentés most") | PASS | POST /api/backup/run → 3 DBs dumped: romm-mariadb 38.5 KB/14 tbl, paperless-postgres 360 KB/67 tbl, immich-postgres 43.2 MB/60 tbl; 43.6 MB total in 2.18s; status.success=true, last_run 2026-06-22T13:27:26Z. Files on disk fresh 13:28: …/primary/immich/db-dumps/immich-postgres.sql (45.3 MB), …/primary/romm/db-dumps/romm-mariadb.sql (39 KB) |
| Cross-drive copy (Tier-2) | PASS | POST /api/backup/tier2 → 8 HDD apps copied to /mnt/sys_drive/felhom-data/backups/secondary/<app> (immich 155.3 MB verified 156 MB on disk; romm has appdata + recovery-unit{db-dump,manifest,compose}). crossdrive_completed events pushed |
| Backup page reflects status | PASS | page shows "Adatmentés aktív", 3 DBs, schedule 02:30, last DB backup timestamp; status API success=true |
| Restore flow (per-app) | PASS | POST /backup/restore stack_name=romm → RestoreFromRecoveryUnit: compose down → redeploy from recovery unit (6 env, 3 encrypted) → "Imported DB dump romm-mariadb.sql into romm-db" → containers up → "Restore-from-unit completed: romm" |
| Restore is non-hollow | PASS | post-restore romm DB has real platforms table + 17 tables; all 3 romm containers Up (healthy). Marker-table test: planted restore_test_marker(id=42) pre-restore → survived restore (see finding) but real snapshot tables were reimported; marker dropped afterward (DB back to 17-table baseline) |
Findings:
GET /api/backup/snapshotshas no server handler (api-router returns "endpoint not found"). The restore UI's snapshot dropdown (backups.html:682) can't populate → a pure-UI restore is blocked at snapshot selection. Restore itself works viaPOST /backup/restore(the form's action). Flag for fix.- Restore DB import is additive —
RestoreFromRecoveryUnitreplays the dump but does not drop tables absent from it (my marker table persisted). Real snapshot data is restored; stale/extra tables are not pruned. Minor behavioral note. - Whole-server ("Rendszermentés") backup shows "A host-ügynök jelenleg nem elérhető" (agent local API unreachable from controller) — same root as the monitoring host-metrics gap; tracked in Phase 6.
Phase 4 — DR rebuild from backup — PASS (non-destructive to 9201)
State: 9201 healthy throughout, controller up, tunnel up (verified before + after).
| Check | Result | Evidence |
|---|---|---|
Agent DR primitive (--selftest=restore-test) |
PASS | restored felhom-pbs:backup/ct/9201/2026-06-22T12:45:37Z into scratch 990000 (band 990000-990009, no collision) → bind-mounts neutralized → net link-down → boot → verified: boot+running → torn down. pass:true, 2m56s, self-cleaning |
| Deep DR restore into throwaway 9300 | PASS | pct restore 9300 <archive> --storage local-lvm. Safety: before start, removed shared binds mp3:/mnt/felhom-drives + mp9:/etc/felhom-bootstrap, removed net0, set onboot 0 — verified config clean (only restored mp0 docker-data + rootfs remain). |
| 9300 boots + controller process up | PASS | pct status 9300: running; felhom-controller Up (healthy) inside |
| Restored volumes + app.yaml + stacks intact | PASS | /opt/docker/stacks/ has all 57 stack dirs; romm app.yaml restored (deployed:true, env, locked_fields) |
docker ps inside 9300 |
PASS | 25 containers up from restored volumes — identical set to 9201 (filebrowser, vikunja, uptime-kuma, romm×3, radarr, privatebin, paperless×3, mealie, komga, jellyfin, immich×4, gokapi, calibre-web, audiobookshelf, actualbudget, felhom-controller healthy, cloudflared, traefik) |
| Restored data non-hollow | PASS | 9300 romm-db: 17 tables (the real restored DB) |
| No collision with live 9201 | PASS | 9300 has no eth0 (net removed) → cloudflared cannot reach Cloudflare/hub; live 9201 cloudflared Up 5 days unchanged throughout |
| Cleanup | PASS | pct stop 9300 && pct destroy 9300 → both LVs removed; pct list shows only 9001/9201/9999; local-lvm freed to 9.71% |
Core DR claim proven end-to-end: PBS whole-CT restore → guest boots → controller + all app stacks + DB data recovered from restored volumes — twice (agent self-cleaning primitive + manual deep inspection), both collision-safe per §0.6.
Phase 5 — Storage lifecycle — PARTIAL (blocked by a controller→agent connectivity defect)
State: 9201 healthy, controller up, tunnel up. Agent (felhom-agent v0.39.0) healthy on the host.
🔴 HEADLINE FINDING — controller cannot reach the agent local API (8443)
The controller container's calls to the agent local API fail: agentapi: GET /disks: dial tcp 192.168.0.162:8443: connect: cannot assign requested address (and same for /host/metrics). Reachability matrix from the controller container:
192.168.0.162:8006(Proxmox pveproxy, binds0.0.0.0) → 200 (reachable)192.168.0.162:8443(felhom-agent local API, binds192.168.0.162specifically) → refused/EADDRNOTAVAIL192.168.0.141:443(guest's own traefik) → 404 (reachable) From the guest host namespace,192.168.0.162:8443is OPEN. So the agent is up and listening, but its 8443 socket is unreachable specifically from the docker container path while 8006 on the same host is fine — consistent with the agent binding only to the host LAN IP vs a NAT/interface mismatch on the container→host path. Confirmed:ssshows agentLISTEN 192.168.0.162:8443(bound to the specific LAN IP, not0.0.0.0like pveproxy:8006); no iptables rule references 8443; pve-firewall disabled. So it is not a firewall — likely a container→host-IP routing/source-bind mismatch (the docker-NAT'd path to the host's own.162fails for the agent socket while pveproxy's0.0.0.0socket is reachable). Suggested fix direction (supervised): bind the agent local API to0.0.0.0(or the vmbr0 bridge address the guest routes through), or point the controller'slocal_api.endpointat a container-reachable address. Impact: the whole agent-backed feature set is down in this demo — storage management UI (/api/disks,/api/storage), host metrics (monitoring "gazdagép metrikái nem elérhetők"), and whole-guest backup ("A host-ügynök jelenleg nem elérhető"). Needs supervised network/bind diagnosis (likely agent listen-address or PVE firewall on 8443). Not fixed unattended (network reconfig is out of scope per §0.1/§0.8).
| Check | Result | Evidence |
|---|---|---|
| Scan / observe / health / FS / model / durable_id (agent tier) | PASS | --selftest=storage: 5 targets — felhom-flash (usb, attached, class=slow, uuid:81a26531…, /dev/sdc1, /mnt/felhom-flash); felhom-usb (local-dir, uuid:da9e7089…, /dev/sdb1, SMART=PASSED temp=35 poh=2736 realloc=0); local-lvm (thin data=9.7% meta=0.8%); felhom-pbs (21%); local. All reach=true |
| Label edit + revert (via UI) | SKIPPED | blocked — controller→agent path down (headline finding); UI storage mgmt non-functional |
| Eject → re-enroll cycle (via UI) | SKIPPED | blocked — same; not driven directly at agent-tier (would mutate live-drive state via the broken path; unattended caution per §0.7) |
| Destructive loopback (scan→format→mount→migrate→decommission) | SKIPPED | the init wizard drives the agent via the broken path; doing it purely agent-side needs the per-guest local-API token + TLS-pin reconstruction — high-effort/low-confidence unattended. Deferred to supervised. No scratch loopback was created (nothing to clean up). |
Phase 6 — Monitoring / alerts / notifications / settings / hub — PASS
State: 9201 healthy, controller up, tunnel up.
| Check | Result | Evidence |
|---|---|---|
| Monitoring — host/server metrics | PASS | /monitoring: CPU 12.4%, load 0.35/0.76/1.35, mem 7.6/15.4 GB (host's 16 GB → /proc/LXC kernel-shared view, not the agent API), CPU temp 43 °C, uptime 5d21h, "Frissítve 16:03:12". (The Phase-0 momentary "gazdagép metrikái nem elérhetők" did not persist.) |
| Monitoring — container/app metrics | PASS | "Memória eloszlás" per-container breakdown (felhom-controller 1.1 GB, paperless 594 MB, immich-server 414 MB, komga 387 MB, gitea 85 MB, …26 containers); "Alkalmazás erőforrások" per-app CPU/MEM charts; time-range charts 1h/6h/24h/7d/30d |
| Monitoring — storage capacity | PASS | all 5 stores with capacities + felhom-usb SMART temp 35 °C (cached from a prior agent poll) |
| Alerts — state-based (komga) | PASS | controller logs: komga Health probes: 0 ok, 1 unhealthy before fix → Health probes: N ok after (Phase 1); UI badge red→green; route-warning cleared |
| Notification path (exercised ONCE) | PASS | set email + 2 events + cooldown=2h → POST /settings/notifications 200 → hub log Notification preferences updated for demo-felhom. Test: SendTest() → hub log Event from demo-felhom: test (info) → Test email sent to nagyfenyvesi.viktor@gmail.com for demo-felhom = controller→hub→Resend→email end-to-end. Only one email sent (§0.5). |
| Cooldown / dup-suppression | PARTIAL | cooldown_hours=2 persisted + synced to hub (mechanism present); not stress-tested (would require generating duplicate alerts — avoided per §0.5) |
| Settings persistence | PASS | notification prefs set → persisted to settings.json (email/cooldown/events) → reverted (email cleared, events None) — both directions verified |
| Settings — password change | N/A | password protection is operator-only and not configured ("A jelszavas védelem nincs beállítva. Kérd az üzemeltetőt"); no customer-UI password change to test. Dashboard currently serves with no auth wall. |
| Hub — report received | PASS | hub logs: Received report from demo-felhom (11983 bytes) every 15 min; host-report from demo-felhom-01 (3 guests, 5 storage targets, 1 backups, 1 restore-tests, 9 pbs-snapshots); UI "Hub kapcsolat: Kapcsolódva, utolsó jelentés 8 perce" |
| Hub — DR recipe carries PBS coord | PASS | hub DB dr_recipe (demo-felhom): host-half pbs={"repo_id":"felhom-pbs","namespace":"root","latest_snapshot_id":"9201"}; app-half customer demo-felhom / 15 apps. No secrets in either half. |
| Hub — update panel | PASS | settings "Verzió és frissítés": current 0.73.0, latest "0.73.0 — naprakész", last check 2026-06-22T10:54:46Z |
| Hub GUI tiles (analytics/healthchecks) | SKIPPED(no operator creds) | verified hub-side state via logs + DB instead of the auth-gated operator GUI |
Note (reconciles Phase 5): the agent local-API path stays persistently down (6/6 EADDRNOTAVAIL on /api/disks). Monitoring host metrics work anyway because they are /proc-derived (LXC shares the host kernel); the agent gap only blocks features needing a live agent call (raw-disk view, storage mgmt, whole-guest backup trigger).