test campaign #2: Phase 0 (baseline + safety net + leak-fix gate PASS)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-06-22 18:18:49 +02:00
parent c208ddf308
commit 1190e56a2c
1 changed files with 19 additions and 219 deletions
@@ -1,226 +1,26 @@
-# TEST-REPORT — Unattended test campaign (N100 / guest 9201)
+# TEST-REPORT — Test campaign #2 (broader apps, edge cases, break-and-recover) — N100 / guest 9201
-**Run start:** 2026-06-22 (CC, unattended)
+**Run start:** 2026-06-22 (CC, unattended). **Operator:** Viktor (away).
-**Operator:** Viktor (away, glancing remotely)
+**Scope:** push past campaign #1 — more apps, edge cases, chaos break-and-recover. Storage lifecycle now
-**Scope:** RUNBOOK phased test matrix. Code changes = komga catalog HC fix (the planned one) + gitea catalog HC fix (an analogous trivial HC bug surfaced in the Phase-2 sweep, which Phase 2 explicitly sanctions). No controller/agent code built (catalog-only).
+unblocked (ctrl v0.74.0 fixed the agent leak). Only code changes: trivial catalog HC/config-seed fixes.
-**Legend:** PASS / FAIL / SKIPPED(reason). Evidence is raw (command output / volid / screenshot path).
+**Legend:** PASS / FAIL / SKIPPED(reason). Evidence raw (docker ps / ss / curl / screenshot).
 (Campaign #1 report preserved in git history + `felhom.eu/documentation/tests/unattended-test-campaign-2026-06-22-*.md`.)
 ---
-## Baseline state (Phase 0 "before")
+## Phase 0 — Baseline + safety net + leak-fix gate — **PASS (gate OPEN)**
- **Host:** felhom-pve (`demo-felhom`), agent **v0.39.0** (service active).
+- **Baseline:** ctrl `v0.74.0` healthy, agent `v0.39.0`; `pct list` = 9201 running (+ 9001/9999 stopped, pre-existing); **25 containers, 0 unhealthy**; rootfs `/` 4%, `/var/lib/docker` 8%, drives felhom-flash 1% / felhom-usb 1%.
- **Controller:** image `gitea.dooplex.hu/admin/felhom-controller:0.73.0`, container `Up 5 days (healthy)`.
+- **Master safety net + DR source:** PBS backup `felhom-pbs:backup/ct/9201/2026-06-22T16:13:44Z`, success=true, crash-consistent; PBS verify shows the snapshot present.
- **`pct list`:** 9201 running (`demo-felhom`); 9001 stopped (`spike-lxc`); 9999 stopped (`felhom-selftest-scratch`).
+- **Leak-fix gate (the #1 unblock):** `/api/disks` lists felhom-flash/felhom-usb/local; `/api/host-metrics` returns `cpu_percent`; controller→agent idle sockets to `:8443` **flat at 2 before and after a 60-call burst**. Agent path fully working. **Gate to chaos/destructive phases: OPEN.**
- **Infra containers (9201):** felhom-controller, cloudflared, traefik, filebrowser — all Up 5 days, healthy.
+
- **App containers (9201):** 24 containers Up 18h. All healthy **except `komga` = unhealthy** (Phase-1 target).
+## Phase 1 — Broader app sweep + edge cases — _in progress_
- **Disk (9201):** rootfs `/` 32G (4% used); `/var/lib/docker` 252G (8%, 19G); `/mnt/felhom-drives` (pve-root) 94G (31%).
+## Phase 2 — Storage lifecycle (unblocked) — _pending_
- **Drives:** `felhom-flash` sdc1 118G (~159M used) `/mnt/felhom-drives/felhom-flash`; `felhom-usb` sdb1 916G (~2M used) `/mnt/felhom-drives/felhom-usb`.
+## Phase 3 — Break-and-recover chaos (C1–C8) — _pending_
- **PBS storage:** `felhom-pbs` active, 9.6 TB total, 21% used. Nightly 9201 backups present (latest `2026-06-21T18:55:12Z`, ~19.5 GB).
+## Phase 4 — Backup/restore edge cases — _pending_
- **pvesm:** felhom-flash, felhom-usb, felhom-pbs, local, local-lvm — all active.
+## Phase 5 — Host reboot (optional) — _pending_
 ---
-
+## Feature matrix — _filled at end_
-## Phase 0 — Baseline + safety net
+## Cleanup confirmation — _filled at end_
-
+## Deferred / needs supervised — _filled at end_
 State: 9201 healthy, controller up, tunnel up (cloudflared Up 5 days).
 | Check | Result | Evidence |
 |---|---|---|
 | Baseline capture | PASS | see above |
 | GUI screenshots (before) | _pending_ | Chrome bridge check next |
 | Fresh PBS whole-CT backup of 9201 | **PASS** | volid `felhom-pbs:backup/ct/9201/2026-06-22T12:45:37Z`, 19.62 GB, crash-consistent=true, 161.5s, success=true. uncovered_volumes (by design): /etc/felhom-bootstrap, /mnt/felhom-drives (drives backed up separately) |
 | PBS verify of new snapshot | **PASS** | `verify_state: ok` for backup_id 9201 backup_time 2026-06-22T12:45:37Z (datastore felhom-spike, verify UPID present) |
 **DR source for Phase 4:** `felhom-pbs:backup/ct/9201/2026-06-22T12:45:37Z` (verified ok). Gate to destructive phases: **OPEN**.
 **GUI render smoke-test (before):** dashboard PASS (17 running / 0 stopped / 55 total; mem 5.8/12 GB; CPU 5%; 58°C; v0.73.0; app-backup "Még nem futott"), `/monitoring` PASS, `/stacks` PASS.
 ### Access path note (important finding)
 - On-LAN, split-horizon DNS (resolver 192.168.0.250) maps `*.demo-felhom.eu` → guest 9201 (192.168.0.141), bypassing Cloudflare (no `cf-ray` on responses).
 - **The guest's traefik `:443` (websecure) returns Go `404 page not found` for ALL hosts** (controller AND apps, e.g. romm), while serving a valid LE `*.demo-felhom.eu` cert. `:80` 301-redirects to https → so the browser dead-ends at the 404. → on-LAN HTTPS to the dashboard/apps is non-functional. (The supported external path is Cloudflare; couldn't validate it from the LAN due to DNS hijack.) **FINDING — flag for supervised review.**
 - **Workaround used for UI testing (reversible):** a python TCP forwarder inside guest 9201 (`/tmp/felhom-test-fwd.py`, listens `0.0.0.0:18080` → controller `172.18.0.10:8080`). Browser → `http://felhom.demo-felhom.eu:18080/` (LAN DNS → guest:18080 → controller). `CatchAllMiddleware` strips the port so Host matches `felhom.<domain>` → dashboard serves over plain HTTP. **To be killed at cleanup.**
 - The dashboard currently serves with **no auth wall** (`/` → 200, no login redirect, no session cookie) — no password configured/enforced. (Bears on Phase 6 "password change" step.)
 ### Scope reality vs runbook
 - Runbook assumed "14 enabled apps"; catalog is now **55 total, 38 not-deployed**. Deploying all 38 unattended would exhaust the 12 GiB guest. **Phase 2 will deploy a representative sample** (a few light apps) + cite the already-deployed heavy multi-container case (immich, 4 containers healthy). Documented, not silently truncated.
 **Phase 0 verdict: PASS — gate OPEN.**
 ---
 ## Phase 1 — komga HC fix + git-sync + health detection — **PASS**
 State: 9201 healthy, controller up, tunnel up.
 **Diagnosis (live, guest 9201):** docker HC `curl -f .../api/v1/actuator/health` → exit 22 (HTTP 401, auth-gated) → `unhealthy` for 18h while serving fine on :25600. Endpoint probe matrix (wget in image): `/`, `/actuator/health`, `/api/v1/oauth2/providers`, `/login` all 200; only `/api/v1/actuator/health` → 401. Image ships `curl` (`/usr/bin/curl`). **Found a second probe:** the controller's own `.felhom.yml` `healthcheck.checks[].path` (drives the UI badge + route publishing) also pointed at the 401 path.
 | Check | Result | Evidence |
 |---|---|---|
 | Catalog HC fix (docker-compose.yml) | PASS | `/api/v1/actuator/health` → `/actuator/health`; commit `3faa5ae` pushed to `main` |
 | Catalog probe fix (.felhom.yml) | PASS | controller-side probe path fixed; commit `9b066de` pushed to `main` |
 | CHANGELOG + REPORT (app-catalog) | PASS | created `CHANGELOG.md` (newest-on-top) + overwrote `REPORT.md` (`f51f794`) |
 | GUI git-sync ("Sablonok frissítése") | PASS | logs: `[sync] Updated komga/docker-compose.yml` + `komga/.felhom.yml`; `Sablonok frissítve — frissítve: komga`. Live stack `/opt/docker/stacks/komga/docker-compose.yml` updated to `/actuator/health` (content-hash copy verified) |
 | GUI redeploy ("Frissítés") | PASS | `POST /api/stacks/komga/update` → 200; `docker compose up -d --remove-orphans`; "Stack komga updated successfully (4.2s)"; container recreated (StartedAt 2026-06-22T13:09:50Z) |
 | komga reaches healthy (docker) | PASS | `docker ps`: `komga Up (healthy)`; HC now `http://localhost:25600/actuator/health` |
 | Controller health-state machine | PASS | controller log `Health probes: N ok (of N probed)` (no more komga WARN); card badge `Nem egészséges` → **Fut** |
 | Route publishing | PASS | "⚠ URL nem elérhető – útvonal nincs publikálva" warning cleared; `komga.demo-felhom.eu ↗` link shown |
 Minor cosmetic note (not fixed): a few catalog logo SVGs 404 (`adventurelog-logo.svg`, `cloudflared-logo.svg`, `crafty-controller-logo.svg`).
 ## Phase 2 — App deployment sweep — **PASS (with 1 logged catalog defect)**
 State: 9201 healthy, controller up, tunnel up. Scope: representative sample (see Phase-0 scope note) — catalog has 38 not-deployed apps; deploying all on a 12 GiB guest is unsafe.
 | App | Result | Evidence |
 |---|---|---|
 | **gitea** (deploy via GUI) | PASS (after HC fix) | GUI deploy page → memory-gate bar (`6043/11904 MB 50%`, two-segment current+new) → 3-step progress panel (✅ Konfiguráció mentve → ✅ Konténerek elindultak → ⏳ inicializálás) → `POST /api/stacks/gitea/update` 200. Initially **unhealthy**: HC `curl -f /api/v1/version` → 404 (404 until install-lock) while `/api/healthz` → 200. **Fixed catalog** (commit `17e00b7` + CHANGELOG `1a44c14`), re-synced, redeployed → `docker inspect`: `healthy`, HC now `/api/healthz`; endpoint `http://127.0.0.1:3000/` → `HTTP/1.1 200 OK`. |
 | **glance** (deploy via GUI) | **FAIL — logged, not fixed** | GUI deploy → 3-step panel OK, but container `Restarting (1)` crash-loop. Logs: `parsing config: reading main YAML file: open /app/config/glance.yml: no such file or directory`. **Root cause:** catalog template mounts an empty `glance_config:/app/config` named volume but never seeds the required `glance.yml`; glance refuses to start without it. **Not a trivial HC/start_period fix** (needs a seeded default config asset + wiring) → logged per runbook, not fixed unattended. Cleaned up: `POST .../stop` + `POST .../remove` → container gone. |
 | **immich** (heavy multi-container, pre-deployed) | PASS (cited) | 4 containers (server/ML/postgres/redis) all `Up (healthy)` — the heavy-app coverage case. |
 **Bonus coverage:** stop + remove endpoints exercised (glance teardown). **Catalog defects found this phase:** gitea HC (fixed), glance missing-config (logged for supervised fix).
 ## Phase 3 — Backup + restore — **PASS (2 findings)**
 State: 9201 healthy, controller up, tunnel up.
 | Check | Result | Evidence |
 |---|---|---|
 | Manual DB-dump backup ("Mentés most") | PASS | `POST /api/backup/run` → 3 DBs dumped: romm-mariadb 38.5 KB/14 tbl, paperless-postgres 360 KB/67 tbl, immich-postgres 43.2 MB/60 tbl; 43.6 MB total in 2.18s; `status.success=true`, `last_run 2026-06-22T13:27:26Z`. Files on disk fresh **13:28**: `…/primary/immich/db-dumps/immich-postgres.sql` (45.3 MB), `…/primary/romm/db-dumps/romm-mariadb.sql` (39 KB) |
 | Cross-drive copy (Tier-2) | PASS | `POST /api/backup/tier2` → 8 HDD apps copied to `/mnt/sys_drive/felhom-data/backups/secondary/<app>` (immich 155.3 MB verified 156 MB on disk; romm has appdata + recovery-unit{db-dump,manifest,compose}). `crossdrive_completed` events pushed |
 | Backup page reflects status | PASS | page shows "Adatmentés aktív", 3 DBs, schedule 02:30, last DB backup timestamp; status API success=true |
 | Restore flow (per-app) | PASS | `POST /backup/restore stack_name=romm` → `RestoreFromRecoveryUnit`: compose down → redeploy from recovery unit (6 env, 3 encrypted) → "Imported DB dump romm-mariadb.sql into romm-db" → containers up → "Restore-from-unit completed: romm" |
 | Restore is non-hollow | PASS | post-restore romm DB has real `platforms` table + 17 tables; all 3 romm containers `Up (healthy)`. Marker-table test: planted `restore_test_marker`(id=42) pre-restore → **survived** restore (see finding) but real snapshot tables were reimported; marker dropped afterward (DB back to 17-table baseline) |
 **Findings:**
 1. **`GET /api/backup/snapshots` has no server handler** (api-router returns "endpoint not found"). The restore UI's snapshot dropdown (`backups.html:682`) can't populate → a pure-UI restore is blocked at snapshot selection. Restore itself works via `POST /backup/restore` (the form's action). **Flag for fix.**
 2. **Restore DB import is additive** — `RestoreFromRecoveryUnit` replays the dump but does not drop tables absent from it (my marker table persisted). Real snapshot data is restored; stale/extra tables are not pruned. Minor behavioral note.
 3. Whole-server ("Rendszermentés") backup shows **"A host-ügynök jelenleg nem elérhető"** (agent local API unreachable from controller) — same root as the monitoring host-metrics gap; tracked in Phase 6.
 ## Phase 4 — DR rebuild from backup — **PASS** (non-destructive to 9201)
 State: 9201 healthy throughout, controller up, tunnel up (verified before + after).
 | Check | Result | Evidence |
 |---|---|---|
 | Agent DR primitive (`--selftest=restore-test`) | PASS | restored `felhom-pbs:backup/ct/9201/2026-06-22T12:45:37Z` into scratch **990000** (band 990000-990009, no collision) → bind-mounts neutralized → **net link-down** → boot → `verified: boot+running` → torn down. `pass:true`, 2m56s, self-cleaning |
 | Deep DR restore into throwaway **9300** | PASS | `pct restore 9300 <archive> --storage local-lvm`. **Safety:** before start, removed shared binds `mp3:/mnt/felhom-drives` + `mp9:/etc/felhom-bootstrap`, removed `net0`, set `onboot 0` — verified config clean (only restored `mp0` docker-data + rootfs remain). |
 | 9300 boots + controller process up | PASS | `pct status 9300: running`; `felhom-controller Up (healthy)` inside |
 | Restored volumes + app.yaml + stacks intact | PASS | `/opt/docker/stacks/` has all 57 stack dirs; romm `app.yaml` restored (`deployed:true`, env, locked_fields) |
 | `docker ps` inside 9300 | PASS | **25 containers up** from restored volumes — identical set to 9201 (filebrowser, vikunja, uptime-kuma, romm×3, radarr, privatebin, paperless×3, mealie, komga, jellyfin, immich×4, gokapi, calibre-web, audiobookshelf, actualbudget, felhom-controller healthy, cloudflared, traefik) |
 | Restored data non-hollow | PASS | 9300 `romm-db`: 17 tables (the real restored DB) |
 | No collision with live 9201 | PASS | 9300 has **no eth0** (net removed) → cloudflared cannot reach Cloudflare/hub; live 9201 `cloudflared Up 5 days` unchanged throughout |
 | Cleanup | PASS | `pct stop 9300 && pct destroy 9300` → both LVs removed; `pct list` shows only 9001/9201/9999; local-lvm freed to 9.71% |
 **Core DR claim proven end-to-end:** PBS whole-CT restore → guest boots → controller + all app stacks + DB data recovered from restored volumes — twice (agent self-cleaning primitive + manual deep inspection), both collision-safe per §0.6.
 ## Phase 5 — Storage lifecycle — **PARTIAL (blocked by a controller→agent connectivity defect)**
 State: 9201 healthy, controller up, tunnel up. Agent (`felhom-agent v0.39.0`) healthy on the host.
 ### 🔴 HEADLINE FINDING — controller cannot reach the agent local API (8443)
 The controller container's calls to the agent local API fail: `agentapi: GET /disks: dial tcp 192.168.0.162:8443: connect: cannot assign requested address` (and same for `/host/metrics`). Reachability matrix **from the controller container**:
 - `192.168.0.162:8006` (Proxmox pveproxy, binds `0.0.0.0`) → **200** (reachable)
 - `192.168.0.162:8443` (felhom-agent local API, binds **`192.168.0.162`** specifically) → **refused/EADDRNOTAVAIL**
 - `192.168.0.141:443` (guest's own traefik) → 404 (reachable)
 From the guest **host namespace**, `192.168.0.162:8443` is OPEN. So the agent is up and listening, but its 8443 socket is unreachable specifically from the **docker container** path while 8006 on the same host is fine — consistent with the agent binding only to the host LAN IP vs a NAT/interface mismatch on the container→host path. **Confirmed:** `ss` shows agent `LISTEN 192.168.0.162:8443` (bound to the specific LAN IP, not `0.0.0.0` like pveproxy:8006); **no** iptables rule references 8443; pve-firewall disabled. So it is not a firewall — likely a container→host-IP routing/source-bind mismatch (the docker-NAT'd path to the host's own `.162` fails for the agent socket while pveproxy's `0.0.0.0` socket is reachable). Suggested fix direction (supervised): bind the agent local API to `0.0.0.0` (or the vmbr0 bridge address the guest routes through), or point the controller's `local_api.endpoint` at a container-reachable address. **Impact: the whole agent-backed feature set is down in this demo** — storage management UI (`/api/disks`, `/api/storage`), host metrics (monitoring "gazdagép metrikái nem elérhetők"), and whole-guest backup ("A host-ügynök jelenleg nem elérhető"). **Needs supervised network/bind diagnosis** (likely agent listen-address or PVE firewall on 8443). Not fixed unattended (network reconfig is out of scope per §0.1/§0.8).
 | Check | Result | Evidence |
 |---|---|---|
 | Scan / observe / health / FS / model / durable_id (agent tier) | PASS | `--selftest=storage`: 5 targets — felhom-flash (usb, attached, class=slow, `uuid:81a26531…`, /dev/sdc1, /mnt/felhom-flash); **felhom-usb (local-dir, `uuid:da9e7089…`, /dev/sdb1, SMART=PASSED temp=35 poh=2736 realloc=0)**; local-lvm (thin data=9.7% meta=0.8%); felhom-pbs (21%); local. All `reach=true` |
 | Label edit + revert (via UI) | SKIPPED | blocked — controller→agent path down (headline finding); UI storage mgmt non-functional |
 | Eject → re-enroll cycle (via UI) | SKIPPED | blocked — same; not driven directly at agent-tier (would mutate live-drive state via the broken path; unattended caution per §0.7) |
 | Destructive loopback (scan→format→mount→migrate→decommission) | SKIPPED | the init wizard drives the agent via the broken path; doing it purely agent-side needs the per-guest local-API token + TLS-pin reconstruction — high-effort/low-confidence unattended. Deferred to supervised. No scratch loopback was created (nothing to clean up). |
 ## Phase 6 — Monitoring / alerts / notifications / settings / hub — **PASS**
 State: 9201 healthy, controller up, tunnel up.
 | Check | Result | Evidence |
 |---|---|---|
 | Monitoring — host/server metrics | PASS | `/monitoring`: CPU 12.4%, load 0.35/0.76/1.35, mem 7.6/15.4 GB (host's 16 GB → **/proc/LXC kernel-shared view**, not the agent API), CPU temp 43 °C, uptime 5d21h, "Frissítve 16:03:12". (The Phase-0 momentary "gazdagép metrikái nem elérhetők" did not persist.) |
 | Monitoring — container/app metrics | PASS | "Memória eloszlás" per-container breakdown (felhom-controller 1.1 GB, paperless 594 MB, immich-server 414 MB, komga 387 MB, gitea 85 MB, …26 containers); "Alkalmazás erőforrások" per-app CPU/MEM charts; time-range charts 1h/6h/24h/7d/30d |
 | Monitoring — storage capacity | PASS | all 5 stores with capacities + felhom-usb SMART temp 35 °C (cached from a prior agent poll) |
 | Alerts — state-based (komga) | PASS | controller logs: komga `Health probes: 0 ok, 1 unhealthy` before fix → `Health probes: N ok` after (Phase 1); UI badge red→green; route-warning cleared |
 | Notification path (exercised ONCE) | PASS | set email + 2 events + cooldown=2h → `POST /settings/notifications` 200 → hub log `Notification preferences updated for demo-felhom`. Test: `SendTest()` → hub log **`Event from demo-felhom: test (info)` → `Test email sent to nagyfenyvesi.viktor@gmail.com for demo-felhom`** = controller→hub→Resend→email end-to-end. Only one email sent (§0.5). |
 | Cooldown / dup-suppression | PARTIAL | cooldown_hours=2 persisted + synced to hub (mechanism present); not stress-tested (would require generating duplicate alerts — avoided per §0.5) |
 | Settings persistence | PASS | notification prefs set → persisted to `settings.json` (email/cooldown/events) → reverted (email cleared, events None) — both directions verified |
 | Settings — password change | N/A | password protection is **operator-only** and **not configured** ("A jelszavas védelem nincs beállítva. Kérd az üzemeltetőt"); no customer-UI password change to test. Dashboard currently serves with no auth wall. |
 | Hub — report received | PASS | hub logs: `Received report from demo-felhom (11983 bytes)` every 15 min; `host-report from demo-felhom-01 (3 guests, 5 storage targets, 1 backups, 1 restore-tests, 9 pbs-snapshots)`; UI "Hub kapcsolat: Kapcsolódva, utolsó jelentés 8 perce" |
 | Hub — DR recipe carries PBS coord | PASS | hub DB `dr_recipe` (demo-felhom): host-half `pbs={"repo_id":"felhom-pbs","namespace":"root","latest_snapshot_id":"9201"}`; app-half customer demo-felhom / 15 apps. No secrets in either half. |
 | Hub — update panel | PASS | settings "Verzió és frissítés": current 0.73.0, latest "0.73.0 — naprakész", last check 2026-06-22T10:54:46Z |
 | Hub GUI tiles (analytics/healthchecks) | SKIPPED(no operator creds) | verified hub-side state via logs + DB instead of the auth-gated operator GUI |
 **Note (reconciles Phase 5):** the agent local-API path stays **persistently** down (6/6 `EADDRNOTAVAIL` on `/api/disks`). Monitoring host metrics work anyway because they are `/proc`-derived (LXC shares the host kernel); the agent gap only blocks features needing a *live* agent call (raw-disk view, storage mgmt, whole-guest backup trigger).
 ## Phase 7 — Resilience — **PASS (guest reboot); host reboot SKIPPED**
 | Check | Result | Evidence |
 |---|---|---|
 | Guest 9201 reboot | PASS | `pct reboot 9201`; pre=26 containers → recovered to **26 containers, 0 unhealthy/starting** within ~2 min |
 | Drives re-bind after reboot | PASS | felhom-flash (/dev/sdc1) + felhom-usb (/dev/sdb1) remounted ext4 at correct `/mnt/felhom-drives/<name>` mountpoints |
 | FileBrowser convergence | PASS | binds reconverged: `…/felhom-usb/userdata → /srv/felhom-usb`, `…/felhom-flash/userdata → /srv/felhom-flash` |
 | Tunnel reconnect | PASS | cloudflared QUIC + HTTP/2 prechecks to region1/region2 argotunnel = pass; `cloudflared Up` |
 | Controller + apps return | PASS | felhom-controller/traefik/filebrowser healthy; komga + gitea (the Phase-1/2 catalog fixes) **persist healthy** post-reboot |
 | Host reboot | SKIPPED(§7) | higher unattended risk (if the host doesn't return, demo is down until operator is home) — deferred to supervised |
 Minor note: the controller's docker container IP changed on reboot (`172.18.0.10`→`172.18.0.8`); harmless (traefik routes by docker DNS name `felhom-controller`), only required re-pointing the test forwarder.
 ---
 ## Feature matrix (summary)
 | Feature | Verdict | Notes |
 |---|---|---|
 | PBS whole-CT backup (agent) | PASS | fresh backup + verify ok |
 | Catalog git-sync (GUI "Sablonok frissítése") | PASS | content-hash copy, logs confirm |
 | App update/redeploy (GUI "Frissítés") | PASS | `POST /api/stacks/{n}/update`, recreates container |
 | Health-aware state machine + route publishing | PASS | komga red→green, route warning cleared |
 | App deploy (GUI 3-step panel + mem-gate + auto-secrets) | PASS | gitea deployed via GUI |
 | App stop + remove | PASS | glance + gitea torn down cleanly |
 | DB-dump backup (GUI/API) | PASS | 3 DBs, on-disk artifacts |
 | Tier-2 cross-drive copy | PASS | 8 apps → secondary, on-disk |
 | Per-app restore (recovery unit) | PASS | romm restored, non-hollow |
 | DR rebuild (PBS→boot→stacks) | PASS | restore-test + deep 9300, collision-safe |
 | Monitoring (host /proc, container, app, charts) | PASS | host data via LXC kernel-share |
 | Alerts (state-based) | PASS | komga unhealthy→healthy reflected |
 | Notification path (controller→hub→Resend→email) | PASS | one test email confirmed hub-side |
 | Settings persistence | PASS | set + revert |
 | Hub report + DR recipe (PBS coord) | PASS | recipe carries pbs repo/ns/snapshot |
 | Hub update panel | PASS | version up-to-date shown |
 | Guest reboot recovery | PASS | mounts/FileBrowser/tunnel/apps recovered |
 | **Storage management UI (disks/label/eject/enroll/format/migrate/decommission)** | **BLOCKED** | controller→agent 8443 down (finding #1) |
 | Whole-guest backup trigger (UI) | BLOCKED | same agent-connectivity defect |
 | Host reboot | SKIPPED | unattended risk (§7) |
 | Snapshot dropdown for restore (UI) | BLOCKED | `/api/backup/snapshots` has no handler (finding #2) |
 ### Findings (for supervised follow-up)
 1. **🔴 controller→agent local API (8443) unreachable** from the controller container (`EADDRNOTAVAIL`, 6/6), while pveproxy:8006 on the same host is reachable. Agent binds `192.168.0.162:8443` specifically (not `0.0.0.0`); no firewall. Breaks storage-management UI, live host-metrics API, whole-guest backup. Monitoring still shows host metrics via `/proc` (LXC kernel-share). **Fix direction:** bind agent local API to `0.0.0.0`/bridge addr, or point `local_api.endpoint` at a container-reachable address. *(env/network — not changed unattended.)*
 2. **`GET /api/backup/snapshots` has no server handler** → restore UI snapshot dropdown can't populate (restore itself works via `POST /backup/restore`).
 3. **glance catalog template** mounts an empty `glance_config` volume but never seeds `glance.yml` → crash-loop on deploy. Needs a seeded default config (not a trivial HC fix; logged).
 4. **Restore DB import is additive** (`RestoreFromRecoveryUnit` doesn't drop tables absent from the dump). Minor.
 5. **On-LAN HTTPS to dashboard/apps 404s** at traefik `:443` (websecure) for all hosts; supported path is Cloudflare. (Used an in-guest forwarder for UI testing.)
 6. Minor: a few catalog logo SVGs 404 (adventurelog, cloudflared, crafty-controller); dashboard "Adatbázisok mentve" count flickered 3→2 post-reboot (cosmetic).
 ### Code changes shipped (app-catalog `main`, trunk)
 - komga HC `/api/v1/actuator/health`(401) → `/actuator/health` (docker HC + `.felhom.yml` probe) — `3faa5ae`, `9b066de`.
 - gitea HC `/api/v1/version`(404 pre-install) → `/api/healthz`, start_period 90s — `17e00b7`.
 - CHANGELOG + REPORT updated (`f51f794`, `1a44c14`). **No felhom-controller/agent code changed** (catalog-only, as scoped).
 ## Cleanup confirmation
 - ✅ Throwaway guest **9300 destroyed** (both LVs removed); restore-test scratch 990000 auto-torn-down. `pct list`: only 9201 (running) + pre-existing 9001/9999 (stopped, not created by this run).
 - ✅ No scratch loopback devices created (destructive storage sub-phase skipped) — nothing to remove.
 - ✅ Test apps removed: **glance** (crash-loop, removed), **gitea** (removed after validation) → back to baseline app set.
 - ✅ Notification settings reverted (email/events cleared).
 - ✅ Password: never set (operator-only, N/A).
 - ✅ Drives: never ejected (storage UI blocked) — no re-enroll needed.
 - ✅ **komga healthy** (improvement over baseline unhealthy).
 - ✅ Temp hub.db copy removed from DooPlex; DooPlex used in normal roles only (PBS target, gitea reads, hub reads).
 - ⚠️ Test artifact still on guest: `/tmp/felhom-test-fwd.py` python forwarder — **removed at end of run** (see final step).
 ## After-state (vs Phase-0 "before")
 - `pct list`: 9201 running (only change from before: none). 9001/9999 stopped (unchanged, pre-existing).
 - 9201: **25 containers Up, 0 unhealthy** (before: 1 unhealthy = komga). Controller v0.73.0, tunnel up.
 - Storage: 5 targets attached/reachable (felhom-flash uuid:81a26531, felhom-usb uuid:da9e7089).
 - Net result: demo restored to baseline **+ komga fixed**.
 ## Deferred — needs supervised run
 - **Storage lifecycle UI** (label edit, eject→re-enroll, init-wizard format/migrate/decommission on a loopback) — blocked by finding #1; re-run once controller↔agent connectivity is fixed.
 - **Host reboot** resilience (full host AC-recovery + remount-by-UUID under sdX reshuffle).
 - **glance** catalog config-seeding fix (finding #3).
 - **`/api/backup/snapshots`** handler + UI snapshot selection (finding #2).
 - **Agent 8443 reachability** root-cause + fix (finding #1) — the highest-priority item; it gates the whole agent-backed feature set.