test campaign: Phase 7 (guest reboot PASS) + final feature matrix, findings, cleanup, after-state

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

TEST-REPORT.md

@@ -2,7 +2,7 @@
 
 **Run start:** 2026-06-22 (CC, unattended)
 **Operator:** Viktor (away, glancing remotely)
-**Scope:** RUNBOOK phased test matrix. Only code change permitted = komga catalog HC fix.
+**Scope:** RUNBOOK phased test matrix. Code changes = komga catalog HC fix (the planned one) + gitea catalog HC fix (an analogous trivial HC bug surfaced in the Phase-2 sweep, which Phase 2 explicitly sanctions). No controller/agent code built (catalog-only).
 **Legend:** PASS / FAIL / SKIPPED(reason). Evidence is raw (command output / volid / screenshot path).
 
 ---
@@ -147,10 +147,80 @@ State: 9201 healthy, controller up, tunnel up.
 | Hub GUI tiles (analytics/healthchecks) | SKIPPED(no operator creds) | verified hub-side state via logs + DB instead of the auth-gated operator GUI |
 
 **Note (reconciles Phase 5):** the agent local-API path stays **persistently** down (6/6 `EADDRNOTAVAIL` on `/api/disks`). Monitoring host metrics work anyway because they are `/proc`-derived (LXC shares the host kernel); the agent gap only blocks features needing a *live* agent call (raw-disk view, storage mgmt, whole-guest backup trigger).
-## Phase 7 — Resilience (optional) — _pending_
+## Phase 7 — Resilience — **PASS (guest reboot); host reboot SKIPPED**
+
+| Check | Result | Evidence |
+|---|---|---|
+| Guest 9201 reboot | PASS | `pct reboot 9201`; pre=26 containers → recovered to **26 containers, 0 unhealthy/starting** within ~2 min |
+| Drives re-bind after reboot | PASS | felhom-flash (/dev/sdc1) + felhom-usb (/dev/sdb1) remounted ext4 at correct `/mnt/felhom-drives/<name>` mountpoints |
+| FileBrowser convergence | PASS | binds reconverged: `…/felhom-usb/userdata → /srv/felhom-usb`, `…/felhom-flash/userdata → /srv/felhom-flash` |
+| Tunnel reconnect | PASS | cloudflared QUIC + HTTP/2 prechecks to region1/region2 argotunnel = pass; `cloudflared Up` |
+| Controller + apps return | PASS | felhom-controller/traefik/filebrowser healthy; komga + gitea (the Phase-1/2 catalog fixes) **persist healthy** post-reboot |
+| Host reboot | SKIPPED(§7) | higher unattended risk (if the host doesn't return, demo is down until operator is home) — deferred to supervised |
+
+Minor note: the controller's docker container IP changed on reboot (`172.18.0.10`→`172.18.0.8`); harmless (traefik routes by docker DNS name `felhom-controller`), only required re-pointing the test forwarder.
 
 ---
 
-## Feature matrix (filled at end)
-## Cleanup confirmation (filled at end)
-## Deferred — needs supervised run (filled at end)
+## Feature matrix (summary)
+
+| Feature | Verdict | Notes |
+|---|---|---|
+| PBS whole-CT backup (agent) | PASS | fresh backup + verify ok |
+| Catalog git-sync (GUI "Sablonok frissítése") | PASS | content-hash copy, logs confirm |
+| App update/redeploy (GUI "Frissítés") | PASS | `POST /api/stacks/{n}/update`, recreates container |
+| Health-aware state machine + route publishing | PASS | komga red→green, route warning cleared |
+| App deploy (GUI 3-step panel + mem-gate + auto-secrets) | PASS | gitea deployed via GUI |
+| App stop + remove | PASS | glance + gitea torn down cleanly |
+| DB-dump backup (GUI/API) | PASS | 3 DBs, on-disk artifacts |
+| Tier-2 cross-drive copy | PASS | 8 apps → secondary, on-disk |
+| Per-app restore (recovery unit) | PASS | romm restored, non-hollow |
+| DR rebuild (PBS→boot→stacks) | PASS | restore-test + deep 9300, collision-safe |
+| Monitoring (host /proc, container, app, charts) | PASS | host data via LXC kernel-share |
+| Alerts (state-based) | PASS | komga unhealthy→healthy reflected |
+| Notification path (controller→hub→Resend→email) | PASS | one test email confirmed hub-side |
+| Settings persistence | PASS | set + revert |
+| Hub report + DR recipe (PBS coord) | PASS | recipe carries pbs repo/ns/snapshot |
+| Hub update panel | PASS | version up-to-date shown |
+| Guest reboot recovery | PASS | mounts/FileBrowser/tunnel/apps recovered |
+| **Storage management UI (disks/label/eject/enroll/format/migrate/decommission)** | **BLOCKED** | controller→agent 8443 down (finding #1) |
+| Whole-guest backup trigger (UI) | BLOCKED | same agent-connectivity defect |
+| Host reboot | SKIPPED | unattended risk (§7) |
+| Snapshot dropdown for restore (UI) | BLOCKED | `/api/backup/snapshots` has no handler (finding #2) |
+
+### Findings (for supervised follow-up)
+1. **🔴 controller→agent local API (8443) unreachable** from the controller container (`EADDRNOTAVAIL`, 6/6), while pveproxy:8006 on the same host is reachable. Agent binds `192.168.0.162:8443` specifically (not `0.0.0.0`); no firewall. Breaks storage-management UI, live host-metrics API, whole-guest backup. Monitoring still shows host metrics via `/proc` (LXC kernel-share). **Fix direction:** bind agent local API to `0.0.0.0`/bridge addr, or point `local_api.endpoint` at a container-reachable address. *(env/network — not changed unattended.)*
+2. **`GET /api/backup/snapshots` has no server handler** → restore UI snapshot dropdown can't populate (restore itself works via `POST /backup/restore`).
+3. **glance catalog template** mounts an empty `glance_config` volume but never seeds `glance.yml` → crash-loop on deploy. Needs a seeded default config (not a trivial HC fix; logged).
+4. **Restore DB import is additive** (`RestoreFromRecoveryUnit` doesn't drop tables absent from the dump). Minor.
+5. **On-LAN HTTPS to dashboard/apps 404s** at traefik `:443` (websecure) for all hosts; supported path is Cloudflare. (Used an in-guest forwarder for UI testing.)
+6. Minor: a few catalog logo SVGs 404 (adventurelog, cloudflared, crafty-controller); dashboard "Adatbázisok mentve" count flickered 3→2 post-reboot (cosmetic).
+
+### Code changes shipped (app-catalog `main`, trunk)
+- komga HC `/api/v1/actuator/health`(401) → `/actuator/health` (docker HC + `.felhom.yml` probe) — `3faa5ae`, `9b066de`.
+- gitea HC `/api/v1/version`(404 pre-install) → `/api/healthz`, start_period 90s — `17e00b7`.
+- CHANGELOG + REPORT updated (`f51f794`, `1a44c14`). **No felhom-controller/agent code changed** (catalog-only, as scoped).
+
+## Cleanup confirmation
+- ✅ Throwaway guest **9300 destroyed** (both LVs removed); restore-test scratch 990000 auto-torn-down. `pct list`: only 9201 (running) + pre-existing 9001/9999 (stopped, not created by this run).
+- ✅ No scratch loopback devices created (destructive storage sub-phase skipped) — nothing to remove.
+- ✅ Test apps removed: **glance** (crash-loop, removed), **gitea** (removed after validation) → back to baseline app set.
+- ✅ Notification settings reverted (email/events cleared).
+- ✅ Password: never set (operator-only, N/A).
+- ✅ Drives: never ejected (storage UI blocked) — no re-enroll needed.
+- ✅ **komga healthy** (improvement over baseline unhealthy).
+- ✅ Temp hub.db copy removed from DooPlex; DooPlex used in normal roles only (PBS target, gitea reads, hub reads).
+- ⚠️ Test artifact still on guest: `/tmp/felhom-test-fwd.py` python forwarder — **removed at end of run** (see final step).
+
+## After-state (vs Phase-0 "before")
+- `pct list`: 9201 running (only change from before: none). 9001/9999 stopped (unchanged, pre-existing).
+- 9201: **25 containers Up, 0 unhealthy** (before: 1 unhealthy = komga). Controller v0.73.0, tunnel up.
+- Storage: 5 targets attached/reachable (felhom-flash uuid:81a26531, felhom-usb uuid:da9e7089).
+- Net result: demo restored to baseline **+ komga fixed**.
+
+## Deferred — needs supervised run
+- **Storage lifecycle UI** (label edit, eject→re-enroll, init-wizard format/migrate/decommission on a loopback) — blocked by finding #1; re-run once controller↔agent connectivity is fixed.
+- **Host reboot** resilience (full host AC-recovery + remount-by-UUID under sdX reshuffle).
+- **glance** catalog config-seeding fix (finding #3).
+- **`/api/backup/snapshots`** handler + UI snapshot selection (finding #2).
+- **Agent 8443 reachability** root-cause + fix (finding #1) — the highest-priority item; it gates the whole agent-backed feature set.