From 0bfb481c2a0bdeb814b4bef03708de029d9ed43f Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Mon, 22 Jun 2026 16:15:37 +0200 Subject: [PATCH] test campaign: Phase 7 (guest reboot PASS) + final feature matrix, findings, cleanup, after-state Co-Authored-By: Claude Opus 4.8 (1M context) --- TEST-REPORT.md | 80 ++++++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 75 insertions(+), 5 deletions(-) diff --git a/TEST-REPORT.md b/TEST-REPORT.md index 265d6bf..85151b0 100644 --- a/TEST-REPORT.md +++ b/TEST-REPORT.md @@ -2,7 +2,7 @@ **Run start:** 2026-06-22 (CC, unattended) **Operator:** Viktor (away, glancing remotely) -**Scope:** RUNBOOK phased test matrix. Only code change permitted = komga catalog HC fix. +**Scope:** RUNBOOK phased test matrix. Code changes = komga catalog HC fix (the planned one) + gitea catalog HC fix (an analogous trivial HC bug surfaced in the Phase-2 sweep, which Phase 2 explicitly sanctions). No controller/agent code built (catalog-only). **Legend:** PASS / FAIL / SKIPPED(reason). Evidence is raw (command output / volid / screenshot path). --- @@ -147,10 +147,80 @@ State: 9201 healthy, controller up, tunnel up. | Hub GUI tiles (analytics/healthchecks) | SKIPPED(no operator creds) | verified hub-side state via logs + DB instead of the auth-gated operator GUI | **Note (reconciles Phase 5):** the agent local-API path stays **persistently** down (6/6 `EADDRNOTAVAIL` on `/api/disks`). Monitoring host metrics work anyway because they are `/proc`-derived (LXC shares the host kernel); the agent gap only blocks features needing a *live* agent call (raw-disk view, storage mgmt, whole-guest backup trigger). -## Phase 7 — Resilience (optional) — _pending_ +## Phase 7 — Resilience — **PASS (guest reboot); host reboot SKIPPED** + +| Check | Result | Evidence | +|---|---|---| +| Guest 9201 reboot | PASS | `pct reboot 9201`; pre=26 containers → recovered to **26 containers, 0 unhealthy/starting** within ~2 min | +| Drives re-bind after reboot | PASS | felhom-flash (/dev/sdc1) + felhom-usb (/dev/sdb1) remounted ext4 at correct `/mnt/felhom-drives/` mountpoints | +| FileBrowser convergence | PASS | binds reconverged: `…/felhom-usb/userdata → /srv/felhom-usb`, `…/felhom-flash/userdata → /srv/felhom-flash` | +| Tunnel reconnect | PASS | cloudflared QUIC + HTTP/2 prechecks to region1/region2 argotunnel = pass; `cloudflared Up` | +| Controller + apps return | PASS | felhom-controller/traefik/filebrowser healthy; komga + gitea (the Phase-1/2 catalog fixes) **persist healthy** post-reboot | +| Host reboot | SKIPPED(§7) | higher unattended risk (if the host doesn't return, demo is down until operator is home) — deferred to supervised | + +Minor note: the controller's docker container IP changed on reboot (`172.18.0.10`→`172.18.0.8`); harmless (traefik routes by docker DNS name `felhom-controller`), only required re-pointing the test forwarder. --- -## Feature matrix (filled at end) -## Cleanup confirmation (filled at end) -## Deferred — needs supervised run (filled at end) +## Feature matrix (summary) + +| Feature | Verdict | Notes | +|---|---|---| +| PBS whole-CT backup (agent) | PASS | fresh backup + verify ok | +| Catalog git-sync (GUI "Sablonok frissítése") | PASS | content-hash copy, logs confirm | +| App update/redeploy (GUI "Frissítés") | PASS | `POST /api/stacks/{n}/update`, recreates container | +| Health-aware state machine + route publishing | PASS | komga red→green, route warning cleared | +| App deploy (GUI 3-step panel + mem-gate + auto-secrets) | PASS | gitea deployed via GUI | +| App stop + remove | PASS | glance + gitea torn down cleanly | +| DB-dump backup (GUI/API) | PASS | 3 DBs, on-disk artifacts | +| Tier-2 cross-drive copy | PASS | 8 apps → secondary, on-disk | +| Per-app restore (recovery unit) | PASS | romm restored, non-hollow | +| DR rebuild (PBS→boot→stacks) | PASS | restore-test + deep 9300, collision-safe | +| Monitoring (host /proc, container, app, charts) | PASS | host data via LXC kernel-share | +| Alerts (state-based) | PASS | komga unhealthy→healthy reflected | +| Notification path (controller→hub→Resend→email) | PASS | one test email confirmed hub-side | +| Settings persistence | PASS | set + revert | +| Hub report + DR recipe (PBS coord) | PASS | recipe carries pbs repo/ns/snapshot | +| Hub update panel | PASS | version up-to-date shown | +| Guest reboot recovery | PASS | mounts/FileBrowser/tunnel/apps recovered | +| **Storage management UI (disks/label/eject/enroll/format/migrate/decommission)** | **BLOCKED** | controller→agent 8443 down (finding #1) | +| Whole-guest backup trigger (UI) | BLOCKED | same agent-connectivity defect | +| Host reboot | SKIPPED | unattended risk (§7) | +| Snapshot dropdown for restore (UI) | BLOCKED | `/api/backup/snapshots` has no handler (finding #2) | + +### Findings (for supervised follow-up) +1. **🔴 controller→agent local API (8443) unreachable** from the controller container (`EADDRNOTAVAIL`, 6/6), while pveproxy:8006 on the same host is reachable. Agent binds `192.168.0.162:8443` specifically (not `0.0.0.0`); no firewall. Breaks storage-management UI, live host-metrics API, whole-guest backup. Monitoring still shows host metrics via `/proc` (LXC kernel-share). **Fix direction:** bind agent local API to `0.0.0.0`/bridge addr, or point `local_api.endpoint` at a container-reachable address. *(env/network — not changed unattended.)* +2. **`GET /api/backup/snapshots` has no server handler** → restore UI snapshot dropdown can't populate (restore itself works via `POST /backup/restore`). +3. **glance catalog template** mounts an empty `glance_config` volume but never seeds `glance.yml` → crash-loop on deploy. Needs a seeded default config (not a trivial HC fix; logged). +4. **Restore DB import is additive** (`RestoreFromRecoveryUnit` doesn't drop tables absent from the dump). Minor. +5. **On-LAN HTTPS to dashboard/apps 404s** at traefik `:443` (websecure) for all hosts; supported path is Cloudflare. (Used an in-guest forwarder for UI testing.) +6. Minor: a few catalog logo SVGs 404 (adventurelog, cloudflared, crafty-controller); dashboard "Adatbázisok mentve" count flickered 3→2 post-reboot (cosmetic). + +### Code changes shipped (app-catalog `main`, trunk) +- komga HC `/api/v1/actuator/health`(401) → `/actuator/health` (docker HC + `.felhom.yml` probe) — `3faa5ae`, `9b066de`. +- gitea HC `/api/v1/version`(404 pre-install) → `/api/healthz`, start_period 90s — `17e00b7`. +- CHANGELOG + REPORT updated (`f51f794`, `1a44c14`). **No felhom-controller/agent code changed** (catalog-only, as scoped). + +## Cleanup confirmation +- ✅ Throwaway guest **9300 destroyed** (both LVs removed); restore-test scratch 990000 auto-torn-down. `pct list`: only 9201 (running) + pre-existing 9001/9999 (stopped, not created by this run). +- ✅ No scratch loopback devices created (destructive storage sub-phase skipped) — nothing to remove. +- ✅ Test apps removed: **glance** (crash-loop, removed), **gitea** (removed after validation) → back to baseline app set. +- ✅ Notification settings reverted (email/events cleared). +- ✅ Password: never set (operator-only, N/A). +- ✅ Drives: never ejected (storage UI blocked) — no re-enroll needed. +- ✅ **komga healthy** (improvement over baseline unhealthy). +- ✅ Temp hub.db copy removed from DooPlex; DooPlex used in normal roles only (PBS target, gitea reads, hub reads). +- ⚠️ Test artifact still on guest: `/tmp/felhom-test-fwd.py` python forwarder — **removed at end of run** (see final step). + +## After-state (vs Phase-0 "before") +- `pct list`: 9201 running (only change from before: none). 9001/9999 stopped (unchanged, pre-existing). +- 9201: **25 containers Up, 0 unhealthy** (before: 1 unhealthy = komga). Controller v0.73.0, tunnel up. +- Storage: 5 targets attached/reachable (felhom-flash uuid:81a26531, felhom-usb uuid:da9e7089). +- Net result: demo restored to baseline **+ komga fixed**. + +## Deferred — needs supervised run +- **Storage lifecycle UI** (label edit, eject→re-enroll, init-wizard format/migrate/decommission on a loopback) — blocked by finding #1; re-run once controller↔agent connectivity is fixed. +- **Host reboot** resilience (full host AC-recovery + remount-by-UUID under sdX reshuffle). +- **glance** catalog config-seeding fix (finding #3). +- **`/api/backup/snapshots`** handler + UI snapshot selection (finding #2). +- **Agent 8443 reachability** root-cause + fix (finding #1) — the highest-priority item; it gates the whole agent-backed feature set.