felhom.eu/documentation/tests/unattended-test-campaign-2026-06-22-findings.md

# Unattended test campaign — full-feature validation (demo guest 9201) — 2026-06-22

> **Status: empirical findings from an unattended live test campaign (2026-06-22).** Operator away,
> glancing remotely. Goal: maximize feature coverage across the running demo (Felhom on Proxmox:
> host agent `felhom-agent v0.39.0` on `felhom-pve`, in-guest controller `felhom-controller v0.73.0`
> in LXC 9201, hub `felhom-hub` on k3s/DooPlex) **without leaving the demo unrecoverably broken.**
> Only code change permitted: the komga catalog healthcheck fix (+ one analogous gitea HC fix the
> sweep surfaced, which the runbook sanctions). No controller/agent build. All destructive DR work
> targeted throwaway guests; the live demo was restored to baseline (and is healthier — komga fixed).
>
> Companion live report (per-check raw evidence, continuously updated during the run):
> `felhom-controller/TEST-REPORT.md`. This document is the consolidated findings writeup.

## Topology under test

- **Host / agent:** `felhom-pve` (`demo-felhom`, PVE 9.2.2, Intel N100, 16 GB), `felhom-agent v0.39.0`
  (systemd active). SSH `root@felhom-pve`.
- **Guest / controller:** LXC **9201** (`demo-felhom`, 12 GB, 32 GB rootfs + 256 GB `/var/lib/docker`
  vol), `felhom-controller:0.73.0` (docker container, no auth wall currently). Customer id `demo-felhom`.
- **Hub:** `felhom-hub` on k3s @ DooPlex `192.168.0.180`; `hub.felhom.eu`.
- **Backup target:** PBS `felhom-pbs` (datastore `felhom-spike` on DooPlex), 9.6 TB, 21 % used.
- **Drives (enrolled):** `felhom-flash` (USB, `uuid:81a26531…`, `/mnt/felhom-drives/felhom-flash`,
  117 GB, holds app userdata for 8 apps) · `felhom-usb` (local-dir, `uuid:da9e7089…`,
  `/mnt/felhom-drives/felhom-usb`, 916 GB, empty) · `local-lvm` · `local` · `felhom-pbs`.
- **Baseline apps:** 14 deployed app stacks (25 containers incl. 4 infra: controller, traefik,
  cloudflared, filebrowser); 55 in catalog, 38 installable. komga `unhealthy` at baseline.

### Access note — how the UI was driven
On the LAN, split-horizon DNS (resolver `192.168.0.250`) maps `*.demo-felhom.eu` → guest 9201
(`192.168.0.141`), bypassing Cloudflare. The guest's traefik **`:443` (websecure) returns Go
`404 page not found` for every host** (controller and apps alike) while serving a valid LE
`*.demo-felhom.eu` cert; `:80` 301-redirects to https → so a browser dead-ends at the 404 on-LAN.
**Workaround:** a short-lived python TCP forwarder inside 9201 (`0.0.0.0:18080` → controller
`<container-ip>:8080`); the browser hit `http://felhom.demo-felhom.eu:18080/` (LAN DNS → guest:18080),
and the controller's `CatchAllMiddleware` strips the port so the Host matches `felhom.<domain>` and the
dashboard serves over plain HTTP. Forwarder removed at end of run. The controller dashboard is
reachable internally at traefik file-route `Host(felhom.demo-felhom.eu) → felhom-controller:8080`.

---

## Phase 0 — Baseline + PBS safety-net backup

| Check | Result | Evidence |
|---|---|---|
| Baseline capture | ✅ | agent v0.39.0; ctrl v0.73.0; 25 containers, komga unhealthy; drives + pvesm healthy |
| Fresh PBS whole-CT backup of 9201 | ✅ | `felhom-agent --selftest=backup -vmid=9201` → `felhom-pbs:backup/ct/9201/2026-06-22T12:45:37Z`, 19.62 GB, crash-consistent, 161 s. Uncovered (by design): `/etc/felhom-bootstrap`, `/mnt/felhom-drives` (drives backed up separately) |
| PBS verify | ✅ | `--selftest=pbs-verify` → `verify_state: ok` for the new snapshot |

The verified backup is the DR source for Phase 4. **Gate to destructive phases: opened.**

---

## Phase 1 — komga healthcheck fix + git-sync + health state machine

**Diagnosis (live):** the docker HC `curl -f http://localhost:25600/api/v1/actuator/health` exited 22
(HTTP **401** — the `/api/v1` prefix is auth-gated) → `unhealthy` for 18 h while komga served fine.
Endpoint probe matrix (wget inside the image): `/`, `/actuator/health`, `/api/v1/oauth2/providers`,
`/login` → all 200; only `/api/v1/actuator/health` → 401. **Komga's actuator is served unauthenticated
at `/actuator/health`, off the `/api/v1` API prefix.** A second probe matters too: the controller's own
`.felhom.yml` `healthcheck.checks[].path` (which drives the dashboard badge + route publishing) pointed
at the same 401 path. The `gotson/komga:1.20.0` image ships `curl` (verified).

**Fix (app-catalog `main`):** repoint **both** probes to `/actuator/health` — docker HC
(`3faa5ae`) and `.felhom.yml` (`9b066de`).

**Validation (end-to-end through the real UI + server pipeline):**
- GUI "Sablonok frissítése" → controller git-sync: `[sync] Updated komga/docker-compose.yml` +
  `komga/.felhom.yml`; content-hash copy landed in `/opt/docker/stacks/komga/`.
- GUI "Frissítés" → `POST /api/stacks/komga/update` 200 → `docker compose up -d --remove-orphans`
  recreated the container (4.2 s).
- Result: `docker ps` → `komga Up (healthy)`, HC now `/actuator/health`; controller log
  `Health probes: N ok`; dashboard badge `Nem egészséges` → **Fut**; the
  "⚠ URL nem elérhető – útvonal nincs publikálva" route warning cleared.

This one phase exercised: catalog fix, git-sync (content-hash copy), redeploy, the health-aware state
machine, and route publishing.

---

## Phase 2 — App deployment sweep

Scope note: catalog is **55 apps, 38 not deployed** (the runbook's "14 enabled" was stale). Deploying
all 38 would exhaust the 12 GB guest, so a representative sample was deployed via the GUI; the heavy
multi-container case (immich, 4 containers) is already deployed + healthy (cited).

| App | Result | Evidence |
|---|---|---|
| **gitea** (GUI deploy) | ✅ after HC fix | GUI deploy page → memory-gate bar (two-segment current+new) → 3-step progress panel → containers up. Initially `unhealthy`: HC `curl -f /api/v1/version` → **404** (404 until install-lock) while `/api/healthz` → 200. **Fixed catalog** (`17e00b7`, start_period 30→90 s) → re-sync → redeploy → `healthy`; endpoint `:3000/` → 200 |
| **glance** (GUI deploy) | ❌ logged, not fixed | crash-loop `Restarting (1)`. Log: `open /app/config/glance.yml: no such file or directory`. Catalog template mounts an empty `glance_config` named volume but **never seeds the required `glance.yml`**. Not a trivial HC fix → logged. Removed cleanly (`stop`+`remove`) |
| **immich** (pre-deployed) | ✅ cited | 4 containers healthy = heavy-app coverage |

Bonus coverage: the stop + remove endpoints (glance + gitea teardown). Both test deployments were
removed at cleanup to restore the baseline app set.

---

## Phase 3 — Backup + restore (per-app)

| Check | Result | Evidence |
|---|---|---|
| Manual DB-dump backup | ✅ | `POST /api/backup/run` → 3 DBs: romm-mariadb (14 tbl, 38.5 KB), paperless-postgres (67 tbl, 360 KB), immich-postgres (60 tbl, 43.2 MB); 43.6 MB / 2.18 s. On-disk fresh (`…/primary/<app>/db-dumps/*.sql`) |
| Tier-2 cross-drive copy | ✅ | `POST /api/backup/tier2` → 8 HDD apps → `/mnt/sys_drive/felhom-data/backups/secondary/<app>` (immich 156 MB on disk; romm appdata + recovery-unit{db-dump,manifest,compose}); `crossdrive_completed` events pushed to hub |
| Per-app restore | ✅ | `POST /backup/restore stack_name=romm` → `RestoreFromRecoveryUnit`: compose down → redeploy (6 env, 3 encrypted recovered) → `Imported DB dump romm-mariadb.sql` → containers up → "Restore-from-unit completed" |
| Restore non-hollow | ✅ | planted a marker table pre-restore; post-restore romm DB has the real `platforms` + 17 tables, 3 containers healthy; marker dropped afterward to restore baseline |

**Findings:**
- **`GET /api/backup/snapshots` has no server handler** (the api-router returns "endpoint not found").
  The restore UI's snapshot dropdown (`backups.html:682`) therefore cannot populate → a **pure-UI
  restore is blocked at snapshot selection.** The restore itself works via the form action
  `POST /backup/restore` (which calls `RestoreFromRecoveryUnit(stack)` and ignores `snapshot_id`'s value).
- **Restore DB import is additive** — `RestoreFromRecoveryUnit` replays the dump but does not drop
  tables absent from it (the planted marker table survived). Snapshot data is restored; stale/extra
  tables are not pruned. Minor behavioral note.

---

## Phase 4 — DR rebuild from backup (non-destructive to 9201)

Proven **twice**, both collision-safe (no tunnel/hub identity clash with the live 9201):

1. **Agent DR primitive** — `felhom-agent --selftest=restore-test -archive=<Phase-0 volid>`: restored
   into scratch **990000** → bind-mounts neutralized → **net link-down** → boot → `verified: boot+running`
   → torn down. `pass:true`, 2 m 56 s, self-cleaning.
2. **Deep manual restore into throwaway 9300** — `pct restore 9300 <archive> --storage local-lvm`.
   **Safety before start:** removed the shared binds `mp3:/mnt/felhom-drives` + `mp9:/etc/felhom-bootstrap`,
   removed `net0`, set `onboot 0` (kept only the restored `mp0` docker-data + rootfs — independent
   copies). Then started and inspected:
   - `pct status: running`; `felhom-controller Up (healthy)` inside.
   - `/opt/docker/stacks/` had all 57 stack dirs; romm `app.yaml` restored (`deployed:true`, env, locked_fields).
   - **`docker ps` inside 9300: 25 containers up** from restored volumes — identical set to 9201.
   - Restored data non-hollow: 9300 `romm-db` has 17 tables.
   - No collision: 9300 had **no eth0**; live 9201 `cloudflared Up` unchanged throughout.
   - Cleanup: `pct stop 9300 && pct destroy 9300` (both LVs removed); local-lvm freed.

**Core DR claim proven end-to-end:** PBS whole-CT restore → guest boots → controller + all app stacks +
DB data recovered from restored volumes. The bind-mount-neutralization recipe (remove host-path binds +
`net0` + `onboot` before start) is the safe way to inspect a restored customer image alongside a live
one.

---

## Phase 5 — Storage lifecycle — **BLOCKED by a controller→agent connectivity defect**

### 🔴 FINDING #1 (highest priority) — controller cannot reach the agent local API (8443)
The controller container's calls to the agent local API fail: `agentapi: GET /disks: dial tcp
192.168.0.162:8443: connect: cannot assign requested address` (EADDRNOTAVAIL, persistent — 6/6). The
same error surfaces in the Settings UI ("Meghajtók (ügynök nézet)") and in the backup page
("A host-ügynök jelenleg nem elérhető").

Reachability matrix **from the controller container**:
- `192.168.0.162:8006` (pveproxy, binds `0.0.0.0`) → **200** (reachable)
- `192.168.0.162:8443` (felhom-agent local API) → **refused / EADDRNOTAVAIL**
- `192.168.0.141:443` (guest's own traefik) → 404 (reachable)

From the guest **host namespace**, `192.168.0.162:8443` is OPEN. Confirmed: `ss` shows the agent
`LISTEN 192.168.0.162:8443` — **bound to the specific host LAN IP, not `0.0.0.0`** like pveproxy;
**no iptables rule** references 8443; **pve-firewall disabled**. The agent itself is healthy
(`--selftest=storage` enumerates all 5 targets with SMART/FS/durable_id fine). So this is a
**container→host-IP routing / source-bind mismatch on port 8443**, not a firewall and not an agent
crash.

**Impact — the entire agent-backed feature set is down in this demo:** storage management UI
(`/api/disks`, `/api/storage`: scan/label/eject/enroll/format/migrate/decommission), the live
host-metrics API (`/api/host-metrics`), and whole-guest backup trigger. (Monitoring still shows host
CPU/mem/temp — those are read from `/proc`, since the LXC shares the host kernel, not via the agent.)

**Fix direction (supervised):** bind the agent local API to `0.0.0.0` (or the `vmbr0` bridge address the
guest routes through), or repoint the controller's `local_api.endpoint` (bootstrap.json) at a
container-reachable address; then re-run Phase 5's mutating checks. Not changed in this run (network
reconfig is out of scope for an unattended test).

| Check | Result | Evidence |
|---|---|---|
| Scan / health / FS / model / durable_id (agent tier) | ✅ | `--selftest=storage`: 5 targets; felhom-usb SMART **PASSED** temp=35 poh=2736 realloc=0; felhom-flash uuid:81a26531; thin-pool data=9.7 % |
| Label edit + revert (UI) | ⛔ blocked | controller→agent down (finding #1) |
| Eject → re-enroll cycle (UI) | ⛔ blocked | same |
| Destructive loopback (scan→format→mount→migrate→decommission) | ⛔ skipped | wizard drives the agent via the broken path; agent-direct needs the per-guest token + TLS-pin reconstruction → high-effort/low-confidence unattended. **No scratch loopback created** (nothing to clean up). Deferred to supervised. |

---

## Phase 6 — Monitoring / alerts / notifications / settings / hub

| Check | Result | Evidence |
|---|---|---|
| Monitoring — host metrics | ✅ | CPU/load/mem (7.6/15.4 GB — host's 16 GB via `/proc`/LXC kernel-share)/temp/uptime |
| Monitoring — container/app metrics | ✅ | per-container memory breakdown (26 containers), per-app CPU/MEM charts, time-range charts 1h–30d |
| Alerts — state-based | ✅ | komga `0 ok, 1 unhealthy` before → `N ok` after the Phase-1 fix; badge red→green |
| Notification path (exercised ONCE) | ✅ | set email+events+cooldown → hub `Notification preferences updated for demo-felhom`. Test → hub **`Event from demo-felhom: test` → `Test email sent to nagyfenyvesi.viktor@gmail.com`** = controller→hub→Resend→email confirmed. One email only (§0.5) |
| Settings persistence | ✅ | notification prefs set → persisted to `settings.json` → reverted (both directions) |
| Settings — password change | n/a | password protection is **operator-only and not configured** ("A jelszavas védelem nincs beállítva"); dashboard currently serves with no auth wall |
| Hub — report received | ✅ | `Received report from demo-felhom (11983 bytes)` every 15 min; `host-report (3 guests, 5 storage targets, 1 backups, 1 restore-tests, 9 pbs-snapshots)`; UI "Hub: Kapcsolódva" |
| Hub — DR recipe carries the PBS coord | ✅ | `dr_recipe` (demo-felhom) host-half `pbs={"repo_id":"felhom-pbs","namespace":"root","latest_snapshot_id":"9201"}`; app-half 15 apps; **no secrets in either half** |
| Hub — update panel | ✅ | version 0.73.0, "naprakész", last check timestamp shown |
| Hub GUI tiles (analytics/healthchecks) | skipped | no operator GUI creds — verified hub-side state via logs + DB instead |

The DR-recipe app-half + host-half are both stored at the hub each report (`DR-recipe …-half stored for
customer demo-felhom (v1)`), and the assembled recipe carries the live PBS coordinate (the agent v0.39.0
live-reporter feature) — verified directly in the hub DB.

---

## Phase 7 — Resilience

| Check | Result | Evidence |
|---|---|---|
| Guest 9201 reboot | ✅ | `pct reboot 9201`; recovered to 26→**25** containers, 0 unhealthy in ~2 min |
| Drives re-bind | ✅ | felhom-flash (sdc1) + felhom-usb (sdb1) remounted ext4 at correct mountpoints |
| FileBrowser convergence | ✅ | binds reconverged to `<drive>/userdata → /srv/<drive>` |
| Tunnel reconnect | ✅ | cloudflared QUIC + HTTP/2 prechecks to region1/2 argotunnel pass |
| Controller + apps + Phase-1/2 fixes persist | ✅ | komga + gitea still healthy post-reboot |
| Host reboot | ⏭ skipped (§7) | unattended risk — deferred to supervised |

Incidental: the controller's docker container IP changed across the reboot
(`172.18.0.10`→`172.18.0.8`); harmless to the system (traefik routes by docker DNS name), only required
re-pointing the test forwarder.

---

## Consolidated findings (for supervised follow-up)

1. **🔴 controller→agent local-API (8443) unreachable** (Phase 5) — gates storage UI + host-metrics API
   + whole-guest backup. Agent binds `192.168.0.162` not `0.0.0.0`; no firewall; container reaches
   `:8006` but not `:8443` on the same host. **Highest priority.** Fix: bind `0.0.0.0`/bridge or
   repoint `local_api.endpoint`.
2. **`GET /api/backup/snapshots` has no handler** (Phase 3) — restore UI snapshot dropdown can't
   populate (restore works via `POST /backup/restore`).
3. **glance catalog template** never seeds `glance.yml` → crash-loop on deploy (Phase 2). Needs a
   default-config seed (init step / entrypoint), not a trivial HC fix.
4. **Restore DB import is additive** (Phase 3) — doesn't drop tables absent from the dump. Minor.
5. **On-LAN HTTPS to dashboard/apps 404s** at traefik `:443` for all hosts (cert valid) — supported
   path is Cloudflare; on-LAN HTTPS appears non-functional. Worth confirming the Cloudflare path is the
   intended sole ingress.
6. Minor: a few catalog logo SVGs 404 (adventurelog, cloudflared, crafty-controller); dashboard
   "Adatbázisok mentve" count flickered 3→2 post-reboot (cosmetic).

## Code shipped (app-catalog `main`, trunk)
- komga HC `/api/v1/actuator/health`(401) → `/actuator/health` (docker HC + `.felhom.yml` probe) —
  `3faa5ae`, `9b066de`.
- gitea HC `/api/v1/version`(404 pre-install) → `/api/healthz`, start_period 90 s — `17e00b7`.
- CHANGELOG + REPORT updated. **No felhom-controller / felhom-agent code changed** (catalog-only).

## Cleanup confirmation
- Throwaway guest **9300 destroyed**; restore-test scratch 990000 auto-torn-down; `pct list` =
  9201 (running) + pre-existing 9001/9999 (stopped, not created by this run).
- **No scratch loopback devices created** (destructive storage sub-phase skipped).
- Test apps removed (glance, gitea) → baseline app set.
- Notification settings reverted; password never set; drives never ejected.
- Test forwarder (`/tmp/felhom-test-fwd.py`) removed from the guest; temp `hub.db` copy removed from
  DooPlex. **DooPlex used in normal roles only** (PBS target, gitea/hub reads).
- **komga healthy** (improvement over the unhealthy baseline).

## After-state vs before
9201 running; **25 containers Up, 0 unhealthy** (before: komga unhealthy); controller v0.73.0; tunnel up;
5 storage targets reachable. Net result: demo restored to baseline **+ komga fixed**.

## Deferred — needs a supervised run
- **Agent 8443 reachability** root-cause + fix (finding #1) — gates the whole agent-backed subsystem.
- **Storage lifecycle UI** (label edit, eject→re-enroll, init-wizard format/migrate/decommission on a
  loopback) — re-run once finding #1 is fixed.
- **Host reboot** resilience (full host AC-recovery + remount-by-UUID under sdX reshuffle).
- **glance** config-seeding fix (finding #3); **`/api/backup/snapshots`** handler + UI snapshot
  selection (finding #2).