felhom.eu/documentation/audits/live-drive-findings-2026-06-14.md

LIVE-DRIVE-FINDINGS — felhom-controller end-to-end live drive

• Date: 2026-06-14
• Branch: audit/2026-06-14-live-drive
• Versions under test: controller v0.60.0, agent v0.30.0, hub v0.11.0 (per runbook)
• Target: demo guest 9201 (demo-felhom) on Proxmox host felhom-pve (192.168.0.162)
• Mode: unattended discovery drive — find bugs, do NOT fix. Destructive ops authorized (demo disposable).

Control-plane access method used

The demo dashboard has no password set, so the controller's RequireAuth and CsrfProtect middleware both skip (auth.go / csrf.go). The full REST API + web forms are therefore open over the public URL https://felhom.demo-felhom.eu. I drive the controller as a user would via that URL (NOT the container IP, which 404s on a network/mux nuance). Host SSH (root@felhom-pve) + pct exec/docker exec are used only for baseline setup and ground-truth verification — never to perform the operation under test.

• Verified: GET /api/health → {"ok":true}; GET /settings and GET / → 200 without auth.

---

EXECUTIVE SUMMARY

Data-migration verdict (up front): Cannot be exercised — and the current provisioning is broken for it. The felhom-usb HDD is mounted on the Proxmox host but is not passed through to guest 9201 (no LXC mountpoint), so there is no second/HDD drive to migrate to or from. Worse, /api/disks and /api/host-metrics still advertise the HDD as "attached/user-data/reachable" while the controller genuinely can't see it. HDD-requiring apps (13 of 55) are undeployable; an "HDD app" deployed anyway lands silently on the 32G OS rootfs. (F9, F11)

What works (verified end-to-end):

• App deploy orchestration (single + multi-container DB apps), lifecycle (start/stop/restart/update/logs/optional-config), removal + protected-stack guards. (F12, F18)
• DNS/routing for healthy apps (Traefik + Cloudflare tunnel + public URL → 200). (F5)
• Health detection: unhealthy and crash-loop states are correctly surfaced (no false "running"). (F5, F10)
• CTRL-T2-1: crash mid-image-pull leaves the stack not-deployed and cleanly redeployable. (F14)
• CTRL-001: malicious .fab (app_name traversal and tar-member traversal) rejected; nothing written outside the stacks dir. (F16)
• Import/export round-trip; backup tier-1 (recovery units + DB dumps) and PBS offsite whole-guest snapshots; hub reporting; sync; per-container metrics. (F16, F13, F19)
• AGENT-001 gate: refuses without confirmation, refuses a wrong durable_id, and authorizes only the correct durable_id-resolved device. (F20)

What's broken / risky:

• CRITICAL — Restore does not restore data. The per-app restore re-creates config + recovers secrets but never imports the DB dump and recovery units don't capture volume data. Only the PBS whole-guest snapshot can actually recover data. (F17)
• CRITICAL — HDD not attached to guest (data-migration verdict above). (F9)
• HIGH — Disk wipe execution: format errors are swallowed (ok:true on a failed mkfs) and a large-disk format times out and leaves the disk corrupt (mkfs killed mid-run). The security gate itself is fine. (F20)
• HIGH — Broken healthcheck = total outage. A wrong catalog healthcheck (uptime-kuma) makes the container "unhealthy", and Traefik then refuses to publish its route → the app URL is a hard 404 even though the app is running. (F5)
• HIGH — Memory metric reports host RAM (16GB), not the guest's 2GB cap → unsafe deploy-headroom basis. (F1)
• MEDIUM — 3-2-1 backup is weakened (tier-1 lives on the same OS disk as the data; no 2nd drive) (F13); restore snapshot_id ignored (F17); controller not auto-restarted after a manual stop (F15); durable_id scheme mismatch across disk APIs (F20).

Feature pass/fail tally:

#	Feature	Verdict
1	Multi-app deployment	PASS (optimistic POST msg — F6)
2	Health detection (incl. crash-loop)	PASS (F10) — but unhealthy→404 cascade (F5)
3	DNS / routing	PASS for healthy; design risk F5
4	Lifecycle ops	PASS (F12)
5	Backup (3-2-1)	PARTIAL — tier1+PBS work; 3-2-1 weak (F13)
6	Restore	FAIL — no data restore (F17)
7	App data migration (SSD↔HDD)	BLOCKED/N-A — no HDD attached (F9)
8	Removal	PASS (F18)
9	Import/export + CTRL-001	PASS (F16)
10a	CTRL-T2-1 crash-window	PASS (F14)
10b	AGENT-001 wipe	MIXED — gate PASS, execution broken (F20)
11	Monitoring/settings/sync/hub	PASS (F19) — except system mem metric (F1)

---

Progress log (timestamped, Europe/Budapest)

• ~start — Baseline captured. Controller :0.60.0 Up (healthy), agent 0.30.0 active. 55 stacks in catalog; deployed/running: actualbudget (only customer app) + protected infra traefik, cloudflared, filebrowser. RomM is currently not_deployed (memory/CONTEXT said it was the live HDD app — state has since changed). Headroom: rootfs 32G (29G free), docker-data 252G (238G free). Disks: felhom-usb (user-data, data_bearing, /dev/sdb1, uuid:277a2179…), local + local-lvm (system), felhom-pbs (backup, 192.168.0.180:felhom-spike). Hub-side online status TBD.
• drive complete — All 11 areas exercised. Deployed vikunja, gokapi, privatebin, uptime-kuma, mealie, romm (+ code-server for the crash test, since removed). Findings F1–F20 recorded with host-verified evidence. Destructive ops (restore, removal, AGENT-001 wipe) executed last. felhom-usb reformatted clean host-side after the authorized wipe. Hub reporting confirmed live (HTTP 200, report pushed). Demo left healthy and usable for the operator's UI pass; HDD passthrough still missing (F9).

---

BASELINE

Item	Value
Controller image/status	gitea.dooplex.hu/admin/felhom-controller:0.60.0 Up (healthy)
Agent	felhom-agent 0.30.0, systemd active
Guest rootfs	32G total, 29G avail (4% used)
Guest docker-data (/var/lib/docker)	252G total, 238G avail (1% used)
Guest RAM (LXC cgroup)	2048 MB (config memory: 2048, swap 512)
Deployed customer apps	actualbudget
Protected infra running	traefik, cloudflared, filebrowser, felhom-controller
Disks	felhom-usb (HDD, user-data), local/local-lvm (system), felhom-pbs (offsite backup)

---

FINDINGS

F1 — Memory metric reports HOST RAM (16GB), not the guest's 2GB cap — HIGH

• Area: §11 monitoring accuracy / deploy headroom guard
• Action: GET /api/system/info; cross-checked with free inside the LXC and the LXC config.
• Expected: memory total ≈ the guest's 2048 MB cgroup limit.
• Actual: /api/system/info → total_mem_mb: 15771, avail_mem_mb: 13577. The LXC is capped at 2048 MB (/etc/pve/lxc/9201.conf: memory: 2048); free -m inside the LXC correctly shows 2048. The controller Docker container reads the host /proc/meminfo (MemTotal: 16150380 kB ≈ 15772 MB) because lxcfs is not bind-mounted into the container and its cgroup memory.max = max (no Docker memory limit).
• Evidence:
  • LXC free -m: Mem: 2048 ... available 1829
  • controller container head -1 /proc/meminfo: MemTotal: 16150380 kB; cat /sys/fs/cgroup/memory.max → max
  • /etc/pve/lxc/9201.conf: memory: 2048, swap: 512
• Impact: The monitoring memory bar and (critically) the deploy memory-headroom guard believe the guest has ~13.5 GB free when only ~1.8 GB is usable. An operator deploying apps by that figure will OOM-kill the 2 GB guest. The 3-app bound in the runbook exists precisely because the real ceiling is 2 GB, which the UI hides.
• Verdict: broken (misleading metric + unsafe headroom basis). Severity: HIGH.

F2 — hdd_configured: false despite an attached user-data HDD — LOW (verify)

• /api/system/info reports hdd_configured: false even though felhom-usb (HDD, role user-data, data_bearing) is attached and mounted at /mnt/felhom-usb. To confirm whether this gates any HDD-app deploy UI. Severity TBD.

F3 — Hungarian catalog text double-UTF-8-encoded in API JSON — LOW (verify)

• GET /api/stacks returns descriptions like Személyes pénzügyek (mojibake for "Személyes pénzügyek"). Looks like double-encoding. Need to confirm whether the rendered HTML UI is affected or only the JSON API. Severity TBD.

F5 — uptime-kuma: broken catalog healthcheck → permanently unhealthy → total 404 outage via Traefik — HIGH

• Area: §2 deploy / §2b health detection / §3 routing — a cascade.
• Action: deployed uptime-kuma (POST /api/stacks/uptime-kuma/deploy), observed state, then traced the 404.
• Expected: deploy → healthy → status.demo-felhom.eu serves the app (200).
• Actual / evidence — the cascade:
  1. Catalog bug: /opt/docker/stacks/uptime-kuma/docker-compose.yml defines healthcheck.test: ["CMD","node","/app/extra/healthcheck.mjs"], but in the louislam/uptime-kuma:2 image that file does not exist → docker inspect health log: Error: Cannot find module '/app/extra/healthcheck.mjs', Health=unhealthy FailingStreak=5.
  2. The app process is actually fine — curl http://uptime-kuma:3001/ from a peer container → 302 (serving).
  3. Traefik gates route registration on Docker health. On the websecure (443) entrypoint, status.demo-felhom.eu → 404, while every healthy app (tasks/recipes/share/paste) → 200. The HTTP→HTTPS redirect on :80 returns 301 for all hosts (global catch-all), which masks the missing 443 router. So the unhealthy container's TLS router is never published.
  4. Net result: uptime-kuma is completely unreachable at its URL (404) for the customer, despite the app running — purely because of a wrong healthcheck path in the catalog.
• Verdict: broken. Severity: HIGH. Two issues to file: (a) catalog healthcheck wrong for uptime-kuma:2; (b) design risk — any app with a broken/too-slow healthcheck doesn't just show "unhealthy", it becomes a hard 404 outage. The controller's deploy returns success and the dashboard shows "deployed (unhealthy)", giving no hint that the URL is dead. Consider surfacing "route not published because unhealthy" to the operator.
• Good part: the controller did correctly detect and surface unhealthy (GET /api/stacks/uptime-kuma → state=unhealthy) — health detection itself works.

F6 — Deploy POST returns "deployed" optimistically, before compose completes / before health is known — MEDIUM (API contract)

• Action: timed POST /api/stacks/<app>/deploy vs controller logs.
• Evidence: vikunja POST returned ~0s but log shows compose took 3.4s; uptime-kuma POST returned ~0s but compose pull took 31.7s and the app ended unhealthy. The response {"ok":true,"message":"Stack <app> deployed"} is sent before the container is up and regardless of eventual health.
• Impact: This is the documented in-memory Deployed=true-before-compose up pattern (avoids the card flipping back mid-pull), and the UI compensates by polling GET /api/stacks/<app>. But the API message "deployed" is misleading — an API consumer (or a script) that trusts the POST result will think a broken/unhealthy app succeeded (see F5). Verdict: clunky/misleading message; not a data-integrity bug. Severity: MEDIUM.

F7 — API state lags Docker health by ~10s after deploy — LOW

• mealie's container reported (healthy) in docker ps ~12s before GET /api/stacks/mealie flipped from starting to running. Cosmetic polling lag; transient. Severity: LOW.

F8 — Infrastructure secrets stored in plaintext on the guest rootfs — LOW/INFO

• The cloudflared infra stack's compose holds TUNNEL_TOKEN=<redacted> in plaintext; /etc/felhom-controller/controller.yaml holds hub.api_key, infrastructure.cf_api_token, and infrastructure.cf_tunnel_token in plaintext (not enc:-wrapped like app.yaml secrets). These live on the guest rootfs (in PBS snapshots) and the controller needs them at runtime, so it may be acceptable, but it diverges from the "secrets encrypted at rest" posture applied to app secrets. All redacted here. Severity: LOW/INFO.

F19 — §11 misc (monitoring, settings, sync, geo, assets, hub reporting) — mostly PASS — INFO

• Per-container metrics: accurate. /api/metrics/containers/summary reports romm mem_usage_mb=278.1, limit 512; docker stats romm = 277.8MiB/512MiB — matches, and uses the correct per-container cgroup limits. /api/metrics/containers/{name} returns a CPU/memory time-series (works). Contrast with F1: only the system/guest memory metric is wrong (host 16GB), not the container metrics.
• Settings page /settings → 200. Sync (POST /api/sync, "Sablonok frissítése") → works ("Sablonok naprakészek — nincs változás"). Geo /api/geo/status → reachable (allowed:[HU], cf_configured:true, enabled:false). Asset sync /api/assets/status → not_configured (assets not synced from hub on this demo — note, not necessarily a bug).
• Hub reporting: works. Logs: Hub connectivity: https://hub.felhom.eu reachable (HTTP 200), Hub report pushed successfully (7658 bytes), periodic hub-report every 15m, startup report sent. Severity: INFO (PASS).

F9 — felhom-usb HDD is NOT passed through to guest 9201 → HDD apps undeployable; disk APIs misrepresent it as available — CRITICAL

• Area: §2 HDD deploy / §7b data migration / §11 storage.
• Action: POST /api/stacks/romm/deploy with HDD_PATH=/mnt/felhom-usb (the path /api/disks reports). Then traced the guest/container mount topology.
• Expected: an HDD-requiring app (romm) deploys onto the user-data HDD.
• Actual:
  • Deploy rejected at field validation: {"ok":false,"error":"path \"/mnt/felhom-usb\" does not exist for field \"Adattárolási útvonal\""}.
  • The HDD is not in the guest. /etc/pve/lxc/9201.conf has only mp0 (docker-data → /var/lib/docker) and mp9 (bootstrap, ro). No mountpoint for felhom-usb. findmnt in the guest shows /mnt is just pve-vm-9201-disk-0[/mnt] — a subdir of the 32G rootfs.
  • The controller's only "drive" is /mnt/sys_drive/felhom-data, which df inside the controller confirms is on /dev/mapper/pve-vm--9201--disk--0 (the 32G OS rootfs), not a data drive.
  • The HDD does exist on the Proxmox host: findmnt /mnt/felhom-usb → /dev/sdb1 ext4 915.8G, containing real data (felhom-data, images, storage, Dokumentumok, …). It is host-mounted but never bind-mounted into the LXC.
• Source-of-truth disconnect: /api/disks reports felhom-usb state:attached, role:user-data, data_bearing:true, mount_path:/mnt/felhom-usb; /api/host-metrics reports the same drive as content: "backup". Both are the agent's host-side view and say "attached/reachable", but the guest/controller genuinely cannot see it. Meanwhile /api/system/info correctly reports hdd_configured:false (the controller's local view). Three endpoints, two contradictory stories about whether the customer has usable external storage.
• Impact: All HDD-requiring apps (romm, nextcloud, immich, jellyfin, paperless-ngx, plex, navidrome, audiobookshelf, komga, calibre-web, emby, sonarr, radarr — 13 of 55) are undeployable. A user who reads /api/disks and enters /mnt/felhom-usb gets a confusing rejection. If they instead pick the offered sys_drive, their "external storage" app data silently lands on the 32G OS disk — the exact failure the OS/data-split slice was meant to prevent.
• Root context: the 2026-06-13 re-provision of 9201 left "RomM/USB re-enroll" as a documented pending step (CONTEXT.md); this drive confirms it was never completed and that the agent's disk APIs do not reflect the gap — they advertise the HDD as usable. The honest customer-facing status should be "external drive present on host but not attached to your server".
• Verdict: broken. Severity: CRITICAL (blocks a quarter of the catalog + risks silent data-on-OS-disk placement; API misrepresents storage availability). This is the data-migration verdict: cannot be exercised — there is no second/HDD drive attached to the guest to migrate to or from.

F10 — Crash-loop detection works (controller reports restarting, not false "running") — PASS (with a brief initial window) — INFO

• Area: §2b — the crash-loop-exit-0 trap.
• Method: created a fixture stack brokentest (alpine, command: sh -c 'echo booting; sleep 3; echo crashing; exit 1', restart: always) in the stacks dir, picked up on a controller restart, then deployed it via POST /api/stacks/brokentest/deploy and watched the state.
• Result: docker compose up -d returned exit 0 and the deploy POST returned {"ok":true,"message":"Stack brokentest deployed"} (F6 again). For the first ~10s the API reported state=running (the container was genuinely "Up" during its initial 3s sleep). Once it began crash-looping, the controller correctly transitioned to state=restarting and held there (docker oscillating Up↔Restarting (1)). It never falsely reported a stable "running"/healthy for a crashing container.
• Verdict: works. Caveat: there's a brief post-deploy window where a stack that will crash shows "running"; an operator watching the 3-step panel could see a momentary green before it flips to restarting. Severity: INFO (the lesson-#6 trap is handled).
• Note (deploy-page scan): a hand-created stack dir is only picked up by ScanStacks at controller startup — POST /api/sync (git catalog sync) does not rescan the local stacks dir, and there is no exposed manual-rescan endpoint (see F4). Minor operational gap.

F11 — "HDD-requiring" app deploys silently onto the 32G OS rootfs when no HDD is attached — MEDIUM

• Area: §2 / storage placement (related to F9).
• Action: with felhom-usb unavailable to the guest (F9), deployed romm with HDD_PATH=/mnt/sys_drive/felhom-data.
• Actual: accepted; romm + romm-db (MariaDB) + romm-redis came up healthy, route arcade.demo-felhom.eu → 200. Appdata landed at /mnt/sys_drive/felhom-data/appdata/romm, which df confirms is the 32G OS rootfs (disk-0). No warning that an HDD-class app is being placed on the OS disk.
• Impact: for a real customer with a media app (jellyfin/immich/plex), bulk media would fill the 32G rootfs and the OS-headroom guard becomes the only backstop. The deploy UI offered no "this is your OS disk, not external storage" caution. Severity: MEDIUM.
• Positive: the multi-container DB app (MariaDB + redis + app) deployed and reached healthy cleanly — the deploy orchestration itself is solid.

F12 — Lifecycle ops (start/stop/restart/update/logs/optional-config) — PASS — INFO

• stop (vikunja): state→stopped; container removed from docker ps -a (i.e. "stop" performs a compose down, not a stop that leaves an Exited container — semantic note, functionally fine).
• start / restart: container back Up; tasks.demo-felhom.eu → 200 after restart (route survived).
• update (gokapi): completed in ~1s; container not recreated (Up 18 minutes preserved) because the image was unchanged — correct no-op behavior (compose pull + up with no new image).
• logs (GET /api/stacks/vikunja/logs?raw=1): returns real container logs (migrations, http started), timestamps current.
• optional-config (romm IGDB_CLIENT_ID): POST /api/stacks/romm/optional-config → persisted to app.yaml (plaintext, correct for a non-secret text field), romm restarted, and the env var was verified inside the running container (IGDB_CLIENT_ID=testclientid12345). Full round-trip works.
• remove guard: POST /api/stacks/brokentest/remove while running → correctly refused ("still running — stop it first"). After stop, remove succeeded; the stack directory is intentionally left on disk (so a catalog app can be redeployed) — by design.
• Verdict: all lifecycle operations work and stay consistent with ground truth. Severity: INFO (PASS).

F13 — Backup tiers: tier-1 + PBS offsite WORK; tier-2 honest no-target; but "primary" lives on the same OS disk as the data — MIXED (3-2-1 weakened by F9)

• Area: §5 backup.
• Tier-1 (primary recovery units + DB dumps): POST /api/backup/run → success. Recovery units (compose/ + manifest.json) created for all 7 deployed apps; romm's MariaDB dumped to backups/primary/romm/db-dumps/romm-mariadb.sql (44K). db_dump count=1 is correct — mealie runs embedded SQLite (single container, no postgres sidecar), so it has no SQL dump (captured as a volume in its unit). Verdict: works.
• Tier-2 (off-drive secondary): POST /api/backup/tier2 → ran, processed 1 HDD app (romm), and honestly refused: log Tier 2 for romm: no off-drive target — nincs másik fizikai meghajtó — a 2. mentéshez 2. meghajtó szükséges; backup-data shows secondary/romm/rsync exists:false. No silent failure, no rootfs-fill. Verdict: works (correct no-target handling).
• Tier-3 (PBS offsite, whole-guest): POST /api/guest-backup/trigger → created a new PBS snapshot. GET /api/guest-backup/status → phase done, started 2026-06-13T22:28:55Z, success, size 5.99GB, mode snapshot, crash_consistent. Verified in PBS itself (read-only pvesm list felhom-pbs --vmid 9201): new snapshot felhom-pbs:backup/ct/9201/2026-06-13T22:28:56Z listed (3 snapshots total). Verdict: works end-to-end.
• The 3-2-1 weakness (HIGH-ish, consequence of F9): the "primary" tier (recovery units + DB dumps) is stored under /mnt/sys_drive/felhom-data/backups, which is the same 32G OS rootfs (disk-0) that holds the app data itself. So tier-1 gives zero protection against disk-0 loss — source and backup share a disk. With no second drive attached (F9), the only genuine off-host copy is PBS. Effective reality is "2 copies on the same disk + 1 PBS", not 3-2-1. The UI/dashboard 3-2-1 framing would overstate resilience here.
• Minor: /api/backup/status reports only db_dump + running/enabled — it does not surface recovery-unit or tier-2 status, so "is my backup complete?" requires multiple endpoints. The earlier guest-backup status briefly showed a stale prior run (20:27Z) before my new run appeared — momentary staleness, not a clock bug.

F14 — CTRL-T2-1 crash-window: crash mid-image-pull leaves stack NOT-deployed and redeployable — PASS — INFO

• Area: CTRL-T2-1 regression.
• Method: deployed code-server (uncached ~1GB image) with a password, confirmed the docker compose up -d pull subprocess was running, then docker kill felhom-controller mid-pull.
• On-disk state during the crash (read directly from the guest fs while controller down): app.yaml had deployed: false (only deployed_at was stamped at start; the deployed:true flag is written only on success).
• After restart: GET /api/stacks/code-server → state=not_deployed, deployed=False; no orphaned code-server container.
• Redeploy: allowed (no "already deployed; use update" refusal) → returned deployed → polled to running/healthy, code.demo-felhom.eu → 302, deployed: true persisted on success.
• Verdict: the fix works — no ghost-stuck stack. Severity: INFO (PASS).
• First-attempt note: an initial attempt without the (required) PASSWORD was rejected at validation before any pull — so a missing required field is caught pre-pull (good), and I had to supply the password to actually exercise the pull window.

F15 — docker kill/stop does NOT auto-restart the controller; bootstrap service is oneshot/Restart=no — MEDIUM (operational expectation)

• Area: §3 crash recovery / operational.
• Finding: the runbook states the bootstrap service "restarts it within seconds." In reality: felhom-controller-bootstrap.service is Type=oneshot, Restart=no (it ran once and exited). The container has --restart unless-stopped, but Docker treats an explicit docker kill/docker stop as a manual stop, so unless-stopped does not restart it (observed: RestartCount=0, stayed Exited (137) indefinitely; I had to docker start it manually).
• Impact: unless-stopped will restart the controller after a genuine unexpected death (panic/OOM/host reboot) — so real crash-recovery is covered. But there is no watchdog that recovers from a manual stop or a clean exit, and the bootstrap service does not re-run. An operator who docker stops the controller (or a tool that does) will find it stays down. The "auto-restarts within seconds" expectation is only partially true. Severity: MEDIUM (recovery assumption is narrower than documented).

F16 — Import/export round-trip + CTRL-001 path-traversal — PASS — INFO

• Export (vikunja → /mnt/sys_drive/felhom-data/exports/vikunja_*.fab): estimate → start → status all clean; .fab is a gzip tar (manifest.json, config/{.felhom.yml,app.yaml,docker-compose.yml}, data/volumes); manifest readable via POST /api/export/manifest (app_name, volume_names, config_files, encrypted flag). Works.
• Import round-trip: POST /api/export/import of the same .fab (over the live vikunja) succeeded; vikunja ended running, tasks.demo-felhom.eu → 200. Works.
• CTRL-001 — app_name traversal: crafted .fab with manifest.app_name = "../../../tmp/evil_appname" → import rejected at manifest parse: Manifest hiba: appexport: app_name "../../../tmp/evil_appname" must not contain a path separator; verified nothing written to /tmp, /opt, /opt/docker. PASS.
• CTRL-001 — tar-member traversal (tar-slip): crafted .fab with a valid app_name but a tar entry ../../../../tmp/evil_member → the member was not written to /tmp/evil_member (extraction sanitized the traversal). PASS. Minor: that bundle left an empty /opt/docker/stacks/eviltar dir (import got partway then failed "stack not found") — cleanup debt, not a security issue (I removed it).
• Verdict: export/import works; CTRL-001 is robustly enforced on both vectors with a clear error message. Severity: INFO (PASS).

F17 — Per-app restore does NOT restore data (DB dump never imported, volume tars not captured) — CRITICAL

• Area: §7 restore — the core disaster-recovery path.
• Method (real data-marker round-trip on romm/MariaDB):
  1. Inserted a marker row into romm's DB: felhom_restore_marker (1, 'MARKER-BEFORE-BACKUP-9921').
  2. POST /api/backup/run → confirmed the marker is in the captured dump (grep MARKER… romm-mariadb.sql → 1 hit).
  3. Dropped the table (simulated data loss) — confirmed gone.
  4. POST /backup/restore (stack_name=romm).
  5. Result: the marker table did NOT return — ERROR 1146: Table 'romm.felhom_restore_marker' doesn't exist after restore.
• What the restore actually did (from logs): stopped romm (compose down, volumes kept), secrets recovered=3/3, "Redeploying romm from recovery unit", compose up -d, "Restore-from-unit completed". The DB dump was never imported; the app simply came back on its surviving (post-drop) volume.
• Code confirmation: backup/restore_unit.go: RestoreFromRecoveryUnit → stackProvider.RecreateStackFromUnit(...) → (main.go:953) copies only docker-compose.yml + .felhom.yml from the unit and calls RedeployFromEnv. There is no DB-import (pg_restore/mysql </.sql replay) anywhere in internal/backup/*, and recovery units do not capture volume tars (F13). So the per-app restore is config + secrets only.
• Impact: the headline disaster-recovery feature does not recover data. A customer who loses DB content (corruption, bad migration, accidental delete) and clicks "restore" gets the app back with whatever was already in the volume — if the volume is intact, their loss is unchanged; if the volume is gone, the DB comes up empty. The carefully-captured DB dumps and recovery-unit manifests are effectively decorative for restore purposes. The only path that actually restores data is the PBS whole-guest snapshot (the agent's), which restores the entire LXC incl. the docker-data volume.
• What does work: secret recovery (3/3 from the guest's own app.yaml), the fail-closed data-key gate (Phase 2b, proven previously), and the app coming back healthy. The orchestration is fine; the data-replay step is simply missing.
• Verdict: broken. Severity: CRITICAL (restore advertised as data recovery does not restore data). Also: /backup/restore reads snapshot_id but ignores it (RestoreFromRecoveryUnit takes only the stack name) — there is no point-in-time selection; "restore" always means "re-create from the single latest unit."

F18 — Removal + protected-stack guards — PASS — INFO

• Protected stacks (traefik, cloudflared, filebrowser, felhom-controller): both stop and remove refused server-side with clear messages (cannot stop protected stack X / stack "X" is protected and cannot be removed); all confirmed still running. PASS.
• Full removal (code-server): stop → remove → container gone, no orphaned volume (docker volume ls empty), state=not_deployed, route code.demo-felhom.eu → 404 (Traefik route withdrawn with the container). Remove response carries explicit data-handling fields (volumes_removed, hdd_paths_removed, hdd_paths_preserved). PASS.
• Verdict: removal and protection guards work correctly. Severity: INFO (PASS).

F20 — AGENT-001 wipe gate: refusals + correct-device targeting PASS; but format error-reporting + large-disk timeout are broken — MIXED (gate PASS, execution HIGH)

• Area: §6 / AGENT-001.
• Refusal A (no confirmation): POST /api/disks/format {device:/dev/sdb1, fstype:ext4} → formatted:false, needs_confirmation:true, "customer confirmation required", HTTP 409. No mkfs. PASS.
• Refusal B (wrong durable_id): +confirmed:true, durable_id:byid:wwn-0xDEADBEEF… → formatted:false, "operator authorization required", HTTP 409 (agent log: reason=binding_mismatch). A non-matching confirmation does not authorize. PASS.
• Gate authorization (correct durable_id): with confirmed:true, durable_id:byid:wwn-0x5000039ddb108568-part1 the agent gate progressed pending_confirmation → binding_mismatch → customer_confirmed allowed=true and targeted the correct re-resolved device /dev/sdb1. The anti-retarget durable_id resolution works. PASS (gate logic).
• BUG 1 — swallowed format error (HIGH): the first confirmed attempt failed because /dev/sdb1 was mounted (mkfs.ext4 … is mounted; will not make a filesystem here!, agent logged level=ERROR), yet the API returned ok:true with formatted:false, device:"", reason:"". The failure was not propagated — a caller/UI would read it as success while nothing happened.
• BUG 2 — durable_id scheme mismatch across endpoints (MEDIUM): /api/disks reports this drive's durable_id as uuid:277a2179-…, but the format gate expects/returns byid:wwn-0x5000039ddb108568-part1. A customer/UI copying the id from the disk list would always be refused (binding_mismatch). The two APIs disagree on the durable_id scheme.
• BUG 3 — large-disk format times out and leaves the disk corrupt (HIGH): after unmounting (the gate won't unmount itself), the confirmed format on the 915GB drive ran mkfs, but the controller→agent call timed out (agentapi: POST /disks/format … context deadline exceeded, HTTP 502). The cancelled request context killed the in-progress mkfs (agent log: mkfs.ext4 … signal: killed). Result: /dev/sdb1 is left half-formatted — old fs/UUID/LABEL destroyed, blkid shows only PARTUUID, mount fails bad superblock. The disk is neither cleanly wiped nor usable. For a real customer wiping a large drive this corrupts it.
• Net: AGENT-001's security properties (refuse without confirmation, refuse wrong durable_id, target the durable_id-resolved device) are intact and were proven on the correct device. But the format execution path has two real defects (silent failure on mounted device; timeout-kill leaving a corrupt fs on large disks). Severity: gate PASS; execution HIGH.
• Disk final state: felhom-usb (/dev/sdb1) old data destroyed, left without a valid filesystem (see "Final state" section).

F4 — /api/stacks/rescan returns "stack not found: rescan" — LOW

• The runbook's documented rescan endpoint GET /api/stacks/rescan is routed as a stack name lookup → {"ok":false,"error":"stack not found: rescan"}. Either the route was removed/renamed or the runbook is stale. (Sync/rescan is reachable via POST /api/sync.) Cosmetic but documents a stale/missing endpoint.

---

PRIORITIZED ISSUE LIST (severity-ordered, with repro)

1. [CRITICAL] F17 — Per-app restore does not restore data. Restore re-creates config + recovers secrets but never imports the captured DB dump; recovery units don't capture volume tars. Repro: insert a row in romm's DB → POST /api/backup/run (row is in romm-mariadb.sql) → drop the row → POST /backup/restore stack_name=romm → row does NOT return. Only PBS whole-guest restore recovers data.
2. [CRITICAL] F9 — HDD not passed through to guest; HDD apps undeployable; disk APIs misrepresent it. Repro: POST /api/stacks/romm/deploy HDD_PATH=/mnt/felhom-usb → path … does not exist; /etc/pve/lxc/9201.conf has no mp for felhom-usb; /api/disks still lists it attached/user-data.
3. [HIGH] F20 (BUG1+BUG3) — Disk wipe execution. (a) mkfs failure on a mounted device returns ok:true (swallowed error). (b) Large-disk format times out (agentapi … context deadline exceeded), the cancelled context kills mkfs → disk left corrupt (bad superblock). Repro: POST /api/disks/format confirmed on a 915GB drive → 502 + half-formatted device. (The gate logic itself is correct.)
4. [HIGH] F5 — Broken healthcheck → total 404 outage. uptime-kuma's catalog healthcheck points to a nonexistent file → container "unhealthy" → Traefik never publishes the route → status.demo-felhom.eu = 404 though the app runs. Repro: deploy uptime-kuma; curl https://status.demo-felhom.eu/ → 404; docker inspect health log shows Cannot find module '/app/extra/healthcheck.mjs'.
5. [HIGH] F1 — Memory metric reports host RAM (16GB), not the guest's 2GB cap. /api/system/info → total_mem_mb:15771; LXC is capped at 2048. Unsafe deploy-headroom basis.
6. [MEDIUM] F13 — 3-2-1 weakened: tier-1 backups live on the same OS disk as the data; no 2nd drive (consequence of F9). Only PBS is genuinely off-host.
7. [MEDIUM] F15 — Controller not auto-restarted after a manual docker stop/kill (bootstrap is oneshot/Restart=no; unless-stopped ignores manual stops). Real crashes (panic/OOM) are covered.
8. [MEDIUM] F20 BUG2 — durable_id scheme mismatch: /api/disks uses uuid:…, the format gate uses byid:wwn:…; copying the disk-list id into a format confirmation is always refused.
9. [MEDIUM] F6 — Deploy POST returns "deployed" before compose completes / regardless of health (misleading for API/script consumers; UI compensates by polling).
10. [LOW] F17b — /backup/restore ignores snapshot_id (no point-in-time selection).
11. [LOW] F2 — hdd_configured:false vs /api/disks listing the HDD (two sources disagree; F9 root).
12. [LOW] F8 — Infra secrets in plaintext (cloudflared TUNNEL_TOKEN; controller.yaml cf/hub tokens).
13. [LOW] F7 — API state lags Docker health ~10s post-deploy. F4 — stale rescan endpoint. F3 — JSON API double-encodes Hungarian text (HTML UI fine).

---

UX / "this felt wrong" observations (for the operator's UI pass tomorrow)

• uptime-kuma will show as deployed-but-unhealthy with a dead URL (404). Check whether the dashboard gives any hint that the route is unpublished — currently it just looks "unhealthy". This is the most confusing customer-facing case. (F5)
• Storage page vs reality: the dashboard/disk views advertise the felhom-usb HDD as available, but no app can actually use it. Watch how the deploy "drive picker" presents storage — does it offer felhom-usb (which would fail) and/or the OS-disk "sys_drive" (which silently fills the 32G OS disk)? (F9, F11)
• Memory bar likely shows ~16GB. The deploy page memory bar / headroom is based on host RAM, not the 2GB guest cap — looks like tons of free RAM. (F1)
• Restore UI implies data recovery. Confirm what the restore screen promises; in practice it only restores config/secrets, not data. A customer could believe their data is safe via per-app restore when it isn't. (F17)
• Format/wipe result: an attempted format can report success (ok:true) when nothing happened, or spin/time out on a large disk. Watch the wipe UI's success/failure messaging and progress on a big drive. (F20)
• Deploy 3-step panel may briefly show "running"/green before a doomed stack flips to restarting; and the per-card "deployed" flips on before health is known. (F6, F10)
• Hungarian text in the JSON API is mojibake (Személyes), though the rendered HTML is correct — only relevant if any UI reads the JSON directly. (F3)

---

What I could NOT test (and why)

• App data migration SSD↔HDD — no HDD is attached to the guest (F9); there is no second drive to migrate to/from. The feature is effectively untestable in the current provisioning state; this is the answer.
• Full readable-data restore e2e for a large app (e.g. AdventureLog) — independently moot given F17 (restore doesn't replay data), and historically blocked by the 8/32G rootfs.
• HDD-app real behavior (RomM with an imported library, media apps) — undeployable to a real HDD (F9); romm was exercised on the OS-disk fallback only.
• Cloudflare geo-restriction enforcement live — enabled:false on the demo; only confirmed the geo subsystem is reachable/configured (HU allowlist), not active WAF behavior. (F19)
• Asset sync from hub — not_configured on this demo. (F19)
• Tier-2 off-drive backup actually copying — no off-disk target exists (F9); only the honest no-target path was exercised. (F13)
• Genuine unexpected-crash auto-restart (panic/OOM) — could not induce a real panic; confirmed docker kill does NOT trigger restart and reasoned that unless-stopped covers genuine deaths. (F15)

---

FINAL STATE OF THE DEMO

Usable as-is for the operator's UI testing — does NOT need a Proxmox redeploy (but the HDD passthrough must be re-established for full functionality).

• Controller gitea.dooplex.hu/admin/felhom-controller:0.60.0 Up (healthy); agent 0.30.0 active; GET /api/health ok.
• Deployed customer apps (all healthy except uptime-kuma): actualbudget, gokapi, mealie, privatebin, romm (+romm-db, romm-redis), vikunja. uptime-kuma is deployed but unhealthy with a 404 URL — left in place intentionally as a live example of F5 for the operator to inspect.
• Protected infra healthy: traefik, cloudflared, filebrowser.
• Headroom: rootfs 32G (29G free), docker-data 252G (234G free), RAM 2048M (~1187M available).
• felhom-usb HDD: its old data was destroyed by the authorized AGENT-001 wipe test; I reformatted it host-side to a clean empty ext4 (new UUID ce9d1d81-…, LABEL hdd_1) and remounted it at /mnt/felhom-usb on the host. It remains NOT attached to guest 9201 (the F9 gap is pre-existing; I did not change the LXC config). To make HDD apps and data-migration work, an mp for felhom-usb must be added to the guest.
• Test artifacts cleaned up: brokentest fixture (container + dir) removed; code-server removed; malicious .fab bundles and the empty eviltar dir removed; romm's test marker table is gone (expected — it demonstrated F17).
• romm's app data lives on the OS rootfs (/mnt/sys_drive/felhom-data, F11) — fine for a demo, not where it should be in production.

Bottom line: the control plane, deploy/lifecycle, routing, health detection, import/export + path-traversal defense, CTRL-T2-1, backup-to-PBS, and the wipe gate are solid. The two things a customer would most need in a crisis — restoring their data and using their external drive — do not work in the current build/provisioning (F17, F9), and the disk-wipe execution can corrupt a large drive (F20). Those three should be the focus before this is customer-ready.