# LIVE-DRIVE-FINDINGS — felhom-controller end-to-end live drive

- **Date:** 2026-06-14
- **Branch:** `audit/2026-06-14-live-drive`
- **Versions under test:** controller **v0.60.0**, agent **v0.30.0**, hub v0.11.0 (per runbook)
- **Target:** demo guest **9201** (`demo-felhom`) on Proxmox host `felhom-pve` (192.168.0.162)
- **Mode:** unattended discovery drive — find bugs, do NOT fix. Destructive ops authorized (demo disposable).

## Control-plane access method used

The demo dashboard has **no password set**, so the controller's `RequireAuth` and `CsrfProtect` middleware both skip (auth.go / csrf.go). The full REST API + web forms are therefore **open over the public URL** `https://felhom.demo-felhom.eu`. I drive the controller as a user would via that URL (NOT the container IP, which 404s on a network/mux nuance). Host SSH (`root@felhom-pve`) + `pct exec`/`docker exec` are used **only** for baseline setup and ground-truth verification — never to perform the operation under test.

- Verified: `GET /api/health` → `{"ok":true}`; `GET /settings` and `GET /` → 200 without auth.

---

## Progress log (timestamped, Europe/Budapest)

- **~start** — Baseline captured. Controller `:0.60.0 Up (healthy)`, agent `0.30.0 active`. 55 stacks in catalog; deployed/running: **actualbudget** (only customer app) + protected infra **traefik, cloudflared, filebrowser**. RomM is currently **not_deployed** (memory/CONTEXT said it was the live HDD app — state has since changed). Headroom: rootfs 32G (29G free), docker-data 252G (238G free). **Disks:** felhom-usb (user-data, data_bearing, /dev/sdb1, uuid:277a2179…), local + local-lvm (system), felhom-pbs (backup, 192.168.0.180:felhom-spike). Hub-side online status TBD.

---

## BASELINE

| Item | Value |
|---|---|
| Controller image/status | `gitea.dooplex.hu/admin/felhom-controller:0.60.0` Up (healthy) |
| Agent | `felhom-agent 0.30.0`, systemd active |
| Guest rootfs | 32G total, 29G avail (4% used) |
| Guest docker-data (/var/lib/docker) | 252G total, 238G avail (1% used) |
| Guest RAM (LXC cgroup) | **2048 MB** (config `memory: 2048`, swap 512) |
| Deployed customer apps | actualbudget |
| Protected infra running | traefik, cloudflared, filebrowser, felhom-controller |
| Disks | felhom-usb (HDD, user-data), local/local-lvm (system), felhom-pbs (offsite backup) |

---

## FINDINGS

### F1 — Memory metric reports HOST RAM (16GB), not the guest's 2GB cap — HIGH
- **Area:** §11 monitoring accuracy / deploy headroom guard
- **Action:** `GET /api/system/info`; cross-checked with `free` inside the LXC and the LXC config.
- **Expected:** memory total ≈ the guest's 2048 MB cgroup limit.
- **Actual:** `/api/system/info` → `total_mem_mb: 15771`, `avail_mem_mb: 13577`. The LXC is capped at **2048 MB** (`/etc/pve/lxc/9201.conf: memory: 2048`); `free -m` inside the LXC correctly shows 2048. The controller **Docker container** reads the host `/proc/meminfo` (`MemTotal: 16150380 kB` ≈ 15772 MB) because lxcfs is not bind-mounted into the container and its cgroup `memory.max = max` (no Docker memory limit).
- **Evidence:**
  - LXC `free -m`: `Mem: 2048 ... available 1829`
  - controller container `head -1 /proc/meminfo`: `MemTotal: 16150380 kB`; `cat /sys/fs/cgroup/memory.max` → `max`
  - `/etc/pve/lxc/9201.conf`: `memory: 2048`, `swap: 512`
- **Impact:** The monitoring memory bar and (critically) the **deploy memory-headroom guard** believe the guest has ~13.5 GB free when only ~1.8 GB is usable. An operator deploying apps by that figure will OOM-kill the 2 GB guest. The 3-app bound in the runbook exists precisely because the real ceiling is 2 GB, which the UI hides.
- **Verdict:** broken (misleading metric + unsafe headroom basis). **Severity: HIGH.**

### F2 — `hdd_configured: false` despite an attached user-data HDD — LOW (verify)
- `/api/system/info` reports `hdd_configured: false` even though felhom-usb (HDD, role user-data, data_bearing) is attached and mounted at /mnt/felhom-usb. To confirm whether this gates any HDD-app deploy UI. Severity TBD.

### F3 — Hungarian catalog text double-UTF-8-encoded in API JSON — LOW (verify)
- `GET /api/stacks` returns descriptions like `SzemĂ©lyes pĂ©nzĂĽgyek` (mojibake for "Személyes pénzügyek"). Looks like double-encoding. Need to confirm whether the rendered HTML UI is affected or only the JSON API. Severity TBD.

### F5 — uptime-kuma: broken catalog healthcheck → permanently unhealthy → **total 404 outage via Traefik** — HIGH
- **Area:** §2 deploy / §2b health detection / §3 routing — a cascade.
- **Action:** deployed uptime-kuma (`POST /api/stacks/uptime-kuma/deploy`), observed state, then traced the 404.
- **Expected:** deploy → healthy → status.demo-felhom.eu serves the app (200).
- **Actual / evidence — the cascade:**
  1. **Catalog bug:** `/opt/docker/stacks/uptime-kuma/docker-compose.yml` defines `healthcheck.test: ["CMD","node","/app/extra/healthcheck.mjs"]`, but in the `louislam/uptime-kuma:2` image that file does not exist → `docker inspect` health log: `Error: Cannot find module '/app/extra/healthcheck.mjs'`, `Health=unhealthy FailingStreak=5`.
  2. The **app process is actually fine** — `curl http://uptime-kuma:3001/` from a peer container → **302** (serving).
  3. **Traefik gates route registration on Docker health.** On the websecure (443) entrypoint, `status.demo-felhom.eu → 404`, while every healthy app (tasks/recipes/share/paste) → 200. The HTTP→HTTPS redirect on :80 returns 301 for *all* hosts (global catch-all), which masks the missing 443 router. So the unhealthy container's TLS router is never published.
  4. **Net result:** uptime-kuma is **completely unreachable at its URL (404)** for the customer, despite the app running — purely because of a wrong healthcheck path in the catalog.
- **Verdict:** broken. **Severity: HIGH.** Two issues to file: (a) catalog healthcheck wrong for uptime-kuma:2; (b) **design risk** — *any* app with a broken/too-slow healthcheck doesn't just show "unhealthy", it becomes a hard 404 outage. The controller's deploy returns success and the dashboard shows "deployed (unhealthy)", giving no hint that the URL is dead. Consider surfacing "route not published because unhealthy" to the operator.
- **Good part:** the controller **did correctly detect and surface `unhealthy`** (`GET /api/stacks/uptime-kuma` → `state=unhealthy`) — health detection itself works.

### F6 — Deploy POST returns `"deployed"` optimistically, before compose completes / before health is known — MEDIUM (API contract)
- **Action:** timed `POST /api/stacks/<app>/deploy` vs controller logs.
- **Evidence:** vikunja POST returned ~0s but log shows compose took 3.4s; uptime-kuma POST returned ~0s but compose pull took **31.7s** and the app ended **unhealthy**. The response `{"ok":true,"message":"Stack <app> deployed"}` is sent before the container is up and regardless of eventual health.
- **Impact:** This is the documented in-memory `Deployed=true`-before-`compose up` pattern (avoids the card flipping back mid-pull), and the UI compensates by polling `GET /api/stacks/<app>`. But the **API message "deployed" is misleading** — an API consumer (or a script) that trusts the POST result will think a broken/unhealthy app succeeded (see F5). Verdict: clunky/misleading message; not a data-integrity bug. **Severity: MEDIUM.**

### F7 — API `state` lags Docker health by ~10s after deploy — LOW
- mealie's container reported `(healthy)` in `docker ps` ~12s before `GET /api/stacks/mealie` flipped from `starting` to `running`. Cosmetic polling lag; transient. **Severity: LOW.**

### F8 — cloudflared `TUNNEL_TOKEN` stored in plaintext in docker-compose.yml — LOW/INFO
- The cloudflared infra stack's compose holds `TUNNEL_TOKEN=<redacted>` in plaintext on disk (not `enc:`-wrapped like app.yaml secrets). May be acceptable for an infra/base-bringup stack, but worth confirming against the "secrets must be encrypted at rest" posture. Token redacted here. **Severity: LOW/INFO.**

### F9 — felhom-usb HDD is NOT passed through to guest 9201 → HDD apps undeployable; disk APIs misrepresent it as available — CRITICAL
- **Area:** §2 HDD deploy / §7b data migration / §11 storage.
- **Action:** `POST /api/stacks/romm/deploy` with `HDD_PATH=/mnt/felhom-usb` (the path `/api/disks` reports). Then traced the guest/container mount topology.
- **Expected:** an HDD-requiring app (romm) deploys onto the user-data HDD.
- **Actual:**
  - Deploy **rejected** at field validation: `{"ok":false,"error":"path \"/mnt/felhom-usb\" does not exist for field \"Adattárolási útvonal\""}`.
  - **The HDD is not in the guest.** `/etc/pve/lxc/9201.conf` has only `mp0` (docker-data → /var/lib/docker) and `mp9` (bootstrap, ro). **No mountpoint for felhom-usb.** `findmnt` in the guest shows `/mnt` is just `pve-vm-9201-disk-0[/mnt]` — a subdir of the **32G rootfs**.
  - The controller's only "drive" is `/mnt/sys_drive/felhom-data`, which `df` inside the controller confirms is on `/dev/mapper/pve-vm--9201--disk--0` (**the 32G OS rootfs**), not a data drive.
  - **The HDD does exist on the Proxmox host**: `findmnt /mnt/felhom-usb` → `/dev/sdb1 ext4 915.8G`, containing real data (`felhom-data`, `images`, `storage`, `Dokumentumok`, …). It is host-mounted but never bind-mounted into the LXC.
- **Source-of-truth disconnect:** `/api/disks` reports felhom-usb `state:attached, role:user-data, data_bearing:true, mount_path:/mnt/felhom-usb`; `/api/host-metrics` reports the *same* drive as `content: "backup"`. Both are the **agent's host-side view** and say "attached/reachable", but the guest/controller genuinely cannot see it. Meanwhile `/api/system/info` correctly reports `hdd_configured:false` (the controller's local view). Three endpoints, two contradictory stories about whether the customer has usable external storage.
- **Impact:** **All HDD-requiring apps (romm, nextcloud, immich, jellyfin, paperless-ngx, plex, navidrome, audiobookshelf, komga, calibre-web, emby, sonarr, radarr — 13 of 55) are undeployable.** A user who reads `/api/disks` and enters `/mnt/felhom-usb` gets a confusing rejection. If they instead pick the offered `sys_drive`, their "external storage" app data silently lands on the **32G OS disk** — the exact failure the OS/data-split slice was meant to prevent.
- **Root context:** the 2026-06-13 re-provision of 9201 left "RomM/USB re-enroll" as a documented pending step (CONTEXT.md); this drive confirms it was never completed **and** that the agent's disk APIs do not reflect the gap — they advertise the HDD as usable. The honest customer-facing status should be "external drive present on host but not attached to your server".
- **Verdict:** broken. **Severity: CRITICAL** (blocks a quarter of the catalog + risks silent data-on-OS-disk placement; API misrepresents storage availability). **This is the data-migration verdict: cannot be exercised — there is no second/HDD drive attached to the guest to migrate to or from.**

### F10 — Crash-loop detection works (controller reports `restarting`, not false "running") — PASS (with a brief initial window) — INFO
- **Area:** §2b — the crash-loop-exit-0 trap.
- **Method:** created a fixture stack `brokentest` (`alpine`, `command: sh -c 'echo booting; sleep 3; echo crashing; exit 1'`, `restart: always`) in the stacks dir, picked up on a controller restart, then deployed it via `POST /api/stacks/brokentest/deploy` and watched the state.
- **Result:** `docker compose up -d` returned exit 0 and the deploy POST returned `{"ok":true,"message":"Stack brokentest deployed"}` (F6 again). For the first ~10s the API reported `state=running` (the container was genuinely "Up" during its initial 3s sleep). Once it began crash-looping, the controller **correctly transitioned to `state=restarting`** and held there (docker oscillating Up↔`Restarting (1)`). It never falsely reported a stable "running"/healthy for a crashing container.
- **Verdict:** works. Caveat: there's a brief post-deploy window where a stack that will crash shows "running"; an operator watching the 3-step panel could see a momentary green before it flips to restarting. **Severity: INFO** (the lesson-#6 trap is handled).
- **Note (deploy-page scan):** a hand-created stack dir is only picked up by `ScanStacks` at controller **startup** — `POST /api/sync` (git catalog sync) does not rescan the local stacks dir, and there is no exposed manual-rescan endpoint (see F4). Minor operational gap.

### F11 — "HDD-requiring" app deploys silently onto the 32G OS rootfs when no HDD is attached — MEDIUM
- **Area:** §2 / storage placement (related to F9).
- **Action:** with felhom-usb unavailable to the guest (F9), deployed romm with `HDD_PATH=/mnt/sys_drive/felhom-data`.
- **Actual:** accepted; romm + romm-db (MariaDB) + romm-redis came up **healthy**, route `arcade.demo-felhom.eu` → 200. Appdata landed at `/mnt/sys_drive/felhom-data/appdata/romm`, which `df` confirms is the **32G OS rootfs (disk-0)**. No warning that an HDD-class app is being placed on the OS disk.
- **Impact:** for a real customer with a media app (jellyfin/immich/plex), bulk media would fill the 32G rootfs and the OS-headroom guard becomes the only backstop. The deploy UI offered no "this is your OS disk, not external storage" caution. **Severity: MEDIUM.**
- **Positive:** the multi-container DB app (MariaDB + redis + app) deployed and reached healthy cleanly — the deploy orchestration itself is solid.

### F12 — Lifecycle ops (start/stop/restart/update/logs/optional-config) — PASS — INFO
- **stop** (vikunja): `state→stopped`; container removed from `docker ps -a` (i.e. "stop" performs a compose `down`, not a `stop` that leaves an Exited container — semantic note, functionally fine).
- **start / restart**: container back Up; `tasks.demo-felhom.eu` → 200 after restart (route survived).
- **update** (gokapi): completed in ~1s; container **not** recreated (`Up 18 minutes` preserved) because the image was unchanged — correct no-op behavior (compose pull + up with no new image).
- **logs** (`GET /api/stacks/vikunja/logs?raw=1`): returns real container logs (migrations, http started), timestamps current.
- **optional-config** (romm `IGDB_CLIENT_ID`): `POST /api/stacks/romm/optional-config` → persisted to app.yaml (plaintext, correct for a non-secret text field), romm restarted, and the env var was verified **inside the running container** (`IGDB_CLIENT_ID=testclientid12345`). Full round-trip works.
- **remove guard**: `POST /api/stacks/brokentest/remove` while running → correctly **refused** (`"still running — stop it first"`). After stop, remove succeeded; the stack **directory is intentionally left** on disk (so a catalog app can be redeployed) — by design.
- **Verdict:** all lifecycle operations work and stay consistent with ground truth. **Severity: INFO (PASS).**

### F13 — Backup tiers: tier-1 + PBS offsite WORK; tier-2 honest no-target; but "primary" lives on the same OS disk as the data — MIXED (3-2-1 weakened by F9)
- **Area:** §5 backup.
- **Tier-1 (primary recovery units + DB dumps):** `POST /api/backup/run` → success. Recovery units (`compose/` + `manifest.json`) created for **all 7 deployed apps**; romm's MariaDB dumped to `backups/primary/romm/db-dumps/romm-mariadb.sql` (44K). `db_dump count=1` is **correct** — mealie runs embedded SQLite (single container, no postgres sidecar), so it has no SQL dump (captured as a volume in its unit). **Verdict: works.**
- **Tier-2 (off-drive secondary):** `POST /api/backup/tier2` → ran, processed 1 HDD app (romm), and **honestly refused**: log `Tier 2 for romm: no off-drive target — nincs másik fizikai meghajtó — a 2. mentéshez 2. meghajtó szükséges`; `backup-data` shows `secondary/romm/rsync exists:false`. No silent failure, no rootfs-fill. **Verdict: works (correct no-target handling).**
- **Tier-3 (PBS offsite, whole-guest):** `POST /api/guest-backup/trigger` → created a **new** PBS snapshot. `GET /api/guest-backup/status` → `phase done, started 2026-06-13T22:28:55Z, success, size 5.99GB, mode snapshot, crash_consistent`. Verified in PBS itself (read-only `pvesm list felhom-pbs --vmid 9201`): new snapshot `felhom-pbs:backup/ct/9201/2026-06-13T22:28:56Z` listed (3 snapshots total). **Verdict: works end-to-end.**
- **The 3-2-1 weakness (HIGH-ish, consequence of F9):** the "primary" tier (recovery units + DB dumps) is stored under `/mnt/sys_drive/felhom-data/backups`, which is the **same 32G OS rootfs (disk-0)** that holds the app data itself. So tier-1 gives **zero protection against disk-0 loss** — source and backup share a disk. With no second drive attached (F9), the only genuine off-host copy is PBS. Effective reality is **"2 copies on the same disk + 1 PBS"**, not 3-2-1. The UI/dashboard 3-2-1 framing would overstate resilience here.
- **Minor:** `/api/backup/status` reports only `db_dump` + `running`/`enabled` — it does not surface recovery-unit or tier-2 status, so "is my backup complete?" requires multiple endpoints. The earlier guest-backup status briefly showed a **stale prior run** (20:27Z) before my new run appeared — momentary staleness, not a clock bug.

### F14 — CTRL-T2-1 crash-window: crash mid-image-pull leaves stack NOT-deployed and redeployable — PASS — INFO
- **Area:** CTRL-T2-1 regression.
- **Method:** deployed `code-server` (uncached ~1GB image) with a password, confirmed the `docker compose up -d` pull subprocess was running, then `docker kill felhom-controller` mid-pull.
- **On-disk state during the crash (read directly from the guest fs while controller down):** `app.yaml` had `deployed: false` (only `deployed_at` was stamped at start; the `deployed:true` flag is written **only on success**).
- **After restart:** `GET /api/stacks/code-server` → `state=not_deployed, deployed=False`; **no orphaned code-server container**.
- **Redeploy:** allowed (no "already deployed; use update" refusal) → returned deployed → polled to `running`/healthy, `code.demo-felhom.eu` → 302, `deployed: true` persisted on success.
- **Verdict:** the fix works — no ghost-stuck stack. **Severity: INFO (PASS).**
- **First-attempt note:** an initial attempt without the (required) PASSWORD was rejected at validation **before any pull** — so a missing required field is caught pre-pull (good), and I had to supply the password to actually exercise the pull window.

### F15 — `docker kill`/`stop` does NOT auto-restart the controller; bootstrap service is oneshot/Restart=no — MEDIUM (operational expectation)
- **Area:** §3 crash recovery / operational.
- **Finding:** the runbook states the bootstrap service "restarts it within seconds." In reality: `felhom-controller-bootstrap.service` is `Type=oneshot, Restart=no` (it ran once and exited). The container has `--restart unless-stopped`, but Docker treats an explicit `docker kill`/`docker stop` as a **manual stop**, so `unless-stopped` does **not** restart it (observed: `RestartCount=0`, stayed `Exited (137)` indefinitely; I had to `docker start` it manually).
- **Impact:** `unless-stopped` *will* restart the controller after a genuine unexpected death (panic/OOM/host reboot) — so real crash-recovery is covered. But there is **no watchdog** that recovers from a manual stop or a clean exit, and the bootstrap service does not re-run. An operator who `docker stop`s the controller (or a tool that does) will find it stays down. The "auto-restarts within seconds" expectation is only partially true. **Severity: MEDIUM** (recovery assumption is narrower than documented).

### F16 — Import/export round-trip + CTRL-001 path-traversal — PASS — INFO
- **Export** (vikunja → `/mnt/sys_drive/felhom-data/exports/vikunja_*.fab`): estimate → start → status all clean; `.fab` is a gzip tar (`manifest.json`, `config/{.felhom.yml,app.yaml,docker-compose.yml}`, `data/volumes`); manifest readable via `POST /api/export/manifest` (app_name, volume_names, config_files, encrypted flag). **Works.**
- **Import round-trip:** `POST /api/export/import` of the same `.fab` (over the live vikunja) succeeded; vikunja ended `running`, `tasks.demo-felhom.eu` → 200. **Works.**
- **CTRL-001 — app_name traversal:** crafted `.fab` with `manifest.app_name = "../../../tmp/evil_appname"` → import **rejected at manifest parse**: `Manifest hiba: appexport: app_name "../../../tmp/evil_appname" must not contain a path separator`; verified **nothing written** to `/tmp`, `/opt`, `/opt/docker`. **PASS.**
- **CTRL-001 — tar-member traversal (tar-slip):** crafted `.fab` with a valid app_name but a tar entry `../../../../tmp/evil_member` → the member was **not** written to `/tmp/evil_member` (extraction sanitized the traversal). **PASS.** Minor: that bundle left an **empty `/opt/docker/stacks/eviltar` dir** (import got partway then failed "stack not found") — cleanup debt, not a security issue (I removed it).
- **Verdict:** export/import works; CTRL-001 is robustly enforced on both vectors with a clear error message. **Severity: INFO (PASS).**

### F17 — Per-app restore does NOT restore data (DB dump never imported, volume tars not captured) — CRITICAL
- **Area:** §7 restore — the core disaster-recovery path.
- **Method (real data-marker round-trip on romm/MariaDB):**
  1. Inserted a marker row into romm's DB: `felhom_restore_marker (1, 'MARKER-BEFORE-BACKUP-9921')`.
  2. `POST /api/backup/run` → confirmed the marker **is in the captured dump** (`grep MARKER… romm-mariadb.sql` → 1 hit).
  3. Dropped the table (simulated data loss) — confirmed gone.
  4. `POST /backup/restore` (`stack_name=romm`).
  5. **Result: the marker table did NOT return** — `ERROR 1146: Table 'romm.felhom_restore_marker' doesn't exist` after restore.
- **What the restore actually did (from logs):** stopped romm (`compose down`, volumes kept), `secrets recovered=3/3`, "Redeploying romm from recovery unit", `compose up -d`, "Restore-from-unit completed". The DB dump was **never imported**; the app simply came back on its surviving (post-drop) volume.
- **Code confirmation:** `backup/restore_unit.go: RestoreFromRecoveryUnit` → `stackProvider.RecreateStackFromUnit(...)` → (`main.go:953`) copies only `docker-compose.yml` + `.felhom.yml` from the unit and calls `RedeployFromEnv`. There is **no DB-import** (`pg_restore`/`mysql <`/`.sql` replay) anywhere in `internal/backup/*`, and recovery units **do not capture volume tars** (F13). So the per-app restore is **config + secrets only**.
- **Impact:** the headline disaster-recovery feature does not recover data. A customer who loses DB content (corruption, bad migration, accidental delete) and clicks "restore" gets the app back with **whatever was already in the volume** — if the volume is intact, their loss is unchanged; if the volume is gone, the DB comes up **empty**. The carefully-captured DB dumps and recovery-unit manifests are effectively decorative for restore purposes. The only path that actually restores data is the **PBS whole-guest snapshot** (the agent's), which restores the entire LXC incl. the docker-data volume.
- **What does work:** secret recovery (3/3 from the guest's own app.yaml), the fail-closed data-key gate (Phase 2b, proven previously), and the app coming back healthy. The orchestration is fine; the data-replay step is simply missing.
- **Verdict:** broken. **Severity: CRITICAL** (restore advertised as data recovery does not restore data). Also: `/backup/restore` reads `snapshot_id` but **ignores it** (`RestoreFromRecoveryUnit` takes only the stack name) — there is no point-in-time selection; "restore" always means "re-create from the single latest unit."

### F4 — `/api/stacks/rescan` returns "stack not found: rescan" — LOW
- The runbook's documented rescan endpoint `GET /api/stacks/rescan` is routed as a stack name lookup → `{"ok":false,"error":"stack not found: rescan"}`. Either the route was removed/renamed or the runbook is stale. (Sync/rescan is reachable via `POST /api/sync`.) Cosmetic but documents a stale/missing endpoint.