Live drive: finalize report — exec summary, prioritized issues, UX, couldn't-test, final state
This commit is contained in:
@@ -14,9 +14,49 @@ The demo dashboard has **no password set**, so the controller's `RequireAuth` an
|
||||
|
||||
---
|
||||
|
||||
## EXECUTIVE SUMMARY
|
||||
|
||||
**Data-migration verdict (up front):** **Cannot be exercised — and the current provisioning is broken for it.** The felhom-usb HDD is mounted on the Proxmox *host* but is **not passed through to guest 9201** (no LXC mountpoint), so there is no second/HDD drive to migrate to or from. Worse, `/api/disks` and `/api/host-metrics` still advertise the HDD as "attached/user-data/reachable" while the controller genuinely can't see it. HDD-requiring apps (13 of 55) are undeployable; an "HDD app" deployed anyway lands silently on the 32G OS rootfs. (F9, F11)
|
||||
|
||||
**What works (verified end-to-end):**
|
||||
- App **deploy** orchestration (single + multi-container DB apps), **lifecycle** (start/stop/restart/update/logs/optional-config), **removal** + protected-stack guards. (F12, F18)
|
||||
- **DNS/routing** for healthy apps (Traefik + Cloudflare tunnel + public URL → 200). (F5)
|
||||
- **Health detection**: unhealthy and crash-loop states are correctly surfaced (no false "running"). (F5, F10)
|
||||
- **CTRL-T2-1**: crash mid-image-pull leaves the stack not-deployed and cleanly redeployable. (F14)
|
||||
- **CTRL-001**: malicious `.fab` (app_name traversal **and** tar-member traversal) rejected; nothing written outside the stacks dir. (F16)
|
||||
- **Import/export** round-trip; **backup tier-1** (recovery units + DB dumps) and **PBS offsite** whole-guest snapshots; **hub reporting**; **sync**; per-container **metrics**. (F16, F13, F19)
|
||||
- **AGENT-001 gate**: refuses without confirmation, refuses a wrong durable_id, and authorizes only the correct durable_id-resolved device. (F20)
|
||||
|
||||
**What's broken / risky:**
|
||||
- **CRITICAL — Restore does not restore data.** The per-app restore re-creates config + recovers secrets but **never imports the DB dump** and recovery units **don't capture volume data**. Only the PBS whole-guest snapshot can actually recover data. (F17)
|
||||
- **CRITICAL — HDD not attached to guest** (data-migration verdict above). (F9)
|
||||
- **HIGH — Disk wipe execution**: format errors are swallowed (`ok:true` on a failed mkfs) and a large-disk format **times out and leaves the disk corrupt** (mkfs killed mid-run). The security gate itself is fine. (F20)
|
||||
- **HIGH — Broken healthcheck = total outage.** A wrong catalog healthcheck (uptime-kuma) makes the container "unhealthy", and Traefik then refuses to publish its route → the app URL is a hard **404** even though the app is running. (F5)
|
||||
- **HIGH — Memory metric reports host RAM (16GB), not the guest's 2GB cap** → unsafe deploy-headroom basis. (F1)
|
||||
- **MEDIUM** — 3-2-1 backup is weakened (tier-1 lives on the same OS disk as the data; no 2nd drive) (F13); restore `snapshot_id` ignored (F17); controller not auto-restarted after a manual stop (F15); durable_id scheme mismatch across disk APIs (F20).
|
||||
|
||||
**Feature pass/fail tally:**
|
||||
| # | Feature | Verdict |
|
||||
|---|---|---|
|
||||
| 1 | Multi-app deployment | PASS (optimistic POST msg — F6) |
|
||||
| 2 | Health detection (incl. crash-loop) | PASS (F10) — but unhealthy→404 cascade (F5) |
|
||||
| 3 | DNS / routing | PASS for healthy; design risk F5 |
|
||||
| 4 | Lifecycle ops | PASS (F12) |
|
||||
| 5 | Backup (3-2-1) | PARTIAL — tier1+PBS work; 3-2-1 weak (F13) |
|
||||
| 6 | **Restore** | **FAIL — no data restore (F17)** |
|
||||
| 7 | **App data migration (SSD↔HDD)** | **BLOCKED/N-A — no HDD attached (F9)** |
|
||||
| 8 | Removal | PASS (F18) |
|
||||
| 9 | Import/export + CTRL-001 | PASS (F16) |
|
||||
| 10a | CTRL-T2-1 crash-window | PASS (F14) |
|
||||
| 10b | AGENT-001 wipe | MIXED — gate PASS, execution broken (F20) |
|
||||
| 11 | Monitoring/settings/sync/hub | PASS (F19) — except system mem metric (F1) |
|
||||
|
||||
---
|
||||
|
||||
## Progress log (timestamped, Europe/Budapest)
|
||||
|
||||
- **~start** — Baseline captured. Controller `:0.60.0 Up (healthy)`, agent `0.30.0 active`. 55 stacks in catalog; deployed/running: **actualbudget** (only customer app) + protected infra **traefik, cloudflared, filebrowser**. RomM is currently **not_deployed** (memory/CONTEXT said it was the live HDD app — state has since changed). Headroom: rootfs 32G (29G free), docker-data 252G (238G free). **Disks:** felhom-usb (user-data, data_bearing, /dev/sdb1, uuid:277a2179…), local + local-lvm (system), felhom-pbs (backup, 192.168.0.180:felhom-spike). Hub-side online status TBD.
|
||||
- **drive complete** — All 11 areas exercised. Deployed vikunja, gokapi, privatebin, uptime-kuma, mealie, romm (+ code-server for the crash test, since removed). Findings F1–F20 recorded with host-verified evidence. Destructive ops (restore, removal, AGENT-001 wipe) executed last. felhom-usb reformatted clean host-side after the authorized wipe. Hub reporting confirmed live (HTTP 200, report pushed). Demo left healthy and usable for the operator's UI pass; HDD passthrough still missing (F9).
|
||||
|
||||
---
|
||||
|
||||
@@ -181,3 +221,61 @@ The demo dashboard has **no password set**, so the controller's `RequireAuth` an
|
||||
|
||||
### F4 — `/api/stacks/rescan` returns "stack not found: rescan" — LOW
|
||||
- The runbook's documented rescan endpoint `GET /api/stacks/rescan` is routed as a stack name lookup → `{"ok":false,"error":"stack not found: rescan"}`. Either the route was removed/renamed or the runbook is stale. (Sync/rescan is reachable via `POST /api/sync`.) Cosmetic but documents a stale/missing endpoint.
|
||||
|
||||
---
|
||||
|
||||
## PRIORITIZED ISSUE LIST (severity-ordered, with repro)
|
||||
|
||||
1. **[CRITICAL] F17 — Per-app restore does not restore data.** Restore re-creates config + recovers secrets but never imports the captured DB dump; recovery units don't capture volume tars. **Repro:** insert a row in romm's DB → `POST /api/backup/run` (row is in `romm-mariadb.sql`) → drop the row → `POST /backup/restore stack_name=romm` → row does NOT return. Only PBS whole-guest restore recovers data.
|
||||
2. **[CRITICAL] F9 — HDD not passed through to guest; HDD apps undeployable; disk APIs misrepresent it.** **Repro:** `POST /api/stacks/romm/deploy HDD_PATH=/mnt/felhom-usb` → `path … does not exist`; `/etc/pve/lxc/9201.conf` has no mp for felhom-usb; `/api/disks` still lists it `attached/user-data`.
|
||||
3. **[HIGH] F20 (BUG1+BUG3) — Disk wipe execution.** (a) mkfs failure on a mounted device returns `ok:true` (swallowed error). (b) Large-disk format times out (`agentapi … context deadline exceeded`), the cancelled context kills mkfs → disk left corrupt (`bad superblock`). **Repro:** `POST /api/disks/format` confirmed on a 915GB drive → 502 + half-formatted device. (The gate logic itself is correct.)
|
||||
4. **[HIGH] F5 — Broken healthcheck → total 404 outage.** uptime-kuma's catalog healthcheck points to a nonexistent file → container "unhealthy" → Traefik never publishes the route → `status.demo-felhom.eu` = 404 though the app runs. **Repro:** deploy uptime-kuma; `curl https://status.demo-felhom.eu/` → 404; `docker inspect` health log shows `Cannot find module '/app/extra/healthcheck.mjs'`.
|
||||
5. **[HIGH] F1 — Memory metric reports host RAM (16GB), not the guest's 2GB cap.** `/api/system/info` → `total_mem_mb:15771`; LXC is capped at 2048. Unsafe deploy-headroom basis.
|
||||
6. **[MEDIUM] F13 — 3-2-1 weakened:** tier-1 backups live on the same OS disk as the data; no 2nd drive (consequence of F9). Only PBS is genuinely off-host.
|
||||
7. **[MEDIUM] F15 — Controller not auto-restarted after a manual `docker stop`/`kill`** (bootstrap is oneshot/Restart=no; `unless-stopped` ignores manual stops). Real crashes (panic/OOM) are covered.
|
||||
8. **[MEDIUM] F20 BUG2 — durable_id scheme mismatch:** `/api/disks` uses `uuid:…`, the format gate uses `byid:wwn:…`; copying the disk-list id into a format confirmation is always refused.
|
||||
9. **[MEDIUM] F6 — Deploy POST returns `"deployed"` before compose completes / regardless of health** (misleading for API/script consumers; UI compensates by polling).
|
||||
10. **[LOW] F17b — `/backup/restore` ignores `snapshot_id`** (no point-in-time selection).
|
||||
11. **[LOW] F2 — `hdd_configured:false`** vs `/api/disks` listing the HDD (two sources disagree; F9 root).
|
||||
12. **[LOW] F8 — Infra secrets in plaintext** (cloudflared TUNNEL_TOKEN; controller.yaml cf/hub tokens).
|
||||
13. **[LOW] F7 — API state lags Docker health ~10s post-deploy.** F4 — stale rescan endpoint. F3 — JSON API double-encodes Hungarian text (HTML UI fine).
|
||||
|
||||
---
|
||||
|
||||
## UX / "this felt wrong" observations (for the operator's UI pass tomorrow)
|
||||
|
||||
- **uptime-kuma will show as deployed-but-unhealthy with a dead URL (404).** Check whether the dashboard gives any hint that the *route* is unpublished — currently it just looks "unhealthy". This is the most confusing customer-facing case. (F5)
|
||||
- **Storage page vs reality:** the dashboard/disk views advertise the felhom-usb HDD as available, but no app can actually use it. Watch how the deploy "drive picker" presents storage — does it offer felhom-usb (which would fail) and/or the OS-disk "sys_drive" (which silently fills the 32G OS disk)? (F9, F11)
|
||||
- **Memory bar likely shows ~16GB.** The deploy page memory bar / headroom is based on host RAM, not the 2GB guest cap — looks like tons of free RAM. (F1)
|
||||
- **Restore UI implies data recovery.** Confirm what the restore screen promises; in practice it only restores config/secrets, not data. A customer could believe their data is safe via per-app restore when it isn't. (F17)
|
||||
- **Format/wipe result:** an attempted format can report success (`ok:true`) when nothing happened, or spin/time out on a large disk. Watch the wipe UI's success/failure messaging and progress on a big drive. (F20)
|
||||
- **Deploy 3-step panel** may briefly show "running"/green before a doomed stack flips to restarting; and the per-card "deployed" flips on before health is known. (F6, F10)
|
||||
- Hungarian text in the **JSON API** is mojibake (`SzemĂ©lyes`), though the rendered HTML is correct — only relevant if any UI reads the JSON directly. (F3)
|
||||
|
||||
---
|
||||
|
||||
## What I could NOT test (and why)
|
||||
|
||||
- **App data migration SSD↔HDD** — no HDD is attached to the guest (F9); there is no second drive to migrate to/from. The feature is effectively untestable in the current provisioning state; this *is* the answer.
|
||||
- **Full readable-data restore e2e for a large app** (e.g. AdventureLog) — independently moot given F17 (restore doesn't replay data), and historically blocked by the 8/32G rootfs.
|
||||
- **HDD-app real behavior** (RomM with an imported library, media apps) — undeployable to a real HDD (F9); romm was exercised on the OS-disk fallback only.
|
||||
- **Cloudflare geo-restriction enforcement live** — `enabled:false` on the demo; only confirmed the geo subsystem is reachable/configured (HU allowlist), not active WAF behavior. (F19)
|
||||
- **Asset sync from hub** — `not_configured` on this demo. (F19)
|
||||
- **Tier-2 off-drive backup actually copying** — no off-disk target exists (F9); only the honest no-target path was exercised. (F13)
|
||||
- **Genuine unexpected-crash auto-restart** (panic/OOM) — could not induce a real panic; confirmed `docker kill` does NOT trigger restart and reasoned that `unless-stopped` covers genuine deaths. (F15)
|
||||
|
||||
---
|
||||
|
||||
## FINAL STATE OF THE DEMO
|
||||
|
||||
**Usable as-is for the operator's UI testing — does NOT need a Proxmox redeploy** (but the HDD passthrough must be re-established for full functionality).
|
||||
|
||||
- Controller `gitea.dooplex.hu/admin/felhom-controller:0.60.0` **Up (healthy)**; agent **0.30.0 active**; `GET /api/health` ok.
|
||||
- **Deployed customer apps (all healthy except uptime-kuma):** actualbudget, gokapi, mealie, privatebin, romm (+romm-db, romm-redis), vikunja. **uptime-kuma is deployed but `unhealthy` with a 404 URL** — left in place intentionally as a live example of F5 for the operator to inspect.
|
||||
- Protected infra healthy: traefik, cloudflared, filebrowser.
|
||||
- Headroom: rootfs 32G (29G free), docker-data 252G (234G free), RAM 2048M (~1187M available).
|
||||
- **felhom-usb HDD:** its old data was destroyed by the authorized AGENT-001 wipe test; I reformatted it host-side to a clean empty ext4 (new UUID `ce9d1d81-…`, LABEL `hdd_1`) and remounted it at `/mnt/felhom-usb` on the host. **It remains NOT attached to guest 9201** (the F9 gap is pre-existing; I did not change the LXC config). To make HDD apps and data-migration work, an `mp` for felhom-usb must be added to the guest.
|
||||
- Test artifacts cleaned up: brokentest fixture (container + dir) removed; code-server removed; malicious `.fab` bundles and the empty `eviltar` dir removed; romm's test marker table is gone (expected — it demonstrated F17).
|
||||
- romm's app data lives on the **OS rootfs** (`/mnt/sys_drive/felhom-data`, F11) — fine for a demo, not where it should be in production.
|
||||
|
||||
**Bottom line:** the control plane, deploy/lifecycle, routing, health detection, import/export + path-traversal defense, CTRL-T2-1, backup-to-PBS, and the wipe *gate* are solid. The two things a customer would most need in a crisis — **restoring their data** and **using their external drive** — do not work in the current build/provisioning (F17, F9), and the **disk-wipe execution** can corrupt a large drive (F20). Those three should be the focus before this is customer-ready.
|
||||
|
||||
Reference in New Issue
Block a user