chore: relocate live-drive findings/fixspec to felhom.eu/documentation/audits; drop transient session logs
The live-drive findings + fixspec now live in felhom.eu/documentation/audits/ (with the other audit records); they no longer belong loose at the controller repo root. Transient SESSION-*.md working logs dropped (their lasting content is in CHANGELOG/CONTEXT and the audit records). Trunk-based no-branches cleanup.
This commit is contained in:
@@ -1,281 +0,0 @@
|
||||
# LIVE-DRIVE-FINDINGS — felhom-controller end-to-end live drive
|
||||
|
||||
- **Date:** 2026-06-14
|
||||
- **Branch:** `audit/2026-06-14-live-drive`
|
||||
- **Versions under test:** controller **v0.60.0**, agent **v0.30.0**, hub v0.11.0 (per runbook)
|
||||
- **Target:** demo guest **9201** (`demo-felhom`) on Proxmox host `felhom-pve` (192.168.0.162)
|
||||
- **Mode:** unattended discovery drive — find bugs, do NOT fix. Destructive ops authorized (demo disposable).
|
||||
|
||||
## Control-plane access method used
|
||||
|
||||
The demo dashboard has **no password set**, so the controller's `RequireAuth` and `CsrfProtect` middleware both skip (auth.go / csrf.go). The full REST API + web forms are therefore **open over the public URL** `https://felhom.demo-felhom.eu`. I drive the controller as a user would via that URL (NOT the container IP, which 404s on a network/mux nuance). Host SSH (`root@felhom-pve`) + `pct exec`/`docker exec` are used **only** for baseline setup and ground-truth verification — never to perform the operation under test.
|
||||
|
||||
- Verified: `GET /api/health` → `{"ok":true}`; `GET /settings` and `GET /` → 200 without auth.
|
||||
|
||||
---
|
||||
|
||||
## EXECUTIVE SUMMARY
|
||||
|
||||
**Data-migration verdict (up front):** **Cannot be exercised — and the current provisioning is broken for it.** The felhom-usb HDD is mounted on the Proxmox *host* but is **not passed through to guest 9201** (no LXC mountpoint), so there is no second/HDD drive to migrate to or from. Worse, `/api/disks` and `/api/host-metrics` still advertise the HDD as "attached/user-data/reachable" while the controller genuinely can't see it. HDD-requiring apps (13 of 55) are undeployable; an "HDD app" deployed anyway lands silently on the 32G OS rootfs. (F9, F11)
|
||||
|
||||
**What works (verified end-to-end):**
|
||||
- App **deploy** orchestration (single + multi-container DB apps), **lifecycle** (start/stop/restart/update/logs/optional-config), **removal** + protected-stack guards. (F12, F18)
|
||||
- **DNS/routing** for healthy apps (Traefik + Cloudflare tunnel + public URL → 200). (F5)
|
||||
- **Health detection**: unhealthy and crash-loop states are correctly surfaced (no false "running"). (F5, F10)
|
||||
- **CTRL-T2-1**: crash mid-image-pull leaves the stack not-deployed and cleanly redeployable. (F14)
|
||||
- **CTRL-001**: malicious `.fab` (app_name traversal **and** tar-member traversal) rejected; nothing written outside the stacks dir. (F16)
|
||||
- **Import/export** round-trip; **backup tier-1** (recovery units + DB dumps) and **PBS offsite** whole-guest snapshots; **hub reporting**; **sync**; per-container **metrics**. (F16, F13, F19)
|
||||
- **AGENT-001 gate**: refuses without confirmation, refuses a wrong durable_id, and authorizes only the correct durable_id-resolved device. (F20)
|
||||
|
||||
**What's broken / risky:**
|
||||
- **CRITICAL — Restore does not restore data.** The per-app restore re-creates config + recovers secrets but **never imports the DB dump** and recovery units **don't capture volume data**. Only the PBS whole-guest snapshot can actually recover data. (F17)
|
||||
- **CRITICAL — HDD not attached to guest** (data-migration verdict above). (F9)
|
||||
- **HIGH — Disk wipe execution**: format errors are swallowed (`ok:true` on a failed mkfs) and a large-disk format **times out and leaves the disk corrupt** (mkfs killed mid-run). The security gate itself is fine. (F20)
|
||||
- **HIGH — Broken healthcheck = total outage.** A wrong catalog healthcheck (uptime-kuma) makes the container "unhealthy", and Traefik then refuses to publish its route → the app URL is a hard **404** even though the app is running. (F5)
|
||||
- **HIGH — Memory metric reports host RAM (16GB), not the guest's 2GB cap** → unsafe deploy-headroom basis. (F1)
|
||||
- **MEDIUM** — 3-2-1 backup is weakened (tier-1 lives on the same OS disk as the data; no 2nd drive) (F13); restore `snapshot_id` ignored (F17); controller not auto-restarted after a manual stop (F15); durable_id scheme mismatch across disk APIs (F20).
|
||||
|
||||
**Feature pass/fail tally:**
|
||||
| # | Feature | Verdict |
|
||||
|---|---|---|
|
||||
| 1 | Multi-app deployment | PASS (optimistic POST msg — F6) |
|
||||
| 2 | Health detection (incl. crash-loop) | PASS (F10) — but unhealthy→404 cascade (F5) |
|
||||
| 3 | DNS / routing | PASS for healthy; design risk F5 |
|
||||
| 4 | Lifecycle ops | PASS (F12) |
|
||||
| 5 | Backup (3-2-1) | PARTIAL — tier1+PBS work; 3-2-1 weak (F13) |
|
||||
| 6 | **Restore** | **FAIL — no data restore (F17)** |
|
||||
| 7 | **App data migration (SSD↔HDD)** | **BLOCKED/N-A — no HDD attached (F9)** |
|
||||
| 8 | Removal | PASS (F18) |
|
||||
| 9 | Import/export + CTRL-001 | PASS (F16) |
|
||||
| 10a | CTRL-T2-1 crash-window | PASS (F14) |
|
||||
| 10b | AGENT-001 wipe | MIXED — gate PASS, execution broken (F20) |
|
||||
| 11 | Monitoring/settings/sync/hub | PASS (F19) — except system mem metric (F1) |
|
||||
|
||||
---
|
||||
|
||||
## Progress log (timestamped, Europe/Budapest)
|
||||
|
||||
- **~start** — Baseline captured. Controller `:0.60.0 Up (healthy)`, agent `0.30.0 active`. 55 stacks in catalog; deployed/running: **actualbudget** (only customer app) + protected infra **traefik, cloudflared, filebrowser**. RomM is currently **not_deployed** (memory/CONTEXT said it was the live HDD app — state has since changed). Headroom: rootfs 32G (29G free), docker-data 252G (238G free). **Disks:** felhom-usb (user-data, data_bearing, /dev/sdb1, uuid:277a2179…), local + local-lvm (system), felhom-pbs (backup, 192.168.0.180:felhom-spike). Hub-side online status TBD.
|
||||
- **drive complete** — All 11 areas exercised. Deployed vikunja, gokapi, privatebin, uptime-kuma, mealie, romm (+ code-server for the crash test, since removed). Findings F1–F20 recorded with host-verified evidence. Destructive ops (restore, removal, AGENT-001 wipe) executed last. felhom-usb reformatted clean host-side after the authorized wipe. Hub reporting confirmed live (HTTP 200, report pushed). Demo left healthy and usable for the operator's UI pass; HDD passthrough still missing (F9).
|
||||
|
||||
---
|
||||
|
||||
## BASELINE
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| Controller image/status | `gitea.dooplex.hu/admin/felhom-controller:0.60.0` Up (healthy) |
|
||||
| Agent | `felhom-agent 0.30.0`, systemd active |
|
||||
| Guest rootfs | 32G total, 29G avail (4% used) |
|
||||
| Guest docker-data (/var/lib/docker) | 252G total, 238G avail (1% used) |
|
||||
| Guest RAM (LXC cgroup) | **2048 MB** (config `memory: 2048`, swap 512) |
|
||||
| Deployed customer apps | actualbudget |
|
||||
| Protected infra running | traefik, cloudflared, filebrowser, felhom-controller |
|
||||
| Disks | felhom-usb (HDD, user-data), local/local-lvm (system), felhom-pbs (offsite backup) |
|
||||
|
||||
---
|
||||
|
||||
## FINDINGS
|
||||
|
||||
### F1 — Memory metric reports HOST RAM (16GB), not the guest's 2GB cap — HIGH
|
||||
- **Area:** §11 monitoring accuracy / deploy headroom guard
|
||||
- **Action:** `GET /api/system/info`; cross-checked with `free` inside the LXC and the LXC config.
|
||||
- **Expected:** memory total ≈ the guest's 2048 MB cgroup limit.
|
||||
- **Actual:** `/api/system/info` → `total_mem_mb: 15771`, `avail_mem_mb: 13577`. The LXC is capped at **2048 MB** (`/etc/pve/lxc/9201.conf: memory: 2048`); `free -m` inside the LXC correctly shows 2048. The controller **Docker container** reads the host `/proc/meminfo` (`MemTotal: 16150380 kB` ≈ 15772 MB) because lxcfs is not bind-mounted into the container and its cgroup `memory.max = max` (no Docker memory limit).
|
||||
- **Evidence:**
|
||||
- LXC `free -m`: `Mem: 2048 ... available 1829`
|
||||
- controller container `head -1 /proc/meminfo`: `MemTotal: 16150380 kB`; `cat /sys/fs/cgroup/memory.max` → `max`
|
||||
- `/etc/pve/lxc/9201.conf`: `memory: 2048`, `swap: 512`
|
||||
- **Impact:** The monitoring memory bar and (critically) the **deploy memory-headroom guard** believe the guest has ~13.5 GB free when only ~1.8 GB is usable. An operator deploying apps by that figure will OOM-kill the 2 GB guest. The 3-app bound in the runbook exists precisely because the real ceiling is 2 GB, which the UI hides.
|
||||
- **Verdict:** broken (misleading metric + unsafe headroom basis). **Severity: HIGH.**
|
||||
|
||||
### F2 — `hdd_configured: false` despite an attached user-data HDD — LOW (verify)
|
||||
- `/api/system/info` reports `hdd_configured: false` even though felhom-usb (HDD, role user-data, data_bearing) is attached and mounted at /mnt/felhom-usb. To confirm whether this gates any HDD-app deploy UI. Severity TBD.
|
||||
|
||||
### F3 — Hungarian catalog text double-UTF-8-encoded in API JSON — LOW (verify)
|
||||
- `GET /api/stacks` returns descriptions like `SzemĂ©lyes pĂ©nzĂĽgyek` (mojibake for "Személyes pénzügyek"). Looks like double-encoding. Need to confirm whether the rendered HTML UI is affected or only the JSON API. Severity TBD.
|
||||
|
||||
### F5 — uptime-kuma: broken catalog healthcheck → permanently unhealthy → **total 404 outage via Traefik** — HIGH
|
||||
- **Area:** §2 deploy / §2b health detection / §3 routing — a cascade.
|
||||
- **Action:** deployed uptime-kuma (`POST /api/stacks/uptime-kuma/deploy`), observed state, then traced the 404.
|
||||
- **Expected:** deploy → healthy → status.demo-felhom.eu serves the app (200).
|
||||
- **Actual / evidence — the cascade:**
|
||||
1. **Catalog bug:** `/opt/docker/stacks/uptime-kuma/docker-compose.yml` defines `healthcheck.test: ["CMD","node","/app/extra/healthcheck.mjs"]`, but in the `louislam/uptime-kuma:2` image that file does not exist → `docker inspect` health log: `Error: Cannot find module '/app/extra/healthcheck.mjs'`, `Health=unhealthy FailingStreak=5`.
|
||||
2. The **app process is actually fine** — `curl http://uptime-kuma:3001/` from a peer container → **302** (serving).
|
||||
3. **Traefik gates route registration on Docker health.** On the websecure (443) entrypoint, `status.demo-felhom.eu → 404`, while every healthy app (tasks/recipes/share/paste) → 200. The HTTP→HTTPS redirect on :80 returns 301 for *all* hosts (global catch-all), which masks the missing 443 router. So the unhealthy container's TLS router is never published.
|
||||
4. **Net result:** uptime-kuma is **completely unreachable at its URL (404)** for the customer, despite the app running — purely because of a wrong healthcheck path in the catalog.
|
||||
- **Verdict:** broken. **Severity: HIGH.** Two issues to file: (a) catalog healthcheck wrong for uptime-kuma:2; (b) **design risk** — *any* app with a broken/too-slow healthcheck doesn't just show "unhealthy", it becomes a hard 404 outage. The controller's deploy returns success and the dashboard shows "deployed (unhealthy)", giving no hint that the URL is dead. Consider surfacing "route not published because unhealthy" to the operator.
|
||||
- **Good part:** the controller **did correctly detect and surface `unhealthy`** (`GET /api/stacks/uptime-kuma` → `state=unhealthy`) — health detection itself works.
|
||||
|
||||
### F6 — Deploy POST returns `"deployed"` optimistically, before compose completes / before health is known — MEDIUM (API contract)
|
||||
- **Action:** timed `POST /api/stacks/<app>/deploy` vs controller logs.
|
||||
- **Evidence:** vikunja POST returned ~0s but log shows compose took 3.4s; uptime-kuma POST returned ~0s but compose pull took **31.7s** and the app ended **unhealthy**. The response `{"ok":true,"message":"Stack <app> deployed"}` is sent before the container is up and regardless of eventual health.
|
||||
- **Impact:** This is the documented in-memory `Deployed=true`-before-`compose up` pattern (avoids the card flipping back mid-pull), and the UI compensates by polling `GET /api/stacks/<app>`. But the **API message "deployed" is misleading** — an API consumer (or a script) that trusts the POST result will think a broken/unhealthy app succeeded (see F5). Verdict: clunky/misleading message; not a data-integrity bug. **Severity: MEDIUM.**
|
||||
|
||||
### F7 — API `state` lags Docker health by ~10s after deploy — LOW
|
||||
- mealie's container reported `(healthy)` in `docker ps` ~12s before `GET /api/stacks/mealie` flipped from `starting` to `running`. Cosmetic polling lag; transient. **Severity: LOW.**
|
||||
|
||||
### F8 — Infrastructure secrets stored in plaintext on the guest rootfs — LOW/INFO
|
||||
- The cloudflared infra stack's compose holds `TUNNEL_TOKEN=<redacted>` in plaintext; `/etc/felhom-controller/controller.yaml` holds `hub.api_key`, `infrastructure.cf_api_token`, and `infrastructure.cf_tunnel_token` in **plaintext** (not `enc:`-wrapped like app.yaml secrets). These live on the guest rootfs (in PBS snapshots) and the controller needs them at runtime, so it may be acceptable, but it diverges from the "secrets encrypted at rest" posture applied to app secrets. All redacted here. **Severity: LOW/INFO.**
|
||||
|
||||
### F19 — §11 misc (monitoring, settings, sync, geo, assets, hub reporting) — mostly PASS — INFO
|
||||
- **Per-container metrics: accurate.** `/api/metrics/containers/summary` reports romm `mem_usage_mb=278.1, limit 512`; `docker stats romm` = `277.8MiB/512MiB` — matches, and uses the correct **per-container cgroup limits**. `/api/metrics/containers/{name}` returns a CPU/memory time-series (works). **Contrast with F1:** only the *system/guest* memory metric is wrong (host 16GB), not the container metrics.
|
||||
- **Settings page** `/settings` → 200. **Sync** (`POST /api/sync`, "Sablonok frissítése") → works ("Sablonok naprakészek — nincs változás"). **Geo** `/api/geo/status` → reachable (`allowed:[HU], cf_configured:true, enabled:false`). **Asset sync** `/api/assets/status` → `not_configured` (assets not synced from hub on this demo — note, not necessarily a bug).
|
||||
- **Hub reporting: works.** Logs: `Hub connectivity: https://hub.felhom.eu reachable (HTTP 200)`, `Hub report pushed successfully (7658 bytes)`, periodic `hub-report every 15m`, startup report sent. **Severity: INFO (PASS).**
|
||||
|
||||
### F9 — felhom-usb HDD is NOT passed through to guest 9201 → HDD apps undeployable; disk APIs misrepresent it as available — CRITICAL
|
||||
- **Area:** §2 HDD deploy / §7b data migration / §11 storage.
|
||||
- **Action:** `POST /api/stacks/romm/deploy` with `HDD_PATH=/mnt/felhom-usb` (the path `/api/disks` reports). Then traced the guest/container mount topology.
|
||||
- **Expected:** an HDD-requiring app (romm) deploys onto the user-data HDD.
|
||||
- **Actual:**
|
||||
- Deploy **rejected** at field validation: `{"ok":false,"error":"path \"/mnt/felhom-usb\" does not exist for field \"Adattárolási útvonal\""}`.
|
||||
- **The HDD is not in the guest.** `/etc/pve/lxc/9201.conf` has only `mp0` (docker-data → /var/lib/docker) and `mp9` (bootstrap, ro). **No mountpoint for felhom-usb.** `findmnt` in the guest shows `/mnt` is just `pve-vm-9201-disk-0[/mnt]` — a subdir of the **32G rootfs**.
|
||||
- The controller's only "drive" is `/mnt/sys_drive/felhom-data`, which `df` inside the controller confirms is on `/dev/mapper/pve-vm--9201--disk--0` (**the 32G OS rootfs**), not a data drive.
|
||||
- **The HDD does exist on the Proxmox host**: `findmnt /mnt/felhom-usb` → `/dev/sdb1 ext4 915.8G`, containing real data (`felhom-data`, `images`, `storage`, `Dokumentumok`, …). It is host-mounted but never bind-mounted into the LXC.
|
||||
- **Source-of-truth disconnect:** `/api/disks` reports felhom-usb `state:attached, role:user-data, data_bearing:true, mount_path:/mnt/felhom-usb`; `/api/host-metrics` reports the *same* drive as `content: "backup"`. Both are the **agent's host-side view** and say "attached/reachable", but the guest/controller genuinely cannot see it. Meanwhile `/api/system/info` correctly reports `hdd_configured:false` (the controller's local view). Three endpoints, two contradictory stories about whether the customer has usable external storage.
|
||||
- **Impact:** **All HDD-requiring apps (romm, nextcloud, immich, jellyfin, paperless-ngx, plex, navidrome, audiobookshelf, komga, calibre-web, emby, sonarr, radarr — 13 of 55) are undeployable.** A user who reads `/api/disks` and enters `/mnt/felhom-usb` gets a confusing rejection. If they instead pick the offered `sys_drive`, their "external storage" app data silently lands on the **32G OS disk** — the exact failure the OS/data-split slice was meant to prevent.
|
||||
- **Root context:** the 2026-06-13 re-provision of 9201 left "RomM/USB re-enroll" as a documented pending step (CONTEXT.md); this drive confirms it was never completed **and** that the agent's disk APIs do not reflect the gap — they advertise the HDD as usable. The honest customer-facing status should be "external drive present on host but not attached to your server".
|
||||
- **Verdict:** broken. **Severity: CRITICAL** (blocks a quarter of the catalog + risks silent data-on-OS-disk placement; API misrepresents storage availability). **This is the data-migration verdict: cannot be exercised — there is no second/HDD drive attached to the guest to migrate to or from.**
|
||||
|
||||
### F10 — Crash-loop detection works (controller reports `restarting`, not false "running") — PASS (with a brief initial window) — INFO
|
||||
- **Area:** §2b — the crash-loop-exit-0 trap.
|
||||
- **Method:** created a fixture stack `brokentest` (`alpine`, `command: sh -c 'echo booting; sleep 3; echo crashing; exit 1'`, `restart: always`) in the stacks dir, picked up on a controller restart, then deployed it via `POST /api/stacks/brokentest/deploy` and watched the state.
|
||||
- **Result:** `docker compose up -d` returned exit 0 and the deploy POST returned `{"ok":true,"message":"Stack brokentest deployed"}` (F6 again). For the first ~10s the API reported `state=running` (the container was genuinely "Up" during its initial 3s sleep). Once it began crash-looping, the controller **correctly transitioned to `state=restarting`** and held there (docker oscillating Up↔`Restarting (1)`). It never falsely reported a stable "running"/healthy for a crashing container.
|
||||
- **Verdict:** works. Caveat: there's a brief post-deploy window where a stack that will crash shows "running"; an operator watching the 3-step panel could see a momentary green before it flips to restarting. **Severity: INFO** (the lesson-#6 trap is handled).
|
||||
- **Note (deploy-page scan):** a hand-created stack dir is only picked up by `ScanStacks` at controller **startup** — `POST /api/sync` (git catalog sync) does not rescan the local stacks dir, and there is no exposed manual-rescan endpoint (see F4). Minor operational gap.
|
||||
|
||||
### F11 — "HDD-requiring" app deploys silently onto the 32G OS rootfs when no HDD is attached — MEDIUM
|
||||
- **Area:** §2 / storage placement (related to F9).
|
||||
- **Action:** with felhom-usb unavailable to the guest (F9), deployed romm with `HDD_PATH=/mnt/sys_drive/felhom-data`.
|
||||
- **Actual:** accepted; romm + romm-db (MariaDB) + romm-redis came up **healthy**, route `arcade.demo-felhom.eu` → 200. Appdata landed at `/mnt/sys_drive/felhom-data/appdata/romm`, which `df` confirms is the **32G OS rootfs (disk-0)**. No warning that an HDD-class app is being placed on the OS disk.
|
||||
- **Impact:** for a real customer with a media app (jellyfin/immich/plex), bulk media would fill the 32G rootfs and the OS-headroom guard becomes the only backstop. The deploy UI offered no "this is your OS disk, not external storage" caution. **Severity: MEDIUM.**
|
||||
- **Positive:** the multi-container DB app (MariaDB + redis + app) deployed and reached healthy cleanly — the deploy orchestration itself is solid.
|
||||
|
||||
### F12 — Lifecycle ops (start/stop/restart/update/logs/optional-config) — PASS — INFO
|
||||
- **stop** (vikunja): `state→stopped`; container removed from `docker ps -a` (i.e. "stop" performs a compose `down`, not a `stop` that leaves an Exited container — semantic note, functionally fine).
|
||||
- **start / restart**: container back Up; `tasks.demo-felhom.eu` → 200 after restart (route survived).
|
||||
- **update** (gokapi): completed in ~1s; container **not** recreated (`Up 18 minutes` preserved) because the image was unchanged — correct no-op behavior (compose pull + up with no new image).
|
||||
- **logs** (`GET /api/stacks/vikunja/logs?raw=1`): returns real container logs (migrations, http started), timestamps current.
|
||||
- **optional-config** (romm `IGDB_CLIENT_ID`): `POST /api/stacks/romm/optional-config` → persisted to app.yaml (plaintext, correct for a non-secret text field), romm restarted, and the env var was verified **inside the running container** (`IGDB_CLIENT_ID=testclientid12345`). Full round-trip works.
|
||||
- **remove guard**: `POST /api/stacks/brokentest/remove` while running → correctly **refused** (`"still running — stop it first"`). After stop, remove succeeded; the stack **directory is intentionally left** on disk (so a catalog app can be redeployed) — by design.
|
||||
- **Verdict:** all lifecycle operations work and stay consistent with ground truth. **Severity: INFO (PASS).**
|
||||
|
||||
### F13 — Backup tiers: tier-1 + PBS offsite WORK; tier-2 honest no-target; but "primary" lives on the same OS disk as the data — MIXED (3-2-1 weakened by F9)
|
||||
- **Area:** §5 backup.
|
||||
- **Tier-1 (primary recovery units + DB dumps):** `POST /api/backup/run` → success. Recovery units (`compose/` + `manifest.json`) created for **all 7 deployed apps**; romm's MariaDB dumped to `backups/primary/romm/db-dumps/romm-mariadb.sql` (44K). `db_dump count=1` is **correct** — mealie runs embedded SQLite (single container, no postgres sidecar), so it has no SQL dump (captured as a volume in its unit). **Verdict: works.**
|
||||
- **Tier-2 (off-drive secondary):** `POST /api/backup/tier2` → ran, processed 1 HDD app (romm), and **honestly refused**: log `Tier 2 for romm: no off-drive target — nincs másik fizikai meghajtó — a 2. mentéshez 2. meghajtó szükséges`; `backup-data` shows `secondary/romm/rsync exists:false`. No silent failure, no rootfs-fill. **Verdict: works (correct no-target handling).**
|
||||
- **Tier-3 (PBS offsite, whole-guest):** `POST /api/guest-backup/trigger` → created a **new** PBS snapshot. `GET /api/guest-backup/status` → `phase done, started 2026-06-13T22:28:55Z, success, size 5.99GB, mode snapshot, crash_consistent`. Verified in PBS itself (read-only `pvesm list felhom-pbs --vmid 9201`): new snapshot `felhom-pbs:backup/ct/9201/2026-06-13T22:28:56Z` listed (3 snapshots total). **Verdict: works end-to-end.**
|
||||
- **The 3-2-1 weakness (HIGH-ish, consequence of F9):** the "primary" tier (recovery units + DB dumps) is stored under `/mnt/sys_drive/felhom-data/backups`, which is the **same 32G OS rootfs (disk-0)** that holds the app data itself. So tier-1 gives **zero protection against disk-0 loss** — source and backup share a disk. With no second drive attached (F9), the only genuine off-host copy is PBS. Effective reality is **"2 copies on the same disk + 1 PBS"**, not 3-2-1. The UI/dashboard 3-2-1 framing would overstate resilience here.
|
||||
- **Minor:** `/api/backup/status` reports only `db_dump` + `running`/`enabled` — it does not surface recovery-unit or tier-2 status, so "is my backup complete?" requires multiple endpoints. The earlier guest-backup status briefly showed a **stale prior run** (20:27Z) before my new run appeared — momentary staleness, not a clock bug.
|
||||
|
||||
### F14 — CTRL-T2-1 crash-window: crash mid-image-pull leaves stack NOT-deployed and redeployable — PASS — INFO
|
||||
- **Area:** CTRL-T2-1 regression.
|
||||
- **Method:** deployed `code-server` (uncached ~1GB image) with a password, confirmed the `docker compose up -d` pull subprocess was running, then `docker kill felhom-controller` mid-pull.
|
||||
- **On-disk state during the crash (read directly from the guest fs while controller down):** `app.yaml` had `deployed: false` (only `deployed_at` was stamped at start; the `deployed:true` flag is written **only on success**).
|
||||
- **After restart:** `GET /api/stacks/code-server` → `state=not_deployed, deployed=False`; **no orphaned code-server container**.
|
||||
- **Redeploy:** allowed (no "already deployed; use update" refusal) → returned deployed → polled to `running`/healthy, `code.demo-felhom.eu` → 302, `deployed: true` persisted on success.
|
||||
- **Verdict:** the fix works — no ghost-stuck stack. **Severity: INFO (PASS).**
|
||||
- **First-attempt note:** an initial attempt without the (required) PASSWORD was rejected at validation **before any pull** — so a missing required field is caught pre-pull (good), and I had to supply the password to actually exercise the pull window.
|
||||
|
||||
### F15 — `docker kill`/`stop` does NOT auto-restart the controller; bootstrap service is oneshot/Restart=no — MEDIUM (operational expectation)
|
||||
- **Area:** §3 crash recovery / operational.
|
||||
- **Finding:** the runbook states the bootstrap service "restarts it within seconds." In reality: `felhom-controller-bootstrap.service` is `Type=oneshot, Restart=no` (it ran once and exited). The container has `--restart unless-stopped`, but Docker treats an explicit `docker kill`/`docker stop` as a **manual stop**, so `unless-stopped` does **not** restart it (observed: `RestartCount=0`, stayed `Exited (137)` indefinitely; I had to `docker start` it manually).
|
||||
- **Impact:** `unless-stopped` *will* restart the controller after a genuine unexpected death (panic/OOM/host reboot) — so real crash-recovery is covered. But there is **no watchdog** that recovers from a manual stop or a clean exit, and the bootstrap service does not re-run. An operator who `docker stop`s the controller (or a tool that does) will find it stays down. The "auto-restarts within seconds" expectation is only partially true. **Severity: MEDIUM** (recovery assumption is narrower than documented).
|
||||
|
||||
### F16 — Import/export round-trip + CTRL-001 path-traversal — PASS — INFO
|
||||
- **Export** (vikunja → `/mnt/sys_drive/felhom-data/exports/vikunja_*.fab`): estimate → start → status all clean; `.fab` is a gzip tar (`manifest.json`, `config/{.felhom.yml,app.yaml,docker-compose.yml}`, `data/volumes`); manifest readable via `POST /api/export/manifest` (app_name, volume_names, config_files, encrypted flag). **Works.**
|
||||
- **Import round-trip:** `POST /api/export/import` of the same `.fab` (over the live vikunja) succeeded; vikunja ended `running`, `tasks.demo-felhom.eu` → 200. **Works.**
|
||||
- **CTRL-001 — app_name traversal:** crafted `.fab` with `manifest.app_name = "../../../tmp/evil_appname"` → import **rejected at manifest parse**: `Manifest hiba: appexport: app_name "../../../tmp/evil_appname" must not contain a path separator`; verified **nothing written** to `/tmp`, `/opt`, `/opt/docker`. **PASS.**
|
||||
- **CTRL-001 — tar-member traversal (tar-slip):** crafted `.fab` with a valid app_name but a tar entry `../../../../tmp/evil_member` → the member was **not** written to `/tmp/evil_member` (extraction sanitized the traversal). **PASS.** Minor: that bundle left an **empty `/opt/docker/stacks/eviltar` dir** (import got partway then failed "stack not found") — cleanup debt, not a security issue (I removed it).
|
||||
- **Verdict:** export/import works; CTRL-001 is robustly enforced on both vectors with a clear error message. **Severity: INFO (PASS).**
|
||||
|
||||
### F17 — Per-app restore does NOT restore data (DB dump never imported, volume tars not captured) — CRITICAL
|
||||
- **Area:** §7 restore — the core disaster-recovery path.
|
||||
- **Method (real data-marker round-trip on romm/MariaDB):**
|
||||
1. Inserted a marker row into romm's DB: `felhom_restore_marker (1, 'MARKER-BEFORE-BACKUP-9921')`.
|
||||
2. `POST /api/backup/run` → confirmed the marker **is in the captured dump** (`grep MARKER… romm-mariadb.sql` → 1 hit).
|
||||
3. Dropped the table (simulated data loss) — confirmed gone.
|
||||
4. `POST /backup/restore` (`stack_name=romm`).
|
||||
5. **Result: the marker table did NOT return** — `ERROR 1146: Table 'romm.felhom_restore_marker' doesn't exist` after restore.
|
||||
- **What the restore actually did (from logs):** stopped romm (`compose down`, volumes kept), `secrets recovered=3/3`, "Redeploying romm from recovery unit", `compose up -d`, "Restore-from-unit completed". The DB dump was **never imported**; the app simply came back on its surviving (post-drop) volume.
|
||||
- **Code confirmation:** `backup/restore_unit.go: RestoreFromRecoveryUnit` → `stackProvider.RecreateStackFromUnit(...)` → (`main.go:953`) copies only `docker-compose.yml` + `.felhom.yml` from the unit and calls `RedeployFromEnv`. There is **no DB-import** (`pg_restore`/`mysql <`/`.sql` replay) anywhere in `internal/backup/*`, and recovery units **do not capture volume tars** (F13). So the per-app restore is **config + secrets only**.
|
||||
- **Impact:** the headline disaster-recovery feature does not recover data. A customer who loses DB content (corruption, bad migration, accidental delete) and clicks "restore" gets the app back with **whatever was already in the volume** — if the volume is intact, their loss is unchanged; if the volume is gone, the DB comes up **empty**. The carefully-captured DB dumps and recovery-unit manifests are effectively decorative for restore purposes. The only path that actually restores data is the **PBS whole-guest snapshot** (the agent's), which restores the entire LXC incl. the docker-data volume.
|
||||
- **What does work:** secret recovery (3/3 from the guest's own app.yaml), the fail-closed data-key gate (Phase 2b, proven previously), and the app coming back healthy. The orchestration is fine; the data-replay step is simply missing.
|
||||
- **Verdict:** broken. **Severity: CRITICAL** (restore advertised as data recovery does not restore data). Also: `/backup/restore` reads `snapshot_id` but **ignores it** (`RestoreFromRecoveryUnit` takes only the stack name) — there is no point-in-time selection; "restore" always means "re-create from the single latest unit."
|
||||
|
||||
### F18 — Removal + protected-stack guards — PASS — INFO
|
||||
- **Protected stacks** (traefik, cloudflared, filebrowser, felhom-controller): both `stop` and `remove` **refused server-side** with clear messages (`cannot stop protected stack X` / `stack "X" is protected and cannot be removed`); all confirmed still running. **PASS.**
|
||||
- **Full removal** (code-server): stop → remove → container gone, no orphaned volume (`docker volume ls` empty), `state=not_deployed`, route `code.demo-felhom.eu` → 404 (Traefik route withdrawn with the container). Remove response carries explicit data-handling fields (`volumes_removed`, `hdd_paths_removed`, `hdd_paths_preserved`). **PASS.**
|
||||
- **Verdict:** removal and protection guards work correctly. **Severity: INFO (PASS).**
|
||||
|
||||
### F20 — AGENT-001 wipe gate: refusals + correct-device targeting PASS; but format error-reporting + large-disk timeout are broken — MIXED (gate PASS, execution HIGH)
|
||||
- **Area:** §6 / AGENT-001.
|
||||
- **Refusal A (no confirmation):** `POST /api/disks/format {device:/dev/sdb1, fstype:ext4}` → `formatted:false, needs_confirmation:true, "customer confirmation required"`, **HTTP 409**. No mkfs. **PASS.**
|
||||
- **Refusal B (wrong durable_id):** `+confirmed:true, durable_id:byid:wwn-0xDEADBEEF…` → `formatted:false, "operator authorization required"`, **HTTP 409** (agent log: `reason=binding_mismatch`). A non-matching confirmation does not authorize. **PASS.**
|
||||
- **Gate authorization (correct durable_id):** with `confirmed:true, durable_id:byid:wwn-0x5000039ddb108568-part1` the agent gate progressed `pending_confirmation → binding_mismatch → customer_confirmed allowed=true` and targeted the **correct re-resolved device** `/dev/sdb1`. The anti-retarget durable_id resolution works. **PASS (gate logic).**
|
||||
- **BUG 1 — swallowed format error (HIGH):** the first confirmed attempt failed because `/dev/sdb1` was mounted (`mkfs.ext4 … is mounted; will not make a filesystem here!`, agent logged `level=ERROR`), yet the **API returned `ok:true`** with `formatted:false, device:"", reason:""`. The failure was not propagated — a caller/UI would read it as success while nothing happened.
|
||||
- **BUG 2 — durable_id scheme mismatch across endpoints (MEDIUM):** `/api/disks` reports this drive's durable_id as `uuid:277a2179-…`, but the format gate expects/returns `byid:wwn-0x5000039ddb108568-part1`. A customer/UI copying the id from the disk list would always be refused (`binding_mismatch`). The two APIs disagree on the durable_id scheme.
|
||||
- **BUG 3 — large-disk format times out and leaves the disk corrupt (HIGH):** after unmounting (the gate won't unmount itself), the confirmed format on the 915GB drive ran mkfs, but the **controller→agent call timed out** (`agentapi: POST /disks/format … context deadline exceeded`, HTTP 502). The cancelled request context **killed the in-progress mkfs** (agent log: `mkfs.ext4 … signal: killed`). Result: `/dev/sdb1` is left **half-formatted** — old fs/UUID/LABEL destroyed, `blkid` shows only PARTUUID, `mount` fails `bad superblock`. The disk is neither cleanly wiped nor usable. For a real customer wiping a large drive this corrupts it.
|
||||
- **Net:** AGENT-001's *security* properties (refuse without confirmation, refuse wrong durable_id, target the durable_id-resolved device) are intact and were proven on the correct device. But the *format execution path* has two real defects (silent failure on mounted device; timeout-kill leaving a corrupt fs on large disks). **Severity: gate PASS; execution HIGH.**
|
||||
- **Disk final state:** felhom-usb (/dev/sdb1) old data destroyed, left without a valid filesystem (see "Final state" section).
|
||||
|
||||
### F4 — `/api/stacks/rescan` returns "stack not found: rescan" — LOW
|
||||
- The runbook's documented rescan endpoint `GET /api/stacks/rescan` is routed as a stack name lookup → `{"ok":false,"error":"stack not found: rescan"}`. Either the route was removed/renamed or the runbook is stale. (Sync/rescan is reachable via `POST /api/sync`.) Cosmetic but documents a stale/missing endpoint.
|
||||
|
||||
---
|
||||
|
||||
## PRIORITIZED ISSUE LIST (severity-ordered, with repro)
|
||||
|
||||
1. **[CRITICAL] F17 — Per-app restore does not restore data.** Restore re-creates config + recovers secrets but never imports the captured DB dump; recovery units don't capture volume tars. **Repro:** insert a row in romm's DB → `POST /api/backup/run` (row is in `romm-mariadb.sql`) → drop the row → `POST /backup/restore stack_name=romm` → row does NOT return. Only PBS whole-guest restore recovers data.
|
||||
2. **[CRITICAL] F9 — HDD not passed through to guest; HDD apps undeployable; disk APIs misrepresent it.** **Repro:** `POST /api/stacks/romm/deploy HDD_PATH=/mnt/felhom-usb` → `path … does not exist`; `/etc/pve/lxc/9201.conf` has no mp for felhom-usb; `/api/disks` still lists it `attached/user-data`.
|
||||
3. **[HIGH] F20 (BUG1+BUG3) — Disk wipe execution.** (a) mkfs failure on a mounted device returns `ok:true` (swallowed error). (b) Large-disk format times out (`agentapi … context deadline exceeded`), the cancelled context kills mkfs → disk left corrupt (`bad superblock`). **Repro:** `POST /api/disks/format` confirmed on a 915GB drive → 502 + half-formatted device. (The gate logic itself is correct.)
|
||||
4. **[HIGH] F5 — Broken healthcheck → total 404 outage.** uptime-kuma's catalog healthcheck points to a nonexistent file → container "unhealthy" → Traefik never publishes the route → `status.demo-felhom.eu` = 404 though the app runs. **Repro:** deploy uptime-kuma; `curl https://status.demo-felhom.eu/` → 404; `docker inspect` health log shows `Cannot find module '/app/extra/healthcheck.mjs'`.
|
||||
5. **[HIGH] F1 — Memory metric reports host RAM (16GB), not the guest's 2GB cap.** `/api/system/info` → `total_mem_mb:15771`; LXC is capped at 2048. Unsafe deploy-headroom basis.
|
||||
6. **[MEDIUM] F13 — 3-2-1 weakened:** tier-1 backups live on the same OS disk as the data; no 2nd drive (consequence of F9). Only PBS is genuinely off-host.
|
||||
7. **[MEDIUM] F15 — Controller not auto-restarted after a manual `docker stop`/`kill`** (bootstrap is oneshot/Restart=no; `unless-stopped` ignores manual stops). Real crashes (panic/OOM) are covered.
|
||||
8. **[MEDIUM] F20 BUG2 — durable_id scheme mismatch:** `/api/disks` uses `uuid:…`, the format gate uses `byid:wwn:…`; copying the disk-list id into a format confirmation is always refused.
|
||||
9. **[MEDIUM] F6 — Deploy POST returns `"deployed"` before compose completes / regardless of health** (misleading for API/script consumers; UI compensates by polling).
|
||||
10. **[LOW] F17b — `/backup/restore` ignores `snapshot_id`** (no point-in-time selection).
|
||||
11. **[LOW] F2 — `hdd_configured:false`** vs `/api/disks` listing the HDD (two sources disagree; F9 root).
|
||||
12. **[LOW] F8 — Infra secrets in plaintext** (cloudflared TUNNEL_TOKEN; controller.yaml cf/hub tokens).
|
||||
13. **[LOW] F7 — API state lags Docker health ~10s post-deploy.** F4 — stale rescan endpoint. F3 — JSON API double-encodes Hungarian text (HTML UI fine).
|
||||
|
||||
---
|
||||
|
||||
## UX / "this felt wrong" observations (for the operator's UI pass tomorrow)
|
||||
|
||||
- **uptime-kuma will show as deployed-but-unhealthy with a dead URL (404).** Check whether the dashboard gives any hint that the *route* is unpublished — currently it just looks "unhealthy". This is the most confusing customer-facing case. (F5)
|
||||
- **Storage page vs reality:** the dashboard/disk views advertise the felhom-usb HDD as available, but no app can actually use it. Watch how the deploy "drive picker" presents storage — does it offer felhom-usb (which would fail) and/or the OS-disk "sys_drive" (which silently fills the 32G OS disk)? (F9, F11)
|
||||
- **Memory bar likely shows ~16GB.** The deploy page memory bar / headroom is based on host RAM, not the 2GB guest cap — looks like tons of free RAM. (F1)
|
||||
- **Restore UI implies data recovery.** Confirm what the restore screen promises; in practice it only restores config/secrets, not data. A customer could believe their data is safe via per-app restore when it isn't. (F17)
|
||||
- **Format/wipe result:** an attempted format can report success (`ok:true`) when nothing happened, or spin/time out on a large disk. Watch the wipe UI's success/failure messaging and progress on a big drive. (F20)
|
||||
- **Deploy 3-step panel** may briefly show "running"/green before a doomed stack flips to restarting; and the per-card "deployed" flips on before health is known. (F6, F10)
|
||||
- Hungarian text in the **JSON API** is mojibake (`SzemĂ©lyes`), though the rendered HTML is correct — only relevant if any UI reads the JSON directly. (F3)
|
||||
|
||||
---
|
||||
|
||||
## What I could NOT test (and why)
|
||||
|
||||
- **App data migration SSD↔HDD** — no HDD is attached to the guest (F9); there is no second drive to migrate to/from. The feature is effectively untestable in the current provisioning state; this *is* the answer.
|
||||
- **Full readable-data restore e2e for a large app** (e.g. AdventureLog) — independently moot given F17 (restore doesn't replay data), and historically blocked by the 8/32G rootfs.
|
||||
- **HDD-app real behavior** (RomM with an imported library, media apps) — undeployable to a real HDD (F9); romm was exercised on the OS-disk fallback only.
|
||||
- **Cloudflare geo-restriction enforcement live** — `enabled:false` on the demo; only confirmed the geo subsystem is reachable/configured (HU allowlist), not active WAF behavior. (F19)
|
||||
- **Asset sync from hub** — `not_configured` on this demo. (F19)
|
||||
- **Tier-2 off-drive backup actually copying** — no off-disk target exists (F9); only the honest no-target path was exercised. (F13)
|
||||
- **Genuine unexpected-crash auto-restart** (panic/OOM) — could not induce a real panic; confirmed `docker kill` does NOT trigger restart and reasoned that `unless-stopped` covers genuine deaths. (F15)
|
||||
|
||||
---
|
||||
|
||||
## FINAL STATE OF THE DEMO
|
||||
|
||||
**Usable as-is for the operator's UI testing — does NOT need a Proxmox redeploy** (but the HDD passthrough must be re-established for full functionality).
|
||||
|
||||
- Controller `gitea.dooplex.hu/admin/felhom-controller:0.60.0` **Up (healthy)**; agent **0.30.0 active**; `GET /api/health` ok.
|
||||
- **Deployed customer apps (all healthy except uptime-kuma):** actualbudget, gokapi, mealie, privatebin, romm (+romm-db, romm-redis), vikunja. **uptime-kuma is deployed but `unhealthy` with a 404 URL** — left in place intentionally as a live example of F5 for the operator to inspect.
|
||||
- Protected infra healthy: traefik, cloudflared, filebrowser.
|
||||
- Headroom: rootfs 32G (29G free), docker-data 252G (234G free), RAM 2048M (~1187M available).
|
||||
- **felhom-usb HDD:** its old data was destroyed by the authorized AGENT-001 wipe test; I reformatted it host-side to a clean empty ext4 (new UUID `ce9d1d81-…`, LABEL `hdd_1`) and remounted it at `/mnt/felhom-usb` on the host. **It remains NOT attached to guest 9201** (the F9 gap is pre-existing; I did not change the LXC config). To make HDD apps and data-migration work, an `mp` for felhom-usb must be added to the guest.
|
||||
- Test artifacts cleaned up: brokentest fixture (container + dir) removed; code-server removed; malicious `.fab` bundles and the empty `eviltar` dir removed; romm's test marker table is gone (expected — it demonstrated F17).
|
||||
- romm's app data lives on the **OS rootfs** (`/mnt/sys_drive/felhom-data`, F11) — fine for a demo, not where it should be in production.
|
||||
|
||||
**Bottom line:** the control plane, deploy/lifecycle, routing, health detection, import/export + path-traversal defense, CTRL-T2-1, backup-to-PBS, and the wipe *gate* are solid. The two things a customer would most need in a crisis — **restoring their data** and **using their external drive** — do not work in the current build/provisioning (F17, F9), and the **disk-wipe execution** can corrupt a large drive (F20). Those three should be the focus before this is customer-ready.
|
||||
@@ -1,472 +0,0 @@
|
||||
# LIVE-DRIVE-FIXSPEC — triage of the 2026-06-14 live-drive findings
|
||||
|
||||
- **Date:** 2026-06-14
|
||||
- **Branch:** `audit/2026-06-14-live-drive` (companion to `LIVE-DRIVE-FINDINGS-2026-06-14.md`)
|
||||
- **Scope:** re-diagnose F1–F20 against **current source** at file:line, spec a fix for each worth fixing, then a sequenced plan. **Diagnose + spec only — no code changes, no deploys, no `main` commits.**
|
||||
- **Method:** every finding's stated cause was treated as a claim and re-rooted in code. Two report causes were found **wrong** (F20-BUG1 location; F17 volume-capture sub-claim) and one is likely a measurement artifact (F3). Runtime-only facts reference the findings report's captured evidence.
|
||||
|
||||
## Progress log
|
||||
|
||||
- **t0** — Confirmed branch + repos (`felhom-controller`, `felhom-agent`, `app-catalog-felhom.eu`, `felhom.eu`). Launched parallel code investigations.
|
||||
- **t1** — F17 + F20(agent) re-diagnosed: F17 DB-replay claim TRUE but volume-capture sub-claim mis-attributed; **F20-BUG1 relocated** (agent is correct; bug is in the controller's agentapi client).
|
||||
- **t2** — F9 + F2 re-diagnosed: provisioning never binds user-data drives; "misreporting" is a missing guest-attached field; F2 is the expected symptom.
|
||||
- **t3** — F1/F11/F13 + F5 catalog + controller-misc (F3/F4/F6/F7/F8/F15 + BUG1 controller mapping) re-diagnosed. F1 confirmed to defeat the deploy memory guard (not display-only). F5 catalog survey found additional at-risk healthchecks.
|
||||
- **t4** — Wrote per-finding specs + sequenced batches. (this doc)
|
||||
|
||||
---
|
||||
|
||||
## BATCH 1 (+F17) IMPLEMENTATION STATUS — shipped v0.61.0 (2026-06-14)
|
||||
|
||||
All controller-side Batch-1 items + F17 implemented, tested, built, deployed to demo guest 9201, and the
|
||||
key fixes live-verified. (F9, F20-BUG2, F20-BUG3 remain for the SUPERVISED agent/golden session.)
|
||||
|
||||
| Finding | Status | Commit | Live verification |
|
||||
|---|---|---|---|
|
||||
| F1 | **FIXED** | `0550b31`+`4989513` | `/api/system/info total_mem_mb=2048` (was 15771) |
|
||||
| F20-BUG1 | **FIXED** | `2cf3fad` | unit test (502→non-nil err) |
|
||||
| F4 | **FIXED** | `56fe574` | `GET /api/stacks/rescan` → 405 |
|
||||
| F6 | **FIXED** | `56fe574` | 202 + "Telepítés elindítva" (code/UI) |
|
||||
| F7 | **FIXED** | `4938cc8` | status-refresh 10s |
|
||||
| F8 | **FIXED** | `6868489` | unit test (mode 0600 on Linux) |
|
||||
| F5 (catalog) | **FIXED** | app-catalog `main` | uptime-kuma healthy → route 302 (was 404) |
|
||||
| F5 (dashboard) | **FIXED** | `803ce50` | funcmap + template render tests |
|
||||
| F17 | **FIXED** | `0b9450e` | **marker DB round-trip PASSED live** (escape hatch cleared) |
|
||||
| F9, F20-BUG2, F20-BUG3 | DEFERRED | — | SUPERVISED next session (agent/golden) |
|
||||
|
||||
Note on F1: the FIXSPEC's "read the cgroup limit" approach proved a **no-op on the demo** — the
|
||||
controller container's own cgroup is unlimited (the 2 GB cap is on the LXC ancestor, hidden) and there is
|
||||
no lxcfs in the container. The working source is `docker info` MemTotal (the daemon runs in the LXC), with
|
||||
the deploy guard's "used" switched to the controller's committed-memory accounting (host RSS is not
|
||||
observable per-guest). cgroup-limit reading is retained for the non-nested case.
|
||||
|
||||
---
|
||||
|
||||
## RE-DIAGNOSIS SUMMARY (verdict counts)
|
||||
|
||||
| Verdict | Count | Findings |
|
||||
|---|---|---|
|
||||
| **real** (fix as specced) | 11 | F1, F5, F9, F11, F17, F20-BUG1, F20-BUG2, F20-BUG3, F6, F7, F8 |
|
||||
| **mis-attributed** (report cause wrong → re-rooted) | 3 | **F20-BUG1** (controller not agent), **F17 sub-claim** (units DO capture volumes), **F3** (not in code) |
|
||||
| **already-fine / no independent action** | 9 | F10, F12, F14, F16, F18, F19 (PASS), **F13** (consequence of F9), **F2** (expected symptom of F9), **F15** (standard Docker semantics) |
|
||||
| **needs-runtime-confirmation** | 2 | **F3** (byte-level recheck), **F17 capture-side** (are volume tars actually written by `backup/run`?) |
|
||||
| **trivial / doc-only** | 1 | F4 |
|
||||
|
||||
> Findings whose **report cause was WRONG against the code** (the most important output): **F20-BUG1** (report blamed the agent's format handler; the agent is correct — the bug is the controller's `agentapi.FormatDisk` swallowing the agent's 502). **F17** (report said "recovery units don't capture volume data"; they do — `restoreDockerVolumes` + `VolumeDumps` exist; the real gap is the `.sql` dump is never replayed). **F3** (report said the JSON API double-encodes; no transcoding exists in the code path — likely a terminal/`curl` display artifact).
|
||||
|
||||
---
|
||||
|
||||
## CRITICAL / HIGH — full specs
|
||||
|
||||
### F17 — Per-app restore re-creates config + restores Docker volumes but never replays the `.sql` DB dump
|
||||
```
|
||||
Report claim: "per-app restore never imports the DB dump; recovery units don't capture volume data"
|
||||
Re-diagnosis: real (DB-replay gap) + mis-attributed (volume-capture sub-claim is FALSE)
|
||||
True cause: POST /backup/restore → web/handlers.go:785 RestoreFromRecoveryUnit →
|
||||
backup/restore_unit.go:125-131 does StopStack + restoreDockerVolumes(:128) +
|
||||
RecreateStackFromUnit(:131). RecreateStackFromUnit (cmd/controller/main.go:953)
|
||||
copies docker-compose.yml/.felhom.yml + RedeployFromEnv (compose up -d). NO .sql
|
||||
replay anywhere in internal/backup/*. The captured romm-mariadb.sql is enumerated
|
||||
into the unit manifest (recovery_unit.go:105) but never piped into the DB.
|
||||
The reusable importer EXISTS but only in the .fab path: appexport/restore.go —
|
||||
importDBDump(:971), findDBServiceInCompose(:816), waitForDB(:928), findDumpFile(:795),
|
||||
restoreDatabase(:722) — all UNEXPORTED, package appexport.
|
||||
Correction to the report: RestoreFromRecoveryUnit DOES restore Docker named-volume
|
||||
tars (restore.go:84-137) and capture DOES populate VolumeDumps (recovery_unit.go:44).
|
||||
So "units don't capture volumes" is wrong; the romm row was lost because (a) the .sql
|
||||
is never replayed AND (b) romm's MariaDB data dir is under the appdata bind namespace
|
||||
(live dir, not a named-volume tar), so the volume-restore path doesn't cover it either.
|
||||
Severity: CRITICAL (data-loss on the headline disaster-recovery feature; matches report)
|
||||
Repo / layer: controller — internal/backup (+ reuse internal/appexport DB-import logic)
|
||||
Fix approach: 1) Extract the four free funcs (importDBDump/findDBServiceInCompose/waitForDB/
|
||||
findDumpFile) into a neutral pkg (e.g. internal/dbrestore) and have BOTH appexport
|
||||
and backup call them (no duplication). 2) In RestoreFromRecoveryUnit, after
|
||||
RecreateStackFromUnit + DB service healthy, for each <unit>/db-dumps/*.sql replay it
|
||||
via the shared importer (bring DB svc up, waitForDB, importDBDump, then continue the
|
||||
stack). 3) Make volume-restore failure NOT silently non-fatal: restore.go:64 currently
|
||||
logs WARN + continues and RestoreApp returns nil regardless — surface a partial-restore
|
||||
error/flag to the caller so the UI can't claim success on a silent data-restore failure.
|
||||
Effort: L (cross-package extraction + restore-flow change + tests)
|
||||
Risk: unattended-safe to WRITE/test (no live deploy); behaviour change is destructive-restore,
|
||||
so validate against a scratch app before shipping.
|
||||
Depends on / blocks: independent. Co-edits restore_unit.go/restore.go (see F-collision note: none in B1).
|
||||
Proof of fix: unit test: capture marker row → drop → RestoreFromRecoveryUnit → row present.
|
||||
Re-run the live romm marker repro from the findings report.
|
||||
Open question: For an app whose DB lives in a named volume AND has a .sql dump, replaying both could
|
||||
double-apply — decide precedence (prefer .sql replay for logical consistency, or skip
|
||||
.sql when the DB's named volume tar was restored). Needs a one-line policy decision.
|
||||
```
|
||||
|
||||
### F9 — felhom-usb HDD never bound into the guest; provisioning omits user-data binds; disk APIs lack a "guest-attached" signal
|
||||
```
|
||||
Report claim: "HDD not passed to guest; HDD apps undeployable; disk APIs misrepresent it as available"
|
||||
Re-diagnosis: real (two layers) — provisioning gap is primary; "misreporting" is really a MISSING field
|
||||
True cause: LAYER A (provisioning): agent reconcile/bringup.go runBringUp (:158-294) writes only
|
||||
mp0 (docker-data, :236-250) + spec.Mounts as fresh PVE volume specs (:348-356) + mp9
|
||||
(bootstrap). It NEVER adds a host-path bind for an external user-data drive. That bind
|
||||
lives only in a separate, explicit enroll step: localapi/disks.go handleDiskGuestAttach
|
||||
(:238-287) → guestbind.go AttachBind (:51-71, `pct set <vmid> -mpN <host>/felhom-data`)
|
||||
→ requires guest reboot (:77-84). It was never re-run after 9201 was re-provisioned.
|
||||
A self-heal watchdog exists (disks.go:535-559) but only reconciles drives previously
|
||||
ENROLLED (intent recorded) — a never-enrolled drive is invisible to it.
|
||||
LAYER B (reporting): /disks state/reachable (storage/observe.go:290-299) and role
|
||||
(storage/role.go:105-126) faithfully describe HOST presence/topology; /host-metrics
|
||||
(host_metrics.go:28-49) emits the raw PVE content string. There is NO field anywhere
|
||||
that asserts "bound into guest N", so host-presence reads as availability. The role
|
||||
(user-data) vs content (backup) "disagreement" is two orthogonal attributes, not a bug.
|
||||
Severity: CRITICAL (blocks 13/55 apps; silent OS-disk placement; unblocks F13 + data-migration)
|
||||
Repo / layer: agent — internal/reconcile (provision) + internal/localapi (enroll, reporting);
|
||||
+ golden/bootstrap; + small controller/agent reporting field
|
||||
Fix approach: A) Make user-data drive attachment part of (re)provisioning OR auto-re-enroll on
|
||||
bring-up: record the drive's durable-id intent at first enroll (already done) and have
|
||||
runBringUp re-assert known user-data binds for the guest after recreating it (reuse
|
||||
AttachBind + the intent store the watchdog already reads), so destroy+recreate restores
|
||||
the mp. B) Add a `guest_attached bool` (per-guest) to the DiskInfo/storage-target the
|
||||
agent returns, set by checking the guest LXC config for an mp whose source is the
|
||||
drive's mount_path; surface it in the controller so /api/disks and the UI distinguish
|
||||
"present on host" from "usable by your server". This also fixes the F2 disagreement
|
||||
presentation. (Operationally for THIS demo: run guest-attach + reboot for felhom-usb.)
|
||||
Effort: L (provisioning change + new reporting field across agent+controller)
|
||||
Risk: SUPERVISED — agent/provisioning/golden; touches pct/LXC config on a live host.
|
||||
Depends on / blocks: blocks F13 (3-2-1 needs a real 2nd drive), F11 (real HDD class), data-migration.
|
||||
Proof of fix: re-provision a scratch guest with an enrolled user-data drive → mp re-appears in
|
||||
<vmid>.conf and the drive is writable in-guest; /api/disks shows guest_attached=true;
|
||||
an HDD app deploys onto it.
|
||||
Open question: Should bring-up auto-re-enroll ALL previously-enrolled drives unattended, or require an
|
||||
operator confirm (a bind is a data-path change)? Recommend auto-re-assert for drives
|
||||
with a recorded intent + durable-id match; flag mismatches. Needs operator sign-off.
|
||||
```
|
||||
|
||||
### F20-BUG1 — Controller's `agentapi.FormatDisk` swallows the agent's error and returns `ok:true` (zero-value)
|
||||
```
|
||||
Report claim: "agent format handler maps mkfs failure to ok:true (swallowed error)"
|
||||
Re-diagnosis: real, but MIS-ATTRIBUTED — the AGENT is correct; the bug is in the CONTROLLER client
|
||||
True cause: Agent handleDiskFormat returns ok:false / HTTP 502 on mkfs failure (localapi/disks.go
|
||||
:437-440, :476-480 → writeErr → {"ok":false} 502) — correct. The controller's
|
||||
agentapi/client.go FormatDisk (:372-398) calls postWithStatus, which by design does
|
||||
NOT surface an ok:false business error (comment :402). On the agent's 502, env.Data is
|
||||
JSON `null` → json.Unmarshal leaves a zero-value FormatResult; the 403/NeedsConfirmation
|
||||
guards (:389,:393) don't match a 502, so it falls through to `return out, nil` (:397).
|
||||
web/agent_disk_handlers.go:191-207 then sees err==nil and writes
|
||||
{"data":{...zeros...},"ok":true} HTTP 200 — exactly the observed response.
|
||||
Severity: HIGH (a destructive op silently reports success on failure)
|
||||
Repo / layer: controller — internal/agentapi/client.go (FormatDisk / postWithStatus)
|
||||
Fix approach: Have postWithStatus also return the envelope's error string + ok flag; in FormatDisk,
|
||||
after the NeedsConfirmation/403 handling, add: if status >= 300 (or !env.OK) and not a
|
||||
confirmation case → return out, fmt.Errorf("agentapi: format HTTP %d: %s", status, msg).
|
||||
The existing web handler `if err != nil → 502` branch then surfaces it correctly.
|
||||
Effort: S
|
||||
Risk: unattended-safe (controller-only error propagation)
|
||||
Depends on / blocks: co-edits client.go FormatDisk with F20-BUG3 (async rework) — see collision note.
|
||||
Proof of fix: unit test: stub agent 502 → FormatDisk returns non-nil err; handler returns ok:false.
|
||||
Open question: none.
|
||||
```
|
||||
|
||||
### F20-BUG3 — Synchronous format; controller's 15s client timeout cancels the request context → SIGKILLs in-flight mkfs → corrupt disk
|
||||
```
|
||||
Report claim: "large-disk format times out → cancelled context kills mkfs → corrupt disk"
|
||||
Re-diagnosis: real (confirmed end-to-end)
|
||||
True cause: agentapi/client.go:87 sets http.Client{Timeout: 15s} for ALL agent calls incl.
|
||||
/disks/format. Agent runs mkfs under the REQUEST context: handleDiskFormat passes
|
||||
r.Context() (disks.go:437,:476) → hostops.go:346 → proxmox/privileged.go:57/59
|
||||
exec.CommandContext(ctx,...). When the 15s client timeout fires, Go cancels the request
|
||||
→ r.Context() done → CommandContext sends SIGKILL to mkfs mid-write → corrupt fs
|
||||
(matches agent log "mkfs.ext4 ... signal: killed").
|
||||
Severity: HIGH (destructive: corrupts a large drive)
|
||||
Repo / layer: agent — internal/localapi (format handler) + internal/storage; controller — agentapi
|
||||
Fix approach: NOT "raise the timeout". Make format a detached job + status poll, modelled on the
|
||||
agent's existing patterns: handleGuestReboot (disks.go:298-326) already runs a long op
|
||||
in a goroutine off s.baseCtx and returns 202; the backup job (StartBackup/BackupStatus,
|
||||
client.go:106-207, phases idle|running|...|done) is the status-poll template. Run mkfs
|
||||
under s.baseCtx (NOT r.Context()) so a dropped HTTP request can't kill it; add
|
||||
/disks/format → {job_id} + /disks/format/status; controller polls. The async worker
|
||||
MUST re-resolve the durable-id at execution time (preserve AGENT-001 anti-retarget,
|
||||
wipe_reresolve.go).
|
||||
Effort: L
|
||||
Risk: SUPERVISED — agent destructive path; design + live validation on a scratch device.
|
||||
Depends on / blocks: co-edits client.go FormatDisk with F20-BUG1.
|
||||
Proof of fix: format a large scratch device; drop the HTTP client mid-op → mkfs continues to
|
||||
completion; status endpoint reports done; device mounts clean.
|
||||
Open question: Status persistence across an agent restart (like BackupRecord)? Recommend yes, reuse
|
||||
the backup record pattern. Confirm with operator whether a format must survive agent
|
||||
restart.
|
||||
```
|
||||
|
||||
### F5 — uptime-kuma catalog healthcheck points at a nonexistent file → permanent unhealthy → Traefik withholds route → 404
|
||||
```
|
||||
Report claim: "broken catalog healthcheck → unhealthy → Traefik 404; design: unhealthy = total outage"
|
||||
Re-diagnosis: real (data fix) + a broader catalog risk + an unaddressed design question
|
||||
True cause: app-catalog-felhom.eu/templates/uptime-kuma/docker-compose.yml:24-29 overrides
|
||||
healthcheck with `["CMD","node","/app/extra/healthcheck.mjs"]`; image louislam/
|
||||
uptime-kuma:2 ships its healthcheck as a COMPILED Go binary at /app/extra/healthcheck
|
||||
(no .mjs) and already defines a working built-in HEALTHCHECK. The override fails →
|
||||
permanent unhealthy. Survey found MANY other apps using wget/curl healthchecks on
|
||||
non-Alpine bases (claper, calcom, rallly, wanderer, wishlist, plant-it, papra, gokapi,
|
||||
termix; kimai, komga, crafty-controller) where the tool may be absent — same failure
|
||||
class, unconfirmed. Several migration-heavy apps use start_period:30s (calcom, outline,
|
||||
docmost, rallly, tandoor, nextcloud, wger) → transient 404 right after deploy.
|
||||
Severity: HIGH (one confirmed total-outage; broader latent set)
|
||||
Repo / layer: app-catalog-felhom.eu (data) + DESIGN question spanning Traefik/controller dashboard
|
||||
Fix approach: (data) Delete the uptime-kuma compose healthcheck block (lines 24-29) → inherit the
|
||||
image's built-in; or set test:["CMD","extra/healthcheck"]. Then audit the wget/curl
|
||||
healthchecks: for each, `docker run --rm <image> which wget||which curl`; replace
|
||||
absent-tool checks with inline `node -e`/`python3` socket checks (mealie's pattern is
|
||||
the precedent already in-catalog). Bump start_period to 60-120s for migration apps.
|
||||
Effort: S (uptime-kuma alone) / M (full catalog audit)
|
||||
Risk: unattended-safe (catalog data; auto-syncs to controllers within 15m).
|
||||
Depends on / blocks: independent.
|
||||
Proof of fix: redeploy uptime-kuma → healthy within start_period → status.<domain> → 200.
|
||||
Open question: DESIGN (needs operator decision, do NOT silently pick): an unhealthy container makes
|
||||
Traefik withhold the route → hard 404, indistinguishable from "not deployed". Options:
|
||||
(a) leave as-is (fail-safe: don't route to unhealthy); (b) publish the route even when
|
||||
unhealthy so the app is reachable while degraded; (c) keep gating but surface
|
||||
"route unpublished because unhealthy" distinctly in the dashboard so it's not a silent
|
||||
404. Recommend (c). Decision required.
|
||||
```
|
||||
|
||||
### F1 — Memory metric reads host `/proc/meminfo` (16GB), not the LXC 2GB cgroup cap — and it defeats the deploy memory guard
|
||||
```
|
||||
Report claim: "system memory reports host RAM not the 2GB guest cap; unsafe headroom"
|
||||
Re-diagnosis: real — AND confirmed it is NOT display-only: it disables the deploy memory hard-block
|
||||
True cause: system/info_linux.go readMemInfo (:80-113) parses /proc/meminfo MemTotal/MemAvailable
|
||||
directly. No cgroup awareness anywhere (repo-wide: zero matches for cgroup/memory.max/
|
||||
limit_in_bytes/Sysinfo/lxcfs). The container has no lxcfs and unbounded cgroup
|
||||
memory.max → reads host 16GB. Consumed by the deploy gate: stacks/deploy.go:162
|
||||
system.GetMemoryMB() → hard block at :173 (usedMB+newReqMB > usableMB) and overcommit
|
||||
warning at :188. With totalMB=15771 the hard block effectively never fires → the
|
||||
controller will deploy apps the 2GB LXC cannot run. Memory bar UI uses the same value.
|
||||
Severity: HIGH (defeats the deploy OOM guard; not merely cosmetic)
|
||||
Repo / layer: controller — internal/system/info_linux.go
|
||||
Fix approach: Add a cgroup-limit read used as min(cgroupLimit, meminfoTotal): cgroup v2
|
||||
/sys/fs/cgroup/memory.max ("max" → fall back), v1 /sys/fs/cgroup/memory/
|
||||
memory.limit_in_bytes (ignore the ~unbounded sentinel). For used/available use v2
|
||||
memory.current / v1 memory.usage_in_bytes. Wire into readMemInfo so TotalMemMB/
|
||||
AvailMemMB/UsedMemMB/MemPercent reflect the cap; all accessors (GetMemoryMB,
|
||||
GetTotalMemoryMB, GetInfo) become correct with no signature change.
|
||||
Effort: S
|
||||
Risk: unattended-safe (read-only metric change in the controller).
|
||||
Depends on / blocks: independent.
|
||||
Proof of fix: unit test with a fake cgroup file → reports the limit; on the 2GB guest /api/system/info
|
||||
shows ~2048; deploy hard-block triggers when committed mem approaches 2GB.
|
||||
Open question: none.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## MEDIUM — specs
|
||||
|
||||
### F11 — `needs_hdd` app deploys onto the OS rootfs with no drive-class guard
|
||||
```
|
||||
Report claim: "HDD-required app silently lands on the 32G OS disk"
|
||||
Re-diagnosis: real
|
||||
True cause: Path validation is existence-only: stacks/deploy.go:278-283 (os.Stat of the value).
|
||||
isValidDrivePath (web/handler_export.go:356-364) only checks string-equality against a
|
||||
registered StoragePath. StoragePath has no drive-class attribute; the deploy dropdown
|
||||
(web/handlers.go:322-333 from GetSchedulableStoragePaths) has no class filter. NeedsHDD
|
||||
(stacks/metadata.go:58) is a UI badge only, never cross-checked against the chosen
|
||||
path's physical drive. sys_drive is the SSD fallback namespace
|
||||
(config.go:265 /mnt/sys_drive; appbackup/paths.go:24-29 + backup.go:105-107).
|
||||
Severity: MEDIUM (data-placement footgun; on a real customer it fills the OS disk)
|
||||
Repo / layer: controller — internal/stacks (deploy) + internal/web (dropdown) + internal/settings
|
||||
Fix approach: Classify each StoragePath by backing device (compare Stat_t.Dev against the rootfs
|
||||
device; system.SamePhysicalDevice already exists for Tier-2, and agentapi exposes
|
||||
durable-id/device info). In DeployStack after the path check (deploy.go:283): if
|
||||
meta.Resources.NeedsHDD and the chosen path resolves to the OS/rootfs device → return
|
||||
a deployWarning (the function already has that channel, :160/:191) or refuse. Annotate
|
||||
the dropdown likewise.
|
||||
Effort: M
|
||||
Risk: unattended-safe (controller-only).
|
||||
Depends on / blocks: best AFTER F9 (so a real HDD class exists to steer toward); works standalone as a
|
||||
warning even now.
|
||||
Proof of fix: deploy a needs_hdd app with only sys_drive available → warning/refusal surfaced.
|
||||
Open question: warn-and-allow vs hard-refuse when no real HDD exists? Recommend warn-and-allow (demo/
|
||||
single-SSD nodes are legitimate). Minor product call.
|
||||
```
|
||||
|
||||
### F20-BUG2 — durable_id scheme mismatch: `/disks` emits `uuid:…`, the wipe gate expects `byid:`/`byuuid:…`
|
||||
```
|
||||
Report claim: "durable_id scheme mismatch across endpoints; customer copying the disk-list id is refused"
|
||||
Re-diagnosis: real
|
||||
True cause: List path: storage/durableid.go:54-63 emits `uuid:`+fs-uuid for usb/local-dir
|
||||
(→ DiskInfo.DurableID in /disks). Gate path: storage/durable_device.go:34-51
|
||||
DeviceDurableID returns `byid:`+wwn (:43) or `byuuid:`+uuid (:47); ResolveDurableDevice
|
||||
(:55-74, the AGENT-001 anti-retarget resolver) only accepts byid:/byuuid: and refuses a
|
||||
bare/uuid: scheme. Gate compares the customer's ConfirmDurableID against DeviceDurableID
|
||||
(localapi/disks.go:450,454-457) → binding_mismatch. Three prefixes (uuid: vs byid: vs
|
||||
byuuid:) from two functions.
|
||||
Severity: MEDIUM (usability: the documented confirm flow can't succeed with the advertised id)
|
||||
Repo / layer: agent — internal/storage (durableid.go + durable_device.go); reflected in controller
|
||||
Fix approach: Make the disk-LIST advertise the SAME durable-id the gate will accept (the byid:/byuuid:
|
||||
scheme from DeviceDurableID), or include both an `id` (display) and a `wipe_durable_id`
|
||||
(gate-accepted) field so the UI/customer always confirm with the gate's scheme. Do NOT
|
||||
relax the gate to accept uuid: (that weakens AGENT-001 anti-retarget). Single source of
|
||||
truth: have handleDisks call DeviceDurableID for the wipe id.
|
||||
Effort: M
|
||||
Risk: SUPERVISED — touches the wipe binding semantics; validate against AGENT-001 invariants.
|
||||
Depends on / blocks: same files as F20-BUG3 region (agent storage/localapi) — sequence with Batch 3.
|
||||
Proof of fix: read /disks id → POST /disks/format confirmed with it → gate authorizes (no
|
||||
binding_mismatch) on a scratch device.
|
||||
Open question: expose one canonical id or a display+wipe pair? Recommend the pair (UI clarity).
|
||||
```
|
||||
|
||||
### F8 — Infra secrets in `controller.yaml` are plaintext at `0644`
|
||||
```
|
||||
Report claim: "cf_api_token / cf_tunnel_token / hub api_key plaintext (not enc: like app secrets)"
|
||||
Re-diagnosis: real, but largely by-design for runtime config; the concrete weakness is file perms
|
||||
True cause: config/config.go fields (CFTunnelToken/CFAPIToken :72-75, HubConfig.APIKey :175,
|
||||
GitConfig.Token :96, WebConfig.SessionSecret :88) are plain yaml; loadAndParse
|
||||
(:206-224) does ReadFile→ExpandEnv→Unmarshal, no decryption. App.yaml secrets ARE
|
||||
enc:-wrapped (deploy.go:683-692). Config is written 0644 on apply (router.go:1007).
|
||||
Severity: LOW-MEDIUM (these are runtime-required creds the guest legitimately holds; the file
|
||||
mode is the real gap)
|
||||
Repo / layer: controller — internal/config / config apply
|
||||
Fix approach: Write/chmod controller.yaml to 0600. (Encrypting at rest with a key co-located on the
|
||||
same rootfs adds little; if wanted later, derive the key from an env-injected secret,
|
||||
not a file on the guest.)
|
||||
Effort: S
|
||||
Risk: unattended-safe.
|
||||
Depends on / blocks: independent.
|
||||
Proof of fix: stat controller.yaml → 0600.
|
||||
Open question: Is at-rest encryption of infra creds a requirement, or is 0600 sufficient given the
|
||||
threat model (guest-rootfs compromise already = game over)? Recommend 0600 now.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## LOW / TRIVIAL — brief specs
|
||||
|
||||
### F6 — Deploy POST returns `"deployed"` before compose finishes (async by design; message misleads)
|
||||
```
|
||||
Re-diagnosis: real (by design). router.go:381 returns 200 after validation; DeployStack launches
|
||||
compose in a goroutine (deploy.go:339) — the documented anti-stale-button pattern; UI
|
||||
polls GET /api/stacks/{name} every 3s. Only the WORDING is wrong.
|
||||
Fix: change message to "Telepítés elindítva" / return 202 Accepted. Do NOT make it sync.
|
||||
Severity: LOW. Effort S. unattended-safe. controller internal/api.
|
||||
```
|
||||
|
||||
### F7 — Dashboard state lags Docker health by ~10s
|
||||
```
|
||||
Re-diagnosis: real (cache cadence). status-refresh ticker is 30s (cmd/controller/main.go:256);
|
||||
stack-scan 2min (:259); the dashboard list serves the in-memory map. Health is read
|
||||
correctly (manager.go:437), just polled slowly. (Deploy page already polls per-stack 3s.)
|
||||
Fix: lower status-refresh to ~10s, and/or fire an extra RefreshStatus() shortly after
|
||||
deploy/start completes (runComposeDeploy already calls it once at :401).
|
||||
Severity: LOW. Effort S. unattended-safe. controller.
|
||||
```
|
||||
|
||||
### F4 — `GET /api/stacks/rescan` → "stack not found: rescan"
|
||||
```
|
||||
Re-diagnosis: real but trivial. A rescan route EXISTS but POST-only (router.go:112 rescanStacks);
|
||||
GET falls through to GET /stacks/{name} (:116) → getStack("rescan") → "not found" (:314).
|
||||
Reachable via POST /api/stacks/rescan or POST /api/sync.
|
||||
Fix: add a default 405 for /stacks/rescan on non-POST so it doesn't masquerade as a stack
|
||||
lookup; fix the runbook to use POST. (Doc-first.)
|
||||
Severity: TRIVIAL. Effort S. unattended-safe. controller.
|
||||
```
|
||||
|
||||
### F3 — JSON API shows double-encoded Hungarian (`SzemĂ©lyes`)
|
||||
```
|
||||
Re-diagnosis: MIS-ATTRIBUTED / not-in-code → needs-runtime-confirmation. Source .felhom.yml is clean
|
||||
single-UTF-8 (verified hexdump); yaml.Unmarshal (metadata.go:145) and
|
||||
json.NewEncoder (router.go:1079) are UTF-8-native; git-sync copies bytes verbatim
|
||||
(sync.go:379-411). No transcoding exists. The rendered HTML being correct (same source
|
||||
string) is strong evidence the bytes aren't actually doubled — the mojibake is most
|
||||
likely a terminal/curl display artifact in the live measurement, OR a stale cache.
|
||||
Fix: none in code. Re-measure: `curl .../api/stacks | xxd | grep -i saj` and compare bytes to
|
||||
source; only if doubled, re-sync the catalog cache on the guest.
|
||||
Severity: LOW. needs-runtime-confirmation.
|
||||
```
|
||||
|
||||
### F2 — `hdd_configured:false` disagrees with `/api/disks` listing the HDD
|
||||
```
|
||||
Re-diagnosis: already-fine (expected symptom of F9, not a bug). hdd_configured comes from the
|
||||
controller's own config plane: system/info_linux.go:32-35 (true iff hddPath != ""),
|
||||
fed by cfg.Paths.HDDPath (router.go:711). /api/disks is the agent's HOST view. They
|
||||
legitimately differ because the HDD is host-present but NOT guest-bound and NOT in
|
||||
controller.yaml.
|
||||
Fix: no independent fix; the F9 "guest_attached" reporting field resolves the user-facing
|
||||
confusion. Optionally make /api/system/info reflect any registered guest storage path
|
||||
(it currently ignores GetStoragePaths fallback that monitor/report use).
|
||||
Severity: LOW (folds into F9).
|
||||
```
|
||||
|
||||
### F15 — Controller not auto-restarted after a manual `docker stop`/`kill`
|
||||
```
|
||||
Re-diagnosis: already-fine / standard semantics. Golden bakes `docker run --restart unless-stopped`
|
||||
(felhom-agent/configs/build-golden.sh:179) — restarts on crash/daemon-restart but NOT
|
||||
after a manual stop (by design). The systemd unit is Type=oneshot/RemainAfterExit
|
||||
(:189-204) — runs the bootstrap once, not a supervisor. Genuine crashes ARE covered.
|
||||
Fix: none required. If "survives manual stop" becomes a requirement: add a periodic
|
||||
agent/hub desired-state reconcile that re-asserts the container is running (preferred
|
||||
over fighting unless-stopped with systemd Restart=always).
|
||||
Severity: LOW (acceptable as-is).
|
||||
```
|
||||
|
||||
### F13 — 3-2-1 weakened: primary recovery units share the app's disk
|
||||
```
|
||||
Re-diagnosis: already-fine as code; pure consequence of F9. Backup path helpers
|
||||
(appbackup/paths.go:31-70) root the primary unit at the app's own nsRoot by design;
|
||||
the OFF-drive copy is Tier-2 (internal/backup/tier2.go) which already targets a
|
||||
different physical disk (system.SamePhysicalDevice) once one exists.
|
||||
Fix: no path-helper change. Resolved by F9 (attach a real 2nd drive → Tier-2 has an off-disk
|
||||
target). Until then Tier-2 correctly reports "needs 2nd drive".
|
||||
Severity: MEDIUM consequence, but no independent action (→ F9).
|
||||
```
|
||||
|
||||
### PASS findings — confirmed working, no action
|
||||
```
|
||||
F10 (crash-loop detection), F12 (lifecycle ops), F14 (CTRL-T2-1 crash-window), F16 (import/export +
|
||||
CTRL-001 traversal defense), F18 (removal + protected-stack guards), F19 (monitoring/settings/sync/
|
||||
hub reporting). Confirmed real PASSes against code + the report's evidence. Residual sub-issues that
|
||||
DID get specs: F6 (optimistic POST wording), F11 (OS-disk footgun), F7 (state lag). No other residuals.
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## SEQUENCED FIX PLAN
|
||||
|
||||
### Batch 1 — quick contained wins → ship as controller **v0.61.0** (all S, unattended-safe)
|
||||
- **F1** (cgroup memory read — `system/info_linux.go`) — restores the deploy OOM guard; highest value-per-effort.
|
||||
- **F20-BUG1** (surface agent format error — `agentapi/client.go: FormatDisk`) — stop silent destructive-op "success".
|
||||
- **F6** (deploy message wording / 202 — `api/router.go`).
|
||||
- **F7** (status-refresh cadence 30s→10s — `cmd/controller/main.go`).
|
||||
- **F8** (controller.yaml → 0600 — config apply).
|
||||
- **F4** (405 for non-POST `/stacks/rescan` — `api/router.go`).
|
||||
- **Ship alongside (separate repo, auto-syncs): F5 uptime-kuma** healthcheck fix in `app-catalog-felhom.eu` — delete the broken override.
|
||||
- *Grouping rationale:* all controller-internal, no agent/provisioning, no destructive-path behaviour change, independent files. Each is independently testable and low-risk.
|
||||
- *File-collision warnings:* **F4 and F6 both edit `internal/api/router.go`** (different cases — coordinate one PR). **F20-BUG1 edits `agentapi/client.go: FormatDisk`, which Batch-3 F20-BUG3 will re-architect** — land BUG1 first (small error-surface change), and have BUG3 build on it (BUG3's async path must keep the error-surfacing).
|
||||
|
||||
### Batch 2 — controller, needs-spec (M/L), no live host risk
|
||||
- **F17** (replay `.sql` on per-app restore; extract shared `dbrestore` pkg from `appexport`; make volume-restore failure non-silent). The CRITICAL data-recovery fix.
|
||||
- **F11** (drive-class guard in `DeployStack` + dropdown).
|
||||
- *Sequencing:* F17 first (it's the critical correctness fix). F11 is best AFTER F9 lands (real HDD class to steer toward) but can ship a warning now.
|
||||
- *File-collision warnings:* **F17 co-edits `internal/backup/restore_unit.go` + `restore.go` + extracts from `internal/appexport/restore.go`** — keep this as one PR; nothing else in Batch 1/2 touches `internal/backup` or `internal/appexport`. F11 touches `internal/stacks/deploy.go` + `internal/web/handlers.go` + `internal/settings` — disjoint from F17.
|
||||
- *Runtime-confirm before/with F17:* verify whether `POST /backup/run` actually writes volume tars to `<unit>/volume-dumps/` (the live drive saw none for romm). If capture is also missing for some apps, the F17 fix must ensure capture, not just replay.
|
||||
|
||||
### Batch 3 — foundational, SUPERVISED (agent / provisioning / golden / destructive)
|
||||
- **F9** (provisioning auto-re-enroll user-data binds + `guest_attached` reporting field) — agent `reconcile` + `localapi` + golden/bootstrap + small controller field. Unblocks data-migration, F13 (3-2-1), and the real-HDD half of F11/F2.
|
||||
- **F20-BUG3** (async detached mkfs + `/disks/format/status`; run under `s.baseCtx`; preserve AGENT-001 re-resolve) — agent destructive path + controller `agentapi` polling.
|
||||
- **F20-BUG2** (durable_id scheme consistency: `/disks` advertises the gate-accepted id) — agent `storage`.
|
||||
- *Why last + supervised:* all touch the host agent / LXC config / destructive wipe path and need live validation on scratch guests/devices, not unattended pushes. They also carry the highest blast radius (provisioning, disk-wipe).
|
||||
- *File-collision warnings:* **F20-BUG2 and F20-BUG3 both touch `felhom-agent/internal/storage` + `internal/localapi/disks.go`** (durable-id derivation + the format handler) — do them in one coordinated agent PR. **F20-BUG3 re-edits controller `agentapi/client.go: FormatDisk`** already changed in Batch 1 (BUG1) — rebase BUG3 on the BUG1 error-surfacing. **F9 touches `reconcile/bringup.go` + `localapi/disks.go` + `guestbind.go`** — `disks.go` is also touched by F20-BUG2/3, so sequence F9 and the F20 agent work to avoid stepping on `disks.go` (recommend: F9 first, then F20-BUG2/BUG3 rebased).
|
||||
|
||||
---
|
||||
|
||||
## WRAP-UP
|
||||
|
||||
**Re-diagnosis verdict counts:** real = **11** (F1, F5, F9, F11, F17, F20-BUG1, F20-BUG2, F20-BUG3, F6, F7, F8); mis-attributed = **3** (F20-BUG1 relocated to controller, F17 volume-capture sub-claim, F3 not-in-code); already-fine/no-independent-action = **9** (F10, F12, F14, F16, F18, F19, F13, F2, F15); needs-runtime-confirmation = **2** (F3 byte recheck, F17 capture-side volume-tar question); trivial/doc = **1** (F4).
|
||||
|
||||
**Batch 1 (ship first, v0.61.0):** F1 (cgroup memory → restores deploy OOM guard), F20-BUG1 (stop silent format "success"), F6 (deploy message wording), F7 (status cadence), F8 (controller.yaml 0600), F4 (405 on rescan) + F5 uptime-kuma healthcheck in the catalog repo.
|
||||
|
||||
**Report-causes found WRONG against the code (the key output):**
|
||||
- **F20-BUG1** — report blamed the *agent's* format handler; the agent correctly returns 502. The bug is the *controller's* `agentapi.FormatDisk` (`client.go:393-397`) swallowing the agent's error and returning a zero-value `ok:true`.
|
||||
- **F17** — report said "recovery units don't capture volume data"; they do (`restoreDockerVolumes` + `VolumeDumps`). The real gap is the captured `.sql` dump is **never replayed** on per-app restore (and the DB-import logic already exists, unused, in `appexport`).
|
||||
- **F3** — report said the JSON API double-encodes Hungarian; no transcoding exists in the code path (source is clean UTF-8, encoder is UTF-8-native, sync copies bytes). Most likely a terminal/`curl` display artifact — re-measure with `xxd`.
|
||||
- (Refinement, not contradiction) **F9** — the disk APIs don't "misreport"; they faithfully describe *host* presence. The real defect is the *absence* of a guest-attached signal, plus provisioning never binding user-data drives.
|
||||
|
||||
**Open design questions needing the operator's decision before coding:**
|
||||
1. **F5 (Traefik/unhealthy):** publish routes for unhealthy containers, keep gating, or keep gating but surface "route unpublished because unhealthy" distinctly? (Recommend the last.)
|
||||
2. **F9 (re-enroll):** may bring-up auto-re-assert previously-enrolled user-data binds unattended, or must an operator confirm each data-path bind? (Recommend auto for durable-id-matched intents.)
|
||||
3. **F17 (double-apply):** when an app has BOTH a named-volume tar and a `.sql` dump, which wins? (Recommend `.sql` replay for logical consistency.)
|
||||
4. **F11 (no-HDD nodes):** warn-and-allow vs hard-refuse a needs_hdd app on a single-SSD node? (Recommend warn-and-allow.)
|
||||
5. **F20-BUG3:** must an in-progress format survive an agent restart (persisted job record)? (Recommend yes, reuse the backup-record pattern.)
|
||||
6. **F8:** is at-rest encryption of infra creds required, or is `0600` sufficient under the current threat model? (Recommend 0600 now.)
|
||||
@@ -1,35 +0,0 @@
|
||||
# Session 2026-06-13 — backlog-Medium cleanup + doc currency + live-drive runbook
|
||||
|
||||
Working/checkpoint log. Commit/push after each Part. Controller-only (agent v0.30.0 left alone).
|
||||
|
||||
## Progress log
|
||||
|
||||
- 23:1x — Start. Controller main `d6aacf2`, demo 9201 on v0.59.0 healthy. Verified the 6 backlog Mediums.
|
||||
- **Backlog-Medium verdicts (commit d6aacf2):**
|
||||
- **M4** (SubdomainInUse I/O under RLock) — **FIXED** @ stacks/deploy.go:67-80 (collects dirs under RLock, RUnlock, then LoadAppConfig with lock released).
|
||||
- **M5** (MigrateEncryption encKey nil-check race) — **FIXED** @ stacks/manager.go:129-131 (nil-check now inside `m.mu.Lock()`; SetEncryptionKey writes under same lock).
|
||||
- **M6** (MigrateEncryption holds lock across I/O) — **FIXED-in-practice** @ stacks/manager.go:128-191 — lock still spans the I/O loop, but runs once at startup (main.go:154) strictly before scheduler (`sched.Start`) and HTTP server; nothing concurrent to block. Acceptable, no action.
|
||||
- **M18** (ListDumpFiles re-validates every dump every 5min) — **LIVE** @ appbackup/dbdump.go:469. Perf; cross-package-entangled (settings cache has no size/modtime; signature change + bridge). → escape-hatch branch.
|
||||
- **M19** (deriveStackName naive suffix-strip) — **LIVE** @ appbackup/dbdump.go:536-551. Low-incidence correctness (misattributes a stack literally named `<x>-cache`/`-db`); needs deployed-stack list injected into appbackup. → escape-hatch branch.
|
||||
- **M25** (Server integrationMgr data race) — **LIVE** @ web/server.go:119-120 (constructor launches `go SyncFileBrowserMounts`) vs :162 `SetIntegrationManager` (post-construction, unsynchronized) read at handlers.go:358,360,1433. Real race. → clean fix (atomic.Pointer), ship in v0.60.0.
|
||||
|
||||
## Plan
|
||||
- Ship **M25** (atomic.Pointer) in controller **v0.60.0** + concurrency `-race` test.
|
||||
- **M18 / M19** → `fix/m18-*` / `fix/m19-*` branches (pushed, NOT deployed) with implemented fix + notes (escape hatch: entangled cross-package changes, not forced at 2am).
|
||||
- Part 2 docs, Part 3 runbook.
|
||||
|
||||
## PART 1 — DONE
|
||||
- M4/M5/M6: FIXED (no action). M25: fixed (atomic.Pointer) `6953899`; CHANGELOG v0.60.0 `7dd1aba`.
|
||||
- M25 `-race` proven clean on the build server. M18/M19: notes + fix plan on `fix/m18-dump-validation-cache` / `fix/m19-stackname-crossref` (pushed, NOT deployed).
|
||||
- **Deployed v0.60.0** to demo 9201 (rollback ref v0.59.0). Verified: `0.60.0 Up (healthy)`, self-test 6/0, "Controller elindult (0.60.0)", public `/api/health` 200, no errors. No rollback.
|
||||
|
||||
## PART 2 — docs: DONE
|
||||
- `felhom-agent/CLAUDE.md`: stale v0.13.0 banner → v0.30.0 current-state summary (agent `238ec02`).
|
||||
- `felhom-controller/CLAUDE.md`: corrected the WRONG "bulk strip has NOT happened" status → de-priv DONE (slice 8C); `CONTEXT.md` → v0.60.0 banner (ctrl `c67c53b`).
|
||||
- Doc-gaps closed (felhom.eu `4c0eb2f`): hub v0.6.3→v0.11.0 (05 doc); onlyoffice:nextcloud occ internals (auth-hub-sync doc); metricsDBPath **verified** on the live guest (coincides with volume-backed data_dir → persists; hardcoding is latent fragility only) + storage doc note. REORG-NOTES updated (gaps 2/3 closed, 5 partial). Remaining gaps (volume-dump-consistency #1, selfUpdateAuthMiddleware #4, design-drafts 01/03/04) left flagged.
|
||||
- controller/README.md slim-down: DEFERRED again (1987-line rewrite; banner already points to central docs — safety/budget).
|
||||
|
||||
## PART 3 — runbook: DONE
|
||||
- `felhom-controller/RUNBOOK-e2e-live-drive.md` written for tomorrow's supervised drive. 8 sections, non-destructive first (baseline → deploy happy → **CTRL-T2-1 crash-window probe** → **CTRL-001 import + traversal-negative** → backup 3-2-1 → **AGENT-001 wipe-gate refusals**), then **[DESTRUCTIVE] restore** last. Each step: exact UI path/API call, what "good" looks like, what to watch for. All routes verified against current source.
|
||||
|
||||
## ALL PARTS DONE.
|
||||
@@ -1,30 +0,0 @@
|
||||
# Session 2026-06-13 — audit-fix + docs-centralize working log
|
||||
|
||||
Working/checkpoint file for the unattended fix+docs session. Commit/push after each Part.
|
||||
|
||||
## Progress log
|
||||
|
||||
- **Part 1 — controller fixes → deploy: DONE & VERIFIED.**
|
||||
- Fix 1 CTRL-001 (import path traversal) — `appexport/validate.go` + `UnmarshalManifest` guard + join-loop guards + regression test. Commit `c20ff56`.
|
||||
- Fix 2 CTRL-T2-1 + H10 (deploy.go slice) — transitional `Deployed:false` until compose succeeds; `SaveAppConfig` fail-closed on encrypt error. Regression tests. Commit `5a80739`.
|
||||
- Fix 3 M2 (init-only stackProvider, lock removed + documented). Commit `092cbbe`.
|
||||
- CHANGELOG v0.59.0. Commit `7c45fec`. Pushed to main (`eea235b..7c45fec`).
|
||||
- Green gate: `go build/vet/test ./...` all PASS before each commit.
|
||||
- **Deploy:** built `felhom-controller:0.59.0` on 192.168.0.180; deployed to guest **9201 (demo-felhom)** via the bootstrap mechanism (`/etc/felhom-controller-image` + `systemctl restart felhom-controller-bootstrap.service`). Rollback ref was `0.58.0`.
|
||||
- **Verify:** `docker ps` → `0.59.0 Up (healthy)`; self-test 6 passed/0 failed; "Controller elindult (0.59.0)"; hub report pushed; public URL `https://felhom.demo-felhom.eu/api/health` → HTTP 200 `{"ok":true}`. NO rollback needed.
|
||||
- Deploy-target note: CLAUDE.md's `/opt/docker` compose path is stale (re-platform). Live demo controller runs in LXC guest 9201 on felhom-pve, bootstrap-managed. felhotest (router.abonet.hu:33022) was unreachable (timeout) — skipped.
|
||||
|
||||
- **Part 2 — agent AGENT-001 fix (branch only): DONE.** Branch `fix/agent-001-wipe-durable-reresolve` @ `d96e5bd` (agent repo), pushed, NOT merged/deployed. `antiRetargetResolve` (resolve→re-derive+match→re-inspect, format the re-resolved device); injectable `reresolveWipe` seam; unit test covers mismatch/gone/blank/empty refusals; existing integration test green via stub. `AGENT-001-FIX-NOTES.md` on the branch (5-line review checklist). Full agent `go build/vet/test ./...` green.
|
||||
|
||||
- **Part 3 — audit/BUGHUNT records: DONE.** Audit records → `felhom.eu/documentation/audits/{deep-sweep,bughunt-reconcile}-2026-06-13.md` with per-finding remediation-status headers (felhom.eu `4320c01`). Controller root `BUGHUNT.md` reconciliation banner (H10 fixed, pointer) — controller `6cd1fc9`.
|
||||
|
||||
- **Part 4 — docs centralize + refresh: DONE.** `felhom.eu/documentation/` now has: top-level `README.md` index; `controller/` subtree (README, module-map superseding the v0.33 planning map, + 4 code-verified feature docs: deploy/stack-lifecycle, backup-architecture, storage-monitoring-metrics, auth-hub-sync-integrations); `REORG-NOTES-2026-06-13.md` (rationale + verification ledger + flagged doc-gaps). felhom.eu `21d0e7c`/`e68fe3e`. Controller `README.md` banner → central docs; `CONTEXT.md` banner refreshed to v0.59.0; stray `controller/mnt/` artifact removed — controller `86a61fb`. Agent/hub design-drafts left as-is (flagged drift in REORG-NOTES per the cardinal rule).
|
||||
|
||||
## Final state
|
||||
- Controller main: fixes + docs through `86a61fb`; v0.59.0 deployed+verified on demo 9201.
|
||||
- Agent: fix on branch `fix/agent-001-wipe-durable-reresolve` `d96e5bd` (PENDING REVIEW).
|
||||
- felhom.eu main: docs through `e68fe3e`.
|
||||
|
||||
## Assumptions
|
||||
- "The demo node" = guest 9201 (runs the v0.58.0 the task cited; bootstrap-managed). felhotest unreachable, not deployed there.
|
||||
- M2 included in v0.59.0 (trivial, zero-runtime-change) since Fixes 1-2 were green; no risk signal.
|
||||
@@ -1,13 +0,0 @@
|
||||
# SESSION 2026-06-14 — Fix Batch 1 (+F17) → controller v0.61.0
|
||||
|
||||
Branch: `audit/2026-06-14-live-drive`. Implementing controller-side live-drive fixes per `LIVE-DRIVE-FIXSPEC-2026-06-14.md`.
|
||||
Exclusions: F9, F20-BUG2, F20-BUG3 (agent/golden — supervised next session).
|
||||
Operator decisions applied: F17 `.sql` wins; F5 gate route but surface "route unpublished"; F8 0600; F11 warn-and-allow (note: F11 not in this batch's fix list — only F1, F20-BUG1, F4, F6, F7, F8, F5, F17).
|
||||
|
||||
## Progress log
|
||||
- **t0** — Read CLAUDE.md (build workflow), FIXSPEC, target source files. Setting up.
|
||||
- **Test approach (documented assumption):** controller code is largely `//go:build linux`; dev host is Windows. Per-commit gate = `CGO_ENABLED=0 GOOS=linux go build ./... && go vet ./...` locally (compiles + type-checks code AND `_test.go`). Authoritative `go test ./...` runs on the Linux build server (192.168.0.180) against the pushed branch BEFORE the image build/deploy. Windows cannot execute linux test binaries, so this is the faithful interpretation of the per-commit green gate.
|
||||
- **t2 — quick wins DONE** (separate commits): F20-BUG1 (agentapi.FormatDisk surfaces non-2xx/ok:false instead of zero-value success; test passes, fails on old code), F4 (405 for non-POST /stacks/rescan), F6 (deploy POST → 202 "Telepítés elindítva"; UI checks data.ok so 202 safe), F7 (status-refresh 30s→10s), F8 (writeConfig0600 helper enforces 0600 even on a pre-existing file; test skips on Windows, asserts on Linux). F5 catalog: uptime-kuma healthcheck fixed (→ `extra/healthcheck` binary + 180s start_period) committed+pushed to app-catalog main. F5 dashboard: `routeUnpublished` funcmap helper + indicator on stacks/dashboard cards + CSS; tests (func + real-template parse + fragment render) pass.
|
||||
- **t3 — F17 DONE.** Reuse decision: put `ImportDump`/`waitDBReady` in `appbackup` (the DB-domain home with DiscoverDatabases/DumpOne/getMariaDBPassword) rather than reusing appexport's unexported copies — appexport→appbackup already exists, so the reverse would CYCLE; appbackup is the clean shared home and `DiscoveredDB` already carries the live container's creds (no env threading). `backup.reimportDBDumps` (injectable discover/import seams) runs after volume restore in both RestoreFromRecoveryUnit and RestoreApp → `.sql` WINS. Volume-restore + DB-import failures now SURFACE (restore returns error). Unit tests pass; full backup/appbackup/agentapi/web suites green locally. **F17 escape hatch:** shipping it in v0.61.0 contingent on the LIVE DB round-trip passing post-deploy; if it fails, revert F17 to branch fix/f17-restore-db-reimport and ship the rest.
|
||||
- **t4 — DEPLOY + LIVE VERIFY.** Full `go test ./...` GREEN on Linux build server (incl. linux `system`/`api` tests). Built+pushed `:0.61.0`, deployed to 9201 (golden/bootstrap). Healthy, /api/health + dashboard 200. **F17 escape hatch CLEARED** — live romm marker round-trip PASSED (drop after backup → `/backup/restore` → "replayed 1 DB dump(s)" → row returned). **F1 reroot:** discovered the cgroup approach was a no-op (container cgroup=max; cap is on the LXC ancestor; no lxcfs) → reworked to `docker info` MemTotal + committed-memory guard → rebuilt+redeployed → **`total_mem_mb=2048`** live (was 15771). F4 → 405 live. F5 → uptime-kuma healthy → route 302 live (was 404). Docs updated (CHANGELOG, README, FIXSPEC statuses, REPORT). DONE — all Batch-1 + F17 shipped in v0.61.0; F9/F20-BUG2/F20-BUG3 deferred to supervised session.
|
||||
- **t1 — F1 (cgroup memory)** DONE. `internal/system/info_linux.go`: `readMemInfo` now prefers the cgroup memory LIMIT (v2 `memory.max`, v1 `memory.limit_in_bytes`, "max"/near-uint64-max = unlimited→fallback) when finite and below the host `/proc/meminfo` total; used = `memory.current`/`usage_in_bytes`. Restores the deploy OOM guard (deploy.go:162-185 reads GetMemoryMB). Test `info_cgroup_test.go` (4 cases) — fails on pre-fix code (ignored cgroup). GOOS=linux build+vet OK.
|
||||
Reference in New Issue
Block a user