From 84979932e7c86ad952cb687e45a6db8b7dbda3e1 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sun, 14 Jun 2026 10:25:00 +0200 Subject: [PATCH] docs: v0.61.0 CHANGELOG + README (F1 mem/F17 restore/F5 route) + FIXSPEC statuses + REPORT --- CHANGELOG.md | 46 ++++++++++++++ LIVE-DRIVE-FIXSPEC-2026-06-14.md | 26 ++++++++ REPORT.md | 100 +++++++++++++++---------------- SESSION-2026-06-14-fixbatch.md | 1 + controller/README.md | 31 ++++++++-- 5 files changed, 148 insertions(+), 56 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6a6b47d..23e9a0f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,51 @@ ## Changelog +### v0.61.0 — live-drive Batch 1 (+F17) fixes (2026-06-14) + +Controller-side fixes triaged in `LIVE-DRIVE-FIXSPEC-2026-06-14.md` from the 2026-06-14 live-drive +findings. Each fix has a regression test that fails on the pre-fix code. Built, deployed to demo guest +9201, and the key fixes live-verified. (F9, F20-BUG2, F20-BUG3 are the SUPERVISED agent/golden next +session — not in this batch.) + +- **F17 (CRITICAL) — per-app restore now replays the captured `.sql` DB dump.** `RestoreFromRecoveryUnit` + (and the `RestoreApp` fallback) repopulated Docker volume tars but NEVER replayed the captured + `-.sql`, so DB-resident data did not come back. New `appbackup.ImportDump` (read-side + counterpart to `DumpOne`, reuses `DiscoveredDB`'s own discovered credentials) + `backup.reimportDBDumps` + replay the dump AFTER volume restore + stack bring-up, so the logical dump **wins** over any volume-tar + copy of the DB (operator-chosen precedence). Volume-restore and DB-import failures now **surface** (the + restore returns an error) instead of a swallowed WARN. **Live-validated** on guest 9201: a marker row + dropped after backup was restored by `/backup/restore` (log: "replayed 1 DB dump(s)"). Reuse note: + `ImportDump` lives in `appbackup` (the DB-domain package) — `appexport→appbackup` already exists so + reusing appexport's unexported copies would cycle; appbackup is the clean shared home. +- **F1 (HIGH) — guest RAM cap read from the Docker daemon; deploy guard uses committed memory.** The + controller container reported the Proxmox host's 16 GB (no lxcfs in the container; its own cgroup is + unlimited — the 2 GB cap is on the LXC ancestor), defeating the deploy memory-headroom hard-block. + `system` now sources the cap from `docker info` MemTotal (the daemon runs in the LXC → reports the + guest's real RAM; cgroup limit still preferred when present). The deploy guard now uses the controller's + own committed-app memory (sum of running mem requests) for "used" — accurate and cheap — instead of + host RSS. `/api/system/info` reports the guest cap + committed used. **Live-verified:** `total_mem_mb` + 2048 (was 15771). +- **F20-BUG1 (HIGH) — `agentapi.FormatDisk` surfaces the agent's error.** A failed format (agent 502 + "device is mounted", `ok:false`, `data:null`) fell through to `return out, nil`, so the web layer + reported a zero-value result as `ok:true` — a failed DESTRUCTIVE op read as success. Now returns a + non-nil error on any non-2xx/`ok:false` that is not a recognized refusal (403/needs-confirmation). +- **F5 (HIGH) — broken healthcheck → 404, two parts.** (catalog, `app-catalog-felhom.eu`) uptime-kuma's + healthcheck pointed at a v1-era `node /app/extra/healthcheck.mjs` absent in `:2`, so the container + stayed unhealthy and Traefik withheld the route (404 though running) — fixed to the v2 compiled + `extra/healthcheck` binary + 180s start_period. (dashboard) new `routeUnpublished` helper + a distinct + "URL nem elérhető – útvonal nincs publikálva" indicator on the dashboard/stacks cards for + unhealthy/restarting deployed apps (operator decision: keep gating the route, surface it distinctly). + **Live-verified:** uptime-kuma healthy → route publishes → status URL 302 (was 404). +- **F8 (LOW-MED) — `controller.yaml` persisted 0600.** It holds infra credentials (cf/hub tokens) in + plaintext; the Hub config-apply handler wrote 0644. New `writeConfig0600` enforces 0600 even on a + pre-existing 0644 file. +- **F6 (LOW) — deploy POST reports "started", not "deployed".** The deploy runs async (UI polls); the + POST now returns 202 Accepted + "Telepítés elindítva…" so API/script consumers aren't told a deploy + finished before it has. +- **F7 (LOW) — dashboard state lag.** `status-refresh` tightened 30s → 10s (cheap docker-ps refresh). +- **F4 (TRIVIAL) — `GET /api/stacks/rescan`** now returns 405 + `Allow: POST` instead of the misleading + "stack not found: rescan" fall-through. + ### v0.60.0 — M25 data-race fix (backlog-Medium cleanup) (2026-06-13) Backlog-Medium reconciliation from the 2026-06-13 BUGHUNT reconcile. M4/M5/M6 verified already FIXED diff --git a/LIVE-DRIVE-FIXSPEC-2026-06-14.md b/LIVE-DRIVE-FIXSPEC-2026-06-14.md index b25cb35..1d0e050 100644 --- a/LIVE-DRIVE-FIXSPEC-2026-06-14.md +++ b/LIVE-DRIVE-FIXSPEC-2026-06-14.md @@ -15,6 +15,32 @@ --- +## BATCH 1 (+F17) IMPLEMENTATION STATUS — shipped v0.61.0 (2026-06-14) + +All controller-side Batch-1 items + F17 implemented, tested, built, deployed to demo guest 9201, and the +key fixes live-verified. (F9, F20-BUG2, F20-BUG3 remain for the SUPERVISED agent/golden session.) + +| Finding | Status | Commit | Live verification | +|---|---|---|---| +| F1 | **FIXED** | `0550b31`+`4989513` | `/api/system/info total_mem_mb=2048` (was 15771) | +| F20-BUG1 | **FIXED** | `2cf3fad` | unit test (502→non-nil err) | +| F4 | **FIXED** | `56fe574` | `GET /api/stacks/rescan` → 405 | +| F6 | **FIXED** | `56fe574` | 202 + "Telepítés elindítva" (code/UI) | +| F7 | **FIXED** | `4938cc8` | status-refresh 10s | +| F8 | **FIXED** | `6868489` | unit test (mode 0600 on Linux) | +| F5 (catalog) | **FIXED** | app-catalog `main` | uptime-kuma healthy → route 302 (was 404) | +| F5 (dashboard) | **FIXED** | `803ce50` | funcmap + template render tests | +| F17 | **FIXED** | `0b9450e` | **marker DB round-trip PASSED live** (escape hatch cleared) | +| F9, F20-BUG2, F20-BUG3 | DEFERRED | — | SUPERVISED next session (agent/golden) | + +Note on F1: the FIXSPEC's "read the cgroup limit" approach proved a **no-op on the demo** — the +controller container's own cgroup is unlimited (the 2 GB cap is on the LXC ancestor, hidden) and there is +no lxcfs in the container. The working source is `docker info` MemTotal (the daemon runs in the LXC), with +the deploy guard's "used" switched to the controller's committed-memory accounting (host RSS is not +observable per-guest). cgroup-limit reading is retained for the non-nested case. + +--- + ## RE-DIAGNOSIS SUMMARY (verdict counts) | Verdict | Count | Findings | diff --git a/REPORT.md b/REPORT.md index 34edcc5..4d7d209 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,56 +1,56 @@ -# REPORT — felhom-controller v0.58.0 (infra-protection prevention layer for the OS/Docker-data split) +# REPORT — live-drive Batch 1 (+F17) → controller v0.61.0 (2026-06-14) -Phase 2 of the OS/Docker-data storage-split slice (Phase 1 = felhom-agent v0.29.0: golden + provision). -The controller guest's OS rootfs and Docker data are now split onto separate `local-lvm` volumes for -resilience; infra (controller/traefik/cloudflared/filebrowser) shares the one Docker data-root and is -protected by **prevention, not placement**. Built, deployed, and **live-validated on a freshly -re-provisioned guest 9201**. +Implemented the controller-side fixes from `LIVE-DRIVE-FIXSPEC-2026-06-14.md` (Batch 1 + the CRITICAL +F17), shipped as **v0.61.0** to demo guest 9201. Each fix has a regression test that fails on the +pre-fix code; the authoritative `go build && go vet && go test ./...` is green on the Linux build server. -## What shipped (v0.58.0) -- **Reserved-buffer headroom guard** (`internal/system/dockervol.go`): `GetDockerVolumeHeadroom()` - measures the Docker-data volume via `statfs("/")` — the controller container's root overlay is the - upperdir on the guest's `/var/lib/docker` volume (true with the **overlay2** driver; see the agent - report), so `/` reports the data volume. Reserve floor `DockerVolumeReserveGB = max(5 GB, 10%)`. - Fail-open on a measurement error. -- **Deploy-time hard gate** (`internal/api/router.go` `deployStack`): a new deploy is **refused (HTTP - 507** + Hungarian message) when free space on the Docker-data volume is at/under the reserved buffer. -- **Deploy-page surfacing** (`deploy.html`): a new deploy below the buffer shows a Hungarian warning and - **disables** the "Telepítés indítása" button; the API gate is the hard backstop. -- **Runtime monitoring** (`monitor/healthcheck.go`): confirmed `DiskPercent` watches the Docker-data - volume (statfs `/`); warn 80% / crit 90% trip ABOVE the 10%-free buffer, so the customer is warned - before the gate engages. Clarifying comment added. -- **Log rotation** baked into the golden's `daemon.json` (agent side; `max-size 10m`, `max-file 3`). -- Tests: `DockerVolumeReserveGB` floor/scale. +## Fixes (each its own commit) -## Live validation (guest 9201, freshly re-provisioned from the split golden) -9201 was **destroyed and re-provisioned** from the new split golden (32 GB OS rootfs + 256 GB Docker-data -volume, `backup=1`), via `felhom-agent --selftest=provision` + a reboot. The controller bootstrapped -from baked images (no pull), **pulled its config from the hub** (catalog synced — 55 app defs — hub -HTTP 200, CF token configured, hub report pushed). Then: +| Finding | Sev | Change | Test | Live-verified | +|---|---|---|---|---| +| **F17** | CRIT | per-app restore now **replays the captured `.sql` dump** (new `appbackup.ImportDump` + `backup.reimportDBDumps`), AFTER volume restore so the **dump wins**; volume/DB failures now surface | `restore_db_test.go` (injectable seams) | **marker DB round-trip PASSED** on romm | +| **F1** | HIGH | guest RAM cap from `docker info` (daemon runs in LXC); deploy guard "used" = committed memory | `info_cgroup_test.go` (cgroup + docker-info paths) | `total_mem_mb=2048` (was 15771) | +| **F20-BUG1** | HIGH | `agentapi.FormatDisk` surfaces non-2xx/`ok:false` instead of zero-value success | `disks_test.go` (502→err) | unit | +| **F5 (catalog)** | HIGH | uptime-kuma healthcheck → v2 `extra/healthcheck` binary + 180s start_period | — (catalog) | uptime-kuma **healthy → route 302** (was 404) | +| **F5 (dashboard)** | HIGH | `routeUnpublished` helper + distinct indicator for unhealthy/restarting deployed apps | `route_unpublished_test.go` | render test | +| **F8** | LOW-MED | `controller.yaml` persisted 0600 (holds infra tokens) | `config_perms_test.go` | unit (Linux) | +| **F6** | LOW | deploy POST → 202 + "Telepítés elindítva" (was "deployed" before compose) | — | code/UI | +| **F7** | LOW | `status-refresh` 30s → 10s | — | cadence | +| **F4** | TRIV | `GET /api/stacks/rescan` → 405 + `Allow: POST` | — | `→ 405` | -- **Split layout:** controller image 0.58.0, **Storage Driver overlay2**, `Docker Root Dir - /var/lib/docker`; images on the data volume (`/var/lib/docker/overlay2` 1.7 GB), `/var/lib/containerd` - idle (380 K); `df`: `/` 935 MB/32 GB (4%, lean OS rootfs), `/var/lib/docker` 256 GB. -- **Prevention gate (the headline):** with ample space the deploy page shows **no** gate banner; after - `fallocate`-filling the data volume to 99% (3.2 GB free < 25.6 GB reserve), a `POST /api/stacks/.../deploy` - returned **HTTP 507** with the Hungarian "Nincs elég szabad tárhely" message — proven on the real - 256 GB data volume. -- **Regression:** `/`, `/stacks`, `/backups`, `/monitoring`, `/stacks/{n}/deploy`, `/stacks/{n}/backup` - all HTTP 200; A1 host-storage list still ordered + friendly-labelled (felhom-usb → local-lvm → local - → felhom-pbs); A2 Tier-2 panel route serves. -- **Deploy path + DB-on-data-volume (step 3):** deployed ActualBudget (HTTP 200, container up); its named - volume landed at `/var/lib/docker/volumes/actualbudget_actualbudget_data` = the data volume. -- **External access:** via Cloudflare the controller returns HTTP 200 for vmid 9201 (tunnel + traefik - route healthy). (A local-DNS override on the dev machine points the hostname at a stale LAN IP — a - red herring; the real public path works.) +## Key implementation notes -## OS isolation (resilience — the reason for the split), proven on the provisioned guest -Filling the Docker-data volume to 100% (239 GB) left the OS rootfs at 4% and fully writable, the guest -healthy throughout — the data volume cannot starve the OS. +- **F17 reuse decision:** `ImportDump` lives in `internal/appbackup` (the DB-domain package with + `DiscoverDatabases`/`DumpOne`/`getMariaDBPassword`), NOT reusing `appexport`'s unexported copies — + `appexport→appbackup` already exists, so the reverse would import-cycle. `DiscoveredDB` carries the live + container's own creds, so no env threading. Precedence: replay runs after volume restore → `.sql` wins. +- **F1 reroot:** the FIXSPEC's cgroup-limit approach was a **no-op** on the demo (the container's own + cgroup is unlimited; the 2 GB cap is on the LXC ancestor, hidden; no lxcfs in the container). The + working source is `docker info` MemTotal. The deploy guard's "used" switched from host RSS (not + per-guest observable) to the controller's committed-memory accounting. -## Outstanding (demo restoration, not slice validation) -- **RomM (HDD app) + USB re-enroll:** RomM's data is safe on the host USB (`/mnt/felhom-usb/felhom-data`, - untouched by the re-provision). Restoring it is the slice-10 enroll flow (assign → guest-attach → - reboot to activate the bind → register storage → deploy). With the split, the USB binds to a free slot - (mp1+) since **mp0 is now the Docker-data volume** — no collision. Documented as the final restore step; - not required for slice validation (ActualBudget covered the deploy path; the USB bind was not touched). +## Deploy & verification + +- Built+pushed `gitea.dooplex.hu/admin/felhom-controller:0.61.0`; deployed via the guest-9201 + golden/bootstrap mechanism (`/etc/felhom-controller-image` + restart bootstrap). Healthy, `/api/health` + + dashboard 200, no startup errors. +- Live-verified: F1 (`total_mem_mb=2048`), F4 (405), F5 (uptime-kuma healthy → route 302), **F17 (marker + row dropped after backup was restored by `/backup/restore`)**. + +## Test approach + +Controller code is largely `//go:build linux`; dev host is Windows. Per-commit gate = +`CGO_ENABLED=0 GOOS=linux go build && go vet` locally (compiles code + tests); authoritative +`go test ./...` ran on the Linux build server (all green, incl. the linux `system` tests). + +## Not in this batch (SUPERVISED, next session — agent/golden) + +**F9** (HDD provisioning/guest-attach + reporting), **F20-BUG2** (durable_id scheme), **F20-BUG3** +(async mkfs). These touch the host agent, golden template, and bring-up (blast radius beyond the +disposable demo). + +## Final state + +Demo guest 9201 on `:0.61.0`, healthy; agent unchanged. Apps: actualbudget, gokapi, mealie, privatebin, +romm(+db,+redis), uptime-kuma (now **healthy**), vikunja — all healthy. felhom-usb still not attached to +the guest (F9, deferred). diff --git a/SESSION-2026-06-14-fixbatch.md b/SESSION-2026-06-14-fixbatch.md index f4cba8d..30c0ef9 100644 --- a/SESSION-2026-06-14-fixbatch.md +++ b/SESSION-2026-06-14-fixbatch.md @@ -9,4 +9,5 @@ Operator decisions applied: F17 `.sql` wins; F5 gate route but surface "route un - **Test approach (documented assumption):** controller code is largely `//go:build linux`; dev host is Windows. Per-commit gate = `CGO_ENABLED=0 GOOS=linux go build ./... && go vet ./...` locally (compiles + type-checks code AND `_test.go`). Authoritative `go test ./...` runs on the Linux build server (192.168.0.180) against the pushed branch BEFORE the image build/deploy. Windows cannot execute linux test binaries, so this is the faithful interpretation of the per-commit green gate. - **t2 — quick wins DONE** (separate commits): F20-BUG1 (agentapi.FormatDisk surfaces non-2xx/ok:false instead of zero-value success; test passes, fails on old code), F4 (405 for non-POST /stacks/rescan), F6 (deploy POST → 202 "Telepítés elindítva"; UI checks data.ok so 202 safe), F7 (status-refresh 30s→10s), F8 (writeConfig0600 helper enforces 0600 even on a pre-existing file; test skips on Windows, asserts on Linux). F5 catalog: uptime-kuma healthcheck fixed (→ `extra/healthcheck` binary + 180s start_period) committed+pushed to app-catalog main. F5 dashboard: `routeUnpublished` funcmap helper + indicator on stacks/dashboard cards + CSS; tests (func + real-template parse + fragment render) pass. - **t3 — F17 DONE.** Reuse decision: put `ImportDump`/`waitDBReady` in `appbackup` (the DB-domain home with DiscoverDatabases/DumpOne/getMariaDBPassword) rather than reusing appexport's unexported copies — appexport→appbackup already exists, so the reverse would CYCLE; appbackup is the clean shared home and `DiscoveredDB` already carries the live container's creds (no env threading). `backup.reimportDBDumps` (injectable discover/import seams) runs after volume restore in both RestoreFromRecoveryUnit and RestoreApp → `.sql` WINS. Volume-restore + DB-import failures now SURFACE (restore returns error). Unit tests pass; full backup/appbackup/agentapi/web suites green locally. **F17 escape hatch:** shipping it in v0.61.0 contingent on the LIVE DB round-trip passing post-deploy; if it fails, revert F17 to branch fix/f17-restore-db-reimport and ship the rest. +- **t4 — DEPLOY + LIVE VERIFY.** Full `go test ./...` GREEN on Linux build server (incl. linux `system`/`api` tests). Built+pushed `:0.61.0`, deployed to 9201 (golden/bootstrap). Healthy, /api/health + dashboard 200. **F17 escape hatch CLEARED** — live romm marker round-trip PASSED (drop after backup → `/backup/restore` → "replayed 1 DB dump(s)" → row returned). **F1 reroot:** discovered the cgroup approach was a no-op (container cgroup=max; cap is on the LXC ancestor; no lxcfs) → reworked to `docker info` MemTotal + committed-memory guard → rebuilt+redeployed → **`total_mem_mb=2048`** live (was 15771). F4 → 405 live. F5 → uptime-kuma healthy → route 302 live (was 404). Docs updated (CHANGELOG, README, FIXSPEC statuses, REPORT). DONE — all Batch-1 + F17 shipped in v0.61.0; F9/F20-BUG2/F20-BUG3 deferred to supervised session. - **t1 — F1 (cgroup memory)** DONE. `internal/system/info_linux.go`: `readMemInfo` now prefers the cgroup memory LIMIT (v2 `memory.max`, v1 `memory.limit_in_bytes`, "max"/near-uint64-max = unlimited→fallback) when finite and below the host `/proc/meminfo` total; used = `memory.current`/`usage_in_bytes`. Restores the deploy OOM guard (deploy.go:162-185 reads GetMemoryMB). Test `info_cgroup_test.go` (4 cases) — fails on pre-fix code (ignored cgroup). GOOS=linux build+vet OK. diff --git a/controller/README.md b/controller/README.md index 3ecc0d2..371dfd8 100644 --- a/controller/README.md +++ b/controller/README.md @@ -160,12 +160,16 @@ The app catalog lives in a separate Git repository. The controller: - User-configurable inputs (admin password, language, storage path) remain editable - Section header prompts the user to note down any passwords they need 3. `checkBeforeDeploy()` JS guard fetches live state first (prevents double-deploy from another tab) -4. **Memory validation** uses real system memory from `/proc/meminfo`: - - `usable_memory = total_ram - reserved_memory_mb` (default 384MB reserved) - - `system.GetMemoryMB()` returns real-time total and used memory (not declared reservations) - - Hard block if `used_mb + new_request > usable_memory` - - `CommittedMemory()` (declared sum) still used for soft overcommit warning only - - Deploy page shows real memory usage bar (not declared reservations) +4. **Memory validation** (F1, v0.61.0): the controller runs as a Docker container inside an LXC, where + `/proc/meminfo` shows the **Proxmox host's** RAM (no lxcfs in the container) and the container's own + cgroup is unlimited (the guest cap lives on the LXC ancestor). So the guest cap is read from the + **Docker daemon** (`system.GuestMemTotalMB()` → `docker info` MemTotal — the daemon runs in the LXC and + reports the guest's lxcfs-backed RAM; the cgroup limit is preferred when present, e.g. non-nested): + - `usable_memory = guest_cap - reserved_memory_mb` (default 384MB reserved) + - Hard block if `committed_used + new_request > usable_memory`, where `committed_used` = + `CommittedMemory()` (sum of running apps' mem requests) — the guest-wide RSS is not observable from + the container, so the controller's own committed accounting is the accurate, cheap "used". + - `/api/system/info` reports the guest cap as total and committed memory as used. 4b. **Docker-data volume reserved-buffer gate (v0.58.0, storage-split prevention layer):** the OS rootfs and Docker data are split onto separate volumes; infra (controller/traefik/cloudflared/filebrowser) shares the one Docker data-root (`/var/lib/docker`) and is protected by **prevention, not placement**. @@ -265,6 +269,12 @@ When app templates are updated (e.g., a new `APP_KEY` secret is added to `.felho | Restarting | Yellow | "Ujrainditas..." | Restart loop | | Not deployed | Gray | "Nincs telepitve" | Compose file exists, not deployed | +**Route-unpublished indicator (F5, v0.61.0).** Traefik's Docker provider only publishes a route to a +container that is healthy (or has no healthcheck), so an `unhealthy`/`restarting` deployed app returns a +hard **404** at its URL even though the container is running. The `routeUnpublished` template helper +(`funcmap.go`) drives a distinct "URL nem elérhető – útvonal nincs publikálva" indicator on the dashboard +and stacks cards for such apps, so a dead URL isn't mistaken for a merely-degraded-but-reachable one. + #### Controller-side Health Probes (`internal/stacks/healthprobe.go`) For apps that declare a `healthcheck:` section in `.felhom.yml`, the controller probes the container directly over the Docker network (both are on `traefik-public`). This complements Docker-level healthchecks and is the **only** health mechanism for distroless/scratch images that lack shell utilities. @@ -404,6 +414,15 @@ backups/primary// env comes from `StackDataProvider.GetStackRecoveryInfo` (excludes secret-named + encrypted values, so the capture never touches a secret). `data_key` fields are marked in `.felhom.yml` (`DeployField.DataKey`). +- **Restore replays the DB dump (F17, v0.61.0).** `RestoreFromRecoveryUnit` (and the `RestoreApp` + fallback) stops the app → restores named-volume tars → recreates the compose definition + redeploys + with the recovered env → **replays each `db-dumps/*.sql` into the now-running DB** via + `backup.reimportDBDumps` → `appbackup.ImportDump` (psql / mariadb client, using the live container's own + discovered credentials). The DB replay runs AFTER the volume restore, so the **logical SQL dump wins** + over any volume-tar copy of the database (the dumps use DROP/CREATE — `pg_dump --clean --if-exists`, + `mariadb-dump` default `--add-drop-table` — so replay is idempotent). Volume-restore and DB-import + failures now **surface** (restore returns an error) instead of a swallowed WARN. Prior to v0.61.0 the + per-app restore never replayed the `.sql`, so DB-resident data did not come back. #### Tier 2 — off-drive copy (Phase 3, v0.55.x)