docs: v0.61.0 CHANGELOG + README (F1 mem/F17 restore/F5 route) + FIXSPEC statuses + REPORT

This commit is contained in:
2026-06-14 10:25:00 +02:00
parent 4989513a96
commit 84979932e7
5 changed files with 148 additions and 56 deletions
+50 -50
View File
@@ -1,56 +1,56 @@
# REPORT — felhom-controller v0.58.0 (infra-protection prevention layer for the OS/Docker-data split)
# REPORT — live-drive Batch 1 (+F17) → controller v0.61.0 (2026-06-14)
Phase 2 of the OS/Docker-data storage-split slice (Phase 1 = felhom-agent v0.29.0: golden + provision).
The controller guest's OS rootfs and Docker data are now split onto separate `local-lvm` volumes for
resilience; infra (controller/traefik/cloudflared/filebrowser) shares the one Docker data-root and is
protected by **prevention, not placement**. Built, deployed, and **live-validated on a freshly
re-provisioned guest 9201**.
Implemented the controller-side fixes from `LIVE-DRIVE-FIXSPEC-2026-06-14.md` (Batch 1 + the CRITICAL
F17), shipped as **v0.61.0** to demo guest 9201. Each fix has a regression test that fails on the
pre-fix code; the authoritative `go build && go vet && go test ./...` is green on the Linux build server.
## What shipped (v0.58.0)
- **Reserved-buffer headroom guard** (`internal/system/dockervol.go`): `GetDockerVolumeHeadroom()`
measures the Docker-data volume via `statfs("/")` — the controller container's root overlay is the
upperdir on the guest's `/var/lib/docker` volume (true with the **overlay2** driver; see the agent
report), so `/` reports the data volume. Reserve floor `DockerVolumeReserveGB = max(5 GB, 10%)`.
Fail-open on a measurement error.
- **Deploy-time hard gate** (`internal/api/router.go` `deployStack`): a new deploy is **refused (HTTP
507** + Hungarian message) when free space on the Docker-data volume is at/under the reserved buffer.
- **Deploy-page surfacing** (`deploy.html`): a new deploy below the buffer shows a Hungarian warning and
**disables** the "Telepítés indítása" button; the API gate is the hard backstop.
- **Runtime monitoring** (`monitor/healthcheck.go`): confirmed `DiskPercent` watches the Docker-data
volume (statfs `/`); warn 80% / crit 90% trip ABOVE the 10%-free buffer, so the customer is warned
before the gate engages. Clarifying comment added.
- **Log rotation** baked into the golden's `daemon.json` (agent side; `max-size 10m`, `max-file 3`).
- Tests: `DockerVolumeReserveGB` floor/scale.
## Fixes (each its own commit)
## Live validation (guest 9201, freshly re-provisioned from the split golden)
9201 was **destroyed and re-provisioned** from the new split golden (32 GB OS rootfs + 256 GB Docker-data
volume, `backup=1`), via `felhom-agent --selftest=provision` + a reboot. The controller bootstrapped
from baked images (no pull), **pulled its config from the hub** (catalog synced — 55 app defs — hub
HTTP 200, CF token configured, hub report pushed). Then:
| Finding | Sev | Change | Test | Live-verified |
|---|---|---|---|---|
| **F17** | CRIT | per-app restore now **replays the captured `.sql` dump** (new `appbackup.ImportDump` + `backup.reimportDBDumps`), AFTER volume restore so the **dump wins**; volume/DB failures now surface | `restore_db_test.go` (injectable seams) | **marker DB round-trip PASSED** on romm |
| **F1** | HIGH | guest RAM cap from `docker info` (daemon runs in LXC); deploy guard "used" = committed memory | `info_cgroup_test.go` (cgroup + docker-info paths) | `total_mem_mb=2048` (was 15771) |
| **F20-BUG1** | HIGH | `agentapi.FormatDisk` surfaces non-2xx/`ok:false` instead of zero-value success | `disks_test.go` (502→err) | unit |
| **F5 (catalog)** | HIGH | uptime-kuma healthcheck → v2 `extra/healthcheck` binary + 180s start_period | — (catalog) | uptime-kuma **healthy → route 302** (was 404) |
| **F5 (dashboard)** | HIGH | `routeUnpublished` helper + distinct indicator for unhealthy/restarting deployed apps | `route_unpublished_test.go` | render test |
| **F8** | LOW-MED | `controller.yaml` persisted 0600 (holds infra tokens) | `config_perms_test.go` | unit (Linux) |
| **F6** | LOW | deploy POST → 202 + "Telepítés elindítva" (was "deployed" before compose) | — | code/UI |
| **F7** | LOW | `status-refresh` 30s → 10s | — | cadence |
| **F4** | TRIV | `GET /api/stacks/rescan` → 405 + `Allow: POST` | — | `→ 405` |
- **Split layout:** controller image 0.58.0, **Storage Driver overlay2**, `Docker Root Dir
/var/lib/docker`; images on the data volume (`/var/lib/docker/overlay2` 1.7 GB), `/var/lib/containerd`
idle (380 K); `df`: `/` 935 MB/32 GB (4%, lean OS rootfs), `/var/lib/docker` 256 GB.
- **Prevention gate (the headline):** with ample space the deploy page shows **no** gate banner; after
`fallocate`-filling the data volume to 99% (3.2 GB free < 25.6 GB reserve), a `POST /api/stacks/.../deploy`
returned **HTTP 507** with the Hungarian "Nincs elég szabad tárhely" message — proven on the real
256 GB data volume.
- **Regression:** `/`, `/stacks`, `/backups`, `/monitoring`, `/stacks/{n}/deploy`, `/stacks/{n}/backup`
all HTTP 200; A1 host-storage list still ordered + friendly-labelled (felhom-usb → local-lvm → local
→ felhom-pbs); A2 Tier-2 panel route serves.
- **Deploy path + DB-on-data-volume (step 3):** deployed ActualBudget (HTTP 200, container up); its named
volume landed at `/var/lib/docker/volumes/actualbudget_actualbudget_data` = the data volume.
- **External access:** via Cloudflare the controller returns HTTP 200 for vmid 9201 (tunnel + traefik
route healthy). (A local-DNS override on the dev machine points the hostname at a stale LAN IP — a
red herring; the real public path works.)
## Key implementation notes
## OS isolation (resilience — the reason for the split), proven on the provisioned guest
Filling the Docker-data volume to 100% (239 GB) left the OS rootfs at 4% and fully writable, the guest
healthy throughout — the data volume cannot starve the OS.
- **F17 reuse decision:** `ImportDump` lives in `internal/appbackup` (the DB-domain package with
`DiscoverDatabases`/`DumpOne`/`getMariaDBPassword`), NOT reusing `appexport`'s unexported copies —
`appexport→appbackup` already exists, so the reverse would import-cycle. `DiscoveredDB` carries the live
container's own creds, so no env threading. Precedence: replay runs after volume restore → `.sql` wins.
- **F1 reroot:** the FIXSPEC's cgroup-limit approach was a **no-op** on the demo (the container's own
cgroup is unlimited; the 2 GB cap is on the LXC ancestor, hidden; no lxcfs in the container). The
working source is `docker info` MemTotal. The deploy guard's "used" switched from host RSS (not
per-guest observable) to the controller's committed-memory accounting.
## Outstanding (demo restoration, not slice validation)
- **RomM (HDD app) + USB re-enroll:** RomM's data is safe on the host USB (`/mnt/felhom-usb/felhom-data`,
untouched by the re-provision). Restoring it is the slice-10 enroll flow (assign → guest-attach →
reboot to activate the bind → register storage → deploy). With the split, the USB binds to a free slot
(mp1+) since **mp0 is now the Docker-data volume** — no collision. Documented as the final restore step;
not required for slice validation (ActualBudget covered the deploy path; the USB bind was not touched).
## Deploy & verification
- Built+pushed `gitea.dooplex.hu/admin/felhom-controller:0.61.0`; deployed via the guest-9201
golden/bootstrap mechanism (`/etc/felhom-controller-image` + restart bootstrap). Healthy, `/api/health`
+ dashboard 200, no startup errors.
- Live-verified: F1 (`total_mem_mb=2048`), F4 (405), F5 (uptime-kuma healthy → route 302), **F17 (marker
row dropped after backup was restored by `/backup/restore`)**.
## Test approach
Controller code is largely `//go:build linux`; dev host is Windows. Per-commit gate =
`CGO_ENABLED=0 GOOS=linux go build && go vet` locally (compiles code + tests); authoritative
`go test ./...` ran on the Linux build server (all green, incl. the linux `system` tests).
## Not in this batch (SUPERVISED, next session — agent/golden)
**F9** (HDD provisioning/guest-attach + reporting), **F20-BUG2** (durable_id scheme), **F20-BUG3**
(async mkfs). These touch the host agent, golden template, and bring-up (blast radius beyond the
disposable demo).
## Final state
Demo guest 9201 on `:0.61.0`, healthy; agent unchanged. Apps: actualbudget, gokapi, mealie, privatebin,
romm(+db,+redis), uptime-kuma (now **healthy**), vikunja — all healthy. felhom-usb still not attached to
the guest (F9, deferred).