docs: v0.61.0 CHANGELOG + README (F1 mem/F17 restore/F5 route) + FIXSPEC statuses + REPORT
This commit is contained in:
@@ -1,56 +1,56 @@
|
||||
# REPORT — felhom-controller v0.58.0 (infra-protection prevention layer for the OS/Docker-data split)
|
||||
# REPORT — live-drive Batch 1 (+F17) → controller v0.61.0 (2026-06-14)
|
||||
|
||||
Phase 2 of the OS/Docker-data storage-split slice (Phase 1 = felhom-agent v0.29.0: golden + provision).
|
||||
The controller guest's OS rootfs and Docker data are now split onto separate `local-lvm` volumes for
|
||||
resilience; infra (controller/traefik/cloudflared/filebrowser) shares the one Docker data-root and is
|
||||
protected by **prevention, not placement**. Built, deployed, and **live-validated on a freshly
|
||||
re-provisioned guest 9201**.
|
||||
Implemented the controller-side fixes from `LIVE-DRIVE-FIXSPEC-2026-06-14.md` (Batch 1 + the CRITICAL
|
||||
F17), shipped as **v0.61.0** to demo guest 9201. Each fix has a regression test that fails on the
|
||||
pre-fix code; the authoritative `go build && go vet && go test ./...` is green on the Linux build server.
|
||||
|
||||
## What shipped (v0.58.0)
|
||||
- **Reserved-buffer headroom guard** (`internal/system/dockervol.go`): `GetDockerVolumeHeadroom()`
|
||||
measures the Docker-data volume via `statfs("/")` — the controller container's root overlay is the
|
||||
upperdir on the guest's `/var/lib/docker` volume (true with the **overlay2** driver; see the agent
|
||||
report), so `/` reports the data volume. Reserve floor `DockerVolumeReserveGB = max(5 GB, 10%)`.
|
||||
Fail-open on a measurement error.
|
||||
- **Deploy-time hard gate** (`internal/api/router.go` `deployStack`): a new deploy is **refused (HTTP
|
||||
507** + Hungarian message) when free space on the Docker-data volume is at/under the reserved buffer.
|
||||
- **Deploy-page surfacing** (`deploy.html`): a new deploy below the buffer shows a Hungarian warning and
|
||||
**disables** the "Telepítés indítása" button; the API gate is the hard backstop.
|
||||
- **Runtime monitoring** (`monitor/healthcheck.go`): confirmed `DiskPercent` watches the Docker-data
|
||||
volume (statfs `/`); warn 80% / crit 90% trip ABOVE the 10%-free buffer, so the customer is warned
|
||||
before the gate engages. Clarifying comment added.
|
||||
- **Log rotation** baked into the golden's `daemon.json` (agent side; `max-size 10m`, `max-file 3`).
|
||||
- Tests: `DockerVolumeReserveGB` floor/scale.
|
||||
## Fixes (each its own commit)
|
||||
|
||||
## Live validation (guest 9201, freshly re-provisioned from the split golden)
|
||||
9201 was **destroyed and re-provisioned** from the new split golden (32 GB OS rootfs + 256 GB Docker-data
|
||||
volume, `backup=1`), via `felhom-agent --selftest=provision` + a reboot. The controller bootstrapped
|
||||
from baked images (no pull), **pulled its config from the hub** (catalog synced — 55 app defs — hub
|
||||
HTTP 200, CF token configured, hub report pushed). Then:
|
||||
| Finding | Sev | Change | Test | Live-verified |
|
||||
|---|---|---|---|---|
|
||||
| **F17** | CRIT | per-app restore now **replays the captured `.sql` dump** (new `appbackup.ImportDump` + `backup.reimportDBDumps`), AFTER volume restore so the **dump wins**; volume/DB failures now surface | `restore_db_test.go` (injectable seams) | **marker DB round-trip PASSED** on romm |
|
||||
| **F1** | HIGH | guest RAM cap from `docker info` (daemon runs in LXC); deploy guard "used" = committed memory | `info_cgroup_test.go` (cgroup + docker-info paths) | `total_mem_mb=2048` (was 15771) |
|
||||
| **F20-BUG1** | HIGH | `agentapi.FormatDisk` surfaces non-2xx/`ok:false` instead of zero-value success | `disks_test.go` (502→err) | unit |
|
||||
| **F5 (catalog)** | HIGH | uptime-kuma healthcheck → v2 `extra/healthcheck` binary + 180s start_period | — (catalog) | uptime-kuma **healthy → route 302** (was 404) |
|
||||
| **F5 (dashboard)** | HIGH | `routeUnpublished` helper + distinct indicator for unhealthy/restarting deployed apps | `route_unpublished_test.go` | render test |
|
||||
| **F8** | LOW-MED | `controller.yaml` persisted 0600 (holds infra tokens) | `config_perms_test.go` | unit (Linux) |
|
||||
| **F6** | LOW | deploy POST → 202 + "Telepítés elindítva" (was "deployed" before compose) | — | code/UI |
|
||||
| **F7** | LOW | `status-refresh` 30s → 10s | — | cadence |
|
||||
| **F4** | TRIV | `GET /api/stacks/rescan` → 405 + `Allow: POST` | — | `→ 405` |
|
||||
|
||||
- **Split layout:** controller image 0.58.0, **Storage Driver overlay2**, `Docker Root Dir
|
||||
/var/lib/docker`; images on the data volume (`/var/lib/docker/overlay2` 1.7 GB), `/var/lib/containerd`
|
||||
idle (380 K); `df`: `/` 935 MB/32 GB (4%, lean OS rootfs), `/var/lib/docker` 256 GB.
|
||||
- **Prevention gate (the headline):** with ample space the deploy page shows **no** gate banner; after
|
||||
`fallocate`-filling the data volume to 99% (3.2 GB free < 25.6 GB reserve), a `POST /api/stacks/.../deploy`
|
||||
returned **HTTP 507** with the Hungarian "Nincs elég szabad tárhely" message — proven on the real
|
||||
256 GB data volume.
|
||||
- **Regression:** `/`, `/stacks`, `/backups`, `/monitoring`, `/stacks/{n}/deploy`, `/stacks/{n}/backup`
|
||||
all HTTP 200; A1 host-storage list still ordered + friendly-labelled (felhom-usb → local-lvm → local
|
||||
→ felhom-pbs); A2 Tier-2 panel route serves.
|
||||
- **Deploy path + DB-on-data-volume (step 3):** deployed ActualBudget (HTTP 200, container up); its named
|
||||
volume landed at `/var/lib/docker/volumes/actualbudget_actualbudget_data` = the data volume.
|
||||
- **External access:** via Cloudflare the controller returns HTTP 200 for vmid 9201 (tunnel + traefik
|
||||
route healthy). (A local-DNS override on the dev machine points the hostname at a stale LAN IP — a
|
||||
red herring; the real public path works.)
|
||||
## Key implementation notes
|
||||
|
||||
## OS isolation (resilience — the reason for the split), proven on the provisioned guest
|
||||
Filling the Docker-data volume to 100% (239 GB) left the OS rootfs at 4% and fully writable, the guest
|
||||
healthy throughout — the data volume cannot starve the OS.
|
||||
- **F17 reuse decision:** `ImportDump` lives in `internal/appbackup` (the DB-domain package with
|
||||
`DiscoverDatabases`/`DumpOne`/`getMariaDBPassword`), NOT reusing `appexport`'s unexported copies —
|
||||
`appexport→appbackup` already exists, so the reverse would import-cycle. `DiscoveredDB` carries the live
|
||||
container's own creds, so no env threading. Precedence: replay runs after volume restore → `.sql` wins.
|
||||
- **F1 reroot:** the FIXSPEC's cgroup-limit approach was a **no-op** on the demo (the container's own
|
||||
cgroup is unlimited; the 2 GB cap is on the LXC ancestor, hidden; no lxcfs in the container). The
|
||||
working source is `docker info` MemTotal. The deploy guard's "used" switched from host RSS (not
|
||||
per-guest observable) to the controller's committed-memory accounting.
|
||||
|
||||
## Outstanding (demo restoration, not slice validation)
|
||||
- **RomM (HDD app) + USB re-enroll:** RomM's data is safe on the host USB (`/mnt/felhom-usb/felhom-data`,
|
||||
untouched by the re-provision). Restoring it is the slice-10 enroll flow (assign → guest-attach →
|
||||
reboot to activate the bind → register storage → deploy). With the split, the USB binds to a free slot
|
||||
(mp1+) since **mp0 is now the Docker-data volume** — no collision. Documented as the final restore step;
|
||||
not required for slice validation (ActualBudget covered the deploy path; the USB bind was not touched).
|
||||
## Deploy & verification
|
||||
|
||||
- Built+pushed `gitea.dooplex.hu/admin/felhom-controller:0.61.0`; deployed via the guest-9201
|
||||
golden/bootstrap mechanism (`/etc/felhom-controller-image` + restart bootstrap). Healthy, `/api/health`
|
||||
+ dashboard 200, no startup errors.
|
||||
- Live-verified: F1 (`total_mem_mb=2048`), F4 (405), F5 (uptime-kuma healthy → route 302), **F17 (marker
|
||||
row dropped after backup was restored by `/backup/restore`)**.
|
||||
|
||||
## Test approach
|
||||
|
||||
Controller code is largely `//go:build linux`; dev host is Windows. Per-commit gate =
|
||||
`CGO_ENABLED=0 GOOS=linux go build && go vet` locally (compiles code + tests); authoritative
|
||||
`go test ./...` ran on the Linux build server (all green, incl. the linux `system` tests).
|
||||
|
||||
## Not in this batch (SUPERVISED, next session — agent/golden)
|
||||
|
||||
**F9** (HDD provisioning/guest-attach + reporting), **F20-BUG2** (durable_id scheme), **F20-BUG3**
|
||||
(async mkfs). These touch the host agent, golden template, and bring-up (blast radius beyond the
|
||||
disposable demo).
|
||||
|
||||
## Final state
|
||||
|
||||
Demo guest 9201 on `:0.61.0`, healthy; agent unchanged. Apps: actualbudget, gokapi, mealie, privatebin,
|
||||
romm(+db,+redis), uptime-kuma (now **healthy**), vikunja — all healthy. felhom-usb still not attached to
|
||||
the guest (F9, deferred).
|
||||
|
||||
Reference in New Issue
Block a user