diff --git a/CONTEXT.md b/CONTEXT.md index 3ba67bf..9e304a7 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -13,6 +13,17 @@ Last updated: 2026-06-12 (storage UX polish) > is tracked in `CHANGELOG.md`, `controller/README.md`, and the auto-memory `MEMORY.md`. Live version: > **v0.45.0**. > +> **2026-06-13 — v0.58.0 OS/Docker-data split prevention layer (Phase 2; Phase 1 = agent v0.29.0):** +> - OS rootfs + Docker data split onto separate local-lvm volumes (golden bakes 32G rootfs + 256G +> /var/lib/docker, backup=1, **overlay2** so images live on the data vol). Infra protected by +> PREVENTION not placement: `system.GetDockerVolumeHeadroom()` (statfs "/" = data vol) reserves +> max(5GB,10%); `deployStack` refuses HTTP 507 below buffer; deploy.html banner+disable; monitor +> (warn80/crit90) watches the same vol; log rotation baked into the golden daemon.json. +> - Live-validated by DESTROYING + RE-PROVISIONING 9201 from the split golden: split layout, overlay2, +> images on data vol, lean rootfs, OS isolation (data vol 100% → rootfs healthy), deploy gate 507, +> ActualBudget volume on data vol, hub config pull, external access via CF. RomM/USB re-enroll = the +> documented final restore step (RomM data safe on host USB). See memory [[os-data-split]]. +> > **2026-06-13 — v0.57.0 UI fixes (Part A of the UI-fixes/storage-spike spec):** > - A1: fixed the RIGHT storage list — `#host-storage-bars` (the JS-filled, agent-PVE-storage list: > `local`/`local-lvm`/`felhom-pbs`/`felhom-usb`), which reordered on every poll. Now diff --git a/REPORT.md b/REPORT.md index 22564f9..34edcc5 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,63 +1,56 @@ -# REPORT — felhom-controller v0.57.0 (UI fixes: stable host-storage list + Tier-2 config panel) +# REPORT — felhom-controller v0.58.0 (infra-protection prevention layer for the OS/Docker-data split) -Part A of the "UI fixes (Part A) + storage-restructure spike (Part B)" spec. Part A ships a single -controller version bump (v0.57.0); Part B builds nothing (findings report — see below). Built, -deployed, and live-validated on guest 9201. +Phase 2 of the OS/Docker-data storage-split slice (Phase 1 = felhom-agent v0.29.0: golden + provision). +The controller guest's OS rootfs and Docker data are now split onto separate `local-lvm` volumes for +resilience; infra (controller/traefik/cloudflared/filebrowser) shares the one Docker data-root and is +protected by **prevention, not placement**. Built, deployed, and **live-validated on a freshly +re-provisioned guest 9201**. -## A1 — host-storage list no longer reorders (item 2) -The monitoring page has two storage sections. v0.56.0's 4C sorted the **server-rendered, user-data-only** -list (`buildStorageBars`). The list the customer actually saw reordering was the **other** one: -`#host-storage-bars`, filled client-side from the agent's PVE-storage list (`local`, `local-lvm`, -`felhom-pbs`, `felhom-usb` with thin-pool % + temperature). The agent enumerates `pvesm` in a -non-deterministic order and this list never passed through a Go sort, so it reshuffled on every 8 s poll. +## What shipped (v0.58.0) +- **Reserved-buffer headroom guard** (`internal/system/dockervol.go`): `GetDockerVolumeHeadroom()` + measures the Docker-data volume via `statfs("/")` — the controller container's root overlay is the + upperdir on the guest's `/var/lib/docker` volume (true with the **overlay2** driver; see the agent + report), so `/` reports the data volume. Reserve floor `DockerVolumeReserveGB = max(5 GB, 10%)`. + Fail-open on a measurement error. +- **Deploy-time hard gate** (`internal/api/router.go` `deployStack`): a new deploy is **refused (HTTP + 507** + Hungarian message) when free space on the Docker-data volume is at/under the reserved buffer. +- **Deploy-page surfacing** (`deploy.html`): a new deploy below the buffer shows a Hungarian warning and + **disables** the "Telepítés indítása" button; the API gate is the hard backstop. +- **Runtime monitoring** (`monitor/healthcheck.go`): confirmed `DiskPercent` watches the Docker-data + volume (statfs `/`); warn 80% / crit 90% trip ABOVE the 10%-free buffer, so the customer is warned + before the gate engages. Clarifying comment added. +- **Log rotation** baked into the golden's `daemon.json` (agent side; `max-size 10m`, `max-file 3`). +- Tests: `DockerVolumeReserveGB` floor/scale. -Fix (`internal/web/agent_host_metrics_handler.go`): `enrichHostStorageTargets` sorts the -`/api/host-metrics` response **server-side** — user-data (`usb`/`local-dir`) → system+apps -(`lvmthin`/`lvm`) → builtin local (`local`) → backup (`pbs`/`nfs`/`cifs`) → other; alphabetical by id -within a tier — and attaches a **friendly Hungarian label + one-line purpose** per entry. The raw PVE id -stays in `Name` and is rendered muted in `monitoring.html`. **Display labels only — PVE storage ids are -never renamed** (vzdump/PBS targets reference them by name). Decision flagged for the owner: a follow-up -could replace the raw breakdown with customer-meaningful aggregates and push the raw `local`/`local-lvm`/ -`pbs` detail to an operator (hub) view; this pass takes the smallest change that fixes the confusion -(friendly labels + descriptions on the existing list). +## Live validation (guest 9201, freshly re-provisioned from the split golden) +9201 was **destroyed and re-provisioned** from the new split golden (32 GB OS rootfs + 256 GB Docker-data +volume, `backup=1`), via `felhom-agent --selftest=provision` + a reboot. The controller bootstrapped +from baked images (no pull), **pulled its config from the hub** (catalog synced — 55 app defs — hub +HTTP 200, CF token configured, hub report pushed). Then: -## A2 — per-app Tier-2 config panel (item 4) -The "2. mentés" row's **Beállítás** button linked to the app deploy page (no backup-location setting — a -dead end). New surface: `GET/POST /stacks/{name}/backup` (`tier2_config_handler.go` + -`templates/tier2_config.html`), wired in `server.go` behind RequireAuth + CsrfProtect; the button is -repointed there on **every** "2. mentés" branch (configured / disconnected / inactive / unconfigured / -disabled). The panel shows the current/effective off-drive target, whether it's the size-limited internal -SSD, the last-run reason, and lets the customer **pin a different registered drive** or **turn Tier 2 -off**. It is **always visible**: with only the internal SSD it shows "automatikus: belső SSD — csak -DB/konfiguráció" + the rootfs-headroom note; for a non-HDD app it shows honest "already in the PBS -whole-guest snapshot; the off-drive copy is supplementary" context (no active control, since Tier 2 does -not run for rootfs apps). +- **Split layout:** controller image 0.58.0, **Storage Driver overlay2**, `Docker Root Dir + /var/lib/docker`; images on the data volume (`/var/lib/docker/overlay2` 1.7 GB), `/var/lib/containerd` + idle (380 K); `df`: `/` 935 MB/32 GB (4%, lean OS rootfs), `/var/lib/docker` 256 GB. +- **Prevention gate (the headline):** with ample space the deploy page shows **no** gate banner; after + `fallocate`-filling the data volume to 99% (3.2 GB free < 25.6 GB reserve), a `POST /api/stacks/.../deploy` + returned **HTTP 507** with the Hungarian "Nincs elég szabad tárhely" message — proven on the real + 256 GB data volume. +- **Regression:** `/`, `/stacks`, `/backups`, `/monitoring`, `/stacks/{n}/deploy`, `/stacks/{n}/backup` + all HTTP 200; A1 host-storage list still ordered + friendly-labelled (felhom-usb → local-lvm → local + → felhom-pbs); A2 Tier-2 panel route serves. +- **Deploy path + DB-on-data-volume (step 3):** deployed ActualBudget (HTTP 200, container up); its named + volume landed at `/var/lib/docker/volumes/actualbudget_actualbudget_data` = the data volume. +- **External access:** via Cloudflare the controller returns HTTP 200 for vmid 9201 (tunnel + traefik + route healthy). (A local-DNS override on the dev machine points the hostname at a stale LAN IP — a + red herring; the real public path works.) -Persistence: two preference fields on `settings.CrossDriveBackup` — `UserDisabled` + `PreferredTarget` -— set via `SetTier2Preference` and **preserved across the Tier-2 runner's status writes** -(`withTier2Prefs` in `tier2.go`). `selectTier2Target` now honors a valid pinned target (registered, -schedulable, off physical disk) before the auto-pick; an invalid pin silently falls back to auto. -`RunTier2` skips a customer-disabled app. Saving with Tier 2 on for an HDD app triggers an immediate run. +## OS isolation (resilience — the reason for the split), proven on the provisioned guest +Filling the Docker-data volume to 100% (239 GB) left the OS rootfs at 4% and fully writable, the guest +healthy throughout — the data volume cannot starve the OS. -## Live validation (guest 9201, v0.57.0, public URL https://felhom.demo-felhom.eu) -- **A1:** `/api/host-metrics` returns a stable order across repeated polls — `felhom-usb` → - `local-lvm` → `local` → `felhom-pbs` — each entry carrying `label` + `purpose` (e.g. `local-lvm` → - "Belső SSD – rendszer és alkalmazások"). Confirmed 3 consecutive polls identical. -- **A2 (HDD app, RomM):** panel shows Tier 2 **Bekapcsolva**, effective target "belső SSD (rendszer) — - csak DB/konfiguráció", the SSD-only banner, the "Nincs másik adatmeghajtó" help (single-drive demo), - and the enabled checkbox + target dropdown. POST disable → 303 + flash → GET shows **Kikapcsolva**; - POST re-enable → GET shows **Bekapcsolva** and the controller log records an immediate successful - Tier 2 run (RomM → SSD, 77 KB, DB/config only). -- **A2 (non-HDD app, ActualBudget):** panel shows the PBS-coverage context ("már szerepelnek a teljes - rendszermentésben (PBS) … nincs külön teendő"), no active form. -- Both `/backups` "Beállítás" buttons now target `/stacks/{name}/backup`; logs clean (no errors/panics). - -## Tests -`enrichHostStorageTargets` (order, labels, determinism, unknown-type fallback); -`selectTier2Target` (honors a pin / falls back on an invalid pin); status writes preserve the -preference. Full `go test ./...` green. - -## Part B — storage-restructure spike -Build-nothing investigation; the findings report gating the provisioning spec is at -`felhom-agent/REPORT-storage-split-spike.md` (and summarised in that repo's CHANGELOG). Nothing in the -controller changed for Part B. +## Outstanding (demo restoration, not slice validation) +- **RomM (HDD app) + USB re-enroll:** RomM's data is safe on the host USB (`/mnt/felhom-usb/felhom-data`, + untouched by the re-provision). Restoring it is the slice-10 enroll flow (assign → guest-attach → + reboot to activate the bind → register storage → deploy). With the split, the USB binds to a free slot + (mp1+) since **mp0 is now the Docker-data volume** — no collision. Documented as the final restore step; + not required for slice validation (ActualBudget covered the deploy path; the USB bind was not touched). diff --git a/controller/README.md b/controller/README.md index 76d387f..be2485d 100644 --- a/controller/README.md +++ b/controller/README.md @@ -149,6 +149,16 @@ The app catalog lives in a separate Git repository. The controller: - Hard block if `used_mb + new_request > usable_memory` - `CommittedMemory()` (declared sum) still used for soft overcommit warning only - Deploy page shows real memory usage bar (not declared reservations) +4b. **Docker-data volume reserved-buffer gate (v0.58.0, storage-split prevention layer):** the OS rootfs + and Docker data are split onto separate volumes; infra (controller/traefik/cloudflared/filebrowser) + shares the one Docker data-root (`/var/lib/docker`) and is protected by **prevention, not placement**. + `system.GetDockerVolumeHeadroom()` measures the Docker-data volume via `statfs("/")` (the controller + container's overlay root is the upperdir on that volume — true with the golden's **overlay2** driver) + and reserves `max(5 GB, 10%)`. `deployStack` **refuses a new deploy (HTTP 507)** when free space is + at/under the buffer; the deploy page shows the warning + disables the button. Fail-open on a statfs + error. The runtime disk monitor (`healthcheck.go`, warn 80% / crit 90%) watches the same volume and + trips above the buffer. (Assumes the split guest's large data volume; the golden bakes overlay2 + log + rotation so images+volumes live on the data volume, not `/var/lib/containerd`.) 5. Pre-generated secret values are submitted as hidden form inputs so the **same values** the user saw are saved to `app.yaml` (no silent re-generation on submit). Controller saves `app.yaml`, sets in-memory `Deployed` + `Deploying` flags, then runs `docker compose up -d` **asynchronously** in a goroutine — API returns immediately so the UI switches to the progress panel without waiting for image pulls. On failure the goroutine reverts both disk and in-memory state and sets `DeployError`. 6. 3-step progress panel polls `GET /api/stacks/{name}` every 3s: config saved → `deploying` (pulling images) → containers starting → health check passed. New `StateDeploying` state shown while compose-up is in progress (no containers yet). 7. Post-deploy: locked fields (DB_PASSWORD, etc.) become read-only; the "Automatikusan generált értékek" section continues to show the saved values on the settings page