docs: REPORT/README/CONTEXT for v0.58.0 prevention layer (9201 re-provision validated)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

CONTEXT.md

@@ -13,6 +13,17 @@ Last updated: 2026-06-12 (storage UX polish)
 > is tracked in `CHANGELOG.md`, `controller/README.md`, and the auto-memory `MEMORY.md`. Live version:
 > **v0.45.0**.
 >
+> **2026-06-13 — v0.58.0 OS/Docker-data split prevention layer (Phase 2; Phase 1 = agent v0.29.0):**
+> - OS rootfs + Docker data split onto separate local-lvm volumes (golden bakes 32G rootfs + 256G
+>   /var/lib/docker, backup=1, **overlay2** so images live on the data vol). Infra protected by
+>   PREVENTION not placement: `system.GetDockerVolumeHeadroom()` (statfs "/" = data vol) reserves
+>   max(5GB,10%); `deployStack` refuses HTTP 507 below buffer; deploy.html banner+disable; monitor
+>   (warn80/crit90) watches the same vol; log rotation baked into the golden daemon.json.
+> - Live-validated by DESTROYING + RE-PROVISIONING 9201 from the split golden: split layout, overlay2,
+>   images on data vol, lean rootfs, OS isolation (data vol 100% → rootfs healthy), deploy gate 507,
+>   ActualBudget volume on data vol, hub config pull, external access via CF. RomM/USB re-enroll = the
+>   documented final restore step (RomM data safe on host USB). See memory [[os-data-split]].
+>
 > **2026-06-13 — v0.57.0 UI fixes (Part A of the UI-fixes/storage-spike spec):**
 > - A1: fixed the RIGHT storage list — `#host-storage-bars` (the JS-filled, agent-PVE-storage list:
 >   `local`/`local-lvm`/`felhom-pbs`/`felhom-usb`), which reordered on every poll. Now

REPORT.md

@@ -1,63 +1,56 @@
-# REPORT — felhom-controller v0.57.0 (UI fixes: stable host-storage list + Tier-2 config panel)
+# REPORT — felhom-controller v0.58.0 (infra-protection prevention layer for the OS/Docker-data split)
 
-Part A of the "UI fixes (Part A) + storage-restructure spike (Part B)" spec. Part A ships a single
-controller version bump (v0.57.0); Part B builds nothing (findings report — see below). Built,
-deployed, and live-validated on guest 9201.
+Phase 2 of the OS/Docker-data storage-split slice (Phase 1 = felhom-agent v0.29.0: golden + provision).
+The controller guest's OS rootfs and Docker data are now split onto separate `local-lvm` volumes for
+resilience; infra (controller/traefik/cloudflared/filebrowser) shares the one Docker data-root and is
+protected by **prevention, not placement**. Built, deployed, and **live-validated on a freshly
+re-provisioned guest 9201**.
 
-## A1 — host-storage list no longer reorders (item 2)
-The monitoring page has two storage sections. v0.56.0's 4C sorted the **server-rendered, user-data-only**
-list (`buildStorageBars`). The list the customer actually saw reordering was the **other** one:
-`#host-storage-bars`, filled client-side from the agent's PVE-storage list (`local`, `local-lvm`,
-`felhom-pbs`, `felhom-usb` with thin-pool % + temperature). The agent enumerates `pvesm` in a
-non-deterministic order and this list never passed through a Go sort, so it reshuffled on every 8 s poll.
+## What shipped (v0.58.0)
+- **Reserved-buffer headroom guard** (`internal/system/dockervol.go`): `GetDockerVolumeHeadroom()`
+  measures the Docker-data volume via `statfs("/")` — the controller container's root overlay is the
+  upperdir on the guest's `/var/lib/docker` volume (true with the **overlay2** driver; see the agent
+  report), so `/` reports the data volume. Reserve floor `DockerVolumeReserveGB = max(5 GB, 10%)`.
+  Fail-open on a measurement error.
+- **Deploy-time hard gate** (`internal/api/router.go` `deployStack`): a new deploy is **refused (HTTP
+  507** + Hungarian message) when free space on the Docker-data volume is at/under the reserved buffer.
+- **Deploy-page surfacing** (`deploy.html`): a new deploy below the buffer shows a Hungarian warning and
+  **disables** the "Telepítés indítása" button; the API gate is the hard backstop.
+- **Runtime monitoring** (`monitor/healthcheck.go`): confirmed `DiskPercent` watches the Docker-data
+  volume (statfs `/`); warn 80% / crit 90% trip ABOVE the 10%-free buffer, so the customer is warned
+  before the gate engages. Clarifying comment added.
+- **Log rotation** baked into the golden's `daemon.json` (agent side; `max-size 10m`, `max-file 3`).
+- Tests: `DockerVolumeReserveGB` floor/scale.
 
-Fix (`internal/web/agent_host_metrics_handler.go`): `enrichHostStorageTargets` sorts the
-`/api/host-metrics` response **server-side** — user-data (`usb`/`local-dir`) → system+apps
-(`lvmthin`/`lvm`) → builtin local (`local`) → backup (`pbs`/`nfs`/`cifs`) → other; alphabetical by id
-within a tier — and attaches a **friendly Hungarian label + one-line purpose** per entry. The raw PVE id
-stays in `Name` and is rendered muted in `monitoring.html`. **Display labels only — PVE storage ids are
-never renamed** (vzdump/PBS targets reference them by name). Decision flagged for the owner: a follow-up
-could replace the raw breakdown with customer-meaningful aggregates and push the raw `local`/`local-lvm`/
-`pbs` detail to an operator (hub) view; this pass takes the smallest change that fixes the confusion
-(friendly labels + descriptions on the existing list).
+## Live validation (guest 9201, freshly re-provisioned from the split golden)
+9201 was **destroyed and re-provisioned** from the new split golden (32 GB OS rootfs + 256 GB Docker-data
+volume, `backup=1`), via `felhom-agent --selftest=provision` + a reboot. The controller bootstrapped
+from baked images (no pull), **pulled its config from the hub** (catalog synced — 55 app defs — hub
+HTTP 200, CF token configured, hub report pushed). Then:
 
-## A2 — per-app Tier-2 config panel (item 4)
-The "2. mentés" row's **Beállítás** button linked to the app deploy page (no backup-location setting — a
-dead end). New surface: `GET/POST /stacks/{name}/backup` (`tier2_config_handler.go` +
-`templates/tier2_config.html`), wired in `server.go` behind RequireAuth + CsrfProtect; the button is
-repointed there on **every** "2. mentés" branch (configured / disconnected / inactive / unconfigured /
-disabled). The panel shows the current/effective off-drive target, whether it's the size-limited internal
-SSD, the last-run reason, and lets the customer **pin a different registered drive** or **turn Tier 2
-off**. It is **always visible**: with only the internal SSD it shows "automatikus: belső SSD — csak
-DB/konfiguráció" + the rootfs-headroom note; for a non-HDD app it shows honest "already in the PBS
-whole-guest snapshot; the off-drive copy is supplementary" context (no active control, since Tier 2 does
-not run for rootfs apps).
+- **Split layout:** controller image 0.58.0, **Storage Driver overlay2**, `Docker Root Dir
+  /var/lib/docker`; images on the data volume (`/var/lib/docker/overlay2` 1.7 GB), `/var/lib/containerd`
+  idle (380 K); `df`: `/` 935 MB/32 GB (4%, lean OS rootfs), `/var/lib/docker` 256 GB.
+- **Prevention gate (the headline):** with ample space the deploy page shows **no** gate banner; after
+  `fallocate`-filling the data volume to 99% (3.2 GB free < 25.6 GB reserve), a `POST /api/stacks/.../deploy`
+  returned **HTTP 507** with the Hungarian "Nincs elég szabad tárhely" message — proven on the real
+  256 GB data volume.
+- **Regression:** `/`, `/stacks`, `/backups`, `/monitoring`, `/stacks/{n}/deploy`, `/stacks/{n}/backup`
+  all HTTP 200; A1 host-storage list still ordered + friendly-labelled (felhom-usb → local-lvm → local
+  → felhom-pbs); A2 Tier-2 panel route serves.
+- **Deploy path + DB-on-data-volume (step 3):** deployed ActualBudget (HTTP 200, container up); its named
+  volume landed at `/var/lib/docker/volumes/actualbudget_actualbudget_data` = the data volume.
+- **External access:** via Cloudflare the controller returns HTTP 200 for vmid 9201 (tunnel + traefik
+  route healthy). (A local-DNS override on the dev machine points the hostname at a stale LAN IP — a
+  red herring; the real public path works.)
 
-Persistence: two preference fields on `settings.CrossDriveBackup` — `UserDisabled` + `PreferredTarget`
-— set via `SetTier2Preference` and **preserved across the Tier-2 runner's status writes**
-(`withTier2Prefs` in `tier2.go`). `selectTier2Target` now honors a valid pinned target (registered,
-schedulable, off physical disk) before the auto-pick; an invalid pin silently falls back to auto.
-`RunTier2` skips a customer-disabled app. Saving with Tier 2 on for an HDD app triggers an immediate run.
+## OS isolation (resilience — the reason for the split), proven on the provisioned guest
+Filling the Docker-data volume to 100% (239 GB) left the OS rootfs at 4% and fully writable, the guest
+healthy throughout — the data volume cannot starve the OS.
 
-## Live validation (guest 9201, v0.57.0, public URL https://felhom.demo-felhom.eu)
-- **A1:** `/api/host-metrics` returns a stable order across repeated polls — `felhom-usb` →
-  `local-lvm` → `local` → `felhom-pbs` — each entry carrying `label` + `purpose` (e.g. `local-lvm` →
-  "Belső SSD – rendszer és alkalmazások"). Confirmed 3 consecutive polls identical.
-- **A2 (HDD app, RomM):** panel shows Tier 2 **Bekapcsolva**, effective target "belső SSD (rendszer) —
-  csak DB/konfiguráció", the SSD-only banner, the "Nincs másik adatmeghajtó" help (single-drive demo),
-  and the enabled checkbox + target dropdown. POST disable → 303 + flash → GET shows **Kikapcsolva**;
-  POST re-enable → GET shows **Bekapcsolva** and the controller log records an immediate successful
-  Tier 2 run (RomM → SSD, 77 KB, DB/config only).
-- **A2 (non-HDD app, ActualBudget):** panel shows the PBS-coverage context ("már szerepelnek a teljes
-  rendszermentésben (PBS) … nincs külön teendő"), no active form.
-- Both `/backups` "Beállítás" buttons now target `/stacks/{name}/backup`; logs clean (no errors/panics).
-
-## Tests
-`enrichHostStorageTargets` (order, labels, determinism, unknown-type fallback);
-`selectTier2Target` (honors a pin / falls back on an invalid pin); status writes preserve the
-preference. Full `go test ./...` green.
-
-## Part B — storage-restructure spike
-Build-nothing investigation; the findings report gating the provisioning spec is at
-`felhom-agent/REPORT-storage-split-spike.md` (and summarised in that repo's CHANGELOG). Nothing in the
-controller changed for Part B.
+## Outstanding (demo restoration, not slice validation)
+- **RomM (HDD app) + USB re-enroll:** RomM's data is safe on the host USB (`/mnt/felhom-usb/felhom-data`,
+  untouched by the re-provision). Restoring it is the slice-10 enroll flow (assign → guest-attach →
+  reboot to activate the bind → register storage → deploy). With the split, the USB binds to a free slot
+  (mp1+) since **mp0 is now the Docker-data volume** — no collision. Documented as the final restore step;
+  not required for slice validation (ActualBudget covered the deploy path; the USB bind was not touched).

controller/README.md

@@ -149,6 +149,16 @@ The app catalog lives in a separate Git repository. The controller:
    - Hard block if `used_mb + new_request > usable_memory`
    - `CommittedMemory()` (declared sum) still used for soft overcommit warning only
    - Deploy page shows real memory usage bar (not declared reservations)
+4b. **Docker-data volume reserved-buffer gate (v0.58.0, storage-split prevention layer):** the OS rootfs
+   and Docker data are split onto separate volumes; infra (controller/traefik/cloudflared/filebrowser)
+   shares the one Docker data-root (`/var/lib/docker`) and is protected by **prevention, not placement**.
+   `system.GetDockerVolumeHeadroom()` measures the Docker-data volume via `statfs("/")` (the controller
+   container's overlay root is the upperdir on that volume — true with the golden's **overlay2** driver)
+   and reserves `max(5 GB, 10%)`. `deployStack` **refuses a new deploy (HTTP 507)** when free space is
+   at/under the buffer; the deploy page shows the warning + disables the button. Fail-open on a statfs
+   error. The runtime disk monitor (`healthcheck.go`, warn 80% / crit 90%) watches the same volume and
+   trips above the buffer. (Assumes the split guest's large data volume; the golden bakes overlay2 + log
+   rotation so images+volumes live on the data volume, not `/var/lib/containerd`.)
 5. Pre-generated secret values are submitted as hidden form inputs so the **same values** the user saw are saved to `app.yaml` (no silent re-generation on submit). Controller saves `app.yaml`, sets in-memory `Deployed` + `Deploying` flags, then runs `docker compose up -d` **asynchronously** in a goroutine — API returns immediately so the UI switches to the progress panel without waiting for image pulls. On failure the goroutine reverts both disk and in-memory state and sets `DeployError`.
 6. 3-step progress panel polls `GET /api/stacks/{name}` every 3s: config saved → `deploying` (pulling images) → containers starting → health check passed. New `StateDeploying` state shown while compose-up is in progress (no containers yet).
 7. Post-deploy: locked fields (DB_PASSWORD, etc.) become read-only; the "Automatikusan generált értékek" section continues to show the saved values on the settings page