docs: REPORT for v0.29.0 storage split (live-validated via 9201 re-provision)

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

REPORT.md

@@ -1,32 +1,58 @@
-# REPORT — felhom-agent v0.28.0
+# REPORT — felhom-agent v0.29.0 (OS / Docker-data storage split: golden + provision)
 
-Backup re-target → offsite PBS (real DR) + operator-signed decommission + restore-test bind-mount fix.
-Implemented from the "whole-guest backup → felhom-pbs" CC SPEC (PHASE 0 gate → PHASE 1/3 + a validation-discovered fix). Live-validated on demo-felhom (PVE 9.2.2, guest 9201). Secrets (PBS token/fingerprint, encryption key) redacted / stored out-of-band.
+Phase 1 of the storage-split slice (Phase 2 = felhom-controller v0.58.0 prevention layer). The
+controller guest's OS rootfs and Docker data are carved onto separate `local-lvm` volumes for
+**resilience** — an isolated OS rootfs stays bootable + agent-recoverable if the Docker volume fills.
+Built, the golden re-baked, and **live-validated by destroying + re-provisioning guest 9201**.
 
-## PHASE 0 — GATE (proven live before any re-target code)
-- **Backup → PBS, snapshot mode** fires the `create storage snapshot 'vzdump'` marker → 8B.2 early-resume/quiesce survives a PBS target (marker is mode-driven, not target-driven).
-- **Restore-test enumerates PBS** through the SAME generic `StorageContent` (`/nodes/<node>/storage/felhom-pbs/content` returns `content:"backup"` + ctime/vmid/volid) — `PickRestoreCandidate`/`latestArchive` need NO PBS-client change.
-- **pct restore from a PBS volid** round-trips (storage.cfg encryption key applied transparently). PBS gotchas (`ignore-verified`, node-from-UPID, privsep) touch only the verify-API path.
+## What shipped (v0.29.0)
+- **`configs/build-golden.sh`** — split baked in: `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}` (default
+  **32**, was hardcoded 8) **plus** `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`
+  (default 16). `daemon.json` bakes the **classic overlay2 driver** (`features.containerd-snapshotter:
+  false`) + log rotation (`max-size 10m`, `max-file 3`). Guards: aborts unless `/var/lib/docker` is a
+  separate mount, the driver is overlay2, and vzdump **includes** mp0 (the B3 trap).
+- **`internal/reconcile/bringup.go`** — `GuestMount.Backup` emits `,backup=1` (closes the spike-B3/B5
+  silent-DB-loss trap at the mount builder). `BringUpSpec.{DataVolGrowGB,DataVolMount}` grow the
+  golden-carried Docker-data volume (default `mp0`) online to the per-customer target rather than
+  attaching a fresh empty volume that would shadow the baked images. Plus `RootfsGrowGB`.
+- **CLI:** `--selftest=bring-up|provision` gain `-rootfs-grow` / `-datavol-grow` / `-datavol-mount`.
+- **`RUNBOOK-provisioning-storage.md`** — provisioning procedure + fresh-PVE-install thin-pool carving
+  (`hdsize`/`maxroot`/`maxvz`) + the per-customer sizing seam (flags now; hub storage manifest later).
+- Tests: `buildBringUpConfig` backup=1 emission; bring-up issues rootfs + data-volume resizes.
 
-## PHASE 1 — backup re-target → felhom-pbs
-- `config.BackupConfig.BackupTarget()` returns `backup.local_backup_target` or, when empty, the new default `felhom-pbs` (separate hardware = real DR). Configurable; no call site hardcodes it. All `NewBackupRunner` sites + restore-test `SourceTier` route through it.
-- Live: demo `agent.json` `local_backup_target` "local" → **felhom-pbs**; `--selftest=backup -vmid 9201` → `target=felhom-pbs`, snapshot, crash-consistent ✓. Daemon `/backup/status` now reports `target_id=felhom-pbs`.
-- **PBS prune default** set on the storage: `prune-backups keep-daily=7,keep-weekly=4,keep-monthly=3` (was keep-all at 73% full).
+## The critical fix validation caught (overlay2)
+Docker 29's **default containerd-snapshotter keeps the image store at `/var/lib/containerd`** (on the OS
+rootfs), so mounting the data volume at `/var/lib/docker` only moved named volumes — **1.2 GB of images
+stayed on the rootfs** (validated live), defeating the split and breaking the controller's `statfs("/")`
+guard. Fix: the classic **overlay2** driver stores everything (images + overlay + volumes) under
+data-root = the data volume. A start-vs-restart trap (docker-ce auto-starts on install) was also fixed
+(daemon.json needs a `restart`). After the fix, a provisioned guest showed images on the data volume
+(`/var/lib/docker/overlay2`), `/var/lib/containerd` idle, and a lean OS rootfs.
 
-## Restore-test bind-mount fix (validation-discovered; scope-approved)
-- A slice-10 enrolled guest's data drive is a host **bind-mount** `mp0`; vzrestore refuses it under the privsep token ("restoring 'mpN' to bind mount is only possible for root") — so the restore-test failed for EVERY enrolled guest, independent of backup tier.
-- Fix: the restore-test reads the SOURCE guest config (vmid parsed from the archive volid) and passes `RestoreLXCOptions.MountOverrides` converting each bind-mount `mpN` → a throwaway 1G volume on the restore storage, plus a `rootfs` override sized from the source (PVE requires rootfs when mp params are present). Boot-verify doesn't need the data. Storage-backed mounts restore normally; best-effort if the source config is unreadable.
-- Live: daemon **scheduled restore-test from felhom-pbs → `pass:true, verified:"boot+running", source_tier:"pbs"`** (44s, scratch torn down clean).
+## Live validation
+1. **Re-baked the golden** with the split (overlay2): docker OK on overlay2, `/var/lib/docker` a separate
+   ext4 mount, vzdump "including mount point mp0 ('/var/lib/docker')", 580 MB archive carrying the images.
+2. **Throwaway provision (9301)** from the golden: booted in 39 s; `pct config` mp0=`/var/lib/docker,backup=1,size=256G`,
+   rootfs 32G; overlay2; baked images on the data volume; **OS isolation** — filled the data volume to
+   100%, the OS rootfs stayed at 4% + writable; log rotation effective.
+3. **Destroyed + re-provisioned the live 9201** from the same golden via `--selftest=provision`
+   (customer `demo-felhom`, retrieval passphrase reused from its old bootstrap; `-datavol-grow 240` →
+   256 GB) + a reboot: bring-up booted in 39 s, the back-half minted a token + attached the bootstrap
+   mp9, and after reboot the controller-bootstrap deployed the baked controller, which pulled its config
+   from the hub. All four infra containers came up **from baked images with no pull**. Split + overlay2 +
+   lean rootfs confirmed on the real guest; the controller's deploy gate refused (507) on the full data
+   volume. (Phase-2 details in the controller REPORT.)
 
-## PHASE 3 — operator-signed decommission (reachable now)
-- The previously-unreachable `IntentDecommissioned` is now reached ONLY via a gate-VERIFIED operator signature (classified destructive). New `signedjobs.DecommissionExecutor` (op `decommission`) → `IntentStore.SetDecommissioned`, keyed by the drive's **storage** durable-id (the watchdog's key, e.g. `uuid:<fs-uuid>` — NOT the device-level `byid:/byuuid:` of `storage_wipe`), so the intent actually gates remounts. New `ExecutorChain` serves both wipe + decommission; runner wiring moved below the intent-store open. `felhom-opsign` builds decommission params from `-durable-id`. No customer/controller UI — operator path is hub jobs-queue → signed-jobs runner. Distinct from a customer-confirmable safe eject.
+## Provisioning facts confirmed (for the spec/runbook)
+- The restore carries all mountpoints (golden mp0 travels in); `freeMountSlot` picks the lowest free slot
+  for a later USB enroll, so **mp0 = Docker-data, mp9 = bootstrap, USB = mp1+** — no collision.
+- Attaching/flag-changing a mountpoint on a running unprivileged guest is **pending until reboot** — the
+  provision flow's post-provision reboot activates the bootstrap mp9 (and would activate a USB bind).
+- `backup=1` on the Docker-data mp is mandatory twice over: keeps named-volume DBs in PBS, AND captures
+  the volume (with baked images) into the golden archive.
 
-## Tests
-- `internal/signedjobs`: decommission happy-path / foreign-op / unbound-refusal / no-store; ExecutorChain dispatch.
-- `internal/reconcile`: `archiveVMID` (PBS ct/vm + vzdump forms), `bindMountOverrides`, `rootfsSizeGB`/`sizeToGB`.
-- Full `go build ./...` + package tests green.
-
-## Live deploy
-- `felhom-agent 0.28.0` active on demo-felhom (prior binary kept as `felhom-agent.bak-0.27.0`); config `local_backup_target=felhom-pbs`, restore-test cadence restored to 86400s.
-- **Golden prune** (housekeeping): kept the newest 2 golden (VMID 9100) archives on `local`, freed the older 11.
-- **Note (not actioned):** the drive's legacy bare-metal `felhom_data` (underscore) dir is orphaned under the new `felhom-data` (hyphen) namespace on `/mnt/felhom-usb`; flagged for a real-customer migration path (out of scope here).
+## Outstanding (demo restoration, not slice validation)
+RomM (the HDD app) + USB re-enroll: its data is safe on the host USB (`/mnt/felhom-usb/felhom-data`,
+untouched). Restoring it is the slice-10 enroll flow (assign → guest-attach → reboot → register storage
+→ deploy); with the split it binds to mp1+ (mp0 is the Docker-data volume). Documented as the final
+restore step.