felhom-agent/RUNBOOK-provisioning-storage.md

RUNBOOK — provisioning the OS / Docker-data / user-data storage split

How the controller guest's OS rootfs, Docker data, and SSD user-data are carved onto three separate local-lvm volumes, and how to size them per node/customer. Grounded by the storage-split spike (REPORT-storage-split-spike.md). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 200 GB Docker-data

• 50 GB user-data**, rest of the thin pool free for other guests.

The design (locked)

Three thin volumes are baked into the golden and grown at provision:

mp slot	Mount	Holds	Golden default	Provision grow knob
rootfs	/	OS only	OS_SIZE_GB=32	-rootfs-grow
mp0	/var/lib/docker	ALL Docker images + overlay + named volumes	GOLDEN_DOCKER_GB=16	-datavol-grow
mp1	/mnt/sys_drive	SSD user-data (the controller's system_data_path; <sys_drive>/felhom-data = SSD-only app data + DBs)	GOLDEN_SYSDATA_GB=8	-sysdata-grow

• Split for resilience, not backup size. An isolated OS rootfs stays bootable + agent-recoverable if the Docker volume fills. All three volumes ride into PBS regardless (so the backup size is unchanged) — the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
• One Docker daemon, one data-root, one volume at /var/lib/docker for ALL images + overlay (controller, traefik, cloudflared, filebrowser, and customer apps). Infra is NOT relocated to the OS partition — infra/user separation under one daemon isn't cleanly possible and isn't needed. Infra is protected by the controller's prevention layer (reserved buffer + deploy gate + disk alerts + baked log rotation), not by placement.
• /mnt/sys_drive is a static CT volume, NOT an enrolled drive. It never enrolls, ejects, or decommissions — it is part of the golden's baked layout. The controller auto-discovers <sys_drive>/felhom-data and, via system.IsMountPoint, warns when /mnt/sys_drive is not its own mount (selftest.go). Making mp1 a real mount clears that warning with zero controller change; the controller is untouched by this slice. The mp mounted under the guest's /mnt propagates into the controller container through the existing -v /mnt:/mnt:rslave bind (the same path enrolled drives use).
• backup=1 on the Docker-data AND user-data volumes is non-negotiable. Extra LXC mountpoints default to backup=0 = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs / SSD app data fall out of the snapshot AND (for mp0) the baked images are excluded from the golden archive.

Golden build (per controller-image baseline)

configs/build-golden.sh builds the template guest with the split baked in:

# defaults shown; override via env
OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 GOLDEN_SYSDATA_GB=8 \
  ./build-golden.sh <VMID> <TEMPLATE> <ROOTFS_STORAGE> <ARCHIVE_STORAGE> <BRIDGE> <CONTROLLER_IMAGE>

• --rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}
  • --mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1
  • --mp1 ${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1.
• Keep GOLDEN_DOCKER_GB / GOLDEN_SYSDATA_GB small (the golden ships them near-empty) — provision grows both. The controller creates <sys_drive>/felhom-data itself once /mnt/sys_drive is a real mount; the golden does NOT pre-create it.
• The script bakes /etc/docker/daemon.json with features.containerd-snapshotter: false (the classic overlay2 driver) + log rotation (max-size 10m, max-file 3), verifies /var/lib/docker AND /mnt/sys_drive are separate mounts + the driver is overlay2, and aborts if vzdump excludes mp0 or mp1 (the B3 trap).
• Why overlay2 (not the Docker-29 default containerd-snapshotter): containerd-snapshotter keeps the image content store under /var/lib/containerd, which is NOT /var/lib/docker — so it would stay on the OS rootfs and the split would only move named volumes, leaving images (the bulk) on the rootfs (validated live: 1.2 GB of images landed on the rootfs). overlay2 stores everything (images + overlay + volumes) under data-root = the data volume, which is what the split + the controller's statfs("/") prevention guard both require.

Provision (per customer guest)

The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data + mp1 user-data volumes + baked images), then sizes per customer. Each volume is grown (not re-attached) so the baked images are never shadowed:

felhom-agent --selftest=provision \
  -archive <golden-volid> -vmid <VMID> \
  -customer-id <id> -hub-password <retrieval-passphrase> \
  -rootfs-grow <N>   # GiB to add to the OS rootfs beyond the golden's 32 (0 = keep 32)
  -datavol-grow <N>  # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 184 → 200 total)
  -datavol-mount mp0 # the golden's Docker-data slot (default mp0)
  -sysdata-grow <N>  # GiB to add to the user-data volume beyond the golden's 8 (e.g. 42 → 50 total)
  -sysdata-mount mp1 # the golden's user-data slot (default mp1)

In code: reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount, SysDataGrowGB, SysDataMount} — all grows are online (pct resize, spike B4), each its own task. Separateness, not size, clears the controller warning — -sysdata-grow 0 still yields a separate, warning-free mount at the golden size; the grow is an orthogonal knob. GuestMount.Backup (→ ,backup=1) is set on any data-bearing additive mount (bringup.go buildBringUpConfig). The bootstrap mount is mp9, the shared parent bind is mp8; USB/data drive enroll picks the lowest free slot (freeMountSlot), which auto-skips the baked mp0/mp1.

Per-customer sizing source: flags today; the slice-10 hub storage manifest is the intended home (marked at bringup.go GuestMount / the slice 10 wires the hub storage manifest comment) — the controller already pulls customer-scoped config from the hub, so OS/data sizes belong there.

Default sizing guidance

• ~512 GB SSD: 32 GB rootfs + 200 GB Docker-data + 50 GB user-data (-rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42), leaving the rest of the thin pool free for other guests.
• Other SSDs: size the Docker-data + user-data volumes to "most of the free thin pool minus a reserve for other guests." local-lvm is thin-provisioned, so an over-allocation only consumes space as it's written.

Fresh PVE host install — thin-pool carving (host-level, do this at install)

This sizes the host's local root vs the local-lvm thin pool (the pool the guests' volumes are carved from). It is NOT the guest split — do it on the Proxmox ISO installer's Advanced LVM options:

• hdsize — total disk PVE uses (leave headroom on multi-disk nodes).
• maxroot — cap on the host / (pve/root). Keep small (e.g. 20–40 GB) — the host OS is thin.
• maxvz — cap on the data thin pool (local-lvm). Set to 0 / leave default to let the pool take the rest, OR cap it to reserve unallocated VG space.
• minfree — reserved free space in the VG (thin-pool safety).
• swapsize — host swap.

Never repartition a live host root. Per-guest carving (rootfs + Docker-data) is done at provision time via pct against the existing thin pool — no host repartition needed (the demo's pool already has ~350 GB free).

In-place migration of an EXISTING guest (NOT used by this slice)

This slice rebuilds clean (fresh re-provision), so the move-then-verify in-place migration is not used here. It remains documented in REPORT-storage-split-spike.md §B2 for a future in-place node upgrade: stop apps → stop docker → rsync -aHAX --numeric-ids /var/lib/docker/ <new-vol>/ (check rsync's OWN exit code) → switch data-root → verify volumes+DBs+boot → only then reclaim. backup=1 on the new mount and a CT restart to activate it are required (spike B3/B4).