felhom-agent/RUNBOOK-provisioning-storage.md

RUNBOOK — provisioning the OS / Docker-data storage split

How the controller guest's OS rootfs and Docker data are carved onto separate local-lvm volumes, and how to size them per node/customer. Grounded by the storage-split spike (REPORT-storage-split-spike.md). Defaults target a ~512 GB SSD: 32 GB OS rootfs + 256 GB Docker-data, rest of the thin pool free for other guests.

The design (locked)

• Split for resilience, not backup size. An isolated OS rootfs stays bootable + agent-recoverable if the Docker volume fills. Both volumes ride into PBS regardless (so the backup size is unchanged) — the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
• One Docker daemon, one data-root, one volume at /var/lib/docker for ALL images + overlay (controller, traefik, cloudflared, filebrowser, and customer apps). Infra is NOT relocated to the OS partition — infra/user separation under one daemon isn't cleanly possible and isn't needed. Infra is protected by the controller's prevention layer (reserved buffer + deploy gate + disk alerts + baked log rotation), not by placement.
• backup=1 on the Docker-data volume is non-negotiable. Extra LXC mountpoints default to backup=0 = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs fall out of the snapshot AND the volume (with baked images) is excluded from the golden archive.

Golden build (per controller-image baseline)

configs/build-golden.sh builds the template guest with the split baked in:

# defaults shown; override via env
OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 \
  ./build-golden.sh <VMID> <TEMPLATE> <ROOTFS_STORAGE> <ARCHIVE_STORAGE> <BRIDGE> <CONTROLLER_IMAGE>

• --rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB} + --mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1.
• Keep GOLDEN_DOCKER_GB small (just the baked images + headroom) — provision grows it.
• The script bakes /etc/docker/daemon.json with features.containerd-snapshotter: false (the classic overlay2 driver) + log rotation (max-size 10m, max-file 3), verifies /var/lib/docker is a separate mount + the driver is overlay2, and aborts if vzdump excludes mp0 (the B3 trap).
• Why overlay2 (not the Docker-29 default containerd-snapshotter): containerd-snapshotter keeps the image content store under /var/lib/containerd, which is NOT /var/lib/docker — so it would stay on the OS rootfs and the split would only move named volumes, leaving images (the bulk) on the rootfs (validated live: 1.2 GB of images landed on the rootfs). overlay2 stores everything (images + overlay + volumes) under data-root = the data volume, which is what the split + the controller's statfs("/") prevention guard both require.

Provision (per customer guest)

The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data volume + baked images), then sizes per customer. The Docker-data volume is grown (not re-attached) so the baked images are never shadowed:

felhom-agent --selftest=provision \
  -archive <golden-volid> -vmid <VMID> \
  -customer-id <id> -hub-password <retrieval-passphrase> \
  -rootfs-grow <N>   # GiB to add to the OS rootfs beyond the golden's 32 (0 = keep 32)
  -datavol-grow <N>  # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 240 → 256 total)
  -datavol-mount mp0 # the golden's Docker-data slot (default mp0)

In code: reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount} — both grows are online (pct resize, spike B4), each its own task. GuestMount.Backup (→ ,backup=1) is set on any data-bearing additive mount (bringup.go buildConfigParams). The bootstrap mount is mp9; USB/data drive enroll picks the lowest free slot (freeMountSlot), so it never collides with the docker mp0.

Per-customer sizing source: flags today; the slice-10 hub storage manifest is the intended home (marked at bringup.go GuestMount / the slice 10 wires the hub storage manifest comment) — the controller already pulls customer-scoped config from the hub, so OS/data sizes belong there.

Default sizing guidance

• ~512 GB SSD: 32 GB rootfs + 256 GB Docker-data (-rootfs-grow 0 -datavol-grow 240), leaving the rest of the thin pool free for other guests.
• Other SSDs: size the data volume to "most of the free thin pool minus a reserve for other guests." local-lvm is thin-provisioned, so an over-allocation only consumes space as it's written.

Fresh PVE host install — thin-pool carving (host-level, do this at install)

This sizes the host's local root vs the local-lvm thin pool (the pool the guests' volumes are carved from). It is NOT the guest split — do it on the Proxmox ISO installer's Advanced LVM options:

• hdsize — total disk PVE uses (leave headroom on multi-disk nodes).
• maxroot — cap on the host / (pve/root). Keep small (e.g. 20–40 GB) — the host OS is thin.
• maxvz — cap on the data thin pool (local-lvm). Set to 0 / leave default to let the pool take the rest, OR cap it to reserve unallocated VG space.
• minfree — reserved free space in the VG (thin-pool safety).
• swapsize — host swap.

Never repartition a live host root. Per-guest carving (rootfs + Docker-data) is done at provision time via pct against the existing thin pool — no host repartition needed (the demo's pool already has ~350 GB free).

In-place migration of an EXISTING guest (NOT used by this slice)

This slice rebuilds clean (fresh re-provision), so the move-then-verify in-place migration is not used here. It remains documented in REPORT-storage-split-spike.md §B2 for a future in-place node upgrade: stop apps → stop docker → rsync -aHAX --numeric-ids /var/lib/docker/ <new-vol>/ (check rsync's OWN exit code) → switch data-root → verify volumes+DBs+boot → only then reclaim. backup=1 on the new mount and a CT restart to activate it are required (spike B3/B4).