felhom-agent/RUNBOOK-provisioning-storage.md

# RUNBOOK — provisioning the OS / Docker-data storage split

How the controller guest's **OS rootfs** and **Docker data** are carved onto separate `local-lvm`
volumes, and how to size them per node/customer. Grounded by the storage-split spike
(`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 256 GB
Docker-data**, rest of the thin pool free for other guests.

## The design (locked)

- **Split for resilience, not backup size.** An isolated OS rootfs stays bootable + agent-recoverable
  if the Docker volume fills. Both volumes ride into PBS regardless (so the backup size is unchanged) —
  the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
- **One Docker daemon, one data-root, one volume** at `/var/lib/docker` for ALL images + overlay
  (controller, traefik, cloudflared, filebrowser, and customer apps). Infra is NOT relocated to the OS
  partition — infra/user separation under one daemon isn't cleanly possible and isn't needed. Infra is
  protected by the controller's **prevention layer** (reserved buffer + deploy gate + disk alerts +
  baked log rotation), not by placement.
- **`backup=1` on the Docker-data volume is non-negotiable.** Extra LXC mountpoints default to
  `backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs fall out of the
  snapshot AND the volume (with baked images) is excluded from the golden archive.

## Golden build (per controller-image baseline)

`configs/build-golden.sh` builds the template guest with the split baked in:

```
# defaults shown; override via env
OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 \
  ./build-golden.sh <VMID> <TEMPLATE> <ROOTFS_STORAGE> <ARCHIVE_STORAGE> <BRIDGE> <CONTROLLER_IMAGE>
```

- `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}` + `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`.
- Keep `GOLDEN_DOCKER_GB` small (just the baked images + headroom) — provision grows it.
- The script bakes `/etc/docker/daemon.json` with **`features.containerd-snapshotter: false`** (the
  classic **overlay2** driver) + log rotation (`max-size 10m`, `max-file 3`), verifies `/var/lib/docker`
  is a separate mount + the driver is overlay2, and **aborts if vzdump excludes mp0** (the B3 trap).
- **Why overlay2 (not the Docker-29 default containerd-snapshotter):** containerd-snapshotter keeps the
  image content store under `/var/lib/containerd`, which is NOT `/var/lib/docker` — so it would stay on
  the OS rootfs and the split would only move named volumes, leaving images (the bulk) on the rootfs
  (validated live: 1.2 GB of images landed on the rootfs). overlay2 stores **everything** (images +
  overlay + volumes) under data-root = the data volume, which is what the split + the controller's
  `statfs("/")` prevention guard both require.

## Provision (per customer guest)

The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data volume + baked images),
then sizes per customer. The Docker-data volume is **grown** (not re-attached) so the baked images are
never shadowed:

```
felhom-agent --selftest=provision \
  -archive <golden-volid> -vmid <VMID> \
  -customer-id <id> -hub-password <retrieval-passphrase> \
  -rootfs-grow <N>   # GiB to add to the OS rootfs beyond the golden's 32 (0 = keep 32)
  -datavol-grow <N>  # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 240 → 256 total)
  -datavol-mount mp0 # the golden's Docker-data slot (default mp0)
```

In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount}` — both grows are online
(`pct resize`, spike B4), each its own task. `GuestMount.Backup` (→ `,backup=1`) is set on any
data-bearing additive mount (`bringup.go` `buildConfigParams`). The bootstrap mount is `mp9`; USB/data
drive enroll picks the lowest free slot (`freeMountSlot`), so it never collides with the docker mp0.

**Per-customer sizing source:** flags today; the **slice-10 hub storage manifest** is the intended
home (marked at `bringup.go` `GuestMount` / the `slice 10 wires the hub storage manifest` comment) —
the controller already pulls customer-scoped config from the hub, so OS/data sizes belong there.

### Default sizing guidance
- ~512 GB SSD: 32 GB rootfs + 256 GB Docker-data (`-rootfs-grow 0 -datavol-grow 240`), leaving the rest
  of the thin pool free for other guests.
- Other SSDs: size the data volume to "most of the free thin pool minus a reserve for other guests."
  `local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's written.

## Fresh PVE host install — thin-pool carving (host-level, do this at install)

This sizes the **host's** `local` root vs the `local-lvm` thin pool (the pool the guests' volumes are
carved from). It is NOT the guest split — do it on the Proxmox ISO installer's **Advanced LVM options**:

- `hdsize` — total disk PVE uses (leave headroom on multi-disk nodes).
- `maxroot` — cap on the host `/` (`pve/root`). Keep small (e.g. 20–40 GB) — the host OS is thin.
- `maxvz` — cap on the `data` thin pool (`local-lvm`). Set to 0 / leave default to let the pool take the
  rest, OR cap it to reserve unallocated VG space.
- `minfree` — reserved free space in the VG (thin-pool safety).
- `swapsize` — host swap.

**Never repartition a live host root.** Per-guest carving (rootfs + Docker-data) is done at provision
time via `pct` against the existing thin pool — no host repartition needed (the demo's pool already has
~350 GB free).

## In-place migration of an EXISTING guest (NOT used by this slice)

This slice rebuilds clean (fresh re-provision), so the move-then-verify in-place migration is **not
used here**. It remains documented in `REPORT-storage-split-spike.md` §B2 for a future in-place node
upgrade: stop apps → stop docker → `rsync -aHAX --numeric-ids /var/lib/docker/ <new-vol>/` (check
rsync's OWN exit code) → switch data-root → verify volumes+DBs+boot → only then reclaim. `backup=1` on
the new mount and a CT restart to activate it are required (spike B3/B4).