# RUNBOOK — provisioning the OS / Docker-data / user-data storage split

How the controller guest's **OS rootfs**, **Docker data**, and **SSD user-data** are carved onto three
separate `local-lvm` volumes, and how to size them per node/customer. Grounded by the storage-split spike
(`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 200 GB Docker-data
+ 50 GB user-data**, rest of the thin pool free for other guests.

## The design (locked)

Three thin volumes are baked into the golden and grown at provision:

| mp slot | Mount | Holds | Golden default | Provision grow knob |
|---------|-------|-------|----------------|---------------------|
| rootfs | `/` | OS only | `OS_SIZE_GB=32` | `-rootfs-grow` |
| `mp0` | `/var/lib/docker` | ALL Docker images + overlay + named volumes | `GOLDEN_DOCKER_GB=16` | `-datavol-grow` |
| `mp1` | `/mnt/sys_drive` | SSD user-data (the controller's `system_data_path`; `<sys_drive>/felhom-data` = SSD-only app data + DBs) | `GOLDEN_SYSDATA_GB=8` | `-sysdata-grow` |

- **Split for resilience, not backup size.** An isolated OS rootfs stays bootable + agent-recoverable
  if the Docker volume fills. All three volumes ride into PBS regardless (so the backup size is
  unchanged) — the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
- **One Docker daemon, one data-root, one volume** at `/var/lib/docker` for ALL images + overlay
  (controller, traefik, cloudflared, filebrowser, and customer apps). Infra is NOT relocated to the OS
  partition — infra/user separation under one daemon isn't cleanly possible and isn't needed. Infra is
  protected by the controller's **prevention layer** (reserved buffer + deploy gate + disk alerts +
  baked log rotation), not by placement.
- **`/mnt/sys_drive` is a static CT volume, NOT an enrolled drive.** It never enrolls, ejects, or
  decommissions — it is part of the golden's baked layout. The controller auto-discovers
  `<sys_drive>/felhom-data` and, via `system.IsMountPoint`, **warns when `/mnt/sys_drive` is not its own
  mount** (`selftest.go`). Making `mp1` a real mount clears that warning with **zero controller change**;
  the controller is untouched by this slice. The `mp` mounted under the guest's `/mnt` propagates into
  the controller container through the existing `-v /mnt:/mnt:rslave` bind (the same path enrolled drives
  use).
- **`backup=1` on the Docker-data AND user-data volumes is non-negotiable.** Extra LXC mountpoints
  default to `backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs / SSD app
  data fall out of the snapshot AND (for mp0) the baked images are excluded from the golden archive.

## Golden build (per controller-image baseline)

`configs/build-golden.sh` builds the template guest with the split baked in:

```
# defaults shown; override via env
OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 GOLDEN_SYSDATA_GB=8 \
  ./build-golden.sh <VMID> <TEMPLATE> <ROOTFS_STORAGE> <ARCHIVE_STORAGE> <BRIDGE> <CONTROLLER_IMAGE>
```

- `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}`
  + `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`
  + `--mp1 ${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1`.
- Keep `GOLDEN_DOCKER_GB` / `GOLDEN_SYSDATA_GB` small (the golden ships them near-empty) — provision
  grows both. The controller creates `<sys_drive>/felhom-data` itself once `/mnt/sys_drive` is a real
  mount; the golden does NOT pre-create it.
- The script bakes `/etc/docker/daemon.json` with **`features.containerd-snapshotter: false`** (the
  classic **overlay2** driver) + log rotation (`max-size 10m`, `max-file 3`), verifies `/var/lib/docker`
  AND `/mnt/sys_drive` are separate mounts + the driver is overlay2, and **aborts if vzdump excludes mp0
  or mp1** (the B3 trap).
- **Why overlay2 (not the Docker-29 default containerd-snapshotter):** containerd-snapshotter keeps the
  image content store under `/var/lib/containerd`, which is NOT `/var/lib/docker` — so it would stay on
  the OS rootfs and the split would only move named volumes, leaving images (the bulk) on the rootfs
  (validated live: 1.2 GB of images landed on the rootfs). overlay2 stores **everything** (images +
  overlay + volumes) under data-root = the data volume, which is what the split + the controller's
  `statfs("/")` prevention guard both require.

## Provision (per customer guest)

The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data + mp1 user-data volumes +
baked images), then sizes per customer. Each volume is **grown** (not re-attached) so the baked images
are never shadowed:

```
felhom-agent --selftest=provision \
  -archive <golden-volid> -vmid <VMID> \
  -customer-id <id> -hub-password <retrieval-passphrase> \
  -rootfs-grow <N>   # GiB to add to the OS rootfs beyond the golden's 32 (0 = keep 32)
  -datavol-grow <N>  # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 184 → 200 total)
  -datavol-mount mp0 # the golden's Docker-data slot (default mp0)
  -sysdata-grow <N>  # GiB to add to the user-data volume beyond the golden's 8 (e.g. 42 → 50 total)
  -sysdata-mount mp1 # the golden's user-data slot (default mp1)
```

In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount, SysDataGrowGB, SysDataMount}`
— all grows are online (`pct resize`, spike B4), each its own task. **Separateness, not size, clears the
controller warning** — `-sysdata-grow 0` still yields a separate, warning-free mount at the golden size;
the grow is an orthogonal knob. `GuestMount.Backup` (→ `,backup=1`) is set on any data-bearing additive
mount (`bringup.go` `buildBringUpConfig`). The bootstrap mount is `mp9`, the shared parent bind is `mp8`;
USB/data drive enroll picks the lowest free slot (`freeMountSlot`), which auto-skips the baked mp0/mp1.

**Auto-start on host reboot (`onboot:1`).** The provision back-half sets the customer guest to
**`onboot:1`** (`provision/backhalf.go`, v0.41.0) so the whole home-server (controller + apps) comes back
on its own after a host reboot/power-cut. The **golden template stays `onboot:0`** (`build-golden.sh`) —
a template must not auto-start. No `startup`/boot-order is used: the v0.75 mountpoint-gate covers the
drive-bind race at boot, so the controller won't write app data onto the rootfs while drives re-bind.
(Guests provisioned before v0.41.0 need a one-time `pct set <vmid> -onboot 1`.)

**Per-customer sizing source:** flags today; the **slice-10 hub storage manifest** is the intended
home (marked at `bringup.go` `GuestMount` / the `slice 10 wires the hub storage manifest` comment) —
the controller already pulls customer-scoped config from the hub, so OS/data sizes belong there.

### Default sizing guidance
- ~512 GB SSD: 32 GB rootfs + 200 GB Docker-data + 50 GB user-data
  (`-rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42`), leaving the rest of the thin pool free for
  other guests.
- Other SSDs: size the Docker-data + user-data volumes to "most of the free thin pool minus a reserve
  for other guests." `local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's
  written.

## Fresh PVE host install — thin-pool carving (host-level, do this at install)

This sizes the **host's** `local` root vs the `local-lvm` thin pool (the pool the guests' volumes are
carved from). It is NOT the guest split — do it on the Proxmox ISO installer's **Advanced LVM options**:

- `hdsize` — total disk PVE uses (leave headroom on multi-disk nodes).
- `maxroot` — cap on the host `/` (`pve/root`). Keep small (e.g. 20–40 GB) — the host OS is thin.
- `maxvz` — cap on the `data` thin pool (`local-lvm`). Set to 0 / leave default to let the pool take the
  rest, OR cap it to reserve unallocated VG space.
- `minfree` — reserved free space in the VG (thin-pool safety).
- `swapsize` — host swap.

**Never repartition a live host root.** Per-guest carving (rootfs + Docker-data) is done at provision
time via `pct` against the existing thin pool — no host repartition needed (the demo's pool already has
~350 GB free).

## In-place migration of an EXISTING guest (NOT used by this slice)

This slice rebuilds clean (fresh re-provision), so the move-then-verify in-place migration is **not
used here**. It remains documented in `REPORT-storage-split-spike.md` §B2 for a future in-place node
upgrade: stop apps → stop docker → `rsync -aHAX --numeric-ids /var/lib/docker/ <new-vol>/` (check
rsync's OWN exit code) → switch data-root → verify volumes+DBs+boot → only then reclaim. `backup=1` on
the new mount and a CT restart to activate it are required (spike B3/B4).