admin
166a1c8bcb
The golden bakes --onboot 0 (template safety) and the back-half never overrode it, so every provisioned customer guest was onboot:0 -> after a host reboot/power-cut the customer's whole home-server stayed stopped until a manual pct start. Add a fatal 'pct set <vmid> -onboot 1' step to BackHalf.Provision (right after the config-mount attach), mirroring the existing pct set ops. No startup/boot-order: the v0.75 mountpoint-gate covers the drive-bind race at boot. Golden build-golden.sh unchanged (templates must not auto-start). Unit-tested (TestProvision_SetsOnbootOne + red-proof). RUNBOOK note added. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01FpBYrZCt9sFDqLgbG5GRGD
130 lines
8.5 KiB
Markdown
130 lines
8.5 KiB
Markdown
# RUNBOOK — provisioning the OS / Docker-data / user-data storage split
|
||
|
||
How the controller guest's **OS rootfs**, **Docker data**, and **SSD user-data** are carved onto three
|
||
separate `local-lvm` volumes, and how to size them per node/customer. Grounded by the storage-split spike
|
||
(`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 200 GB Docker-data
|
||
+ 50 GB user-data**, rest of the thin pool free for other guests.
|
||
|
||
## The design (locked)
|
||
|
||
Three thin volumes are baked into the golden and grown at provision:
|
||
|
||
| mp slot | Mount | Holds | Golden default | Provision grow knob |
|
||
|---------|-------|-------|----------------|---------------------|
|
||
| rootfs | `/` | OS only | `OS_SIZE_GB=32` | `-rootfs-grow` |
|
||
| `mp0` | `/var/lib/docker` | ALL Docker images + overlay + named volumes | `GOLDEN_DOCKER_GB=16` | `-datavol-grow` |
|
||
| `mp1` | `/mnt/sys_drive` | SSD user-data (the controller's `system_data_path`; `<sys_drive>/felhom-data` = SSD-only app data + DBs) | `GOLDEN_SYSDATA_GB=8` | `-sysdata-grow` |
|
||
|
||
- **Split for resilience, not backup size.** An isolated OS rootfs stays bootable + agent-recoverable
|
||
if the Docker volume fills. All three volumes ride into PBS regardless (so the backup size is
|
||
unchanged) — the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
|
||
- **One Docker daemon, one data-root, one volume** at `/var/lib/docker` for ALL images + overlay
|
||
(controller, traefik, cloudflared, filebrowser, and customer apps). Infra is NOT relocated to the OS
|
||
partition — infra/user separation under one daemon isn't cleanly possible and isn't needed. Infra is
|
||
protected by the controller's **prevention layer** (reserved buffer + deploy gate + disk alerts +
|
||
baked log rotation), not by placement.
|
||
- **`/mnt/sys_drive` is a static CT volume, NOT an enrolled drive.** It never enrolls, ejects, or
|
||
decommissions — it is part of the golden's baked layout. The controller auto-discovers
|
||
`<sys_drive>/felhom-data` and, via `system.IsMountPoint`, **warns when `/mnt/sys_drive` is not its own
|
||
mount** (`selftest.go`). Making `mp1` a real mount clears that warning with **zero controller change**;
|
||
the controller is untouched by this slice. The `mp` mounted under the guest's `/mnt` propagates into
|
||
the controller container through the existing `-v /mnt:/mnt:rslave` bind (the same path enrolled drives
|
||
use).
|
||
- **`backup=1` on the Docker-data AND user-data volumes is non-negotiable.** Extra LXC mountpoints
|
||
default to `backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs / SSD app
|
||
data fall out of the snapshot AND (for mp0) the baked images are excluded from the golden archive.
|
||
|
||
## Golden build (per controller-image baseline)
|
||
|
||
`configs/build-golden.sh` builds the template guest with the split baked in:
|
||
|
||
```
|
||
# defaults shown; override via env
|
||
OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 GOLDEN_SYSDATA_GB=8 \
|
||
./build-golden.sh <VMID> <TEMPLATE> <ROOTFS_STORAGE> <ARCHIVE_STORAGE> <BRIDGE> <CONTROLLER_IMAGE>
|
||
```
|
||
|
||
- `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}`
|
||
+ `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`
|
||
+ `--mp1 ${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1`.
|
||
- Keep `GOLDEN_DOCKER_GB` / `GOLDEN_SYSDATA_GB` small (the golden ships them near-empty) — provision
|
||
grows both. The controller creates `<sys_drive>/felhom-data` itself once `/mnt/sys_drive` is a real
|
||
mount; the golden does NOT pre-create it.
|
||
- The script bakes `/etc/docker/daemon.json` with **`features.containerd-snapshotter: false`** (the
|
||
classic **overlay2** driver) + log rotation (`max-size 10m`, `max-file 3`), verifies `/var/lib/docker`
|
||
AND `/mnt/sys_drive` are separate mounts + the driver is overlay2, and **aborts if vzdump excludes mp0
|
||
or mp1** (the B3 trap).
|
||
- **Why overlay2 (not the Docker-29 default containerd-snapshotter):** containerd-snapshotter keeps the
|
||
image content store under `/var/lib/containerd`, which is NOT `/var/lib/docker` — so it would stay on
|
||
the OS rootfs and the split would only move named volumes, leaving images (the bulk) on the rootfs
|
||
(validated live: 1.2 GB of images landed on the rootfs). overlay2 stores **everything** (images +
|
||
overlay + volumes) under data-root = the data volume, which is what the split + the controller's
|
||
`statfs("/")` prevention guard both require.
|
||
|
||
## Provision (per customer guest)
|
||
|
||
The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data + mp1 user-data volumes +
|
||
baked images), then sizes per customer. Each volume is **grown** (not re-attached) so the baked images
|
||
are never shadowed:
|
||
|
||
```
|
||
felhom-agent --selftest=provision \
|
||
-archive <golden-volid> -vmid <VMID> \
|
||
-customer-id <id> -hub-password <retrieval-passphrase> \
|
||
-rootfs-grow <N> # GiB to add to the OS rootfs beyond the golden's 32 (0 = keep 32)
|
||
-datavol-grow <N> # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 184 → 200 total)
|
||
-datavol-mount mp0 # the golden's Docker-data slot (default mp0)
|
||
-sysdata-grow <N> # GiB to add to the user-data volume beyond the golden's 8 (e.g. 42 → 50 total)
|
||
-sysdata-mount mp1 # the golden's user-data slot (default mp1)
|
||
```
|
||
|
||
In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount, SysDataGrowGB, SysDataMount}`
|
||
— all grows are online (`pct resize`, spike B4), each its own task. **Separateness, not size, clears the
|
||
controller warning** — `-sysdata-grow 0` still yields a separate, warning-free mount at the golden size;
|
||
the grow is an orthogonal knob. `GuestMount.Backup` (→ `,backup=1`) is set on any data-bearing additive
|
||
mount (`bringup.go` `buildBringUpConfig`). The bootstrap mount is `mp9`, the shared parent bind is `mp8`;
|
||
USB/data drive enroll picks the lowest free slot (`freeMountSlot`), which auto-skips the baked mp0/mp1.
|
||
|
||
**Auto-start on host reboot (`onboot:1`).** The provision back-half sets the customer guest to
|
||
**`onboot:1`** (`provision/backhalf.go`, v0.41.0) so the whole home-server (controller + apps) comes back
|
||
on its own after a host reboot/power-cut. The **golden template stays `onboot:0`** (`build-golden.sh`) —
|
||
a template must not auto-start. No `startup`/boot-order is used: the v0.75 mountpoint-gate covers the
|
||
drive-bind race at boot, so the controller won't write app data onto the rootfs while drives re-bind.
|
||
(Guests provisioned before v0.41.0 need a one-time `pct set <vmid> -onboot 1`.)
|
||
|
||
**Per-customer sizing source:** flags today; the **slice-10 hub storage manifest** is the intended
|
||
home (marked at `bringup.go` `GuestMount` / the `slice 10 wires the hub storage manifest` comment) —
|
||
the controller already pulls customer-scoped config from the hub, so OS/data sizes belong there.
|
||
|
||
### Default sizing guidance
|
||
- ~512 GB SSD: 32 GB rootfs + 200 GB Docker-data + 50 GB user-data
|
||
(`-rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42`), leaving the rest of the thin pool free for
|
||
other guests.
|
||
- Other SSDs: size the Docker-data + user-data volumes to "most of the free thin pool minus a reserve
|
||
for other guests." `local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's
|
||
written.
|
||
|
||
## Fresh PVE host install — thin-pool carving (host-level, do this at install)
|
||
|
||
This sizes the **host's** `local` root vs the `local-lvm` thin pool (the pool the guests' volumes are
|
||
carved from). It is NOT the guest split — do it on the Proxmox ISO installer's **Advanced LVM options**:
|
||
|
||
- `hdsize` — total disk PVE uses (leave headroom on multi-disk nodes).
|
||
- `maxroot` — cap on the host `/` (`pve/root`). Keep small (e.g. 20–40 GB) — the host OS is thin.
|
||
- `maxvz` — cap on the `data` thin pool (`local-lvm`). Set to 0 / leave default to let the pool take the
|
||
rest, OR cap it to reserve unallocated VG space.
|
||
- `minfree` — reserved free space in the VG (thin-pool safety).
|
||
- `swapsize` — host swap.
|
||
|
||
**Never repartition a live host root.** Per-guest carving (rootfs + Docker-data) is done at provision
|
||
time via `pct` against the existing thin pool — no host repartition needed (the demo's pool already has
|
||
~350 GB free).
|
||
|
||
## In-place migration of an EXISTING guest (NOT used by this slice)
|
||
|
||
This slice rebuilds clean (fresh re-provision), so the move-then-verify in-place migration is **not
|
||
used here**. It remains documented in `REPORT-storage-split-spike.md` §B2 for a future in-place node
|
||
upgrade: stop apps → stop docker → `rsync -aHAX --numeric-ids /var/lib/docker/ <new-vol>/` (check
|
||
rsync's OWN exit code) → switch data-root → verify volumes+DBs+boot → only then reclaim. `backup=1` on
|
||
the new mount and a CT restart to activate it are required (spike B3/B4).
|