admin/felhom-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.40.0: third CT volume — SSD user-data (/mnt/sys_drive, mp1) baked + -sysdata-grow

Browse Source 
Extends the OS/Docker-data split to a three-volume layout: rootfs + Docker-data
(mp0) + SSD user-data (mp1 @ /mnt/sys_drive, backup=1) = the controller's
system_data_path. Clears the controller's "not a separate drive" warning with
zero controller change (it already auto-discovers <sys_drive>/felhom-data and
warns via system.IsMountPoint; the mp reaches the container via the existing
-v /mnt:/mnt:rslave bind).

- build-golden.sh: --mp1 ...,mp=/mnt/sys_drive,backup=1 (env GOLDEN_SYSDATA_GB=8);
  findmnt /mnt/sys_drive separate-mount guard + vzdump aborts if mp0 OR mp1 excluded.
- bringup.go: DefaultSysDataMount=mp1; BringUpSpec.{SysDataGrowGB,SysDataMount};
  new "4c" online grow-only block mirroring the "4b" Docker-data grow.
- main.go: -sysdata-grow / -sysdata-mount flags wired into all three call sites.
- Tests: SysDataGrow (asserts ResizeLXC mp1 +42G) + SysDataGrowZeroNoResize.
- RUNBOOK extended to the three-volume layout (32 rootfs + 200 docker + 50 user-data).

Static CT volume, NOT an enrolled drive — never enrolls/ejects/decommissions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
This commit is contained in:
authentik Default Admin
2026-06-23 16:25:14 +02:00
parent b908b9a8e5
commit 459dad954b
6 changed files with 210 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
RUNBOOK-provisioning-storage.md
+52 -26
View File
@@ -1,23 +1,38 @@
# RUNBOOK — provisioning the OS / Docker-data storage split
# RUNBOOK — provisioning the OS / Docker-data / user-data storage split
How the controller guest's **OS rootfs** and **Docker data** are carved onto separate `local-lvm`
volumes, and how to size them per node/customer. Grounded by the storage-split spike
(`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 256 GB
Docker-data**, rest of the thin pool free for other guests.
How the controller guest's **OS rootfs**, **Docker data**, and **SSD user-data** are carved onto three
separate `local-lvm` volumes, and how to size them per node/customer. Grounded by the storage-split spike
(`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 200 GB Docker-data
+ 50 GB user-data**, rest of the thin pool free for other guests.
## The design (locked)
Three thin volumes are baked into the golden and grown at provision:
| mp slot | Mount | Holds | Golden default | Provision grow knob |
|---------|-------|-------|----------------|---------------------|
| rootfs | `/` | OS only | `OS_SIZE_GB=32` | `-rootfs-grow` |
| `mp0` | `/var/lib/docker` | ALL Docker images + overlay + named volumes | `GOLDEN_DOCKER_GB=16` | `-datavol-grow` |
| `mp1` | `/mnt/sys_drive` | SSD user-data (the controller's `system_data_path`; `<sys_drive>/felhom-data` = SSD-only app data + DBs) | `GOLDEN_SYSDATA_GB=8` | `-sysdata-grow` |
- **Split for resilience, not backup size.** An isolated OS rootfs stays bootable + agent-recoverable
if the Docker volume fills. Both volumes ride into PBS regardless (so the backup size is unchanged) —
the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
if the Docker volume fills. All three volumes ride into PBS regardless (so the backup size is
unchanged) — the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
- **One Docker daemon, one data-root, one volume** at `/var/lib/docker` for ALL images + overlay
(controller, traefik, cloudflared, filebrowser, and customer apps). Infra is NOT relocated to the OS
partition — infra/user separation under one daemon isn't cleanly possible and isn't needed. Infra is
protected by the controller's **prevention layer** (reserved buffer + deploy gate + disk alerts +
baked log rotation), not by placement.
- **`backup=1` on the Docker-data volume is non-negotiable.** Extra LXC mountpoints default to
`backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs fall out of the
snapshot AND the volume (with baked images) is excluded from the golden archive.
- **`/mnt/sys_drive` is a static CT volume, NOT an enrolled drive.** It never enrolls, ejects, or
decommissions — it is part of the golden's baked layout. The controller auto-discovers
`<sys_drive>/felhom-data` and, via `system.IsMountPoint`, **warns when `/mnt/sys_drive` is not its own
mount** (`selftest.go`). Making `mp1` a real mount clears that warning with **zero controller change**;
the controller is untouched by this slice. The `mp` mounted under the guest's `/mnt` propagates into
the controller container through the existing `-v /mnt:/mnt:rslave` bind (the same path enrolled drives
use).
- **`backup=1` on the Docker-data AND user-data volumes is non-negotiable.** Extra LXC mountpoints
default to `backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs / SSD app
data fall out of the snapshot AND (for mp0) the baked images are excluded from the golden archive.
## Golden build (per controller-image baseline)
@@ -25,15 +40,20 @@ Docker-data**, rest of the thin pool free for other guests.
```
# defaults shown; override via env
OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 \
OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 GOLDEN_SYSDATA_GB=8 \
./build-golden.sh <VMID> <TEMPLATE> <ROOTFS_STORAGE> <ARCHIVE_STORAGE> <BRIDGE> <CONTROLLER_IMAGE>
```
- `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}` + `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`.
- Keep `GOLDEN_DOCKER_GB` small (just the baked images + headroom) — provision grows it.
- `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}`
+ `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`
+ `--mp1 ${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1`.
- Keep `GOLDEN_DOCKER_GB` / `GOLDEN_SYSDATA_GB` small (the golden ships them near-empty) — provision
grows both. The controller creates `<sys_drive>/felhom-data` itself once `/mnt/sys_drive` is a real
mount; the golden does NOT pre-create it.
- The script bakes `/etc/docker/daemon.json` with **`features.containerd-snapshotter: false`** (the
classic **overlay2** driver) + log rotation (`max-size 10m`, `max-file 3`), verifies `/var/lib/docker`
is a separate mount + the driver is overlay2, and **aborts if vzdump excludes mp0** (the B3 trap).
AND `/mnt/sys_drive` are separate mounts + the driver is overlay2, and **aborts if vzdump excludes mp0
or mp1** (the B3 trap).
- **Why overlay2 (not the Docker-29 default containerd-snapshotter):** containerd-snapshotter keeps the
image content store under `/var/lib/containerd`, which is NOT `/var/lib/docker` — so it would stay on
the OS rootfs and the split would only move named volumes, leaving images (the bulk) on the rootfs
@@ -43,33 +63,39 @@ OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 \
## Provision (per customer guest)
The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data volume + baked images),
then sizes per customer. The Docker-data volume is **grown** (not re-attached) so the baked images are
never shadowed:
The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data + mp1 user-data volumes +
baked images), then sizes per customer. Each volume is **grown** (not re-attached) so the baked images
are never shadowed:
```
felhom-agent --selftest=provision \
-archive <golden-volid> -vmid <VMID> \
-customer-id <id> -hub-password <retrieval-passphrase> \
-rootfs-grow <N> # GiB to add to the OS rootfs beyond the golden's 32 (0 = keep 32)
-datavol-grow <N> # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 240 → 256 total)
-datavol-grow <N> # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 184 → 200 total)
-datavol-mount mp0 # the golden's Docker-data slot (default mp0)
-sysdata-grow <N> # GiB to add to the user-data volume beyond the golden's 8 (e.g. 42 → 50 total)
-sysdata-mount mp1 # the golden's user-data slot (default mp1)
```
In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount}` — both grows are online
(`pct resize`, spike B4), each its own task. `GuestMount.Backup` (→ `,backup=1`) is set on any
data-bearing additive mount (`bringup.go` `buildConfigParams`). The bootstrap mount is `mp9`; USB/data
drive enroll picks the lowest free slot (`freeMountSlot`), so it never collides with the docker mp0.
In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount, SysDataGrowGB, SysDataMount}`
— all grows are online (`pct resize`, spike B4), each its own task. **Separateness, not size, clears the
controller warning** — `-sysdata-grow 0` still yields a separate, warning-free mount at the golden size;
the grow is an orthogonal knob. `GuestMount.Backup` (→ `,backup=1`) is set on any data-bearing additive
mount (`bringup.go` `buildBringUpConfig`). The bootstrap mount is `mp9`, the shared parent bind is `mp8`;
USB/data drive enroll picks the lowest free slot (`freeMountSlot`), which auto-skips the baked mp0/mp1.
**Per-customer sizing source:** flags today; the **slice-10 hub storage manifest** is the intended
home (marked at `bringup.go` `GuestMount` / the `slice 10 wires the hub storage manifest` comment) —
the controller already pulls customer-scoped config from the hub, so OS/data sizes belong there.
### Default sizing guidance
- ~512 GB SSD: 32 GB rootfs + 256 GB Docker-data (`-rootfs-grow 0 -datavol-grow 240`), leaving the rest
of the thin pool free for other guests.
- Other SSDs: size the data volume to "most of the free thin pool minus a reserve for other guests."
`local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's written.
- ~512 GB SSD: 32 GB rootfs + 200 GB Docker-data + 50 GB user-data
(`-rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42`), leaving the rest of the thin pool free for
other guests.
- Other SSDs: size the Docker-data + user-data volumes to "most of the free thin pool minus a reserve
for other guests." `local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's
written.
## Fresh PVE host install — thin-pool carving (host-level, do this at install)