v0.40.0: third CT volume — SSD user-data (/mnt/sys_drive, mp1) baked + -sysdata-grow

Extends the OS/Docker-data split to a three-volume layout: rootfs + Docker-data
(mp0) + SSD user-data (mp1 @ /mnt/sys_drive, backup=1) = the controller's
system_data_path. Clears the controller's "not a separate drive" warning with
zero controller change (it already auto-discovers <sys_drive>/felhom-data and
warns via system.IsMountPoint; the mp reaches the container via the existing
-v /mnt:/mnt:rslave bind).

- build-golden.sh: --mp1 ...,mp=/mnt/sys_drive,backup=1 (env GOLDEN_SYSDATA_GB=8);
  findmnt /mnt/sys_drive separate-mount guard + vzdump aborts if mp0 OR mp1 excluded.
- bringup.go: DefaultSysDataMount=mp1; BringUpSpec.{SysDataGrowGB,SysDataMount};
  new "4c" online grow-only block mirroring the "4b" Docker-data grow.
- main.go: -sysdata-grow / -sysdata-mount flags wired into all three call sites.
- Tests: SysDataGrow (asserts ResizeLXC mp1 +42G) + SysDataGrowZeroNoResize.
- RUNBOOK extended to the three-volume layout (32 rootfs + 200 docker + 50 user-data).

Static CT volume, NOT an enrolled drive — never enrolls/ejects/decommissions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
This commit is contained in:
2026-06-23 16:25:14 +02:00
parent b908b9a8e5
commit 459dad954b
6 changed files with 210 additions and 35 deletions
+31
View File
@@ -3,6 +3,37 @@
All notable changes to **felhom-agent** are recorded here. Update on every code All notable changes to **felhom-agent** are recorded here. Update on every code
change that gets pushed. change that gets pushed.
## v0.40.0 — third CT volume: SSD user-data (`/mnt/sys_drive`, mp1) baked + `-sysdata-grow` (2026-06-23)
**The third golden volume.** Extends the OS/Docker-data split (v0.29.x) to a **three-volume layout**:
rootfs + Docker-data (`mp0`) + **SSD user-data (`mp1` @ `/mnt/sys_drive`, `backup=1`)** — the
controller's `system_data_path`. Until now `/mnt/sys_drive` was a plain directory on the 32 GB OS
rootfs, so the controller correctly warned that SSD app data (`<sys_drive>/felhom-data`) lands on the
OS drive. Baking it as its own thin volume clears that warning with **zero controller change** (the
controller already auto-discovers `<sys_drive>/felhom-data` and warns via `system.IsMountPoint`); the
`mp` under the guest's `/mnt` reaches the controller container through the existing
`-v /mnt:/mnt:rslave` bind.
- **`configs/build-golden.sh`** — `pct create` gains
`--mp1 ${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1` (new env
`GOLDEN_SYSDATA_GB=8`, near-empty; provision grows it). The resilience guards are mirrored for `mp1`:
a `findmnt /mnt/sys_drive` separate-mount assertion, and the vzdump-inclusion guard now aborts if
**either** `mp0` **or** `mp1` is EXCLUDED (the B3 trap — extra mountpoints default `backup=0`). The
golden does NOT pre-create `felhom-data`; the controller does once it's a real mountpoint.
- **`internal/reconcile/bringup.go`** — `const DefaultSysDataMount = "mp1"`; `BringUpSpec` gains
`SysDataGrowGB int` + `SysDataMount string`; a new **"4c"** grow block (online, grow-only `ResizeLXC`,
its own task) mirrors the "4b" Docker-data grow. `0 = skip` (separateness comes from the golden, not
the grow — the warning clears regardless of size).
- **`cmd/felhom-agent/main.go`** — `-sysdata-grow` / `-sysdata-mount` flags (mirror
`-datavol-grow`/`-datavol-mount`); `bringUpSizing` carries them into all three bring-up/provision call
sites; `--selftest=provision` help text updated.
- **Static volume, NOT an enrolled drive.** `/mnt/sys_drive` is part of the baked golden layout; it
never enrolls/ejects/decommissions and is deliberately kept off the drive-intent machinery.
`freeMountSlot` auto-skips the baked `mp0`/`mp1` so enrolled drives never collide.
- Tests: `TestRunBringUp_StorageSplit_SysDataGrow` (asserts `ResizeLXC(vmid,"mp1","+42G")`) +
`…_SysDataGrowZeroNoResize` (0 → no mp1 resize). RUNBOOK-provisioning-storage.md extended to the
three-volume layout (default ~512 GB SSD: 32 rootfs + 200 docker-data + 50 user-data).
## v0.39.0 — DR recipe completion: live PBS coord + drop the two unfillable drive fields (2026-06-16) ## v0.39.0 — DR recipe completion: live PBS coord + drop the two unfillable drive fields (2026-06-16)
**DR-recipe agent-half completion.** A live eyeball of the demo recipe (v0.38.0) found three host-half **DR-recipe agent-half completion.** A live eyeball of the demo recipe (v0.38.0) found three host-half
+52 -26
View File
@@ -1,23 +1,38 @@
# RUNBOOK — provisioning the OS / Docker-data storage split # RUNBOOK — provisioning the OS / Docker-data / user-data storage split
How the controller guest's **OS rootfs** and **Docker data** are carved onto separate `local-lvm` How the controller guest's **OS rootfs**, **Docker data**, and **SSD user-data** are carved onto three
volumes, and how to size them per node/customer. Grounded by the storage-split spike separate `local-lvm` volumes, and how to size them per node/customer. Grounded by the storage-split spike
(`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 256 GB (`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 200 GB Docker-data
Docker-data**, rest of the thin pool free for other guests. + 50 GB user-data**, rest of the thin pool free for other guests.
## The design (locked) ## The design (locked)
Three thin volumes are baked into the golden and grown at provision:
| mp slot | Mount | Holds | Golden default | Provision grow knob |
|---------|-------|-------|----------------|---------------------|
| rootfs | `/` | OS only | `OS_SIZE_GB=32` | `-rootfs-grow` |
| `mp0` | `/var/lib/docker` | ALL Docker images + overlay + named volumes | `GOLDEN_DOCKER_GB=16` | `-datavol-grow` |
| `mp1` | `/mnt/sys_drive` | SSD user-data (the controller's `system_data_path`; `<sys_drive>/felhom-data` = SSD-only app data + DBs) | `GOLDEN_SYSDATA_GB=8` | `-sysdata-grow` |
- **Split for resilience, not backup size.** An isolated OS rootfs stays bootable + agent-recoverable - **Split for resilience, not backup size.** An isolated OS rootfs stays bootable + agent-recoverable
if the Docker volume fills. Both volumes ride into PBS regardless (so the backup size is unchanged) — if the Docker volume fills. All three volumes ride into PBS regardless (so the backup size is
the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs. unchanged) — the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
- **One Docker daemon, one data-root, one volume** at `/var/lib/docker` for ALL images + overlay - **One Docker daemon, one data-root, one volume** at `/var/lib/docker` for ALL images + overlay
(controller, traefik, cloudflared, filebrowser, and customer apps). Infra is NOT relocated to the OS (controller, traefik, cloudflared, filebrowser, and customer apps). Infra is NOT relocated to the OS
partition — infra/user separation under one daemon isn't cleanly possible and isn't needed. Infra is partition — infra/user separation under one daemon isn't cleanly possible and isn't needed. Infra is
protected by the controller's **prevention layer** (reserved buffer + deploy gate + disk alerts + protected by the controller's **prevention layer** (reserved buffer + deploy gate + disk alerts +
baked log rotation), not by placement. baked log rotation), not by placement.
- **`backup=1` on the Docker-data volume is non-negotiable.** Extra LXC mountpoints default to - **`/mnt/sys_drive` is a static CT volume, NOT an enrolled drive.** It never enrolls, ejects, or
`backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs fall out of the decommissions — it is part of the golden's baked layout. The controller auto-discovers
snapshot AND the volume (with baked images) is excluded from the golden archive. `<sys_drive>/felhom-data` and, via `system.IsMountPoint`, **warns when `/mnt/sys_drive` is not its own
mount** (`selftest.go`). Making `mp1` a real mount clears that warning with **zero controller change**;
the controller is untouched by this slice. The `mp` mounted under the guest's `/mnt` propagates into
the controller container through the existing `-v /mnt:/mnt:rslave` bind (the same path enrolled drives
use).
- **`backup=1` on the Docker-data AND user-data volumes is non-negotiable.** Extra LXC mountpoints
default to `backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs / SSD app
data fall out of the snapshot AND (for mp0) the baked images are excluded from the golden archive.
## Golden build (per controller-image baseline) ## Golden build (per controller-image baseline)
@@ -25,15 +40,20 @@ Docker-data**, rest of the thin pool free for other guests.
``` ```
# defaults shown; override via env # defaults shown; override via env
OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 \ OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 GOLDEN_SYSDATA_GB=8 \
./build-golden.sh <VMID> <TEMPLATE> <ROOTFS_STORAGE> <ARCHIVE_STORAGE> <BRIDGE> <CONTROLLER_IMAGE> ./build-golden.sh <VMID> <TEMPLATE> <ROOTFS_STORAGE> <ARCHIVE_STORAGE> <BRIDGE> <CONTROLLER_IMAGE>
``` ```
- `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}` + `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`. - `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}`
- Keep `GOLDEN_DOCKER_GB` small (just the baked images + headroom) — provision grows it. + `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`
+ `--mp1 ${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1`.
- Keep `GOLDEN_DOCKER_GB` / `GOLDEN_SYSDATA_GB` small (the golden ships them near-empty) — provision
grows both. The controller creates `<sys_drive>/felhom-data` itself once `/mnt/sys_drive` is a real
mount; the golden does NOT pre-create it.
- The script bakes `/etc/docker/daemon.json` with **`features.containerd-snapshotter: false`** (the - The script bakes `/etc/docker/daemon.json` with **`features.containerd-snapshotter: false`** (the
classic **overlay2** driver) + log rotation (`max-size 10m`, `max-file 3`), verifies `/var/lib/docker` classic **overlay2** driver) + log rotation (`max-size 10m`, `max-file 3`), verifies `/var/lib/docker`
is a separate mount + the driver is overlay2, and **aborts if vzdump excludes mp0** (the B3 trap). AND `/mnt/sys_drive` are separate mounts + the driver is overlay2, and **aborts if vzdump excludes mp0
or mp1** (the B3 trap).
- **Why overlay2 (not the Docker-29 default containerd-snapshotter):** containerd-snapshotter keeps the - **Why overlay2 (not the Docker-29 default containerd-snapshotter):** containerd-snapshotter keeps the
image content store under `/var/lib/containerd`, which is NOT `/var/lib/docker` — so it would stay on image content store under `/var/lib/containerd`, which is NOT `/var/lib/docker` — so it would stay on
the OS rootfs and the split would only move named volumes, leaving images (the bulk) on the rootfs the OS rootfs and the split would only move named volumes, leaving images (the bulk) on the rootfs
@@ -43,33 +63,39 @@ OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 \
## Provision (per customer guest) ## Provision (per customer guest)
The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data volume + baked images), The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data + mp1 user-data volumes +
then sizes per customer. The Docker-data volume is **grown** (not re-attached) so the baked images are baked images), then sizes per customer. Each volume is **grown** (not re-attached) so the baked images
never shadowed: are never shadowed:
``` ```
felhom-agent --selftest=provision \ felhom-agent --selftest=provision \
-archive <golden-volid> -vmid <VMID> \ -archive <golden-volid> -vmid <VMID> \
-customer-id <id> -hub-password <retrieval-passphrase> \ -customer-id <id> -hub-password <retrieval-passphrase> \
-rootfs-grow <N> # GiB to add to the OS rootfs beyond the golden's 32 (0 = keep 32) -rootfs-grow <N> # GiB to add to the OS rootfs beyond the golden's 32 (0 = keep 32)
-datavol-grow <N> # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 240 → 256 total) -datavol-grow <N> # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 184 → 200 total)
-datavol-mount mp0 # the golden's Docker-data slot (default mp0) -datavol-mount mp0 # the golden's Docker-data slot (default mp0)
-sysdata-grow <N> # GiB to add to the user-data volume beyond the golden's 8 (e.g. 42 → 50 total)
-sysdata-mount mp1 # the golden's user-data slot (default mp1)
``` ```
In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount}` — both grows are online In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount, SysDataGrowGB, SysDataMount}`
(`pct resize`, spike B4), each its own task. `GuestMount.Backup` (→ `,backup=1`) is set on any — all grows are online (`pct resize`, spike B4), each its own task. **Separateness, not size, clears the
data-bearing additive mount (`bringup.go` `buildConfigParams`). The bootstrap mount is `mp9`; USB/data controller warning** — `-sysdata-grow 0` still yields a separate, warning-free mount at the golden size;
drive enroll picks the lowest free slot (`freeMountSlot`), so it never collides with the docker mp0. the grow is an orthogonal knob. `GuestMount.Backup` (→ `,backup=1`) is set on any data-bearing additive
mount (`bringup.go` `buildBringUpConfig`). The bootstrap mount is `mp9`, the shared parent bind is `mp8`;
USB/data drive enroll picks the lowest free slot (`freeMountSlot`), which auto-skips the baked mp0/mp1.
**Per-customer sizing source:** flags today; the **slice-10 hub storage manifest** is the intended **Per-customer sizing source:** flags today; the **slice-10 hub storage manifest** is the intended
home (marked at `bringup.go` `GuestMount` / the `slice 10 wires the hub storage manifest` comment) — home (marked at `bringup.go` `GuestMount` / the `slice 10 wires the hub storage manifest` comment) —
the controller already pulls customer-scoped config from the hub, so OS/data sizes belong there. the controller already pulls customer-scoped config from the hub, so OS/data sizes belong there.
### Default sizing guidance ### Default sizing guidance
- ~512 GB SSD: 32 GB rootfs + 256 GB Docker-data (`-rootfs-grow 0 -datavol-grow 240`), leaving the rest - ~512 GB SSD: 32 GB rootfs + 200 GB Docker-data + 50 GB user-data
of the thin pool free for other guests. (`-rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42`), leaving the rest of the thin pool free for
- Other SSDs: size the data volume to "most of the free thin pool minus a reserve for other guests." other guests.
`local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's written. - Other SSDs: size the Docker-data + user-data volumes to "most of the free thin pool minus a reserve
for other guests." `local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's
written.
## Fresh PVE host install — thin-pool carving (host-level, do this at install) ## Fresh PVE host install — thin-pool carving (host-level, do this at install)
+14 -4
View File
@@ -44,7 +44,7 @@ import (
// version is the agent version. Overridable at build time with // version is the agent version. Overridable at build time with
// -ldflags "-X main.version=<v>"; defaults to the in-repo CHANGELOG version. // -ldflags "-X main.version=<v>"; defaults to the in-repo CHANGELOG version.
var version = "0.39.0" var version = "0.40.0"
// runGuestHook is the PVE pre-start hook body (`felhom-agent guest-hook <vmid> <phase>`). On the // runGuestHook is the PVE pre-start hook body (`felhom-agent guest-hook <vmid> <phase>`). On the
// pre-start phase it creates placeholder dirs for any absent bind-mount source so the guest always boots // pre-start phase it creates placeholder dirs for any absent bind-mount source so the guest always boots
@@ -88,6 +88,8 @@ func main() {
rootfsGrow int rootfsGrow int
dataVolGrow int dataVolGrow int
dataVolMount string dataVolMount string
sysDataGrow int
sysDataMount string
pbsStorage string pbsStorage string
paperkey bool paperkey bool
offline bool offline bool
@@ -105,7 +107,7 @@ func main() {
showVersion bool showVersion bool
) )
flag.StringVar(&cfgPath, "config", envOr("FELHOM_AGENT_CONFIG", "/etc/felhom-agent/agent.json"), "path to the agent config file (JSON)") flag.StringVar(&cfgPath, "config", envOr("FELHOM_AGENT_CONFIG", "/etc/felhom-agent/agent.json"), "path to the agent config file (JSON)")
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; keeps the guest)") flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; optional -rootfs-grow/-datavol-grow/-sysdata-grow; keeps the guest)")
flag.IntVar(&vmid, "vmid", 0, "guest VMID for --selftest=task|backup|bring-up") flag.IntVar(&vmid, "vmid", 0, "guest VMID for --selftest=task|backup|bring-up")
flag.DurationVar(&watch, "watch", 0, "for --selftest=storage: run the watchdog verbose for this duration (e.g. 3m) with the re-mount response live; 0 = observe pass only") flag.DurationVar(&watch, "watch", 0, "for --selftest=storage: run the watchdog verbose for this duration (e.g. 3m) with the re-mount response live; 0 = observe pass only")
flag.StringVar(&archive, "archive", "", "for --selftest=restore-test|bring-up: the backup volid to restore (restore-test: default newest on the local target)") flag.StringVar(&archive, "archive", "", "for --selftest=restore-test|bring-up: the backup volid to restore (restore-test: default newest on the local target)")
@@ -115,6 +117,8 @@ func main() {
flag.IntVar(&rootfsGrow, "rootfs-grow", 0, "for --selftest=bring-up|provision: grow the OS rootfs by this many GiB after restore (0 = keep golden size)") flag.IntVar(&rootfsGrow, "rootfs-grow", 0, "for --selftest=bring-up|provision: grow the OS rootfs by this many GiB after restore (0 = keep golden size)")
flag.IntVar(&dataVolGrow, "datavol-grow", 0, "for --selftest=bring-up|provision: grow the golden's Docker-data volume (mp0) by this many GiB (0 = keep golden size)") flag.IntVar(&dataVolGrow, "datavol-grow", 0, "for --selftest=bring-up|provision: grow the golden's Docker-data volume (mp0) by this many GiB (0 = keep golden size)")
flag.StringVar(&dataVolMount, "datavol-mount", "", "for --selftest=bring-up|provision: the mpN slot of the Docker-data volume to grow (default mp0)") flag.StringVar(&dataVolMount, "datavol-mount", "", "for --selftest=bring-up|provision: the mpN slot of the Docker-data volume to grow (default mp0)")
flag.IntVar(&sysDataGrow, "sysdata-grow", 0, "for --selftest=bring-up|provision: grow the golden's SSD user-data volume (mp1, /mnt/sys_drive) by this many GiB (0 = keep golden size)")
flag.StringVar(&sysDataMount, "sysdata-mount", "", "for --selftest=bring-up|provision: the mpN slot of the user-data volume to grow (default mp1)")
flag.StringVar(&pbsStorage, "storage", "", "for --selftest=escrow-create: the pbs storage whose key to escrow (default: escrow.pbs_storage_id)") flag.StringVar(&pbsStorage, "storage", "", "for --selftest=escrow-create: the pbs storage whose key to escrow (default: escrow.pbs_storage_id)")
flag.BoolVar(&paperkey, "paperkey", false, "for --selftest=escrow-create: ALSO emit the raw-key paperkey (opt-in (a); single-factor, unrevocable)") flag.BoolVar(&paperkey, "paperkey", false, "for --selftest=escrow-create: ALSO emit the raw-key paperkey (opt-in (a); single-factor, unrevocable)")
flag.BoolVar(&offline, "offline", false, "for --selftest=escrow-create: ALSO emit the R-wrapped offline copy to print (opt-in (b))") flag.BoolVar(&offline, "offline", false, "for --selftest=escrow-create: ALSO emit the R-wrapped offline copy to print (opt-in (b))")
@@ -170,12 +174,14 @@ func main() {
os.Exit(runSelftestLANResolver(context.Background(), cfg, logger, vmid)) os.Exit(runSelftestLANResolver(context.Background(), cfg, logger, vmid))
case "bring-up": case "bring-up":
os.Exit(runSelftestBringUp(context.Background(), cfg, logger, mode, archive, vmid, hostname, keep, os.Exit(runSelftestBringUp(context.Background(), cfg, logger, mode, archive, vmid, hostname, keep,
bringUpSizing{RootfsGrowGB: rootfsGrow, DataVolGrowGB: dataVolGrow, DataVolMount: dataVolMount})) bringUpSizing{RootfsGrowGB: rootfsGrow, DataVolGrowGB: dataVolGrow, DataVolMount: dataVolMount,
SysDataGrowGB: sysDataGrow, SysDataMount: sysDataMount}))
case "provision": case "provision":
os.Exit(runSelftestProvision(context.Background(), cfg, logger, provisionArgs{ os.Exit(runSelftestProvision(context.Background(), cfg, logger, provisionArgs{
archive: archive, vmid: vmid, hostname: hostname, archive: archive, vmid: vmid, hostname: hostname,
customerID: custID, hubPassword: hubPassword, customerID: custID, hubPassword: hubPassword,
sizing: bringUpSizing{RootfsGrowGB: rootfsGrow, DataVolGrowGB: dataVolGrow, DataVolMount: dataVolMount}, sizing: bringUpSizing{RootfsGrowGB: rootfsGrow, DataVolGrowGB: dataVolGrow, DataVolMount: dataVolMount,
SysDataGrowGB: sysDataGrow, SysDataMount: sysDataMount},
})) }))
case "escrow-create": case "escrow-create":
os.Exit(runSelftestEscrowCreate(context.Background(), cfg, logger, pbsStorage, paperkey, offline, upload, idBundlePath, directivePath)) os.Exit(runSelftestEscrowCreate(context.Background(), cfg, logger, pbsStorage, paperkey, offline, upload, idBundlePath, directivePath))
@@ -1060,6 +1066,8 @@ type bringUpSizing struct {
RootfsGrowGB int RootfsGrowGB int
DataVolGrowGB int DataVolGrowGB int
DataVolMount string DataVolMount string
SysDataGrowGB int
SysDataMount string
} }
func runSelftestBringUp(ctx context.Context, cfg config.Config, logger *slog.Logger, mode, archive string, vmid int, hostname string, keep bool, sizing bringUpSizing) int { func runSelftestBringUp(ctx context.Context, cfg config.Config, logger *slog.Logger, mode, archive string, vmid int, hostname string, keep bool, sizing bringUpSizing) int {
@@ -1119,6 +1127,7 @@ func runSelftestBringUp(ctx context.Context, cfg config.Config, logger *slog.Log
Mode: bmode, Archive: archive, VMID: vmid, RestoreStorage: cfg.Backup.RestoreStorage, Mode: bmode, Archive: archive, VMID: vmid, RestoreStorage: cfg.Backup.RestoreStorage,
Hostname: hostname, KeepMAC: bmode == reconcile.ModeDRGuestLoss, Hostname: hostname, KeepMAC: bmode == reconcile.ModeDRGuestLoss,
RootfsGrowGB: sizing.RootfsGrowGB, DataVolGrowGB: sizing.DataVolGrowGB, DataVolMount: sizing.DataVolMount, RootfsGrowGB: sizing.RootfsGrowGB, DataVolGrowGB: sizing.DataVolGrowGB, DataVolMount: sizing.DataVolMount,
SysDataGrowGB: sizing.SysDataGrowGB, SysDataMount: sizing.SysDataMount,
} }
fmt.Printf(" bringing up %s → vmid %d on %s …\n", archive, vmid, cfg.Backup.RestoreStorage) fmt.Printf(" bringing up %s → vmid %d on %s …\n", archive, vmid, cfg.Backup.RestoreStorage)
res := engine.RunBringUp(ctx, spec) res := engine.RunBringUp(ctx, spec)
@@ -1271,6 +1280,7 @@ func runSelftestProvision(ctx context.Context, cfg config.Config, logger *slog.L
Mode: reconcile.ModeProvision, Archive: a.archive, VMID: a.vmid, Mode: reconcile.ModeProvision, Archive: a.archive, VMID: a.vmid,
RestoreStorage: cfg.Backup.RestoreStorage, Hostname: hostname, RestoreStorage: cfg.Backup.RestoreStorage, Hostname: hostname,
RootfsGrowGB: a.sizing.RootfsGrowGB, DataVolGrowGB: a.sizing.DataVolGrowGB, DataVolMount: a.sizing.DataVolMount, RootfsGrowGB: a.sizing.RootfsGrowGB, DataVolGrowGB: a.sizing.DataVolGrowGB, DataVolMount: a.sizing.DataVolMount,
SysDataGrowGB: a.sizing.SysDataGrowGB, SysDataMount: a.sizing.SysDataMount,
}) })
if res.Err != nil || !res.Pass { if res.Err != nil || !res.Pass {
fmt.Fprintf(os.Stderr, " [FAIL] front-half bring-up (vmid %d): %v\n", a.vmid, res.Err) fmt.Fprintf(os.Stderr, " [FAIL] front-half bring-up (vmid %d): %v\n", a.vmid, res.Err)
+23 -5
View File
@@ -46,13 +46,20 @@ REGISTRY_HOST="${CONTROLLER_IMAGE%%/*}"
# large enough for the baked images + headroom; provision grows it to the per-customer target. # large enough for the baked images + headroom; provision grows it to the per-customer target.
OS_SIZE_GB="${OS_SIZE_GB:-32}" OS_SIZE_GB="${OS_SIZE_GB:-32}"
GOLDEN_DOCKER_GB="${GOLDEN_DOCKER_GB:-16}" GOLDEN_DOCKER_GB="${GOLDEN_DOCKER_GB:-16}"
# The golden's SSD user-data volume (GiB) mounted at /mnt/sys_drive (mp1, backup=1) — the controller's
# system_data_path. Ships small + near-empty (the controller creates <sys_drive>/felhom-data itself once
# it's a real mountpoint); provision GROWS it to the per-customer target (bringup.go SysDataGrowGB). Like
# mp0, backup=1 is MANDATORY: without it vzdump EXCLUDES the volume (extra mountpoints default backup=0 —
# storage-split B3) and the user-data area would silently fall out of PBS coverage.
GOLDEN_SYSDATA_GB="${GOLDEN_SYSDATA_GB:-8}"
echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G @ /var/lib/docker, backup=1) …" echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G @ /var/lib/docker + user-data ${GOLDEN_SYSDATA_GB}G @ /mnt/sys_drive, both backup=1) …"
pct create "$VMID" "$TEMPLATE" \ pct create "$VMID" "$TEMPLATE" \
--hostname felhom-golden --unprivileged 1 \ --hostname felhom-golden --unprivileged 1 \
--features nesting=1,keyctl=1 \ --features nesting=1,keyctl=1 \
--rootfs "${ROOTFS_STORAGE}:${OS_SIZE_GB}" --cores 2 --memory 2048 \ --rootfs "${ROOTFS_STORAGE}:${OS_SIZE_GB}" --cores 2 --memory 2048 \
--mp0 "${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1" \ --mp0 "${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1" \
--mp1 "${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1" \
--net0 "name=eth0,bridge=${BRIDGE},ip=dhcp" --onboot 0 --net0 "name=eth0,bridge=${BRIDGE},ip=dhcp" --onboot 0
echo "[golden] starting + installing Docker (official repo, trixie channel) …" echo "[golden] starting + installing Docker (official repo, trixie channel) …"
@@ -99,6 +106,10 @@ pct exec "$VMID" -- bash -c 'systemctl restart docker; sleep 3; docker run --rm
pct exec "$VMID" -- bash -c 'drv=$(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); [ "$drv" = "overlay2" ] || { echo "[golden] FATAL: storage driver is $drv, expected overlay2 — images would not land on the data volume"; exit 1; }' pct exec "$VMID" -- bash -c 'drv=$(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); [ "$drv" = "overlay2" ] || { echo "[golden] FATAL: storage driver is $drv, expected overlay2 — images would not land on the data volume"; exit 1; }'
# Confirm /var/lib/docker is genuinely the dedicated volume, not the rootfs (catch a silent mp miss). # Confirm /var/lib/docker is genuinely the dedicated volume, not the rootfs (catch a silent mp miss).
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo " /var/lib/docker is a separate mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker)" || { echo "[golden] FATAL: /var/lib/docker is NOT a separate mount — the mp0 split did not take"; exit 1; }' pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo " /var/lib/docker is a separate mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker)" || { echo "[golden] FATAL: /var/lib/docker is NOT a separate mount — the mp0 split did not take"; exit 1; }'
# Same guard for the SSD user-data volume (mp1): /mnt/sys_drive must be its own mount, not the rootfs
# device — otherwise the controller's system_data_path lands on the OS drive and it warns (the whole
# point of this volume is to clear that warning).
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . && echo " /mnt/sys_drive is a separate mount: $(findmnt -no SOURCE,FSTYPE /mnt/sys_drive)" || { echo "[golden] FATAL: /mnt/sys_drive is NOT a separate mount — the mp1 split did not take"; exit 1; }'
echo "[golden] baking the in-guest controller image $CONTROLLER_IMAGE (no registry cred at deploy) …" echo "[golden] baking the in-guest controller image $CONTROLLER_IMAGE (no registry cred at deploy) …"
# docker login is used ONCE here on the trusted build host, then logged out before archiving so # docker login is used ONCE here on the trusted build host, then logged out before archiving so
@@ -235,17 +246,24 @@ pct exec "$VMID" -- bash -c '
echo "[golden] stop + archive …" echo "[golden] stop + archive …"
pct stop "$VMID" pct stop "$VMID"
# --mode stop with mp0 backup=1 → the Docker-data volume (with baked images) is INCLUDED. The log # --mode stop with mp0 + mp1 backup=1 → BOTH the Docker-data volume (baked images) and the
# below MUST show "including mount point mp0" — if it shows "excluding … (disabled)" the backup flag # /mnt/sys_drive user-data volume are INCLUDED. The log below MUST show "including mount point mp0"
# was lost and the archive carries no images (storage-split B3 trap). # AND "including mount point mp1" — if either shows "excluding … (disabled)" the backup flag was lost
# and the archive carries no images / no user-data volume (storage-split B3 trap).
vzdump "$VMID" --storage "$ARCHIVE_STORAGE" --mode stop --compress zstd 2>&1 | tee /tmp/golden-vzdump.log | grep -iE "including mount point|excluding|archive file size|Finished Backup" || true vzdump "$VMID" --storage "$ARCHIVE_STORAGE" --mode stop --compress zstd 2>&1 | tee /tmp/golden-vzdump.log | grep -iE "including mount point|excluding|archive file size|Finished Backup" || true
if grep -q "excluding volume mount point mp0" /tmp/golden-vzdump.log; then if grep -q "excluding volume mount point mp0" /tmp/golden-vzdump.log; then
echo "[golden] FATAL: mp0 (/var/lib/docker) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no images. Aborting." echo "[golden] FATAL: mp0 (/var/lib/docker) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no images. Aborting."
exit 1 exit 1
fi fi
if grep -q "excluding volume mount point mp1" /tmp/golden-vzdump.log; then
echo "[golden] FATAL: mp1 (/mnt/sys_drive) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no user-data volume. Aborting."
exit 1
fi
grep -q "including mount point mp0" /tmp/golden-vzdump.log \ grep -q "including mount point mp0" /tmp/golden-vzdump.log \
|| echo "[golden] WARN: could not confirm mp0 inclusion in the vzdump log — verify manually before using this archive." || echo "[golden] WARN: could not confirm mp0 inclusion in the vzdump log — verify manually before using this archive."
grep -q "including mount point mp1" /tmp/golden-vzdump.log \
|| echo "[golden] WARN: could not confirm mp1 inclusion in the vzdump log — verify manually before using this archive."
VOLID=$(pvesm list "$ARCHIVE_STORAGE" --content backup 2>/dev/null | awk -v v="$VMID" '$1 ~ ("vzdump-lxc-" v "-") {print $1}' | sort | tail -1) VOLID=$(pvesm list "$ARCHIVE_STORAGE" --content backup 2>/dev/null | awk -v v="$VMID" '$1 ~ ("vzdump-lxc-" v "-") {print $1}' | sort | tail -1)
echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G, both in the archive)" echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G + user-data ${GOLDEN_SYSDATA_GB}G, all in the archive)"
echo "[golden] (the build guest $VMID is stopped; destroy it with: pct destroy $VMID --purge)" echo "[golden] (the build guest $VMID is stopped; destroy it with: pct destroy $VMID --purge)"
+32
View File
@@ -43,6 +43,11 @@ const bringUpKind = "bring_up"
// DefaultDataVolMount is the mpN slot the golden bakes the Docker-data volume (/var/lib/docker) at. // DefaultDataVolMount is the mpN slot the golden bakes the Docker-data volume (/var/lib/docker) at.
const DefaultDataVolMount = "mp0" const DefaultDataVolMount = "mp0"
// DefaultSysDataMount is the mpN slot the golden bakes the SSD user-data volume (/mnt/sys_drive) at.
// This is the controller's system_data_path; provision grows it (SysDataGrowGB) like the Docker-data
// volume. mp1 is the natural next bring-up slot (mp8/mp9 are added by the provision back-half).
const DefaultSysDataMount = "mp1"
// configLockMaxAttempts bounds the F4 config-lock retry. configLockBackoff is a package var so // configLockMaxAttempts bounds the F4 config-lock retry. configLockBackoff is a package var so
// tests can shrink it (the production value gives PVE time to release its async config lock). // tests can shrink it (the production value gives PVE time to release its async config lock).
const configLockMaxAttempts = 5 const configLockMaxAttempts = 5
@@ -81,6 +86,13 @@ type BringUpSpec struct {
DataVolGrowGB int DataVolGrowGB int
// DataVolMount is the mpN slot of the golden's Docker-data volume to grow; "" → DefaultDataVolMount ("mp0"). // DataVolMount is the mpN slot of the golden's Docker-data volume to grow; "" → DefaultDataVolMount ("mp0").
DataVolMount string DataVolMount string
// SysDataGrowGB grows the golden-carried SSD user-data volume (SysDataMount, default mp1, mounted at
// /mnt/sys_drive = the controller's system_data_path) to the per-customer target. Same online,
// grow-only mechanism as DataVolGrowGB. 0 = skip (keep the golden's small size — the volume is still
// a separate mount, so the controller's "not a separate drive" warning clears regardless of grow).
SysDataGrowGB int
// SysDataMount is the mpN slot of the golden's user-data volume to grow; "" → DefaultSysDataMount ("mp1").
SysDataMount string
Mounts []GuestMount // additive mpN mounts (slice 7 may pass empty/test) Mounts []GuestMount // additive mpN mounts (slice 7 may pass empty/test)
KeepMAC bool // DR knob: keep the archived MAC (true) unless a source may be live KeepMAC bool // DR knob: keep the archived MAC (true) unless a source may be live
BootTimeout time.Duration // 0 → DefaultBootTimeout; bounds the link-up liveness wait BootTimeout time.Duration // 0 → DefaultBootTimeout; bounds the link-up liveness wait
@@ -249,6 +261,26 @@ func (e *Engine) runBringUp(ctx context.Context, spec BringUpSpec, res *BringUpR
} }
} }
// 4c. Grow the golden-carried SSD user-data volume (mp1, /mnt/sys_drive = the controller's
// system_data_path) to the per-customer target. Same shape as the Docker-data grow: grow-only,
// online, its OWN call. The volume came in with the restore (separate mount, backup=1), so we
// grow it rather than attach a fresh one.
if spec.SysDataGrowGB > 0 {
mount := spec.SysDataMount
if mount == "" {
mount = DefaultSysDataMount
}
supid, err := e.api.ResizeLXC(ctx, spec.VMID, mount, fmt.Sprintf("+%dG", spec.SysDataGrowGB))
if err != nil {
res.Err = fmt.Errorf("reconcile: bring-up sys-data resize (%s): %w", mount, err)
return
}
if _, err := e.waitTask(ctx, supid, proxmox.WaitOptions{}); err != nil {
res.Err = fmt.Errorf("reconcile: bring-up sys-data resize task (%s): %w", mount, err)
return
}
}
// Capture the post-reset MAC for the result (fresh for provision; archived for DR keep). // Capture the post-reset MAC for the result (fresh for provision; archived for DR keep).
if cfg2, err := e.api.GuestConfig(ctx, spec.VMID); err == nil { if cfg2, err := e.api.GuestConfig(ctx, spec.VMID); err == nil {
res.AssignedMAC = net0MAC(cfg2) res.AssignedMAC = net0MAC(cfg2)
+58
View File
@@ -126,6 +126,64 @@ func TestRunBringUp_StorageSplit_DataVolGrow(t *testing.T) {
} }
} }
// The golden-carried SSD user-data volume (/mnt/sys_drive) is grown via a SEPARATE resize on its
// mpN slot (mp1), independent of the rootfs and Docker-data grows. With SysDataGrowGB=0 NO mp1
// resize is issued (the volume stays at the golden size, still a separate mount).
func TestRunBringUp_StorageSplit_SysDataGrow(t *testing.T) {
const vmid = 8051
api := &fakeAPI{cfg: map[int]proxmox.GuestConfig{vmid: scratchCfg()}}
e, _, q := newEngine(t, api, EmptyProvider{})
defer q.Close()
res := e.RunBringUp(context.Background(), BringUpSpec{
Mode: ModeProvision, Archive: "local:backup/golden.tar.zst", VMID: vmid,
RestoreStorage: "local-lvm", Hostname: "felhom-prov-8051",
DataVolGrowGB: 240, SysDataGrowGB: 42, // grows mp0 AND mp1 (DefaultSysDataMount)
})
if res.Err != nil || !res.Pass {
t.Fatalf("provision must pass, got %+v", res)
}
// TWO resizes here: Docker-data mp0 +240G and the user-data volume mp1 +42G (no rootfs grow).
if len(api.resizes) != 2 {
t.Fatalf("expected data-volume + sys-data resizes, got %+v", api.resizes)
}
var sawData, sawSys bool
for _, r := range api.resizes {
if r.disk == "mp0" && r.size == "+240G" {
sawData = true
}
if r.disk == "mp1" && r.size == "+42G" {
sawSys = true
}
}
if !sawData || !sawSys {
t.Errorf("want mp0 +240G AND mp1 +42G, got %+v", api.resizes)
}
}
// SysDataGrowGB=0 must issue NO mp1 resize (grow is an orthogonal knob; separateness comes from the
// golden, not the grow).
func TestRunBringUp_StorageSplit_SysDataGrowZeroNoResize(t *testing.T) {
const vmid = 8052
api := &fakeAPI{cfg: map[int]proxmox.GuestConfig{vmid: scratchCfg()}}
e, _, q := newEngine(t, api, EmptyProvider{})
defer q.Close()
res := e.RunBringUp(context.Background(), BringUpSpec{
Mode: ModeProvision, Archive: "local:backup/golden.tar.zst", VMID: vmid,
RestoreStorage: "local-lvm", Hostname: "felhom-prov-8052",
SysDataGrowGB: 0, // no sys-data grow
})
if res.Err != nil || !res.Pass {
t.Fatalf("provision must pass, got %+v", res)
}
for _, r := range api.resizes {
if r.disk == "mp1" {
t.Errorf("SysDataGrowGB=0 must NOT resize mp1, got %+v", api.resizes)
}
}
}
func TestRunBringUp_CompensatingRollback(t *testing.T) { func TestRunBringUp_CompensatingRollback(t *testing.T) {
const vmid = 8000 const vmid = 8000
lockBackoffFast(t) lockBackoffFast(t)