v0.40.0: third CT volume — SSD user-data (/mnt/sys_drive, mp1) baked + -sysdata-grow

Extends the OS/Docker-data split to a three-volume layout: rootfs + Docker-data
(mp0) + SSD user-data (mp1 @ /mnt/sys_drive, backup=1) = the controller's
system_data_path. Clears the controller's "not a separate drive" warning with
zero controller change (it already auto-discovers <sys_drive>/felhom-data and
warns via system.IsMountPoint; the mp reaches the container via the existing
-v /mnt:/mnt:rslave bind).

- build-golden.sh: --mp1 ...,mp=/mnt/sys_drive,backup=1 (env GOLDEN_SYSDATA_GB=8);
  findmnt /mnt/sys_drive separate-mount guard + vzdump aborts if mp0 OR mp1 excluded.
- bringup.go: DefaultSysDataMount=mp1; BringUpSpec.{SysDataGrowGB,SysDataMount};
  new "4c" online grow-only block mirroring the "4b" Docker-data grow.
- main.go: -sysdata-grow / -sysdata-mount flags wired into all three call sites.
- Tests: SysDataGrow (asserts ResizeLXC mp1 +42G) + SysDataGrowZeroNoResize.
- RUNBOOK extended to the three-volume layout (32 rootfs + 200 docker + 50 user-data).

Static CT volume, NOT an enrolled drive — never enrolls/ejects/decommissions.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
This commit is contained in:
2026-06-23 16:25:14 +02:00
parent b908b9a8e5
commit 459dad954b
6 changed files with 210 additions and 35 deletions
+31
View File
@@ -3,6 +3,37 @@
All notable changes to **felhom-agent** are recorded here. Update on every code
change that gets pushed.
## v0.40.0 — third CT volume: SSD user-data (`/mnt/sys_drive`, mp1) baked + `-sysdata-grow` (2026-06-23)
**The third golden volume.** Extends the OS/Docker-data split (v0.29.x) to a **three-volume layout**:
rootfs + Docker-data (`mp0`) + **SSD user-data (`mp1` @ `/mnt/sys_drive`, `backup=1`)** — the
controller's `system_data_path`. Until now `/mnt/sys_drive` was a plain directory on the 32 GB OS
rootfs, so the controller correctly warned that SSD app data (`<sys_drive>/felhom-data`) lands on the
OS drive. Baking it as its own thin volume clears that warning with **zero controller change** (the
controller already auto-discovers `<sys_drive>/felhom-data` and warns via `system.IsMountPoint`); the
`mp` under the guest's `/mnt` reaches the controller container through the existing
`-v /mnt:/mnt:rslave` bind.
- **`configs/build-golden.sh`** — `pct create` gains
`--mp1 ${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1` (new env
`GOLDEN_SYSDATA_GB=8`, near-empty; provision grows it). The resilience guards are mirrored for `mp1`:
a `findmnt /mnt/sys_drive` separate-mount assertion, and the vzdump-inclusion guard now aborts if
**either** `mp0` **or** `mp1` is EXCLUDED (the B3 trap — extra mountpoints default `backup=0`). The
golden does NOT pre-create `felhom-data`; the controller does once it's a real mountpoint.
- **`internal/reconcile/bringup.go`** — `const DefaultSysDataMount = "mp1"`; `BringUpSpec` gains
`SysDataGrowGB int` + `SysDataMount string`; a new **"4c"** grow block (online, grow-only `ResizeLXC`,
its own task) mirrors the "4b" Docker-data grow. `0 = skip` (separateness comes from the golden, not
the grow — the warning clears regardless of size).
- **`cmd/felhom-agent/main.go`** — `-sysdata-grow` / `-sysdata-mount` flags (mirror
`-datavol-grow`/`-datavol-mount`); `bringUpSizing` carries them into all three bring-up/provision call
sites; `--selftest=provision` help text updated.
- **Static volume, NOT an enrolled drive.** `/mnt/sys_drive` is part of the baked golden layout; it
never enrolls/ejects/decommissions and is deliberately kept off the drive-intent machinery.
`freeMountSlot` auto-skips the baked `mp0`/`mp1` so enrolled drives never collide.
- Tests: `TestRunBringUp_StorageSplit_SysDataGrow` (asserts `ResizeLXC(vmid,"mp1","+42G")`) +
`…_SysDataGrowZeroNoResize` (0 → no mp1 resize). RUNBOOK-provisioning-storage.md extended to the
three-volume layout (default ~512 GB SSD: 32 rootfs + 200 docker-data + 50 user-data).
## v0.39.0 — DR recipe completion: live PBS coord + drop the two unfillable drive fields (2026-06-16)
**DR-recipe agent-half completion.** A live eyeball of the demo recipe (v0.38.0) found three host-half
+52 -26
View File
@@ -1,23 +1,38 @@
# RUNBOOK — provisioning the OS / Docker-data storage split
# RUNBOOK — provisioning the OS / Docker-data / user-data storage split
How the controller guest's **OS rootfs** and **Docker data** are carved onto separate `local-lvm`
volumes, and how to size them per node/customer. Grounded by the storage-split spike
(`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 256 GB
Docker-data**, rest of the thin pool free for other guests.
How the controller guest's **OS rootfs**, **Docker data**, and **SSD user-data** are carved onto three
separate `local-lvm` volumes, and how to size them per node/customer. Grounded by the storage-split spike
(`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 200 GB Docker-data
+ 50 GB user-data**, rest of the thin pool free for other guests.
## The design (locked)
Three thin volumes are baked into the golden and grown at provision:
| mp slot | Mount | Holds | Golden default | Provision grow knob |
|---------|-------|-------|----------------|---------------------|
| rootfs | `/` | OS only | `OS_SIZE_GB=32` | `-rootfs-grow` |
| `mp0` | `/var/lib/docker` | ALL Docker images + overlay + named volumes | `GOLDEN_DOCKER_GB=16` | `-datavol-grow` |
| `mp1` | `/mnt/sys_drive` | SSD user-data (the controller's `system_data_path`; `<sys_drive>/felhom-data` = SSD-only app data + DBs) | `GOLDEN_SYSDATA_GB=8` | `-sysdata-grow` |
- **Split for resilience, not backup size.** An isolated OS rootfs stays bootable + agent-recoverable
if the Docker volume fills. Both volumes ride into PBS regardless (so the backup size is unchanged) —
the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
if the Docker volume fills. All three volumes ride into PBS regardless (so the backup size is
unchanged) — the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
- **One Docker daemon, one data-root, one volume** at `/var/lib/docker` for ALL images + overlay
(controller, traefik, cloudflared, filebrowser, and customer apps). Infra is NOT relocated to the OS
partition — infra/user separation under one daemon isn't cleanly possible and isn't needed. Infra is
protected by the controller's **prevention layer** (reserved buffer + deploy gate + disk alerts +
baked log rotation), not by placement.
- **`backup=1` on the Docker-data volume is non-negotiable.** Extra LXC mountpoints default to
`backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs fall out of the
snapshot AND the volume (with baked images) is excluded from the golden archive.
- **`/mnt/sys_drive` is a static CT volume, NOT an enrolled drive.** It never enrolls, ejects, or
decommissions — it is part of the golden's baked layout. The controller auto-discovers
`<sys_drive>/felhom-data` and, via `system.IsMountPoint`, **warns when `/mnt/sys_drive` is not its own
mount** (`selftest.go`). Making `mp1` a real mount clears that warning with **zero controller change**;
the controller is untouched by this slice. The `mp` mounted under the guest's `/mnt` propagates into
the controller container through the existing `-v /mnt:/mnt:rslave` bind (the same path enrolled drives
use).
- **`backup=1` on the Docker-data AND user-data volumes is non-negotiable.** Extra LXC mountpoints
default to `backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs / SSD app
data fall out of the snapshot AND (for mp0) the baked images are excluded from the golden archive.
## Golden build (per controller-image baseline)
@@ -25,15 +40,20 @@ Docker-data**, rest of the thin pool free for other guests.
```
# defaults shown; override via env
OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 \
OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 GOLDEN_SYSDATA_GB=8 \
./build-golden.sh <VMID> <TEMPLATE> <ROOTFS_STORAGE> <ARCHIVE_STORAGE> <BRIDGE> <CONTROLLER_IMAGE>
```
- `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}` + `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`.
- Keep `GOLDEN_DOCKER_GB` small (just the baked images + headroom) — provision grows it.
- `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}`
+ `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`
+ `--mp1 ${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1`.
- Keep `GOLDEN_DOCKER_GB` / `GOLDEN_SYSDATA_GB` small (the golden ships them near-empty) — provision
grows both. The controller creates `<sys_drive>/felhom-data` itself once `/mnt/sys_drive` is a real
mount; the golden does NOT pre-create it.
- The script bakes `/etc/docker/daemon.json` with **`features.containerd-snapshotter: false`** (the
classic **overlay2** driver) + log rotation (`max-size 10m`, `max-file 3`), verifies `/var/lib/docker`
is a separate mount + the driver is overlay2, and **aborts if vzdump excludes mp0** (the B3 trap).
AND `/mnt/sys_drive` are separate mounts + the driver is overlay2, and **aborts if vzdump excludes mp0
or mp1** (the B3 trap).
- **Why overlay2 (not the Docker-29 default containerd-snapshotter):** containerd-snapshotter keeps the
image content store under `/var/lib/containerd`, which is NOT `/var/lib/docker` — so it would stay on
the OS rootfs and the split would only move named volumes, leaving images (the bulk) on the rootfs
@@ -43,33 +63,39 @@ OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 \
## Provision (per customer guest)
The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data volume + baked images),
then sizes per customer. The Docker-data volume is **grown** (not re-attached) so the baked images are
never shadowed:
The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data + mp1 user-data volumes +
baked images), then sizes per customer. Each volume is **grown** (not re-attached) so the baked images
are never shadowed:
```
felhom-agent --selftest=provision \
-archive <golden-volid> -vmid <VMID> \
-customer-id <id> -hub-password <retrieval-passphrase> \
-rootfs-grow <N> # GiB to add to the OS rootfs beyond the golden's 32 (0 = keep 32)
-datavol-grow <N> # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 240 → 256 total)
-datavol-grow <N> # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 184 → 200 total)
-datavol-mount mp0 # the golden's Docker-data slot (default mp0)
-sysdata-grow <N> # GiB to add to the user-data volume beyond the golden's 8 (e.g. 42 → 50 total)
-sysdata-mount mp1 # the golden's user-data slot (default mp1)
```
In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount}` — both grows are online
(`pct resize`, spike B4), each its own task. `GuestMount.Backup` (→ `,backup=1`) is set on any
data-bearing additive mount (`bringup.go` `buildConfigParams`). The bootstrap mount is `mp9`; USB/data
drive enroll picks the lowest free slot (`freeMountSlot`), so it never collides with the docker mp0.
In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount, SysDataGrowGB, SysDataMount}`
— all grows are online (`pct resize`, spike B4), each its own task. **Separateness, not size, clears the
controller warning** — `-sysdata-grow 0` still yields a separate, warning-free mount at the golden size;
the grow is an orthogonal knob. `GuestMount.Backup` (→ `,backup=1`) is set on any data-bearing additive
mount (`bringup.go` `buildBringUpConfig`). The bootstrap mount is `mp9`, the shared parent bind is `mp8`;
USB/data drive enroll picks the lowest free slot (`freeMountSlot`), which auto-skips the baked mp0/mp1.
**Per-customer sizing source:** flags today; the **slice-10 hub storage manifest** is the intended
home (marked at `bringup.go` `GuestMount` / the `slice 10 wires the hub storage manifest` comment) —
the controller already pulls customer-scoped config from the hub, so OS/data sizes belong there.
### Default sizing guidance
- ~512 GB SSD: 32 GB rootfs + 256 GB Docker-data (`-rootfs-grow 0 -datavol-grow 240`), leaving the rest
of the thin pool free for other guests.
- Other SSDs: size the data volume to "most of the free thin pool minus a reserve for other guests."
`local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's written.
- ~512 GB SSD: 32 GB rootfs + 200 GB Docker-data + 50 GB user-data
(`-rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42`), leaving the rest of the thin pool free for
other guests.
- Other SSDs: size the Docker-data + user-data volumes to "most of the free thin pool minus a reserve
for other guests." `local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's
written.
## Fresh PVE host install — thin-pool carving (host-level, do this at install)
+14 -4
View File
@@ -44,7 +44,7 @@ import (
// version is the agent version. Overridable at build time with
// -ldflags "-X main.version=<v>"; defaults to the in-repo CHANGELOG version.
var version = "0.39.0"
var version = "0.40.0"
// runGuestHook is the PVE pre-start hook body (`felhom-agent guest-hook <vmid> <phase>`). On the
// pre-start phase it creates placeholder dirs for any absent bind-mount source so the guest always boots
@@ -88,6 +88,8 @@ func main() {
rootfsGrow int
dataVolGrow int
dataVolMount string
sysDataGrow int
sysDataMount string
pbsStorage string
paperkey bool
offline bool
@@ -105,7 +107,7 @@ func main() {
showVersion bool
)
flag.StringVar(&cfgPath, "config", envOr("FELHOM_AGENT_CONFIG", "/etc/felhom-agent/agent.json"), "path to the agent config file (JSON)")
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; keeps the guest)")
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; optional -rootfs-grow/-datavol-grow/-sysdata-grow; keeps the guest)")
flag.IntVar(&vmid, "vmid", 0, "guest VMID for --selftest=task|backup|bring-up")
flag.DurationVar(&watch, "watch", 0, "for --selftest=storage: run the watchdog verbose for this duration (e.g. 3m) with the re-mount response live; 0 = observe pass only")
flag.StringVar(&archive, "archive", "", "for --selftest=restore-test|bring-up: the backup volid to restore (restore-test: default newest on the local target)")
@@ -115,6 +117,8 @@ func main() {
flag.IntVar(&rootfsGrow, "rootfs-grow", 0, "for --selftest=bring-up|provision: grow the OS rootfs by this many GiB after restore (0 = keep golden size)")
flag.IntVar(&dataVolGrow, "datavol-grow", 0, "for --selftest=bring-up|provision: grow the golden's Docker-data volume (mp0) by this many GiB (0 = keep golden size)")
flag.StringVar(&dataVolMount, "datavol-mount", "", "for --selftest=bring-up|provision: the mpN slot of the Docker-data volume to grow (default mp0)")
flag.IntVar(&sysDataGrow, "sysdata-grow", 0, "for --selftest=bring-up|provision: grow the golden's SSD user-data volume (mp1, /mnt/sys_drive) by this many GiB (0 = keep golden size)")
flag.StringVar(&sysDataMount, "sysdata-mount", "", "for --selftest=bring-up|provision: the mpN slot of the user-data volume to grow (default mp1)")
flag.StringVar(&pbsStorage, "storage", "", "for --selftest=escrow-create: the pbs storage whose key to escrow (default: escrow.pbs_storage_id)")
flag.BoolVar(&paperkey, "paperkey", false, "for --selftest=escrow-create: ALSO emit the raw-key paperkey (opt-in (a); single-factor, unrevocable)")
flag.BoolVar(&offline, "offline", false, "for --selftest=escrow-create: ALSO emit the R-wrapped offline copy to print (opt-in (b))")
@@ -170,12 +174,14 @@ func main() {
os.Exit(runSelftestLANResolver(context.Background(), cfg, logger, vmid))
case "bring-up":
os.Exit(runSelftestBringUp(context.Background(), cfg, logger, mode, archive, vmid, hostname, keep,
bringUpSizing{RootfsGrowGB: rootfsGrow, DataVolGrowGB: dataVolGrow, DataVolMount: dataVolMount}))
bringUpSizing{RootfsGrowGB: rootfsGrow, DataVolGrowGB: dataVolGrow, DataVolMount: dataVolMount,
SysDataGrowGB: sysDataGrow, SysDataMount: sysDataMount}))
case "provision":
os.Exit(runSelftestProvision(context.Background(), cfg, logger, provisionArgs{
archive: archive, vmid: vmid, hostname: hostname,
customerID: custID, hubPassword: hubPassword,
sizing: bringUpSizing{RootfsGrowGB: rootfsGrow, DataVolGrowGB: dataVolGrow, DataVolMount: dataVolMount},
sizing: bringUpSizing{RootfsGrowGB: rootfsGrow, DataVolGrowGB: dataVolGrow, DataVolMount: dataVolMount,
SysDataGrowGB: sysDataGrow, SysDataMount: sysDataMount},
}))
case "escrow-create":
os.Exit(runSelftestEscrowCreate(context.Background(), cfg, logger, pbsStorage, paperkey, offline, upload, idBundlePath, directivePath))
@@ -1060,6 +1066,8 @@ type bringUpSizing struct {
RootfsGrowGB int
DataVolGrowGB int
DataVolMount string
SysDataGrowGB int
SysDataMount string
}
func runSelftestBringUp(ctx context.Context, cfg config.Config, logger *slog.Logger, mode, archive string, vmid int, hostname string, keep bool, sizing bringUpSizing) int {
@@ -1119,6 +1127,7 @@ func runSelftestBringUp(ctx context.Context, cfg config.Config, logger *slog.Log
Mode: bmode, Archive: archive, VMID: vmid, RestoreStorage: cfg.Backup.RestoreStorage,
Hostname: hostname, KeepMAC: bmode == reconcile.ModeDRGuestLoss,
RootfsGrowGB: sizing.RootfsGrowGB, DataVolGrowGB: sizing.DataVolGrowGB, DataVolMount: sizing.DataVolMount,
SysDataGrowGB: sizing.SysDataGrowGB, SysDataMount: sizing.SysDataMount,
}
fmt.Printf(" bringing up %s → vmid %d on %s …\n", archive, vmid, cfg.Backup.RestoreStorage)
res := engine.RunBringUp(ctx, spec)
@@ -1271,6 +1280,7 @@ func runSelftestProvision(ctx context.Context, cfg config.Config, logger *slog.L
Mode: reconcile.ModeProvision, Archive: a.archive, VMID: a.vmid,
RestoreStorage: cfg.Backup.RestoreStorage, Hostname: hostname,
RootfsGrowGB: a.sizing.RootfsGrowGB, DataVolGrowGB: a.sizing.DataVolGrowGB, DataVolMount: a.sizing.DataVolMount,
SysDataGrowGB: a.sizing.SysDataGrowGB, SysDataMount: a.sizing.SysDataMount,
})
if res.Err != nil || !res.Pass {
fmt.Fprintf(os.Stderr, " [FAIL] front-half bring-up (vmid %d): %v\n", a.vmid, res.Err)
+23 -5
View File
@@ -46,13 +46,20 @@ REGISTRY_HOST="${CONTROLLER_IMAGE%%/*}"
# large enough for the baked images + headroom; provision grows it to the per-customer target.
OS_SIZE_GB="${OS_SIZE_GB:-32}"
GOLDEN_DOCKER_GB="${GOLDEN_DOCKER_GB:-16}"
# The golden's SSD user-data volume (GiB) mounted at /mnt/sys_drive (mp1, backup=1) — the controller's
# system_data_path. Ships small + near-empty (the controller creates <sys_drive>/felhom-data itself once
# it's a real mountpoint); provision GROWS it to the per-customer target (bringup.go SysDataGrowGB). Like
# mp0, backup=1 is MANDATORY: without it vzdump EXCLUDES the volume (extra mountpoints default backup=0 —
# storage-split B3) and the user-data area would silently fall out of PBS coverage.
GOLDEN_SYSDATA_GB="${GOLDEN_SYSDATA_GB:-8}"
echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G @ /var/lib/docker, backup=1) …"
echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G @ /var/lib/docker + user-data ${GOLDEN_SYSDATA_GB}G @ /mnt/sys_drive, both backup=1) …"
pct create "$VMID" "$TEMPLATE" \
--hostname felhom-golden --unprivileged 1 \
--features nesting=1,keyctl=1 \
--rootfs "${ROOTFS_STORAGE}:${OS_SIZE_GB}" --cores 2 --memory 2048 \
--mp0 "${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1" \
--mp1 "${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1" \
--net0 "name=eth0,bridge=${BRIDGE},ip=dhcp" --onboot 0
echo "[golden] starting + installing Docker (official repo, trixie channel) …"
@@ -99,6 +106,10 @@ pct exec "$VMID" -- bash -c 'systemctl restart docker; sleep 3; docker run --rm
pct exec "$VMID" -- bash -c 'drv=$(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); [ "$drv" = "overlay2" ] || { echo "[golden] FATAL: storage driver is $drv, expected overlay2 — images would not land on the data volume"; exit 1; }'
# Confirm /var/lib/docker is genuinely the dedicated volume, not the rootfs (catch a silent mp miss).
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo " /var/lib/docker is a separate mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker)" || { echo "[golden] FATAL: /var/lib/docker is NOT a separate mount — the mp0 split did not take"; exit 1; }'
# Same guard for the SSD user-data volume (mp1): /mnt/sys_drive must be its own mount, not the rootfs
# device — otherwise the controller's system_data_path lands on the OS drive and it warns (the whole
# point of this volume is to clear that warning).
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . && echo " /mnt/sys_drive is a separate mount: $(findmnt -no SOURCE,FSTYPE /mnt/sys_drive)" || { echo "[golden] FATAL: /mnt/sys_drive is NOT a separate mount — the mp1 split did not take"; exit 1; }'
echo "[golden] baking the in-guest controller image $CONTROLLER_IMAGE (no registry cred at deploy) …"
# docker login is used ONCE here on the trusted build host, then logged out before archiving so
@@ -235,17 +246,24 @@ pct exec "$VMID" -- bash -c '
echo "[golden] stop + archive …"
pct stop "$VMID"
# --mode stop with mp0 backup=1 → the Docker-data volume (with baked images) is INCLUDED. The log
# below MUST show "including mount point mp0" — if it shows "excluding … (disabled)" the backup flag
# was lost and the archive carries no images (storage-split B3 trap).
# --mode stop with mp0 + mp1 backup=1 → BOTH the Docker-data volume (baked images) and the
# /mnt/sys_drive user-data volume are INCLUDED. The log below MUST show "including mount point mp0"
# AND "including mount point mp1" — if either shows "excluding … (disabled)" the backup flag was lost
# and the archive carries no images / no user-data volume (storage-split B3 trap).
vzdump "$VMID" --storage "$ARCHIVE_STORAGE" --mode stop --compress zstd 2>&1 | tee /tmp/golden-vzdump.log | grep -iE "including mount point|excluding|archive file size|Finished Backup" || true
if grep -q "excluding volume mount point mp0" /tmp/golden-vzdump.log; then
echo "[golden] FATAL: mp0 (/var/lib/docker) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no images. Aborting."
exit 1
fi
if grep -q "excluding volume mount point mp1" /tmp/golden-vzdump.log; then
echo "[golden] FATAL: mp1 (/mnt/sys_drive) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no user-data volume. Aborting."
exit 1
fi
grep -q "including mount point mp0" /tmp/golden-vzdump.log \
|| echo "[golden] WARN: could not confirm mp0 inclusion in the vzdump log — verify manually before using this archive."
grep -q "including mount point mp1" /tmp/golden-vzdump.log \
|| echo "[golden] WARN: could not confirm mp1 inclusion in the vzdump log — verify manually before using this archive."
VOLID=$(pvesm list "$ARCHIVE_STORAGE" --content backup 2>/dev/null | awk -v v="$VMID" '$1 ~ ("vzdump-lxc-" v "-") {print $1}' | sort | tail -1)
echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G, both in the archive)"
echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G + user-data ${GOLDEN_SYSDATA_GB}G, all in the archive)"
echo "[golden] (the build guest $VMID is stopped; destroy it with: pct destroy $VMID --purge)"
+32
View File
@@ -43,6 +43,11 @@ const bringUpKind = "bring_up"
// DefaultDataVolMount is the mpN slot the golden bakes the Docker-data volume (/var/lib/docker) at.
const DefaultDataVolMount = "mp0"
// DefaultSysDataMount is the mpN slot the golden bakes the SSD user-data volume (/mnt/sys_drive) at.
// This is the controller's system_data_path; provision grows it (SysDataGrowGB) like the Docker-data
// volume. mp1 is the natural next bring-up slot (mp8/mp9 are added by the provision back-half).
const DefaultSysDataMount = "mp1"
// configLockMaxAttempts bounds the F4 config-lock retry. configLockBackoff is a package var so
// tests can shrink it (the production value gives PVE time to release its async config lock).
const configLockMaxAttempts = 5
@@ -81,6 +86,13 @@ type BringUpSpec struct {
DataVolGrowGB int
// DataVolMount is the mpN slot of the golden's Docker-data volume to grow; "" → DefaultDataVolMount ("mp0").
DataVolMount string
// SysDataGrowGB grows the golden-carried SSD user-data volume (SysDataMount, default mp1, mounted at
// /mnt/sys_drive = the controller's system_data_path) to the per-customer target. Same online,
// grow-only mechanism as DataVolGrowGB. 0 = skip (keep the golden's small size — the volume is still
// a separate mount, so the controller's "not a separate drive" warning clears regardless of grow).
SysDataGrowGB int
// SysDataMount is the mpN slot of the golden's user-data volume to grow; "" → DefaultSysDataMount ("mp1").
SysDataMount string
Mounts []GuestMount // additive mpN mounts (slice 7 may pass empty/test)
KeepMAC bool // DR knob: keep the archived MAC (true) unless a source may be live
BootTimeout time.Duration // 0 → DefaultBootTimeout; bounds the link-up liveness wait
@@ -249,6 +261,26 @@ func (e *Engine) runBringUp(ctx context.Context, spec BringUpSpec, res *BringUpR
}
}
// 4c. Grow the golden-carried SSD user-data volume (mp1, /mnt/sys_drive = the controller's
// system_data_path) to the per-customer target. Same shape as the Docker-data grow: grow-only,
// online, its OWN call. The volume came in with the restore (separate mount, backup=1), so we
// grow it rather than attach a fresh one.
if spec.SysDataGrowGB > 0 {
mount := spec.SysDataMount
if mount == "" {
mount = DefaultSysDataMount
}
supid, err := e.api.ResizeLXC(ctx, spec.VMID, mount, fmt.Sprintf("+%dG", spec.SysDataGrowGB))
if err != nil {
res.Err = fmt.Errorf("reconcile: bring-up sys-data resize (%s): %w", mount, err)
return
}
if _, err := e.waitTask(ctx, supid, proxmox.WaitOptions{}); err != nil {
res.Err = fmt.Errorf("reconcile: bring-up sys-data resize task (%s): %w", mount, err)
return
}
}
// Capture the post-reset MAC for the result (fresh for provision; archived for DR keep).
if cfg2, err := e.api.GuestConfig(ctx, spec.VMID); err == nil {
res.AssignedMAC = net0MAC(cfg2)
+58
View File
@@ -126,6 +126,64 @@ func TestRunBringUp_StorageSplit_DataVolGrow(t *testing.T) {
}
}
// The golden-carried SSD user-data volume (/mnt/sys_drive) is grown via a SEPARATE resize on its
// mpN slot (mp1), independent of the rootfs and Docker-data grows. With SysDataGrowGB=0 NO mp1
// resize is issued (the volume stays at the golden size, still a separate mount).
func TestRunBringUp_StorageSplit_SysDataGrow(t *testing.T) {
const vmid = 8051
api := &fakeAPI{cfg: map[int]proxmox.GuestConfig{vmid: scratchCfg()}}
e, _, q := newEngine(t, api, EmptyProvider{})
defer q.Close()
res := e.RunBringUp(context.Background(), BringUpSpec{
Mode: ModeProvision, Archive: "local:backup/golden.tar.zst", VMID: vmid,
RestoreStorage: "local-lvm", Hostname: "felhom-prov-8051",
DataVolGrowGB: 240, SysDataGrowGB: 42, // grows mp0 AND mp1 (DefaultSysDataMount)
})
if res.Err != nil || !res.Pass {
t.Fatalf("provision must pass, got %+v", res)
}
// TWO resizes here: Docker-data mp0 +240G and the user-data volume mp1 +42G (no rootfs grow).
if len(api.resizes) != 2 {
t.Fatalf("expected data-volume + sys-data resizes, got %+v", api.resizes)
}
var sawData, sawSys bool
for _, r := range api.resizes {
if r.disk == "mp0" && r.size == "+240G" {
sawData = true
}
if r.disk == "mp1" && r.size == "+42G" {
sawSys = true
}
}
if !sawData || !sawSys {
t.Errorf("want mp0 +240G AND mp1 +42G, got %+v", api.resizes)
}
}
// SysDataGrowGB=0 must issue NO mp1 resize (grow is an orthogonal knob; separateness comes from the
// golden, not the grow).
func TestRunBringUp_StorageSplit_SysDataGrowZeroNoResize(t *testing.T) {
const vmid = 8052
api := &fakeAPI{cfg: map[int]proxmox.GuestConfig{vmid: scratchCfg()}}
e, _, q := newEngine(t, api, EmptyProvider{})
defer q.Close()
res := e.RunBringUp(context.Background(), BringUpSpec{
Mode: ModeProvision, Archive: "local:backup/golden.tar.zst", VMID: vmid,
RestoreStorage: "local-lvm", Hostname: "felhom-prov-8052",
SysDataGrowGB: 0, // no sys-data grow
})
if res.Err != nil || !res.Pass {
t.Fatalf("provision must pass, got %+v", res)
}
for _, r := range api.resizes {
if r.disk == "mp1" {
t.Errorf("SysDataGrowGB=0 must NOT resize mp1, got %+v", api.resizes)
}
}
}
func TestRunBringUp_CompensatingRollback(t *testing.T) {
const vmid = 8000
lockBackoffFast(t)