v0.40.0: third CT volume — SSD user-data (/mnt/sys_drive, mp1) baked + -sysdata-grow
Extends the OS/Docker-data split to a three-volume layout: rootfs + Docker-data
(mp0) + SSD user-data (mp1 @ /mnt/sys_drive, backup=1) = the controller's
system_data_path. Clears the controller's "not a separate drive" warning with
zero controller change (it already auto-discovers <sys_drive>/felhom-data and
warns via system.IsMountPoint; the mp reaches the container via the existing
-v /mnt:/mnt:rslave bind).
- build-golden.sh: --mp1 ...,mp=/mnt/sys_drive,backup=1 (env GOLDEN_SYSDATA_GB=8);
findmnt /mnt/sys_drive separate-mount guard + vzdump aborts if mp0 OR mp1 excluded.
- bringup.go: DefaultSysDataMount=mp1; BringUpSpec.{SysDataGrowGB,SysDataMount};
new "4c" online grow-only block mirroring the "4b" Docker-data grow.
- main.go: -sysdata-grow / -sysdata-mount flags wired into all three call sites.
- Tests: SysDataGrow (asserts ResizeLXC mp1 +42G) + SysDataGrowZeroNoResize.
- RUNBOOK extended to the three-volume layout (32 rootfs + 200 docker + 50 user-data).
Static CT volume, NOT an enrolled drive — never enrolls/ejects/decommissions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
This commit is contained in:
@@ -3,6 +3,37 @@
|
||||
All notable changes to **felhom-agent** are recorded here. Update on every code
|
||||
change that gets pushed.
|
||||
|
||||
## v0.40.0 — third CT volume: SSD user-data (`/mnt/sys_drive`, mp1) baked + `-sysdata-grow` (2026-06-23)
|
||||
|
||||
**The third golden volume.** Extends the OS/Docker-data split (v0.29.x) to a **three-volume layout**:
|
||||
rootfs + Docker-data (`mp0`) + **SSD user-data (`mp1` @ `/mnt/sys_drive`, `backup=1`)** — the
|
||||
controller's `system_data_path`. Until now `/mnt/sys_drive` was a plain directory on the 32 GB OS
|
||||
rootfs, so the controller correctly warned that SSD app data (`<sys_drive>/felhom-data`) lands on the
|
||||
OS drive. Baking it as its own thin volume clears that warning with **zero controller change** (the
|
||||
controller already auto-discovers `<sys_drive>/felhom-data` and warns via `system.IsMountPoint`); the
|
||||
`mp` under the guest's `/mnt` reaches the controller container through the existing
|
||||
`-v /mnt:/mnt:rslave` bind.
|
||||
|
||||
- **`configs/build-golden.sh`** — `pct create` gains
|
||||
`--mp1 ${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1` (new env
|
||||
`GOLDEN_SYSDATA_GB=8`, near-empty; provision grows it). The resilience guards are mirrored for `mp1`:
|
||||
a `findmnt /mnt/sys_drive` separate-mount assertion, and the vzdump-inclusion guard now aborts if
|
||||
**either** `mp0` **or** `mp1` is EXCLUDED (the B3 trap — extra mountpoints default `backup=0`). The
|
||||
golden does NOT pre-create `felhom-data`; the controller does once it's a real mountpoint.
|
||||
- **`internal/reconcile/bringup.go`** — `const DefaultSysDataMount = "mp1"`; `BringUpSpec` gains
|
||||
`SysDataGrowGB int` + `SysDataMount string`; a new **"4c"** grow block (online, grow-only `ResizeLXC`,
|
||||
its own task) mirrors the "4b" Docker-data grow. `0 = skip` (separateness comes from the golden, not
|
||||
the grow — the warning clears regardless of size).
|
||||
- **`cmd/felhom-agent/main.go`** — `-sysdata-grow` / `-sysdata-mount` flags (mirror
|
||||
`-datavol-grow`/`-datavol-mount`); `bringUpSizing` carries them into all three bring-up/provision call
|
||||
sites; `--selftest=provision` help text updated.
|
||||
- **Static volume, NOT an enrolled drive.** `/mnt/sys_drive` is part of the baked golden layout; it
|
||||
never enrolls/ejects/decommissions and is deliberately kept off the drive-intent machinery.
|
||||
`freeMountSlot` auto-skips the baked `mp0`/`mp1` so enrolled drives never collide.
|
||||
- Tests: `TestRunBringUp_StorageSplit_SysDataGrow` (asserts `ResizeLXC(vmid,"mp1","+42G")`) +
|
||||
`…_SysDataGrowZeroNoResize` (0 → no mp1 resize). RUNBOOK-provisioning-storage.md extended to the
|
||||
three-volume layout (default ~512 GB SSD: 32 rootfs + 200 docker-data + 50 user-data).
|
||||
|
||||
## v0.39.0 — DR recipe completion: live PBS coord + drop the two unfillable drive fields (2026-06-16)
|
||||
|
||||
**DR-recipe agent-half completion.** A live eyeball of the demo recipe (v0.38.0) found three host-half
|
||||
|
||||
@@ -1,23 +1,38 @@
|
||||
# RUNBOOK — provisioning the OS / Docker-data storage split
|
||||
# RUNBOOK — provisioning the OS / Docker-data / user-data storage split
|
||||
|
||||
How the controller guest's **OS rootfs** and **Docker data** are carved onto separate `local-lvm`
|
||||
volumes, and how to size them per node/customer. Grounded by the storage-split spike
|
||||
(`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 256 GB
|
||||
Docker-data**, rest of the thin pool free for other guests.
|
||||
How the controller guest's **OS rootfs**, **Docker data**, and **SSD user-data** are carved onto three
|
||||
separate `local-lvm` volumes, and how to size them per node/customer. Grounded by the storage-split spike
|
||||
(`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 200 GB Docker-data
|
||||
+ 50 GB user-data**, rest of the thin pool free for other guests.
|
||||
|
||||
## The design (locked)
|
||||
|
||||
Three thin volumes are baked into the golden and grown at provision:
|
||||
|
||||
| mp slot | Mount | Holds | Golden default | Provision grow knob |
|
||||
|---------|-------|-------|----------------|---------------------|
|
||||
| rootfs | `/` | OS only | `OS_SIZE_GB=32` | `-rootfs-grow` |
|
||||
| `mp0` | `/var/lib/docker` | ALL Docker images + overlay + named volumes | `GOLDEN_DOCKER_GB=16` | `-datavol-grow` |
|
||||
| `mp1` | `/mnt/sys_drive` | SSD user-data (the controller's `system_data_path`; `<sys_drive>/felhom-data` = SSD-only app data + DBs) | `GOLDEN_SYSDATA_GB=8` | `-sysdata-grow` |
|
||||
|
||||
- **Split for resilience, not backup size.** An isolated OS rootfs stays bootable + agent-recoverable
|
||||
if the Docker volume fills. Both volumes ride into PBS regardless (so the backup size is unchanged) —
|
||||
the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
|
||||
if the Docker volume fills. All three volumes ride into PBS regardless (so the backup size is
|
||||
unchanged) — the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
|
||||
- **One Docker daemon, one data-root, one volume** at `/var/lib/docker` for ALL images + overlay
|
||||
(controller, traefik, cloudflared, filebrowser, and customer apps). Infra is NOT relocated to the OS
|
||||
partition — infra/user separation under one daemon isn't cleanly possible and isn't needed. Infra is
|
||||
protected by the controller's **prevention layer** (reserved buffer + deploy gate + disk alerts +
|
||||
baked log rotation), not by placement.
|
||||
- **`backup=1` on the Docker-data volume is non-negotiable.** Extra LXC mountpoints default to
|
||||
`backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs fall out of the
|
||||
snapshot AND the volume (with baked images) is excluded from the golden archive.
|
||||
- **`/mnt/sys_drive` is a static CT volume, NOT an enrolled drive.** It never enrolls, ejects, or
|
||||
decommissions — it is part of the golden's baked layout. The controller auto-discovers
|
||||
`<sys_drive>/felhom-data` and, via `system.IsMountPoint`, **warns when `/mnt/sys_drive` is not its own
|
||||
mount** (`selftest.go`). Making `mp1` a real mount clears that warning with **zero controller change**;
|
||||
the controller is untouched by this slice. The `mp` mounted under the guest's `/mnt` propagates into
|
||||
the controller container through the existing `-v /mnt:/mnt:rslave` bind (the same path enrolled drives
|
||||
use).
|
||||
- **`backup=1` on the Docker-data AND user-data volumes is non-negotiable.** Extra LXC mountpoints
|
||||
default to `backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs / SSD app
|
||||
data fall out of the snapshot AND (for mp0) the baked images are excluded from the golden archive.
|
||||
|
||||
## Golden build (per controller-image baseline)
|
||||
|
||||
@@ -25,15 +40,20 @@ Docker-data**, rest of the thin pool free for other guests.
|
||||
|
||||
```
|
||||
# defaults shown; override via env
|
||||
OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 \
|
||||
OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 GOLDEN_SYSDATA_GB=8 \
|
||||
./build-golden.sh <VMID> <TEMPLATE> <ROOTFS_STORAGE> <ARCHIVE_STORAGE> <BRIDGE> <CONTROLLER_IMAGE>
|
||||
```
|
||||
|
||||
- `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}` + `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`.
|
||||
- Keep `GOLDEN_DOCKER_GB` small (just the baked images + headroom) — provision grows it.
|
||||
- `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}`
|
||||
+ `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`
|
||||
+ `--mp1 ${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1`.
|
||||
- Keep `GOLDEN_DOCKER_GB` / `GOLDEN_SYSDATA_GB` small (the golden ships them near-empty) — provision
|
||||
grows both. The controller creates `<sys_drive>/felhom-data` itself once `/mnt/sys_drive` is a real
|
||||
mount; the golden does NOT pre-create it.
|
||||
- The script bakes `/etc/docker/daemon.json` with **`features.containerd-snapshotter: false`** (the
|
||||
classic **overlay2** driver) + log rotation (`max-size 10m`, `max-file 3`), verifies `/var/lib/docker`
|
||||
is a separate mount + the driver is overlay2, and **aborts if vzdump excludes mp0** (the B3 trap).
|
||||
AND `/mnt/sys_drive` are separate mounts + the driver is overlay2, and **aborts if vzdump excludes mp0
|
||||
or mp1** (the B3 trap).
|
||||
- **Why overlay2 (not the Docker-29 default containerd-snapshotter):** containerd-snapshotter keeps the
|
||||
image content store under `/var/lib/containerd`, which is NOT `/var/lib/docker` — so it would stay on
|
||||
the OS rootfs and the split would only move named volumes, leaving images (the bulk) on the rootfs
|
||||
@@ -43,33 +63,39 @@ OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 \
|
||||
|
||||
## Provision (per customer guest)
|
||||
|
||||
The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data volume + baked images),
|
||||
then sizes per customer. The Docker-data volume is **grown** (not re-attached) so the baked images are
|
||||
never shadowed:
|
||||
The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data + mp1 user-data volumes +
|
||||
baked images), then sizes per customer. Each volume is **grown** (not re-attached) so the baked images
|
||||
are never shadowed:
|
||||
|
||||
```
|
||||
felhom-agent --selftest=provision \
|
||||
-archive <golden-volid> -vmid <VMID> \
|
||||
-customer-id <id> -hub-password <retrieval-passphrase> \
|
||||
-rootfs-grow <N> # GiB to add to the OS rootfs beyond the golden's 32 (0 = keep 32)
|
||||
-datavol-grow <N> # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 240 → 256 total)
|
||||
-datavol-grow <N> # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 184 → 200 total)
|
||||
-datavol-mount mp0 # the golden's Docker-data slot (default mp0)
|
||||
-sysdata-grow <N> # GiB to add to the user-data volume beyond the golden's 8 (e.g. 42 → 50 total)
|
||||
-sysdata-mount mp1 # the golden's user-data slot (default mp1)
|
||||
```
|
||||
|
||||
In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount}` — both grows are online
|
||||
(`pct resize`, spike B4), each its own task. `GuestMount.Backup` (→ `,backup=1`) is set on any
|
||||
data-bearing additive mount (`bringup.go` `buildConfigParams`). The bootstrap mount is `mp9`; USB/data
|
||||
drive enroll picks the lowest free slot (`freeMountSlot`), so it never collides with the docker mp0.
|
||||
In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount, SysDataGrowGB, SysDataMount}`
|
||||
— all grows are online (`pct resize`, spike B4), each its own task. **Separateness, not size, clears the
|
||||
controller warning** — `-sysdata-grow 0` still yields a separate, warning-free mount at the golden size;
|
||||
the grow is an orthogonal knob. `GuestMount.Backup` (→ `,backup=1`) is set on any data-bearing additive
|
||||
mount (`bringup.go` `buildBringUpConfig`). The bootstrap mount is `mp9`, the shared parent bind is `mp8`;
|
||||
USB/data drive enroll picks the lowest free slot (`freeMountSlot`), which auto-skips the baked mp0/mp1.
|
||||
|
||||
**Per-customer sizing source:** flags today; the **slice-10 hub storage manifest** is the intended
|
||||
home (marked at `bringup.go` `GuestMount` / the `slice 10 wires the hub storage manifest` comment) —
|
||||
the controller already pulls customer-scoped config from the hub, so OS/data sizes belong there.
|
||||
|
||||
### Default sizing guidance
|
||||
- ~512 GB SSD: 32 GB rootfs + 256 GB Docker-data (`-rootfs-grow 0 -datavol-grow 240`), leaving the rest
|
||||
of the thin pool free for other guests.
|
||||
- Other SSDs: size the data volume to "most of the free thin pool minus a reserve for other guests."
|
||||
`local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's written.
|
||||
- ~512 GB SSD: 32 GB rootfs + 200 GB Docker-data + 50 GB user-data
|
||||
(`-rootfs-grow 0 -datavol-grow 184 -sysdata-grow 42`), leaving the rest of the thin pool free for
|
||||
other guests.
|
||||
- Other SSDs: size the Docker-data + user-data volumes to "most of the free thin pool minus a reserve
|
||||
for other guests." `local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's
|
||||
written.
|
||||
|
||||
## Fresh PVE host install — thin-pool carving (host-level, do this at install)
|
||||
|
||||
|
||||
@@ -44,7 +44,7 @@ import (
|
||||
|
||||
// version is the agent version. Overridable at build time with
|
||||
// -ldflags "-X main.version=<v>"; defaults to the in-repo CHANGELOG version.
|
||||
var version = "0.39.0"
|
||||
var version = "0.40.0"
|
||||
|
||||
// runGuestHook is the PVE pre-start hook body (`felhom-agent guest-hook <vmid> <phase>`). On the
|
||||
// pre-start phase it creates placeholder dirs for any absent bind-mount source so the guest always boots
|
||||
@@ -88,6 +88,8 @@ func main() {
|
||||
rootfsGrow int
|
||||
dataVolGrow int
|
||||
dataVolMount string
|
||||
sysDataGrow int
|
||||
sysDataMount string
|
||||
pbsStorage string
|
||||
paperkey bool
|
||||
offline bool
|
||||
@@ -105,7 +107,7 @@ func main() {
|
||||
showVersion bool
|
||||
)
|
||||
flag.StringVar(&cfgPath, "config", envOr("FELHOM_AGENT_CONFIG", "/etc/felhom-agent/agent.json"), "path to the agent config file (JSON)")
|
||||
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; keeps the guest)")
|
||||
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; optional -rootfs-grow/-datavol-grow/-sysdata-grow; keeps the guest)")
|
||||
flag.IntVar(&vmid, "vmid", 0, "guest VMID for --selftest=task|backup|bring-up")
|
||||
flag.DurationVar(&watch, "watch", 0, "for --selftest=storage: run the watchdog verbose for this duration (e.g. 3m) with the re-mount response live; 0 = observe pass only")
|
||||
flag.StringVar(&archive, "archive", "", "for --selftest=restore-test|bring-up: the backup volid to restore (restore-test: default newest on the local target)")
|
||||
@@ -115,6 +117,8 @@ func main() {
|
||||
flag.IntVar(&rootfsGrow, "rootfs-grow", 0, "for --selftest=bring-up|provision: grow the OS rootfs by this many GiB after restore (0 = keep golden size)")
|
||||
flag.IntVar(&dataVolGrow, "datavol-grow", 0, "for --selftest=bring-up|provision: grow the golden's Docker-data volume (mp0) by this many GiB (0 = keep golden size)")
|
||||
flag.StringVar(&dataVolMount, "datavol-mount", "", "for --selftest=bring-up|provision: the mpN slot of the Docker-data volume to grow (default mp0)")
|
||||
flag.IntVar(&sysDataGrow, "sysdata-grow", 0, "for --selftest=bring-up|provision: grow the golden's SSD user-data volume (mp1, /mnt/sys_drive) by this many GiB (0 = keep golden size)")
|
||||
flag.StringVar(&sysDataMount, "sysdata-mount", "", "for --selftest=bring-up|provision: the mpN slot of the user-data volume to grow (default mp1)")
|
||||
flag.StringVar(&pbsStorage, "storage", "", "for --selftest=escrow-create: the pbs storage whose key to escrow (default: escrow.pbs_storage_id)")
|
||||
flag.BoolVar(&paperkey, "paperkey", false, "for --selftest=escrow-create: ALSO emit the raw-key paperkey (opt-in (a); single-factor, unrevocable)")
|
||||
flag.BoolVar(&offline, "offline", false, "for --selftest=escrow-create: ALSO emit the R-wrapped offline copy to print (opt-in (b))")
|
||||
@@ -170,12 +174,14 @@ func main() {
|
||||
os.Exit(runSelftestLANResolver(context.Background(), cfg, logger, vmid))
|
||||
case "bring-up":
|
||||
os.Exit(runSelftestBringUp(context.Background(), cfg, logger, mode, archive, vmid, hostname, keep,
|
||||
bringUpSizing{RootfsGrowGB: rootfsGrow, DataVolGrowGB: dataVolGrow, DataVolMount: dataVolMount}))
|
||||
bringUpSizing{RootfsGrowGB: rootfsGrow, DataVolGrowGB: dataVolGrow, DataVolMount: dataVolMount,
|
||||
SysDataGrowGB: sysDataGrow, SysDataMount: sysDataMount}))
|
||||
case "provision":
|
||||
os.Exit(runSelftestProvision(context.Background(), cfg, logger, provisionArgs{
|
||||
archive: archive, vmid: vmid, hostname: hostname,
|
||||
customerID: custID, hubPassword: hubPassword,
|
||||
sizing: bringUpSizing{RootfsGrowGB: rootfsGrow, DataVolGrowGB: dataVolGrow, DataVolMount: dataVolMount},
|
||||
sizing: bringUpSizing{RootfsGrowGB: rootfsGrow, DataVolGrowGB: dataVolGrow, DataVolMount: dataVolMount,
|
||||
SysDataGrowGB: sysDataGrow, SysDataMount: sysDataMount},
|
||||
}))
|
||||
case "escrow-create":
|
||||
os.Exit(runSelftestEscrowCreate(context.Background(), cfg, logger, pbsStorage, paperkey, offline, upload, idBundlePath, directivePath))
|
||||
@@ -1060,6 +1066,8 @@ type bringUpSizing struct {
|
||||
RootfsGrowGB int
|
||||
DataVolGrowGB int
|
||||
DataVolMount string
|
||||
SysDataGrowGB int
|
||||
SysDataMount string
|
||||
}
|
||||
|
||||
func runSelftestBringUp(ctx context.Context, cfg config.Config, logger *slog.Logger, mode, archive string, vmid int, hostname string, keep bool, sizing bringUpSizing) int {
|
||||
@@ -1119,6 +1127,7 @@ func runSelftestBringUp(ctx context.Context, cfg config.Config, logger *slog.Log
|
||||
Mode: bmode, Archive: archive, VMID: vmid, RestoreStorage: cfg.Backup.RestoreStorage,
|
||||
Hostname: hostname, KeepMAC: bmode == reconcile.ModeDRGuestLoss,
|
||||
RootfsGrowGB: sizing.RootfsGrowGB, DataVolGrowGB: sizing.DataVolGrowGB, DataVolMount: sizing.DataVolMount,
|
||||
SysDataGrowGB: sizing.SysDataGrowGB, SysDataMount: sizing.SysDataMount,
|
||||
}
|
||||
fmt.Printf(" bringing up %s → vmid %d on %s …\n", archive, vmid, cfg.Backup.RestoreStorage)
|
||||
res := engine.RunBringUp(ctx, spec)
|
||||
@@ -1271,6 +1280,7 @@ func runSelftestProvision(ctx context.Context, cfg config.Config, logger *slog.L
|
||||
Mode: reconcile.ModeProvision, Archive: a.archive, VMID: a.vmid,
|
||||
RestoreStorage: cfg.Backup.RestoreStorage, Hostname: hostname,
|
||||
RootfsGrowGB: a.sizing.RootfsGrowGB, DataVolGrowGB: a.sizing.DataVolGrowGB, DataVolMount: a.sizing.DataVolMount,
|
||||
SysDataGrowGB: a.sizing.SysDataGrowGB, SysDataMount: a.sizing.SysDataMount,
|
||||
})
|
||||
if res.Err != nil || !res.Pass {
|
||||
fmt.Fprintf(os.Stderr, " [FAIL] front-half bring-up (vmid %d): %v\n", a.vmid, res.Err)
|
||||
|
||||
+23
-5
@@ -46,13 +46,20 @@ REGISTRY_HOST="${CONTROLLER_IMAGE%%/*}"
|
||||
# large enough for the baked images + headroom; provision grows it to the per-customer target.
|
||||
OS_SIZE_GB="${OS_SIZE_GB:-32}"
|
||||
GOLDEN_DOCKER_GB="${GOLDEN_DOCKER_GB:-16}"
|
||||
# The golden's SSD user-data volume (GiB) mounted at /mnt/sys_drive (mp1, backup=1) — the controller's
|
||||
# system_data_path. Ships small + near-empty (the controller creates <sys_drive>/felhom-data itself once
|
||||
# it's a real mountpoint); provision GROWS it to the per-customer target (bringup.go SysDataGrowGB). Like
|
||||
# mp0, backup=1 is MANDATORY: without it vzdump EXCLUDES the volume (extra mountpoints default backup=0 —
|
||||
# storage-split B3) and the user-data area would silently fall out of PBS coverage.
|
||||
GOLDEN_SYSDATA_GB="${GOLDEN_SYSDATA_GB:-8}"
|
||||
|
||||
echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G @ /var/lib/docker, backup=1) …"
|
||||
echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G @ /var/lib/docker + user-data ${GOLDEN_SYSDATA_GB}G @ /mnt/sys_drive, both backup=1) …"
|
||||
pct create "$VMID" "$TEMPLATE" \
|
||||
--hostname felhom-golden --unprivileged 1 \
|
||||
--features nesting=1,keyctl=1 \
|
||||
--rootfs "${ROOTFS_STORAGE}:${OS_SIZE_GB}" --cores 2 --memory 2048 \
|
||||
--mp0 "${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1" \
|
||||
--mp1 "${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1" \
|
||||
--net0 "name=eth0,bridge=${BRIDGE},ip=dhcp" --onboot 0
|
||||
|
||||
echo "[golden] starting + installing Docker (official repo, trixie channel) …"
|
||||
@@ -99,6 +106,10 @@ pct exec "$VMID" -- bash -c 'systemctl restart docker; sleep 3; docker run --rm
|
||||
pct exec "$VMID" -- bash -c 'drv=$(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); [ "$drv" = "overlay2" ] || { echo "[golden] FATAL: storage driver is $drv, expected overlay2 — images would not land on the data volume"; exit 1; }'
|
||||
# Confirm /var/lib/docker is genuinely the dedicated volume, not the rootfs (catch a silent mp miss).
|
||||
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo " /var/lib/docker is a separate mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker)" || { echo "[golden] FATAL: /var/lib/docker is NOT a separate mount — the mp0 split did not take"; exit 1; }'
|
||||
# Same guard for the SSD user-data volume (mp1): /mnt/sys_drive must be its own mount, not the rootfs
|
||||
# device — otherwise the controller's system_data_path lands on the OS drive and it warns (the whole
|
||||
# point of this volume is to clear that warning).
|
||||
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . && echo " /mnt/sys_drive is a separate mount: $(findmnt -no SOURCE,FSTYPE /mnt/sys_drive)" || { echo "[golden] FATAL: /mnt/sys_drive is NOT a separate mount — the mp1 split did not take"; exit 1; }'
|
||||
|
||||
echo "[golden] baking the in-guest controller image $CONTROLLER_IMAGE (no registry cred at deploy) …"
|
||||
# docker login is used ONCE here on the trusted build host, then logged out before archiving so
|
||||
@@ -235,17 +246,24 @@ pct exec "$VMID" -- bash -c '
|
||||
|
||||
echo "[golden] stop + archive …"
|
||||
pct stop "$VMID"
|
||||
# --mode stop with mp0 backup=1 → the Docker-data volume (with baked images) is INCLUDED. The log
|
||||
# below MUST show "including mount point mp0" — if it shows "excluding … (disabled)" the backup flag
|
||||
# was lost and the archive carries no images (storage-split B3 trap).
|
||||
# --mode stop with mp0 + mp1 backup=1 → BOTH the Docker-data volume (baked images) and the
|
||||
# /mnt/sys_drive user-data volume are INCLUDED. The log below MUST show "including mount point mp0"
|
||||
# AND "including mount point mp1" — if either shows "excluding … (disabled)" the backup flag was lost
|
||||
# and the archive carries no images / no user-data volume (storage-split B3 trap).
|
||||
vzdump "$VMID" --storage "$ARCHIVE_STORAGE" --mode stop --compress zstd 2>&1 | tee /tmp/golden-vzdump.log | grep -iE "including mount point|excluding|archive file size|Finished Backup" || true
|
||||
if grep -q "excluding volume mount point mp0" /tmp/golden-vzdump.log; then
|
||||
echo "[golden] FATAL: mp0 (/var/lib/docker) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no images. Aborting."
|
||||
exit 1
|
||||
fi
|
||||
if grep -q "excluding volume mount point mp1" /tmp/golden-vzdump.log; then
|
||||
echo "[golden] FATAL: mp1 (/mnt/sys_drive) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no user-data volume. Aborting."
|
||||
exit 1
|
||||
fi
|
||||
grep -q "including mount point mp0" /tmp/golden-vzdump.log \
|
||||
|| echo "[golden] WARN: could not confirm mp0 inclusion in the vzdump log — verify manually before using this archive."
|
||||
grep -q "including mount point mp1" /tmp/golden-vzdump.log \
|
||||
|| echo "[golden] WARN: could not confirm mp1 inclusion in the vzdump log — verify manually before using this archive."
|
||||
|
||||
VOLID=$(pvesm list "$ARCHIVE_STORAGE" --content backup 2>/dev/null | awk -v v="$VMID" '$1 ~ ("vzdump-lxc-" v "-") {print $1}' | sort | tail -1)
|
||||
echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G, both in the archive)"
|
||||
echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G + user-data ${GOLDEN_SYSDATA_GB}G, all in the archive)"
|
||||
echo "[golden] (the build guest $VMID is stopped; destroy it with: pct destroy $VMID --purge)"
|
||||
|
||||
@@ -43,6 +43,11 @@ const bringUpKind = "bring_up"
|
||||
// DefaultDataVolMount is the mpN slot the golden bakes the Docker-data volume (/var/lib/docker) at.
|
||||
const DefaultDataVolMount = "mp0"
|
||||
|
||||
// DefaultSysDataMount is the mpN slot the golden bakes the SSD user-data volume (/mnt/sys_drive) at.
|
||||
// This is the controller's system_data_path; provision grows it (SysDataGrowGB) like the Docker-data
|
||||
// volume. mp1 is the natural next bring-up slot (mp8/mp9 are added by the provision back-half).
|
||||
const DefaultSysDataMount = "mp1"
|
||||
|
||||
// configLockMaxAttempts bounds the F4 config-lock retry. configLockBackoff is a package var so
|
||||
// tests can shrink it (the production value gives PVE time to release its async config lock).
|
||||
const configLockMaxAttempts = 5
|
||||
@@ -81,6 +86,13 @@ type BringUpSpec struct {
|
||||
DataVolGrowGB int
|
||||
// DataVolMount is the mpN slot of the golden's Docker-data volume to grow; "" → DefaultDataVolMount ("mp0").
|
||||
DataVolMount string
|
||||
// SysDataGrowGB grows the golden-carried SSD user-data volume (SysDataMount, default mp1, mounted at
|
||||
// /mnt/sys_drive = the controller's system_data_path) to the per-customer target. Same online,
|
||||
// grow-only mechanism as DataVolGrowGB. 0 = skip (keep the golden's small size — the volume is still
|
||||
// a separate mount, so the controller's "not a separate drive" warning clears regardless of grow).
|
||||
SysDataGrowGB int
|
||||
// SysDataMount is the mpN slot of the golden's user-data volume to grow; "" → DefaultSysDataMount ("mp1").
|
||||
SysDataMount string
|
||||
Mounts []GuestMount // additive mpN mounts (slice 7 may pass empty/test)
|
||||
KeepMAC bool // DR knob: keep the archived MAC (true) unless a source may be live
|
||||
BootTimeout time.Duration // 0 → DefaultBootTimeout; bounds the link-up liveness wait
|
||||
@@ -249,6 +261,26 @@ func (e *Engine) runBringUp(ctx context.Context, spec BringUpSpec, res *BringUpR
|
||||
}
|
||||
}
|
||||
|
||||
// 4c. Grow the golden-carried SSD user-data volume (mp1, /mnt/sys_drive = the controller's
|
||||
// system_data_path) to the per-customer target. Same shape as the Docker-data grow: grow-only,
|
||||
// online, its OWN call. The volume came in with the restore (separate mount, backup=1), so we
|
||||
// grow it rather than attach a fresh one.
|
||||
if spec.SysDataGrowGB > 0 {
|
||||
mount := spec.SysDataMount
|
||||
if mount == "" {
|
||||
mount = DefaultSysDataMount
|
||||
}
|
||||
supid, err := e.api.ResizeLXC(ctx, spec.VMID, mount, fmt.Sprintf("+%dG", spec.SysDataGrowGB))
|
||||
if err != nil {
|
||||
res.Err = fmt.Errorf("reconcile: bring-up sys-data resize (%s): %w", mount, err)
|
||||
return
|
||||
}
|
||||
if _, err := e.waitTask(ctx, supid, proxmox.WaitOptions{}); err != nil {
|
||||
res.Err = fmt.Errorf("reconcile: bring-up sys-data resize task (%s): %w", mount, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// Capture the post-reset MAC for the result (fresh for provision; archived for DR keep).
|
||||
if cfg2, err := e.api.GuestConfig(ctx, spec.VMID); err == nil {
|
||||
res.AssignedMAC = net0MAC(cfg2)
|
||||
|
||||
@@ -126,6 +126,64 @@ func TestRunBringUp_StorageSplit_DataVolGrow(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// The golden-carried SSD user-data volume (/mnt/sys_drive) is grown via a SEPARATE resize on its
|
||||
// mpN slot (mp1), independent of the rootfs and Docker-data grows. With SysDataGrowGB=0 NO mp1
|
||||
// resize is issued (the volume stays at the golden size, still a separate mount).
|
||||
func TestRunBringUp_StorageSplit_SysDataGrow(t *testing.T) {
|
||||
const vmid = 8051
|
||||
api := &fakeAPI{cfg: map[int]proxmox.GuestConfig{vmid: scratchCfg()}}
|
||||
e, _, q := newEngine(t, api, EmptyProvider{})
|
||||
defer q.Close()
|
||||
|
||||
res := e.RunBringUp(context.Background(), BringUpSpec{
|
||||
Mode: ModeProvision, Archive: "local:backup/golden.tar.zst", VMID: vmid,
|
||||
RestoreStorage: "local-lvm", Hostname: "felhom-prov-8051",
|
||||
DataVolGrowGB: 240, SysDataGrowGB: 42, // grows mp0 AND mp1 (DefaultSysDataMount)
|
||||
})
|
||||
if res.Err != nil || !res.Pass {
|
||||
t.Fatalf("provision must pass, got %+v", res)
|
||||
}
|
||||
// TWO resizes here: Docker-data mp0 +240G and the user-data volume mp1 +42G (no rootfs grow).
|
||||
if len(api.resizes) != 2 {
|
||||
t.Fatalf("expected data-volume + sys-data resizes, got %+v", api.resizes)
|
||||
}
|
||||
var sawData, sawSys bool
|
||||
for _, r := range api.resizes {
|
||||
if r.disk == "mp0" && r.size == "+240G" {
|
||||
sawData = true
|
||||
}
|
||||
if r.disk == "mp1" && r.size == "+42G" {
|
||||
sawSys = true
|
||||
}
|
||||
}
|
||||
if !sawData || !sawSys {
|
||||
t.Errorf("want mp0 +240G AND mp1 +42G, got %+v", api.resizes)
|
||||
}
|
||||
}
|
||||
|
||||
// SysDataGrowGB=0 must issue NO mp1 resize (grow is an orthogonal knob; separateness comes from the
|
||||
// golden, not the grow).
|
||||
func TestRunBringUp_StorageSplit_SysDataGrowZeroNoResize(t *testing.T) {
|
||||
const vmid = 8052
|
||||
api := &fakeAPI{cfg: map[int]proxmox.GuestConfig{vmid: scratchCfg()}}
|
||||
e, _, q := newEngine(t, api, EmptyProvider{})
|
||||
defer q.Close()
|
||||
|
||||
res := e.RunBringUp(context.Background(), BringUpSpec{
|
||||
Mode: ModeProvision, Archive: "local:backup/golden.tar.zst", VMID: vmid,
|
||||
RestoreStorage: "local-lvm", Hostname: "felhom-prov-8052",
|
||||
SysDataGrowGB: 0, // no sys-data grow
|
||||
})
|
||||
if res.Err != nil || !res.Pass {
|
||||
t.Fatalf("provision must pass, got %+v", res)
|
||||
}
|
||||
for _, r := range api.resizes {
|
||||
if r.disk == "mp1" {
|
||||
t.Errorf("SysDataGrowGB=0 must NOT resize mp1, got %+v", api.resizes)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunBringUp_CompensatingRollback(t *testing.T) {
|
||||
const vmid = 8000
|
||||
lockBackoffFast(t)
|
||||
|
||||
Reference in New Issue
Block a user