v0.29.0: OS/Docker-data storage split — golden + provision (Phase 1)
build-golden.sh bakes a small OS rootfs + a dedicated /var/lib/docker volume (mp0, backup=1) carrying the baked images, plus Docker log rotation. bringup.go grows the golden's data volume to the per-customer target (DataVolGrowGB) and emits backup=1 on data mounts (GuestMount.Backup) — closing the spike-B3 silent DB-loss trap. CLI gains -rootfs-grow/-datavol-grow/-datavol-mount. New RUNBOOK-provisioning-storage.md. Phase 2 = felhom-controller v0.58.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -3,6 +3,31 @@
|
||||
All notable changes to **felhom-agent** are recorded here. Update on every code
|
||||
change that gets pushed.
|
||||
|
||||
## v0.29.0 — OS / Docker-data storage split: golden + provision (2026-06-13)
|
||||
|
||||
Phase 1 of the storage-split slice (Phase 2 = felhom-controller v0.58.0 prevention layer). The
|
||||
controller guest's OS rootfs and Docker data are carved onto separate `local-lvm` volumes for
|
||||
RESILIENCE — an isolated OS rootfs stays bootable + agent-recoverable if the Docker volume fills.
|
||||
|
||||
- **`configs/build-golden.sh` — split baked in:** `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}` (default
|
||||
**32**, was hardcoded 8) **plus** `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`
|
||||
(default 16). The baked controller + infra images land on the data volume and travel inside the
|
||||
golden archive (no empty-volume shadowing, no deploy-time pull). `backup=1` is MANDATORY — extra LXC
|
||||
mountpoints default to `backup=0` = EXCLUDED from vzdump (spike B3), which would drop the images from
|
||||
the archive entirely. The script now also bakes Docker **log rotation** into `daemon.json`
|
||||
(`max-size 10m`, `max-file 3` — prevention layer 2D), asserts `/var/lib/docker` is a separate mount,
|
||||
and **aborts if vzdump excludes mp0**.
|
||||
- **`internal/reconcile/bringup.go` — sized provision:** `GuestMount` gains `Backup` (emits `,backup=1`
|
||||
— closes the spike-B3/B5 silent-DB-loss trap at the mount builder). `BringUpSpec` gains
|
||||
`DataVolGrowGB` + `DataVolMount` (default `mp0`): provision GROWS the golden-carried Docker-data
|
||||
volume online to the per-customer target (grow-only, spike B4) rather than attaching a fresh empty
|
||||
volume that would shadow the baked images. Plus `RootfsGrowGB` for the OS rootfs.
|
||||
- **CLI seam:** `--selftest=bring-up|provision` gain `-rootfs-grow` / `-datavol-grow` / `-datavol-mount`
|
||||
flags. Per-customer sizing source = flags now, the slice-10 hub storage manifest later.
|
||||
- **`RUNBOOK-provisioning-storage.md`** (new): the split provisioning procedure + fresh-PVE-install
|
||||
thin-pool carving knobs (`hdsize`/`maxroot`/`maxvz`, spike B4) + the per-customer sizing seam.
|
||||
- Tests: `buildBringUpConfig` backup=1 emission; bring-up issues rootfs + data-volume resizes.
|
||||
|
||||
## (no version) — storage OS/data-split spike findings (2026-06-13)
|
||||
|
||||
Investigation only — **no code changed**. Findings report: `REPORT-storage-split-spike.md` (gates the
|
||||
|
||||
@@ -0,0 +1,89 @@
|
||||
# RUNBOOK — provisioning the OS / Docker-data storage split
|
||||
|
||||
How the controller guest's **OS rootfs** and **Docker data** are carved onto separate `local-lvm`
|
||||
volumes, and how to size them per node/customer. Grounded by the storage-split spike
|
||||
(`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 256 GB
|
||||
Docker-data**, rest of the thin pool free for other guests.
|
||||
|
||||
## The design (locked)
|
||||
|
||||
- **Split for resilience, not backup size.** An isolated OS rootfs stays bootable + agent-recoverable
|
||||
if the Docker volume fills. Both volumes ride into PBS regardless (so the backup size is unchanged) —
|
||||
the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
|
||||
- **One Docker daemon, one data-root, one volume** at `/var/lib/docker` for ALL images + overlay
|
||||
(controller, traefik, cloudflared, filebrowser, and customer apps). Infra is NOT relocated to the OS
|
||||
partition — infra/user separation under one daemon isn't cleanly possible and isn't needed. Infra is
|
||||
protected by the controller's **prevention layer** (reserved buffer + deploy gate + disk alerts +
|
||||
baked log rotation), not by placement.
|
||||
- **`backup=1` on the Docker-data volume is non-negotiable.** Extra LXC mountpoints default to
|
||||
`backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs fall out of the
|
||||
snapshot AND the volume (with baked images) is excluded from the golden archive.
|
||||
|
||||
## Golden build (per controller-image baseline)
|
||||
|
||||
`configs/build-golden.sh` builds the template guest with the split baked in:
|
||||
|
||||
```
|
||||
# defaults shown; override via env
|
||||
OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 \
|
||||
./build-golden.sh <VMID> <TEMPLATE> <ROOTFS_STORAGE> <ARCHIVE_STORAGE> <BRIDGE> <CONTROLLER_IMAGE>
|
||||
```
|
||||
|
||||
- `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}` + `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`.
|
||||
- Keep `GOLDEN_DOCKER_GB` small (just the baked images + headroom) — provision grows it.
|
||||
- The script bakes `/etc/docker/daemon.json` log rotation (`max-size 10m`, `max-file 3`), verifies
|
||||
`/var/lib/docker` is a separate mount, and **aborts if vzdump excludes mp0** (the B3 trap).
|
||||
|
||||
## Provision (per customer guest)
|
||||
|
||||
The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data volume + baked images),
|
||||
then sizes per customer. The Docker-data volume is **grown** (not re-attached) so the baked images are
|
||||
never shadowed:
|
||||
|
||||
```
|
||||
felhom-agent --selftest=provision \
|
||||
-archive <golden-volid> -vmid <VMID> \
|
||||
-customer-id <id> -hub-password <retrieval-passphrase> \
|
||||
-rootfs-grow <N> # GiB to add to the OS rootfs beyond the golden's 32 (0 = keep 32)
|
||||
-datavol-grow <N> # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 240 → 256 total)
|
||||
-datavol-mount mp0 # the golden's Docker-data slot (default mp0)
|
||||
```
|
||||
|
||||
In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount}` — both grows are online
|
||||
(`pct resize`, spike B4), each its own task. `GuestMount.Backup` (→ `,backup=1`) is set on any
|
||||
data-bearing additive mount (`bringup.go` `buildConfigParams`). The bootstrap mount is `mp9`; USB/data
|
||||
drive enroll picks the lowest free slot (`freeMountSlot`), so it never collides with the docker mp0.
|
||||
|
||||
**Per-customer sizing source:** flags today; the **slice-10 hub storage manifest** is the intended
|
||||
home (marked at `bringup.go` `GuestMount` / the `slice 10 wires the hub storage manifest` comment) —
|
||||
the controller already pulls customer-scoped config from the hub, so OS/data sizes belong there.
|
||||
|
||||
### Default sizing guidance
|
||||
- ~512 GB SSD: 32 GB rootfs + 256 GB Docker-data (`-rootfs-grow 0 -datavol-grow 240`), leaving the rest
|
||||
of the thin pool free for other guests.
|
||||
- Other SSDs: size the data volume to "most of the free thin pool minus a reserve for other guests."
|
||||
`local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's written.
|
||||
|
||||
## Fresh PVE host install — thin-pool carving (host-level, do this at install)
|
||||
|
||||
This sizes the **host's** `local` root vs the `local-lvm` thin pool (the pool the guests' volumes are
|
||||
carved from). It is NOT the guest split — do it on the Proxmox ISO installer's **Advanced LVM options**:
|
||||
|
||||
- `hdsize` — total disk PVE uses (leave headroom on multi-disk nodes).
|
||||
- `maxroot` — cap on the host `/` (`pve/root`). Keep small (e.g. 20–40 GB) — the host OS is thin.
|
||||
- `maxvz` — cap on the `data` thin pool (`local-lvm`). Set to 0 / leave default to let the pool take the
|
||||
rest, OR cap it to reserve unallocated VG space.
|
||||
- `minfree` — reserved free space in the VG (thin-pool safety).
|
||||
- `swapsize` — host swap.
|
||||
|
||||
**Never repartition a live host root.** Per-guest carving (rootfs + Docker-data) is done at provision
|
||||
time via `pct` against the existing thin pool — no host repartition needed (the demo's pool already has
|
||||
~350 GB free).
|
||||
|
||||
## In-place migration of an EXISTING guest (NOT used by this slice)
|
||||
|
||||
This slice rebuilds clean (fresh re-provision), so the move-then-verify in-place migration is **not
|
||||
used here**. It remains documented in `REPORT-storage-split-spike.md` §B2 for a future in-place node
|
||||
upgrade: stop apps → stop docker → `rsync -aHAX --numeric-ids /var/lib/docker/ <new-vol>/` (check
|
||||
rsync's OWN exit code) → switch data-root → verify volumes+DBs+boot → only then reclaim. `backup=1` on
|
||||
the new mount and a CT restart to activate it are required (spike B3/B4).
|
||||
@@ -43,7 +43,7 @@ import (
|
||||
|
||||
// version is the agent version. Overridable at build time with
|
||||
// -ldflags "-X main.version=<v>"; defaults to the in-repo CHANGELOG version.
|
||||
var version = "0.28.0"
|
||||
var version = "0.29.0"
|
||||
|
||||
func main() {
|
||||
var (
|
||||
@@ -55,6 +55,9 @@ func main() {
|
||||
mode string
|
||||
hostname string
|
||||
keep bool
|
||||
rootfsGrow int
|
||||
dataVolGrow int
|
||||
dataVolMount string
|
||||
pbsStorage string
|
||||
paperkey bool
|
||||
offline bool
|
||||
@@ -79,6 +82,9 @@ func main() {
|
||||
flag.StringVar(&mode, "mode", "provision", "for --selftest=bring-up: `provision` (golden, fresh identity) | `dr` (customer backup, preserve continuity)")
|
||||
flag.StringVar(&hostname, "hostname", "", "for --selftest=bring-up provision: the hostname to set on the new guest")
|
||||
flag.BoolVar(&keep, "keep", false, "for --selftest=bring-up: KEEP the guest instead of tearing it down at the end")
|
||||
flag.IntVar(&rootfsGrow, "rootfs-grow", 0, "for --selftest=bring-up|provision: grow the OS rootfs by this many GiB after restore (0 = keep golden size)")
|
||||
flag.IntVar(&dataVolGrow, "datavol-grow", 0, "for --selftest=bring-up|provision: grow the golden's Docker-data volume (mp0) by this many GiB (0 = keep golden size)")
|
||||
flag.StringVar(&dataVolMount, "datavol-mount", "", "for --selftest=bring-up|provision: the mpN slot of the Docker-data volume to grow (default mp0)")
|
||||
flag.StringVar(&pbsStorage, "storage", "", "for --selftest=escrow-create: the pbs storage whose key to escrow (default: escrow.pbs_storage_id)")
|
||||
flag.BoolVar(&paperkey, "paperkey", false, "for --selftest=escrow-create: ALSO emit the raw-key paperkey (opt-in (a); single-factor, unrevocable)")
|
||||
flag.BoolVar(&offline, "offline", false, "for --selftest=escrow-create: ALSO emit the R-wrapped offline copy to print (opt-in (b))")
|
||||
@@ -133,11 +139,13 @@ func main() {
|
||||
case "lanresolver":
|
||||
os.Exit(runSelftestLANResolver(context.Background(), cfg, logger, vmid))
|
||||
case "bring-up":
|
||||
os.Exit(runSelftestBringUp(context.Background(), cfg, logger, mode, archive, vmid, hostname, keep))
|
||||
os.Exit(runSelftestBringUp(context.Background(), cfg, logger, mode, archive, vmid, hostname, keep,
|
||||
bringUpSizing{RootfsGrowGB: rootfsGrow, DataVolGrowGB: dataVolGrow, DataVolMount: dataVolMount}))
|
||||
case "provision":
|
||||
os.Exit(runSelftestProvision(context.Background(), cfg, logger, provisionArgs{
|
||||
archive: archive, vmid: vmid, hostname: hostname,
|
||||
customerID: custID, hubPassword: hubPassword,
|
||||
sizing: bringUpSizing{RootfsGrowGB: rootfsGrow, DataVolGrowGB: dataVolGrow, DataVolMount: dataVolMount},
|
||||
}))
|
||||
case "escrow-create":
|
||||
os.Exit(runSelftestEscrowCreate(context.Background(), cfg, logger, pbsStorage, paperkey, offline, upload, idBundlePath, directivePath))
|
||||
@@ -950,7 +958,16 @@ func runSelftestRestoreTest(ctx context.Context, cfg config.Config, logger *slog
|
||||
// (a selftest must not leave a guest running) unless -keep. -mode picks provision (golden, fresh
|
||||
// identity) or dr (customer backup, preserve continuity). It first Recovers, so a leaked guest
|
||||
// from a prior crashed bring-up is reaped before this run.
|
||||
func runSelftestBringUp(ctx context.Context, cfg config.Config, logger *slog.Logger, mode, archive string, vmid int, hostname string, keep bool) int {
|
||||
// bringUpSizing carries the OS-rootfs / Docker-data sizing knobs from the CLI into a bring-up.
|
||||
// Defaults (0/"") keep the golden's baked sizes; the provisioning spec sources these per-customer
|
||||
// (flags now; the slice-10 hub storage manifest later — see bringup.go GuestMount comment).
|
||||
type bringUpSizing struct {
|
||||
RootfsGrowGB int
|
||||
DataVolGrowGB int
|
||||
DataVolMount string
|
||||
}
|
||||
|
||||
func runSelftestBringUp(ctx context.Context, cfg config.Config, logger *slog.Logger, mode, archive string, vmid int, hostname string, keep bool, sizing bringUpSizing) int {
|
||||
if err := cfg.Validate(); err != nil {
|
||||
fmt.Fprintln(os.Stderr, "selftest: proxmox not configured:", err)
|
||||
return 1
|
||||
@@ -1006,6 +1023,7 @@ func runSelftestBringUp(ctx context.Context, cfg config.Config, logger *slog.Log
|
||||
spec := reconcile.BringUpSpec{
|
||||
Mode: bmode, Archive: archive, VMID: vmid, RestoreStorage: cfg.Backup.RestoreStorage,
|
||||
Hostname: hostname, KeepMAC: bmode == reconcile.ModeDRGuestLoss,
|
||||
RootfsGrowGB: sizing.RootfsGrowGB, DataVolGrowGB: sizing.DataVolGrowGB, DataVolMount: sizing.DataVolMount,
|
||||
}
|
||||
fmt.Printf(" bringing up %s → vmid %d on %s …\n", archive, vmid, cfg.Backup.RestoreStorage)
|
||||
res := engine.RunBringUp(ctx, spec)
|
||||
@@ -1049,6 +1067,7 @@ type provisionArgs struct {
|
||||
hostname string
|
||||
customerID string // baked into bootstrap (the hub config-pull target)
|
||||
hubPassword string // the customer's hub retrieval passphrase (SECRET) — baked into bootstrap
|
||||
sizing bringUpSizing // OS-rootfs / Docker-data sizing for the bring-up half
|
||||
}
|
||||
|
||||
// sanitizeHostname makes s a DNS-safe LXC hostname (RFC 1123 label-ish): lowercase, any run of
|
||||
@@ -1156,6 +1175,7 @@ func runSelftestProvision(ctx context.Context, cfg config.Config, logger *slog.L
|
||||
res := engine.RunBringUp(ctx, reconcile.BringUpSpec{
|
||||
Mode: reconcile.ModeProvision, Archive: a.archive, VMID: a.vmid,
|
||||
RestoreStorage: cfg.Backup.RestoreStorage, Hostname: hostname,
|
||||
RootfsGrowGB: a.sizing.RootfsGrowGB, DataVolGrowGB: a.sizing.DataVolGrowGB, DataVolMount: a.sizing.DataVolMount,
|
||||
})
|
||||
if res.Err != nil || !res.Pass {
|
||||
fmt.Fprintf(os.Stderr, " [FAIL] front-half bring-up (vmid %d): %v\n", a.vmid, res.Err)
|
||||
|
||||
+42
-6
@@ -23,6 +23,16 @@
|
||||
# Usage: build-golden.sh [VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] [CONTROLLER_IMAGE]
|
||||
# Build-time registry login for the controller pull (used ONCE inside the build guest, then logged
|
||||
# out — never baked): set REGISTRY_USER + REGISTRY_TOKEN in the environment.
|
||||
#
|
||||
# OS / Docker-data SPLIT (storage-split slice): the golden is built with a SMALL OS rootfs and a
|
||||
# SEPARATE Docker-data volume mounted at /var/lib/docker (mp0, backup=1). The baked controller +
|
||||
# infra images land on that volume and travel INSIDE the golden archive — so provisioned guests boot
|
||||
# from baked images with no registry pull. The split is for RESILIENCE: an isolated OS rootfs stays
|
||||
# bootable + agent-recoverable if the Docker volume fills (the controller's prevention layer keeps it
|
||||
# from filling). Sizes are env-overridable (OS_SIZE_GB / GOLDEN_DOCKER_GB); provision GROWS the data
|
||||
# volume to the per-customer target (bringup.go DataVolGrowGB). backup=1 is MANDATORY on the data mp:
|
||||
# without it vzdump EXCLUDES the volume (extra LXC mountpoints default backup=0 — storage-split B3),
|
||||
# so the archive would carry NO images and provisioned guests would boot imageless.
|
||||
set -euo pipefail
|
||||
|
||||
VMID="${1:-9100}"
|
||||
@@ -32,12 +42,17 @@ ARCHIVE_STORAGE="${4:-local}"
|
||||
BRIDGE="${5:-vmbr0}"
|
||||
CONTROLLER_IMAGE="${6:-gitea.dooplex.hu/admin/felhom-controller:0.43.0}"
|
||||
REGISTRY_HOST="${CONTROLLER_IMAGE%%/*}"
|
||||
# OS rootfs size (GiB) and the golden's Docker-data volume size (GiB). Keep GOLDEN_DOCKER_GB just
|
||||
# large enough for the baked images + headroom; provision grows it to the per-customer target.
|
||||
OS_SIZE_GB="${OS_SIZE_GB:-32}"
|
||||
GOLDEN_DOCKER_GB="${GOLDEN_DOCKER_GB:-16}"
|
||||
|
||||
echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged) …"
|
||||
echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G @ /var/lib/docker, backup=1) …"
|
||||
pct create "$VMID" "$TEMPLATE" \
|
||||
--hostname felhom-golden --unprivileged 1 \
|
||||
--features nesting=1,keyctl=1 \
|
||||
--rootfs "${ROOTFS_STORAGE}:8" --cores 2 --memory 2048 \
|
||||
--rootfs "${ROOTFS_STORAGE}:${OS_SIZE_GB}" --cores 2 --memory 2048 \
|
||||
--mp0 "${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1" \
|
||||
--net0 "name=eth0,bridge=${BRIDGE},ip=dhcp" --onboot 0
|
||||
|
||||
echo "[golden] starting + installing Docker (official repo, trixie channel) …"
|
||||
@@ -59,8 +74,20 @@ pct exec "$VMID" -- bash -c '
|
||||
apt-get update -qq
|
||||
apt-get install -y -qq docker-ce docker-ce-cli containerd.io >/dev/null
|
||||
'
|
||||
echo "[golden] verifying Docker works in the build guest …"
|
||||
pct exec "$VMID" -- bash -c 'systemctl start docker; sleep 2; docker run --rm hello-world >/dev/null && echo " docker OK ($(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"))"'
|
||||
echo "[golden] baking Docker log rotation into daemon.json (prevention layer: kills unbounded container logs for every guest) …"
|
||||
# /var/lib/docker is the mp0 mount (mounted empty before docker installs), so data-root needs NO
|
||||
# override — the existing image pulls + this config land on the volume automatically. Only the
|
||||
# json-file log caps are set (the most common runaway). Every container inherits these defaults.
|
||||
pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json <<JSON
|
||||
{
|
||||
"log-driver": "json-file",
|
||||
"log-opts": { "max-size": "10m", "max-file": "3" }
|
||||
}
|
||||
JSON'
|
||||
echo "[golden] verifying Docker works in the build guest (storage driver should be overlayfs on the ext4 data volume) …"
|
||||
pct exec "$VMID" -- bash -c 'systemctl start docker; sleep 2; docker run --rm hello-world >/dev/null && echo " docker OK ($(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); data-root $(docker info 2>/dev/null | sed -n "s/.*Docker Root Dir: //p"))"'
|
||||
# Confirm /var/lib/docker is genuinely the dedicated volume, not the rootfs (catch a silent mp miss).
|
||||
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo " /var/lib/docker is a separate mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker)" || { echo "[golden] FATAL: /var/lib/docker is NOT a separate mount — the mp0 split did not take"; exit 1; }'
|
||||
|
||||
echo "[golden] baking the in-guest controller image $CONTROLLER_IMAGE (no registry cred at deploy) …"
|
||||
# docker login is used ONCE here on the trusted build host, then logged out before archiving so
|
||||
@@ -197,8 +224,17 @@ pct exec "$VMID" -- bash -c '
|
||||
|
||||
echo "[golden] stop + archive …"
|
||||
pct stop "$VMID"
|
||||
vzdump "$VMID" --storage "$ARCHIVE_STORAGE" --mode stop --compress zstd
|
||||
# --mode stop with mp0 backup=1 → the Docker-data volume (with baked images) is INCLUDED. The log
|
||||
# below MUST show "including mount point mp0" — if it shows "excluding … (disabled)" the backup flag
|
||||
# was lost and the archive carries no images (storage-split B3 trap).
|
||||
vzdump "$VMID" --storage "$ARCHIVE_STORAGE" --mode stop --compress zstd 2>&1 | tee /tmp/golden-vzdump.log | grep -iE "including mount point|excluding|archive file size|Finished Backup" || true
|
||||
if grep -q "excluding volume mount point mp0" /tmp/golden-vzdump.log; then
|
||||
echo "[golden] FATAL: mp0 (/var/lib/docker) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no images. Aborting."
|
||||
exit 1
|
||||
fi
|
||||
grep -q "including mount point mp0" /tmp/golden-vzdump.log \
|
||||
|| echo "[golden] WARN: could not confirm mp0 inclusion in the vzdump log — verify manually before using this archive."
|
||||
|
||||
VOLID=$(pvesm list "$ARCHIVE_STORAGE" --content backup 2>/dev/null | awk -v v="$VMID" '$1 ~ ("vzdump-lxc-" v "-") {print $1}' | sort | tail -1)
|
||||
echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>}"
|
||||
echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G, both in the archive)"
|
||||
echo "[golden] (the build guest $VMID is stopped; destroy it with: pct destroy $VMID --purge)"
|
||||
|
||||
@@ -40,6 +40,9 @@ const (
|
||||
|
||||
const bringUpKind = "bring_up"
|
||||
|
||||
// DefaultDataVolMount is the mpN slot the golden bakes the Docker-data volume (/var/lib/docker) at.
|
||||
const DefaultDataVolMount = "mp0"
|
||||
|
||||
// configLockMaxAttempts bounds the F4 config-lock retry. configLockBackoff is a package var so
|
||||
// tests can shrink it (the production value gives PVE time to release its async config lock).
|
||||
const configLockMaxAttempts = 5
|
||||
@@ -52,6 +55,12 @@ type GuestMount struct {
|
||||
Storage string // PVE storage id (e.g. "local-lvm")
|
||||
SizeGB int // new-volume size in GiB
|
||||
MountPoint string // in-guest path (e.g. "/mnt/data")
|
||||
// Backup includes this mountpoint in vzdump/PBS. MANDATORY for any data-bearing mount (DB
|
||||
// volumes), because extra LXC mountpoints default to backup=0 = EXCLUDED from the snapshot
|
||||
// (storage-split finding B3). The Docker-data volume normally rides in from the golden archive
|
||||
// (already backup=1) and is grown via DataVolGrowGB rather than attached here, but any data
|
||||
// mount attached through spec.Mounts MUST set this or its contents silently fall out of PBS.
|
||||
Backup bool
|
||||
}
|
||||
|
||||
// BringUpSpec is the input to one bring-up. The caller resolves it (the selftest, or slice-10
|
||||
@@ -65,7 +74,14 @@ type BringUpSpec struct {
|
||||
Cores int // 0 = leave as restored
|
||||
MemoryMB int // 0 = leave as restored
|
||||
RootfsGrowGB int // optional grow-only rootfs resize (0 = skip)
|
||||
Mounts []GuestMount // additive mpN mounts (slice 7 may pass empty/test)
|
||||
// DataVolGrowGB grows the golden-carried Docker-data volume (DataVolMount, default mp0) to the
|
||||
// per-customer target. The golden ships a small data volume with the baked images; provision
|
||||
// grows it online (grow-only, storage-split B4) rather than attaching a fresh empty volume that
|
||||
// would shadow the baked images. 0 = skip (keep the golden's size).
|
||||
DataVolGrowGB int
|
||||
// DataVolMount is the mpN slot of the golden's Docker-data volume to grow; "" → DefaultDataVolMount ("mp0").
|
||||
DataVolMount string
|
||||
Mounts []GuestMount // additive mpN mounts (slice 7 may pass empty/test)
|
||||
KeepMAC bool // DR knob: keep the archived MAC (true) unless a source may be live
|
||||
BootTimeout time.Duration // 0 → DefaultBootTimeout; bounds the link-up liveness wait
|
||||
}
|
||||
@@ -213,6 +229,26 @@ func (e *Engine) runBringUp(ctx context.Context, spec BringUpSpec, res *BringUpR
|
||||
}
|
||||
}
|
||||
|
||||
// 4b. Grow the golden-carried Docker-data volume (mp0) to the per-customer target. Grow-only,
|
||||
// online (storage-split B4); its OWN call like the rootfs resize. The volume + baked images
|
||||
// came in with the restore, so we grow it rather than attach a fresh one that would shadow
|
||||
// the baked images.
|
||||
if spec.DataVolGrowGB > 0 {
|
||||
mount := spec.DataVolMount
|
||||
if mount == "" {
|
||||
mount = DefaultDataVolMount
|
||||
}
|
||||
dupid, err := e.api.ResizeLXC(ctx, spec.VMID, mount, fmt.Sprintf("+%dG", spec.DataVolGrowGB))
|
||||
if err != nil {
|
||||
res.Err = fmt.Errorf("reconcile: bring-up data-volume resize (%s): %w", mount, err)
|
||||
return
|
||||
}
|
||||
if _, err := e.waitTask(ctx, dupid, proxmox.WaitOptions{}); err != nil {
|
||||
res.Err = fmt.Errorf("reconcile: bring-up data-volume resize task (%s): %w", mount, err)
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
// Capture the post-reset MAC for the result (fresh for provision; archived for DR keep).
|
||||
if cfg2, err := e.api.GuestConfig(ctx, spec.VMID); err == nil {
|
||||
res.AssignedMAC = net0MAC(cfg2)
|
||||
@@ -310,7 +346,13 @@ func buildBringUpConfig(spec BringUpSpec, cfg proxmox.GuestConfig) map[string]st
|
||||
params["memory"] = strconv.Itoa(spec.MemoryMB)
|
||||
}
|
||||
for i, m := range spec.Mounts {
|
||||
params[fmt.Sprintf("mp%d", i)] = fmt.Sprintf("%s:%d,mp=%s", m.Storage, m.SizeGB, m.MountPoint)
|
||||
// backup=1 for data-bearing mounts: extra LXC mountpoints default to backup=0 = EXCLUDED
|
||||
// from vzdump/PBS (storage-split B3), which would silently drop their DBs from the snapshot.
|
||||
spec := fmt.Sprintf("%s:%d,mp=%s", m.Storage, m.SizeGB, m.MountPoint)
|
||||
if m.Backup {
|
||||
spec += ",backup=1"
|
||||
}
|
||||
params[fmt.Sprintf("mp%d", i)] = spec
|
||||
}
|
||||
return params
|
||||
}
|
||||
|
||||
@@ -74,6 +74,58 @@ func TestRunBringUp_ProvisionHappyPath(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
// A data-bearing additive mount must carry backup=1 (so its DBs stay in PBS — storage-split B3);
|
||||
// a non-backup mount must NOT. Pure-function check on buildBringUpConfig.
|
||||
func TestBuildBringUpConfig_BackupFlagOnDataMount(t *testing.T) {
|
||||
params := buildBringUpConfig(BringUpSpec{
|
||||
Mode: ModeProvision,
|
||||
Mounts: []GuestMount{
|
||||
{Storage: "local-lvm", SizeGB: 2, MountPoint: "/mnt/data", Backup: true},
|
||||
{Storage: "local-lvm", SizeGB: 1, MountPoint: "/mnt/scratch"}, // no backup
|
||||
},
|
||||
}, scratchCfg())
|
||||
if params["mp0"] != "local-lvm:2,mp=/mnt/data,backup=1" {
|
||||
t.Errorf("data mount must carry backup=1: mp0=%q", params["mp0"])
|
||||
}
|
||||
if params["mp1"] != "local-lvm:1,mp=/mnt/scratch" {
|
||||
t.Errorf("non-backup mount must NOT carry backup=1: mp1=%q", params["mp1"])
|
||||
}
|
||||
}
|
||||
|
||||
// The golden-carried Docker-data volume is grown via a SEPARATE resize on its mpN slot (B4),
|
||||
// alongside (but distinct from) the rootfs grow.
|
||||
func TestRunBringUp_StorageSplit_DataVolGrow(t *testing.T) {
|
||||
const vmid = 8050
|
||||
api := &fakeAPI{cfg: map[int]proxmox.GuestConfig{vmid: scratchCfg()}}
|
||||
e, _, q := newEngine(t, api, EmptyProvider{})
|
||||
defer q.Close()
|
||||
|
||||
res := e.RunBringUp(context.Background(), BringUpSpec{
|
||||
Mode: ModeProvision, Archive: "local:backup/golden.tar.zst", VMID: vmid,
|
||||
RestoreStorage: "local-lvm", Hostname: "felhom-prov-8050",
|
||||
RootfsGrowGB: 8, DataVolGrowGB: 240, // grows mp0 (DefaultDataVolMount)
|
||||
})
|
||||
if res.Err != nil || !res.Pass {
|
||||
t.Fatalf("provision must pass, got %+v", res)
|
||||
}
|
||||
// TWO resizes: rootfs +8G and the Docker-data volume mp0 +240G.
|
||||
if len(api.resizes) != 2 {
|
||||
t.Fatalf("expected rootfs + data-volume resizes, got %+v", api.resizes)
|
||||
}
|
||||
var sawRootfs, sawData bool
|
||||
for _, r := range api.resizes {
|
||||
if r.disk == "rootfs" && r.size == "+8G" {
|
||||
sawRootfs = true
|
||||
}
|
||||
if r.disk == "mp0" && r.size == "+240G" {
|
||||
sawData = true
|
||||
}
|
||||
}
|
||||
if !sawRootfs || !sawData {
|
||||
t.Errorf("want rootfs +8G AND mp0 +240G, got %+v", api.resizes)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunBringUp_CompensatingRollback(t *testing.T) {
|
||||
const vmid = 8000
|
||||
lockBackoffFast(t)
|
||||
|
||||
Reference in New Issue
Block a user