build-golden.sh 2.1.0 -> 3.0.0: a layout change is a major. The golden ships ONE data volume at a NEUTRAL path (/var/lib/felhom); both /var/lib/docker and /mnt/sys_drive are binds of subdirectories of it. mp1 is gone. The variant was chosen on measurement. Three candidates were built and rebooted (SPIKE-r165-phase0-2026-08-03.md); all three boot, reboot 3/3, give ONE df figure and keep a container's statfs(/) on the merged volume — the ordering worry that motivated the probe did not materialise. They differ only in which guarantee they break: volume-at-docker puts customer backups inside Docker's data-root; volume-at-sys_drive puts Docker's ENTIRE data-root under /mnt, which the controller container mounts wholesale (measured: it then sees /mnt/sys_drive/docker). V-c breaks neither. The four assertions were RETARGETED, never deleted, and each was RUN against a deliberately wrong shape — a real split guest and a real archive of it: 8 checks, 8 passed. A new 2b asserts both paths are ONE filesystem, which catches the S2 shape the spike ranked worse than the split. Assertion 5 replaces the old "was mp1 excluded?" guard, whose pattern could no longer match — a guard that cannot match has silently stopped guarding. Provisioning: one volume, one grow. SysDataGrowGB is FOLDED IN rather than dropped, because a census established that felhom-host-install.sh passes -sysdata-grow and the two do not upgrade in the same instant; dropping it would silently shrink every appliance by 42 of 250 GiB. The flags stay accepted for the same reason. The existing test was retargeted to pin the fold, and it caught the change before I did.
This commit is contained in:
+110
-39
@@ -26,20 +26,42 @@
|
||||
# Build-time registry login for the controller pull (used ONCE inside the build guest, then logged
|
||||
# out — never baked): set REGISTRY_USER + REGISTRY_TOKEN in the environment.
|
||||
#
|
||||
# OS / Docker-data SPLIT (storage-split slice): the golden is built with a SMALL OS rootfs and a
|
||||
# SEPARATE Docker-data volume mounted at /var/lib/docker (mp0, backup=1). The baked controller +
|
||||
# infra images land on that volume and travel INSIDE the golden archive — so provisioned guests boot
|
||||
# from baked images with no registry pull. The split is for RESILIENCE: an isolated OS rootfs stays
|
||||
# bootable + agent-recoverable if the Docker volume fills (the controller's prevention layer keeps it
|
||||
# from filling). Sizes are env-overridable (OS_SIZE_GB / GOLDEN_DOCKER_GB); provision GROWS the data
|
||||
# volume to the per-customer target (bringup.go DataVolGrowGB). backup=1 is MANDATORY on the data mp:
|
||||
# without it vzdump EXCLUDES the volume (extra LXC mountpoints default backup=0 — storage-split B3),
|
||||
# so the archive would carry NO images and provisioned guests would boot imageless.
|
||||
# OS / DATA SPLIT, and since v3.0.0 ONE DATA VOLUME (R-165, decision D-a + variant V-c).
|
||||
#
|
||||
# The golden is built with a SMALL OS rootfs and a SINGLE data volume (mp0, backup=1) mounted at a
|
||||
# NEUTRAL path, /var/lib/felhom. Both consumer paths are binds of subdirectories of it:
|
||||
#
|
||||
# /var/lib/felhom/docker --bind--> /var/lib/docker (Docker's data-root)
|
||||
# /var/lib/felhom/sys_drive --bind--> /mnt/sys_drive (the controller's system_data_path)
|
||||
#
|
||||
# WHAT THIS REPLACED, AND WHY. Until v2.1.0 these were TWO volumes (mp0 16 G at /var/lib/docker,
|
||||
# mp1 8 G at /mnt/sys_drive, grown separately at provision). The second one was a fixed ceiling: an
|
||||
# app whose local recovery unit outgrew it stopped being backed up even with free space next door.
|
||||
# D-a removed the wall rather than moving it — one volume, one free-space figure, no ceiling.
|
||||
#
|
||||
# WHY A NEUTRAL MOUNT AND NOT SIMPLY NESTING ONE PATH INSIDE THE OTHER. Both simpler shapes were
|
||||
# built and measured (SPIKE-r165-phase0-2026-08-03.md); both boot and reboot cleanly, and each breaks
|
||||
# a different documented guarantee:
|
||||
# * volume at /var/lib/docker -> customer backups live INSIDE Docker's data-root, so `du` there
|
||||
# stops meaning what it says and the ordinary "clear /var/lib/docker to fix Docker" reflex
|
||||
# destroys every local recovery unit on the box;
|
||||
# * volume at /mnt/sys_drive -> Docker's ENTIRE data-root lands under /mnt, which the controller
|
||||
# container mounts wholesale (`-v /mnt:/mnt:rslave`). Measured: the container then sees
|
||||
# /mnt/sys_drive/docker. The bootstrap's own claim that /mnt "holds only Felhom's
|
||||
# felhom-data-namespace mounts" would become false.
|
||||
# The neutral mount breaks neither, for one extra path and one extra fstab line.
|
||||
#
|
||||
# The split from the OS rootfs is still for RESILIENCE: an isolated rootfs stays bootable +
|
||||
# agent-recoverable if the data volume fills (the controller's prevention layer, and since
|
||||
# controller v0.192.0 the capture floor, keep it from filling). Size is env-overridable
|
||||
# (OS_SIZE_GB / GOLDEN_VOLUME_GB); provision GROWS the one volume (bringup.go DataVolGrowGB).
|
||||
# backup=1 is MANDATORY: without it vzdump EXCLUDES the volume (extra LXC mountpoints default
|
||||
# backup=0 — storage-split B3), so the archive would carry no images AND no user data.
|
||||
set -euo pipefail
|
||||
|
||||
# Script provenance — logged into every bake transcript next to the baked controller tag, so an
|
||||
# archive can always be traced to the script that produced it. Bump on any behavior change.
|
||||
GOLDEN_SCRIPT_VERSION="2.1.0"
|
||||
GOLDEN_SCRIPT_VERSION="3.0.0"
|
||||
|
||||
VMID="${1:-9100}"
|
||||
TEMPLATE="${2:-local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst}"
|
||||
@@ -59,25 +81,28 @@ if [ -z "$CONTROLLER_IMAGE" ]; then
|
||||
exit 1
|
||||
fi
|
||||
REGISTRY_HOST="${CONTROLLER_IMAGE%%/*}"
|
||||
# OS rootfs size (GiB) and the golden's Docker-data volume size (GiB). Keep GOLDEN_DOCKER_GB just
|
||||
# large enough for the baked images + headroom; provision grows it to the per-customer target.
|
||||
# OS rootfs size (GiB) and the golden's SINGLE data volume size (GiB).
|
||||
#
|
||||
# ONE VOLUME MEANS ONE NUMBER (v3.0.0). The retired GOLDEN_SYSDATA_GB has no successor: there is
|
||||
# nothing left to size separately. Keep GOLDEN_VOLUME_GB just large enough for the baked images plus
|
||||
# headroom for the controller's felhom-data skeleton; provision grows the one volume to the
|
||||
# per-customer target (bringup.go DataVolGrowGB).
|
||||
OS_SIZE_GB="${OS_SIZE_GB:-32}"
|
||||
GOLDEN_DOCKER_GB="${GOLDEN_DOCKER_GB:-16}"
|
||||
# The golden's SSD user-data volume (GiB) mounted at /mnt/sys_drive (mp1, backup=1) — the controller's
|
||||
# system_data_path. Ships small + near-empty (the controller creates <sys_drive>/felhom-data itself once
|
||||
# it's a real mountpoint); provision GROWS it to the per-customer target (bringup.go SysDataGrowGB). Like
|
||||
# mp0, backup=1 is MANDATORY: without it vzdump EXCLUDES the volume (extra mountpoints default backup=0 —
|
||||
# storage-split B3) and the user-data area would silently fall out of PBS coverage.
|
||||
GOLDEN_SYSDATA_GB="${GOLDEN_SYSDATA_GB:-8}"
|
||||
# 24 = the retired pair's 16 (docker) + 8 (user-data), so a golden archive carries the same content it
|
||||
# did before the merge. It is deliberately NOT a per-customer size: provision grows it.
|
||||
GOLDEN_VOLUME_GB="${GOLDEN_VOLUME_GB:-24}"
|
||||
# The neutral mount path of the single volume. Both consumer paths are binds of subdirectories of it.
|
||||
GOLDEN_VOLUME_MP="/var/lib/felhom"
|
||||
|
||||
echo "[golden] build-golden.sh v${GOLDEN_SCRIPT_VERSION} — baking controller ${CONTROLLER_IMAGE}"
|
||||
echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G @ /var/lib/docker + user-data ${GOLDEN_SYSDATA_GB}G @ /mnt/sys_drive, both backup=1) …"
|
||||
echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + ONE data volume ${GOLDEN_VOLUME_GB}G @ ${GOLDEN_VOLUME_MP}, backup=1) …"
|
||||
# ONE mpN slot. There is deliberately no mp1: that slot held the retired user-data volume, and the
|
||||
# whole point of R-165 is that it stops existing rather than being made bigger.
|
||||
pct create "$VMID" "$TEMPLATE" \
|
||||
--hostname felhom-golden --unprivileged 1 \
|
||||
--features nesting=1,keyctl=1 \
|
||||
--rootfs "${ROOTFS_STORAGE}:${OS_SIZE_GB}" --cores 2 --memory 2048 \
|
||||
--mp0 "${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1" \
|
||||
--mp1 "${ROOTFS_STORAGE}:${GOLDEN_SYSDATA_GB},mp=/mnt/sys_drive,backup=1" \
|
||||
--mp0 "${ROOTFS_STORAGE}:${GOLDEN_VOLUME_GB},mp=${GOLDEN_VOLUME_MP},backup=1" \
|
||||
--net0 "name=eth0,bridge=${BRIDGE},ip=dhcp" --onboot 0
|
||||
|
||||
echo "[golden] starting + installing Docker (official repo, trixie channel) …"
|
||||
@@ -106,8 +131,10 @@ echo "[golden] baking daemon.json: classic overlay2 driver (containerd-snapshott
|
||||
# The classic overlay2 driver stores EVERYTHING (images + overlay + volumes) under data-root
|
||||
# (/var/lib/docker) = the data volume, which is exactly what "one data-root = one partition for all
|
||||
# images + overlay" requires. It also makes the controller's statfs("/") (its overlay root) report the
|
||||
# DATA volume, which the prevention layer depends on. /var/lib/docker is the mp0 mount (mounted empty
|
||||
# before docker installs), so data-root needs no override. Log caps kill the most common runaway.
|
||||
# DATA volume, which the prevention layer depends on — MEASURED to still hold under the v3.0.0 merged
|
||||
# layout (a container's `df /` reports the single volume, phase-0 spike). Since v3.0.0 /var/lib/docker
|
||||
# is a BIND of <volume>/docker rather than the mp0 mount itself, wired immediately below; data-root
|
||||
# still needs no override because the path is unchanged. Log caps kill the most common runaway.
|
||||
pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json <<JSON
|
||||
{
|
||||
"features": { "containerd-snapshotter": false },
|
||||
@@ -115,6 +142,32 @@ pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json
|
||||
"log-opts": { "max-size": "10m", "max-file": "3" }
|
||||
}
|
||||
JSON'
|
||||
echo "[golden] wiring the single data volume (R-165 variant V-c): ${GOLDEN_VOLUME_MP}/{docker,sys_drive} -> binds …"
|
||||
# docker-ce has already populated /var/lib/docker ON THE ROOTFS by now (it auto-starts on install), so
|
||||
# the content is MOVED onto the volume before the bind is laid over the top. Doing it the other way
|
||||
# round would hide those files under the bind and silently ship a golden whose baked images are on the
|
||||
# rootfs — the exact failure class the assertions below exist to catch.
|
||||
#
|
||||
# /etc/fstab, not a hand-run mount: systemd's fstab generator orders both binds under local-fs.target,
|
||||
# which precedes basic.target and therefore docker.service. MEASURED across 3 reboots per variant in
|
||||
# the phase-0 spike — the ordering worry that motivated the probe did not materialise.
|
||||
pct exec "$VMID" -- bash -c "
|
||||
set -e
|
||||
systemctl stop docker docker.socket containerd 2>/dev/null || true
|
||||
mkdir -p '${GOLDEN_VOLUME_MP}/docker' '${GOLDEN_VOLUME_MP}/sys_drive'
|
||||
if [ -d /var/lib/docker ] && [ -n \"\$(ls -A /var/lib/docker 2>/dev/null)\" ]; then
|
||||
cp -a /var/lib/docker/. '${GOLDEN_VOLUME_MP}/docker'/
|
||||
rm -rf /var/lib/docker/*
|
||||
fi
|
||||
mkdir -p /var/lib/docker /mnt/sys_drive
|
||||
printf '%s /var/lib/docker none bind 0 0\n' '${GOLDEN_VOLUME_MP}/docker' >> /etc/fstab
|
||||
printf '%s /mnt/sys_drive none bind 0 0\n' '${GOLDEN_VOLUME_MP}/sys_drive' >> /etc/fstab
|
||||
systemctl daemon-reload
|
||||
mount /var/lib/docker
|
||||
mount /mnt/sys_drive
|
||||
systemctl start containerd
|
||||
"
|
||||
|
||||
echo "[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …"
|
||||
# RESTART (not start): docker-ce auto-starts on install with the DEFAULT config, so it is already
|
||||
# running by now; only a restart picks up the daemon.json just written (overlay2 + log caps).
|
||||
@@ -122,12 +175,20 @@ pct exec "$VMID" -- bash -c 'systemctl restart docker; sleep 3; docker run --rm
|
||||
# Guard: the image store MUST be on the data volume now. /var/lib/containerd holding the images would
|
||||
# mean containerd-snapshotter is still on (the split would leave images on the rootfs).
|
||||
pct exec "$VMID" -- bash -c 'drv=$(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); [ "$drv" = "overlay2" ] || { echo "[golden] FATAL: storage driver is $drv, expected overlay2 — images would not land on the data volume"; exit 1; }'
|
||||
# Confirm /var/lib/docker is genuinely the dedicated volume, not the rootfs (catch a silent mp miss).
|
||||
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo " /var/lib/docker is a separate mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker)" || { echo "[golden] FATAL: /var/lib/docker is NOT a separate mount — the mp0 split did not take"; exit 1; }'
|
||||
# Same guard for the SSD user-data volume (mp1): /mnt/sys_drive must be its own mount, not the rootfs
|
||||
# device — otherwise the controller's system_data_path lands on the OS drive and it warns (the whole
|
||||
# point of this volume is to clear that warning).
|
||||
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . && echo " /mnt/sys_drive is a separate mount: $(findmnt -no SOURCE,FSTYPE /mnt/sys_drive)" || { echo "[golden] FATAL: /mnt/sys_drive is NOT a separate mount — the mp1 split did not take"; exit 1; }'
|
||||
# ASSERTION 1 (RETARGETED v3.0.0, not removed). /var/lib/docker must be a real mount — now the V-c
|
||||
# bind of <volume>/docker rather than the mp0 mount itself. Still fails closed on the same failure:
|
||||
# if the bind did not take, Docker's data-root silently sits on the OS rootfs and the golden ships
|
||||
# its baked images there.
|
||||
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo " /var/lib/docker is a real mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker | head -1)" || { echo "[golden] FATAL: /var/lib/docker is NOT a mount — the V-c docker bind did not take, so the baked images would land on the OS rootfs"; exit 1; }'
|
||||
# ASSERTION 2 (RETARGETED v3.0.0). /mnt/sys_drive must be a real mount — now the V-c bind of
|
||||
# <volume>/sys_drive. Otherwise the controller's system_data_path lands on the OS drive and it warns
|
||||
# (clearing that warning is the whole point of the volume).
|
||||
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /mnt/sys_drive | grep -q . && echo " /mnt/sys_drive is a real mount: $(findmnt -no SOURCE,FSTYPE /mnt/sys_drive | head -1)" || { echo "[golden] FATAL: /mnt/sys_drive is NOT a mount — the V-c sys_drive bind did not take, so the controller system_data_path would be the OS rootfs"; exit 1; }'
|
||||
# ASSERTION 2b (NEW v3.0.0 — the invariant the merge is FOR). Both paths must be backed by the SAME
|
||||
# device, i.e. ONE filesystem with ONE free-space figure. Two devices here is the S2 shape the R-165
|
||||
# spike ranked strictly WORSE than the split it replaced: every assertion satisfied, the ceiling still
|
||||
# there, and a shared pool neither `df` can see coming.
|
||||
pct exec "$VMID" -- bash -c 'n=$(df --output=source /var/lib/docker /mnt/sys_drive | tail -n +2 | sort -u | wc -l); [ "$n" = "1" ] && echo " both paths are ONE filesystem: $(df --output=source,avail /var/lib/docker | tail -1)" || { echo "[golden] FATAL: /var/lib/docker and /mnt/sys_drive are on $n DIFFERENT filesystems — that is the S2 shape (two ceilings), not the R-165 merge"; exit 1; }'
|
||||
|
||||
echo "[golden] baking the in-guest controller image $CONTROLLER_IMAGE (no registry cred at deploy) …"
|
||||
# docker login is used ONCE here on the trusted build host, then logged out before archiving so
|
||||
@@ -307,26 +368,36 @@ pct exec "$VMID" -- bash -c '
|
||||
|
||||
echo "[golden] stop + archive …"
|
||||
pct stop "$VMID"
|
||||
# --mode stop with mp0 + mp1 backup=1 → BOTH the Docker-data volume (baked images) and the
|
||||
# /mnt/sys_drive user-data volume are INCLUDED. The log below MUST show "including mount point mp0"
|
||||
# AND "including mount point mp1" — if either shows "excluding … (disabled)" the backup flag was lost
|
||||
# and the archive carries no images / no user-data volume (storage-split B3 trap).
|
||||
# --mode stop with mp0 backup=1 → the SINGLE data volume (baked images AND the user-data area) is
|
||||
# INCLUDED. The log MUST show "including mount point mp0" and must NOT show it being excluded — an
|
||||
# exclusion means the backup flag was lost and the archive carries neither (storage-split B3 trap).
|
||||
# Since v3.0.0 there is no mp1; the guard that covered it is retargeted below rather than deleted,
|
||||
# because a guard whose pattern can no longer match is a guard that has silently stopped guarding.
|
||||
vzdump "$VMID" --storage "$ARCHIVE_STORAGE" --mode stop --compress zstd 2>&1 | tee /tmp/golden-vzdump.log | grep -iE "including mount point|excluding|archive file size|Finished Backup" || true
|
||||
if grep -q "excluding volume mount point mp0" /tmp/golden-vzdump.log; then
|
||||
echo "[golden] FATAL: mp0 (/var/lib/docker) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no images. Aborting."
|
||||
exit 1
|
||||
fi
|
||||
if grep -q "excluding volume mount point mp1" /tmp/golden-vzdump.log; then
|
||||
echo "[golden] FATAL: mp1 (/mnt/sys_drive) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no user-data volume. Aborting."
|
||||
# ASSERTION 4 (RETARGETED v3.0.0). The mp1 guard used to catch "the user-data volume fell out of the
|
||||
# archive". After the merge there is no mp1 — so the same failure now looks like the volume being
|
||||
# mounted at the WRONG PATH, which would carry the images but not the user-data area. Assert the
|
||||
# inclusion line names the volume's actual mount path.
|
||||
if ! grep -q "including mount point mp0 ('${GOLDEN_VOLUME_MP}')" /tmp/golden-vzdump.log; then
|
||||
echo "[golden] FATAL: the archive's mp0 is not ${GOLDEN_VOLUME_MP} — the single data volume is mounted somewhere unexpected, so the archive would not carry both the baked images and the user-data area. Aborting."
|
||||
grep -iE "mount point" /tmp/golden-vzdump.log || true
|
||||
exit 1
|
||||
fi
|
||||
# ASSERTION 5 (RETARGETED v3.0.0). There must be NO mp1 in the archive at all. A leftover second
|
||||
# volume means the merge did not take and this golden would ship the very ceiling R-165 removed.
|
||||
if grep -qE "mount point mp1" /tmp/golden-vzdump.log; then
|
||||
echo "[golden] FATAL: the archive still carries an mp1 — the R-165 merge did not take and this golden would ship a second, ceilinged volume. Aborting."
|
||||
exit 1
|
||||
fi
|
||||
grep -q "including mount point mp0" /tmp/golden-vzdump.log \
|
||||
|| echo "[golden] WARN: could not confirm mp0 inclusion in the vzdump log — verify manually before using this archive."
|
||||
grep -q "including mount point mp1" /tmp/golden-vzdump.log \
|
||||
|| echo "[golden] WARN: could not confirm mp1 inclusion in the vzdump log — verify manually before using this archive."
|
||||
|
||||
VOLID=$(pvesm list "$ARCHIVE_STORAGE" --content backup 2>/dev/null | awk -v v="$VMID" '$1 ~ ("vzdump-lxc-" v "-") {print $1}' | sort | tail -1)
|
||||
echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G + user-data ${GOLDEN_SYSDATA_GB}G, all in the archive)"
|
||||
echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + ONE data volume ${GOLDEN_VOLUME_GB}G @ ${GOLDEN_VOLUME_MP}, all in the archive)"
|
||||
|
||||
#-------------------------------------------------------------------------------
|
||||
# Publish to Gitea (BUNDLE slice) — make this golden fetchable by the host-bootstrap script.
|
||||
|
||||
Reference in New Issue
Block a user