diff --git a/RUNBOOK-provisioning-storage.md b/RUNBOOK-provisioning-storage.md index 54ad330..e339d4b 100644 --- a/RUNBOOK-provisioning-storage.md +++ b/RUNBOOK-provisioning-storage.md @@ -31,8 +31,15 @@ OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 \ - `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}` + `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`. - Keep `GOLDEN_DOCKER_GB` small (just the baked images + headroom) — provision grows it. -- The script bakes `/etc/docker/daemon.json` log rotation (`max-size 10m`, `max-file 3`), verifies - `/var/lib/docker` is a separate mount, and **aborts if vzdump excludes mp0** (the B3 trap). +- The script bakes `/etc/docker/daemon.json` with **`features.containerd-snapshotter: false`** (the + classic **overlay2** driver) + log rotation (`max-size 10m`, `max-file 3`), verifies `/var/lib/docker` + is a separate mount + the driver is overlay2, and **aborts if vzdump excludes mp0** (the B3 trap). +- **Why overlay2 (not the Docker-29 default containerd-snapshotter):** containerd-snapshotter keeps the + image content store under `/var/lib/containerd`, which is NOT `/var/lib/docker` — so it would stay on + the OS rootfs and the split would only move named volumes, leaving images (the bulk) on the rootfs + (validated live: 1.2 GB of images landed on the rootfs). overlay2 stores **everything** (images + + overlay + volumes) under data-root = the data volume, which is what the split + the controller's + `statfs("/")` prevention guard both require. ## Provision (per customer guest) diff --git a/configs/build-golden.sh b/configs/build-golden.sh index e8d6b77..cc2092d 100644 --- a/configs/build-golden.sh +++ b/configs/build-golden.sh @@ -74,18 +74,27 @@ pct exec "$VMID" -- bash -c ' apt-get update -qq apt-get install -y -qq docker-ce docker-ce-cli containerd.io >/dev/null ' -echo "[golden] baking Docker log rotation into daemon.json (prevention layer: kills unbounded container logs for every guest) …" -# /var/lib/docker is the mp0 mount (mounted empty before docker installs), so data-root needs NO -# override — the existing image pulls + this config land on the volume automatically. Only the -# json-file log caps are set (the most common runaway). Every container inherits these defaults. +echo "[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …" +# containerd-snapshotter (Docker 28+/29 default) keeps the IMAGE content store under +# /var/lib/containerd — which is NOT /var/lib/docker, so it would stay on the OS rootfs and the split +# would only move named volumes, defeating the whole point (validated: images landed on the rootfs). +# The classic overlay2 driver stores EVERYTHING (images + overlay + volumes) under data-root +# (/var/lib/docker) = the data volume, which is exactly what "one data-root = one partition for all +# images + overlay" requires. It also makes the controller's statfs("/") (its overlay root) report the +# DATA volume, which the prevention layer depends on. /var/lib/docker is the mp0 mount (mounted empty +# before docker installs), so data-root needs no override. Log caps kill the most common runaway. pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json </dev/null && echo " docker OK ($(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); data-root $(docker info 2>/dev/null | sed -n "s/.*Docker Root Dir: //p"))"' +# Guard: the image store MUST be on the data volume now. /var/lib/containerd holding the images would +# mean containerd-snapshotter is still on (the split would leave images on the rootfs). +pct exec "$VMID" -- bash -c 'drv=$(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); [ "$drv" = "overlay2" ] || { echo "[golden] FATAL: storage driver is $drv, expected overlay2 — images would not land on the data volume"; exit 1; }' # Confirm /var/lib/docker is genuinely the dedicated volume, not the rootfs (catch a silent mp miss). pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo " /var/lib/docker is a separate mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker)" || { echo "[golden] FATAL: /var/lib/docker is NOT a separate mount — the mp0 split did not take"; exit 1; }'