golden: use classic overlay2 (containerd-snapshotter off) so images land on the data volume

Validation caught that Docker 29's default containerd-snapshotter keeps the
image store at /var/lib/containerd (on the rootfs), so mounting the data volume
at /var/lib/docker only moved named volumes — images (1.2G) stayed on the
rootfs, defeating the split. overlay2 stores images+overlay+volumes under
data-root = the data volume, which the split and the controller's statfs(/)
guard both require. Golden daemon.json now sets features.containerd-snapshotter
false + a driver guard.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

RUNBOOK-provisioning-storage.md

@@ -31,8 +31,15 @@ OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 \
 
 - `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}` + `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`.
 - Keep `GOLDEN_DOCKER_GB` small (just the baked images + headroom) — provision grows it.
-- The script bakes `/etc/docker/daemon.json` log rotation (`max-size 10m`, `max-file 3`), verifies
-  `/var/lib/docker` is a separate mount, and **aborts if vzdump excludes mp0** (the B3 trap).
+- The script bakes `/etc/docker/daemon.json` with **`features.containerd-snapshotter: false`** (the
+  classic **overlay2** driver) + log rotation (`max-size 10m`, `max-file 3`), verifies `/var/lib/docker`
+  is a separate mount + the driver is overlay2, and **aborts if vzdump excludes mp0** (the B3 trap).
+- **Why overlay2 (not the Docker-29 default containerd-snapshotter):** containerd-snapshotter keeps the
+  image content store under `/var/lib/containerd`, which is NOT `/var/lib/docker` — so it would stay on
+  the OS rootfs and the split would only move named volumes, leaving images (the bulk) on the rootfs
+  (validated live: 1.2 GB of images landed on the rootfs). overlay2 stores **everything** (images +
+  overlay + volumes) under data-root = the data volume, which is what the split + the controller's
+  `statfs("/")` prevention guard both require.
 
 ## Provision (per customer guest)
 

configs/build-golden.sh

@@ -74,18 +74,27 @@ pct exec "$VMID" -- bash -c '
   apt-get update -qq
   apt-get install -y -qq docker-ce docker-ce-cli containerd.io >/dev/null
 '
-echo "[golden] baking Docker log rotation into daemon.json (prevention layer: kills unbounded container logs for every guest) …"
-# /var/lib/docker is the mp0 mount (mounted empty before docker installs), so data-root needs NO
-# override — the existing image pulls + this config land on the volume automatically. Only the
-# json-file log caps are set (the most common runaway). Every container inherits these defaults.
+echo "[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …"
+# containerd-snapshotter (Docker 28+/29 default) keeps the IMAGE content store under
+# /var/lib/containerd — which is NOT /var/lib/docker, so it would stay on the OS rootfs and the split
+# would only move named volumes, defeating the whole point (validated: images landed on the rootfs).
+# The classic overlay2 driver stores EVERYTHING (images + overlay + volumes) under data-root
+# (/var/lib/docker) = the data volume, which is exactly what "one data-root = one partition for all
+# images + overlay" requires. It also makes the controller's statfs("/") (its overlay root) report the
+# DATA volume, which the prevention layer depends on. /var/lib/docker is the mp0 mount (mounted empty
+# before docker installs), so data-root needs no override. Log caps kill the most common runaway.
 pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json <<JSON
 {
+  "features": { "containerd-snapshotter": false },
   "log-driver": "json-file",
   "log-opts": { "max-size": "10m", "max-file": "3" }
 }
 JSON'
-echo "[golden] verifying Docker works in the build guest (storage driver should be overlayfs on the ext4 data volume) …"
+echo "[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …"
 pct exec "$VMID" -- bash -c 'systemctl start docker; sleep 2; docker run --rm hello-world >/dev/null && echo "  docker OK ($(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); data-root $(docker info 2>/dev/null | sed -n "s/.*Docker Root Dir: //p"))"'
+# Guard: the image store MUST be on the data volume now. /var/lib/containerd holding the images would
+# mean containerd-snapshotter is still on (the split would leave images on the rootfs).
+pct exec "$VMID" -- bash -c 'drv=$(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); [ "$drv" = "overlay2" ] || { echo "[golden] FATAL: storage driver is $drv, expected overlay2 — images would not land on the data volume"; exit 1; }'
 # Confirm /var/lib/docker is genuinely the dedicated volume, not the rootfs (catch a silent mp miss).
 pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo "  /var/lib/docker is a separate mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker)" || { echo "[golden] FATAL: /var/lib/docker is NOT a separate mount — the mp0 split did not take"; exit 1; }'