felhom-agent/REPORT-storage-split-spike.md

# Findings — OS/Docker-data storage-split spike (Part B)

**Date:** 2026-06-13 · **Host:** demo-felhom (PVE 9.2.2) · **Throwaway:** unprivileged LXC 9300 (created, exercised, destroyed) · **Builds nothing** — this report gates the provisioning spec.

**Goal:** prove the mechanics of splitting the controller guest's **OS rootfs** from its **Docker/data** onto separate `local-lvm` volumes, and the live-migration path, before writing the provisioning spec. Target shape to validate: a small OS rootfs (~32 GB default) + a separate large Docker-data volume (~256 GB), sizes configurable at install.

**Method:** all risky steps were run on a throwaway unprivileged LXC (9300) replicating guest 9201's config (`unprivileged: 1`, `features nesting=1,keyctl=1`, rootfs on `local-lvm`), never on 9201's live data. Docker 29.5.3 — **identical to production 9201** (verified: both `overlayfs` driver + `io.containerd.snapshotter.v1`).

---

## Verdict summary

| Item | Verdict |
|---|---|
| B1 — second volume + Docker `data-root` on unprivileged LXC | ✅ **Works.** overlayfs on the ext4 `local-lvm` mountpoint; no idmap/ownership problem; survives reboot. |
| B2 — live move-then-verify migration | ✅ **Works**, copy-not-move is the safety net. Caveat: rsync must exist + its exit code must be checked. |
| B3 — vzdump/PBS coverage of the new mountpoint | ⚠️ **NOT by default — contradicts the spec's premise.** Extra CT mountpoints are **excluded** from vzdump unless `backup=1` is set **and the CT restarted**. |
| B4 — live resize vs fresh-install carving | ✅ Live grow is online/non-disruptive (`pct resize`). Fresh-install carving = installer `hdsize`/`maxroot`/`maxvz` (host-level, documented below). |
| B5 — provisioning seams | ✅ Mapped: golden rootfs size + `BringUpSpec.RootfsGrowGB`/`Mounts`; `bringup.go:313` is the exact spot that today omits `backup=1`. |

---

## B1 — Second volume + Docker `data-root` on an unprivileged LXC ✅

- Added a second `local-lvm` mountpoint: `pct set 9300 -mp0 local-lvm:16,mp=/mnt/dockerdata`. PVE created `vm-9300-disk-1`, formatted it **ext4**, mounted it; in-container it is `root:root 0755` — **no idmap/ownership fix needed** (PVE maps the unprivileged offset for a fresh volume automatically).
- Pointed Docker at it via `/etc/docker/daemon.json` `{"data-root":"/mnt/dockerdata"}` → after `systemctl restart docker`, `docker info` reports `Docker Root Dir: /mnt/dockerdata`, `Storage Driver: overlayfs`.
- **overlay works on the mountpoint's filesystem** (ext4): pulled an image + created a named volume + wrote to both the volume and the container's overlay layer. Image/volume bytes landed on `/mnt/dockerdata` (228K → 64M) while the rootfs `/var/lib/docker` stayed flat (232K).
- **Survives reboot:** after `pct reboot`, `data-root` is still `/mnt/dockerdata`, the volume's marker file persists, the image is present.

**Gotcha (Docker 29 / containerd-snapshotter):** the daemon already auto-starts at boot, so `systemctl start docker` is a no-op and won't pick up a freshly-written `daemon.json` — use `systemctl restart docker`. Also note the driver is the new **`overlayfs`** (containerd-snapshotter), not legacy `overlay2` — relevant to B2.

## B2 — Live migration sequence (move-then-verify) ✅

Proven on the throwaway with seeded **running** apps (postgres with a known row `migrate-survive` in a named volume + nginx). The safe sequence:

1. `docker stop <apps>` 2. `systemctl stop docker docker.socket` 3. **copy, not move:** `rsync -aHAX --numeric-ids /var/lib/docker/ /mnt/dockerdata/` 4. write `data-root` → `/mnt/dockerdata` in `daemon.json` 5. `systemctl start docker` 6. **verify** images + `docker volume ls` + start apps + query the DB row 7. **only then** reclaim the old location (`rm -rf /var/lib/docker/*`).

Result: post-switch the `pgdata` volume was present, both apps booted, and the DB row was intact. Reclaiming the old path while apps ran on the new `data-root` did not disturb them — the new `data-root` is fully self-sufficient.

**Two hard lessons (both nearly caused a silent false-positive):**
- **`rsync` was not installed** on the fresh guest; the first attempt's `rsync … | grep …` swallowed the `127` exit (pipe returns grep's status, `set -e` doesn't fire), so the migration "completed" having copied **nothing**. The migration script MUST install rsync and **check rsync's own exit code**, not a piped one.
- With **containerd-snapshotter** the image content store sits under `data-root/containerd…`; switching `data-root` to an empty volume made images *appear* present (they resolve from the snapshotter's own addressing) while volumes/containers were empty — a misleading partial state. The reliable approach is to **rsync the entire `data-root` with docker stopped** (captures the containerd store + volumes + containers) and verify functional integrity (volume + DB row + boot), or treat images as re-pullable and migrate only the volumes (the irreplaceable data). The copy-not-move discipline meant the source was always intact and recoverable — I reverted `data-root` and the original apps + DB row came straight back.

## B3 — Backup coverage after the split ⚠️ (the load-bearing finding)

**The spec's premise is wrong.** It assumed the new mountpoint "should be [captured], as a CT mountpoint unless `backup=0`." The opposite is true on PVE 9.2.2:

```
# mp0 with NO backup flag:
INFO: excluding volume mount point mp0 ('/mnt/dockerdata') from backup (disabled)
```

Additional LXC mountpoints are **excluded from vzdump by default**. They are included **only** when `backup=1` is set on the mountpoint. And setting it on a running CT is a **pending change** (`[pve:pending]`) — vzdump kept excluding mp0 until the **CT was restarted**, after which:

```
# mp0 with backup=1, post-restart:
INFO: including mount point mp0 ('/mnt/dockerdata') in backup
```

Confirmed end-to-end: the resulting archive contains `./mnt/dockerdata/volumes/pgdata/_data/…` — the DB data is in PBS/vzdump only once `backup=1` is active.

**Implication:** if the OS/data split lands without `backup=1` on the docker-data mountpoint, **every named-volume database silently falls out of the PBS whole-guest snapshot** — exactly the class of silent-default trap the spike exists to catch (cf. the PBS `ignore-verified` default). This is the single must-fix for the provisioning spec.

Honest note (as the spec asked): because **both** volumes stay in PBS once `backup=1` is set, the backup **size does not shrink** — the win of the split is independent sizing/growth and a smaller, faster-to-restore OS rootfs, **not** a smaller backup. (Excluding images from PBS would shrink it but would drop DB coverage — out of scope; not done.)

## B4 — Resize paths: live vs fresh install ✅

- **Live (within the existing 349 GB `local-lvm`), non-disruptive:** `pct resize 9300 rootfs +2G` and `pct resize 9300 mp0 +4G` both grew the volume **and** the in-guest ext4 online (7.8→9.8 GB rootfs, 16→20 GB docker-data) with the CT running and the DB still queryable — no reboot, no downtime. Adding a brand-new mountpoint (`pct set -mpN`) is also non-disruptive to the rootfs, but the **mount itself only appears after a CT restart** on an unprivileged guest (same pending-activation behaviour seen with `backup=1` and with drive binds — consistent with the agent's existing "activate at next boot" note). Grow-only is safe; shrinking LVM-thin volumes is not supported online and was not attempted. The host root (`local`) was **not** repartitioned on the live system.
- **Fresh install (host-level carving — this is where "configurable at install" lives):** the Proxmox ISO installer's *Advanced LVM options* expose `hdsize` (total of the disk PVE uses), `maxroot` (cap on the host `/` = `pve/root`), `maxvz` (cap on the `data` thin pool = `local-lvm`; set to 0 / leave headroom to keep space unallocated), `minfree` (reserved free space in the VG), and `swapsize`. These decide how the SSD is split between a small **host** root and a large **`local-lvm` thin pool**. Note the distinction the spec's target shape blurs: those installer knobs size the **host's** local-vs-thinpool; the **guest's** 32 GB-rootfs + 256 GB-docker-data split is sized at *provision* time via `pct` (rootfs size + an additive mountpoint), carved out of the thin pool — not by the installer. On the demo, the thin pool already has ~350 GB free, so per-guest carving needs no host repartition.

## B5 — Provisioning integration points ✅

Where the guest is sized today, and where the split + configurable sizes slot in:

- **Golden rootfs size — `configs/build-golden.sh:40`:** `--rootfs "${ROOTFS_STORAGE}:8"` — the OS rootfs is **hardcoded at 8 GB**. The golden also `docker pull`s the controller + app images into `/var/lib/docker` **on the rootfs** (lines ~71–92), so the baked-image set is bounded by this size. `ROOTFS_STORAGE` is the 3rd script arg (default `local-lvm`). The golden runs the same docker-ce as production (→ 29.5.3 overlayfs/containerd-snapshotter).
- **Provision bring-up — `internal/reconcile/bringup.go`:** `BringUpSpec` already carries the right seams: `RestoreStorage` (rootfs target storage), `RootfsGrowGB` (grow-only rootfs resize, applied as its own `ResizeLXC` call at line ~205), and `Mounts []GuestMount{Storage,SizeGB,MountPoint}` (additive `mpN`, attached in `buildConfigParams` at line ~313). So a docker-data volume is a natural `GuestMount`, and the OS rootfs can be grown per-customer via `RootfsGrowGB`.
- **🔴 Exact code spot for the B3 fix — `bringup.go:313`:** today it builds `fmt.Sprintf("%s:%d,mp=%s", m.Storage, m.SizeGB, m.MountPoint)` — **no `backup=` flag**, so any additive mount (including a future docker-data volume) is created `backup=0` = **excluded from PBS**. The docker-data mount MUST be attached with `,backup=1` (and the provision flow must account for the restart-to-activate behaviour). `GuestMount` should gain a `Backup bool` (or always-on for the data mount).
- **Where the size config comes from:** currently caller-provided — the `--selftest=provision`/bring-up path and `cfg.Backup.RestoreStorage`. The in-code marker (`bringup.go:49-50`) states slice 10 wires the **hub storage manifest** into `Mounts`; that is the right home for per-customer sizes, consistent with bootstrap-v2 (the controller already pulls customer-scoped config from the hub). So "configurable at install" = a per-customer field in the hub desired-state → `RootfsGrowGB` + `GuestMount.SizeGB`.

---

## Open questions for the provisioning spec to resolve

1. **Where does `data-root` get redirected for a *fresh* guest?** The golden bakes images into `/var/lib/docker` on the rootfs. Mounting an empty data volume **over** `/var/lib/docker` hides the baked images. Options: (a) bake `daemon.json` `data-root=/mnt/dockerdata` into the golden and attach the (empty) data mount before first Docker start, re-pulling images on first boot; (b) bake the split into the golden itself (data volume present at golden-build time, images baked onto it); (c) first-boot migration step. Pick one; (b) keeps the baked-image fast-start, (a) is simplest but loses it.
2. **`backup=1` is mandatory on the docker-data mount** (B3). Encode it at `bringup.go:313` and decide whether the OS rootfs alone (no app data) is worth a separate lighter backup cadence.
3. **Restart-to-activate** (B3/B4): attaching a mount or flipping `backup=1` on a running unprivileged guest is pending until reboot. Provisioning attaches mounts *before* first start, so this is a non-issue at provision time, but any *post-hoc* migration of an existing customer guest must schedule a reboot.
4. **Default sizes & source:** confirm 32 GB rootfs / 256 GB docker-data defaults, and wire them through the hub storage manifest (per-customer), sized to the actual SSD via the installer carving facts in B4.
5. **Backup size does not shrink** (B3): set expectations — the split is for independent sizing/growth and a smaller OS-rootfs restore, not a smaller PBS footprint.

**Nothing was shipped.** Throwaway LXC 9300 and its test archives were destroyed after the spike.