v0.29.0: OS/Docker-data storage split — golden + provision (Phase 1)
build-golden.sh bakes a small OS rootfs + a dedicated /var/lib/docker volume (mp0, backup=1) carrying the baked images, plus Docker log rotation. bringup.go grows the golden's data volume to the per-customer target (DataVolGrowGB) and emits backup=1 on data mounts (GuestMount.Backup) — closing the spike-B3 silent DB-loss trap. CLI gains -rootfs-grow/-datavol-grow/-datavol-mount. New RUNBOOK-provisioning-storage.md. Phase 2 = felhom-controller v0.58.0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
@@ -3,6 +3,31 @@
|
|||||||
All notable changes to **felhom-agent** are recorded here. Update on every code
|
All notable changes to **felhom-agent** are recorded here. Update on every code
|
||||||
change that gets pushed.
|
change that gets pushed.
|
||||||
|
|
||||||
|
## v0.29.0 — OS / Docker-data storage split: golden + provision (2026-06-13)
|
||||||
|
|
||||||
|
Phase 1 of the storage-split slice (Phase 2 = felhom-controller v0.58.0 prevention layer). The
|
||||||
|
controller guest's OS rootfs and Docker data are carved onto separate `local-lvm` volumes for
|
||||||
|
RESILIENCE — an isolated OS rootfs stays bootable + agent-recoverable if the Docker volume fills.
|
||||||
|
|
||||||
|
- **`configs/build-golden.sh` — split baked in:** `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}` (default
|
||||||
|
**32**, was hardcoded 8) **plus** `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`
|
||||||
|
(default 16). The baked controller + infra images land on the data volume and travel inside the
|
||||||
|
golden archive (no empty-volume shadowing, no deploy-time pull). `backup=1` is MANDATORY — extra LXC
|
||||||
|
mountpoints default to `backup=0` = EXCLUDED from vzdump (spike B3), which would drop the images from
|
||||||
|
the archive entirely. The script now also bakes Docker **log rotation** into `daemon.json`
|
||||||
|
(`max-size 10m`, `max-file 3` — prevention layer 2D), asserts `/var/lib/docker` is a separate mount,
|
||||||
|
and **aborts if vzdump excludes mp0**.
|
||||||
|
- **`internal/reconcile/bringup.go` — sized provision:** `GuestMount` gains `Backup` (emits `,backup=1`
|
||||||
|
— closes the spike-B3/B5 silent-DB-loss trap at the mount builder). `BringUpSpec` gains
|
||||||
|
`DataVolGrowGB` + `DataVolMount` (default `mp0`): provision GROWS the golden-carried Docker-data
|
||||||
|
volume online to the per-customer target (grow-only, spike B4) rather than attaching a fresh empty
|
||||||
|
volume that would shadow the baked images. Plus `RootfsGrowGB` for the OS rootfs.
|
||||||
|
- **CLI seam:** `--selftest=bring-up|provision` gain `-rootfs-grow` / `-datavol-grow` / `-datavol-mount`
|
||||||
|
flags. Per-customer sizing source = flags now, the slice-10 hub storage manifest later.
|
||||||
|
- **`RUNBOOK-provisioning-storage.md`** (new): the split provisioning procedure + fresh-PVE-install
|
||||||
|
thin-pool carving knobs (`hdsize`/`maxroot`/`maxvz`, spike B4) + the per-customer sizing seam.
|
||||||
|
- Tests: `buildBringUpConfig` backup=1 emission; bring-up issues rootfs + data-volume resizes.
|
||||||
|
|
||||||
## (no version) — storage OS/data-split spike findings (2026-06-13)
|
## (no version) — storage OS/data-split spike findings (2026-06-13)
|
||||||
|
|
||||||
Investigation only — **no code changed**. Findings report: `REPORT-storage-split-spike.md` (gates the
|
Investigation only — **no code changed**. Findings report: `REPORT-storage-split-spike.md` (gates the
|
||||||
|
|||||||
@@ -0,0 +1,89 @@
|
|||||||
|
# RUNBOOK — provisioning the OS / Docker-data storage split
|
||||||
|
|
||||||
|
How the controller guest's **OS rootfs** and **Docker data** are carved onto separate `local-lvm`
|
||||||
|
volumes, and how to size them per node/customer. Grounded by the storage-split spike
|
||||||
|
(`REPORT-storage-split-spike.md`). Defaults target a ~512 GB SSD: **32 GB OS rootfs + 256 GB
|
||||||
|
Docker-data**, rest of the thin pool free for other guests.
|
||||||
|
|
||||||
|
## The design (locked)
|
||||||
|
|
||||||
|
- **Split for resilience, not backup size.** An isolated OS rootfs stays bootable + agent-recoverable
|
||||||
|
if the Docker volume fills. Both volumes ride into PBS regardless (so the backup size is unchanged) —
|
||||||
|
the win is independent sizing/growth and a smaller, faster-to-restore OS rootfs.
|
||||||
|
- **One Docker daemon, one data-root, one volume** at `/var/lib/docker` for ALL images + overlay
|
||||||
|
(controller, traefik, cloudflared, filebrowser, and customer apps). Infra is NOT relocated to the OS
|
||||||
|
partition — infra/user separation under one daemon isn't cleanly possible and isn't needed. Infra is
|
||||||
|
protected by the controller's **prevention layer** (reserved buffer + deploy gate + disk alerts +
|
||||||
|
baked log rotation), not by placement.
|
||||||
|
- **`backup=1` on the Docker-data volume is non-negotiable.** Extra LXC mountpoints default to
|
||||||
|
`backup=0` = EXCLUDED from vzdump/PBS (spike B3). Without it, named-volume DBs fall out of the
|
||||||
|
snapshot AND the volume (with baked images) is excluded from the golden archive.
|
||||||
|
|
||||||
|
## Golden build (per controller-image baseline)
|
||||||
|
|
||||||
|
`configs/build-golden.sh` builds the template guest with the split baked in:
|
||||||
|
|
||||||
|
```
|
||||||
|
# defaults shown; override via env
|
||||||
|
OS_SIZE_GB=32 GOLDEN_DOCKER_GB=16 \
|
||||||
|
./build-golden.sh <VMID> <TEMPLATE> <ROOTFS_STORAGE> <ARCHIVE_STORAGE> <BRIDGE> <CONTROLLER_IMAGE>
|
||||||
|
```
|
||||||
|
|
||||||
|
- `--rootfs ${ROOTFS_STORAGE}:${OS_SIZE_GB}` + `--mp0 ${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1`.
|
||||||
|
- Keep `GOLDEN_DOCKER_GB` small (just the baked images + headroom) — provision grows it.
|
||||||
|
- The script bakes `/etc/docker/daemon.json` log rotation (`max-size 10m`, `max-file 3`), verifies
|
||||||
|
`/var/lib/docker` is a separate mount, and **aborts if vzdump excludes mp0** (the B3 trap).
|
||||||
|
|
||||||
|
## Provision (per customer guest)
|
||||||
|
|
||||||
|
The bring-up restores the golden (carrying rootfs + the small mp0 Docker-data volume + baked images),
|
||||||
|
then sizes per customer. The Docker-data volume is **grown** (not re-attached) so the baked images are
|
||||||
|
never shadowed:
|
||||||
|
|
||||||
|
```
|
||||||
|
felhom-agent --selftest=provision \
|
||||||
|
-archive <golden-volid> -vmid <VMID> \
|
||||||
|
-customer-id <id> -hub-password <retrieval-passphrase> \
|
||||||
|
-rootfs-grow <N> # GiB to add to the OS rootfs beyond the golden's 32 (0 = keep 32)
|
||||||
|
-datavol-grow <N> # GiB to add to the Docker-data volume beyond the golden's 16 (e.g. 240 → 256 total)
|
||||||
|
-datavol-mount mp0 # the golden's Docker-data slot (default mp0)
|
||||||
|
```
|
||||||
|
|
||||||
|
In code: `reconcile.BringUpSpec.{RootfsGrowGB, DataVolGrowGB, DataVolMount}` — both grows are online
|
||||||
|
(`pct resize`, spike B4), each its own task. `GuestMount.Backup` (→ `,backup=1`) is set on any
|
||||||
|
data-bearing additive mount (`bringup.go` `buildConfigParams`). The bootstrap mount is `mp9`; USB/data
|
||||||
|
drive enroll picks the lowest free slot (`freeMountSlot`), so it never collides with the docker mp0.
|
||||||
|
|
||||||
|
**Per-customer sizing source:** flags today; the **slice-10 hub storage manifest** is the intended
|
||||||
|
home (marked at `bringup.go` `GuestMount` / the `slice 10 wires the hub storage manifest` comment) —
|
||||||
|
the controller already pulls customer-scoped config from the hub, so OS/data sizes belong there.
|
||||||
|
|
||||||
|
### Default sizing guidance
|
||||||
|
- ~512 GB SSD: 32 GB rootfs + 256 GB Docker-data (`-rootfs-grow 0 -datavol-grow 240`), leaving the rest
|
||||||
|
of the thin pool free for other guests.
|
||||||
|
- Other SSDs: size the data volume to "most of the free thin pool minus a reserve for other guests."
|
||||||
|
`local-lvm` is thin-provisioned, so an over-allocation only consumes space as it's written.
|
||||||
|
|
||||||
|
## Fresh PVE host install — thin-pool carving (host-level, do this at install)
|
||||||
|
|
||||||
|
This sizes the **host's** `local` root vs the `local-lvm` thin pool (the pool the guests' volumes are
|
||||||
|
carved from). It is NOT the guest split — do it on the Proxmox ISO installer's **Advanced LVM options**:
|
||||||
|
|
||||||
|
- `hdsize` — total disk PVE uses (leave headroom on multi-disk nodes).
|
||||||
|
- `maxroot` — cap on the host `/` (`pve/root`). Keep small (e.g. 20–40 GB) — the host OS is thin.
|
||||||
|
- `maxvz` — cap on the `data` thin pool (`local-lvm`). Set to 0 / leave default to let the pool take the
|
||||||
|
rest, OR cap it to reserve unallocated VG space.
|
||||||
|
- `minfree` — reserved free space in the VG (thin-pool safety).
|
||||||
|
- `swapsize` — host swap.
|
||||||
|
|
||||||
|
**Never repartition a live host root.** Per-guest carving (rootfs + Docker-data) is done at provision
|
||||||
|
time via `pct` against the existing thin pool — no host repartition needed (the demo's pool already has
|
||||||
|
~350 GB free).
|
||||||
|
|
||||||
|
## In-place migration of an EXISTING guest (NOT used by this slice)
|
||||||
|
|
||||||
|
This slice rebuilds clean (fresh re-provision), so the move-then-verify in-place migration is **not
|
||||||
|
used here**. It remains documented in `REPORT-storage-split-spike.md` §B2 for a future in-place node
|
||||||
|
upgrade: stop apps → stop docker → `rsync -aHAX --numeric-ids /var/lib/docker/ <new-vol>/` (check
|
||||||
|
rsync's OWN exit code) → switch data-root → verify volumes+DBs+boot → only then reclaim. `backup=1` on
|
||||||
|
the new mount and a CT restart to activate it are required (spike B3/B4).
|
||||||
@@ -43,7 +43,7 @@ import (
|
|||||||
|
|
||||||
// version is the agent version. Overridable at build time with
|
// version is the agent version. Overridable at build time with
|
||||||
// -ldflags "-X main.version=<v>"; defaults to the in-repo CHANGELOG version.
|
// -ldflags "-X main.version=<v>"; defaults to the in-repo CHANGELOG version.
|
||||||
var version = "0.28.0"
|
var version = "0.29.0"
|
||||||
|
|
||||||
func main() {
|
func main() {
|
||||||
var (
|
var (
|
||||||
@@ -55,6 +55,9 @@ func main() {
|
|||||||
mode string
|
mode string
|
||||||
hostname string
|
hostname string
|
||||||
keep bool
|
keep bool
|
||||||
|
rootfsGrow int
|
||||||
|
dataVolGrow int
|
||||||
|
dataVolMount string
|
||||||
pbsStorage string
|
pbsStorage string
|
||||||
paperkey bool
|
paperkey bool
|
||||||
offline bool
|
offline bool
|
||||||
@@ -79,6 +82,9 @@ func main() {
|
|||||||
flag.StringVar(&mode, "mode", "provision", "for --selftest=bring-up: `provision` (golden, fresh identity) | `dr` (customer backup, preserve continuity)")
|
flag.StringVar(&mode, "mode", "provision", "for --selftest=bring-up: `provision` (golden, fresh identity) | `dr` (customer backup, preserve continuity)")
|
||||||
flag.StringVar(&hostname, "hostname", "", "for --selftest=bring-up provision: the hostname to set on the new guest")
|
flag.StringVar(&hostname, "hostname", "", "for --selftest=bring-up provision: the hostname to set on the new guest")
|
||||||
flag.BoolVar(&keep, "keep", false, "for --selftest=bring-up: KEEP the guest instead of tearing it down at the end")
|
flag.BoolVar(&keep, "keep", false, "for --selftest=bring-up: KEEP the guest instead of tearing it down at the end")
|
||||||
|
flag.IntVar(&rootfsGrow, "rootfs-grow", 0, "for --selftest=bring-up|provision: grow the OS rootfs by this many GiB after restore (0 = keep golden size)")
|
||||||
|
flag.IntVar(&dataVolGrow, "datavol-grow", 0, "for --selftest=bring-up|provision: grow the golden's Docker-data volume (mp0) by this many GiB (0 = keep golden size)")
|
||||||
|
flag.StringVar(&dataVolMount, "datavol-mount", "", "for --selftest=bring-up|provision: the mpN slot of the Docker-data volume to grow (default mp0)")
|
||||||
flag.StringVar(&pbsStorage, "storage", "", "for --selftest=escrow-create: the pbs storage whose key to escrow (default: escrow.pbs_storage_id)")
|
flag.StringVar(&pbsStorage, "storage", "", "for --selftest=escrow-create: the pbs storage whose key to escrow (default: escrow.pbs_storage_id)")
|
||||||
flag.BoolVar(&paperkey, "paperkey", false, "for --selftest=escrow-create: ALSO emit the raw-key paperkey (opt-in (a); single-factor, unrevocable)")
|
flag.BoolVar(&paperkey, "paperkey", false, "for --selftest=escrow-create: ALSO emit the raw-key paperkey (opt-in (a); single-factor, unrevocable)")
|
||||||
flag.BoolVar(&offline, "offline", false, "for --selftest=escrow-create: ALSO emit the R-wrapped offline copy to print (opt-in (b))")
|
flag.BoolVar(&offline, "offline", false, "for --selftest=escrow-create: ALSO emit the R-wrapped offline copy to print (opt-in (b))")
|
||||||
@@ -133,11 +139,13 @@ func main() {
|
|||||||
case "lanresolver":
|
case "lanresolver":
|
||||||
os.Exit(runSelftestLANResolver(context.Background(), cfg, logger, vmid))
|
os.Exit(runSelftestLANResolver(context.Background(), cfg, logger, vmid))
|
||||||
case "bring-up":
|
case "bring-up":
|
||||||
os.Exit(runSelftestBringUp(context.Background(), cfg, logger, mode, archive, vmid, hostname, keep))
|
os.Exit(runSelftestBringUp(context.Background(), cfg, logger, mode, archive, vmid, hostname, keep,
|
||||||
|
bringUpSizing{RootfsGrowGB: rootfsGrow, DataVolGrowGB: dataVolGrow, DataVolMount: dataVolMount}))
|
||||||
case "provision":
|
case "provision":
|
||||||
os.Exit(runSelftestProvision(context.Background(), cfg, logger, provisionArgs{
|
os.Exit(runSelftestProvision(context.Background(), cfg, logger, provisionArgs{
|
||||||
archive: archive, vmid: vmid, hostname: hostname,
|
archive: archive, vmid: vmid, hostname: hostname,
|
||||||
customerID: custID, hubPassword: hubPassword,
|
customerID: custID, hubPassword: hubPassword,
|
||||||
|
sizing: bringUpSizing{RootfsGrowGB: rootfsGrow, DataVolGrowGB: dataVolGrow, DataVolMount: dataVolMount},
|
||||||
}))
|
}))
|
||||||
case "escrow-create":
|
case "escrow-create":
|
||||||
os.Exit(runSelftestEscrowCreate(context.Background(), cfg, logger, pbsStorage, paperkey, offline, upload, idBundlePath, directivePath))
|
os.Exit(runSelftestEscrowCreate(context.Background(), cfg, logger, pbsStorage, paperkey, offline, upload, idBundlePath, directivePath))
|
||||||
@@ -950,7 +958,16 @@ func runSelftestRestoreTest(ctx context.Context, cfg config.Config, logger *slog
|
|||||||
// (a selftest must not leave a guest running) unless -keep. -mode picks provision (golden, fresh
|
// (a selftest must not leave a guest running) unless -keep. -mode picks provision (golden, fresh
|
||||||
// identity) or dr (customer backup, preserve continuity). It first Recovers, so a leaked guest
|
// identity) or dr (customer backup, preserve continuity). It first Recovers, so a leaked guest
|
||||||
// from a prior crashed bring-up is reaped before this run.
|
// from a prior crashed bring-up is reaped before this run.
|
||||||
func runSelftestBringUp(ctx context.Context, cfg config.Config, logger *slog.Logger, mode, archive string, vmid int, hostname string, keep bool) int {
|
// bringUpSizing carries the OS-rootfs / Docker-data sizing knobs from the CLI into a bring-up.
|
||||||
|
// Defaults (0/"") keep the golden's baked sizes; the provisioning spec sources these per-customer
|
||||||
|
// (flags now; the slice-10 hub storage manifest later — see bringup.go GuestMount comment).
|
||||||
|
type bringUpSizing struct {
|
||||||
|
RootfsGrowGB int
|
||||||
|
DataVolGrowGB int
|
||||||
|
DataVolMount string
|
||||||
|
}
|
||||||
|
|
||||||
|
func runSelftestBringUp(ctx context.Context, cfg config.Config, logger *slog.Logger, mode, archive string, vmid int, hostname string, keep bool, sizing bringUpSizing) int {
|
||||||
if err := cfg.Validate(); err != nil {
|
if err := cfg.Validate(); err != nil {
|
||||||
fmt.Fprintln(os.Stderr, "selftest: proxmox not configured:", err)
|
fmt.Fprintln(os.Stderr, "selftest: proxmox not configured:", err)
|
||||||
return 1
|
return 1
|
||||||
@@ -1006,6 +1023,7 @@ func runSelftestBringUp(ctx context.Context, cfg config.Config, logger *slog.Log
|
|||||||
spec := reconcile.BringUpSpec{
|
spec := reconcile.BringUpSpec{
|
||||||
Mode: bmode, Archive: archive, VMID: vmid, RestoreStorage: cfg.Backup.RestoreStorage,
|
Mode: bmode, Archive: archive, VMID: vmid, RestoreStorage: cfg.Backup.RestoreStorage,
|
||||||
Hostname: hostname, KeepMAC: bmode == reconcile.ModeDRGuestLoss,
|
Hostname: hostname, KeepMAC: bmode == reconcile.ModeDRGuestLoss,
|
||||||
|
RootfsGrowGB: sizing.RootfsGrowGB, DataVolGrowGB: sizing.DataVolGrowGB, DataVolMount: sizing.DataVolMount,
|
||||||
}
|
}
|
||||||
fmt.Printf(" bringing up %s → vmid %d on %s …\n", archive, vmid, cfg.Backup.RestoreStorage)
|
fmt.Printf(" bringing up %s → vmid %d on %s …\n", archive, vmid, cfg.Backup.RestoreStorage)
|
||||||
res := engine.RunBringUp(ctx, spec)
|
res := engine.RunBringUp(ctx, spec)
|
||||||
@@ -1049,6 +1067,7 @@ type provisionArgs struct {
|
|||||||
hostname string
|
hostname string
|
||||||
customerID string // baked into bootstrap (the hub config-pull target)
|
customerID string // baked into bootstrap (the hub config-pull target)
|
||||||
hubPassword string // the customer's hub retrieval passphrase (SECRET) — baked into bootstrap
|
hubPassword string // the customer's hub retrieval passphrase (SECRET) — baked into bootstrap
|
||||||
|
sizing bringUpSizing // OS-rootfs / Docker-data sizing for the bring-up half
|
||||||
}
|
}
|
||||||
|
|
||||||
// sanitizeHostname makes s a DNS-safe LXC hostname (RFC 1123 label-ish): lowercase, any run of
|
// sanitizeHostname makes s a DNS-safe LXC hostname (RFC 1123 label-ish): lowercase, any run of
|
||||||
@@ -1156,6 +1175,7 @@ func runSelftestProvision(ctx context.Context, cfg config.Config, logger *slog.L
|
|||||||
res := engine.RunBringUp(ctx, reconcile.BringUpSpec{
|
res := engine.RunBringUp(ctx, reconcile.BringUpSpec{
|
||||||
Mode: reconcile.ModeProvision, Archive: a.archive, VMID: a.vmid,
|
Mode: reconcile.ModeProvision, Archive: a.archive, VMID: a.vmid,
|
||||||
RestoreStorage: cfg.Backup.RestoreStorage, Hostname: hostname,
|
RestoreStorage: cfg.Backup.RestoreStorage, Hostname: hostname,
|
||||||
|
RootfsGrowGB: a.sizing.RootfsGrowGB, DataVolGrowGB: a.sizing.DataVolGrowGB, DataVolMount: a.sizing.DataVolMount,
|
||||||
})
|
})
|
||||||
if res.Err != nil || !res.Pass {
|
if res.Err != nil || !res.Pass {
|
||||||
fmt.Fprintf(os.Stderr, " [FAIL] front-half bring-up (vmid %d): %v\n", a.vmid, res.Err)
|
fmt.Fprintf(os.Stderr, " [FAIL] front-half bring-up (vmid %d): %v\n", a.vmid, res.Err)
|
||||||
|
|||||||
+42
-6
@@ -23,6 +23,16 @@
|
|||||||
# Usage: build-golden.sh [VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] [CONTROLLER_IMAGE]
|
# Usage: build-golden.sh [VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] [CONTROLLER_IMAGE]
|
||||||
# Build-time registry login for the controller pull (used ONCE inside the build guest, then logged
|
# Build-time registry login for the controller pull (used ONCE inside the build guest, then logged
|
||||||
# out — never baked): set REGISTRY_USER + REGISTRY_TOKEN in the environment.
|
# out — never baked): set REGISTRY_USER + REGISTRY_TOKEN in the environment.
|
||||||
|
#
|
||||||
|
# OS / Docker-data SPLIT (storage-split slice): the golden is built with a SMALL OS rootfs and a
|
||||||
|
# SEPARATE Docker-data volume mounted at /var/lib/docker (mp0, backup=1). The baked controller +
|
||||||
|
# infra images land on that volume and travel INSIDE the golden archive — so provisioned guests boot
|
||||||
|
# from baked images with no registry pull. The split is for RESILIENCE: an isolated OS rootfs stays
|
||||||
|
# bootable + agent-recoverable if the Docker volume fills (the controller's prevention layer keeps it
|
||||||
|
# from filling). Sizes are env-overridable (OS_SIZE_GB / GOLDEN_DOCKER_GB); provision GROWS the data
|
||||||
|
# volume to the per-customer target (bringup.go DataVolGrowGB). backup=1 is MANDATORY on the data mp:
|
||||||
|
# without it vzdump EXCLUDES the volume (extra LXC mountpoints default backup=0 — storage-split B3),
|
||||||
|
# so the archive would carry NO images and provisioned guests would boot imageless.
|
||||||
set -euo pipefail
|
set -euo pipefail
|
||||||
|
|
||||||
VMID="${1:-9100}"
|
VMID="${1:-9100}"
|
||||||
@@ -32,12 +42,17 @@ ARCHIVE_STORAGE="${4:-local}"
|
|||||||
BRIDGE="${5:-vmbr0}"
|
BRIDGE="${5:-vmbr0}"
|
||||||
CONTROLLER_IMAGE="${6:-gitea.dooplex.hu/admin/felhom-controller:0.43.0}"
|
CONTROLLER_IMAGE="${6:-gitea.dooplex.hu/admin/felhom-controller:0.43.0}"
|
||||||
REGISTRY_HOST="${CONTROLLER_IMAGE%%/*}"
|
REGISTRY_HOST="${CONTROLLER_IMAGE%%/*}"
|
||||||
|
# OS rootfs size (GiB) and the golden's Docker-data volume size (GiB). Keep GOLDEN_DOCKER_GB just
|
||||||
|
# large enough for the baked images + headroom; provision grows it to the per-customer target.
|
||||||
|
OS_SIZE_GB="${OS_SIZE_GB:-32}"
|
||||||
|
GOLDEN_DOCKER_GB="${GOLDEN_DOCKER_GB:-16}"
|
||||||
|
|
||||||
echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged) …"
|
echo "[golden] creating build LXC $VMID (nesting=1,keyctl=1, unprivileged; rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G @ /var/lib/docker, backup=1) …"
|
||||||
pct create "$VMID" "$TEMPLATE" \
|
pct create "$VMID" "$TEMPLATE" \
|
||||||
--hostname felhom-golden --unprivileged 1 \
|
--hostname felhom-golden --unprivileged 1 \
|
||||||
--features nesting=1,keyctl=1 \
|
--features nesting=1,keyctl=1 \
|
||||||
--rootfs "${ROOTFS_STORAGE}:8" --cores 2 --memory 2048 \
|
--rootfs "${ROOTFS_STORAGE}:${OS_SIZE_GB}" --cores 2 --memory 2048 \
|
||||||
|
--mp0 "${ROOTFS_STORAGE}:${GOLDEN_DOCKER_GB},mp=/var/lib/docker,backup=1" \
|
||||||
--net0 "name=eth0,bridge=${BRIDGE},ip=dhcp" --onboot 0
|
--net0 "name=eth0,bridge=${BRIDGE},ip=dhcp" --onboot 0
|
||||||
|
|
||||||
echo "[golden] starting + installing Docker (official repo, trixie channel) …"
|
echo "[golden] starting + installing Docker (official repo, trixie channel) …"
|
||||||
@@ -59,8 +74,20 @@ pct exec "$VMID" -- bash -c '
|
|||||||
apt-get update -qq
|
apt-get update -qq
|
||||||
apt-get install -y -qq docker-ce docker-ce-cli containerd.io >/dev/null
|
apt-get install -y -qq docker-ce docker-ce-cli containerd.io >/dev/null
|
||||||
'
|
'
|
||||||
echo "[golden] verifying Docker works in the build guest …"
|
echo "[golden] baking Docker log rotation into daemon.json (prevention layer: kills unbounded container logs for every guest) …"
|
||||||
pct exec "$VMID" -- bash -c 'systemctl start docker; sleep 2; docker run --rm hello-world >/dev/null && echo " docker OK ($(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"))"'
|
# /var/lib/docker is the mp0 mount (mounted empty before docker installs), so data-root needs NO
|
||||||
|
# override — the existing image pulls + this config land on the volume automatically. Only the
|
||||||
|
# json-file log caps are set (the most common runaway). Every container inherits these defaults.
|
||||||
|
pct exec "$VMID" -- bash -c 'mkdir -p /etc/docker; cat > /etc/docker/daemon.json <<JSON
|
||||||
|
{
|
||||||
|
"log-driver": "json-file",
|
||||||
|
"log-opts": { "max-size": "10m", "max-file": "3" }
|
||||||
|
}
|
||||||
|
JSON'
|
||||||
|
echo "[golden] verifying Docker works in the build guest (storage driver should be overlayfs on the ext4 data volume) …"
|
||||||
|
pct exec "$VMID" -- bash -c 'systemctl start docker; sleep 2; docker run --rm hello-world >/dev/null && echo " docker OK ($(docker info 2>/dev/null | sed -n "s/.*Storage Driver: //p"); data-root $(docker info 2>/dev/null | sed -n "s/.*Docker Root Dir: //p"))"'
|
||||||
|
# Confirm /var/lib/docker is genuinely the dedicated volume, not the rootfs (catch a silent mp miss).
|
||||||
|
pct exec "$VMID" -- bash -c 'findmnt -no SOURCE,FSTYPE /var/lib/docker | grep -q . && echo " /var/lib/docker is a separate mount: $(findmnt -no SOURCE,FSTYPE /var/lib/docker)" || { echo "[golden] FATAL: /var/lib/docker is NOT a separate mount — the mp0 split did not take"; exit 1; }'
|
||||||
|
|
||||||
echo "[golden] baking the in-guest controller image $CONTROLLER_IMAGE (no registry cred at deploy) …"
|
echo "[golden] baking the in-guest controller image $CONTROLLER_IMAGE (no registry cred at deploy) …"
|
||||||
# docker login is used ONCE here on the trusted build host, then logged out before archiving so
|
# docker login is used ONCE here on the trusted build host, then logged out before archiving so
|
||||||
@@ -197,8 +224,17 @@ pct exec "$VMID" -- bash -c '
|
|||||||
|
|
||||||
echo "[golden] stop + archive …"
|
echo "[golden] stop + archive …"
|
||||||
pct stop "$VMID"
|
pct stop "$VMID"
|
||||||
vzdump "$VMID" --storage "$ARCHIVE_STORAGE" --mode stop --compress zstd
|
# --mode stop with mp0 backup=1 → the Docker-data volume (with baked images) is INCLUDED. The log
|
||||||
|
# below MUST show "including mount point mp0" — if it shows "excluding … (disabled)" the backup flag
|
||||||
|
# was lost and the archive carries no images (storage-split B3 trap).
|
||||||
|
vzdump "$VMID" --storage "$ARCHIVE_STORAGE" --mode stop --compress zstd 2>&1 | tee /tmp/golden-vzdump.log | grep -iE "including mount point|excluding|archive file size|Finished Backup" || true
|
||||||
|
if grep -q "excluding volume mount point mp0" /tmp/golden-vzdump.log; then
|
||||||
|
echo "[golden] FATAL: mp0 (/var/lib/docker) was EXCLUDED from the archive — backup=1 was lost; the golden would carry no images. Aborting."
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
grep -q "including mount point mp0" /tmp/golden-vzdump.log \
|
||||||
|
|| echo "[golden] WARN: could not confirm mp0 inclusion in the vzdump log — verify manually before using this archive."
|
||||||
|
|
||||||
VOLID=$(pvesm list "$ARCHIVE_STORAGE" --content backup 2>/dev/null | awk -v v="$VMID" '$1 ~ ("vzdump-lxc-" v "-") {print $1}' | sort | tail -1)
|
VOLID=$(pvesm list "$ARCHIVE_STORAGE" --content backup 2>/dev/null | awk -v v="$VMID" '$1 ~ ("vzdump-lxc-" v "-") {print $1}' | sort | tail -1)
|
||||||
echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>}"
|
echo "[golden] DONE. golden archive volid: ${VOLID:-<check ${ARCHIVE_STORAGE} dump dir>} (rootfs ${OS_SIZE_GB}G + Docker-data ${GOLDEN_DOCKER_GB}G, both in the archive)"
|
||||||
echo "[golden] (the build guest $VMID is stopped; destroy it with: pct destroy $VMID --purge)"
|
echo "[golden] (the build guest $VMID is stopped; destroy it with: pct destroy $VMID --purge)"
|
||||||
|
|||||||
@@ -40,6 +40,9 @@ const (
|
|||||||
|
|
||||||
const bringUpKind = "bring_up"
|
const bringUpKind = "bring_up"
|
||||||
|
|
||||||
|
// DefaultDataVolMount is the mpN slot the golden bakes the Docker-data volume (/var/lib/docker) at.
|
||||||
|
const DefaultDataVolMount = "mp0"
|
||||||
|
|
||||||
// configLockMaxAttempts bounds the F4 config-lock retry. configLockBackoff is a package var so
|
// configLockMaxAttempts bounds the F4 config-lock retry. configLockBackoff is a package var so
|
||||||
// tests can shrink it (the production value gives PVE time to release its async config lock).
|
// tests can shrink it (the production value gives PVE time to release its async config lock).
|
||||||
const configLockMaxAttempts = 5
|
const configLockMaxAttempts = 5
|
||||||
@@ -52,6 +55,12 @@ type GuestMount struct {
|
|||||||
Storage string // PVE storage id (e.g. "local-lvm")
|
Storage string // PVE storage id (e.g. "local-lvm")
|
||||||
SizeGB int // new-volume size in GiB
|
SizeGB int // new-volume size in GiB
|
||||||
MountPoint string // in-guest path (e.g. "/mnt/data")
|
MountPoint string // in-guest path (e.g. "/mnt/data")
|
||||||
|
// Backup includes this mountpoint in vzdump/PBS. MANDATORY for any data-bearing mount (DB
|
||||||
|
// volumes), because extra LXC mountpoints default to backup=0 = EXCLUDED from the snapshot
|
||||||
|
// (storage-split finding B3). The Docker-data volume normally rides in from the golden archive
|
||||||
|
// (already backup=1) and is grown via DataVolGrowGB rather than attached here, but any data
|
||||||
|
// mount attached through spec.Mounts MUST set this or its contents silently fall out of PBS.
|
||||||
|
Backup bool
|
||||||
}
|
}
|
||||||
|
|
||||||
// BringUpSpec is the input to one bring-up. The caller resolves it (the selftest, or slice-10
|
// BringUpSpec is the input to one bring-up. The caller resolves it (the selftest, or slice-10
|
||||||
@@ -65,6 +74,13 @@ type BringUpSpec struct {
|
|||||||
Cores int // 0 = leave as restored
|
Cores int // 0 = leave as restored
|
||||||
MemoryMB int // 0 = leave as restored
|
MemoryMB int // 0 = leave as restored
|
||||||
RootfsGrowGB int // optional grow-only rootfs resize (0 = skip)
|
RootfsGrowGB int // optional grow-only rootfs resize (0 = skip)
|
||||||
|
// DataVolGrowGB grows the golden-carried Docker-data volume (DataVolMount, default mp0) to the
|
||||||
|
// per-customer target. The golden ships a small data volume with the baked images; provision
|
||||||
|
// grows it online (grow-only, storage-split B4) rather than attaching a fresh empty volume that
|
||||||
|
// would shadow the baked images. 0 = skip (keep the golden's size).
|
||||||
|
DataVolGrowGB int
|
||||||
|
// DataVolMount is the mpN slot of the golden's Docker-data volume to grow; "" → DefaultDataVolMount ("mp0").
|
||||||
|
DataVolMount string
|
||||||
Mounts []GuestMount // additive mpN mounts (slice 7 may pass empty/test)
|
Mounts []GuestMount // additive mpN mounts (slice 7 may pass empty/test)
|
||||||
KeepMAC bool // DR knob: keep the archived MAC (true) unless a source may be live
|
KeepMAC bool // DR knob: keep the archived MAC (true) unless a source may be live
|
||||||
BootTimeout time.Duration // 0 → DefaultBootTimeout; bounds the link-up liveness wait
|
BootTimeout time.Duration // 0 → DefaultBootTimeout; bounds the link-up liveness wait
|
||||||
@@ -213,6 +229,26 @@ func (e *Engine) runBringUp(ctx context.Context, spec BringUpSpec, res *BringUpR
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// 4b. Grow the golden-carried Docker-data volume (mp0) to the per-customer target. Grow-only,
|
||||||
|
// online (storage-split B4); its OWN call like the rootfs resize. The volume + baked images
|
||||||
|
// came in with the restore, so we grow it rather than attach a fresh one that would shadow
|
||||||
|
// the baked images.
|
||||||
|
if spec.DataVolGrowGB > 0 {
|
||||||
|
mount := spec.DataVolMount
|
||||||
|
if mount == "" {
|
||||||
|
mount = DefaultDataVolMount
|
||||||
|
}
|
||||||
|
dupid, err := e.api.ResizeLXC(ctx, spec.VMID, mount, fmt.Sprintf("+%dG", spec.DataVolGrowGB))
|
||||||
|
if err != nil {
|
||||||
|
res.Err = fmt.Errorf("reconcile: bring-up data-volume resize (%s): %w", mount, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
if _, err := e.waitTask(ctx, dupid, proxmox.WaitOptions{}); err != nil {
|
||||||
|
res.Err = fmt.Errorf("reconcile: bring-up data-volume resize task (%s): %w", mount, err)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Capture the post-reset MAC for the result (fresh for provision; archived for DR keep).
|
// Capture the post-reset MAC for the result (fresh for provision; archived for DR keep).
|
||||||
if cfg2, err := e.api.GuestConfig(ctx, spec.VMID); err == nil {
|
if cfg2, err := e.api.GuestConfig(ctx, spec.VMID); err == nil {
|
||||||
res.AssignedMAC = net0MAC(cfg2)
|
res.AssignedMAC = net0MAC(cfg2)
|
||||||
@@ -310,7 +346,13 @@ func buildBringUpConfig(spec BringUpSpec, cfg proxmox.GuestConfig) map[string]st
|
|||||||
params["memory"] = strconv.Itoa(spec.MemoryMB)
|
params["memory"] = strconv.Itoa(spec.MemoryMB)
|
||||||
}
|
}
|
||||||
for i, m := range spec.Mounts {
|
for i, m := range spec.Mounts {
|
||||||
params[fmt.Sprintf("mp%d", i)] = fmt.Sprintf("%s:%d,mp=%s", m.Storage, m.SizeGB, m.MountPoint)
|
// backup=1 for data-bearing mounts: extra LXC mountpoints default to backup=0 = EXCLUDED
|
||||||
|
// from vzdump/PBS (storage-split B3), which would silently drop their DBs from the snapshot.
|
||||||
|
spec := fmt.Sprintf("%s:%d,mp=%s", m.Storage, m.SizeGB, m.MountPoint)
|
||||||
|
if m.Backup {
|
||||||
|
spec += ",backup=1"
|
||||||
|
}
|
||||||
|
params[fmt.Sprintf("mp%d", i)] = spec
|
||||||
}
|
}
|
||||||
return params
|
return params
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -74,6 +74,58 @@ func TestRunBringUp_ProvisionHappyPath(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// A data-bearing additive mount must carry backup=1 (so its DBs stay in PBS — storage-split B3);
|
||||||
|
// a non-backup mount must NOT. Pure-function check on buildBringUpConfig.
|
||||||
|
func TestBuildBringUpConfig_BackupFlagOnDataMount(t *testing.T) {
|
||||||
|
params := buildBringUpConfig(BringUpSpec{
|
||||||
|
Mode: ModeProvision,
|
||||||
|
Mounts: []GuestMount{
|
||||||
|
{Storage: "local-lvm", SizeGB: 2, MountPoint: "/mnt/data", Backup: true},
|
||||||
|
{Storage: "local-lvm", SizeGB: 1, MountPoint: "/mnt/scratch"}, // no backup
|
||||||
|
},
|
||||||
|
}, scratchCfg())
|
||||||
|
if params["mp0"] != "local-lvm:2,mp=/mnt/data,backup=1" {
|
||||||
|
t.Errorf("data mount must carry backup=1: mp0=%q", params["mp0"])
|
||||||
|
}
|
||||||
|
if params["mp1"] != "local-lvm:1,mp=/mnt/scratch" {
|
||||||
|
t.Errorf("non-backup mount must NOT carry backup=1: mp1=%q", params["mp1"])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// The golden-carried Docker-data volume is grown via a SEPARATE resize on its mpN slot (B4),
|
||||||
|
// alongside (but distinct from) the rootfs grow.
|
||||||
|
func TestRunBringUp_StorageSplit_DataVolGrow(t *testing.T) {
|
||||||
|
const vmid = 8050
|
||||||
|
api := &fakeAPI{cfg: map[int]proxmox.GuestConfig{vmid: scratchCfg()}}
|
||||||
|
e, _, q := newEngine(t, api, EmptyProvider{})
|
||||||
|
defer q.Close()
|
||||||
|
|
||||||
|
res := e.RunBringUp(context.Background(), BringUpSpec{
|
||||||
|
Mode: ModeProvision, Archive: "local:backup/golden.tar.zst", VMID: vmid,
|
||||||
|
RestoreStorage: "local-lvm", Hostname: "felhom-prov-8050",
|
||||||
|
RootfsGrowGB: 8, DataVolGrowGB: 240, // grows mp0 (DefaultDataVolMount)
|
||||||
|
})
|
||||||
|
if res.Err != nil || !res.Pass {
|
||||||
|
t.Fatalf("provision must pass, got %+v", res)
|
||||||
|
}
|
||||||
|
// TWO resizes: rootfs +8G and the Docker-data volume mp0 +240G.
|
||||||
|
if len(api.resizes) != 2 {
|
||||||
|
t.Fatalf("expected rootfs + data-volume resizes, got %+v", api.resizes)
|
||||||
|
}
|
||||||
|
var sawRootfs, sawData bool
|
||||||
|
for _, r := range api.resizes {
|
||||||
|
if r.disk == "rootfs" && r.size == "+8G" {
|
||||||
|
sawRootfs = true
|
||||||
|
}
|
||||||
|
if r.disk == "mp0" && r.size == "+240G" {
|
||||||
|
sawData = true
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if !sawRootfs || !sawData {
|
||||||
|
t.Errorf("want rootfs +8G AND mp0 +240G, got %+v", api.resizes)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestRunBringUp_CompensatingRollback(t *testing.T) {
|
func TestRunBringUp_CompensatingRollback(t *testing.T) {
|
||||||
const vmid = 8000
|
const vmid = 8000
|
||||||
lockBackoffFast(t)
|
lockBackoffFast(t)
|
||||||
|
|||||||
Reference in New Issue
Block a user