build-golden.sh 2.1.0 -> 3.0.0: a layout change is a major. The golden ships
ONE data volume at a NEUTRAL path (/var/lib/felhom); both /var/lib/docker
and /mnt/sys_drive are binds of subdirectories of it. mp1 is gone.
The variant was chosen on measurement. Three candidates were built and
rebooted (SPIKE-r165-phase0-2026-08-03.md); all three boot, reboot 3/3, give
ONE df figure and keep a container's statfs(/) on the merged volume — the
ordering worry that motivated the probe did not materialise. They differ
only in which guarantee they break: volume-at-docker puts customer backups
inside Docker's data-root; volume-at-sys_drive puts Docker's ENTIRE
data-root under /mnt, which the controller container mounts wholesale
(measured: it then sees /mnt/sys_drive/docker). V-c breaks neither.
The four assertions were RETARGETED, never deleted, and each was RUN against
a deliberately wrong shape — a real split guest and a real archive of it:
8 checks, 8 passed. A new 2b asserts both paths are ONE filesystem, which
catches the S2 shape the spike ranked worse than the split. Assertion 5
replaces the old "was mp1 excluded?" guard, whose pattern could no longer
match — a guard that cannot match has silently stopped guarding.
Provisioning: one volume, one grow. SysDataGrowGB is FOLDED IN rather than
dropped, because a census established that felhom-host-install.sh passes
-sysdata-grow and the two do not upgrade in the same instant; dropping it
would silently shrink every appliance by 42 of 250 GiB. The flags stay
accepted for the same reason. The existing test was retargeted to pin the
fold, and it caught the change before I did.
The live scratch DR exposed the second half of PVE's all-or-nothing
explicit-params restore: mountpoints NOT named in the params are silently
DROPPED - the DR guest came up without its mp0/mp1 data volumes (boot
passed; the customer's world did not ride along). drRestoreOverrides now
derives the COMPLETE param set from the archive's extracted config:
explicit rootfs, every storage-backed mpN passed through (size + in-guest
path + backup flag preserved so vzrestore extracts its content), the two
structural binds replaced by 4d-swapped throwaways; unknown bind mpN or
unparseable size refuses loudly. Snapshot sections never shadow the
current config.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
The live validation hit PVE's all-or-nothing restore rule: mpN params
without an explicit rootfs -> HTTP 500 "mount points configured, but
'rootfs' not set" (the same constraint restoretest.go:211 documents for the
live-config path; the spike never ran an override restore). The lost guest
has no live config, so the rootfs SIZE now comes from the archive's own
embedded config via NEW Client.ExtractArchiveConfig (GET vzdump/
extractconfig - verified live: answers 200 under the scoped agent token;
PBS keys stay server-side, the spike's candidate-1 rejection holds; used
for the SIZE ONLY - the bind layout stays the platform constants).
Unparseable/unreadable archive config -> clean refusal before any restore.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
ModeDRGuestLoss now passes restore-time MountOverrides for the two
platform-constant structural binds (mp8 parent, mp9 bootstrap) via the
shared throwaway-volume format helper - without them a customer-archive
restore under the privsep token fails outright ("restoring 'mp8' to bind
mount is only possible for root"). New post-restore step 4d swaps the real
binds in via the host runner (root pct set, one slot per call), deletes the
displaced unusedN volumes (API config PUT; a scoped-token refusal logs the
residue loudly instead of widening privileges), and respects the
committed/launched rollback envelope. Provision passes nil overrides -
byte-identical behavior (regression contract test).
Engine grows an optional HostRunner + StateDir seam (DR refuses up front
without a runner); selftest bring-up wires the ExecRunner + cleans the
scratch mp9 host dir on teardown; proxmox.GuestConfig.Unused() added.
6 new tests incl. C2 mid-swap rollback + C3 older-archive + 403-warn paths.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Pool membership is what lets the pool-scoped token reach a guest; pct restore
--pool sets it only at CREATE, so a restore over an existing VMID drops the guest
from the felhom pool and 403s the next restore-test/DR on VM.Audit. This empty-pool
state is the true root cause of the campaign's "R1" (bind-mount restore failing was
a symptom — restore-test's existing bind neutralization never ran without config-read).
Add Client.PoolAddVMID (PUT /pools, additive+idempotent, Pool.Allocate) and call it
in bring-up after liveness when spec.Pool!="" — warn-not-fail on a hiccup (liveness
wins). B3 scratch-teardown 403 diagnosed as a cascade (restoretest already passes
Pool). Role/ACL untouched. Tests + red-proof.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6
Campaign pool-effects F1 (HIGH): the bring-up compensating rollback and the
restore-test teardown destroyed the target vmid even when RestoreLXC failed
synchronously without creating anything — destroying a guest the transaction
never made (only the pool ACL 403 contained it). A RestoreLXC UPID is now the
sole destroy authorization in all three destroy paths (in-process bring-up
defer, in-process restore-test teardown, Recover). F2: the restore-test
advances past an 'already exists' band vmid (invisible squatter) instead of
failing + false-alerting; a fully-occupied band Skips.
Red-proof verified: with the gates reverted, the four new tests fail with the
innocent-guest destroy. go build/vet/test clean.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
RestoreLXCOptions.Pool → pct restore --pool (omit-when-empty). New
reconcile.DefaultPool="felhom"; BringUpSpec.Pool threaded to the bring-up
restore; BOTH restore sites pool the guest (provision/DR via spec.Pool set to
DefaultPool by the CLI; restore-test scratch via DefaultPool = SPIKE residual
#2). No agent ACL change (ships in host-install v1.6.0); the pool param is inert
until the token has Pool.Allocate + the pool exists, so publishing is safe ahead
of the coordinated swap. Tests + red-proofs; go build/vet/test clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Extends the OS/Docker-data split to a three-volume layout: rootfs + Docker-data
(mp0) + SSD user-data (mp1 @ /mnt/sys_drive, backup=1) = the controller's
system_data_path. Clears the controller's "not a separate drive" warning with
zero controller change (it already auto-discovers <sys_drive>/felhom-data and
warns via system.IsMountPoint; the mp reaches the container via the existing
-v /mnt:/mnt:rslave bind).
- build-golden.sh: --mp1 ...,mp=/mnt/sys_drive,backup=1 (env GOLDEN_SYSDATA_GB=8);
findmnt /mnt/sys_drive separate-mount guard + vzdump aborts if mp0 OR mp1 excluded.
- bringup.go: DefaultSysDataMount=mp1; BringUpSpec.{SysDataGrowGB,SysDataMount};
new "4c" online grow-only block mirroring the "4b" Docker-data grow.
- main.go: -sysdata-grow / -sysdata-mount flags wired into all three call sites.
- Tests: SysDataGrow (asserts ResizeLXC mp1 +42G) + SysDataGrowZeroNoResize.
- RUNBOOK extended to the three-volume layout (32 rootfs + 200 docker + 50 user-data).
Static CT volume, NOT an enrolled drive — never enrolls/ejects/decommissions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
build-golden.sh bakes a small OS rootfs + a dedicated /var/lib/docker volume
(mp0, backup=1) carrying the baked images, plus Docker log rotation. bringup.go
grows the golden's data volume to the per-customer target (DataVolGrowGB) and
emits backup=1 on data mounts (GuestMount.Backup) — closing the spike-B3 silent
DB-loss trap. CLI gains -rootfs-grow/-datavol-grow/-datavol-mount. New
RUNBOOK-provisioning-storage.md. Phase 2 = felhom-controller v0.58.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared front half of provision and guest-loss DR as a journaled reconcile job
(internal/reconcile/bringup.go), mirroring the restore-test's crash-safety but keeping
the guest on success and applying a scenario-specific identity policy. Agent-only; no
hub/wire change. Grounded by the slice-7 bring-up spike (commit 3342993): F1/F3/F4.
- RunBringUp: restore -> reset identity -> size -> attach mounts -> start link-up;
verdict is liveness (waitRunning), success KEEPS the guest.
- identity policy: provision = fresh MAC (net0 sans hwaddr -> PVE regen) + hostname,
host-side; machine-id/host-keys regenerate guest-side (systemd + baked golden unit).
dr_guest_loss = preserve continuity (keep hostname; keep MAC unless KeepMAC=false).
- compensating rollback: mid-flight failure destroys the just-created guest
(SameTxnCreated provenance, gated); new Rollback journal flag + Recover.recoverBringUp
reap a half-built guest from a crash.
- F4: coalesced config PUT + bounded retry on the transient PVE config-lock 500 only.
- --selftest=bring-up (mode/archive/vmid/hostname/keep).
- configs/build-golden.sh: validated golden recipe incl. the F3 first-boot host-key unit.
- doc-03 §9 + identity-reset settled/implemented.
Deferred (stated): provisioning back half -> slice 8; host-loss DR + escrow consumption
and the BringUpSpec source (hub desired-state) -> slice 10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin