The CONTROLLER_IMAGE default (arg 6) was a stale :0.43.0, so an argument-less
golden build baked an ancient controller (fresh Day-0 boxes started at 0.77).
Bumped the default to the current :0.85.1; always pass it explicitly per rebuild.
Golden rebuilt at 0.85.1 on felhom-pve (volid vzdump-lxc-9100-2026_06_27-11_42_51);
baked-image verify confirmed :0.85.1 in the build guest. No agent binary change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FSZmmSFVzGwEzhYmxbkgBK
Extends the OS/Docker-data split to a three-volume layout: rootfs + Docker-data
(mp0) + SSD user-data (mp1 @ /mnt/sys_drive, backup=1) = the controller's
system_data_path. Clears the controller's "not a separate drive" warning with
zero controller change (it already auto-discovers <sys_drive>/felhom-data and
warns via system.IsMountPoint; the mp reaches the container via the existing
-v /mnt:/mnt:rslave bind).
- build-golden.sh: --mp1 ...,mp=/mnt/sys_drive,backup=1 (env GOLDEN_SYSDATA_GB=8);
findmnt /mnt/sys_drive separate-mount guard + vzdump aborts if mp0 OR mp1 excluded.
- bringup.go: DefaultSysDataMount=mp1; BringUpSpec.{SysDataGrowGB,SysDataMount};
new "4c" online grow-only block mirroring the "4b" Docker-data grow.
- main.go: -sysdata-grow / -sysdata-mount flags wired into all three call sites.
- Tests: SysDataGrow (asserts ResizeLXC mp1 +42G) + SysDataGrowZeroNoResize.
- RUNBOOK extended to the three-volume layout (32 rootfs + 200 docker + 50 user-data).
Static CT volume, NOT an enrolled drive — never enrolls/ejects/decommissions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
docker-ce auto-starts on install with the default config; only a restart picks
up the just-written daemon.json. Same start-vs-restart trap the spike flagged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validation caught that Docker 29's default containerd-snapshotter keeps the
image store at /var/lib/containerd (on the rootfs), so mounting the data volume
at /var/lib/docker only moved named volumes — images (1.2G) stayed on the
rootfs, defeating the split. overlay2 stores images+overlay+volumes under
data-root = the data volume, which the split and the controller's statfs(/)
guard both require. Golden daemon.json now sets features.containerd-snapshotter
false + a driver guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build-golden.sh bakes a small OS rootfs + a dedicated /var/lib/docker volume
(mp0, backup=1) carrying the baked images, plus Docker log rotation. bringup.go
grows the golden's data volume to the per-customer target (DataVolGrowGB) and
emits backup=1 on data mounts (GuestMount.Backup) — closing the spike-B3 silent
DB-loss trap. CLI gains -rootfs-grow/-datavol-grow/-datavol-mount. New
RUNBOOK-provisioning-storage.md. Phase 2 = felhom-controller v0.58.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build-golden.sh bootstrap makes /mnt a shared mount and binds it :rslave into the
controller container so enrolled data drives (and P3 self-heal remounts) propagate
in. Scoped to /mnt (Model A: only felhom-data-namespace mounts). Spike-proven.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validate the customer.id parsed from bootstrap.json to a DNS-safe label (reject
spaces/slashes/leading-dash) and pass it via a quoted array, so a malformed or
hostile id can't smuggle extra docker flags (e.g. --privileged) into the
controller's de-privileged `docker run`. Empty/invalid → no --hostname.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build-golden.sh: same-path /opt/docker/stacks host bind (Section-G fix, breaks
all bind-mounted stacks without it) + --hostname <customer-id> from bootstrap.json
(portable sed parse, no jq) + bake the 3 pinned public base-infra images with a
manifest-inspect hard gate. Provision --selftest defaults -hostname to the
DNS-safe-sanitized customer-id so the CT/LXC is named meaningfully.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lockstep with felhom-controller v0.40.0. The agent now bakes a v2 bootstrap.json
carrying only what the controller needs to PULL its config from the hub:
customer.id + hub.url + hub.retrieval_password + the per-guest local_api. Stops
baking the agent's host hub key/host_id (and customer name/domain/email) into the
guest — the controller gets the customer-scoped key from the hub pull.
- internal/provision/doc.go: SchemaV2; DocCustomer{id}; DocHub{url,retrieval_password}.
- backhalf.go: render v2; require customer.id + hub.url + hub.retrieval_password.
- cmd/.../main.go --selftest=provision: new required -hub-password flag; stop
baking APIKey/HostID; -customer-domain/name/email accepted but not baked.
- configs/build-golden.sh: default CONTROLLER_IMAGE off stale :v0.35.0 -> :0.40.0.
- doc_test.go/backhalf_test.go updated to v2 shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BackupRunner.BackupWithSnapshotHook tails the task log for the 'create storage
snapshot' marker (snapshot mode only) and fires onSnapshot once; localapi flips
/backup/status to 'snapshotted' before 'done' so the controller resumes early.
Phase 0 validated on PVE 9.2.2: marker confirmed, downtime ~24s->~1s (934MB).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Live validation on demo-felhom caught the felhom-regen-hostkeys unit failing with
203/EXEC: ExecStart was /usr/sbin/ssh-keygen but on Debian 13 ssh-keygen is at
/usr/bin/ssh-keygen. Fixed build-golden.sh, rebuilt the golden, re-validated — host
keys now regenerate on first boot by the baked unit (agent issues no ssh-keygen).
All three live scenarios green: provision (fresh MAC, host keys via unit, machine-id,
Docker, DHCP), dr (continuity: hostname + host keys preserved), Recover (killed
mid-restore -> orphan rolled back idempotently). REPORT + CHANGELOG updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared front half of provision and guest-loss DR as a journaled reconcile job
(internal/reconcile/bringup.go), mirroring the restore-test's crash-safety but keeping
the guest on success and applying a scenario-specific identity policy. Agent-only; no
hub/wire change. Grounded by the slice-7 bring-up spike (commit 3342993): F1/F3/F4.
- RunBringUp: restore -> reset identity -> size -> attach mounts -> start link-up;
verdict is liveness (waitRunning), success KEEPS the guest.
- identity policy: provision = fresh MAC (net0 sans hwaddr -> PVE regen) + hostname,
host-side; machine-id/host-keys regenerate guest-side (systemd + baked golden unit).
dr_guest_loss = preserve continuity (keep hostname; keep MAC unless KeepMAC=false).
- compensating rollback: mid-flight failure destroys the just-created guest
(SameTxnCreated provenance, gated); new Rollback journal flag + Recover.recoverBringUp
reap a half-built guest from a crash.
- F4: coalesced config PUT + bounded retry on the transient PVE config-lock 500 only.
- --selftest=bring-up (mode/archive/vmid/hostname/keep).
- configs/build-golden.sh: validated golden recipe incl. the F3 first-boot host-key unit.
- doc-03 §9 + identity-reset settled/implemented.
Deferred (stated): provisioning back half -> slice 8; host-loss DR + escrow consumption
and the BringUpSpec source (hub desired-state) -> slice 10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin