build-golden.sh 2.1.0 -> 3.0.0: a layout change is a major. The golden ships
ONE data volume at a NEUTRAL path (/var/lib/felhom); both /var/lib/docker
and /mnt/sys_drive are binds of subdirectories of it. mp1 is gone.
The variant was chosen on measurement. Three candidates were built and
rebooted (SPIKE-r165-phase0-2026-08-03.md); all three boot, reboot 3/3, give
ONE df figure and keep a container's statfs(/) on the merged volume — the
ordering worry that motivated the probe did not materialise. They differ
only in which guarantee they break: volume-at-docker puts customer backups
inside Docker's data-root; volume-at-sys_drive puts Docker's ENTIRE
data-root under /mnt, which the controller container mounts wholesale
(measured: it then sees /mnt/sys_drive/docker). V-c breaks neither.
The four assertions were RETARGETED, never deleted, and each was RUN against
a deliberately wrong shape — a real split guest and a real archive of it:
8 checks, 8 passed. A new 2b asserts both paths are ONE filesystem, which
catches the S2 shape the spike ranked worse than the split. Assertion 5
replaces the old "was mp1 excluded?" guard, whose pattern could no longer
match — a guard that cannot match has silently stopped guarding.
Provisioning: one volume, one grow. SysDataGrowGB is FOLDED IN rather than
dropped, because a census established that felhom-host-install.sh passes
-sysdata-grow and the two do not upgrade in the same instant; dropping it
would silently shrink every appliance by 42 of 250 GiB. The flags stay
accepted for the same reason. The existing test was retargeted to pin the
fold, and it caught the change before I did.
Enabling Megosztás on a fresh box pulled felhom-samba from the registry with
zero feedback — minutes of silent nothing. Observed live, twice.
Cause: this script carried its own hand-maintained array of three image tags,
with a comment telling the reader to keep it in sync with the controller's
internal/infra constants. It drifted the moment a fourth stack was added.
felhom-samba was never added here, so the golden baked 3 of 4.
The fix is structural rather than a fourth copy of the list: it now comes from
the controller image the bake just pulled (--print-infra-images, backed by
infra.Images(), which derives from the pins themselves). The golden bakes
exactly what THAT controller version will request, so the two cannot disagree.
Ordering fix this exposed: docker logout ran immediately after the controller
pull, but felhom-samba is on the same private registry, so the infra loop would
have 401'd. Logout moved after the loop, plus a hard assertion that no
credential remains in the guest before it is archived.
Pre-0.147.0 controllers have no such flag; the bake falls back to the historical
3-image list and says so loudly — the fallback IS the drift-prone thing being
removed, so it must never pass silently.
No agent version bump: build tooling only. Effective at the next golden build;
the current golden is not rebuilt for this.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
- configs/felhom-agent.service: canonical non-root unit (User=felhom-agent, sudo model);
deliberately NO NoNewPrivileges (breaks sudo) and NO mount-namespacing hardening (breaks
the intermediary-mount drive propagation into guests) — documented inline.
- scripts/publish-agent.sh: build (optional) + PUT binary to Gitea generic + sha256 +
GET round-trip. Pinned version, idempotent (delete-then-PUT).
- configs/build-golden.sh: after vzdump, compute sha256 + PUT golden.tar.zst to Gitea
generic (version = baked controller version). Opt-in; local auto-discovery stays fallback.
- cmd/felhom-agent/main.go: version 0.42.0 -> 0.43.0.
- README: process model now canonical (non-root + publish/install).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CONTROLLER_IMAGE default (arg 6) was a stale :0.43.0, so an argument-less
golden build baked an ancient controller (fresh Day-0 boxes started at 0.77).
Bumped the default to the current :0.85.1; always pass it explicitly per rebuild.
Golden rebuilt at 0.85.1 on felhom-pve (volid vzdump-lxc-9100-2026_06_27-11_42_51);
baked-image verify confirmed :0.85.1 in the build guest. No agent binary change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FSZmmSFVzGwEzhYmxbkgBK
Extends the OS/Docker-data split to a three-volume layout: rootfs + Docker-data
(mp0) + SSD user-data (mp1 @ /mnt/sys_drive, backup=1) = the controller's
system_data_path. Clears the controller's "not a separate drive" warning with
zero controller change (it already auto-discovers <sys_drive>/felhom-data and
warns via system.IsMountPoint; the mp reaches the container via the existing
-v /mnt:/mnt:rslave bind).
- build-golden.sh: --mp1 ...,mp=/mnt/sys_drive,backup=1 (env GOLDEN_SYSDATA_GB=8);
findmnt /mnt/sys_drive separate-mount guard + vzdump aborts if mp0 OR mp1 excluded.
- bringup.go: DefaultSysDataMount=mp1; BringUpSpec.{SysDataGrowGB,SysDataMount};
new "4c" online grow-only block mirroring the "4b" Docker-data grow.
- main.go: -sysdata-grow / -sysdata-mount flags wired into all three call sites.
- Tests: SysDataGrow (asserts ResizeLXC mp1 +42G) + SysDataGrowZeroNoResize.
- RUNBOOK extended to the three-volume layout (32 rootfs + 200 docker + 50 user-data).
Static CT volume, NOT an enrolled drive — never enrolls/ejects/decommissions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
docker-ce auto-starts on install with the default config; only a restart picks
up the just-written daemon.json. Same start-vs-restart trap the spike flagged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validation caught that Docker 29's default containerd-snapshotter keeps the
image store at /var/lib/containerd (on the rootfs), so mounting the data volume
at /var/lib/docker only moved named volumes — images (1.2G) stayed on the
rootfs, defeating the split. overlay2 stores images+overlay+volumes under
data-root = the data volume, which the split and the controller's statfs(/)
guard both require. Golden daemon.json now sets features.containerd-snapshotter
false + a driver guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build-golden.sh bakes a small OS rootfs + a dedicated /var/lib/docker volume
(mp0, backup=1) carrying the baked images, plus Docker log rotation. bringup.go
grows the golden's data volume to the per-customer target (DataVolGrowGB) and
emits backup=1 on data mounts (GuestMount.Backup) — closing the spike-B3 silent
DB-loss trap. CLI gains -rootfs-grow/-datavol-grow/-datavol-mount. New
RUNBOOK-provisioning-storage.md. Phase 2 = felhom-controller v0.58.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build-golden.sh bootstrap makes /mnt a shared mount and binds it :rslave into the
controller container so enrolled data drives (and P3 self-heal remounts) propagate
in. Scoped to /mnt (Model A: only felhom-data-namespace mounts). Spike-proven.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validate the customer.id parsed from bootstrap.json to a DNS-safe label (reject
spaces/slashes/leading-dash) and pass it via a quoted array, so a malformed or
hostile id can't smuggle extra docker flags (e.g. --privileged) into the
controller's de-privileged `docker run`. Empty/invalid → no --hostname.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build-golden.sh: same-path /opt/docker/stacks host bind (Section-G fix, breaks
all bind-mounted stacks without it) + --hostname <customer-id> from bootstrap.json
(portable sed parse, no jq) + bake the 3 pinned public base-infra images with a
manifest-inspect hard gate. Provision --selftest defaults -hostname to the
DNS-safe-sanitized customer-id so the CT/LXC is named meaningfully.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lockstep with felhom-controller v0.40.0. The agent now bakes a v2 bootstrap.json
carrying only what the controller needs to PULL its config from the hub:
customer.id + hub.url + hub.retrieval_password + the per-guest local_api. Stops
baking the agent's host hub key/host_id (and customer name/domain/email) into the
guest — the controller gets the customer-scoped key from the hub pull.
- internal/provision/doc.go: SchemaV2; DocCustomer{id}; DocHub{url,retrieval_password}.
- backhalf.go: render v2; require customer.id + hub.url + hub.retrieval_password.
- cmd/.../main.go --selftest=provision: new required -hub-password flag; stop
baking APIKey/HostID; -customer-domain/name/email accepted but not baked.
- configs/build-golden.sh: default CONTROLLER_IMAGE off stale :v0.35.0 -> :0.40.0.
- doc_test.go/backhalf_test.go updated to v2 shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BackupRunner.BackupWithSnapshotHook tails the task log for the 'create storage
snapshot' marker (snapshot mode only) and fires onSnapshot once; localapi flips
/backup/status to 'snapshotted' before 'done' so the controller resumes early.
Phase 0 validated on PVE 9.2.2: marker confirmed, downtime ~24s->~1s (934MB).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Live validation on demo-felhom caught the felhom-regen-hostkeys unit failing with
203/EXEC: ExecStart was /usr/sbin/ssh-keygen but on Debian 13 ssh-keygen is at
/usr/bin/ssh-keygen. Fixed build-golden.sh, rebuilt the golden, re-validated — host
keys now regenerate on first boot by the baked unit (agent issues no ssh-keygen).
All three live scenarios green: provision (fresh MAC, host keys via unit, machine-id,
Docker, DHCP), dr (continuity: hostname + host keys preserved), Recover (killed
mid-restore -> orphan rolled back idempotently). REPORT + CHANGELOG updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared front half of provision and guest-loss DR as a journaled reconcile job
(internal/reconcile/bringup.go), mirroring the restore-test's crash-safety but keeping
the guest on success and applying a scenario-specific identity policy. Agent-only; no
hub/wire change. Grounded by the slice-7 bring-up spike (commit 3342993): F1/F3/F4.
- RunBringUp: restore -> reset identity -> size -> attach mounts -> start link-up;
verdict is liveness (waitRunning), success KEEPS the guest.
- identity policy: provision = fresh MAC (net0 sans hwaddr -> PVE regen) + hostname,
host-side; machine-id/host-keys regenerate guest-side (systemd + baked golden unit).
dr_guest_loss = preserve continuity (keep hostname; keep MAC unless KeepMAC=false).
- compensating rollback: mid-flight failure destroys the just-created guest
(SameTxnCreated provenance, gated); new Rollback journal flag + Recover.recoverBringUp
reap a half-built guest from a crash.
- F4: coalesced config PUT + bounded retry on the transient PVE config-lock 500 only.
- --selftest=bring-up (mode/archive/vmid/hostname/keep).
- configs/build-golden.sh: validated golden recipe incl. the F3 first-boot host-key unit.
- doc-03 §9 + identity-reset settled/implemented.
Deferred (stated): provisioning back half -> slice 8; host-loss DR + escrow consumption
and the BringUpSpec source (hub desired-state) -> slice 10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin