RestoreLXCOptions.Pool → pct restore --pool (omit-when-empty). New
reconcile.DefaultPool="felhom"; BringUpSpec.Pool threaded to the bring-up
restore; BOTH restore sites pool the guest (provision/DR via spec.Pool set to
DefaultPool by the CLI; restore-test scratch via DefaultPool = SPIKE residual
#2). No agent ACL change (ships in host-install v1.6.0); the pool param is inert
until the token has Pool.Allocate + the pool exists, so publishing is safe ahead
of the coordinated swap. Tests + red-proofs; go build/vet/test clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New -cores N / -memory M (MiB) flags for --selftest=bring-up|provision (0 = keep
golden default), flowed through bringUpSizing into reconcile.BringUpSpec so the
existing buildBringUpConfig emits cores/memory into the pre-start config PUT (the
cap lands before first boot). No engine change. New pure-function test
TestBuildBringUpConfig_ResourceCaps asserts the set + omit-when-zero cases
(red-proof run + reverted). go build/vet/test clean.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The preventive counterpart to host_disk + storage_fill detectors: the periodic local
whole-guest vzdump now prunes its own old archives (keep-last=3, clamped >=1) so a box
can't refill its own root via its own backups. Local target only — PBS never pruned
(resolved via ListStorage; fail-safe skip on unknown). Seeded in host-install.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
Host-side NFS/SMB automount of a bulk-media NAS share under /mnt/felhom-drives/<name>
(propagates into the guest via the existing shared bind), the +100000 uid recipe,
per-share liveness, and add/list/remove local-API endpoints. A NAS is a distinct class
that bypasses the drive enroll/eject/decommission/SMART/watchdog machinery.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HxLA1mZurFq9kt8hneFeCs
F2-a red-proof PASS: parent came up own peer group (shared:94, not
shared:1), no doubling, guest sees both drives. F2-b PASS: reboot with the
exact snapshot-delete stuck state → agent auto-recovered (unlock → removed
real dangling vzdump snapshot → started CT). Zero spurious pages. Two
live-caught fixes noted (?source=active, AllowWarnings).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
Live on 9999: the recovery's start succeeded (guest running) but WaitTask
false-failed on "WARNINGS: 1" (the systemd-nesting notice). Mirror the
restore-test's start step (AllowWarnings) so a booted guest isn't logged
as a start error.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
Live test on felhom-pve (PVE 9.2.2) caught GET /nodes/{node}/tasks?running=1
returning HTTP 400 "property not defined in schema". The fail-safe correctly
left the lock, but the BackupRunning guard always errored. source=active is
the supported filter and returns the RUNNING tasks.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
F2-b: at startup, recover a guest left with a stale vzdump lock by a
reboot-during-backup — pct unlock -> delete dangling vzdump snapshot ->
start iff onboot, guarded by a no-vzdump-running invariant (fail-safe).
New internal/localapi/stalelock.go; proxmox GuestConfig.Lock()/OnBoot(),
ListSnapshots, ListRunningTasks, Snapshot type. New narrow sudoers grant
FELHOM_STALELOCK (pct unlock) + Critical capability stalelock-unlock.
F2-a: EnsureSharedParent only redeployed the boot script when the UNIT
differed, so the v0.36.6 make-private fix never reached hosts whose unit
was current -> /mnt/felhom-drives stayed in root's shared:1 and doubled
every drive bind. New sharedParentInstallStale compares BOTH script and
unit. Boot-time-only; never churns the live mount.
Both root causes confirmed live on felhom-pve before fixing. Green gate
(build/vet/test) all pass.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0162BnMpUXscPsUB1cU8Tr6K
HostReport.LeafFingerprint rides the served fp (from EnsureLeaf) on every report; empty when local
API disabled. Collector.SetLeafFingerprint threads it like Capabilities. Hub watches it for a re-key.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
controllerHealthy reads RestartCount (running&&rc>0 -> not ok) + signals needsDwell for no-healthcheck;
verify requires verifyDwell(=3) consecutive ok polls for a no-healthcheck image (real healthcheck
trusted immediately). Closes the F1 hole (alpine crash-loop passed the point-in-time check). Red-proof
+ dwell + real-image tests. No sudoers/orchestration change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pg8ANF97SEeKYSN5Jxw3qJ
writeImage drops bash -c/printf for GuestExecStdin(img+\n -> tee /etc/felhom-controller-image);
new Runner.RunStdin/GuestExecStdin route stdin through the fenced sudo -n runner. 5 narrow,
auditable sudoers grants (no general pct exec, no bash -c) + capability manifest entries (Critical)
so the self-probe watches them and the build-test asserts coverage (companion red-proof). No
controller change; swap orchestration/rollback/state unchanged. Spike GO.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
New internal/capability: Manifest of required sudo -n grants + Prober that LISTS each
via 'sudo -n -l' (never executes) + binary-exists check → ok/degraded snapshot on the hub
report. Build-time test asserts manifest⊆sudoers (red-proof: dropping lxc-info FAILs the
gate). Startup logs N/N ok + ERROR per degraded. Serve-degraded; no allowlist change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
Root cause of multi-drive flapping: lxc-info absent from the allowlist → empty guest
PID → bound_under_parent=false for every drive → controller gate stops apps. Audit-driven
completeness pass; surfaces GuestExec/pct-exec (controller-swap) for operator decision.
No binary change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EPZ4GJ8L5Jqf8UiPwbn1kt
Bare commas are command separators in sudoers; the lvs/lsblk -o option lists need
escaped commas. The file had never been visudo-validated live (the demo host ran the
agent root+direct). Surfaced by the BUNDLE host-install visudo -cf gate.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- configs/felhom-agent.service: canonical non-root unit (User=felhom-agent, sudo model);
deliberately NO NoNewPrivileges (breaks sudo) and NO mount-namespacing hardening (breaks
the intermediary-mount drive propagation into guests) — documented inline.
- scripts/publish-agent.sh: build (optional) + PUT binary to Gitea generic + sha256 +
GET round-trip. Pinned version, idempotent (delete-then-PUT).
- configs/build-golden.sh: after vzdump, compute sha256 + PUT golden.tar.zst to Gitea
generic (version = baked controller version). Opt-in; local auto-discovery stays fallback.
- cmd/felhom-agent/main.go: version 0.42.0 -> 0.43.0.
- README: process model now canonical (non-root + publish/install).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The CONTROLLER_IMAGE default (arg 6) was a stale :0.43.0, so an argument-less
golden build baked an ancient controller (fresh Day-0 boxes started at 0.77).
Bumped the default to the current :0.85.1; always pass it explicitly per rebuild.
Golden rebuilt at 0.85.1 on felhom-pve (volid vzdump-lxc-9100-2026_06_27-11_42_51);
baked-image verify confirmed :0.85.1 in the build guest. No agent binary change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FSZmmSFVzGwEzhYmxbkgBK
New local-API POST /controller/swap (+ GET /controller/swap/status), withGuest-
scoped: the agent records the previous image, confirms the target is present,
rewrites /etc/felhom-controller-image, restarts felhom-controller-bootstrap.service,
verifies the new controller is healthy (docker inspect, <=90s), and ROLLS BACK to
the previous image if not. Single-flight per guest; strict image-ref validation;
crash-safety state file. GuestBinder.GuestExec is the pct-exec seam.
--selftest=controller-swap exercises it directly.
Tests: happy/rollback-on-unhealthy(+red-proof)/image-absent/no-healthcheck/
bad-image-400/single-flight-409.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtXesNa2LGbMmE4DNL6SE7
Destroyed + re-provisioned 9201 through the real provision chain with v0.41.0: fresh pct config
showed onboot:1 with no manual set. Host reboot of felhom-pve -> 9201 auto-started (running, no
manual pct start); onboot:0 scratch guests stayed stopped; drives re-bound at stable, sys_drive
separate, controller + infra healthy. Demo controller redeployed to v0.82.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FpBYrZCt9sFDqLgbG5GRGD
The golden bakes --onboot 0 (template safety) and the back-half never overrode it, so every
provisioned customer guest was onboot:0 -> after a host reboot/power-cut the customer's whole
home-server stayed stopped until a manual pct start. Add a fatal 'pct set <vmid> -onboot 1' step
to BackHalf.Provision (right after the config-mount attach), mirroring the existing pct set ops.
No startup/boot-order: the v0.75 mountpoint-gate covers the drive-bind race at boot.
Golden build-golden.sh unchanged (templates must not auto-start). Unit-tested
(TestProvision_SetsOnbootOne + red-proof). RUNBOOK note added.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01FpBYrZCt9sFDqLgbG5GRGD
Extends the OS/Docker-data split to a three-volume layout: rootfs + Docker-data
(mp0) + SSD user-data (mp1 @ /mnt/sys_drive, backup=1) = the controller's
system_data_path. Clears the controller's "not a separate drive" warning with
zero controller change (it already auto-discovers <sys_drive>/felhom-data and
warns via system.IsMountPoint; the mp reaches the container via the existing
-v /mnt:/mnt:rslave bind).
- build-golden.sh: --mp1 ...,mp=/mnt/sys_drive,backup=1 (env GOLDEN_SYSDATA_GB=8);
findmnt /mnt/sys_drive separate-mount guard + vzdump aborts if mp0 OR mp1 excluded.
- bringup.go: DefaultSysDataMount=mp1; BringUpSpec.{SysDataGrowGB,SysDataMount};
new "4c" online grow-only block mirroring the "4b" Docker-data grow.
- main.go: -sysdata-grow / -sysdata-mount flags wired into all three call sites.
- Tests: SysDataGrow (asserts ResizeLXC mp1 +42G) + SysDataGrowZeroNoResize.
- RUNBOOK extended to the three-volume layout (32 rootfs + 200 docker + 50 user-data).
Static CT volume, NOT an enrolled drive — never enrolls/ejects/decommissions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_017PsnU2ASocYrvzqE82YDYW
Live PBS coord: new internal/pbs/live_reporter.go (LiveSnapshotReporter implements
hub.PBSReporter via the cheap Client.Snapshots() list with last-known-good fallback,
bounded by an 8s timeout, list-only — never triggers a verify). Closes the gap where
the recipe's pbs block was omitted whenever the verify-loop SnapshotStore was empty
(one-shot collect + the first ~6h after a daemon restart). SnapshotStore.Get added
(per-datastore LKG). Wired into the collector in both runDaemon and runSelftestHub;
the verify loop keeps Recording into the SAME shared store via one hoisted pbsTargets.
v1 host-half drive shape: dropped drives[].role (hub/operator-owned manifest concept,
not host-derivable) and drives[].restic_repo_coord (named a backup tier that doesn't
exist). Drive shape is now {durable_id, mount_path, intent, fs_type?, total_bytes}.
Hub reads drives as json.RawMessage → no hub struct change; goldens re-pinned
byte-identical (agent + hub copies).
Tests: live_reporter_test.go (T1 load-bearing coord-without-verify + T2..T6),
TestDRRecipeHostHalf_V1DriveShape; each companion demonstrated to fail pre-fix then
reverted. go build/vet/test green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
DR recipe slice (agent half), grounded in SPIKE-dr-recipe-2026-06-16. Additive
`dr_recipe` host-report section = the non-secret reconstruction scaffolding the
operator must rebuild before PBS bytes can land. Built by pure
BuildDRRecipeHostHalf from facts the report already collects (no new reads):
guests[] sizing, drives[] (user-data by durable_id/role/mount/intent),
pve_storage[] (storage.cfg), pbs coordinates.
BOUNDARY (Phase-1 lesson): every field is an identifier/intent/size/coordinate —
never a key/password/token/hash/ENC:. PBS key stays in escrow; restic password
stays in escrow; the recipe names only the coordinates the restore targets.
Tests: BuildDRRecipeHostHalf selection, NoPBS, NoSecrets (boundary mirror),
dr_recipe key-set in the cross-repo golden contract test. recipe_version=1,
ignore-unknown on read.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The skip-if-mounted optimization defeated the actual root cause: felhom-usb is
mounted now but its unit is `disabled`, so a host reboot would not auto-mount
it. ReassertEnrolledMounts now skips ONLY the durable steady state (mounted AND
enabled) via the pure shouldReassertMount; a mounted-but-disabled unit is
re-asserted so enable --now re-creates the wants-symlink. Enabled-state read by
privilege-free Lstat of the multi-user.target.wants symlink (unitEnabled) — no
systemctl is-enabled subprocess, no new sudoers entry.
Tests: TestShouldReassertMount (4 combos), TestUnitEnabled (wants-symlink).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TASK A — close out the reboot story (agent half). Root cause (pinned live on
felhom-pve): an enrolled .mount unit left `disabled` by a prior detach never
auto-mounts at boot, and kernel re-enumeration can move a drive's node
(/dev/sdb->sdc). Fix re-asserts every enrolled mount by filesystem UUID at
startup + on the periodic tick.
- ResolveStorageDevice: resolve uuid:<fs-uuid> -> current /dev node via
/dev/disk/by-uuid (never a cached node); errors if absent.
- parseFelhomMountUnit: pure inverse of renderMountUnit (marker-gated).
- (*SudoHostOps).ReassertEnrolledMounts: re-run EnsureMount (enable --now) for
any enrolled unit not in /proc/mounts; idempotent, skips mounted/absent.
- main.go: runs before ReassertGuestBinds at startup + on the 20s tick.
- tests (Linux, seam=device resolution): letter-move tolerated (sdb->sdc) +
red-proof companion, absent/scheme rejection, render->parse round-trip.
TASK A2 verdict: enrolling a NEW drive does NOT need an LXC restart — the path
lands on the live AttachDrive (shared parent, named live slots), not RebootGuest.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
make-private+make-shared only when first creating the self-bind; re-running it
churns the peer group and orphans the guest's slave (propagation dies).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The self-bind inherited /'s shared peer group, so binds under /mnt/felhom-drives
propagated back and doubled. make-private (own group) before make-shared fixes it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
countHostMounts + normalize: no-op only when exactly one bind is guest-visible;
else strip all binds and lay one fresh. Converges a stacked double-bind to one
(the old umount-one+mount-one never did).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A GuestBinder mutex prevents a concurrent reconnect + periodic reconcile from
both passing isHostMountpoint and double-binding a stable path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Detach now removes ALL stacked binds at a stable path, not just one layer, so an
eject fully detaches even with >1 bind (keeping fail-close intact).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Eject now DetachDrive's the bind under the parent but leaves the raw /mnt/<name>
mounted, so disconnect->reconnect re-binds on a non-removable drive. Tests updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The decommission unmounted the raw /mnt/<name>, orphaning a non-removable drive
so re-enroll bound an empty dir. Now DetachDrive only (bind under parent); raw
stays mounted so re-enroll re-binds. Test: raw NOT unmounted + DetachDrive called.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin