TASK A — close out the reboot story (agent half). Root cause (pinned live on
felhom-pve): an enrolled .mount unit left `disabled` by a prior detach never
auto-mounts at boot, and kernel re-enumeration can move a drive's node
(/dev/sdb->sdc). Fix re-asserts every enrolled mount by filesystem UUID at
startup + on the periodic tick.
- ResolveStorageDevice: resolve uuid:<fs-uuid> -> current /dev node via
/dev/disk/by-uuid (never a cached node); errors if absent.
- parseFelhomMountUnit: pure inverse of renderMountUnit (marker-gated).
- (*SudoHostOps).ReassertEnrolledMounts: re-run EnsureMount (enable --now) for
any enrolled unit not in /proc/mounts; idempotent, skips mounted/absent.
- main.go: runs before ReassertGuestBinds at startup + on the 20s tick.
- tests (Linux, seam=device resolution): letter-move tolerated (sdb->sdc) +
red-proof companion, absent/scheme rejection, render->parse round-trip.
TASK A2 verdict: enrolling a NEW drive does NOT need an LXC restart — the path
lands on the live AttachDrive (shared parent, named live slots), not RebootGuest.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
make-private+make-shared only when first creating the self-bind; re-running it
churns the peer group and orphans the guest's slave (propagation dies).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The self-bind inherited /'s shared peer group, so binds under /mnt/felhom-drives
propagated back and doubled. make-private (own group) before make-shared fixes it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
countHostMounts + normalize: no-op only when exactly one bind is guest-visible;
else strip all binds and lay one fresh. Converges a stacked double-bind to one
(the old umount-one+mount-one never did).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A GuestBinder mutex prevents a concurrent reconnect + periodic reconcile from
both passing isHostMountpoint and double-binding a stable path.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Detach now removes ALL stacked binds at a stable path, not just one layer, so an
eject fully detaches even with >1 bind (keeping fail-close intact).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Eject now DetachDrive's the bind under the parent but leaves the raw /mnt/<name>
mounted, so disconnect->reconnect re-binds on a non-removable drive. Tests updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The decommission unmounted the raw /mnt/<name>, orphaning a non-removable drive
so re-enroll bound an empty dir. Now DetachDrive only (bind under parent); raw
stays mounted so re-enroll re-binds. Test: raw NOT unmounted + DetachDrive called.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
GET /disks emits guest_boot_id = <host-btime>-<guest-init-starttime>: changes on
every guest/host boot, stable across controller-only restarts. The controller
persists it + deterministically recreates drive-backed apps on change (replaces
the timed state-sample). Non-hollow parser test + companion.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared-parent oneshot never ran before pve-guests (WantedBy=multi-user
wasn't pulled into the boot transaction), so on a host reboot the guest bound a
not-yet-shared parent -> private -> propagation broken. Now WantedBy=pve-guests
(pve-guests pulls it in + Before= orders it first); EnsureSharedParent reinstalls
the unit when content differs so the fix deploys.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A guest's parent bind is non-recursive, so a guest reboot leaves enrolled drives
bound on the HOST but invisible in the fresh guest ns (propagation only delivers
new events). AttachDrive(vmid) now checks GuestSeesMount (/proc/<pid>/mountinfo)
and force re-binds (umount+mount) to re-propagate; a 20s periodic reconcile
self-heals guest reboots without an agent restart; BoundUnderParent reflects guest
visibility (the controller gate's signal). Caught + fixed in the live migration.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replaces the per-drive 'pct set -mpN' bind with ONE permanent parent bind
/mnt/felhom-drives plus host-side felhom-data swaps underneath it (propagates
into the running guest live, no pct, no reboot; C1-immune; confined; fail-closed
when absent). EnsureSharedParent installs a boot unit ordered Before=pve-guests.
ReassertGuestBinds is now a pure host-side reconcile. /disks reports GuestPath +
BoundUnderParent for the controller repoint+gate. Non-hollow tests + companions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Pre-start PVE hookscript (internal/guesthook) creates host-root placeholders for
absent bind-mount sources so the guest always boots (fail-closed); decommission
now pct set --delete's the dead mp (GuestBinder.DetachBind) so a missing source
can't brick the next reboot (B3 C1 bug). Non-hollow tests + companions. Installed
+ registered per-guest by the provision back-half. Transitional ahead of the
intermediary-mount re-architecture which makes C1 structural.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
POST /disks/decommission mirrors eject (withGuest, user-data role gate) — no
operator signature, non-destructive (never formats): sets IntentDecommissioned,
prunes the GuestBindStore entry, unmounts. ReassertGuestBinds is now intent-aware
(skip non-enrolled) so a decommissioned-but-present drive never auto-rebinds on
agent restart — the load-bearing F9-reconnect fix. GuestBindStore.Remove added.
Operator-signed DecommissionExecutor + classify untouched. Non-hollow tests incl.
the intent-aware reassert companion (mutation-proven to fail on intent-blind code).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The format ran mkfs under the HTTP request context, so the controller's 15s client
timeout cancelled it → SIGKILL mid-write → corrupt disk. Now mkfs runs DETACHED off
s.baseCtx (a dropped request can't kill it) via a persisted formatJob record; the handler
still waits to return the synchronous result (backward-compatible with the v0.62.0
controller) but abandoning the wait on client-disconnect leaves the mkfs running to
completion. New GET /disks/format/status surfaces the job (additive). RecoverFormatJob
runs on agent startup: a record left 'running' (agent died mid-format) is re-resolved by
durable-id (anti-retarget — absent/swapped disk NOT re-formatted) and the mkfs re-run; a
blank/path-bound interrupted format is marked failed (retry), never auto-re-run.
Tests: detached run persists running→done + binds durable-id; status endpoint; recovery
re-runs an interrupted durable-id-bound format; skips blank; skips unresolvable durable-id.
Version 0.30.0 → 0.31.0.
The in-guest bind (pct set -mpN) is config state that a destroy+re-provision drops, and
nothing restored it — so a re-provisioned guest came up with its enrolled HDD unattached
(the live-drive F9 finding). New GuestBindStore persists, per guest, the durable-ids of
enrolled drives (recorded at guest-attach); ReassertGuestBinds runs on agent startup (the
host's bring-up/reconcile trigger) and re-adds any bind a guest is MISSING — but ONLY when
the durable-id still resolves to a present, mounted drive (a swapped/absent drive is never
auto-bound) and the guest lacks it (idempotent). The re-added bind activates on the guest's
next reboot, like the enroll flow. Wired in main.go (store opened beside drive-intents.json;
ReassertGuestBinds called before the local API serves).
Tests: restores a missing bind with no manual call (the operator's real-trigger proof);
skips absent/swapped durable-id; no-op when already bound; store survives reopen (restart).
F20-BUG2: the /disks list only carried DurableID in the uuid: scheme (for /disks/assign),
but the wipe gate resolves devices in the byid:/byuuid: scheme — so a customer confirming a
wipe with the advertised id was refused (binding_mismatch). Added a shared s.deviceDurableID
seam used by BOTH handleDisks (new DiskInfo.WipeDurableID) and the format gate, so the id the
customer copies from the list is exactly the id the gate accepts. DurableID (uuid:) is unchanged
(still feeds assign).
F9 (reporting half): added DiskInfo.GuestAttached — whether the drive's namespace is actually
bound into THIS guest's config (guestBoundPaths), distinct from mere host presence (State). This
is the signal whose absence made the HDD look available when it wasn't attached, and resolves the
F2 hdd_configured-vs-/disks disagreement.
Tests: wipe_durable_id is the gate scheme + distinct from uuid:; the list's wipe id matches the
gate's device-id binding (no mismatch); guest_attached true iff bound into the guest.
handleDiskFormat's customer-confirmed branch formatted the mutable req.Device
path; the durable id only bound the confirmation, never the mkfs target. A /dev
reassignment between inspect and mkfs could wipe the wrong physical disk.
Now mirrors signedjobs.WipeExecutor: resolve confirmed durable id -> current
device, re-derive + require exact match, re-inspect (still data-bearing), then
format THAT device. Any refusal -> 409, no mkfs. New antiRetargetResolve helper
(injected deps, unit-tested: mismatch/gone/blank/empty all refuse). Injectable
reresolveWipe seam on Server (defaults to real storage funcs).
BRANCH ONLY — pending supervised review/deploy (see AGENT-001-FIX-NOTES.md).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
After a guest DHCP IP move, the split-horizon resolver kept serving the old IP:
the drop-in (address=/domain/ip) updated but 'systemctl reload dnsmasq' (SIGHUP)
does NOT re-read /etc/dnsmasq.d config — only /etc/hosts + cache. Changed
reload() -> restartDnsmasq() so address= changes actually take effect.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build-golden.sh bakes a small OS rootfs + a dedicated /var/lib/docker volume
(mp0, backup=1) carrying the baked images, plus Docker log rotation. bringup.go
grows the golden's data volume to the per-customer target (DataVolGrowGB) and
emits backup=1 on data mounts (GuestMount.Backup) — closing the spike-B3 silent
DB-loss trap. CLI gains -rootfs-grow/-datavol-grow/-datavol-mount. New
RUNBOOK-provisioning-storage.md. Phase 2 = felhom-controller v0.58.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PVE refuses a restore carrying mountpoint params unless rootfs is also set ("mount
points configured, but 'rootfs' not set"). Size the rootfs override from the source
rootfs (rootfsSizeGB/sizeToGB, round up). Validated manually on the host: restore of
bind-mounted 9201 with --rootfs + --mp0/--mp9 overrides boots + tears down cleanly.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A slice-10 enrolled guest's data drive is a host bind-mount mp0 that the privsep
token can't vzrestore ("bind mount is only possible for root") — so the restore-test
failed for every enrolled guest regardless of backup tier. The restore-test now reads
the source guest config (vmid from the archive volid) and passes RestoreLXC mp
overrides converting each bind-mount mpN to a throwaway 1G volume on the restore
storage (no root needed; boot-verify doesn't need the data). proxmox.RestoreLXC gains
MountOverrides. + unit tests (archiveVMID, bindMountOverrides).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- BackupConfig.BackupTarget() defaults whole-guest backup to felhom-pbs (separate
hardware = real DR), configurable via backup.local_backup_target; all NewBackupRunner
sites route through it. PBS round-trip proven live (snapshot marker + restore-test +
pct-restore) before the re-point.
- signedjobs DecommissionExecutor + ExecutorChain: makes IntentDecommissioned reachable
ONLY via a verified operator signature (keyed by the watchdog's storage durable-id);
felhom-opsign builds decommission params from -durable-id. Runner wiring moved below
the intent-store open.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
POST /disks/guest-attach binds an enrolled drive's felhom-data namespace into
the guest (Model A: felhom-data is the bind source mounted at /mnt/<name>, so
only Felhom's namespace crosses in). GuestBinder does mkdir+chown(100000)+pct set
(RW bind) via the fenced runner. Idempotent, free-slot selection, path-validated.
Spike-proven on 9201. Pairs with controller P2C + golden /mnt:rslave (P2B).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
handleDiskEject now resolves the authoritative role of the storage at `where`
and refuses 403 (no Unmount) unless it is user-data. Fails safe to protected on
ambiguity. Adds roleForMountPath + an injectable HostReader seam for testability.
TestEject_RoleGated asserts protected mounts are refused with no Unmount.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
localapi.DiskInfo gains durable_id (from StorageTarget.DurableID, "uuid:<fs-uuid>"
for usb/local-dir). The de-privileged controller can't read a device's fs UUID
but assign mounts strictly by UUID — this read-only field is the only way it
learns the assign key. No new privilege, no gate change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Host-side dnsmasq the agent manages so LAN clients reach their guest directly
(same hostname + real wildcard cert, no Cloudflare hairpin). Renders local=/
+address=/ per customer (AAAA->NODATA via authoritative zone, wildcard A ->
live guest IP), forwards everything else. Manager ensures dnsmasq+base config,
discovers guest IP (pct exec ip) + domain (controller.yaml), write-if-changed +
reload. Loop (7th daemon goroutine) tracks DHCP IP changes per provisioned
guest. --selftest=lanresolver. FELHOM_DNSMASQ sudoers. Spiked live on felhom-pve.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lockstep with felhom-controller v0.40.0. The agent now bakes a v2 bootstrap.json
carrying only what the controller needs to PULL its config from the hub:
customer.id + hub.url + hub.retrieval_password + the per-guest local_api. Stops
baking the agent's host hub key/host_id (and customer name/domain/email) into the
guest — the controller gets the customer-scoped key from the hub pull.
- internal/provision/doc.go: SchemaV2; DocCustomer{id}; DocHub{url,retrieval_password}.
- backhalf.go: render v2; require customer.id + hub.url + hub.retrieval_password.
- cmd/.../main.go --selftest=provision: new required -hub-password flag; stop
baking APIKey/HostID; -customer-domain/name/email accepted but not baked.
- configs/build-golden.sh: default CONTROLLER_IMAGE off stale :v0.35.0 -> :0.40.0.
- doc_test.go/backhalf_test.go updated to v2 shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Identity escrow wraps {tunnel_token,pbs_token} under the SAME R via age
(scrypt+ChaCha20-Poly1305), reusing the K-escrow pty; wrong R fails closed.
escrow.Create optionally emits the identity blob; escrow-create uploads it +
the non-secret directive; identity-consume recovers it (R by hand, never
logged). K-escrow + 10C Consume untouched. Closes slice 10 with hub v0.11.0;
operator-side rotation model (hub holds no Cloudflare write-power).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add escrow.Consume(blob, R, expectedFingerprint, keyDest): Unwrap -> fingerprint
gate -> atomic 0600 install. Bakes in the spike findings — wrong R fails closed
(no write), the fingerprint gate runs BEFORE any restore (no install on
mismatch), the input blob is read-only (retryable), K is never mutated, R/key
bytes never logged. Zero-knowledge holds: the hub serves all but R (by hand).
--selftest=escrow-consume invokes the real path live. Agent-only; no hub change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A destructive op runs ONLY on a pinned-key-verified, nonce-fresh, in-window,
host-bound, durable-id-bound operator signature. New cmd/felhom-opsign signs
canonical OpBlobs offline via ssh-keygen -Y sign (hardware-ready); the signing
key is never in the hub or agent. New internal/signedjobs runner verifies each
queued blob through the gate and only on all-pass runs the WipeExecutor, which
re-resolves the DURABLE device id + re-inspects (8C) before mkfs — closing the
8C data-bearing-wipe pending_signature gap. New storage durable-device
resolution; authz.CanonicalBlob promoted to production. Real-crypto tests assert
valid executes and forged/replay/expired/retarget/non-pinned are rejected
(executor never called).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The control envelope becomes live: the agent caches the hub's desired-state +
generation and re-fetches GET /hosts/{id}/desired-state only when the
generation advances. A new internal/desired Syncer maps the wire shape into a
reconcile.CachingProvider feeding the engine; benign deltas reconcile, an
explicit guest decommission is gated pending_signature (exec is 10B). Adds the
DesiredStateResponse/WireDesiredState wire types + Client.FetchDesiredState +
the loop EnvelopeObserver seam. Cross-repo golden (envelope + desired-state)
byte-identical with the hub.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a host-wide, token-authed GET /host/metrics local-API endpoint that
re-serves the slice-4 collector's host + per-storage view to the customer
(the de-privileged controller can't read the host itself). Add the one new
collector — CPU/chassis temperature via sysfs hwmon/thermal-zones, graceful-
null — to the shared HostMetrics struct, so the hub report carries cpu_temp_c
too. Cross-repo host-report golden updated byte-identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BackupRunner.BackupWithSnapshotHook tails the task log for the 'create storage
snapshot' marker (snapshot mode only) and fires onSnapshot once; localapi flips
/backup/status to 'snapshotted' before 'done' so the controller resumes early.
Phase 0 validated on PVE 9.2.2: marker confirmed, downtime ~24s->~1s (934MB).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
internal/localapi: real /backup/due (cadence; due when no successful backup or
newest older than backup.backup_cadence_seconds; false in-window after success;
failed doesn't count) + /backup/status phases (idle|running|done|failed + job
id) + POST /backup single-flight with job id. Drives the controller quiesce loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
internal/escrow: zero-knowledge escrow creation. R = 10 EFF-wordlist words
(crypto/rand, ~129 bits); wrap K under R via PBS-native key change-passphrase
driven over a stdlib pty (x/sys/unix; output discarded so R can't leak, F-A2);
self-verify the blob recovers K (fingerprint match) before shipping. Opt-in (b)
R-wrapped offline copy + (a) raw paperkey. Live K is byte-unchanged (operates on
a copy). --selftest=escrow-create (-storage/-paperkey/-offline/-upload). Posture
config field (zero_knowledge default). PBSEncKeyPath helper. Grounded by the
escrow spike findings.
Tests: R entropy>=128/format/uniqueness; integration round-trip (wrap->unwrap
fingerprint match, wrong-R fails, K byte-unchanged) guarded to linux+pbc.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared front half of provision and guest-loss DR as a journaled reconcile job
(internal/reconcile/bringup.go), mirroring the restore-test's crash-safety but keeping
the guest on success and applying a scenario-specific identity policy. Agent-only; no
hub/wire change. Grounded by the slice-7 bring-up spike (commit 3342993): F1/F3/F4.
- RunBringUp: restore -> reset identity -> size -> attach mounts -> start link-up;
verdict is liveness (waitRunning), success KEEPS the guest.
- identity policy: provision = fresh MAC (net0 sans hwaddr -> PVE regen) + hostname,
host-side; machine-id/host-keys regenerate guest-side (systemd + baked golden unit).
dr_guest_loss = preserve continuity (keep hostname; keep MAC unless KeepMAC=false).
- compensating rollback: mid-flight failure destroys the just-created guest
(SameTxnCreated provenance, gated); new Rollback journal flag + Recover.recoverBringUp
reap a half-built guest from a crash.
- F4: coalesced config PUT + bounded retry on the transient PVE config-lock 500 only.
- --selftest=bring-up (mode/archive/vmid/hostname/keep).
- configs/build-golden.sh: validated golden recipe incl. the F3 first-boot host-key unit.
- doc-03 §9 + identity-reset settled/implemented.
Deferred (stated): provisioning back half -> slice 8; host-loss DR + escrow consumption
and the BringUpSpec source (hub desired-state) -> slice 10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the two new RestoreTest wire keys to the shared host-report golden and exercises
them in the bidirectional key-set contract test. Kept byte-identical with felhom-hub's
copy (sha256 e6999d77...). Testdata/tests only — the v0.7.0 binary is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the crying-wolf false-fail surfaced by the live hub-enrollment runbook:
PVE's guest-start task exits "WARNINGS: 1" for the benign systemd-nesting
advisory, and WaitTask treated any non-OK exitstatus as failure, so the verdict
was decided by an advisory exit code before the real boot check ran. Every
modern-distro restore-test reported pass:false.
- proxmox.WaitOptions.AllowWarnings (opt-in; default keeps all callers strict)
- restore-test start step accepts warnings, surfaces them, verdict stays waitRunning
- RestoreTestResult.StartWarnings/.WarningsRecognized + version-free "enable
nesting" recognizer (can't rot back at systemd 258+); GuestAPI.TaskLogTail
- hub.RestoreTest.warnings/.warnings_recognized wire fields (consumed by hub v0.7.5)
- scheduler logs clean / passed-with-recognized / passed-with-unrecognized warnings
- tests: WaitTask warnings matrix; restore-test pass/fail-on-liveness; version-free
regression guard (systemd 256-300)
Single agent bump 0.6.0 -> 0.7.0 covering the agent half of both task phases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live PBS runbook surfaced two gaps: (1) PBS verify defaults to ignore-verified=true and
SKIPS already-verified snapshots, so corruption after the first verify is never caught —
the agent's integrity check now POSTs ignore-verified=false to actually re-read+re-check.
(2) restore-test source_tier was hardcoded 'local'; now derived from the source storage
type ('pbs' for a PBS datastore). Adds a form-POST path to the PBS client.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spike-proven that backup/restore-to-PBS reuse Phase A unchanged; the only new code is
the verify capability, a small PBS-API client, and PBSSnapshot reporting.
- internal/pbs: fingerprint-pinned, token-authed PBS-API client (Verify/Snapshots/
TaskStatus, node-from-UPID; secret read from /etc/pve/priv/storage/<id>.pw at runtime,
never logged) + the verify maintenance loop (own cadence, default 6h, NOT gated/journaled,
like the watchdog) + SnapshotStore.
- hub: PBSSnapshot filled (namespace/type/id/time/size/owner/protected/encrypted/
verify_state/verify_upid); PBSReporter collector seam; cross-repo golden + bidirectional
key-set tests; hub handler parses pbs_snapshots + logs a failed-verify WARN.
- backup: report the ACTUAL vzdump mode (parsed from the task log; PVE may downgrade
snapshot->stop). proxmox.Storage.Username. config PBSVerifyCadence/secret-dir.
--selftest=pbs-verify. Backup/restore-to-PBS unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The slice-6 restore-test teardown failed live: DELETE /lxc/{vmid} with a form body
returns HTTP 501 'Unexpected content for method DELETE', and a booted scratch needs
force=1. Move purge/destroy-unreferenced-disks/force to the query string, send no body.
Regression test locks the contract.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin