build-golden.sh bootstrap makes /mnt a shared mount and binds it :rslave into the
controller container so enrolled data drives (and P3 self-heal remounts) propagate
in. Scoped to /mnt (Model A: only felhom-data-namespace mounts). Spike-proven.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
POST /disks/guest-attach binds an enrolled drive's felhom-data namespace into
the guest (Model A: felhom-data is the bind source mounted at /mnt/<name>, so
only Felhom's namespace crosses in). GuestBinder does mkdir+chown(100000)+pct set
(RW bind) via the fenced runner. Idempotent, free-slot selection, path-validated.
Spike-proven on 9201. Pairs with controller P2C + golden /mnt:rslave (P2B).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
handleDiskEject now resolves the authoritative role of the storage at `where`
and refuses 403 (no Unmount) unless it is user-data. Fails safe to protected on
ambiguity. Adds roleForMountPath + an injectable HostReader seam for testability.
TestEject_RoleGated asserts protected mounts are refused with no Unmount.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
localapi.DiskInfo gains durable_id (from StorageTarget.DurableID, "uuid:<fs-uuid>"
for usb/local-dir). The de-privileged controller can't read a device's fs UUID
but assign mounts strictly by UUID — this read-only field is the only way it
learns the assign key. No new privilege, no gate change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Host-side dnsmasq the agent manages so LAN clients reach their guest directly
(same hostname + real wildcard cert, no Cloudflare hairpin). Renders local=/
+address=/ per customer (AAAA->NODATA via authoritative zone, wildcard A ->
live guest IP), forwards everything else. Manager ensures dnsmasq+base config,
discovers guest IP (pct exec ip) + domain (controller.yaml), write-if-changed +
reload. Loop (7th daemon goroutine) tracks DHCP IP changes per provisioned
guest. --selftest=lanresolver. FELHOM_DNSMASQ sudoers. Spiked live on felhom-pve.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validate the customer.id parsed from bootstrap.json to a DNS-safe label (reject
spaces/slashes/leading-dash) and pass it via a quoted array, so a malformed or
hostile id can't smuggle extra docker flags (e.g. --privileged) into the
controller's de-privileged `docker run`. Empty/invalid → no --hostname.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build-golden.sh: same-path /opt/docker/stacks host bind (Section-G fix, breaks
all bind-mounted stacks without it) + --hostname <customer-id> from bootstrap.json
(portable sed parse, no jq) + bake the 3 pinned public base-infra images with a
manifest-inspect hard gate. Provision --selftest defaults -hostname to the
DNS-safe-sanitized customer-id so the CT/LXC is named meaningfully.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lockstep with felhom-controller v0.40.0. The agent now bakes a v2 bootstrap.json
carrying only what the controller needs to PULL its config from the hub:
customer.id + hub.url + hub.retrieval_password + the per-guest local_api. Stops
baking the agent's host hub key/host_id (and customer name/domain/email) into the
guest — the controller gets the customer-scoped key from the hub pull.
- internal/provision/doc.go: SchemaV2; DocCustomer{id}; DocHub{url,retrieval_password}.
- backhalf.go: render v2; require customer.id + hub.url + hub.retrieval_password.
- cmd/.../main.go --selftest=provision: new required -hub-password flag; stop
baking APIKey/HostID; -customer-domain/name/email accepted but not baked.
- configs/build-golden.sh: default CONTROLLER_IMAGE off stale :v0.35.0 -> :0.40.0.
- doc_test.go/backhalf_test.go updated to v2 shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Identity escrow wraps {tunnel_token,pbs_token} under the SAME R via age
(scrypt+ChaCha20-Poly1305), reusing the K-escrow pty; wrong R fails closed.
escrow.Create optionally emits the identity blob; escrow-create uploads it +
the non-secret directive; identity-consume recovers it (R by hand, never
logged). K-escrow + 10C Consume untouched. Closes slice 10 with hub v0.11.0;
operator-side rotation model (hub holds no Cloudflare write-power).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add escrow.Consume(blob, R, expectedFingerprint, keyDest): Unwrap -> fingerprint
gate -> atomic 0600 install. Bakes in the spike findings — wrong R fails closed
(no write), the fingerprint gate runs BEFORE any restore (no install on
mismatch), the input blob is read-only (retryable), K is never mutated, R/key
bytes never logged. Zero-knowledge holds: the hub serves all but R (by hand).
--selftest=escrow-consume invokes the real path live. Agent-only; no hub change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A destructive op runs ONLY on a pinned-key-verified, nonce-fresh, in-window,
host-bound, durable-id-bound operator signature. New cmd/felhom-opsign signs
canonical OpBlobs offline via ssh-keygen -Y sign (hardware-ready); the signing
key is never in the hub or agent. New internal/signedjobs runner verifies each
queued blob through the gate and only on all-pass runs the WipeExecutor, which
re-resolves the DURABLE device id + re-inspects (8C) before mkfs — closing the
8C data-bearing-wipe pending_signature gap. New storage durable-device
resolution; authz.CanonicalBlob promoted to production. Real-crypto tests assert
valid executes and forged/replay/expired/retarget/non-pinned are rejected
(executor never called).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The control envelope becomes live: the agent caches the hub's desired-state +
generation and re-fetches GET /hosts/{id}/desired-state only when the
generation advances. A new internal/desired Syncer maps the wire shape into a
reconcile.CachingProvider feeding the engine; benign deltas reconcile, an
explicit guest decommission is gated pending_signature (exec is 10B). Adds the
DesiredStateResponse/WireDesiredState wire types + Client.FetchDesiredState +
the loop EnvelopeObserver seam. Cross-repo golden (envelope + desired-state)
byte-identical with the hub.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a host-wide, token-authed GET /host/metrics local-API endpoint that
re-serves the slice-4 collector's host + per-storage view to the customer
(the de-privileged controller can't read the host itself). Add the one new
collector — CPU/chassis temperature via sysfs hwmon/thermal-zones, graceful-
null — to the shared HostMetrics struct, so the hub report carries cpu_temp_c
too. Cross-repo host-report golden updated byte-identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BackupRunner.BackupWithSnapshotHook tails the task log for the 'create storage
snapshot' marker (snapshot mode only) and fires onSnapshot once; localapi flips
/backup/status to 'snapshotted' before 'done' so the controller resumes early.
Phase 0 validated on PVE 9.2.2: marker confirmed, downtime ~24s->~1s (934MB).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
internal/localapi: real /backup/due (cadence; due when no successful backup or
newest older than backup.backup_cadence_seconds; false in-window after success;
failed doesn't count) + /backup/status phases (idle|running|done|failed + job
id) + POST /backup single-flight with job id. Drives the controller quiesce loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Agent half of slice-7 close-out finalized: cross-repo wire-contract test
(escrowUploadRequest mirrors the hub), v0.9.0 CHANGELOG entry + REPORT + CLAUDE
version line. No secrets.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
internal/escrow: zero-knowledge escrow creation. R = 10 EFF-wordlist words
(crypto/rand, ~129 bits); wrap K under R via PBS-native key change-passphrase
driven over a stdlib pty (x/sys/unix; output discarded so R can't leak, F-A2);
self-verify the blob recovers K (fingerprint match) before shipping. Opt-in (b)
R-wrapped offline copy + (a) raw paperkey. Live K is byte-unchanged (operates on
a copy). --selftest=escrow-create (-storage/-paperkey/-offline/-upload). Posture
config field (zero_knowledge default). PBSEncKeyPath helper. Grounded by the
escrow spike findings.
Tests: R entropy>=128/format/uniqueness; integration round-trip (wrap->unwrap
fingerprint match, wrong-R fails, K byte-unchanged) guarded to linux+pbc.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live validation on demo-felhom caught the felhom-regen-hostkeys unit failing with
203/EXEC: ExecStart was /usr/sbin/ssh-keygen but on Debian 13 ssh-keygen is at
/usr/bin/ssh-keygen. Fixed build-golden.sh, rebuilt the golden, re-validated — host
keys now regenerate on first boot by the baked unit (agent issues no ssh-keygen).
All three live scenarios green: provision (fresh MAC, host keys via unit, machine-id,
Docker, DHCP), dr (continuity: hostname + host keys preserved), Recover (killed
mid-restore -> orphan rolled back idempotently). REPORT + CHANGELOG updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared front half of provision and guest-loss DR as a journaled reconcile job
(internal/reconcile/bringup.go), mirroring the restore-test's crash-safety but keeping
the guest on success and applying a scenario-specific identity policy. Agent-only; no
hub/wire change. Grounded by the slice-7 bring-up spike (commit 3342993): F1/F3/F4.
- RunBringUp: restore -> reset identity -> size -> attach mounts -> start link-up;
verdict is liveness (waitRunning), success KEEPS the guest.
- identity policy: provision = fresh MAC (net0 sans hwaddr -> PVE regen) + hostname,
host-side; machine-id/host-keys regenerate guest-side (systemd + baked golden unit).
dr_guest_loss = preserve continuity (keep hostname; keep MAC unless KeepMAC=false).
- compensating rollback: mid-flight failure destroys the just-created guest
(SameTxnCreated provenance, gated); new Rollback journal flag + Recover.recoverBringUp
reap a half-built guest from a crash.
- F4: coalesced config PUT + bounded retry on the transient PVE config-lock 500 only.
- --selftest=bring-up (mode/archive/vmid/hostname/keep).
- configs/build-golden.sh: validated golden recipe incl. the F3 first-boot host-key unit.
- doc-03 §9 + identity-reset settled/implemented.
Deferred (stated): provisioning back half -> slice 8; host-loss DR + escrow consumption
and the BringUpSpec source (hub desired-state) -> slice 10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Adds the two new RestoreTest wire keys to the shared host-report golden and exercises
them in the bidirectional key-set contract test. Kept byte-identical with felhom-hub's
copy (sha256 e6999d77...). Testdata/tests only — the v0.7.0 binary is unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the crying-wolf false-fail surfaced by the live hub-enrollment runbook:
PVE's guest-start task exits "WARNINGS: 1" for the benign systemd-nesting
advisory, and WaitTask treated any non-OK exitstatus as failure, so the verdict
was decided by an advisory exit code before the real boot check ran. Every
modern-distro restore-test reported pass:false.
- proxmox.WaitOptions.AllowWarnings (opt-in; default keeps all callers strict)
- restore-test start step accepts warnings, surfaces them, verdict stays waitRunning
- RestoreTestResult.StartWarnings/.WarningsRecognized + version-free "enable
nesting" recognizer (can't rot back at systemd 258+); GuestAPI.TaskLogTail
- hub.RestoreTest.warnings/.warnings_recognized wire fields (consumed by hub v0.7.5)
- scheduler logs clean / passed-with-recognized / passed-with-unrecognized warnings
- tests: WaitTask warnings matrix; restore-test pass/fail-on-liveness; version-free
regression guard (systemd 256-300)
Single agent bump 0.6.0 -> 0.7.0 covering the agent half of both task phases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live PBS runbook surfaced two gaps: (1) PBS verify defaults to ignore-verified=true and
SKIPS already-verified snapshots, so corruption after the first verify is never caught —
the agent's integrity check now POSTs ignore-verified=false to actually re-read+re-check.
(2) restore-test source_tier was hardcoded 'local'; now derived from the source storage
type ('pbs' for a PBS datastore). Adds a form-POST path to the PBS client.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spike-proven that backup/restore-to-PBS reuse Phase A unchanged; the only new code is
the verify capability, a small PBS-API client, and PBSSnapshot reporting.
- internal/pbs: fingerprint-pinned, token-authed PBS-API client (Verify/Snapshots/
TaskStatus, node-from-UPID; secret read from /etc/pve/priv/storage/<id>.pw at runtime,
never logged) + the verify maintenance loop (own cadence, default 6h, NOT gated/journaled,
like the watchdog) + SnapshotStore.
- hub: PBSSnapshot filled (namespace/type/id/time/size/owner/protected/encrypted/
verify_state/verify_upid); PBSReporter collector seam; cross-repo golden + bidirectional
key-set tests; hub handler parses pbs_snapshots + logs a failed-verify WARN.
- backup: report the ACTUAL vzdump mode (parsed from the task log; PVE may downgrade
snapshot->stop). proxmox.Storage.Username. config PBSVerifyCadence/secret-dir.
--selftest=pbs-verify. Backup/restore-to-PBS unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live: backup of 9001 (crash-consistent, 934MiB), restore-test (restore→boot→verify→
teardown, gate-benign, no leak), AND a real crash-recovery proof — a DestroyLXC DELETE
bug leaked scratch 990000, the fix landed, and Recover reaped the leaked guest on the
next run before a clean test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The slice-6 restore-test teardown failed live: DELETE /lxc/{vmid} with a form body
returns HTTP 501 'Unexpected content for method DELETE', and a booted scratch needs
force=1. Move purge/destroy-unreferenced-disks/force to the query string, send no body.
Regression test locks the contract.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validated v0.5.1 live: observe (durable_id=fs-UUID, class_hint slow, SMART PASSED via
USB-SATA bridge, lvs thin-pool metadata), durable_id stability, and the watchdog cycle —
mount-drop (W2) and reversible device soft-remove/return (W3): detect in ~3s, re-mount
by stable by-UUID systemd unit, and NO re-mount while the device is truly gone. Both
v0.5.1 fixes confirmed live. Physical H1/H2 + end-to-end hub reflection flagged as not done.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Surfaced preparing the live USB validation on demo-felhom:
- observe.go: an unmounted removable dir-storage no longer falls through to the ROOT fs
for its backing device/UUID — durable_id was becoming uuid:<root-uuid> (a DR mis-id that
would re-attach the wrong disk). Now derived only from the target's own mountpoint;
unmounted → no device + stable store:<name> durable_id. Removed containingMountDevice.
- watchdog.go: remember the fs-UUID observed while attached and backfill it onto the
re-mount target, so re-mount works even if the known-set cache refreshed mid-drop
(doc 03 §7 "sourced from the existing definition").
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin