DR recipe slice (agent half), grounded in SPIKE-dr-recipe-2026-06-16. Additive
`dr_recipe` host-report section = the non-secret reconstruction scaffolding the
operator must rebuild before PBS bytes can land. Built by pure
BuildDRRecipeHostHalf from facts the report already collects (no new reads):
guests[] sizing, drives[] (user-data by durable_id/role/mount/intent),
pve_storage[] (storage.cfg), pbs coordinates.
BOUNDARY (Phase-1 lesson): every field is an identifier/intent/size/coordinate —
never a key/password/token/hash/ENC:. PBS key stays in escrow; restic password
stays in escrow; the recipe names only the coordinates the restore targets.
Tests: BuildDRRecipeHostHalf selection, NoPBS, NoSecrets (boundary mirror),
dr_recipe key-set in the cross-repo golden contract test. recipe_version=1,
ignore-unknown on read.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The skip-if-mounted optimization defeated the actual root cause: felhom-usb is
mounted now but its unit is `disabled`, so a host reboot would not auto-mount
it. ReassertEnrolledMounts now skips ONLY the durable steady state (mounted AND
enabled) via the pure shouldReassertMount; a mounted-but-disabled unit is
re-asserted so enable --now re-creates the wants-symlink. Enabled-state read by
privilege-free Lstat of the multi-user.target.wants symlink (unitEnabled) — no
systemctl is-enabled subprocess, no new sudoers entry.
Tests: TestShouldReassertMount (4 combos), TestUnitEnabled (wants-symlink).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
TASK A — close out the reboot story (agent half). Root cause (pinned live on
felhom-pve): an enrolled .mount unit left `disabled` by a prior detach never
auto-mounts at boot, and kernel re-enumeration can move a drive's node
(/dev/sdb->sdc). Fix re-asserts every enrolled mount by filesystem UUID at
startup + on the periodic tick.
- ResolveStorageDevice: resolve uuid:<fs-uuid> -> current /dev node via
/dev/disk/by-uuid (never a cached node); errors if absent.
- parseFelhomMountUnit: pure inverse of renderMountUnit (marker-gated).
- (*SudoHostOps).ReassertEnrolledMounts: re-run EnsureMount (enable --now) for
any enrolled unit not in /proc/mounts; idempotent, skips mounted/absent.
- main.go: runs before ReassertGuestBinds at startup + on the 20s tick.
- tests (Linux, seam=device resolution): letter-move tolerated (sdb->sdc) +
red-proof companion, absent/scheme rejection, render->parse round-trip.
TASK A2 verdict: enrolling a NEW drive does NOT need an LXC restart — the path
lands on the live AttachDrive (shared parent, named live slots), not RebootGuest.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Identity escrow wraps {tunnel_token,pbs_token} under the SAME R via age
(scrypt+ChaCha20-Poly1305), reusing the K-escrow pty; wrong R fails closed.
escrow.Create optionally emits the identity blob; escrow-create uploads it +
the non-secret directive; identity-consume recovers it (R by hand, never
logged). K-escrow + 10C Consume untouched. Closes slice 10 with hub v0.11.0;
operator-side rotation model (hub holds no Cloudflare write-power).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add escrow.Consume(blob, R, expectedFingerprint, keyDest): Unwrap -> fingerprint
gate -> atomic 0600 install. Bakes in the spike findings — wrong R fails closed
(no write), the fingerprint gate runs BEFORE any restore (no install on
mismatch), the input blob is read-only (retryable), K is never mutated, R/key
bytes never logged. Zero-knowledge holds: the hub serves all but R (by hand).
--selftest=escrow-consume invokes the real path live. Agent-only; no hub change.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A destructive op runs ONLY on a pinned-key-verified, nonce-fresh, in-window,
host-bound, durable-id-bound operator signature. New cmd/felhom-opsign signs
canonical OpBlobs offline via ssh-keygen -Y sign (hardware-ready); the signing
key is never in the hub or agent. New internal/signedjobs runner verifies each
queued blob through the gate and only on all-pass runs the WipeExecutor, which
re-resolves the DURABLE device id + re-inspects (8C) before mkfs — closing the
8C data-bearing-wipe pending_signature gap. New storage durable-device
resolution; authz.CanonicalBlob promoted to production. Real-crypto tests assert
valid executes and forged/replay/expired/retarget/non-pinned are rejected
(executor never called).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The control envelope becomes live: the agent caches the hub's desired-state +
generation and re-fetches GET /hosts/{id}/desired-state only when the
generation advances. A new internal/desired Syncer maps the wire shape into a
reconcile.CachingProvider feeding the engine; benign deltas reconcile, an
explicit guest decommission is gated pending_signature (exec is 10B). Adds the
DesiredStateResponse/WireDesiredState wire types + Client.FetchDesiredState +
the loop EnvelopeObserver seam. Cross-repo golden (envelope + desired-state)
byte-identical with the hub.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add a host-wide, token-authed GET /host/metrics local-API endpoint that
re-serves the slice-4 collector's host + per-storage view to the customer
(the de-privileged controller can't read the host itself). Add the one new
collector — CPU/chassis temperature via sysfs hwmon/thermal-zones, graceful-
null — to the shared HostMetrics struct, so the hub report carries cpu_temp_c
too. Cross-repo host-report golden updated byte-identical.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Agent half of slice-7 close-out finalized: cross-repo wire-contract test
(escrowUploadRequest mirrors the hub), v0.9.0 CHANGELOG entry + REPORT + CLAUDE
version line. No secrets.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live validation on demo-felhom caught the felhom-regen-hostkeys unit failing with
203/EXEC: ExecStart was /usr/sbin/ssh-keygen but on Debian 13 ssh-keygen is at
/usr/bin/ssh-keygen. Fixed build-golden.sh, rebuilt the golden, re-validated — host
keys now regenerate on first boot by the baked unit (agent issues no ssh-keygen).
All three live scenarios green: provision (fresh MAC, host keys via unit, machine-id,
Docker, DHCP), dr (continuity: hostname + host keys preserved), Recover (killed
mid-restore -> orphan rolled back idempotently). REPORT + CHANGELOG updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared front half of provision and guest-loss DR as a journaled reconcile job
(internal/reconcile/bringup.go), mirroring the restore-test's crash-safety but keeping
the guest on success and applying a scenario-specific identity policy. Agent-only; no
hub/wire change. Grounded by the slice-7 bring-up spike (commit 3342993): F1/F3/F4.
- RunBringUp: restore -> reset identity -> size -> attach mounts -> start link-up;
verdict is liveness (waitRunning), success KEEPS the guest.
- identity policy: provision = fresh MAC (net0 sans hwaddr -> PVE regen) + hostname,
host-side; machine-id/host-keys regenerate guest-side (systemd + baked golden unit).
dr_guest_loss = preserve continuity (keep hostname; keep MAC unless KeepMAC=false).
- compensating rollback: mid-flight failure destroys the just-created guest
(SameTxnCreated provenance, gated); new Rollback journal flag + Recover.recoverBringUp
reap a half-built guest from a crash.
- F4: coalesced config PUT + bounded retry on the transient PVE config-lock 500 only.
- --selftest=bring-up (mode/archive/vmid/hostname/keep).
- configs/build-golden.sh: validated golden recipe incl. the F3 first-boot host-key unit.
- doc-03 §9 + identity-reset settled/implemented.
Deferred (stated): provisioning back half -> slice 8; host-loss DR + escrow consumption
and the BringUpSpec source (hub desired-state) -> slice 10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fixes the crying-wolf false-fail surfaced by the live hub-enrollment runbook:
PVE's guest-start task exits "WARNINGS: 1" for the benign systemd-nesting
advisory, and WaitTask treated any non-OK exitstatus as failure, so the verdict
was decided by an advisory exit code before the real boot check ran. Every
modern-distro restore-test reported pass:false.
- proxmox.WaitOptions.AllowWarnings (opt-in; default keeps all callers strict)
- restore-test start step accepts warnings, surfaces them, verdict stays waitRunning
- RestoreTestResult.StartWarnings/.WarningsRecognized + version-free "enable
nesting" recognizer (can't rot back at systemd 258+); GuestAPI.TaskLogTail
- hub.RestoreTest.warnings/.warnings_recognized wire fields (consumed by hub v0.7.5)
- scheduler logs clean / passed-with-recognized / passed-with-unrecognized warnings
- tests: WaitTask warnings matrix; restore-test pass/fail-on-liveness; version-free
regression guard (systemd 256-300)
Single agent bump 0.6.0 -> 0.7.0 covering the agent half of both task phases.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Live: backup of 9001 (crash-consistent, 934MiB), restore-test (restore→boot→verify→
teardown, gate-benign, no leak), AND a real crash-recovery proof — a DestroyLXC DELETE
bug leaked scratch 990000, the fix landed, and Recover reaped the leaked guest on the
next run before a clean test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validated v0.5.1 live: observe (durable_id=fs-UUID, class_hint slow, SMART PASSED via
USB-SATA bridge, lvs thin-pool metadata), durable_id stability, and the watchdog cycle —
mount-drop (W2) and reversible device soft-remove/return (W3): detect in ~3s, re-mount
by stable by-UUID systemd unit, and NO re-mount while the device is truly gone. Both
v0.5.1 fixes confirmed live. Physical H1/H2 + end-to-end hub reflection flagged as not done.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fill the slice-3 storage_targets stub and add the fast-poll storage watchdog.
Read-only this phase; the host-root surface (mounts/SMART/grow/destructive gate)
is Phase B. Hub-owned desired manifest is slice 10, so reconcile against it is
built-but-unfed.
- internal/storage: StorageTarget wire contract, durable_id derivation per type,
HostReader seam (procfs/sysfs, root-free), Observer (storage_targets from
ListStorage/NodeStorage + host reads, lvmthin thin-pool fill), and the watchdog
(third daemon goroutine; debounced out-of-band report on a known target's
attach/disconnect transition).
- proxmox.Storage: additive parse-only config fields (durable_id sources).
- collector StorageObserver seam; Loop.SetTrigger out-of-band report; daemon runs
the watchdog as a third goroutine; StorageConfig knobs.
- cross-repo golden kept byte-identical with felhom.eu/hub; bidirectional key-set test.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The security core of slice 4: hub-supplied intent is no longer trusted for
destructive change. The gate fronts the per-guest queue's executor, so every
mutation passes it. Reuses internal/authz for all crypto (surface untouched).
- Classifier (doc 03 §4): benign vs destructive by provenance + data-bearing-
ness, NOT by verb. Destroy/overwrite of customer data is destructive unless
agent-internal provenance (same-journaled-txn create, or agent-tagged scratch)
makes it benign — and that provenance is journal-recorded, NEVER hub-sourced.
Unknown op class fails safe to destructive.
- Reversibility gate: benign -> allowed unsigned; destructive -> requires a
verified, role-scoped, action-bound operator signature, else pending_signature
and never executed. Every decision audited (signal, never the guard).
- Signed-op consuming layer over authz.Verifier.Verify (locked pipeline
untouched): role-scoping (doc 04 §4 — recovery=rotation only, operational=
ordinary destructive + planned rotation) + op-to-action binding (op+host+
guest+params must match the gated action).
- Signed-job orchestration: idempotency dedupe by nonce + journal-wrapped
execution via an injected DestructiveExecutor (nil this slice — inert).
- Crash recovery (Note 1): Engine.Recover consumes the journal InFlight() set at
startup (resume-or-rollback) — covers an op that crashed after the POST and
before its terminal record, which idempotency dedupe alone cannot. Added
TaskStatusOnce to the GuestAPI seam. Wired into daemon startup.
- Note 2: memory comparison canonicalized to MiB (desiredMemoryMiB) so a
non-MiB-aligned MemoryBytes converges in one pass, not perpetual drift.
- Daemon: builds the verifier from config signers (none = nil verifier, the
common slice-4 state), the gate (+SlogAudit), runs Recover before mutating.
Adversarial matrix proven against the REAL authz.Verifier with in-test-minted
SSHSIGs (framing replicated in reconcile's test binary; authz untouched, no
signing added to the verify-only package): unsigned job + unsigned desired-state
delta -> pending_signature; unknown signer/expired/replay-across-restart/wrong
host -> typed authz rejections; wrong guest/op/params -> binding_mismatch;
recovery key on ordinary destructive -> role_denied; hub-supplied scratch tag
ignored -> refused; valid+role+target+fresh nonce -> accepted then replay
rejected. Full module race-clean + vet-clean on the Linux build server.
Inert this slice: no destructive deltas served until slice 10; the destructive
path is classified, gated, and tested but not wired to live execution.
CHECKPOINT: Phase B complete (slice 4 done). Awaiting validation.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
New internal/reconcile package: the agent-side control core's structural half.
- Per-guest serializer Queue (doc 03 §10): the single choke point all mutation
sources funnel through; same-vmid serial in submit order, different vmids
parallel (cond-var FIFO lanes).
- Desired-state model + DesiredProvider seam; EmptyProvider is the only live
source at slice 4 (no hub serving until slice 10) so the live engine computes
an empty action set and performs zero mutations.
- Normalization layer (FieldNormalizers): normalized desired-vs-actual so
Proxmox round-trip quirks don't read as drift. normDesc promoted out of
main.go to reconcile.NormDescription; selftest uses the shared helper.
- Plan (pure diff): minimal benign action set (Start/Stop/SetConfig) for guests
in both desired and actual; provision/destroy out of scope here.
- Engine: dispatches onto the shared queue; honors the dual-mode SetConfig
contract (UPID -> WaitTask; empty UPID -> synchronous success).
- Durable op journal + idempotency store (mirrors authz.FileNonceStore):
in-flight task ids for crash detection + AlreadyApplied dedupe across restart.
- Wired into runDaemon alongside the hub loop, sharing the queue; runs cleanly
with no desired state and no signers.
Full module race-clean and vet-clean on the Linux build server.
CHECKPOINT: Phase A only. Awaiting validation before Phase B (the reversibility
gate + signed-op consuming layer, landing v0.4.0).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Append a reversible SetConfig write+revert to runSelftestTask: read
GuestConfig, write a `description` marker, verify it landed, restore the
original (or delete if absent), verify the restore. Handles PVE's dual-mode
SetConfig return (empty UPID = synchronous; UPID = WaitTask+assert OK).
Live self-gate PASSED on demo-felhom / guest 9999. Findings:
- LXC `description` write is synchronous (empty UPID) — dual-mode modeling
confirmed; empty string is success, not an error.
- PVE appends a trailing newline to `description` on read; slice-4 reconcile
must normalize description comparisons (hence normDesc helper).
First live exercise of the VM.Config.* privilege cluster. Standing operator
token rotated during the run; new secret stored out-of-band, not in the repo.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Also overwrite REPORT.md with the live --selftest=task validation on demo-felhom
(snapshot/rollback/delete on guest 9999, exitstatus=OK under the felhom-agent@pve
privsep token; slice-1 mutating-ops gap closed, slice 4 unblocked). No version bump.
Token secret stored out-of-band, not committed.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
internal/hub: the agent's first daemon — a periodic read-only host-report POSTed to
the hub (the heartbeat; no separate ping).
- HostReport wire contract (shared field-for-field with the hub ingest): host
metrics, guests (vmid + spec), cloudflared status; storage/backups/restore-tests/
pbs/audit collections DEFINED but emitted empty (slices 5/6 fill).
- Collector over a read-only proxmoxReader (adapted to the real proxmox surface;
no proxmox changes) + a CloudflaredProber. Partial-failure: NodeStatus fail = hard
(skip POST); per-guest GuestConfig fail = status "unknown", still report.
- Client: Bearer-auth POST, standard TLS (system roots / optional ca_file), typed
TransportError/HTTPError, token never in errors.
- Loop: immediate first report, adopt hub poll_interval (clamp [60,3600]), resilient
to collect/report errors, clean ctx-cancel shutdown.
- ControlEnvelope: only poll_interval_seconds acted on; blocked/desired_generation/
has_signed_ops parsed-but-ignored (slice 4).
- config: HubConfig + FELHOM_AGENT_HUB_* overlay + mode-aware HubConfig.Validate +
WithDefaults + hub-key redaction; example config updated.
- main: no-selftest mode is now the daemon; added --selftest=hub. Version -> 0.3.0.
Tests: report serialization, client (incl. token-redaction), collector partial-
failure, loop continuation+interval adoption, config. internal/proxmox + internal/
authz untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin