Live validation on demo-felhom caught the felhom-regen-hostkeys unit failing with
203/EXEC: ExecStart was /usr/sbin/ssh-keygen but on Debian 13 ssh-keygen is at
/usr/bin/ssh-keygen. Fixed build-golden.sh, rebuilt the golden, re-validated — host
keys now regenerate on first boot by the baked unit (agent issues no ssh-keygen).
All three live scenarios green: provision (fresh MAC, host keys via unit, machine-id,
Docker, DHCP), dr (continuity: hostname + host keys preserved), Recover (killed
mid-restore -> orphan rolled back idempotently). REPORT + CHANGELOG updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared front half of provision and guest-loss DR as a journaled reconcile job
(internal/reconcile/bringup.go), mirroring the restore-test's crash-safety but keeping
the guest on success and applying a scenario-specific identity policy. Agent-only; no
hub/wire change. Grounded by the slice-7 bring-up spike (commit 3342993): F1/F3/F4.
- RunBringUp: restore -> reset identity -> size -> attach mounts -> start link-up;
verdict is liveness (waitRunning), success KEEPS the guest.
- identity policy: provision = fresh MAC (net0 sans hwaddr -> PVE regen) + hostname,
host-side; machine-id/host-keys regenerate guest-side (systemd + baked golden unit).
dr_guest_loss = preserve continuity (keep hostname; keep MAC unless KeepMAC=false).
- compensating rollback: mid-flight failure destroys the just-created guest
(SameTxnCreated provenance, gated); new Rollback journal flag + Recover.recoverBringUp
reap a half-built guest from a crash.
- F4: coalesced config PUT + bounded retry on the transient PVE config-lock 500 only.
- --selftest=bring-up (mode/archive/vmid/hostname/keep).
- configs/build-golden.sh: validated golden recipe incl. the F3 first-boot host-key unit.
- doc-03 §9 + identity-reset settled/implemented.
Deferred (stated): provisioning back half -> slice 8; host-loss DR + escrow consumption
and the BringUpSpec source (hub desired-state) -> slice 10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spike-proven that backup/restore-to-PBS reuse Phase A unchanged; the only new code is
the verify capability, a small PBS-API client, and PBSSnapshot reporting.
- internal/pbs: fingerprint-pinned, token-authed PBS-API client (Verify/Snapshots/
TaskStatus, node-from-UPID; secret read from /etc/pve/priv/storage/<id>.pw at runtime,
never logged) + the verify maintenance loop (own cadence, default 6h, NOT gated/journaled,
like the watchdog) + SnapshotStore.
- hub: PBSSnapshot filled (namespace/type/id/time/size/owner/protected/encrypted/
verify_state/verify_upid); PBSReporter collector seam; cross-repo golden + bidirectional
key-set tests; hub handler parses pbs_snapshots + logs a failed-verify WARN.
- backup: report the ACTUAL vzdump mode (parsed from the task log; PVE may downgrade
snapshot->stop). proxmox.Storage.Username. config PBSVerifyCadence/secret-dir.
--selftest=pbs-verify. Backup/restore-to-PBS unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
internal/hub: the agent's first daemon — a periodic read-only host-report POSTed to
the hub (the heartbeat; no separate ping).
- HostReport wire contract (shared field-for-field with the hub ingest): host
metrics, guests (vmid + spec), cloudflared status; storage/backups/restore-tests/
pbs/audit collections DEFINED but emitted empty (slices 5/6 fill).
- Collector over a read-only proxmoxReader (adapted to the real proxmox surface;
no proxmox changes) + a CloudflaredProber. Partial-failure: NodeStatus fail = hard
(skip POST); per-guest GuestConfig fail = status "unknown", still report.
- Client: Bearer-auth POST, standard TLS (system roots / optional ca_file), typed
TransportError/HTTPError, token never in errors.
- Loop: immediate first report, adopt hub poll_interval (clamp [60,3600]), resilient
to collect/report errors, clean ctx-cancel shutdown.
- ControlEnvelope: only poll_interval_seconds acted on; blocked/desired_generation/
has_signed_ops parsed-but-ignored (slice 4).
- config: HubConfig + FELHOM_AGENT_HUB_* overlay + mode-aware HubConfig.Validate +
WithDefaults + hub-key redaction; example config updated.
- main: no-selftest mode is now the daemon; added --selftest=hub. Version -> 0.3.0.
Tests: report serialization, client (incl. token-redaction), collector partial-
failure, loop continuation+interval adoption, config. internal/proxmox + internal/
authz untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stand up the felhom-agent project (module gitea.dooplex.hu/admin/felhom-agent,
binary felhom-agent) and the internal/proxmox package: the typed library every
other agent module calls to talk to Proxmox.
- API-first Client (hand-rolled REST over net/http, PVEAPIToken auth) with typed
read ops (version/nodes/status/lxc/config/storage) and async mutating ops
(restore/vzdump/snapshot/rollback/delete-snapshot/setconfig/start/stop), each
returning a UPID. WaitTask polls task status until stopped and asserts
exitstatus OK (authz can surface at task exec, not the POST — phase1-2 §1.3).
- Fenced Privileged (root-CLI) backend for the THREE proven exceptions only
(keyctl pct create, USB mount/fstab, SMART/sensors); each cites why it can't be
the API. Fence is structural (Client never shells out, Privileged never HTTPs)
and asserted in routing_test.go.
- TLS: SHA-256 leaf-cert pinning or CA file; insecure mode explicit + off by
default. No blanket verification disable.
- 403 -> privilege-named APIError; failed task -> privilege-named TaskError.
- JSON config + env overrides (token never logged); slog logging.
- cmd/felhom-agent --selftest (read-only health report) + gated --selftest=task
(reversible snapshot/rollback/delete exercise of WaitTask). No daemon loop yet.
- Types grounded in the spike findings and exact JSON shapes captured live from
demo-felhom (PVE 9.2.2). Unit tests use a mock transport + runner.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin