docker-ce auto-starts on install with the default config; only a restart picks
up the just-written daemon.json. Same start-vs-restart trap the spike flagged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validation caught that Docker 29's default containerd-snapshotter keeps the
image store at /var/lib/containerd (on the rootfs), so mounting the data volume
at /var/lib/docker only moved named volumes — images (1.2G) stayed on the
rootfs, defeating the split. overlay2 stores images+overlay+volumes under
data-root = the data volume, which the split and the controller's statfs(/)
guard both require. Golden daemon.json now sets features.containerd-snapshotter
false + a driver guard.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build-golden.sh bakes a small OS rootfs + a dedicated /var/lib/docker volume
(mp0, backup=1) carrying the baked images, plus Docker log rotation. bringup.go
grows the golden's data volume to the per-customer target (DataVolGrowGB) and
emits backup=1 on data mounts (GuestMount.Backup) — closing the spike-B3 silent
DB-loss trap. CLI gains -rootfs-grow/-datavol-grow/-datavol-mount. New
RUNBOOK-provisioning-storage.md. Phase 2 = felhom-controller v0.58.0.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build-golden.sh bootstrap makes /mnt a shared mount and binds it :rslave into the
controller container so enrolled data drives (and P3 self-heal remounts) propagate
in. Scoped to /mnt (Model A: only felhom-data-namespace mounts). Spike-proven.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Host-side dnsmasq the agent manages so LAN clients reach their guest directly
(same hostname + real wildcard cert, no Cloudflare hairpin). Renders local=/
+address=/ per customer (AAAA->NODATA via authoritative zone, wildcard A ->
live guest IP), forwards everything else. Manager ensures dnsmasq+base config,
discovers guest IP (pct exec ip) + domain (controller.yaml), write-if-changed +
reload. Loop (7th daemon goroutine) tracks DHCP IP changes per provisioned
guest. --selftest=lanresolver. FELHOM_DNSMASQ sudoers. Spiked live on felhom-pve.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validate the customer.id parsed from bootstrap.json to a DNS-safe label (reject
spaces/slashes/leading-dash) and pass it via a quoted array, so a malformed or
hostile id can't smuggle extra docker flags (e.g. --privileged) into the
controller's de-privileged `docker run`. Empty/invalid → no --hostname.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
build-golden.sh: same-path /opt/docker/stacks host bind (Section-G fix, breaks
all bind-mounted stacks without it) + --hostname <customer-id> from bootstrap.json
(portable sed parse, no jq) + bake the 3 pinned public base-infra images with a
manifest-inspect hard gate. Provision --selftest defaults -hostname to the
DNS-safe-sanitized customer-id so the CT/LXC is named meaningfully.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Lockstep with felhom-controller v0.40.0. The agent now bakes a v2 bootstrap.json
carrying only what the controller needs to PULL its config from the hub:
customer.id + hub.url + hub.retrieval_password + the per-guest local_api. Stops
baking the agent's host hub key/host_id (and customer name/domain/email) into the
guest — the controller gets the customer-scoped key from the hub pull.
- internal/provision/doc.go: SchemaV2; DocCustomer{id}; DocHub{url,retrieval_password}.
- backhalf.go: render v2; require customer.id + hub.url + hub.retrieval_password.
- cmd/.../main.go --selftest=provision: new required -hub-password flag; stop
baking APIKey/HostID; -customer-domain/name/email accepted but not baked.
- configs/build-golden.sh: default CONTROLLER_IMAGE off stale :v0.35.0 -> :0.40.0.
- doc_test.go/backhalf_test.go updated to v2 shape.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
BackupRunner.BackupWithSnapshotHook tails the task log for the 'create storage
snapshot' marker (snapshot mode only) and fires onSnapshot once; localapi flips
/backup/status to 'snapshotted' before 'done' so the controller resumes early.
Phase 0 validated on PVE 9.2.2: marker confirmed, downtime ~24s->~1s (934MB).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
internal/localapi: real /backup/due (cadence; due when no successful backup or
newest older than backup.backup_cadence_seconds; false in-window after success;
failed doesn't count) + /backup/status phases (idle|running|done|failed + job
id) + POST /backup single-flight with job id. Drives the controller quiesce loop.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Live validation on demo-felhom caught the felhom-regen-hostkeys unit failing with
203/EXEC: ExecStart was /usr/sbin/ssh-keygen but on Debian 13 ssh-keygen is at
/usr/bin/ssh-keygen. Fixed build-golden.sh, rebuilt the golden, re-validated — host
keys now regenerate on first boot by the baked unit (agent issues no ssh-keygen).
All three live scenarios green: provision (fresh MAC, host keys via unit, machine-id,
Docker, DHCP), dr (continuity: hostname + host keys preserved), Recover (killed
mid-restore -> orphan rolled back idempotently). REPORT + CHANGELOG updated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The shared front half of provision and guest-loss DR as a journaled reconcile job
(internal/reconcile/bringup.go), mirroring the restore-test's crash-safety but keeping
the guest on success and applying a scenario-specific identity policy. Agent-only; no
hub/wire change. Grounded by the slice-7 bring-up spike (commit 3342993): F1/F3/F4.
- RunBringUp: restore -> reset identity -> size -> attach mounts -> start link-up;
verdict is liveness (waitRunning), success KEEPS the guest.
- identity policy: provision = fresh MAC (net0 sans hwaddr -> PVE regen) + hostname,
host-side; machine-id/host-keys regenerate guest-side (systemd + baked golden unit).
dr_guest_loss = preserve continuity (keep hostname; keep MAC unless KeepMAC=false).
- compensating rollback: mid-flight failure destroys the just-created guest
(SameTxnCreated provenance, gated); new Rollback journal flag + Recover.recoverBringUp
reap a half-built guest from a crash.
- F4: coalesced config PUT + bounded retry on the transient PVE config-lock 500 only.
- --selftest=bring-up (mode/archive/vmid/hostname/keep).
- configs/build-golden.sh: validated golden recipe incl. the F3 first-boot host-key unit.
- doc-03 §9 + identity-reset settled/implemented.
Deferred (stated): provisioning back half -> slice 8; host-loss DR + escrow consumption
and the BringUpSpec source (hub desired-state) -> slice 10.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Spike-proven that backup/restore-to-PBS reuse Phase A unchanged; the only new code is
the verify capability, a small PBS-API client, and PBSSnapshot reporting.
- internal/pbs: fingerprint-pinned, token-authed PBS-API client (Verify/Snapshots/
TaskStatus, node-from-UPID; secret read from /etc/pve/priv/storage/<id>.pw at runtime,
never logged) + the verify maintenance loop (own cadence, default 6h, NOT gated/journaled,
like the watchdog) + SnapshotStore.
- hub: PBSSnapshot filled (namespace/type/id/time/size/owner/protected/encrypted/
verify_state/verify_upid); PBSReporter collector seam; cross-repo golden + bidirectional
key-set tests; hub handler parses pbs_snapshots + logs a failed-verify WARN.
- backup: report the ACTUAL vzdump mode (parsed from the task log; PVE may downgrade
snapshot->stop). proxmox.Storage.Username. config PBSVerifyCadence/secret-dir.
--selftest=pbs-verify. Backup/restore-to-PBS unchanged.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
internal/hub: the agent's first daemon — a periodic read-only host-report POSTed to
the hub (the heartbeat; no separate ping).
- HostReport wire contract (shared field-for-field with the hub ingest): host
metrics, guests (vmid + spec), cloudflared status; storage/backups/restore-tests/
pbs/audit collections DEFINED but emitted empty (slices 5/6 fill).
- Collector over a read-only proxmoxReader (adapted to the real proxmox surface;
no proxmox changes) + a CloudflaredProber. Partial-failure: NodeStatus fail = hard
(skip POST); per-guest GuestConfig fail = status "unknown", still report.
- Client: Bearer-auth POST, standard TLS (system roots / optional ca_file), typed
TransportError/HTTPError, token never in errors.
- Loop: immediate first report, adopt hub poll_interval (clamp [60,3600]), resilient
to collect/report errors, clean ctx-cancel shutdown.
- ControlEnvelope: only poll_interval_seconds acted on; blocked/desired_generation/
has_signed_ops parsed-but-ignored (slice 4).
- config: HubConfig + FELHOM_AGENT_HUB_* overlay + mode-aware HubConfig.Validate +
WithDefaults + hub-key redaction; example config updated.
- main: no-selftest mode is now the daemon; added --selftest=hub. Version -> 0.3.0.
Tests: report serialization, client (incl. token-redaction), collector partial-
failure, loop continuation+interval adoption, config. internal/proxmox + internal/
authz untouched.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Stand up the felhom-agent project (module gitea.dooplex.hu/admin/felhom-agent,
binary felhom-agent) and the internal/proxmox package: the typed library every
other agent module calls to talk to Proxmox.
- API-first Client (hand-rolled REST over net/http, PVEAPIToken auth) with typed
read ops (version/nodes/status/lxc/config/storage) and async mutating ops
(restore/vzdump/snapshot/rollback/delete-snapshot/setconfig/start/stop), each
returning a UPID. WaitTask polls task status until stopped and asserts
exitstatus OK (authz can surface at task exec, not the POST — phase1-2 §1.3).
- Fenced Privileged (root-CLI) backend for the THREE proven exceptions only
(keyctl pct create, USB mount/fstab, SMART/sensors); each cites why it can't be
the API. Fence is structural (Client never shells out, Privileged never HTTPs)
and asserted in routing_test.go.
- TLS: SHA-256 leaf-cert pinning or CA file; insecure mode explicit + off by
default. No blanket verification disable.
- 403 -> privilege-named APIError; failed task -> privilege-named TaskError.
- JSON config + env overrides (token never logged); slog logging.
- cmd/felhom-agent --selftest (read-only health report) + gated --selftest=task
(reversible snapshot/rollback/delete exercise of WaitTask). No daemon loop yet.
- Types grounded in the spike findings and exact JSON shapes captured live from
demo-felhom (PVE 9.2.2). Unit tests use a mock transport + runner.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
admin