felhom-agent/docs/architecture/03-host-agent.md

Architecture Part 3 — The Host Agent

Status: design draft (decision content). To be grounded by Claude Code against docs/proxmox-platform.md and docs/architecture/02-controller-module-map.md, then placed at docs/architecture/03-host-agent.md.

Builds on Part 1 (01-topology-and-trust.md) and Part 2 (02-controller-module-map.md). Where this doc and the locked decisions disagree, the locked decisions win and this draft is wrong — flag it.

1. Purpose & scope

The host agent is the operator-tier component that runs on each Proxmox host and owns all Proxmox interaction. It is the trusted host actor: it provisions and restores guests, manages host storage, orchestrates backups and restore-tests, watches the host and the tunnel, talks to the hub, and exposes a narrow local API to the in-guest controllers it deploys.

It is the privileged tier. The controller deliberately holds no Proxmox credentials (Part 1) — the privilege the controller shed by losing storage/ did not disappear, it moved here. That makes the agent's hardening and blast-radius discipline the most security-sensitive part of the platform.

The agent manages a set of guests on its host (usually one customer = one guest, but the multi-tenant/company case is not precluded — the agent's data model is per-host, N-guests, never "the guest").

2. Responsibilities (and explicit non-responsibilities)

Owns:

1. Proxmox lifecycle — create/start/stop/destroy guests, snapshots, storage allocation. Via a scoped Proxmox API token (minimal role from Phase 1) for everything the API covers; raw host ops only where unavoidable.
2. Storage management — attach/classify targets, reconcile the storage manifest, mount USB-by-UUID, present mounts into guests.
3. Backup/restore orchestration — vzdump to the tiers, PBS, snapshot management, and the self-restore-test.
4. Host & tunnel monitoring — host metrics, guest up/down, storage-target status, and cloudflared health; reports the host domain to the hub.
5. Provisioning — build a guest, deploy the controller into it, hand it its bootstrap config.
6. Hub control loop — poll for desired state + signed jobs, reconcile, execute, report, heartbeat.
7. Local API — the per-guest authorization gate the controller calls.
8. Self-update — update itself (carefully — it is a host service) and update the controllers it owns.

Explicitly does not:

• Serve application traffic or sit in the data path. Control plane, not data plane: if the agent dies, apps keep serving (Docker + LXC run without it); only management degrades — no new backups, no provisioning, hub loses the heartbeat.
• Hold or proxy customer application data.
• Run inside a guest. It is the thing that recovers guests and the host; it cannot be one of them.

3. Process model & host integration

• Native Go binary, systemd service on the host: boot-start, Restart=always, systemd watchdog (kill+restart on hang), journald logging, resource limits.
• Root-minimized. Default to a non-root service user with the scoped Proxmox token for API-covered work + a narrow sudoers allowlist for the handful of true host ops (USB mount-by-UUID, systemd mount units). Full root on the crown-jewel host is what a compromise most wants; avoid it where the API or a scoped sudoers entry suffices. (Open: confirm during build which ops genuinely need host root vs. are API-covered — the Phase-1 minimal role is the API floor.)
• cloudflared is a separate systemd service, not embedded in the agent. This is what makes the data path survive control-plane death by construction. The agent manages and health-watches it (see §5) but the tunnel does not live or die with the agent process.

4. Control model — reconcile + signed destructive ops

Two channels, split by reversibility, not by transport.

(a) Desired-state reconciliation — steady state. The hub holds desired state for the host: which guests should exist (and at what spec), the storage manifest, backup/retention policies, controller image versions. The agent runs a reconcile loop converging actual Proxmox state → desired: idempotent, self-healing, and tolerant of missed polls (drift is corrected on the next loop). Provisioning retries, re-attach of a flapping USB target, redeploy of a crashed controller — all fall out of reconciliation for free.

(b) Signed one-shot jobs — operator actions. Restore-now, decommission, force-backup, break-glass-enable. Discrete, run-once (idempotency key), written to the customer-visible audit log, and outside the reconcile loop — they are point-in-time and often destructive, and a reconciler must never re-run a restore because it "sees drift." A one-shot job names a target ("restore guest X from snapshot S"), not a procedure; the agent owns the how.

The reversibility gate (security-critical). "Signed jobs resist hub compromise" only holds if the agent also distrusts hub-supplied desired state for destructive changes. So:

• Irreversible/destructive operations — guest destroy, storage detach/wipe, restore-overwrite, decommission — require a valid operator signature, regardless of whether they arrive as a job or as a desired-state delta. A compromised hub cannot forge them because the signing key is not held by the hub (it lives with the operator / a separate signing path; the hub only queues opaque signed blobs).
• Benign convergence — deploy a guest, attach storage, adjust a non-destructive policy, bump a controller version — runs on normal hub API auth, no signature.

Signed payloads carry a nonce + expiry (anti-replay: a captured "restore" job cannot be re-injected later) and a target binding (host + guest id) so a signature can't be retargeted. Notification-on-destructive-op is an audit signal, never the guard — a compromised hub could both issue and suppress the notice, which is exactly why the signature (not the notification) is the control.

5. Hub ↔ agent protocol (host domain)

Box-initiated poll. The hub never connects inbound. Each poll cycle exchanges:

• Up: heartbeat + a host-domain state report — host CPU/RAM/disk, per-guest up/down + spec, storage-target status (USB connected? NFS/CIFS reachable? PBS reachable?), last backup per target, last restore-test result, cloudflared health, agent + controller versions, audit-log tail.
• Down: the current desired state, any pending signed one-shot jobs, and config (poll interval, update window, policy changes).

Dead-man's-switch (essential, not optional). In a box-initiated model the heartbeat is the liveness signal — a box that stops checking in is otherwise invisible. The hub alerts the operator when an agent misses its expected check-in window. This is the worst failure mode for a managed service, so it gets first-class treatment hub-side.

Break-glass. Standing inbound control is off. But when the poll loop itself is wedged (agent hung, host sick) you cannot fix it through the poll loop. So there is an explicit, off-by-default, customer-consented, fully-audited emergency path: SSH to the host via the Cloudflare Tunnel behind Cloudflare Access (or on-site). Enabling it is itself a signed, logged operation; it auto-expires.

6. Agent ↔ controller local API

The controller (in its LXC) reaches the agent (on the host) over the local bridge.

• Transport: HTTPS to the host's bridge IP on a fixed port.
• Auth: a per-guest local token, minted by the agent when it deploys the controller and written into the guest's bootstrap config. The agent maps token → guest and authorizes per guest: a controller can only act on its own guest. This is the agent acting as the per-guest authorization gate from Part 1.
• Surface (minimal, all scoped to the caller's own guest):
  • GET /storage — mounts available to this guest and their class (fast/slow), so the controller can place hot vs bulk volumes per .felhom.yml. (The agent owns the actual mounts; the controller just binds to the paths it's given.)
  • POST /snapshot — snapshot this guest (the snapshot-before-deploy primitive).
  • POST /rollback — roll this guest back to a named snapshot (post-deploy failure recovery).
  • POST /backup — request a backup-now of this guest (enqueued; non-destructive).
  • GET /backup/status, GET /restore-test/status — read-only status for the controller's UI.

Note what is absent: nothing here lets a controller touch another guest, the host, storage attachment, or restore-overwrite. Destructive/cross-guest power stays operator-signed (§4).

7. Storage manifest & reconciliation

The manifest is the load-bearing contract (it absorbs the disk-state fields that settings.StoragePath carries today — see Part 2). Held in the hub, reconciled by the agent.

Per target:

field	meaning
type	local-dir / usb / nfs / cifs / pbs
durable_id	UUID (USB), server:export (NFS/CIFS), repo+fingerprint (PBS) — survives box loss
class	fast or slow, set once at attach, with an IOPS marker; no runtime speed-test
role	primary / vzdump-target / pbs-offsite / bulk-data
creds	encrypted (NFS/CIFS/PBS); USB has none
policy	schedule + retention for this target
state	attached / disconnected / decommissioned

Reconciliation: ensure each attached target is mounted (USB-by-UUID via the sudoers allowlist), each Proxmox storage entry matches, and disconnected targets are surfaced to the hub (the storage watchdog — detect a USB drop in seconds, not at the next health cycle).

Placement is per-volume, not per-app. Hot volumes (DB/config) → a fast target, enforced; bulk volumes (media) → may live on slow, declared in .felhom.yml. Bulk external mounts are excluded from the guest's vzdump (a per-mount backup flag) and carry their own per-volume policy (file-level to a tier, or explicitly not backed up for re-downloadable media). This is what keeps a 1 TB media drive out of the whole-guest image.

8. Backup/restore orchestration

Tiers double as backup and restore-source priority (fastest surviving source first), per Part 1: snapshot (LVM-thin, transient, whole-guest rollback — not a backup) → local second storage (vzdump to dir/NFS/CIFS) → PBS offsite (the DR substrate).

• Quiescing: the controller stops the app stack (volume-consistent) before a guest vzdump where app-consistency matters; stop-mode/snapshot-mode per Phase 1. Every Proxmox op is async → the agent polls task exitstatus, never trusts the POST return.
• Key custody (PBS): the live PBS key sits on the box so the agent can both back up and run restore-tests. The hub holds only the recovery-code-wrapped escrow copy it cannot open (zero-knowledge default). So: the box can restore-test; the operator cannot read the data; the customer's offsite recovery code is the irreducible residual.
• Self-restore-test: the closing of the "tested restore is the critical gap" theme. The agent periodically restores a backup into a throwaway scratch guest, boots it, runs health checks, reports pass/fail, and tears it down. Zero-knowledge backups can only be restore-tested by the box (the operator lacks the key) — so this lives in the agent by necessity, not just convenience. Integrity-verify (cheap, ciphertext-level) runs more often as the lighter check.

9. Provisioning & DR flows

Provisioning (reconcile-driven). Desired state says "this customer should have guest G with controller C." The agent: enrolls (mints its scoped Proxmox token as root at setup) → creates the LXC (unprivileged, nesting=1,keyctl=1, overlayfs — Phase 0) → deploys the controller → hands it the bootstrap config (identity, hub API key, local-API token, mount map). If any step fails, reconciliation retries; a half-built guest is journaled (§10) and rolled back, never orphaned.

Guest loss. Agent restores G from the fastest surviving tier and resets identity (MAC/hostname) so the restored guest rejoins cleanly.

Host/hardware loss. Re-enroll the new host in restore mode; the hub — the durable source of truth that survives box death — hands the new agent the existing identity, PBS namespace, tunnel token, storage manifest, and a restore directive. Tunnel is reused from the hub record, so DNS stays intact.

10. Concurrency, crash-safety, idempotency

• Per-guest serialization. Reconcile, one-shot jobs, and local-API calls all feed a work queue that serializes mutations per guest (Proxmox dislikes concurrent conflicting ops on the same guest). Independent guests proceed in parallel.
• Operation journaling. Multi-step async ops (provision, restore) are journaled with their in-flight Proxmox task ids. On agent restart, the journal is replayed: resume-or-rollback, so a crash mid-restore never leaves a corrupt or half-built guest.
• Idempotency keys on one-shot jobs (run-once across retries and restarts).

11. Self-update

• Agent (the hard case — a host service, no snapshot-rollback). Atomic binary swap: download → verify signature → atomic rename → restart; keep last-known-good; a watchdog reverts to last-good if the new binary fails to come up healthy. Triggered by a hub signed job within the update window; manual always allowed.
• Controller (the easy case — it's a guest). The agent owns the controller's lifecycle, so the agent updates the controller: snapshot-before-update (free rollback, because the controller is a snapshottable guest) → pull new image → redeploy → health-check → rollback on failure. This resolves the Part-2 selfupdate/ open: the controller is agent-managed, not self-updating; the controller's old self-update path is removed.

12. Secrets at rest on the host

The agent holds, root-only on the host fs: the scoped Proxmox token, the hub API key, the operator's public verify key (for §4 signatures — public, low-risk), the Cloudflare tunnel token, encrypted storage creds (NFS/CIFS/PBS), and the live PBS key. The privilege and the secret footprint that left the controller now concentrate here — which is the whole argument for §3's root-minimization and a small, auditable agent.

13. Open items / what this unblocks

Resolved here: tunnel placement (host, agent-managed, own systemd service), the reconcile-vs-jobs fork (hybrid, gated by reversibility), agent process model, self-update ownership, the local-API surface, and the storage-manifest schema.

Still open:

• Multi-tenant resource fairness on a shared host (per-guest cgroup limits, noisy-neighbor) — deferred to the company-case pass.
• Operator-side signing tooling — where the operator signing key lives operationally and how a destructive op gets signed without undue friction (offline key vs. a small signing service; the security floor is "not in the hub").
• Hub-side desired-state editing UX and the host-domain report schema details — belong to the hub architecture doc.

This doc hands the implementation three contracts it was waiting on:

1. the local-API surface (§6) → the controller's NEW local-API client, snapshot-before-deploy, and self-restore-test wiring (Part 2);
2. the storage-manifest schema (§7) → the settings.StoragePath reshape and per-volume hot/bulk placement (Part 2);
3. the backup contract (§7–8) → the destination for the app-data-backup package extracted in the Part-2 refactor.