felhom.eu/documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md

SPIKE — Day-0 first-boot trust handshake (operator-deploy)

Update 2026-06-28 (BUNDLE slice): Day-0 now self-installs the agent + fetches the golden from Gitea. The host-bootstrap script (scripts/felhom-host-install.sh v1.1.0) fetches the agent binary + golden from Gitea generic packages and verifies each sha256 against a hub-vouched artifact manifest (GET /api/v1/artifacts/{id}, hub v0.16.0) before installing/using them — the fetch credential is the existing config-retrieve git token (no new credential); the checksum trust root is the hub, not Gitea. The "works on a box that isn't felhom-pve" gap is closed: prerequisites are now just install PVE + create the customer in the hub. The agent runs non-root (felhom-agent + sudoers). Remaining follow-ups: per-customer artifact pinning, a make golden that resolves-latest, non-root PBS-key access, the DR mode, and the local-DNS slice.

Date: 2026-06-26 Class: Spike (empirical validation; no production code shipped). Output is this doc only. Question: Does the composed first-boot chain — customer-in-hub → host mint → agent host-auth → controller config-pull — work end-to-end today with existing mechanisms (option A), and what exact contracts/gaps must the host-bootstrap script own? Secondary: is option C worth a hub change?

Verdict (short): Option A works end-to-end today with ZERO new code. All three trust links plus every wrong-case refusal behaved exactly as the source predicts, exercised against the LIVE hub with the real agent hub-client. Option C (collapse to one secret) is feasible and recommended for the production cutover but is a deliberate, small hub change — out of scope for this spike. No blocking gap. The host-bootstrap script can be specified on the contracts below.

---

1. Confirmed baselines (live Gitea = local HEAD, verified by git fetch)

Repo	main @ commit	Version	Role
felhom.eu (hub)	347546e	v0.13.1	mints host (POST /admin/hosts), serves config-pull
felhom-agent	4725396 (code = 166a1c8, v0.41.0)	v0.41.0	consumes host credential; authenticates to hub
felhom-controller	e0cf78b	v0.84.0	in-guest config-pull (the already-proven half)

The controller advanced to v0.84.0 (prompt said v0.82.0); irrelevant — the controller half was re-confirmed at the hub contract, which is unchanged.

Exact config-retrieve route string (the spike needed this):

GET /api/v1/config/{customer_id}
Header: X-Retrieval-Password: <5-word passphrase>
→ 200, Content-Type: text/yaml; charset=utf-8, body = the generated controller.yaml (RAW yaml, not JSON)

Confirmed at hub/internal/api/handler.go:187-189 (router) → handleConfigRetrieve (:1341), and it is the literal command the operator UI prints (internal/web/templates/config_detail.html:115): curl -fsSL https://hub.felhom.eu/api/v1/config/{id} -H "X-Retrieval-Password: {pw}" -o controller.yaml.

Correction to the prompt's grounding: config-retrieve is GET with the customer_id in the PATH and the response is raw YAML — not a POST with a {customer_id} body returning a JSON {ConfigYAML}. The host-bootstrap script spec must use the GET form.

Correction #2: the agent hub env vars are FELHOM_AGENT_HUB_URL / FELHOM_AGENT_HUB_HOST_ID / FELHOM_AGENT_HUB_API_KEY (prefix FELHOM_AGENT_HUB_), confirmed at config.go:446-453 — not FELHOM_HUB_* as the prompt stated.

---

2. Environment & method

• Live hub: hub.felhom.eu on k3s (felhom-system/hub-844754f7b4-...), SQLite at /data/hub.db, config at /etc/felhom-hub/hub.yaml. Global key = api.report_api_key (read out-of-band from the pod config into a shell var on the build box; never printed, never written to disk or this doc).
• Throwaway customer: spike-day0-2026-06-26 (domain spike.demo-felhom.eu, a throwaway 64-hex customer api_key, a throwaway passphrase, config_json carrying a fake tunnel token). Inserted via the hub's own customer_configs schema (the web-form create path is operator-auth-only; the row is byte-identical to what SaveCustomerConfig writes). The real demo-felhom customer was not touched.
• Probes ran from the build box (192.168.0.180, reaches hub.felhom.eu). Link 3 (agent auth) was exercised with the real internal/hub client via a throwaway cmd/spikeprobe built from the agent checkout — not a hand-rolled curl. No Proxmox, no provision, no real customer data.
• All secrets (global key, customer key, passphrase, minted host api_key) stayed resident on the build box in /tmp/spike-day0/; only status codes, the non-secret host_id, lengths, and boolean content checks were surfaced.

---

3. Per-probe results

Probe 1 — Host mint (link 2, option A) — WORKS

POST /api/v1/admin/hosts, handler handleAdminCreateHost (handler.go:599), global-key gated.

Case	Request	Expected	Observed
1a correct	global key, {customer_id: spike-day0-…}	201 + {host_id, api_key}	201, host_id=spike-day0-2026-06-26-6e5dc8, api_key 64-hex
host_id default	omitted host_id	<customer>-<6hex>	matched ^<customer>-[0-9a-f]{6}$
row landed	—	hosts row resolvable	present, api_key length 64 (GetHostByAPIKey works — proven in Probe 2)
1c wrong	customer key (non-global)	403	403
1d wrong	minted per-host key (non-global)	403	403
1e wrong	global key, unknown customer	400 Unknown customer_id	400 "Unknown customer_id"
1f wrong	no Authorization header	403	403

The !isGlobal gate (handler.go:600-604) and the cc == nil → 400 customer-existence check (:624-627) are correct, not bugs.

Probe 2 — Agent host-auth (link 3) with the minted credential — WORKS

Exercised the real internal/hub.Client (client.go) against the live hub with the Probe-1 credential.

Case	Call	Expected	Observed
2a correct	Report() → POST /api/v1/host-report	2xx + control envelope	OK, status="ok", desired_generation=0
2a effect	—	host-report row lands	landed: host_reports row, agent_version=spike-probe
2b correct	FetchDesiredState() → GET /api/v1/hosts/{id}/desired-state	2xx, well-formed (empty ok)	OK, generation=0, 0 guests
2c wrong	Report() with one flipped key char	401/403	HTTP 401 Unauthorized
2d wrong	FetchDesiredState() with empty host_id	client refuses pre-flight	refused: "FetchDesiredState requires a configured host_id" (client.go:119-120)
2e wrong	HubConfig.Validate() empty host_id	error before any call	"hub.host_id is required" (config.go:532-534)

checkAuthHost (handler.go:99-116) resolves the per-host Bearer via GetHostByAPIKey; host-report with isGlobal=false requires body.host_id == authed host_id (handler.go:453-455) — satisfied by the client always sending its configured host_id.

Probe 3 — Controller config-pull (link 1) re-confirm — WORKS (unchanged)

Case	Request	Expected	Observed
3a correct	GET /config/{id} + correct X-Retrieval-Password	200 text/yaml	200, text/yaml; charset=utf-8
3a content	—	domain, hub.url, hub api_key, tunnel token present	all YES (domain spike.demo-felhom.eu; hub.api_key = the customer-scoped key; injected tunnel token; hub.url)
3b wrong	wrong passphrase	401	401
3c wrong	missing X-Retrieval-Password header	401	401
3d wrong	unknown customer (correct-shaped pw)	404	404

Constant-time passphrase compare (handler.go:1365); cfg == nil → 404 (:1359-1362); missing header → 401 (:1347-1351). The controller half is intact and needs no re-architecting. The generated yaml carries the customer-scoped hub api_key (configgen.go:47, = cfg.APIKey) — this is the controller's key, distinct from the agent's per-host key minted in Probe 1.

---

4. End-to-end verdict on Option A

Option A composes today with ZERO new code. The full Day-0 chain is exercisable as:

1. HQ: operator creates the customer in the hub UI (mints customer api_key + 5-word passphrase).
2. HQ: operator mints the host with the global key (POST /admin/hosts) and obtains {host_id, host_api_key}. (The global key never leaves HQ.)
3. On the Proxmox box: the host-bootstrap script writes the agent config (/etc/felhom-agent/agent.json or FELHOM_AGENT_HUB_* env) with {hub.url, host_id, host_api_key}; the agent authenticates (host-report + desired-state) — proven in Probe 2.
4. Provision: the agent's provision back-half renders bootstrap.json into the guest from --customer-id + --hub-password (the passphrase) — (not run in this spike; existing, separately validated).
5. In-guest: the controller pulls controller.yaml via GET /config/{id} + X-Retrieval-Password — proven in Probe 3 — which yields the customer-scoped hub key, domain, and tunnel token; the tunnel comes up.

No link 401/403/404'd unexpectedly. The only thing not physically run start-to-finish is the provision step (4), which is deliberately out of a no-destructive-provision spike and is already validated elsewhere; both its inputs (host credential present; passphrase → config-pull) are proven.

Blocking gaps: none.

---

5. Option A vs C — recommendation

Two secrets cross to the customer box under A; one under C.

• Option A (proven): the operator carries two secrets to the deployment: the per-host api_key (for the agent, step 3) and the retrieval passphrase (for the in-guest controller, steps 4–5). The global key stays at HQ. Works today, no code change.
• Option C (proposed): the passphrase-authed pull also mints-or-returns the host credential, so the operator carries only the passphrase — and no global key participates in the deploy path at all (it stays a pure HQ-admin key). Fewer secrets in flight; the deploy box never holds an operator-tier global key even transiently.

Feasibility of C (confirmed at source): handleConfigRetrieve already has both prerequisites in scope — the customerID (path param) and the h.store handle (which exposes UpsertHost, GetHostByAPIKey, GetHost, ListHosts). The required hub delta:

1. Idempotency lookup (new): there is no GetHostByCustomer today. UpsertHost conflicts on host_id (store.go:1172-1183), so naive re-pull would mint a new random host each call and orphan the prior. C should mint-once-and-reuse: add a GetHostByCustomer(customerID) (or filter ListHosts) so a re-pull returns the existing credential. (Mint-per-pull = rotation; rejected — it would silently invalidate a running agent's key on any controller re-pull.)
2. Response-shape change — keep it backward-compatible: config-retrieve currently returns raw YAML consumed by the in-guest controller. Do not switch the body to JSON (breaks the controller curl). Two safe options, in order of preference:
   • (c-pref) Separate passphrase-authed endpoint, e.g. POST /api/v1/hosts/enroll with X-Retrieval-Password, body {customer_id}, returns {host_id, api_key} (idempotent). Leaves the proven config-retrieve path untouched; the host-bootstrap script calls enroll for the agent and config-retrieve for the guest seed. Cleanest separation of the host-tier and guest-tier pulls.
   • (c-alt) Side-channel headers on the existing GET: emit X-Host-Id / X-Host-Api-Key response headers alongside the YAML body. The in-guest controller ignores them; the script reads them. Fewer endpoints but couples two tiers' secrets into one response.

UPDATE 2026-06-26 — c-pref SHIPPED in hub v0.14.0. The dedicated passphrase-authed endpoint landed as POST /api/v1/host-enroll (mint-once-reuse via the new Store.GetHostByCustomer); the proven GET /config/{id} and the global-key POST /admin/hosts escape hatch are unchanged. See the hub CHANGELOG.md v0.14.0 + README.md "Host Enrollment (Day-0, option C)". The route is the exact path == "/host-enroll" form (not /hosts/enroll, which would collide with the /hosts/ per-host prefix routes).

UPDATE 2026-06-26 — Day-0 host-bootstrap script SHIPPED. This gap inventory is now implemented end-to-end by scripts/felhom-host-install.sh v1.0.0 (provision mode): pveum token → host-enroll → agent config → --selftest=provision → verify. Live-validated by an authorized wipe + re-provision of demo guest 9201 (controller config-pull + public tunnel HTTP 200 + host-report confirmed). DR mode is a documented stub (10D). See scripts/README.md + REPORT.md.

Recommendation: ship Option A now (it is proven and needs nothing). Adopt Option C, variant (c-pref) — a dedicated passphrase-authed host-enroll endpoint, mint-once-reuse — at the production cutover, on the secret-hygiene argument: it removes the operator-tier global key from the field deployment entirely and reduces the operator-carried secret set to one (the passphrase the customer already has). It is a small, isolated change that does not touch the proven controller config-pull. This matches the existing handleAdminCreateHost PROVISIONAL note (handler.go:595-598), which already flags that endpoint for lock-down at the enrollment cutover.

---

6. Gap inventory — the checklist the host-bootstrap script spec consumes

Exact config-retrieve contract: GET https://hub.felhom.eu/api/v1/config/{customer_id}, header X-Retrieval-Password: <passphrase>; 200 → raw controller.yaml (text/yaml), 401 wrong/missing pw, 404 unknown customer. Capture HTTP status and only treat 2xx as success (do not pipe a non-2xx body onward — the spike's recurring lesson; curl -f/-w '%{http_code}', check before use).

Secrets/values the script must obtain, and their source (A vs C):

Value	Source under A	Source under C
hub.url	static (https://hub.felhom.eu)	static
customer_id	operator-supplied (created in hub UI)	operator-supplied
retrieval passphrase	operator-supplied (hub UI) — SECRET	operator-supplied — SECRET (the only carried secret)
host_id	returned by HQ POST /admin/hosts	returned by passphrase-authed enroll
host api_key	returned by HQ POST /admin/hosts — SECRET	returned by passphrase-authed enroll — SECRET
global key	HQ-only; never on the box	not used in deploy path at all
customer-scoped hub key, domain, tunnel token	inside the pulled controller.yaml (not handled directly by the script)	same

Agent config write path on a real host (confirmed): /etc/felhom-agent/agent.json (JSON; flag --config, env FELHOM_AGENT_CONFIG), or override the hub fields via env FELHOM_AGENT_HUB_URL / FELHOM_AGENT_HUB_HOST_ID / FELHOM_AGENT_HUB_API_KEY (config.go:446-453). Host-specific fields: hub.host_id, hub.api_key (+ proxmox endpoint/node/token, which the agent needs anyway). Static fields: hub.url. (Per the live-agent notes, the demo box keeps its config under /root/.config/felhom-agent/ via FELHOM_AGENT_CONFIG; the script must write to whatever path the agent's systemd unit points --config/FELHOM_AGENT_CONFIG at — confirm the unit on the target.)

How the controller gets bootstrap.json (confirmed at source — the script does NOT write it): the agent's provision back-half renders it (felhom-agent/internal/provision/backhalf.go:121, doc.go:23-47), writes it 0600, chowns to the mapped guest-root, and attaches it on the guest config mount. The script's job is only to invoke the agent's provision with --customer-id <id> and --hub-password <passphrase> (cmd/felhom-agent/main.go:131-132); the agent bakes those into bootstrap.json. The bootstrap.json contract (cross-repo, must match the controller's internal/bootstrap.Bootstrap):

{ "schema": "...",
  "customer": { "id": "<customer_id>" },
  "hub": { "url": "<hub_url>", "retrieval_password": "<passphrase, SECRET>" },
  "local_api": { "endpoint": "<host-bridge ip:port>", "fingerprint": "<agent leaf SHA-256>", "token": "<per-guest bearer, SECRET>" } }

Note it carries no host key, no customer-scoped hub key, no CF tokens — those all come from the controller's pull (doc.go:24-26).

Unexpected 401/403/404 / values with no source: none observed. Every refusal was the correct gate.

---

7. Cleanup performed

All throwaway state removed and verified (counts → 0); the real demo-felhom customer confirmed still present and untouched:

• DELETE FROM host_reports / hosts / customer_configs WHERE customer_id='spike-day0-2026-06-26' (before: 1/1/1 → after: 0/0/0).
• Removed scratch cmd/spikeprobe from the build-box agent checkout (git status clean) and /tmp/spike{probe,-day0}, /tmp/*.sh.
• Uninstalled the temporary sqlite package from the hub pod (apk del sqlite).
• No production code, no version bump, no push beyond this doc; no image build; no deploy; no provision.

---

8. Open questions for the host-bootstrap script spec

1. Agent config target path on a production host — the spec must pin where the script writes agent.json vs. relying on FELHOM_AGENT_HUB_* env, and align with the agent's systemd unit --config/FELHOM_AGENT_CONFIG. (On the demo box this is /root/.config/felhom-agent/; production may differ.) Also: the agent needs a Proxmox token in that same config — where does the script source it on a freshly-imaged host? (Likely a pveum token-create step earlier in the runbook; confirm it is in the chain.)
2. Option C decision — if the team wants single-secret Day-0, the hosts/enroll endpoint (variant c-pref, idempotent mint-once-reuse) is the prerequisite and should be specced/shipped before the script depends on it. Until then the script uses Option A (HQ pre-mint).
3. local_api.endpoint/fingerprint provenance — these are filled by the agent at provision time, not by the script; the spec should state the script does not supply them (avoids a false "missing value" in the script's input list).
4. Re-run / idempotency of the whole script — under A, re-running the mint step creates a second host for the same customer (admin mint is not idempotent). The script should either reuse a prior credential or the runbook should make the mint a one-time HQ step. (C with mint-once-reuse fixes this structurally.)

No secrets appear anywhere in this document.