spike(day0): findings — first-boot trust handshake (operator-deploy)

Empirically validates the composed Day-0 chain against the LIVE hub: host
mint (POST /admin/hosts), agent host-auth via the real internal/hub client
(host-report + FetchDesiredState), and controller config-pull
(GET /config/{id} + X-Retrieval-Password). All links + wrong-cases pass.
Option A works end-to-end with zero new code; option C (single-secret enroll)
feasible + recommended for cutover. Throwaway customer/host cleaned up.

No production code, no version bump (spike).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtXesNa2LGbMmE4DNL6SE7

documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md

@@ -0,0 +1,271 @@
+# SPIKE — Day-0 first-boot trust handshake (operator-deploy)
+
+**Date:** 2026-06-26
+**Class:** Spike (empirical validation; no production code shipped). Output is this doc only.
+**Question:** Does the composed first-boot chain — customer-in-hub → host mint → agent host-auth →
+controller config-pull — work end-to-end **today** with existing mechanisms (option A), and what
+exact contracts/gaps must the host-bootstrap script own? Secondary: is option C worth a hub change?
+
+**Verdict (short):** **Option A works end-to-end today with ZERO new code.** All three trust links
+plus every wrong-case refusal behaved exactly as the source predicts, exercised against the LIVE hub
+with the real agent hub-client. Option C (collapse to one secret) is **feasible and recommended for
+the production cutover** but is a deliberate, small hub change — out of scope for this spike. No
+blocking gap. The host-bootstrap script can be specified on the contracts below.
+
+---
+
+## 1. Confirmed baselines (live Gitea = local HEAD, verified by `git fetch`)
+
+| Repo | `main` @ commit | Version | Role |
+|------|-----------------|---------|------|
+| felhom.eu (hub) | `347546e` | v0.13.1 | mints host (`POST /admin/hosts`), serves config-pull |
+| felhom-agent | `4725396` (code = `166a1c8`, v0.41.0) | v0.41.0 | consumes host credential; authenticates to hub |
+| felhom-controller | `e0cf78b` | v0.84.0 | in-guest config-pull (the already-proven half) |
+
+> The controller advanced to v0.84.0 (prompt said v0.82.0); irrelevant — the controller half was
+> re-confirmed at the hub contract, which is unchanged.
+
+**Exact config-retrieve route string (the spike needed this):**
+
+```
+GET /api/v1/config/{customer_id}
+Header: X-Retrieval-Password: <5-word passphrase>
+→ 200, Content-Type: text/yaml; charset=utf-8, body = the generated controller.yaml (RAW yaml, not JSON)
+```
+
+Confirmed at `hub/internal/api/handler.go:187-189` (router) → `handleConfigRetrieve` (`:1341`), and it is
+the literal command the operator UI prints (`internal/web/templates/config_detail.html:115`):
+`curl -fsSL https://hub.felhom.eu/api/v1/config/{id} -H "X-Retrieval-Password: {pw}" -o controller.yaml`.
+
+> **Correction to the prompt's grounding:** config-retrieve is **GET with the customer_id in the PATH**
+> and the response is **raw YAML** — not a POST with a `{customer_id}` body returning a JSON
+> `{ConfigYAML}`. The host-bootstrap script spec must use the GET form.
+
+> **Correction #2:** the agent hub env vars are `FELHOM_AGENT_HUB_URL` / `FELHOM_AGENT_HUB_HOST_ID` /
+> `FELHOM_AGENT_HUB_API_KEY` (prefix `FELHOM_AGENT_HUB_`), confirmed at `config.go:446-453` — **not**
+> `FELHOM_HUB_*` as the prompt stated.
+
+---
+
+## 2. Environment & method
+
+- **Live hub:** `hub.felhom.eu` on k3s (`felhom-system/hub-844754f7b4-...`), SQLite at `/data/hub.db`,
+  config at `/etc/felhom-hub/hub.yaml`. Global key = `api.report_api_key` (read out-of-band from the
+  pod config into a shell var on the build box; **never printed, never written to disk or this doc**).
+- **Throwaway customer:** `spike-day0-2026-06-26` (domain `spike.demo-felhom.eu`, a throwaway 64-hex
+  customer api_key, a throwaway passphrase, config_json carrying a fake tunnel token). Inserted via the
+  hub's own `customer_configs` schema (the web-form create path is operator-auth-only; the row is
+  byte-identical to what `SaveCustomerConfig` writes). The real `demo-felhom` customer was **not**
+  touched.
+- **Probes ran from the build box** (`192.168.0.180`, reaches `hub.felhom.eu`). Link 3 (agent auth) was
+  exercised with the **real** `internal/hub` client via a throwaway `cmd/spikeprobe` built from the
+  agent checkout — **not** a hand-rolled curl. No Proxmox, no provision, no real customer data.
+- All secrets (global key, customer key, passphrase, minted host api_key) stayed resident on the build
+  box in `/tmp/spike-day0/`; only status codes, the non-secret `host_id`, lengths, and boolean content
+  checks were surfaced.
+
+---
+
+## 3. Per-probe results
+
+### Probe 1 — Host mint (link 2, option A) — **WORKS**
+
+`POST /api/v1/admin/hosts`, handler `handleAdminCreateHost` (`handler.go:599`), global-key gated.
+
+| Case | Request | Expected | Observed |
+|------|---------|----------|----------|
+| 1a correct | global key, `{customer_id: spike-day0-…}` | 201 + `{host_id, api_key}` | **201**, `host_id=spike-day0-2026-06-26-6e5dc8`, api_key 64-hex |
+| host_id default | omitted `host_id` | `<customer>-<6hex>` | **matched** `^<customer>-[0-9a-f]{6}$` |
+| row landed | — | `hosts` row resolvable | **present**, api_key length 64 (`GetHostByAPIKey` works — proven in Probe 2) |
+| 1c wrong | **customer** key (non-global) | 403 | **403** |
+| 1d wrong | minted **per-host** key (non-global) | 403 | **403** |
+| 1e wrong | global key, **unknown** customer | 400 `Unknown customer_id` | **400 "Unknown customer_id"** |
+| 1f wrong | no Authorization header | 403 | **403** |
+
+The `!isGlobal` gate (`handler.go:600-604`) and the `cc == nil → 400` customer-existence check
+(`:624-627`) are correct, not bugs.
+
+### Probe 2 — Agent host-auth (link 3) with the minted credential — **WORKS**
+
+Exercised the real `internal/hub.Client` (`client.go`) against the live hub with the Probe-1 credential.
+
+| Case | Call | Expected | Observed |
+|------|------|----------|----------|
+| 2a correct | `Report()` → `POST /api/v1/host-report` | 2xx + control envelope | **OK**, `status="ok"`, `desired_generation=0` |
+| 2a effect | — | host-report row lands | **landed**: `host_reports` row, `agent_version=spike-probe` |
+| 2b correct | `FetchDesiredState()` → `GET /api/v1/hosts/{id}/desired-state` | 2xx, well-formed (empty ok) | **OK**, `generation=0`, 0 guests |
+| 2c wrong | `Report()` with one flipped key char | 401/403 | **HTTP 401 Unauthorized** |
+| 2d wrong | `FetchDesiredState()` with empty host_id | client refuses pre-flight | **refused**: "FetchDesiredState requires a configured host_id" (`client.go:119-120`) |
+| 2e wrong | `HubConfig.Validate()` empty host_id | error before any call | **"hub.host_id is required"** (`config.go:532-534`) |
+
+`checkAuthHost` (`handler.go:99-116`) resolves the per-host Bearer via `GetHostByAPIKey`; host-report
+with `isGlobal=false` requires `body.host_id == authed host_id` (`handler.go:453-455`) — satisfied by
+the client always sending its configured `host_id`.
+
+### Probe 3 — Controller config-pull (link 1) re-confirm — **WORKS (unchanged)**
+
+| Case | Request | Expected | Observed |
+|------|---------|----------|----------|
+| 3a correct | `GET /config/{id}` + correct `X-Retrieval-Password` | 200 text/yaml | **200**, `text/yaml; charset=utf-8` |
+| 3a content | — | domain, hub.url, hub api_key, tunnel token present | **all YES** (domain `spike.demo-felhom.eu`; `hub.api_key` = the customer-scoped key; injected tunnel token; `hub.url`) |
+| 3b wrong | wrong passphrase | 401 | **401** |
+| 3c wrong | missing `X-Retrieval-Password` header | 401 | **401** |
+| 3d wrong | unknown customer (correct-shaped pw) | 404 | **404** |
+
+Constant-time passphrase compare (`handler.go:1365`); `cfg == nil → 404` (`:1359-1362`); missing header
+`→ 401` (`:1347-1351`). The controller half is intact and needs no re-architecting. The generated yaml
+carries the **customer-scoped** hub `api_key` (`configgen.go:47`, = `cfg.APIKey`) — this is the
+controller's key, **distinct** from the agent's per-host key minted in Probe 1.
+
+---
+
+## 4. End-to-end verdict on Option A
+
+**Option A composes today with ZERO new code.** The full Day-0 chain is exercisable as:
+
+1. **HQ:** operator creates the customer in the hub UI (mints customer api_key + 5-word passphrase).
+2. **HQ:** operator mints the host with the **global key** (`POST /admin/hosts`) and obtains
+   `{host_id, host_api_key}`. *(The global key never leaves HQ.)*
+3. **On the Proxmox box:** the host-bootstrap script writes the agent config (`/etc/felhom-agent/agent.json`
+   or `FELHOM_AGENT_HUB_*` env) with `{hub.url, host_id, host_api_key}`; the agent authenticates
+   (host-report + desired-state) — **proven in Probe 2**.
+4. **Provision:** the agent's provision back-half renders `bootstrap.json` into the guest from
+   `--customer-id` + `--hub-password` (the passphrase) — *(not run in this spike; existing, separately
+   validated)*.
+5. **In-guest:** the controller pulls `controller.yaml` via `GET /config/{id}` + `X-Retrieval-Password`
+   — **proven in Probe 3** — which yields the customer-scoped hub key, domain, and tunnel token; the
+   tunnel comes up.
+
+No link 401/403/404'd unexpectedly. The only thing **not** physically run start-to-finish is the
+provision step (4), which is deliberately out of a no-destructive-provision spike and is already
+validated elsewhere; both its inputs (host credential present; passphrase → config-pull) are proven.
+
+**Blocking gaps: none.**
+
+---
+
+## 5. Option A vs C — recommendation
+
+**Two secrets cross to the customer box under A; one under C.**
+
+- **Option A** (proven): the operator carries **two** secrets to the deployment: the **per-host
+  api_key** (for the agent, step 3) *and* the **retrieval passphrase** (for the in-guest controller,
+  steps 4–5). The global key stays at HQ. Works today, no code change.
+- **Option C** (proposed): the passphrase-authed pull *also* mints-or-returns the host credential, so
+  the operator carries **only the passphrase** — and **no global key participates in the deploy path at
+  all** (it stays a pure HQ-admin key). Fewer secrets in flight; the deploy box never holds an
+  operator-tier global key even transiently.
+
+**Feasibility of C (confirmed at source):** `handleConfigRetrieve` already has both prerequisites in
+scope — the `customerID` (path param) and the `h.store` handle (which exposes `UpsertHost`,
+`GetHostByAPIKey`, `GetHost`, `ListHosts`). The required hub delta:
+
+1. **Idempotency lookup (new):** there is **no `GetHostByCustomer`** today. `UpsertHost` conflicts on
+   `host_id` (`store.go:1172-1183`), so naive re-pull would mint a *new* random host each call and
+   orphan the prior. C should **mint-once-and-reuse**: add a `GetHostByCustomer(customerID)` (or filter
+   `ListHosts`) so a re-pull returns the existing credential. *(Mint-per-pull = rotation; rejected — it
+   would silently invalidate a running agent's key on any controller re-pull.)*
+2. **Response-shape change — keep it backward-compatible:** config-retrieve currently returns **raw
+   YAML** consumed by the in-guest controller. Do **not** switch the body to JSON (breaks the controller
+   curl). Two safe options, in order of preference:
+   - **(c-pref) Separate passphrase-authed endpoint**, e.g. `POST /api/v1/hosts/enroll` with
+     `X-Retrieval-Password`, body `{customer_id}`, returns `{host_id, api_key}` (idempotent). Leaves the
+     proven config-retrieve path **untouched**; the host-bootstrap script calls enroll for the agent and
+     config-retrieve for the guest seed. Cleanest separation of the host-tier and guest-tier pulls.
+   - **(c-alt) Side-channel headers** on the existing GET: emit `X-Host-Id` / `X-Host-Api-Key` response
+     headers alongside the YAML body. The in-guest controller ignores them; the script reads them. Fewer
+     endpoints but couples two tiers' secrets into one response.
+
+**Recommendation:** ship **Option A now** (it is proven and needs nothing). Adopt **Option C, variant
+(c-pref) — a dedicated passphrase-authed `hosts/enroll` endpoint, mint-once-reuse — at the production
+cutover**, on the secret-hygiene argument: it removes the operator-tier global key from the field
+deployment entirely and reduces the operator-carried secret set to one (the passphrase the customer
+already has). It is a small, isolated change that does not touch the proven controller config-pull.
+This matches the existing `handleAdminCreateHost` PROVISIONAL note (`handler.go:595-598`), which already
+flags that endpoint for lock-down at the enrollment cutover.
+
+---
+
+## 6. Gap inventory — the checklist the host-bootstrap script spec consumes
+
+**Exact config-retrieve contract:** `GET https://hub.felhom.eu/api/v1/config/{customer_id}`, header
+`X-Retrieval-Password: <passphrase>`; 200 → raw `controller.yaml` (text/yaml), 401 wrong/missing pw,
+404 unknown customer. **Capture HTTP status and only treat 2xx as success** (do not pipe a non-2xx body
+onward — the spike's recurring lesson; curl `-f`/`-w '%{http_code}'`, check before use).
+
+**Secrets/values the script must obtain, and their source (A vs C):**
+
+| Value | Source under A | Source under C |
+|-------|----------------|----------------|
+| `hub.url` | static (`https://hub.felhom.eu`) | static |
+| `customer_id` | operator-supplied (created in hub UI) | operator-supplied |
+| `retrieval passphrase` | operator-supplied (hub UI) — SECRET | operator-supplied — SECRET (the **only** carried secret) |
+| `host_id` | returned by HQ `POST /admin/hosts` | returned by passphrase-authed enroll |
+| host `api_key` | returned by HQ `POST /admin/hosts` — SECRET | returned by passphrase-authed enroll — SECRET |
+| **global key** | **HQ-only; never on the box** | **not used in deploy path at all** |
+| customer-scoped hub key, domain, tunnel token | inside the pulled `controller.yaml` (not handled directly by the script) | same |
+
+**Agent config write path on a real host (confirmed):** `/etc/felhom-agent/agent.json` (JSON; flag
+`--config`, env `FELHOM_AGENT_CONFIG`), or override the hub fields via env
+`FELHOM_AGENT_HUB_URL` / `FELHOM_AGENT_HUB_HOST_ID` / `FELHOM_AGENT_HUB_API_KEY` (`config.go:446-453`).
+**Host-specific fields:** `hub.host_id`, `hub.api_key` (+ proxmox endpoint/node/token, which the agent
+needs anyway). **Static fields:** `hub.url`. (Per the live-agent notes, the demo box keeps its config
+under `/root/.config/felhom-agent/` via `FELHOM_AGENT_CONFIG`; the script must write to whatever path
+the agent's systemd unit points `--config`/`FELHOM_AGENT_CONFIG` at — confirm the unit on the target.)
+
+**How the controller gets `bootstrap.json` (confirmed at source — the script does NOT write it):** the
+**agent's provision back-half** renders it (`felhom-agent/internal/provision/backhalf.go:121`,
+`doc.go:23-47`), writes it 0600, chowns to the mapped guest-root, and attaches it on the guest config
+mount. The script's job is only to invoke the agent's provision with `--customer-id <id>` and
+`--hub-password <passphrase>` (`cmd/felhom-agent/main.go:131-132`); the agent bakes those into
+`bootstrap.json`. The `bootstrap.json` contract (cross-repo, must match the controller's
+`internal/bootstrap.Bootstrap`):
+
+```json
+{ "schema": "...",
+  "customer": { "id": "<customer_id>" },
+  "hub": { "url": "<hub_url>", "retrieval_password": "<passphrase, SECRET>" },
+  "local_api": { "endpoint": "<host-bridge ip:port>", "fingerprint": "<agent leaf SHA-256>", "token": "<per-guest bearer, SECRET>" } }
+```
+
+Note it carries **no host key, no customer-scoped hub key, no CF tokens** — those all come from the
+controller's pull (`doc.go:24-26`).
+
+**Unexpected 401/403/404 / values with no source:** none observed. Every refusal was the correct gate.
+
+---
+
+## 7. Cleanup performed
+
+All throwaway state removed and verified (counts → 0); the real `demo-felhom` customer confirmed still
+present and untouched:
+
+- `DELETE FROM host_reports / hosts / customer_configs WHERE customer_id='spike-day0-2026-06-26'`
+  (before: 1/1/1 → after: 0/0/0).
+- Removed scratch `cmd/spikeprobe` from the build-box agent checkout (`git status` clean) and
+  `/tmp/spike{probe,-day0}`, `/tmp/*.sh`.
+- Uninstalled the temporary `sqlite` package from the hub pod (`apk del sqlite`).
+- No production code, no version bump, no push beyond this doc; no image build; no deploy; no provision.
+
+---
+
+## 8. Open questions for the host-bootstrap script spec
+
+1. **Agent config target path on a production host** — the spec must pin where the script writes
+   `agent.json` vs. relying on `FELHOM_AGENT_HUB_*` env, and align with the agent's systemd unit
+   `--config`/`FELHOM_AGENT_CONFIG`. (On the demo box this is `/root/.config/felhom-agent/`; production
+   may differ.) Also: the agent needs a **Proxmox token** in that same config — where does the script
+   source it on a freshly-imaged host? (Likely a `pveum` token-create step earlier in the runbook;
+   confirm it is in the chain.)
+2. **Option C decision** — if the team wants single-secret Day-0, the `hosts/enroll` endpoint (variant
+   c-pref, idempotent mint-once-reuse) is the prerequisite and should be specced/shipped before the
+   script depends on it. Until then the script uses Option A (HQ pre-mint).
+3. **`local_api.endpoint`/`fingerprint` provenance** — these are filled by the agent at provision time,
+   not by the script; the spec should state the script does not supply them (avoids a false "missing
+   value" in the script's input list).
+4. **Re-run / idempotency of the whole script** — under A, re-running the mint step creates a *second*
+   host for the same customer (admin mint is not idempotent). The script should either reuse a prior
+   credential or the runbook should make the mint a one-time HQ step. (C with mint-once-reuse fixes
+   this structurally.)
+
+**No secrets appear anywhere in this document.**