From 230980f7a8d1d36864377e7fc6ab2884bcca5b41 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Fri, 26 Jun 2026 15:12:27 +0200 Subject: [PATCH] =?UTF-8?q?spike(day0):=20findings=20=E2=80=94=20first-boo?= =?UTF-8?q?t=20trust=20handshake=20(operator-deploy)?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Empirically validates the composed Day-0 chain against the LIVE hub: host mint (POST /admin/hosts), agent host-auth via the real internal/hub client (host-report + FetchDesiredState), and controller config-pull (GET /config/{id} + X-Retrieval-Password). All links + wrong-cases pass. Option A works end-to-end with zero new code; option C (single-secret enroll) feasible + recommended for cutover. Throwaway customer/host cleaned up. No production code, no version bump (spike). Co-Authored-By: Claude Opus 4.8 (1M context) Claude-Session: https://claude.ai/code/session_01TtXesNa2LGbMmE4DNL6SE7 --- ...IKE-day0-firstboot-handshake-2026-06-26.md | 271 ++++++++++++++++++ 1 file changed, 271 insertions(+) create mode 100644 documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md diff --git a/documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md b/documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md new file mode 100644 index 0000000..774147d --- /dev/null +++ b/documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md @@ -0,0 +1,271 @@ +# SPIKE — Day-0 first-boot trust handshake (operator-deploy) + +**Date:** 2026-06-26 +**Class:** Spike (empirical validation; no production code shipped). Output is this doc only. +**Question:** Does the composed first-boot chain — customer-in-hub → host mint → agent host-auth → +controller config-pull — work end-to-end **today** with existing mechanisms (option A), and what +exact contracts/gaps must the host-bootstrap script own? Secondary: is option C worth a hub change? + +**Verdict (short):** **Option A works end-to-end today with ZERO new code.** All three trust links +plus every wrong-case refusal behaved exactly as the source predicts, exercised against the LIVE hub +with the real agent hub-client. Option C (collapse to one secret) is **feasible and recommended for +the production cutover** but is a deliberate, small hub change — out of scope for this spike. No +blocking gap. The host-bootstrap script can be specified on the contracts below. + +--- + +## 1. Confirmed baselines (live Gitea = local HEAD, verified by `git fetch`) + +| Repo | `main` @ commit | Version | Role | +|------|-----------------|---------|------| +| felhom.eu (hub) | `347546e` | v0.13.1 | mints host (`POST /admin/hosts`), serves config-pull | +| felhom-agent | `4725396` (code = `166a1c8`, v0.41.0) | v0.41.0 | consumes host credential; authenticates to hub | +| felhom-controller | `e0cf78b` | v0.84.0 | in-guest config-pull (the already-proven half) | + +> The controller advanced to v0.84.0 (prompt said v0.82.0); irrelevant — the controller half was +> re-confirmed at the hub contract, which is unchanged. + +**Exact config-retrieve route string (the spike needed this):** + +``` +GET /api/v1/config/{customer_id} +Header: X-Retrieval-Password: <5-word passphrase> +→ 200, Content-Type: text/yaml; charset=utf-8, body = the generated controller.yaml (RAW yaml, not JSON) +``` + +Confirmed at `hub/internal/api/handler.go:187-189` (router) → `handleConfigRetrieve` (`:1341`), and it is +the literal command the operator UI prints (`internal/web/templates/config_detail.html:115`): +`curl -fsSL https://hub.felhom.eu/api/v1/config/{id} -H "X-Retrieval-Password: {pw}" -o controller.yaml`. + +> **Correction to the prompt's grounding:** config-retrieve is **GET with the customer_id in the PATH** +> and the response is **raw YAML** — not a POST with a `{customer_id}` body returning a JSON +> `{ConfigYAML}`. The host-bootstrap script spec must use the GET form. + +> **Correction #2:** the agent hub env vars are `FELHOM_AGENT_HUB_URL` / `FELHOM_AGENT_HUB_HOST_ID` / +> `FELHOM_AGENT_HUB_API_KEY` (prefix `FELHOM_AGENT_HUB_`), confirmed at `config.go:446-453` — **not** +> `FELHOM_HUB_*` as the prompt stated. + +--- + +## 2. Environment & method + +- **Live hub:** `hub.felhom.eu` on k3s (`felhom-system/hub-844754f7b4-...`), SQLite at `/data/hub.db`, + config at `/etc/felhom-hub/hub.yaml`. Global key = `api.report_api_key` (read out-of-band from the + pod config into a shell var on the build box; **never printed, never written to disk or this doc**). +- **Throwaway customer:** `spike-day0-2026-06-26` (domain `spike.demo-felhom.eu`, a throwaway 64-hex + customer api_key, a throwaway passphrase, config_json carrying a fake tunnel token). Inserted via the + hub's own `customer_configs` schema (the web-form create path is operator-auth-only; the row is + byte-identical to what `SaveCustomerConfig` writes). The real `demo-felhom` customer was **not** + touched. +- **Probes ran from the build box** (`192.168.0.180`, reaches `hub.felhom.eu`). Link 3 (agent auth) was + exercised with the **real** `internal/hub` client via a throwaway `cmd/spikeprobe` built from the + agent checkout — **not** a hand-rolled curl. No Proxmox, no provision, no real customer data. +- All secrets (global key, customer key, passphrase, minted host api_key) stayed resident on the build + box in `/tmp/spike-day0/`; only status codes, the non-secret `host_id`, lengths, and boolean content + checks were surfaced. + +--- + +## 3. Per-probe results + +### Probe 1 — Host mint (link 2, option A) — **WORKS** + +`POST /api/v1/admin/hosts`, handler `handleAdminCreateHost` (`handler.go:599`), global-key gated. + +| Case | Request | Expected | Observed | +|------|---------|----------|----------| +| 1a correct | global key, `{customer_id: spike-day0-…}` | 201 + `{host_id, api_key}` | **201**, `host_id=spike-day0-2026-06-26-6e5dc8`, api_key 64-hex | +| host_id default | omitted `host_id` | `-<6hex>` | **matched** `^-[0-9a-f]{6}$` | +| row landed | — | `hosts` row resolvable | **present**, api_key length 64 (`GetHostByAPIKey` works — proven in Probe 2) | +| 1c wrong | **customer** key (non-global) | 403 | **403** | +| 1d wrong | minted **per-host** key (non-global) | 403 | **403** | +| 1e wrong | global key, **unknown** customer | 400 `Unknown customer_id` | **400 "Unknown customer_id"** | +| 1f wrong | no Authorization header | 403 | **403** | + +The `!isGlobal` gate (`handler.go:600-604`) and the `cc == nil → 400` customer-existence check +(`:624-627`) are correct, not bugs. + +### Probe 2 — Agent host-auth (link 3) with the minted credential — **WORKS** + +Exercised the real `internal/hub.Client` (`client.go`) against the live hub with the Probe-1 credential. + +| Case | Call | Expected | Observed | +|------|------|----------|----------| +| 2a correct | `Report()` → `POST /api/v1/host-report` | 2xx + control envelope | **OK**, `status="ok"`, `desired_generation=0` | +| 2a effect | — | host-report row lands | **landed**: `host_reports` row, `agent_version=spike-probe` | +| 2b correct | `FetchDesiredState()` → `GET /api/v1/hosts/{id}/desired-state` | 2xx, well-formed (empty ok) | **OK**, `generation=0`, 0 guests | +| 2c wrong | `Report()` with one flipped key char | 401/403 | **HTTP 401 Unauthorized** | +| 2d wrong | `FetchDesiredState()` with empty host_id | client refuses pre-flight | **refused**: "FetchDesiredState requires a configured host_id" (`client.go:119-120`) | +| 2e wrong | `HubConfig.Validate()` empty host_id | error before any call | **"hub.host_id is required"** (`config.go:532-534`) | + +`checkAuthHost` (`handler.go:99-116`) resolves the per-host Bearer via `GetHostByAPIKey`; host-report +with `isGlobal=false` requires `body.host_id == authed host_id` (`handler.go:453-455`) — satisfied by +the client always sending its configured `host_id`. + +### Probe 3 — Controller config-pull (link 1) re-confirm — **WORKS (unchanged)** + +| Case | Request | Expected | Observed | +|------|---------|----------|----------| +| 3a correct | `GET /config/{id}` + correct `X-Retrieval-Password` | 200 text/yaml | **200**, `text/yaml; charset=utf-8` | +| 3a content | — | domain, hub.url, hub api_key, tunnel token present | **all YES** (domain `spike.demo-felhom.eu`; `hub.api_key` = the customer-scoped key; injected tunnel token; `hub.url`) | +| 3b wrong | wrong passphrase | 401 | **401** | +| 3c wrong | missing `X-Retrieval-Password` header | 401 | **401** | +| 3d wrong | unknown customer (correct-shaped pw) | 404 | **404** | + +Constant-time passphrase compare (`handler.go:1365`); `cfg == nil → 404` (`:1359-1362`); missing header +`→ 401` (`:1347-1351`). The controller half is intact and needs no re-architecting. The generated yaml +carries the **customer-scoped** hub `api_key` (`configgen.go:47`, = `cfg.APIKey`) — this is the +controller's key, **distinct** from the agent's per-host key minted in Probe 1. + +--- + +## 4. End-to-end verdict on Option A + +**Option A composes today with ZERO new code.** The full Day-0 chain is exercisable as: + +1. **HQ:** operator creates the customer in the hub UI (mints customer api_key + 5-word passphrase). +2. **HQ:** operator mints the host with the **global key** (`POST /admin/hosts`) and obtains + `{host_id, host_api_key}`. *(The global key never leaves HQ.)* +3. **On the Proxmox box:** the host-bootstrap script writes the agent config (`/etc/felhom-agent/agent.json` + or `FELHOM_AGENT_HUB_*` env) with `{hub.url, host_id, host_api_key}`; the agent authenticates + (host-report + desired-state) — **proven in Probe 2**. +4. **Provision:** the agent's provision back-half renders `bootstrap.json` into the guest from + `--customer-id` + `--hub-password` (the passphrase) — *(not run in this spike; existing, separately + validated)*. +5. **In-guest:** the controller pulls `controller.yaml` via `GET /config/{id}` + `X-Retrieval-Password` + — **proven in Probe 3** — which yields the customer-scoped hub key, domain, and tunnel token; the + tunnel comes up. + +No link 401/403/404'd unexpectedly. The only thing **not** physically run start-to-finish is the +provision step (4), which is deliberately out of a no-destructive-provision spike and is already +validated elsewhere; both its inputs (host credential present; passphrase → config-pull) are proven. + +**Blocking gaps: none.** + +--- + +## 5. Option A vs C — recommendation + +**Two secrets cross to the customer box under A; one under C.** + +- **Option A** (proven): the operator carries **two** secrets to the deployment: the **per-host + api_key** (for the agent, step 3) *and* the **retrieval passphrase** (for the in-guest controller, + steps 4–5). The global key stays at HQ. Works today, no code change. +- **Option C** (proposed): the passphrase-authed pull *also* mints-or-returns the host credential, so + the operator carries **only the passphrase** — and **no global key participates in the deploy path at + all** (it stays a pure HQ-admin key). Fewer secrets in flight; the deploy box never holds an + operator-tier global key even transiently. + +**Feasibility of C (confirmed at source):** `handleConfigRetrieve` already has both prerequisites in +scope — the `customerID` (path param) and the `h.store` handle (which exposes `UpsertHost`, +`GetHostByAPIKey`, `GetHost`, `ListHosts`). The required hub delta: + +1. **Idempotency lookup (new):** there is **no `GetHostByCustomer`** today. `UpsertHost` conflicts on + `host_id` (`store.go:1172-1183`), so naive re-pull would mint a *new* random host each call and + orphan the prior. C should **mint-once-and-reuse**: add a `GetHostByCustomer(customerID)` (or filter + `ListHosts`) so a re-pull returns the existing credential. *(Mint-per-pull = rotation; rejected — it + would silently invalidate a running agent's key on any controller re-pull.)* +2. **Response-shape change — keep it backward-compatible:** config-retrieve currently returns **raw + YAML** consumed by the in-guest controller. Do **not** switch the body to JSON (breaks the controller + curl). Two safe options, in order of preference: + - **(c-pref) Separate passphrase-authed endpoint**, e.g. `POST /api/v1/hosts/enroll` with + `X-Retrieval-Password`, body `{customer_id}`, returns `{host_id, api_key}` (idempotent). Leaves the + proven config-retrieve path **untouched**; the host-bootstrap script calls enroll for the agent and + config-retrieve for the guest seed. Cleanest separation of the host-tier and guest-tier pulls. + - **(c-alt) Side-channel headers** on the existing GET: emit `X-Host-Id` / `X-Host-Api-Key` response + headers alongside the YAML body. The in-guest controller ignores them; the script reads them. Fewer + endpoints but couples two tiers' secrets into one response. + +**Recommendation:** ship **Option A now** (it is proven and needs nothing). Adopt **Option C, variant +(c-pref) — a dedicated passphrase-authed `hosts/enroll` endpoint, mint-once-reuse — at the production +cutover**, on the secret-hygiene argument: it removes the operator-tier global key from the field +deployment entirely and reduces the operator-carried secret set to one (the passphrase the customer +already has). It is a small, isolated change that does not touch the proven controller config-pull. +This matches the existing `handleAdminCreateHost` PROVISIONAL note (`handler.go:595-598`), which already +flags that endpoint for lock-down at the enrollment cutover. + +--- + +## 6. Gap inventory — the checklist the host-bootstrap script spec consumes + +**Exact config-retrieve contract:** `GET https://hub.felhom.eu/api/v1/config/{customer_id}`, header +`X-Retrieval-Password: `; 200 → raw `controller.yaml` (text/yaml), 401 wrong/missing pw, +404 unknown customer. **Capture HTTP status and only treat 2xx as success** (do not pipe a non-2xx body +onward — the spike's recurring lesson; curl `-f`/`-w '%{http_code}'`, check before use). + +**Secrets/values the script must obtain, and their source (A vs C):** + +| Value | Source under A | Source under C | +|-------|----------------|----------------| +| `hub.url` | static (`https://hub.felhom.eu`) | static | +| `customer_id` | operator-supplied (created in hub UI) | operator-supplied | +| `retrieval passphrase` | operator-supplied (hub UI) — SECRET | operator-supplied — SECRET (the **only** carried secret) | +| `host_id` | returned by HQ `POST /admin/hosts` | returned by passphrase-authed enroll | +| host `api_key` | returned by HQ `POST /admin/hosts` — SECRET | returned by passphrase-authed enroll — SECRET | +| **global key** | **HQ-only; never on the box** | **not used in deploy path at all** | +| customer-scoped hub key, domain, tunnel token | inside the pulled `controller.yaml` (not handled directly by the script) | same | + +**Agent config write path on a real host (confirmed):** `/etc/felhom-agent/agent.json` (JSON; flag +`--config`, env `FELHOM_AGENT_CONFIG`), or override the hub fields via env +`FELHOM_AGENT_HUB_URL` / `FELHOM_AGENT_HUB_HOST_ID` / `FELHOM_AGENT_HUB_API_KEY` (`config.go:446-453`). +**Host-specific fields:** `hub.host_id`, `hub.api_key` (+ proxmox endpoint/node/token, which the agent +needs anyway). **Static fields:** `hub.url`. (Per the live-agent notes, the demo box keeps its config +under `/root/.config/felhom-agent/` via `FELHOM_AGENT_CONFIG`; the script must write to whatever path +the agent's systemd unit points `--config`/`FELHOM_AGENT_CONFIG` at — confirm the unit on the target.) + +**How the controller gets `bootstrap.json` (confirmed at source — the script does NOT write it):** the +**agent's provision back-half** renders it (`felhom-agent/internal/provision/backhalf.go:121`, +`doc.go:23-47`), writes it 0600, chowns to the mapped guest-root, and attaches it on the guest config +mount. The script's job is only to invoke the agent's provision with `--customer-id ` and +`--hub-password ` (`cmd/felhom-agent/main.go:131-132`); the agent bakes those into +`bootstrap.json`. The `bootstrap.json` contract (cross-repo, must match the controller's +`internal/bootstrap.Bootstrap`): + +```json +{ "schema": "...", + "customer": { "id": "" }, + "hub": { "url": "", "retrieval_password": "" }, + "local_api": { "endpoint": "", "fingerprint": "", "token": "" } } +``` + +Note it carries **no host key, no customer-scoped hub key, no CF tokens** — those all come from the +controller's pull (`doc.go:24-26`). + +**Unexpected 401/403/404 / values with no source:** none observed. Every refusal was the correct gate. + +--- + +## 7. Cleanup performed + +All throwaway state removed and verified (counts → 0); the real `demo-felhom` customer confirmed still +present and untouched: + +- `DELETE FROM host_reports / hosts / customer_configs WHERE customer_id='spike-day0-2026-06-26'` + (before: 1/1/1 → after: 0/0/0). +- Removed scratch `cmd/spikeprobe` from the build-box agent checkout (`git status` clean) and + `/tmp/spike{probe,-day0}`, `/tmp/*.sh`. +- Uninstalled the temporary `sqlite` package from the hub pod (`apk del sqlite`). +- No production code, no version bump, no push beyond this doc; no image build; no deploy; no provision. + +--- + +## 8. Open questions for the host-bootstrap script spec + +1. **Agent config target path on a production host** — the spec must pin where the script writes + `agent.json` vs. relying on `FELHOM_AGENT_HUB_*` env, and align with the agent's systemd unit + `--config`/`FELHOM_AGENT_CONFIG`. (On the demo box this is `/root/.config/felhom-agent/`; production + may differ.) Also: the agent needs a **Proxmox token** in that same config — where does the script + source it on a freshly-imaged host? (Likely a `pveum` token-create step earlier in the runbook; + confirm it is in the chain.) +2. **Option C decision** — if the team wants single-secret Day-0, the `hosts/enroll` endpoint (variant + c-pref, idempotent mint-once-reuse) is the prerequisite and should be specced/shipped before the + script depends on it. Until then the script uses Option A (HQ pre-mint). +3. **`local_api.endpoint`/`fingerprint` provenance** — these are filled by the agent at provision time, + not by the script; the spec should state the script does not supply them (avoids a false "missing + value" in the script's input list). +4. **Re-run / idempotency of the whole script** — under A, re-running the mint step creates a *second* + host for the same customer (admin mint is not idempotent). The script should either reuse a prior + credential or the runbook should make the mint a one-time HQ step. (C with mint-once-reuse fixes + this structurally.) + +**No secrets appear anywhere in this document.**