# SPIKE — Day-0 first-boot trust handshake (operator-deploy) > **Update 2026-06-28 (BUNDLE slice):** Day-0 now **self-installs the agent** + fetches the golden from > Gitea. The host-bootstrap script (`scripts/felhom-host-install.sh` v1.1.0) fetches the agent binary + > golden from Gitea generic packages and **verifies each sha256 against a hub-vouched artifact manifest** > (`GET /api/v1/artifacts/{id}`, hub v0.16.0) before installing/using them — the fetch credential is the > existing config-retrieve git token (no new credential); the checksum trust root is the **hub**, not > Gitea. The "works on a box that isn't felhom-pve" gap is closed: prerequisites are now just **install > PVE + create the customer in the hub**. The agent runs **non-root** (`felhom-agent` + sudoers). > Remaining follow-ups: per-customer artifact pinning, a `make golden` that resolves-latest, non-root > PBS-key access, the DR mode, and the local-DNS slice. **Date:** 2026-06-26 **Class:** Spike (empirical validation; no production code shipped). Output is this doc only. **Question:** Does the composed first-boot chain — customer-in-hub → host mint → agent host-auth → controller config-pull — work end-to-end **today** with existing mechanisms (option A), and what exact contracts/gaps must the host-bootstrap script own? Secondary: is option C worth a hub change? **Verdict (short):** **Option A works end-to-end today with ZERO new code.** All three trust links plus every wrong-case refusal behaved exactly as the source predicts, exercised against the LIVE hub with the real agent hub-client. Option C (collapse to one secret) is **feasible and recommended for the production cutover** but is a deliberate, small hub change — out of scope for this spike. No blocking gap. The host-bootstrap script can be specified on the contracts below. --- ## 1. Confirmed baselines (live Gitea = local HEAD, verified by `git fetch`) | Repo | `main` @ commit | Version | Role | |------|-----------------|---------|------| | felhom.eu (hub) | `347546e` | v0.13.1 | mints host (`POST /admin/hosts`), serves config-pull | | felhom-agent | `4725396` (code = `166a1c8`, v0.41.0) | v0.41.0 | consumes host credential; authenticates to hub | | felhom-controller | `e0cf78b` | v0.84.0 | in-guest config-pull (the already-proven half) | > The controller advanced to v0.84.0 (prompt said v0.82.0); irrelevant — the controller half was > re-confirmed at the hub contract, which is unchanged. **Exact config-retrieve route string (the spike needed this):** ``` GET /api/v1/config/{customer_id} Header: X-Retrieval-Password: <5-word passphrase> → 200, Content-Type: text/yaml; charset=utf-8, body = the generated controller.yaml (RAW yaml, not JSON) ``` Confirmed at `hub/internal/api/handler.go:187-189` (router) → `handleConfigRetrieve` (`:1341`), and it is the literal command the operator UI prints (`internal/web/templates/config_detail.html:115`): `curl -fsSL https://hub.felhom.eu/api/v1/config/{id} -H "X-Retrieval-Password: {pw}" -o controller.yaml`. > **Correction to the prompt's grounding:** config-retrieve is **GET with the customer_id in the PATH** > and the response is **raw YAML** — not a POST with a `{customer_id}` body returning a JSON > `{ConfigYAML}`. The host-bootstrap script spec must use the GET form. > **Correction #2:** the agent hub env vars are `FELHOM_AGENT_HUB_URL` / `FELHOM_AGENT_HUB_HOST_ID` / > `FELHOM_AGENT_HUB_API_KEY` (prefix `FELHOM_AGENT_HUB_`), confirmed at `config.go:446-453` — **not** > `FELHOM_HUB_*` as the prompt stated. --- ## 2. Environment & method - **Live hub:** `hub.felhom.eu` on k3s (`felhom-system/hub-844754f7b4-...`), SQLite at `/data/hub.db`, config at `/etc/felhom-hub/hub.yaml`. Global key = `api.report_api_key` (read out-of-band from the pod config into a shell var on the build box; **never printed, never written to disk or this doc**). - **Throwaway customer:** `spike-day0-2026-06-26` (domain `spike.demo-felhom.eu`, a throwaway 64-hex customer api_key, a throwaway passphrase, config_json carrying a fake tunnel token). Inserted via the hub's own `customer_configs` schema (the web-form create path is operator-auth-only; the row is byte-identical to what `SaveCustomerConfig` writes). The real `demo-felhom` customer was **not** touched. - **Probes ran from the build box** (`192.168.0.180`, reaches `hub.felhom.eu`). Link 3 (agent auth) was exercised with the **real** `internal/hub` client via a throwaway `cmd/spikeprobe` built from the agent checkout — **not** a hand-rolled curl. No Proxmox, no provision, no real customer data. - All secrets (global key, customer key, passphrase, minted host api_key) stayed resident on the build box in `/tmp/spike-day0/`; only status codes, the non-secret `host_id`, lengths, and boolean content checks were surfaced. --- ## 3. Per-probe results ### Probe 1 — Host mint (link 2, option A) — **WORKS** `POST /api/v1/admin/hosts`, handler `handleAdminCreateHost` (`handler.go:599`), global-key gated. | Case | Request | Expected | Observed | |------|---------|----------|----------| | 1a correct | global key, `{customer_id: spike-day0-…}` | 201 + `{host_id, api_key}` | **201**, `host_id=spike-day0-2026-06-26-6e5dc8`, api_key 64-hex | | host_id default | omitted `host_id` | `-<6hex>` | **matched** `^-[0-9a-f]{6}$` | | row landed | — | `hosts` row resolvable | **present**, api_key length 64 (`GetHostByAPIKey` works — proven in Probe 2) | | 1c wrong | **customer** key (non-global) | 403 | **403** | | 1d wrong | minted **per-host** key (non-global) | 403 | **403** | | 1e wrong | global key, **unknown** customer | 400 `Unknown customer_id` | **400 "Unknown customer_id"** | | 1f wrong | no Authorization header | 403 | **403** | The `!isGlobal` gate (`handler.go:600-604`) and the `cc == nil → 400` customer-existence check (`:624-627`) are correct, not bugs. ### Probe 2 — Agent host-auth (link 3) with the minted credential — **WORKS** Exercised the real `internal/hub.Client` (`client.go`) against the live hub with the Probe-1 credential. | Case | Call | Expected | Observed | |------|------|----------|----------| | 2a correct | `Report()` → `POST /api/v1/host-report` | 2xx + control envelope | **OK**, `status="ok"`, `desired_generation=0` | | 2a effect | — | host-report row lands | **landed**: `host_reports` row, `agent_version=spike-probe` | | 2b correct | `FetchDesiredState()` → `GET /api/v1/hosts/{id}/desired-state` | 2xx, well-formed (empty ok) | **OK**, `generation=0`, 0 guests | | 2c wrong | `Report()` with one flipped key char | 401/403 | **HTTP 401 Unauthorized** | | 2d wrong | `FetchDesiredState()` with empty host_id | client refuses pre-flight | **refused**: "FetchDesiredState requires a configured host_id" (`client.go:119-120`) | | 2e wrong | `HubConfig.Validate()` empty host_id | error before any call | **"hub.host_id is required"** (`config.go:532-534`) | `checkAuthHost` (`handler.go:99-116`) resolves the per-host Bearer via `GetHostByAPIKey`; host-report with `isGlobal=false` requires `body.host_id == authed host_id` (`handler.go:453-455`) — satisfied by the client always sending its configured `host_id`. ### Probe 3 — Controller config-pull (link 1) re-confirm — **WORKS (unchanged)** | Case | Request | Expected | Observed | |------|---------|----------|----------| | 3a correct | `GET /config/{id}` + correct `X-Retrieval-Password` | 200 text/yaml | **200**, `text/yaml; charset=utf-8` | | 3a content | — | domain, hub.url, hub api_key, tunnel token present | **all YES** (domain `spike.demo-felhom.eu`; `hub.api_key` = the customer-scoped key; injected tunnel token; `hub.url`) | | 3b wrong | wrong passphrase | 401 | **401** | | 3c wrong | missing `X-Retrieval-Password` header | 401 | **401** | | 3d wrong | unknown customer (correct-shaped pw) | 404 | **404** | Constant-time passphrase compare (`handler.go:1365`); `cfg == nil → 404` (`:1359-1362`); missing header `→ 401` (`:1347-1351`). The controller half is intact and needs no re-architecting. The generated yaml carries the **customer-scoped** hub `api_key` (`configgen.go:47`, = `cfg.APIKey`) — this is the controller's key, **distinct** from the agent's per-host key minted in Probe 1. --- ## 4. End-to-end verdict on Option A **Option A composes today with ZERO new code.** The full Day-0 chain is exercisable as: 1. **HQ:** operator creates the customer in the hub UI (mints customer api_key + 5-word passphrase). 2. **HQ:** operator mints the host with the **global key** (`POST /admin/hosts`) and obtains `{host_id, host_api_key}`. *(The global key never leaves HQ.)* 3. **On the Proxmox box:** the host-bootstrap script writes the agent config (`/etc/felhom-agent/agent.json` or `FELHOM_AGENT_HUB_*` env) with `{hub.url, host_id, host_api_key}`; the agent authenticates (host-report + desired-state) — **proven in Probe 2**. 4. **Provision:** the agent's provision back-half renders `bootstrap.json` into the guest from `--customer-id` + `--hub-password` (the passphrase) — *(not run in this spike; existing, separately validated)*. 5. **In-guest:** the controller pulls `controller.yaml` via `GET /config/{id}` + `X-Retrieval-Password` — **proven in Probe 3** — which yields the customer-scoped hub key, domain, and tunnel token; the tunnel comes up. No link 401/403/404'd unexpectedly. The only thing **not** physically run start-to-finish is the provision step (4), which is deliberately out of a no-destructive-provision spike and is already validated elsewhere; both its inputs (host credential present; passphrase → config-pull) are proven. **Blocking gaps: none.** --- ## 5. Option A vs C — recommendation **Two secrets cross to the customer box under A; one under C.** - **Option A** (proven): the operator carries **two** secrets to the deployment: the **per-host api_key** (for the agent, step 3) *and* the **retrieval passphrase** (for the in-guest controller, steps 4–5). The global key stays at HQ. Works today, no code change. - **Option C** (proposed): the passphrase-authed pull *also* mints-or-returns the host credential, so the operator carries **only the passphrase** — and **no global key participates in the deploy path at all** (it stays a pure HQ-admin key). Fewer secrets in flight; the deploy box never holds an operator-tier global key even transiently. **Feasibility of C (confirmed at source):** `handleConfigRetrieve` already has both prerequisites in scope — the `customerID` (path param) and the `h.store` handle (which exposes `UpsertHost`, `GetHostByAPIKey`, `GetHost`, `ListHosts`). The required hub delta: 1. **Idempotency lookup (new):** there is **no `GetHostByCustomer`** today. `UpsertHost` conflicts on `host_id` (`store.go:1172-1183`), so naive re-pull would mint a *new* random host each call and orphan the prior. C should **mint-once-and-reuse**: add a `GetHostByCustomer(customerID)` (or filter `ListHosts`) so a re-pull returns the existing credential. *(Mint-per-pull = rotation; rejected — it would silently invalidate a running agent's key on any controller re-pull.)* 2. **Response-shape change — keep it backward-compatible:** config-retrieve currently returns **raw YAML** consumed by the in-guest controller. Do **not** switch the body to JSON (breaks the controller curl). Two safe options, in order of preference: - **(c-pref) Separate passphrase-authed endpoint**, e.g. `POST /api/v1/hosts/enroll` with `X-Retrieval-Password`, body `{customer_id}`, returns `{host_id, api_key}` (idempotent). Leaves the proven config-retrieve path **untouched**; the host-bootstrap script calls enroll for the agent and config-retrieve for the guest seed. Cleanest separation of the host-tier and guest-tier pulls. - **(c-alt) Side-channel headers** on the existing GET: emit `X-Host-Id` / `X-Host-Api-Key` response headers alongside the YAML body. The in-guest controller ignores them; the script reads them. Fewer endpoints but couples two tiers' secrets into one response. > **UPDATE 2026-06-26 — c-pref SHIPPED in hub v0.14.0.** The dedicated passphrase-authed endpoint > landed as `POST /api/v1/host-enroll` (mint-once-reuse via the new `Store.GetHostByCustomer`); the > proven `GET /config/{id}` and the global-key `POST /admin/hosts` escape hatch are unchanged. See the > hub `CHANGELOG.md` v0.14.0 + `README.md` "Host Enrollment (Day-0, option C)". The route is the exact > `path == "/host-enroll"` form (not `/hosts/enroll`, which would collide with the `/hosts/` per-host > prefix routes). > > **UPDATE 2026-06-26 — Day-0 host-bootstrap script SHIPPED.** This gap inventory is now implemented > end-to-end by `scripts/felhom-host-install.sh` v1.0.0 (provision mode): pveum token → `host-enroll` > → agent config → `--selftest=provision` → verify. Live-validated by an authorized wipe + re-provision > of demo guest 9201 (controller config-pull + public tunnel `HTTP 200` + host-report confirmed). DR > mode is a documented stub (10D). See `scripts/README.md` + `REPORT.md`. **Recommendation:** ship **Option A now** (it is proven and needs nothing). Adopt **Option C, variant (c-pref) — a dedicated passphrase-authed `host-enroll` endpoint, mint-once-reuse — at the production cutover**, on the secret-hygiene argument: it removes the operator-tier global key from the field deployment entirely and reduces the operator-carried secret set to one (the passphrase the customer already has). It is a small, isolated change that does not touch the proven controller config-pull. This matches the existing `handleAdminCreateHost` PROVISIONAL note (`handler.go:595-598`), which already flags that endpoint for lock-down at the enrollment cutover. --- ## 6. Gap inventory — the checklist the host-bootstrap script spec consumes **Exact config-retrieve contract:** `GET https://hub.felhom.eu/api/v1/config/{customer_id}`, header `X-Retrieval-Password: `; 200 → raw `controller.yaml` (text/yaml), 401 wrong/missing pw, 404 unknown customer. **Capture HTTP status and only treat 2xx as success** (do not pipe a non-2xx body onward — the spike's recurring lesson; curl `-f`/`-w '%{http_code}'`, check before use). **Secrets/values the script must obtain, and their source (A vs C):** | Value | Source under A | Source under C | |-------|----------------|----------------| | `hub.url` | static (`https://hub.felhom.eu`) | static | | `customer_id` | operator-supplied (created in hub UI) | operator-supplied | | `retrieval passphrase` | operator-supplied (hub UI) — SECRET | operator-supplied — SECRET (the **only** carried secret) | | `host_id` | returned by HQ `POST /admin/hosts` | returned by passphrase-authed enroll | | host `api_key` | returned by HQ `POST /admin/hosts` — SECRET | returned by passphrase-authed enroll — SECRET | | **global key** | **HQ-only; never on the box** | **not used in deploy path at all** | | customer-scoped hub key, domain, tunnel token | inside the pulled `controller.yaml` (not handled directly by the script) | same | **Agent config write path on a real host (confirmed):** `/etc/felhom-agent/agent.json` (JSON; flag `--config`, env `FELHOM_AGENT_CONFIG`), or override the hub fields via env `FELHOM_AGENT_HUB_URL` / `FELHOM_AGENT_HUB_HOST_ID` / `FELHOM_AGENT_HUB_API_KEY` (`config.go:446-453`). **Host-specific fields:** `hub.host_id`, `hub.api_key` (+ proxmox endpoint/node/token, which the agent needs anyway). **Static fields:** `hub.url`. (Per the live-agent notes, the demo box keeps its config under `/root/.config/felhom-agent/` via `FELHOM_AGENT_CONFIG`; the script must write to whatever path the agent's systemd unit points `--config`/`FELHOM_AGENT_CONFIG` at — confirm the unit on the target.) **How the controller gets `bootstrap.json` (confirmed at source — the script does NOT write it):** the **agent's provision back-half** renders it (`felhom-agent/internal/provision/backhalf.go:121`, `doc.go:23-47`), writes it 0600, chowns to the mapped guest-root, and attaches it on the guest config mount. The script's job is only to invoke the agent's provision with `--customer-id ` and `--hub-password ` (`cmd/felhom-agent/main.go:131-132`); the agent bakes those into `bootstrap.json`. The `bootstrap.json` contract (cross-repo, must match the controller's `internal/bootstrap.Bootstrap`): ```json { "schema": "...", "customer": { "id": "" }, "hub": { "url": "", "retrieval_password": "" }, "local_api": { "endpoint": "", "fingerprint": "", "token": "" } } ``` Note it carries **no host key, no customer-scoped hub key, no CF tokens** — those all come from the controller's pull (`doc.go:24-26`). **Unexpected 401/403/404 / values with no source:** none observed. Every refusal was the correct gate. --- ## 7. Cleanup performed All throwaway state removed and verified (counts → 0); the real `demo-felhom` customer confirmed still present and untouched: - `DELETE FROM host_reports / hosts / customer_configs WHERE customer_id='spike-day0-2026-06-26'` (before: 1/1/1 → after: 0/0/0). - Removed scratch `cmd/spikeprobe` from the build-box agent checkout (`git status` clean) and `/tmp/spike{probe,-day0}`, `/tmp/*.sh`. - Uninstalled the temporary `sqlite` package from the hub pod (`apk del sqlite`). - No production code, no version bump, no push beyond this doc; no image build; no deploy; no provision. --- ## 8. Open questions for the host-bootstrap script spec 1. **Agent config target path on a production host** — the spec must pin where the script writes `agent.json` vs. relying on `FELHOM_AGENT_HUB_*` env, and align with the agent's systemd unit `--config`/`FELHOM_AGENT_CONFIG`. (On the demo box this is `/root/.config/felhom-agent/`; production may differ.) Also: the agent needs a **Proxmox token** in that same config — where does the script source it on a freshly-imaged host? (Likely a `pveum` token-create step earlier in the runbook; confirm it is in the chain.) 2. **Option C decision** — if the team wants single-secret Day-0, the `hosts/enroll` endpoint (variant c-pref, idempotent mint-once-reuse) is the prerequisite and should be specced/shipped before the script depends on it. Until then the script uses Option A (HQ pre-mint). 3. **`local_api.endpoint`/`fingerprint` provenance** — these are filled by the agent at provision time, not by the script; the spec should state the script does not supply them (avoids a false "missing value" in the script's input list). 4. **Re-run / idempotency of the whole script** — under A, re-running the mint step creates a *second* host for the same customer (admin mint is not idempotent). The script should either reuse a prior credential or the runbook should make the mint a one-time HQ step. (C with mint-once-reuse fixes this structurally.) **No secrets appear anywhere in this document.**