# SPIKE — Day-0 first-boot trust handshake (operator-deploy)

> **Update 2026-06-28 (BUNDLE slice):** Day-0 now **self-installs the agent** + fetches the golden from
> Gitea. The host-bootstrap script (`scripts/felhom-host-install.sh` v1.1.0) fetches the agent binary +
> golden from Gitea generic packages and **verifies each sha256 against a hub-vouched artifact manifest**
> (`GET /api/v1/artifacts/{id}`, hub v0.16.0) before installing/using them — the fetch credential is the
> existing config-retrieve git token (no new credential); the checksum trust root is the **hub**, not
> Gitea. The "works on a box that isn't felhom-pve" gap is closed: prerequisites are now just **install
> PVE + create the customer in the hub**. The agent runs **non-root** (`felhom-agent` + sudoers).
> Remaining follow-ups: per-customer artifact pinning, a `make golden` that resolves-latest, non-root
> PBS-key access, the DR mode, and the local-DNS slice.

**Date:** 2026-06-26
**Class:** Spike (empirical validation; no production code shipped). Output is this doc only.
**Question:** Does the composed first-boot chain — customer-in-hub → host mint → agent host-auth →
controller config-pull — work end-to-end **today** with existing mechanisms (option A), and what
exact contracts/gaps must the host-bootstrap script own? Secondary: is option C worth a hub change?

**Verdict (short):** **Option A works end-to-end today with ZERO new code.** All three trust links
plus every wrong-case refusal behaved exactly as the source predicts, exercised against the LIVE hub
with the real agent hub-client. Option C (collapse to one secret) is **feasible and recommended for
the production cutover** but is a deliberate, small hub change — out of scope for this spike. No
blocking gap. The host-bootstrap script can be specified on the contracts below.

---

## 1. Confirmed baselines (live Gitea = local HEAD, verified by `git fetch`)

| Repo | `main` @ commit | Version | Role |
|------|-----------------|---------|------|
| felhom.eu (hub) | `347546e` | v0.13.1 | mints host (`POST /admin/hosts`), serves config-pull |
| felhom-agent | `4725396` (code = `166a1c8`, v0.41.0) | v0.41.0 | consumes host credential; authenticates to hub |
| felhom-controller | `e0cf78b` | v0.84.0 | in-guest config-pull (the already-proven half) |

> The controller advanced to v0.84.0 (prompt said v0.82.0); irrelevant — the controller half was
> re-confirmed at the hub contract, which is unchanged.

**Exact config-retrieve route string (the spike needed this):**

```
GET /api/v1/config/{customer_id}
Header: X-Retrieval-Password: <5-word passphrase>
→ 200, Content-Type: text/yaml; charset=utf-8, body = the generated controller.yaml (RAW yaml, not JSON)
```

Confirmed at `hub/internal/api/handler.go:187-189` (router) → `handleConfigRetrieve` (`:1341`), and it is
the literal command the operator UI prints (`internal/web/templates/config_detail.html:115`):
`curl -fsSL https://hub.felhom.eu/api/v1/config/{id} -H "X-Retrieval-Password: {pw}" -o controller.yaml`.

> **Correction to the prompt's grounding:** config-retrieve is **GET with the customer_id in the PATH**
> and the response is **raw YAML** — not a POST with a `{customer_id}` body returning a JSON
> `{ConfigYAML}`. The host-bootstrap script spec must use the GET form.

> **Correction #2:** the agent hub env vars are `FELHOM_AGENT_HUB_URL` / `FELHOM_AGENT_HUB_HOST_ID` /
> `FELHOM_AGENT_HUB_API_KEY` (prefix `FELHOM_AGENT_HUB_`), confirmed at `config.go:446-453` — **not**
> `FELHOM_HUB_*` as the prompt stated.

---

## 2. Environment & method

- **Live hub:** `hub.felhom.eu` on k3s (`felhom-system/hub-844754f7b4-...`), SQLite at `/data/hub.db`,
  config at `/etc/felhom-hub/hub.yaml`. Global key = `api.report_api_key` (read out-of-band from the
  pod config into a shell var on the build box; **never printed, never written to disk or this doc**).
- **Throwaway customer:** `spike-day0-2026-06-26` (domain `spike.demo-felhom.eu`, a throwaway 64-hex
  customer api_key, a throwaway passphrase, config_json carrying a fake tunnel token). Inserted via the
  hub's own `customer_configs` schema (the web-form create path is operator-auth-only; the row is
  byte-identical to what `SaveCustomerConfig` writes). The real `demo-felhom` customer was **not**
  touched.
- **Probes ran from the build box** (`192.168.0.180`, reaches `hub.felhom.eu`). Link 3 (agent auth) was
  exercised with the **real** `internal/hub` client via a throwaway `cmd/spikeprobe` built from the
  agent checkout — **not** a hand-rolled curl. No Proxmox, no provision, no real customer data.
- All secrets (global key, customer key, passphrase, minted host api_key) stayed resident on the build
  box in `/tmp/spike-day0/`; only status codes, the non-secret `host_id`, lengths, and boolean content
  checks were surfaced.

---

## 3. Per-probe results

### Probe 1 — Host mint (link 2, option A) — **WORKS**

`POST /api/v1/admin/hosts`, handler `handleAdminCreateHost` (`handler.go:599`), global-key gated.

| Case | Request | Expected | Observed |
|------|---------|----------|----------|
| 1a correct | global key, `{customer_id: spike-day0-…}` | 201 + `{host_id, api_key}` | **201**, `host_id=spike-day0-2026-06-26-6e5dc8`, api_key 64-hex |
| host_id default | omitted `host_id` | `<customer>-<6hex>` | **matched** `^<customer>-[0-9a-f]{6}$` |
| row landed | — | `hosts` row resolvable | **present**, api_key length 64 (`GetHostByAPIKey` works — proven in Probe 2) |
| 1c wrong | **customer** key (non-global) | 403 | **403** |
| 1d wrong | minted **per-host** key (non-global) | 403 | **403** |
| 1e wrong | global key, **unknown** customer | 400 `Unknown customer_id` | **400 "Unknown customer_id"** |
| 1f wrong | no Authorization header | 403 | **403** |

The `!isGlobal` gate (`handler.go:600-604`) and the `cc == nil → 400` customer-existence check
(`:624-627`) are correct, not bugs.

### Probe 2 — Agent host-auth (link 3) with the minted credential — **WORKS**

Exercised the real `internal/hub.Client` (`client.go`) against the live hub with the Probe-1 credential.

| Case | Call | Expected | Observed |
|------|------|----------|----------|
| 2a correct | `Report()` → `POST /api/v1/host-report` | 2xx + control envelope | **OK**, `status="ok"`, `desired_generation=0` |
| 2a effect | — | host-report row lands | **landed**: `host_reports` row, `agent_version=spike-probe` |
| 2b correct | `FetchDesiredState()` → `GET /api/v1/hosts/{id}/desired-state` | 2xx, well-formed (empty ok) | **OK**, `generation=0`, 0 guests |
| 2c wrong | `Report()` with one flipped key char | 401/403 | **HTTP 401 Unauthorized** |
| 2d wrong | `FetchDesiredState()` with empty host_id | client refuses pre-flight | **refused**: "FetchDesiredState requires a configured host_id" (`client.go:119-120`) |
| 2e wrong | `HubConfig.Validate()` empty host_id | error before any call | **"hub.host_id is required"** (`config.go:532-534`) |

`checkAuthHost` (`handler.go:99-116`) resolves the per-host Bearer via `GetHostByAPIKey`; host-report
with `isGlobal=false` requires `body.host_id == authed host_id` (`handler.go:453-455`) — satisfied by
the client always sending its configured `host_id`.

### Probe 3 — Controller config-pull (link 1) re-confirm — **WORKS (unchanged)**

| Case | Request | Expected | Observed |
|------|---------|----------|----------|
| 3a correct | `GET /config/{id}` + correct `X-Retrieval-Password` | 200 text/yaml | **200**, `text/yaml; charset=utf-8` |
| 3a content | — | domain, hub.url, hub api_key, tunnel token present | **all YES** (domain `spike.demo-felhom.eu`; `hub.api_key` = the customer-scoped key; injected tunnel token; `hub.url`) |
| 3b wrong | wrong passphrase | 401 | **401** |
| 3c wrong | missing `X-Retrieval-Password` header | 401 | **401** |
| 3d wrong | unknown customer (correct-shaped pw) | 404 | **404** |

Constant-time passphrase compare (`handler.go:1365`); `cfg == nil → 404` (`:1359-1362`); missing header
`→ 401` (`:1347-1351`). The controller half is intact and needs no re-architecting. The generated yaml
carries the **customer-scoped** hub `api_key` (`configgen.go:47`, = `cfg.APIKey`) — this is the
controller's key, **distinct** from the agent's per-host key minted in Probe 1.

---

## 4. End-to-end verdict on Option A

**Option A composes today with ZERO new code.** The full Day-0 chain is exercisable as:

1. **HQ:** operator creates the customer in the hub UI (mints customer api_key + 5-word passphrase).
2. **HQ:** operator mints the host with the **global key** (`POST /admin/hosts`) and obtains
   `{host_id, host_api_key}`. *(The global key never leaves HQ.)*
3. **On the Proxmox box:** the host-bootstrap script writes the agent config (`/etc/felhom-agent/agent.json`
   or `FELHOM_AGENT_HUB_*` env) with `{hub.url, host_id, host_api_key}`; the agent authenticates
   (host-report + desired-state) — **proven in Probe 2**.
4. **Provision:** the agent's provision back-half renders `bootstrap.json` into the guest from
   `--customer-id` + `--hub-password` (the passphrase) — *(not run in this spike; existing, separately
   validated)*.
5. **In-guest:** the controller pulls `controller.yaml` via `GET /config/{id}` + `X-Retrieval-Password`
   — **proven in Probe 3** — which yields the customer-scoped hub key, domain, and tunnel token; the
   tunnel comes up.

No link 401/403/404'd unexpectedly. The only thing **not** physically run start-to-finish is the
provision step (4), which is deliberately out of a no-destructive-provision spike and is already
validated elsewhere; both its inputs (host credential present; passphrase → config-pull) are proven.

**Blocking gaps: none.**

---

## 5. Option A vs C — recommendation

**Two secrets cross to the customer box under A; one under C.**

- **Option A** (proven): the operator carries **two** secrets to the deployment: the **per-host
  api_key** (for the agent, step 3) *and* the **retrieval passphrase** (for the in-guest controller,
  steps 4–5). The global key stays at HQ. Works today, no code change.
- **Option C** (proposed): the passphrase-authed pull *also* mints-or-returns the host credential, so
  the operator carries **only the passphrase** — and **no global key participates in the deploy path at
  all** (it stays a pure HQ-admin key). Fewer secrets in flight; the deploy box never holds an
  operator-tier global key even transiently.

**Feasibility of C (confirmed at source):** `handleConfigRetrieve` already has both prerequisites in
scope — the `customerID` (path param) and the `h.store` handle (which exposes `UpsertHost`,
`GetHostByAPIKey`, `GetHost`, `ListHosts`). The required hub delta:

1. **Idempotency lookup (new):** there is **no `GetHostByCustomer`** today. `UpsertHost` conflicts on
   `host_id` (`store.go:1172-1183`), so naive re-pull would mint a *new* random host each call and
   orphan the prior. C should **mint-once-and-reuse**: add a `GetHostByCustomer(customerID)` (or filter
   `ListHosts`) so a re-pull returns the existing credential. *(Mint-per-pull = rotation; rejected — it
   would silently invalidate a running agent's key on any controller re-pull.)*
2. **Response-shape change — keep it backward-compatible:** config-retrieve currently returns **raw
   YAML** consumed by the in-guest controller. Do **not** switch the body to JSON (breaks the controller
   curl). Two safe options, in order of preference:
   - **(c-pref) Separate passphrase-authed endpoint**, e.g. `POST /api/v1/hosts/enroll` with
     `X-Retrieval-Password`, body `{customer_id}`, returns `{host_id, api_key}` (idempotent). Leaves the
     proven config-retrieve path **untouched**; the host-bootstrap script calls enroll for the agent and
     config-retrieve for the guest seed. Cleanest separation of the host-tier and guest-tier pulls.
   - **(c-alt) Side-channel headers** on the existing GET: emit `X-Host-Id` / `X-Host-Api-Key` response
     headers alongside the YAML body. The in-guest controller ignores them; the script reads them. Fewer
     endpoints but couples two tiers' secrets into one response.

> **UPDATE 2026-06-26 — c-pref SHIPPED in hub v0.14.0.** The dedicated passphrase-authed endpoint
> landed as `POST /api/v1/host-enroll` (mint-once-reuse via the new `Store.GetHostByCustomer`); the
> proven `GET /config/{id}` and the global-key `POST /admin/hosts` escape hatch are unchanged. See the
> hub `CHANGELOG.md` v0.14.0 + `README.md` "Host Enrollment (Day-0, option C)". The route is the exact
> `path == "/host-enroll"` form (not `/hosts/enroll`, which would collide with the `/hosts/` per-host
> prefix routes).
>
> **UPDATE 2026-06-26 — Day-0 host-bootstrap script SHIPPED.** This gap inventory is now implemented
> end-to-end by `scripts/felhom-host-install.sh` v1.0.0 (provision mode): pveum token → `host-enroll`
> → agent config → `--selftest=provision` → verify. Live-validated by an authorized wipe + re-provision
> of demo guest 9201 (controller config-pull + public tunnel `HTTP 200` + host-report confirmed). DR
> mode is a documented stub (10D). See `scripts/README.md` + `REPORT.md`.

**Recommendation:** ship **Option A now** (it is proven and needs nothing). Adopt **Option C, variant
(c-pref) — a dedicated passphrase-authed `host-enroll` endpoint, mint-once-reuse — at the production
cutover**, on the secret-hygiene argument: it removes the operator-tier global key from the field
deployment entirely and reduces the operator-carried secret set to one (the passphrase the customer
already has). It is a small, isolated change that does not touch the proven controller config-pull.
This matches the existing `handleAdminCreateHost` PROVISIONAL note (`handler.go:595-598`), which already
flags that endpoint for lock-down at the enrollment cutover.

---

## 6. Gap inventory — the checklist the host-bootstrap script spec consumes

**Exact config-retrieve contract:** `GET https://hub.felhom.eu/api/v1/config/{customer_id}`, header
`X-Retrieval-Password: <passphrase>`; 200 → raw `controller.yaml` (text/yaml), 401 wrong/missing pw,
404 unknown customer. **Capture HTTP status and only treat 2xx as success** (do not pipe a non-2xx body
onward — the spike's recurring lesson; curl `-f`/`-w '%{http_code}'`, check before use).

**Secrets/values the script must obtain, and their source (A vs C):**

| Value | Source under A | Source under C |
|-------|----------------|----------------|
| `hub.url` | static (`https://hub.felhom.eu`) | static |
| `customer_id` | operator-supplied (created in hub UI) | operator-supplied |
| `retrieval passphrase` | operator-supplied (hub UI) — SECRET | operator-supplied — SECRET (the **only** carried secret) |
| `host_id` | returned by HQ `POST /admin/hosts` | returned by passphrase-authed enroll |
| host `api_key` | returned by HQ `POST /admin/hosts` — SECRET | returned by passphrase-authed enroll — SECRET |
| **global key** | **HQ-only; never on the box** | **not used in deploy path at all** |
| customer-scoped hub key, domain, tunnel token | inside the pulled `controller.yaml` (not handled directly by the script) | same |

**Agent config write path on a real host (confirmed):** `/etc/felhom-agent/agent.json` (JSON; flag
`--config`, env `FELHOM_AGENT_CONFIG`), or override the hub fields via env
`FELHOM_AGENT_HUB_URL` / `FELHOM_AGENT_HUB_HOST_ID` / `FELHOM_AGENT_HUB_API_KEY` (`config.go:446-453`).
**Host-specific fields:** `hub.host_id`, `hub.api_key` (+ proxmox endpoint/node/token, which the agent
needs anyway). **Static fields:** `hub.url`. (Per the live-agent notes, the demo box keeps its config
under `/root/.config/felhom-agent/` via `FELHOM_AGENT_CONFIG`; the script must write to whatever path
the agent's systemd unit points `--config`/`FELHOM_AGENT_CONFIG` at — confirm the unit on the target.)

**How the controller gets `bootstrap.json` (confirmed at source — the script does NOT write it):** the
**agent's provision back-half** renders it (`felhom-agent/internal/provision/backhalf.go:121`,
`doc.go:23-47`), writes it 0600, chowns to the mapped guest-root, and attaches it on the guest config
mount. The script's job is only to invoke the agent's provision with `--customer-id <id>` and
`--hub-password <passphrase>` (`cmd/felhom-agent/main.go:131-132`); the agent bakes those into
`bootstrap.json`. The `bootstrap.json` contract (cross-repo, must match the controller's
`internal/bootstrap.Bootstrap`):

```json
{ "schema": "...",
  "customer": { "id": "<customer_id>" },
  "hub": { "url": "<hub_url>", "retrieval_password": "<passphrase, SECRET>" },
  "local_api": { "endpoint": "<host-bridge ip:port>", "fingerprint": "<agent leaf SHA-256>", "token": "<per-guest bearer, SECRET>" } }
```

Note it carries **no host key, no customer-scoped hub key, no CF tokens** — those all come from the
controller's pull (`doc.go:24-26`).

**Unexpected 401/403/404 / values with no source:** none observed. Every refusal was the correct gate.

---

## 7. Cleanup performed

All throwaway state removed and verified (counts → 0); the real `demo-felhom` customer confirmed still
present and untouched:

- `DELETE FROM host_reports / hosts / customer_configs WHERE customer_id='spike-day0-2026-06-26'`
  (before: 1/1/1 → after: 0/0/0).
- Removed scratch `cmd/spikeprobe` from the build-box agent checkout (`git status` clean) and
  `/tmp/spike{probe,-day0}`, `/tmp/*.sh`.
- Uninstalled the temporary `sqlite` package from the hub pod (`apk del sqlite`).
- No production code, no version bump, no push beyond this doc; no image build; no deploy; no provision.

---

## 8. Open questions for the host-bootstrap script spec

1. **Agent config target path on a production host** — the spec must pin where the script writes
   `agent.json` vs. relying on `FELHOM_AGENT_HUB_*` env, and align with the agent's systemd unit
   `--config`/`FELHOM_AGENT_CONFIG`. (On the demo box this is `/root/.config/felhom-agent/`; production
   may differ.) Also: the agent needs a **Proxmox token** in that same config — where does the script
   source it on a freshly-imaged host? (Likely a `pveum` token-create step earlier in the runbook;
   confirm it is in the chain.)
2. **Option C decision** — if the team wants single-secret Day-0, the `hosts/enroll` endpoint (variant
   c-pref, idempotent mint-once-reuse) is the prerequisite and should be specced/shipped before the
   script depends on it. Until then the script uses Option A (HQ pre-mint).
3. **`local_api.endpoint`/`fingerprint` provenance** — these are filled by the agent at provision time,
   not by the script; the spec should state the script does not supply them (avoids a false "missing
   value" in the script's input list).
4. **Re-run / idempotency of the whole script** — under A, re-running the mint step creates a *second*
   host for the same customer (admin mint is not idempotent). The script should either reuse a prior
   credential or the runbook should make the mint a one-time HQ step. (C with mint-once-reuse fixes
   this structurally.)

**No secrets appear anywhere in this document.**
