295 lines
19 KiB
Markdown
295 lines
19 KiB
Markdown
# SPIKE — Day-0 first-boot trust handshake (operator-deploy)
|
||
|
||
> **Update 2026-06-28 (BUNDLE slice):** Day-0 now **self-installs the agent** + fetches the golden from
|
||
> Gitea. The host-bootstrap script (`scripts/felhom-host-install.sh` v1.1.0) fetches the agent binary +
|
||
> golden from Gitea generic packages and **verifies each sha256 against a hub-vouched artifact manifest**
|
||
> (`GET /api/v1/artifacts/{id}`, hub v0.16.0) before installing/using them — the fetch credential is the
|
||
> existing config-retrieve git token (no new credential); the checksum trust root is the **hub**, not
|
||
> Gitea. The "works on a box that isn't felhom-pve" gap is closed: prerequisites are now just **install
|
||
> PVE + create the customer in the hub**. The agent runs **non-root** (`felhom-agent` + sudoers).
|
||
> Remaining follow-ups: per-customer artifact pinning, a `make golden` that resolves-latest, non-root
|
||
> PBS-key access, the DR mode, and the local-DNS slice.
|
||
|
||
**Date:** 2026-06-26
|
||
**Class:** Spike (empirical validation; no production code shipped). Output is this doc only.
|
||
**Question:** Does the composed first-boot chain — customer-in-hub → host mint → agent host-auth →
|
||
controller config-pull — work end-to-end **today** with existing mechanisms (option A), and what
|
||
exact contracts/gaps must the host-bootstrap script own? Secondary: is option C worth a hub change?
|
||
|
||
**Verdict (short):** **Option A works end-to-end today with ZERO new code.** All three trust links
|
||
plus every wrong-case refusal behaved exactly as the source predicts, exercised against the LIVE hub
|
||
with the real agent hub-client. Option C (collapse to one secret) is **feasible and recommended for
|
||
the production cutover** but is a deliberate, small hub change — out of scope for this spike. No
|
||
blocking gap. The host-bootstrap script can be specified on the contracts below.
|
||
|
||
---
|
||
|
||
## 1. Confirmed baselines (live Gitea = local HEAD, verified by `git fetch`)
|
||
|
||
| Repo | `main` @ commit | Version | Role |
|
||
|------|-----------------|---------|------|
|
||
| felhom.eu (hub) | `347546e` | v0.13.1 | mints host (`POST /admin/hosts`), serves config-pull |
|
||
| felhom-agent | `4725396` (code = `166a1c8`, v0.41.0) | v0.41.0 | consumes host credential; authenticates to hub |
|
||
| felhom-controller | `e0cf78b` | v0.84.0 | in-guest config-pull (the already-proven half) |
|
||
|
||
> The controller advanced to v0.84.0 (prompt said v0.82.0); irrelevant — the controller half was
|
||
> re-confirmed at the hub contract, which is unchanged.
|
||
|
||
**Exact config-retrieve route string (the spike needed this):**
|
||
|
||
```
|
||
GET /api/v1/config/{customer_id}
|
||
Header: X-Retrieval-Password: <5-word passphrase>
|
||
→ 200, Content-Type: text/yaml; charset=utf-8, body = the generated controller.yaml (RAW yaml, not JSON)
|
||
```
|
||
|
||
Confirmed at `hub/internal/api/handler.go:187-189` (router) → `handleConfigRetrieve` (`:1341`), and it is
|
||
the literal command the operator UI prints (`internal/web/templates/config_detail.html:115`):
|
||
`curl -fsSL https://hub.felhom.eu/api/v1/config/{id} -H "X-Retrieval-Password: {pw}" -o controller.yaml`.
|
||
|
||
> **Correction to the prompt's grounding:** config-retrieve is **GET with the customer_id in the PATH**
|
||
> and the response is **raw YAML** — not a POST with a `{customer_id}` body returning a JSON
|
||
> `{ConfigYAML}`. The host-bootstrap script spec must use the GET form.
|
||
|
||
> **Correction #2:** the agent hub env vars are `FELHOM_AGENT_HUB_URL` / `FELHOM_AGENT_HUB_HOST_ID` /
|
||
> `FELHOM_AGENT_HUB_API_KEY` (prefix `FELHOM_AGENT_HUB_`), confirmed at `config.go:446-453` — **not**
|
||
> `FELHOM_HUB_*` as the prompt stated.
|
||
|
||
---
|
||
|
||
## 2. Environment & method
|
||
|
||
- **Live hub:** `hub.felhom.eu` on k3s (`felhom-system/hub-844754f7b4-...`), SQLite at `/data/hub.db`,
|
||
config at `/etc/felhom-hub/hub.yaml`. Global key = `api.report_api_key` (read out-of-band from the
|
||
pod config into a shell var on the build box; **never printed, never written to disk or this doc**).
|
||
- **Throwaway customer:** `spike-day0-2026-06-26` (domain `spike.demo-felhom.eu`, a throwaway 64-hex
|
||
customer api_key, a throwaway passphrase, config_json carrying a fake tunnel token). Inserted via the
|
||
hub's own `customer_configs` schema (the web-form create path is operator-auth-only; the row is
|
||
byte-identical to what `SaveCustomerConfig` writes). The real `demo-felhom` customer was **not**
|
||
touched.
|
||
- **Probes ran from the build box** (`192.168.0.180`, reaches `hub.felhom.eu`). Link 3 (agent auth) was
|
||
exercised with the **real** `internal/hub` client via a throwaway `cmd/spikeprobe` built from the
|
||
agent checkout — **not** a hand-rolled curl. No Proxmox, no provision, no real customer data.
|
||
- All secrets (global key, customer key, passphrase, minted host api_key) stayed resident on the build
|
||
box in `/tmp/spike-day0/`; only status codes, the non-secret `host_id`, lengths, and boolean content
|
||
checks were surfaced.
|
||
|
||
---
|
||
|
||
## 3. Per-probe results
|
||
|
||
### Probe 1 — Host mint (link 2, option A) — **WORKS**
|
||
|
||
`POST /api/v1/admin/hosts`, handler `handleAdminCreateHost` (`handler.go:599`), global-key gated.
|
||
|
||
| Case | Request | Expected | Observed |
|
||
|------|---------|----------|----------|
|
||
| 1a correct | global key, `{customer_id: spike-day0-…}` | 201 + `{host_id, api_key}` | **201**, `host_id=spike-day0-2026-06-26-6e5dc8`, api_key 64-hex |
|
||
| host_id default | omitted `host_id` | `<customer>-<6hex>` | **matched** `^<customer>-[0-9a-f]{6}$` |
|
||
| row landed | — | `hosts` row resolvable | **present**, api_key length 64 (`GetHostByAPIKey` works — proven in Probe 2) |
|
||
| 1c wrong | **customer** key (non-global) | 403 | **403** |
|
||
| 1d wrong | minted **per-host** key (non-global) | 403 | **403** |
|
||
| 1e wrong | global key, **unknown** customer | 400 `Unknown customer_id` | **400 "Unknown customer_id"** |
|
||
| 1f wrong | no Authorization header | 403 | **403** |
|
||
|
||
The `!isGlobal` gate (`handler.go:600-604`) and the `cc == nil → 400` customer-existence check
|
||
(`:624-627`) are correct, not bugs.
|
||
|
||
### Probe 2 — Agent host-auth (link 3) with the minted credential — **WORKS**
|
||
|
||
Exercised the real `internal/hub.Client` (`client.go`) against the live hub with the Probe-1 credential.
|
||
|
||
| Case | Call | Expected | Observed |
|
||
|------|------|----------|----------|
|
||
| 2a correct | `Report()` → `POST /api/v1/host-report` | 2xx + control envelope | **OK**, `status="ok"`, `desired_generation=0` |
|
||
| 2a effect | — | host-report row lands | **landed**: `host_reports` row, `agent_version=spike-probe` |
|
||
| 2b correct | `FetchDesiredState()` → `GET /api/v1/hosts/{id}/desired-state` | 2xx, well-formed (empty ok) | **OK**, `generation=0`, 0 guests |
|
||
| 2c wrong | `Report()` with one flipped key char | 401/403 | **HTTP 401 Unauthorized** |
|
||
| 2d wrong | `FetchDesiredState()` with empty host_id | client refuses pre-flight | **refused**: "FetchDesiredState requires a configured host_id" (`client.go:119-120`) |
|
||
| 2e wrong | `HubConfig.Validate()` empty host_id | error before any call | **"hub.host_id is required"** (`config.go:532-534`) |
|
||
|
||
`checkAuthHost` (`handler.go:99-116`) resolves the per-host Bearer via `GetHostByAPIKey`; host-report
|
||
with `isGlobal=false` requires `body.host_id == authed host_id` (`handler.go:453-455`) — satisfied by
|
||
the client always sending its configured `host_id`.
|
||
|
||
### Probe 3 — Controller config-pull (link 1) re-confirm — **WORKS (unchanged)**
|
||
|
||
| Case | Request | Expected | Observed |
|
||
|------|---------|----------|----------|
|
||
| 3a correct | `GET /config/{id}` + correct `X-Retrieval-Password` | 200 text/yaml | **200**, `text/yaml; charset=utf-8` |
|
||
| 3a content | — | domain, hub.url, hub api_key, tunnel token present | **all YES** (domain `spike.demo-felhom.eu`; `hub.api_key` = the customer-scoped key; injected tunnel token; `hub.url`) |
|
||
| 3b wrong | wrong passphrase | 401 | **401** |
|
||
| 3c wrong | missing `X-Retrieval-Password` header | 401 | **401** |
|
||
| 3d wrong | unknown customer (correct-shaped pw) | 404 | **404** |
|
||
|
||
Constant-time passphrase compare (`handler.go:1365`); `cfg == nil → 404` (`:1359-1362`); missing header
|
||
`→ 401` (`:1347-1351`). The controller half is intact and needs no re-architecting. The generated yaml
|
||
carries the **customer-scoped** hub `api_key` (`configgen.go:47`, = `cfg.APIKey`) — this is the
|
||
controller's key, **distinct** from the agent's per-host key minted in Probe 1.
|
||
|
||
---
|
||
|
||
## 4. End-to-end verdict on Option A
|
||
|
||
**Option A composes today with ZERO new code.** The full Day-0 chain is exercisable as:
|
||
|
||
1. **HQ:** operator creates the customer in the hub UI (mints customer api_key + 5-word passphrase).
|
||
2. **HQ:** operator mints the host with the **global key** (`POST /admin/hosts`) and obtains
|
||
`{host_id, host_api_key}`. *(The global key never leaves HQ.)*
|
||
3. **On the Proxmox box:** the host-bootstrap script writes the agent config (`/etc/felhom-agent/agent.json`
|
||
or `FELHOM_AGENT_HUB_*` env) with `{hub.url, host_id, host_api_key}`; the agent authenticates
|
||
(host-report + desired-state) — **proven in Probe 2**.
|
||
4. **Provision:** the agent's provision back-half renders `bootstrap.json` into the guest from
|
||
`--customer-id` + `--hub-password` (the passphrase) — *(not run in this spike; existing, separately
|
||
validated)*.
|
||
5. **In-guest:** the controller pulls `controller.yaml` via `GET /config/{id}` + `X-Retrieval-Password`
|
||
— **proven in Probe 3** — which yields the customer-scoped hub key, domain, and tunnel token; the
|
||
tunnel comes up.
|
||
|
||
No link 401/403/404'd unexpectedly. The only thing **not** physically run start-to-finish is the
|
||
provision step (4), which is deliberately out of a no-destructive-provision spike and is already
|
||
validated elsewhere; both its inputs (host credential present; passphrase → config-pull) are proven.
|
||
|
||
**Blocking gaps: none.**
|
||
|
||
---
|
||
|
||
## 5. Option A vs C — recommendation
|
||
|
||
**Two secrets cross to the customer box under A; one under C.**
|
||
|
||
- **Option A** (proven): the operator carries **two** secrets to the deployment: the **per-host
|
||
api_key** (for the agent, step 3) *and* the **retrieval passphrase** (for the in-guest controller,
|
||
steps 4–5). The global key stays at HQ. Works today, no code change.
|
||
- **Option C** (proposed): the passphrase-authed pull *also* mints-or-returns the host credential, so
|
||
the operator carries **only the passphrase** — and **no global key participates in the deploy path at
|
||
all** (it stays a pure HQ-admin key). Fewer secrets in flight; the deploy box never holds an
|
||
operator-tier global key even transiently.
|
||
|
||
**Feasibility of C (confirmed at source):** `handleConfigRetrieve` already has both prerequisites in
|
||
scope — the `customerID` (path param) and the `h.store` handle (which exposes `UpsertHost`,
|
||
`GetHostByAPIKey`, `GetHost`, `ListHosts`). The required hub delta:
|
||
|
||
1. **Idempotency lookup (new):** there is **no `GetHostByCustomer`** today. `UpsertHost` conflicts on
|
||
`host_id` (`store.go:1172-1183`), so naive re-pull would mint a *new* random host each call and
|
||
orphan the prior. C should **mint-once-and-reuse**: add a `GetHostByCustomer(customerID)` (or filter
|
||
`ListHosts`) so a re-pull returns the existing credential. *(Mint-per-pull = rotation; rejected — it
|
||
would silently invalidate a running agent's key on any controller re-pull.)*
|
||
2. **Response-shape change — keep it backward-compatible:** config-retrieve currently returns **raw
|
||
YAML** consumed by the in-guest controller. Do **not** switch the body to JSON (breaks the controller
|
||
curl). Two safe options, in order of preference:
|
||
- **(c-pref) Separate passphrase-authed endpoint**, e.g. `POST /api/v1/hosts/enroll` with
|
||
`X-Retrieval-Password`, body `{customer_id}`, returns `{host_id, api_key}` (idempotent). Leaves the
|
||
proven config-retrieve path **untouched**; the host-bootstrap script calls enroll for the agent and
|
||
config-retrieve for the guest seed. Cleanest separation of the host-tier and guest-tier pulls.
|
||
- **(c-alt) Side-channel headers** on the existing GET: emit `X-Host-Id` / `X-Host-Api-Key` response
|
||
headers alongside the YAML body. The in-guest controller ignores them; the script reads them. Fewer
|
||
endpoints but couples two tiers' secrets into one response.
|
||
|
||
> **UPDATE 2026-06-26 — c-pref SHIPPED in hub v0.14.0.** The dedicated passphrase-authed endpoint
|
||
> landed as `POST /api/v1/host-enroll` (mint-once-reuse via the new `Store.GetHostByCustomer`); the
|
||
> proven `GET /config/{id}` and the global-key `POST /admin/hosts` escape hatch are unchanged. See the
|
||
> hub `CHANGELOG.md` v0.14.0 + `README.md` "Host Enrollment (Day-0, option C)". The route is the exact
|
||
> `path == "/host-enroll"` form (not `/hosts/enroll`, which would collide with the `/hosts/` per-host
|
||
> prefix routes).
|
||
>
|
||
> **UPDATE 2026-06-26 — Day-0 host-bootstrap script SHIPPED.** This gap inventory is now implemented
|
||
> end-to-end by `scripts/felhom-host-install.sh` v1.0.0 (provision mode): pveum token → `host-enroll`
|
||
> → agent config → `--selftest=provision` → verify. Live-validated by an authorized wipe + re-provision
|
||
> of demo guest 9201 (controller config-pull + public tunnel `HTTP 200` + host-report confirmed). DR
|
||
> mode is a documented stub (10D). See `scripts/README.md` + `REPORT.md`.
|
||
|
||
**Recommendation:** ship **Option A now** (it is proven and needs nothing). Adopt **Option C, variant
|
||
(c-pref) — a dedicated passphrase-authed `host-enroll` endpoint, mint-once-reuse — at the production
|
||
cutover**, on the secret-hygiene argument: it removes the operator-tier global key from the field
|
||
deployment entirely and reduces the operator-carried secret set to one (the passphrase the customer
|
||
already has). It is a small, isolated change that does not touch the proven controller config-pull.
|
||
This matches the existing `handleAdminCreateHost` PROVISIONAL note (`handler.go:595-598`), which already
|
||
flags that endpoint for lock-down at the enrollment cutover.
|
||
|
||
---
|
||
|
||
## 6. Gap inventory — the checklist the host-bootstrap script spec consumes
|
||
|
||
**Exact config-retrieve contract:** `GET https://hub.felhom.eu/api/v1/config/{customer_id}`, header
|
||
`X-Retrieval-Password: <passphrase>`; 200 → raw `controller.yaml` (text/yaml), 401 wrong/missing pw,
|
||
404 unknown customer. **Capture HTTP status and only treat 2xx as success** (do not pipe a non-2xx body
|
||
onward — the spike's recurring lesson; curl `-f`/`-w '%{http_code}'`, check before use).
|
||
|
||
**Secrets/values the script must obtain, and their source (A vs C):**
|
||
|
||
| Value | Source under A | Source under C |
|
||
|-------|----------------|----------------|
|
||
| `hub.url` | static (`https://hub.felhom.eu`) | static |
|
||
| `customer_id` | operator-supplied (created in hub UI) | operator-supplied |
|
||
| `retrieval passphrase` | operator-supplied (hub UI) — SECRET | operator-supplied — SECRET (the **only** carried secret) |
|
||
| `host_id` | returned by HQ `POST /admin/hosts` | returned by passphrase-authed enroll |
|
||
| host `api_key` | returned by HQ `POST /admin/hosts` — SECRET | returned by passphrase-authed enroll — SECRET |
|
||
| **global key** | **HQ-only; never on the box** | **not used in deploy path at all** |
|
||
| customer-scoped hub key, domain, tunnel token | inside the pulled `controller.yaml` (not handled directly by the script) | same |
|
||
|
||
**Agent config write path on a real host (confirmed):** `/etc/felhom-agent/agent.json` (JSON; flag
|
||
`--config`, env `FELHOM_AGENT_CONFIG`), or override the hub fields via env
|
||
`FELHOM_AGENT_HUB_URL` / `FELHOM_AGENT_HUB_HOST_ID` / `FELHOM_AGENT_HUB_API_KEY` (`config.go:446-453`).
|
||
**Host-specific fields:** `hub.host_id`, `hub.api_key` (+ proxmox endpoint/node/token, which the agent
|
||
needs anyway). **Static fields:** `hub.url`. (Per the live-agent notes, the demo box keeps its config
|
||
under `/root/.config/felhom-agent/` via `FELHOM_AGENT_CONFIG`; the script must write to whatever path
|
||
the agent's systemd unit points `--config`/`FELHOM_AGENT_CONFIG` at — confirm the unit on the target.)
|
||
|
||
**How the controller gets `bootstrap.json` (confirmed at source — the script does NOT write it):** the
|
||
**agent's provision back-half** renders it (`felhom-agent/internal/provision/backhalf.go:121`,
|
||
`doc.go:23-47`), writes it 0600, chowns to the mapped guest-root, and attaches it on the guest config
|
||
mount. The script's job is only to invoke the agent's provision with `--customer-id <id>` and
|
||
`--hub-password <passphrase>` (`cmd/felhom-agent/main.go:131-132`); the agent bakes those into
|
||
`bootstrap.json`. The `bootstrap.json` contract (cross-repo, must match the controller's
|
||
`internal/bootstrap.Bootstrap`):
|
||
|
||
```json
|
||
{ "schema": "...",
|
||
"customer": { "id": "<customer_id>" },
|
||
"hub": { "url": "<hub_url>", "retrieval_password": "<passphrase, SECRET>" },
|
||
"local_api": { "endpoint": "<host-bridge ip:port>", "fingerprint": "<agent leaf SHA-256>", "token": "<per-guest bearer, SECRET>" } }
|
||
```
|
||
|
||
Note it carries **no host key, no customer-scoped hub key, no CF tokens** — those all come from the
|
||
controller's pull (`doc.go:24-26`).
|
||
|
||
**Unexpected 401/403/404 / values with no source:** none observed. Every refusal was the correct gate.
|
||
|
||
---
|
||
|
||
## 7. Cleanup performed
|
||
|
||
All throwaway state removed and verified (counts → 0); the real `demo-felhom` customer confirmed still
|
||
present and untouched:
|
||
|
||
- `DELETE FROM host_reports / hosts / customer_configs WHERE customer_id='spike-day0-2026-06-26'`
|
||
(before: 1/1/1 → after: 0/0/0).
|
||
- Removed scratch `cmd/spikeprobe` from the build-box agent checkout (`git status` clean) and
|
||
`/tmp/spike{probe,-day0}`, `/tmp/*.sh`.
|
||
- Uninstalled the temporary `sqlite` package from the hub pod (`apk del sqlite`).
|
||
- No production code, no version bump, no push beyond this doc; no image build; no deploy; no provision.
|
||
|
||
---
|
||
|
||
## 8. Open questions for the host-bootstrap script spec
|
||
|
||
1. **Agent config target path on a production host** — the spec must pin where the script writes
|
||
`agent.json` vs. relying on `FELHOM_AGENT_HUB_*` env, and align with the agent's systemd unit
|
||
`--config`/`FELHOM_AGENT_CONFIG`. (On the demo box this is `/root/.config/felhom-agent/`; production
|
||
may differ.) Also: the agent needs a **Proxmox token** in that same config — where does the script
|
||
source it on a freshly-imaged host? (Likely a `pveum` token-create step earlier in the runbook;
|
||
confirm it is in the chain.)
|
||
2. **Option C decision** — if the team wants single-secret Day-0, the `hosts/enroll` endpoint (variant
|
||
c-pref, idempotent mint-once-reuse) is the prerequisite and should be specced/shipped before the
|
||
script depends on it. Until then the script uses Option A (HQ pre-mint).
|
||
3. **`local_api.endpoint`/`fingerprint` provenance** — these are filled by the agent at provision time,
|
||
not by the script; the spec should state the script does not supply them (avoids a false "missing
|
||
value" in the script's input list).
|
||
4. **Re-run / idempotency of the whole script** — under A, re-running the mint step creates a *second*
|
||
host for the same customer (admin mint is not idempotent). The script should either reuse a prior
|
||
credential or the runbook should make the mint a one-time HQ step. (C with mint-once-reuse fixes
|
||
this structurally.)
|
||
|
||
**No secrets appear anywhere in this document.**
|