scripts: felhom-host-install.sh v1.0.0 — Day-0 host bootstrap (live-validated)

Operator-run Day-0 automation for a fresh PVE host: pveum token -> hub
POST /host-enroll (single secret, option C) -> agent config -> felhom-agent
--selftest=provision -> verify. Idempotent/resumable (state.json), --dry-run,
--resume, --force, --preserve-from; --mode dr is a documented 10D stub.

Live-validated on felhom-pve: authorized wipe of demo guest 9201 + full
re-provision from the golden -> controller config-pull + public tunnel HTTP 200
+ host-report of 9201; idempotent --resume no-op. Found+fixed one ordering bug
during the run (token ACL must be applied AFTER token rotation — token-remove
purges the ACL).

Adds scripts/README.md (operator runbook) + scripts/CHANGELOG.md; REPORT
overwritten; spike doc pointer added. No service code changed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01TtXesNa2LGbMmE4DNL6SE7

REPORT.md

@@ -4,83 +4,98 @@
 
 ---
 
-## hub v0.14.0 — Passphrase-authed host enrollment (Day-0 option C) — 2026-06-26
+## Day-0 host-bootstrap script — built + live wipe-and-provision test — 2026-06-26
 
-**Task:** Implement option C from `documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md`:
-a passphrase-authed, mint-once-reuse host-enrollment endpoint so Day-0 carries **one** secret (the
-retrieval passphrase) and the global operator key never enters the field deploy path. Single repo
-(`felhom.eu`, `hub/`).
+**Task:** Deliver `scripts/felhom-host-install.sh` (operator-run Day-0 bootstrap) and validate it with
+an **operator-authorized destructive live test**: wipe demo guest 9201 and re-provision it from scratch
+via the script. Single repo (`felhom.eu`, new `scripts/`).
 
-### Confirmed baseline
-- felhom.eu (hub) `main` @ `230980f` (= origin/main, verified by `git fetch`), version `v0.13.1`
-  (ldflags-injected; no source constant) → target `v0.14.0`.
-- Reconfirmed every relied-on symbol live: `handleConfigRetrieve` (auth pattern), `handleAdminCreateHost`
-  (mint block — left untouched), `UpsertHost` (`ON CONFLICT(host_id)`), `GetHostByAPIKey`/`GetHost`/
-  `scanHost`/`hostSelectCols`, `idx_hosts_customer`, `configgen.RandomHex`.
+### Confirmed baselines
+- hub `1539746` v0.14.0 (`/host-enroll` option C, just shipped) · felhom-agent `4725396` v0.41.0
+  (`--selftest=provision`) · felhom-controller `e0cf78b` v0.84.0.
+- Live host `felhom-pve` (192.168.0.162): PVE 9.2.2, node `demo-felhom`, agent v0.41.0 at
+  `/root/.config/felhom-agent/agent.json` (systemd unit `-config` path — **not** `/etc/felhom-agent/`),
+  golden `local:backup/vzdump-lxc-9100-2026_06_23-16_38_49.tar.zst` (rootfs 32 / docker 16 / sysdata 8 GiB base).
 
-### Files changed
-| File | Change |
-|------|--------|
-| `hub/internal/store/store.go` | **+** `Store.GetHostByCustomer` (SELECT … WHERE customer_id=? ORDER BY updated_at DESC LIMIT 1; nil-on-not-found; uses `idx_hosts_customer`) |
-| `hub/internal/store/host_test.go` | **+** `TestGetHostByCustomer` (none→nil / one / two→most-recent / other-customer) |
-| `hub/internal/api/handler.go` | **+** route `POST /host-enroll` (exact-match, beside `/host-report`); **+** `handleHostEnroll` (passphrase auth → mint-once-reuse). `handleConfigRetrieve` + `handleAdminCreateHost` **unchanged** |
-| `hub/internal/api/host_enroll_test.go` | **+** 5 scenario tests (mint/reuse/401-no-mint/404/400) |
-| `hub/CHANGELOG.md` | **+** v0.14.0 entry (newest on top) |
-| `hub/README.md` | **+** "Host Enrollment (Day-0, option C)" section |
-| `documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md` | **+** "c-pref SHIPPED in v0.14.0" pointer in §5 |
+### Deliverable
+- `scripts/felhom-host-install.sh` **v1.0.0** — 7 idempotent/resumable steps (pre-flight → pveum token →
+  grows → host-enroll → agent config → provision → verify), single-secret (passphrase no-echo/0600 file),
+  `--dry-run`/`--resume`/`--force`/`--preserve-from`, `--mode dr` documented stub.
+- `scripts/README.md` (operator runbook), `scripts/CHANGELOG.md` (v1.0.0).
+- `documentation/audits/SPIKE-…-2026-06-26.md` — pointer added (script ships the gap inventory).
+- `bash -n` passes (Windows + host). **shellcheck unavailable** on dev box, build box, and felhom-pve — not run.
+
+### Live supervised test (felhom-pve, customer `demo-felhom`, authorized destructive)
+Passphrase sourced operator-side from the hub DB into a 0600 file on the host (never printed); deleted
+(shredded) after. Methods noted per check.
+
+**A. Pre-wipe snapshot:** saved `pct config 9201`, controller `0.84.0 Up (healthy)`, `agent.json.bak`
+(1574 B), token list. Confirmed `demo-felhom` host exists in hub (`demo-felhom-01`, plus two stale
+`dr-drill-host*` cruft rows — `demo-felhom-01` has today's `updated_at`, so `GetHostByCustomer` returns it).
+
+**B. Wipe (authorized):** `pct stop 9201` + `pct destroy 9201 --purge` (all 3 volumes removed); moved
+`agent.json` + cleared state to simulate a fresh host.
+
+**C. Dry-run:** every mutating command printed correctly — pveum role-modify (16 privs), **both** ACL
+lines, token rotate, enroll curl, agent.json write, provision call. Pre-flight green (golden auto-found,
+passphrase valid, vmid free).
+
+**D. Real run — found + fixed a bug, then SUCCESS:**
+- **Bug found live:** first run failed the post-config agent selftest with `HTTP 403 … missing privilege
+  Sys.Audit`. Root cause: the script applied the **token ACL before rotating the token**, and
+  `pveum user token remove` **purges the token's ACL** — so the new token had no ACL. **Fix:** apply both
+  ACL grants **after** the token is (re)created. (`scripts/felhom-host-install.sh` step 2.)
+- **Re-run green:** token rotated (old config token failed selftest → rotate → ACL → selftest **passed**);
+  `host-enroll` → **200 REUSE** of `demo-felhom-01` (idempotency confirmed); agent.json written 0600 +
+  read-only selftest passed + daemon restarted; provision front-half bring-up (1m11s) + back-half bootstrap
+  mount `mp9` → `/etc/felhom-bootstrap` + local-api token minted; guest **9201 running, onboot:1**, layout
+  rootfs 32G / mp0 200G / mp1 50G / mp8 bind — exactly the standard appliance.
+
+**E. Verify (method per check):**
+| Check | Method | Result |
+|-------|--------|--------|
+| guest running + onboot:1 + volumes | `pct status/config` | ✅ running, onboot 1, 32/200/50G + mp8 bind |
+| controller healthy | `pct exec docker ps` | ✅ `Up (healthy)`, image `felhom-controller:0.77.0` (from golden) |
+| **controller pulled config from hub** | controller logs (`pct exec`) | ✅ `bootstrap: pulled config from hub for demo-felhom, merged local_api … coming up configured`; `Hub connectivity … HTTP 200` |
+| serving | `pct exec curl :80` | ✅ HTTP 301 (traefik) |
+| **Cloudflare tunnel connected** | cloudflared logs | ✅ 4 `Registered tunnel connection` (vie06, bud01) |
+| **public URL** | curl from build box (public DNS) | ✅ `felhom.demo-felhom.eu → HTTP 200` via Cloudflare `188.114.97.11` (ingress `*.demo-felhom.eu → traefik`) |
+| real dashboard | curl public | ✅ `<title>Vezérlőpult — Felhom.eu</title>` |
+| **host-report includes guest 9201** | hub sqlite | ✅ `demo-felhom-01/9201` status `running`, last_seen current |
+| storage plane (functional touch) | `pct exec ls` | ✅ `/mnt/felhom-drives` bound, `felhom-flash` + `felhom-usb` visible (data intact); local-api drive plane reachable |
+
+> Public URL returned `000` from felhom-pve and the Windows dev box — split-horizon (the agent's LAN
+> resolver points `*.demo-felhom.eu` at the guest's LAN IP, which serves `:80` not `:443`). Through the
+> Cloudflare edge (build box) it is **200**. Not a provision fault.
+
+**F. Idempotency:** plain re-run **refused** to clobber the live 9201 (exit 1, "pass `--force`");
+`--resume` re-run **skipped** token/grows/enroll/agent_config/provision → verify-only **no-op**, token
+**not** rotated (selftest still passes), guest **not** re-provisioned.
+
+**G. Cleanup:** passphrase file shredded; `agent.json.bak` + snapshot retained; final state — 9201
+running, host_id `demo-felhom-01`, agent active.
+
+### pveum / token automation result
+Role `FelhomAgent` normalised to the authoritative **16 privileges** (incl. `Sys.Audit`, `SDN.Use`); user
+`felhom-agent@pve`; privsep token rotated; **both** ACL grants (user **and** token) applied **after** token
+creation. Read-only `felhom-agent --selftest` passed against the live API with the minted token.
+
+### Observations (out-of-scope — not acted on)
+1. **`-hub-password` on argv:** the agent's only input for the retrieval passphrase during provision is the
+   `-hub-password` flag → briefly visible in `ps` during `--selftest=provision`. **Candidate agent
+   follow-up:** accept it via env/stdin. (Not fixed here — would modify the agent.)
+2. **Golden freshness:** the golden bakes controller **0.77.0**; the freshly provisioned demo runs 0.77.0
+   (pre-wipe was 0.84.0, and the v0.78.0 drive-register stable-path fix is newer than the golden). Rebuild
+   the golden at the current controller for parity; full drive **re-enroll** via the UI needs controller
+   ≥0.78.0. The Day-0 script itself is version-agnostic.
+3. **Multi-host cruft:** `demo-felhom` carries two stale `dr-drill-host*` rows. `GetHostByCustomer`
+   (most-recent-updated) correctly returned `demo-felhom-01` here, but the Day-0 invariant is one host per
+   customer — drill artifacts should be cleaned up (left untouched; out of scope).
+4. **Serving + binary/golden delivery:** the script lives in `scripts/`; serving it at
+   `https://felhom.eu/scripts/…`, plus agent-binary and golden central delivery, are documented
+   productionization hooks (the test used the local binary + golden).
 
 ### Commits
-- `8098237` — `hub v0.14.0: passphrase-authed host enrollment (Day-0 option C)` (code + tests + CHANGELOG + README + spike pointer).
-- `<this REPORT commit>` — REPORT.md overwrite.
+- `<this commit>` — script v1.0.0 + scripts/README + scripts/CHANGELOG + spike pointer + REPORT.
 
-### Tests (temp-sqlite stores; no network)
-- **Per-scenario (all PASS):**
-  - A — first enroll mints → `201`, `host_id=c1-<6hex>`, `api_key=64hex`, exactly 1 host row, `GetHostByAPIKey(minted)` resolves.
-  - B — second enroll **reuses** → `200`, creds **byte-for-byte identical**, still exactly 1 host row.
-  - C — wrong passphrase → `401`, host-row count **unchanged at 0** (auth-before-mint proven).
-  - D — unknown customer → `404`, no mint.
-  - E — missing header → `401`; missing `customer_id` → `400`.
-  - store `TestGetHostByCustomer` — none→nil, one→hit, two→most-recently-updated.
-- **Companion red-proof (§10):** temporarily disabled the reuse branch (always-mint) → Scenario B
-  **FAILED** as required (`second status = 201, want 200` — second call returned a *new* `host_id`
-  `c1-0d44c6`, not the first). Reverted to mint-once-reuse → green. This proves the idempotency test is
-  load-bearing, not hollow.
-- **Test count:** api package 29 → **34** (+5); store package 11 → **12** (+1).
-- **Green gate:** `go build ./... && go vet ./... && go test ./...` — all `ok` (api, configgen, monitor,
-  store), no failures.
-
-### Deploy + pod verification
-- `./build.sh 0.14.0 --push` on the build box (auto-pulls `8098237`) → image
-  `gitea.dooplex.hu/admin/felhom-hub:0.14.0` pushed (digest `sha256:541cdb54…`, 24M).
-- `kubectl set image deploy/hub` → rollout succeeded; pod `hub-6c7d8bcc9-454n9` `1/1 Running`.
-- Startup log: `[INFO] felhom-hub 0.14.0 starting`; deployed image confirmed `…:0.14.0`.
-
-### Live verification against the real endpoint (throwaway `enroll-spike-2026-06-26`)
-Curl-against-real-endpoint (the exact call the future host-bootstrap script makes). Secrets stayed
-resident on the build box; only statuses / non-secret host_id / identical-bytes booleans surfaced.
-
-| Check | Result |
-|-------|--------|
-| 1. first enroll (correct passphrase) | **201**, `host_id=enroll-spike-2026-06-26-c641d7`, api_key 64-hex |
-| 2. second enroll (idempotent) | **200**, host_id identical YES, api_key identical YES, **exactly 1 host row** |
-| 3. wrong passphrase | **401** |
-| 4. unknown customer | **404** |
-| 5. missing customer_id | **400** |
-| 6. **Bearer-auth proof** — `GET /hosts/{host_id}/desired-state` with `Authorization: Bearer <enrolled key>` | **200** (the enrolled credential is a real, working host key end-to-end) |
-
-### Cleanup (throwaway discipline, mirroring the spike)
-- Deleted throwaway `customer_configs` / `hosts` / `host_reports` rows → counts **0/0/0** (verified).
-- Real `demo-felhom` customer confirmed **still present / untouched**.
-- Uninstalled temporary `sqlite` from the hub pod (`apk del`); removed all `/tmp/enroll-spike*` scratch.
-- The global key was read out-of-band into a shell var only; **no secret** written to disk, any
-  committed file, or this report.
-
-### Observations / out-of-scope (not acted on)
-- `POST /admin/hosts` is still **PROVISIONAL** (`handler.go:595`) and is left **as the operator/global
-  escape hatch** — its removal/lock-down is a separate cutover decision, deliberately not done here.
-- `GET /config/{id}` was **not** modified (no host fields, no JSON, raw-YAML body intact) — the in-guest
-  controller pull is unaffected.
-- The host-bootstrap script (Day-0 wiring) is the **next** task — not started here. `host-enroll`'s
-  mint-once-reuse is the idempotency primitive that step will depend on.
-- Under option A, `admin/hosts` re-mint still orphans a prior host (no idempotency there); option C
-  fixes this structurally for the enroll path.
+No secrets in any committed file. PVE-token secret + host api_key live only in the host's 0600 agent config.