diff --git a/REPORT.md b/REPORT.md index 9e282ab..4d86493 100644 --- a/REPORT.md +++ b/REPORT.md @@ -4,83 +4,98 @@ --- -## hub v0.14.0 — Passphrase-authed host enrollment (Day-0 option C) — 2026-06-26 +## Day-0 host-bootstrap script — built + live wipe-and-provision test — 2026-06-26 -**Task:** Implement option C from `documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md`: -a passphrase-authed, mint-once-reuse host-enrollment endpoint so Day-0 carries **one** secret (the -retrieval passphrase) and the global operator key never enters the field deploy path. Single repo -(`felhom.eu`, `hub/`). +**Task:** Deliver `scripts/felhom-host-install.sh` (operator-run Day-0 bootstrap) and validate it with +an **operator-authorized destructive live test**: wipe demo guest 9201 and re-provision it from scratch +via the script. Single repo (`felhom.eu`, new `scripts/`). -### Confirmed baseline -- felhom.eu (hub) `main` @ `230980f` (= origin/main, verified by `git fetch`), version `v0.13.1` - (ldflags-injected; no source constant) → target `v0.14.0`. -- Reconfirmed every relied-on symbol live: `handleConfigRetrieve` (auth pattern), `handleAdminCreateHost` - (mint block — left untouched), `UpsertHost` (`ON CONFLICT(host_id)`), `GetHostByAPIKey`/`GetHost`/ - `scanHost`/`hostSelectCols`, `idx_hosts_customer`, `configgen.RandomHex`. +### Confirmed baselines +- hub `1539746` v0.14.0 (`/host-enroll` option C, just shipped) · felhom-agent `4725396` v0.41.0 + (`--selftest=provision`) · felhom-controller `e0cf78b` v0.84.0. +- Live host `felhom-pve` (192.168.0.162): PVE 9.2.2, node `demo-felhom`, agent v0.41.0 at + `/root/.config/felhom-agent/agent.json` (systemd unit `-config` path — **not** `/etc/felhom-agent/`), + golden `local:backup/vzdump-lxc-9100-2026_06_23-16_38_49.tar.zst` (rootfs 32 / docker 16 / sysdata 8 GiB base). -### Files changed -| File | Change | -|------|--------| -| `hub/internal/store/store.go` | **+** `Store.GetHostByCustomer` (SELECT … WHERE customer_id=? ORDER BY updated_at DESC LIMIT 1; nil-on-not-found; uses `idx_hosts_customer`) | -| `hub/internal/store/host_test.go` | **+** `TestGetHostByCustomer` (none→nil / one / two→most-recent / other-customer) | -| `hub/internal/api/handler.go` | **+** route `POST /host-enroll` (exact-match, beside `/host-report`); **+** `handleHostEnroll` (passphrase auth → mint-once-reuse). `handleConfigRetrieve` + `handleAdminCreateHost` **unchanged** | -| `hub/internal/api/host_enroll_test.go` | **+** 5 scenario tests (mint/reuse/401-no-mint/404/400) | -| `hub/CHANGELOG.md` | **+** v0.14.0 entry (newest on top) | -| `hub/README.md` | **+** "Host Enrollment (Day-0, option C)" section | -| `documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md` | **+** "c-pref SHIPPED in v0.14.0" pointer in §5 | +### Deliverable +- `scripts/felhom-host-install.sh` **v1.0.0** — 7 idempotent/resumable steps (pre-flight → pveum token → + grows → host-enroll → agent config → provision → verify), single-secret (passphrase no-echo/0600 file), + `--dry-run`/`--resume`/`--force`/`--preserve-from`, `--mode dr` documented stub. +- `scripts/README.md` (operator runbook), `scripts/CHANGELOG.md` (v1.0.0). +- `documentation/audits/SPIKE-…-2026-06-26.md` — pointer added (script ships the gap inventory). +- `bash -n` passes (Windows + host). **shellcheck unavailable** on dev box, build box, and felhom-pve — not run. + +### Live supervised test (felhom-pve, customer `demo-felhom`, authorized destructive) +Passphrase sourced operator-side from the hub DB into a 0600 file on the host (never printed); deleted +(shredded) after. Methods noted per check. + +**A. Pre-wipe snapshot:** saved `pct config 9201`, controller `0.84.0 Up (healthy)`, `agent.json.bak` +(1574 B), token list. Confirmed `demo-felhom` host exists in hub (`demo-felhom-01`, plus two stale +`dr-drill-host*` cruft rows — `demo-felhom-01` has today's `updated_at`, so `GetHostByCustomer` returns it). + +**B. Wipe (authorized):** `pct stop 9201` + `pct destroy 9201 --purge` (all 3 volumes removed); moved +`agent.json` + cleared state to simulate a fresh host. + +**C. Dry-run:** every mutating command printed correctly — pveum role-modify (16 privs), **both** ACL +lines, token rotate, enroll curl, agent.json write, provision call. Pre-flight green (golden auto-found, +passphrase valid, vmid free). + +**D. Real run — found + fixed a bug, then SUCCESS:** +- **Bug found live:** first run failed the post-config agent selftest with `HTTP 403 … missing privilege + Sys.Audit`. Root cause: the script applied the **token ACL before rotating the token**, and + `pveum user token remove` **purges the token's ACL** — so the new token had no ACL. **Fix:** apply both + ACL grants **after** the token is (re)created. (`scripts/felhom-host-install.sh` step 2.) +- **Re-run green:** token rotated (old config token failed selftest → rotate → ACL → selftest **passed**); + `host-enroll` → **200 REUSE** of `demo-felhom-01` (idempotency confirmed); agent.json written 0600 + + read-only selftest passed + daemon restarted; provision front-half bring-up (1m11s) + back-half bootstrap + mount `mp9` → `/etc/felhom-bootstrap` + local-api token minted; guest **9201 running, onboot:1**, layout + rootfs 32G / mp0 200G / mp1 50G / mp8 bind — exactly the standard appliance. + +**E. Verify (method per check):** +| Check | Method | Result | +|-------|--------|--------| +| guest running + onboot:1 + volumes | `pct status/config` | ✅ running, onboot 1, 32/200/50G + mp8 bind | +| controller healthy | `pct exec docker ps` | ✅ `Up (healthy)`, image `felhom-controller:0.77.0` (from golden) | +| **controller pulled config from hub** | controller logs (`pct exec`) | ✅ `bootstrap: pulled config from hub for demo-felhom, merged local_api … coming up configured`; `Hub connectivity … HTTP 200` | +| serving | `pct exec curl :80` | ✅ HTTP 301 (traefik) | +| **Cloudflare tunnel connected** | cloudflared logs | ✅ 4 `Registered tunnel connection` (vie06, bud01) | +| **public URL** | curl from build box (public DNS) | ✅ `felhom.demo-felhom.eu → HTTP 200` via Cloudflare `188.114.97.11` (ingress `*.demo-felhom.eu → traefik`) | +| real dashboard | curl public | ✅ `Vezérlőpult — Felhom.eu` | +| **host-report includes guest 9201** | hub sqlite | ✅ `demo-felhom-01/9201` status `running`, last_seen current | +| storage plane (functional touch) | `pct exec ls` | ✅ `/mnt/felhom-drives` bound, `felhom-flash` + `felhom-usb` visible (data intact); local-api drive plane reachable | + +> Public URL returned `000` from felhom-pve and the Windows dev box — split-horizon (the agent's LAN +> resolver points `*.demo-felhom.eu` at the guest's LAN IP, which serves `:80` not `:443`). Through the +> Cloudflare edge (build box) it is **200**. Not a provision fault. + +**F. Idempotency:** plain re-run **refused** to clobber the live 9201 (exit 1, "pass `--force`"); +`--resume` re-run **skipped** token/grows/enroll/agent_config/provision → verify-only **no-op**, token +**not** rotated (selftest still passes), guest **not** re-provisioned. + +**G. Cleanup:** passphrase file shredded; `agent.json.bak` + snapshot retained; final state — 9201 +running, host_id `demo-felhom-01`, agent active. + +### pveum / token automation result +Role `FelhomAgent` normalised to the authoritative **16 privileges** (incl. `Sys.Audit`, `SDN.Use`); user +`felhom-agent@pve`; privsep token rotated; **both** ACL grants (user **and** token) applied **after** token +creation. Read-only `felhom-agent --selftest` passed against the live API with the minted token. + +### Observations (out-of-scope — not acted on) +1. **`-hub-password` on argv:** the agent's only input for the retrieval passphrase during provision is the + `-hub-password` flag → briefly visible in `ps` during `--selftest=provision`. **Candidate agent + follow-up:** accept it via env/stdin. (Not fixed here — would modify the agent.) +2. **Golden freshness:** the golden bakes controller **0.77.0**; the freshly provisioned demo runs 0.77.0 + (pre-wipe was 0.84.0, and the v0.78.0 drive-register stable-path fix is newer than the golden). Rebuild + the golden at the current controller for parity; full drive **re-enroll** via the UI needs controller + ≥0.78.0. The Day-0 script itself is version-agnostic. +3. **Multi-host cruft:** `demo-felhom` carries two stale `dr-drill-host*` rows. `GetHostByCustomer` + (most-recent-updated) correctly returned `demo-felhom-01` here, but the Day-0 invariant is one host per + customer — drill artifacts should be cleaned up (left untouched; out of scope). +4. **Serving + binary/golden delivery:** the script lives in `scripts/`; serving it at + `https://felhom.eu/scripts/…`, plus agent-binary and golden central delivery, are documented + productionization hooks (the test used the local binary + golden). ### Commits -- `8098237` — `hub v0.14.0: passphrase-authed host enrollment (Day-0 option C)` (code + tests + CHANGELOG + README + spike pointer). -- `` — REPORT.md overwrite. +- `` — script v1.0.0 + scripts/README + scripts/CHANGELOG + spike pointer + REPORT. -### Tests (temp-sqlite stores; no network) -- **Per-scenario (all PASS):** - - A — first enroll mints → `201`, `host_id=c1-<6hex>`, `api_key=64hex`, exactly 1 host row, `GetHostByAPIKey(minted)` resolves. - - B — second enroll **reuses** → `200`, creds **byte-for-byte identical**, still exactly 1 host row. - - C — wrong passphrase → `401`, host-row count **unchanged at 0** (auth-before-mint proven). - - D — unknown customer → `404`, no mint. - - E — missing header → `401`; missing `customer_id` → `400`. - - store `TestGetHostByCustomer` — none→nil, one→hit, two→most-recently-updated. -- **Companion red-proof (§10):** temporarily disabled the reuse branch (always-mint) → Scenario B - **FAILED** as required (`second status = 201, want 200` — second call returned a *new* `host_id` - `c1-0d44c6`, not the first). Reverted to mint-once-reuse → green. This proves the idempotency test is - load-bearing, not hollow. -- **Test count:** api package 29 → **34** (+5); store package 11 → **12** (+1). -- **Green gate:** `go build ./... && go vet ./... && go test ./...` — all `ok` (api, configgen, monitor, - store), no failures. - -### Deploy + pod verification -- `./build.sh 0.14.0 --push` on the build box (auto-pulls `8098237`) → image - `gitea.dooplex.hu/admin/felhom-hub:0.14.0` pushed (digest `sha256:541cdb54…`, 24M). -- `kubectl set image deploy/hub` → rollout succeeded; pod `hub-6c7d8bcc9-454n9` `1/1 Running`. -- Startup log: `[INFO] felhom-hub 0.14.0 starting`; deployed image confirmed `…:0.14.0`. - -### Live verification against the real endpoint (throwaway `enroll-spike-2026-06-26`) -Curl-against-real-endpoint (the exact call the future host-bootstrap script makes). Secrets stayed -resident on the build box; only statuses / non-secret host_id / identical-bytes booleans surfaced. - -| Check | Result | -|-------|--------| -| 1. first enroll (correct passphrase) | **201**, `host_id=enroll-spike-2026-06-26-c641d7`, api_key 64-hex | -| 2. second enroll (idempotent) | **200**, host_id identical YES, api_key identical YES, **exactly 1 host row** | -| 3. wrong passphrase | **401** | -| 4. unknown customer | **404** | -| 5. missing customer_id | **400** | -| 6. **Bearer-auth proof** — `GET /hosts/{host_id}/desired-state` with `Authorization: Bearer ` | **200** (the enrolled credential is a real, working host key end-to-end) | - -### Cleanup (throwaway discipline, mirroring the spike) -- Deleted throwaway `customer_configs` / `hosts` / `host_reports` rows → counts **0/0/0** (verified). -- Real `demo-felhom` customer confirmed **still present / untouched**. -- Uninstalled temporary `sqlite` from the hub pod (`apk del`); removed all `/tmp/enroll-spike*` scratch. -- The global key was read out-of-band into a shell var only; **no secret** written to disk, any - committed file, or this report. - -### Observations / out-of-scope (not acted on) -- `POST /admin/hosts` is still **PROVISIONAL** (`handler.go:595`) and is left **as the operator/global - escape hatch** — its removal/lock-down is a separate cutover decision, deliberately not done here. -- `GET /config/{id}` was **not** modified (no host fields, no JSON, raw-YAML body intact) — the in-guest - controller pull is unaffected. -- The host-bootstrap script (Day-0 wiring) is the **next** task — not started here. `host-enroll`'s - mint-once-reuse is the idempotency primitive that step will depend on. -- Under option A, `admin/hosts` re-mint still orphans a prior host (no idempotency there); option C - fixes this structurally for the enroll path. +No secrets in any committed file. PVE-token secret + host api_key live only in the host's 0600 agent config. diff --git a/documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md b/documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md index cfe94be..0bdb1a0 100644 --- a/documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md +++ b/documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md @@ -182,6 +182,12 @@ scope — the `customerID` (path param) and the `h.store` handle (which exposes > hub `CHANGELOG.md` v0.14.0 + `README.md` "Host Enrollment (Day-0, option C)". The route is the exact > `path == "/host-enroll"` form (not `/hosts/enroll`, which would collide with the `/hosts/` per-host > prefix routes). +> +> **UPDATE 2026-06-26 — Day-0 host-bootstrap script SHIPPED.** This gap inventory is now implemented +> end-to-end by `scripts/felhom-host-install.sh` v1.0.0 (provision mode): pveum token → `host-enroll` +> → agent config → `--selftest=provision` → verify. Live-validated by an authorized wipe + re-provision +> of demo guest 9201 (controller config-pull + public tunnel `HTTP 200` + host-report confirmed). DR +> mode is a documented stub (10D). See `scripts/README.md` + `REPORT.md`. **Recommendation:** ship **Option A now** (it is proven and needs nothing). Adopt **Option C, variant (c-pref) — a dedicated passphrase-authed `host-enroll` endpoint, mint-once-reuse — at the production diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md new file mode 100644 index 0000000..ced2825 --- /dev/null +++ b/scripts/CHANGELOG.md @@ -0,0 +1,29 @@ +# Felhom scripts — Changelog + +## felhom-host-install.sh v1.0.0 — Day-0 host bootstrap (provision mode) (2026-06-26) + +First release. A single operator-run script that automates Day-0 on a freshly-PVE-installed +host: Proxmox API token → hub host enrollment (option C, single secret) → agent config → +guest provision → verify. Composes proven mechanisms (the `pveum` role/token sequence, hub +`POST /host-enroll`, `felhom-agent --selftest=provision`); grounded by +`documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md`. + +- **7 steps, idempotent + resumable** via `/var/lib/felhom-install/state.json`: pre-flight → + Proxmox token → compute grows → host-enroll → agent config → provision → verify. +- **Single-secret** (the retrieval passphrase): read no-echo or from a 0600 file, never on + argv/logs/state. The global operator key never touches the box. +- **pveum automation:** 16-priv `FelhomAgent` role (create-or-modify), `felhom-agent@pve` user, + privsep token (reuse-if-working else rotate), and **both** ACL grants applied **after** the + token exists (token-remove purges the token ACL). +- **Auto-discovery:** golden archive (newest `vzdump-lxc-`), PVE node name, vmbr0 + bridge IP for the local-api, and the served-leaf TLS fingerprint pin. +- **Safety:** pre-flight fails fast (root, PVE 9.x, local-lvm headroom, hub reachable, + customer+passphrase valid via read-only `GET /config/{id}`, golden resolvable); refuses to + clobber an existing `--vmid` without `--force`; `--dry-run` previews every mutation; + `--preserve-from` keeps operator infra (PBS/local_api/privileged/authz) on re-deploys. +- **`--mode dr`:** documented 10D stub (restore customer PBS snapshot instead of golden) — not + implemented. +- **Live-validated** end-to-end on `felhom-pve`: authorized wipe of demo guest 9201 → + re-provision from the golden → controller config-pull + public tunnel `HTTP 200` → + host-report of guest 9201 → idempotent `--resume` no-op. (One ordering bug — token ACL + applied before rotation — was found and fixed during the live run.) diff --git a/scripts/README.md b/scripts/README.md new file mode 100644 index 0000000..1f4789f --- /dev/null +++ b/scripts/README.md @@ -0,0 +1,87 @@ +# Felhom host scripts + +Operator-side scripts for standing up a Felhom Proxmox host. + +## `felhom-host-install.sh` — Day-0 host bootstrap (operator-deploy) + +Run on a **freshly-PVE-installed** box to fully automate Day-0: Proxmox API token → +hub host enrollment (single secret) → agent config → guest provision → verify. It +composes already-proven mechanisms (the `pveum` role/token sequence, the hub +`POST /host-enroll` enrollment from option C, and `felhom-agent --selftest=provision`). +The agent renders `bootstrap.json` into the guest and the **controller pulls its own +`controller.yaml`** in-guest — the script never fetches it. + +Grounding: [`documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md`](../documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md). + +### Prerequisites (manual, before running) + +1. **Install Proxmox VE 9.x** on the box. During the installer, use **Advanced → LVM** + sizing so `local-lvm` (the `pve/data` thin pool) has enough room for the appliance + volumes — a useful box wants **≥ ~120 GiB** free on `local-lvm` (rootfs 32G + Docker-data + ~200G + user-data ~50G after grows). The script refuses below the hard minimum. +2. **SSH into the box as root.** +3. **Create the customer in the hub first** (hub UI → new customer). The customer's + **retrieval passphrase** (a 5-word Hungarian phrase) is the only secret you carry to the box. +4. A **golden archive** must exist on the archive storage (newest `vzdump-lxc-`). + If none exists, build one with `felhom-agent/configs/build-golden.sh` first. +5. The **felhom-agent binary** + its systemd unit installed (the script auto-detects the + unit's `-config` path; if the binary is absent it tells you to install it). + +### Usage + +```bash +curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o felhom-host-install.sh +chmod +x felhom-host-install.sh + +# secure no-echo passphrase prompt: +sudo ./felhom-host-install.sh --customer-id + +# or from a 0600 file (no prompt): +sudo ./felhom-host-install.sh --customer-id --passphrase-file /root/.pass + +# preview every mutating command without executing: +sudo ./felhom-host-install.sh --customer-id --dry-run + +# resume after a fixed mid-way failure (skips completed steps): +sudo ./felhom-host-install.sh --customer-id --resume +``` + +The passphrase is read **no-echo** or from a **0600 file** — never a CLI argument, never +echoed, never written to the state file or logs. The minted Proxmox-token secret and the +per-host hub api_key live **only** in the agent config (`0600`, root). + +### Key options + +| Option | Default | Purpose | +|--------|---------|---------| +| `--customer-id ID` | (required) | customer (must already exist in the hub) | +| `--vmid N` | `9201` | guest VMID to provision | +| `--golden VOLID` | newest `vzdump-lxc-` | golden archive | +| `--rootfs/--datavol/--sysdata-grow N` | auto-compute | volume grows (GiB over the golden base 32/16/8) | +| `--passphrase-file PATH` | no-echo prompt | read passphrase from a 0600 file | +| `--preserve-from PATH` | — | merge non-Day-0 sections (PBS/local_api/privileged/authz) from an existing config | +| `--dry-run` / `--resume` / `--force` | off | preview / resume / clobber an existing vmid | +| `--mode provision\|dr` | `provision` | `dr` is a documented 10D stub (not implemented) | + +### Behaviour notes + +- **Idempotent + resumable.** A step-state file (`/var/lib/felhom-install/state.json`) records + completed steps; `--resume` skips them. A plain re-run **refuses** to clobber an existing + `--vmid` (pass `--force` to override). +- **Single-secret enrollment.** `POST /host-enroll` mints on first call (201) and **reuses** + the credential on later calls (200) — re-running never orphans a running agent's key. The + global operator key is never used. +- **Token automation.** Creates/normalises the 16-priv `FelhomAgent` role, the + `felhom-agent@pve` user + privsep token, and **both** ACL grants (user **and** token — the + ACL is applied *after* the token exists, because `pveum user token remove` purges it). +- **DR mode** (`--mode dr`) is a documented seam only — it restores the customer's **own** PBS + whole-CT snapshot instead of the golden. Not implemented (10D). + +### Productionization hooks (not done here) + +- **Serving:** place this file where the felhom.eu site serves it at + `https://felhom.eu/scripts/felhom-host-install.sh` (a static route; verify on deploy). +- **Agent binary delivery:** the script expects the agent pre-installed; a fetch-from-release + step is the documented hook. +- **Golden delivery:** the test used a local golden; central download + checksum verify is the + remaining hook. diff --git a/scripts/felhom-host-install.sh b/scripts/felhom-host-install.sh new file mode 100644 index 0000000..297fd58 --- /dev/null +++ b/scripts/felhom-host-install.sh @@ -0,0 +1,565 @@ +#!/bin/bash +#=============================================================================== +# felhom-host-install.sh v1.0.0 +# Day-0 host-bootstrap for a Felhom Proxmox host (operator-deploy model). +# +# Run by the operator on a FRESHLY-PVE-INSTALLED box (after a manual PVE install +# + SSH in). Given a customer-id + retrieval passphrase, it fully automates +# Day-0: Proxmox API token -> hub host enrollment -> agent config -> guest +# provision -> verify. It composes already-proven mechanisms (the pveum role/ +# token sequence, hub POST /host-enroll [option C], felhom-agent +# --selftest=provision). The agent renders bootstrap.json and the controller +# pulls its own controller.yaml in-guest; this script does NOT fetch it. +# +# Grounding: documentation/audits/SPIKE-day0-firstboot-handshake-2026-06-26.md +# +# Usage: +# sudo ./felhom-host-install.sh --customer-id ID [options] +# +# Required: +# --customer-id ID Customer (must already exist in the hub) +# +# Options: +# --mode provision|dr provision (Day-0, default) | dr (10D stub — not impl.) +# --hub-url URL default https://hub.felhom.eu +# --vmid N guest VMID to provision (default 9201) +# --golden VOLID golden archive volid (default: newest vzdump of the +# golden build VMID on the archive storage) +# --golden-vmid N golden build guest vmid for auto-discovery (default 9100) +# --archive-storage NAME storage holding the golden vzdump (default local) +# --node NAME PVE node name (default: pvesh /nodes, else hostname) +# --bridge-ip IP[:PORT] local-api listen addr (default: vmbr0 IP : 8443) +# --rootfs-grow N grow OS rootfs by N GiB (default: auto-compute) +# --datavol-grow N grow Docker-data vol by N GiB (default: auto-compute) +# --sysdata-grow N grow user-data vol by N GiB (default: auto-compute) +# --passphrase-file PATH read the retrieval passphrase from a 0600 file +# (default: secure no-echo prompt) +# --preserve-from PATH merge non-Day-0 sections (privileged/storage/backup/ +# local_api/authz/lan_resolver) from an existing config +# --force allow provisioning over an EXISTING vmid (destructive) +# --dry-run print every mutating command without executing +# --resume skip steps already recorded in the state file +# -h, --help this help +# +# State (idempotent/resumable): /var/lib/felhom-install/state.json +# Agent config written 0600 to the systemd unit's -config path +# (auto-detected; else /etc/felhom-agent/agent.json). +# +# SECURITY: the passphrase is read no-echo or from a 0600 file — never a CLI arg, +# never echoed, never written to the state file or logs. The minted pve-token +# secret + per-host hub api_key live ONLY in the agent config (0600, root). +#=============================================================================== + +set -euo pipefail + +SCRIPT_VERSION="1.0.0" + +#------------------------------------------------------------------------------- +# Logging (mirrors felhom-controller/scripts/docker-setup.sh) +#------------------------------------------------------------------------------- +RED='\033[0;31m'; GREEN='\033[0;32m'; YELLOW='\033[1;33m' +BLUE='\033[0;34m'; CYAN='\033[0;36m'; NC='\033[0m' +log_info() { echo -e "${GREEN}[INFO]${NC} $1"; } +log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; } +log_error() { echo -e "${RED}[ERROR]${NC} $1" >&2; } +log_step() { echo -e "${BLUE}[STEP]${NC} $1"; } +log_success() { echo -e "${GREEN}[OK]${NC} $1"; } +log_skip() { echo -e "${CYAN}[SKIP]${NC} $1"; } +log_dry() { echo -e "${CYAN}[DRY-RUN]${NC} $1"; } +die() { log_error "$1"; exit 1; } + +#------------------------------------------------------------------------------- +# Defaults +#------------------------------------------------------------------------------- +CUSTOMER_ID="" +MODE="provision" +HUB_URL="https://hub.felhom.eu" +VMID="9201" +GOLDEN_VOLID="" +GOLDEN_VMID="9100" +ARCHIVE_STORAGE="local" +NODE="" +BRIDGE_ADDR="" +ROOTFS_GROW="" +DATAVOL_GROW="" +SYSDATA_GROW="" +PASSPHRASE_FILE="" +PRESERVE_FROM="" +FORCE=false +DRY_RUN=false +RESUME=false + +PVE_USER="felhom-agent@pve" +PVE_TOKENID="agent" +PVE_ROLE="FelhomAgent" +# The authoritative 16 privileges (agent README; VM.Config.CPUMemory is NOT real, SDN.Use IS required). +PVE_PRIVS="VM.Allocate VM.Audit VM.Config.Disk VM.Config.CPU VM.Config.Memory VM.Config.Network VM.Config.Options VM.PowerMgmt VM.Snapshot VM.Snapshot.Rollback VM.Backup Datastore.Allocate Datastore.AllocateSpace Datastore.Audit Sys.Audit SDN.Use" + +STATE_DIR="/var/lib/felhom-install" +STATE_FILE="${STATE_DIR}/state.json" +AGENT_CONFIG="" # resolved in preflight +HARD_MIN_LVM_GIB=120 # a useful appliance won't fit below this on local-lvm + +# Runtime carriers (never logged) +PASSPHRASE="" +PVE_TOKEN="" # felhom-agent@pve!agent= +HOST_ID="" +HOST_API_KEY="" + +#------------------------------------------------------------------------------- +# Helpers +#------------------------------------------------------------------------------- +usage() { sed -n '2,55p' "$0" | sed 's/^# \{0,1\}//'; exit 0; } + +run() { # simple (no pipes/redirects) mutating command + if $DRY_RUN; then log_dry "$*"; else "$@"; fi +} + +# State helpers (robust JSON via python3). +_state_has() { + [[ -f "$STATE_FILE" ]] || return 1 + STATE_FILE="$STATE_FILE" python3 -c "import json,os,sys;f=os.environ['STATE_FILE'];d=json.load(open(f));sys.exit(0 if sys.argv[1] in d.get('completed',[]) else 1)" "$1" 2>/dev/null +} +_state_mark() { + $DRY_RUN && return 0 + mkdir -p "$STATE_DIR" + STATE_FILE="$STATE_FILE" python3 -c "import json,os,sys;f=os.environ['STATE_FILE'];d=json.load(open(f)) if os.path.exists(f) else {'completed':[]};c=d.setdefault('completed',[]);(c.append(sys.argv[1]) if sys.argv[1] not in c else None);json.dump(d,open(f,'w'),indent=2)" "$1" +} +should_skip() { # returns 0 (skip) if --resume AND step already done + if $RESUME && _state_has "$1"; then log_skip "step '$1' already completed"; return 0; fi + return 1 +} + +http_code() { # GET, prints status code only (read-only preflight) + curl -fsS -o /dev/null -w '%{http_code}' "$@" 2>/dev/null || curl -sS -o /dev/null -w '%{http_code}' "$@" 2>/dev/null +} + +#------------------------------------------------------------------------------- +# Arg parse +#------------------------------------------------------------------------------- +while [[ $# -gt 0 ]]; do + case "$1" in + --customer-id) CUSTOMER_ID="$2"; shift 2 ;; + --mode) MODE="$2"; shift 2 ;; + --hub-url) HUB_URL="$2"; shift 2 ;; + --vmid) VMID="$2"; shift 2 ;; + --golden) GOLDEN_VOLID="$2"; shift 2 ;; + --golden-vmid) GOLDEN_VMID="$2"; shift 2 ;; + --archive-storage) ARCHIVE_STORAGE="$2"; shift 2 ;; + --node) NODE="$2"; shift 2 ;; + --bridge-ip) BRIDGE_ADDR="$2"; shift 2 ;; + --rootfs-grow) ROOTFS_GROW="$2"; shift 2 ;; + --datavol-grow) DATAVOL_GROW="$2"; shift 2 ;; + --sysdata-grow) SYSDATA_GROW="$2"; shift 2 ;; + --passphrase-file) PASSPHRASE_FILE="$2"; shift 2 ;; + --preserve-from) PRESERVE_FROM="$2"; shift 2 ;; + --force) FORCE=true; shift ;; + --dry-run) DRY_RUN=true; shift ;; + --resume) RESUME=true; shift ;; + -h|--help) usage ;; + *) die "Unknown option: $1 (use -h)" ;; + esac +done + +#=============================================================================== +# DR MODE — documented seam only (10D). NOT implemented. +#=============================================================================== +if [[ "$MODE" == "dr" ]]; then + log_error "DR mode not yet implemented (10D)." + cat >&2 <<'EOF' + The DR step skeleton (for the future implementer) mirrors provision EXCEPT + the restore source: + 1. pre-flight (root, PVE, hub reachable, customer+passphrase valid) + 2. pveum token (identical to provision) + 3. host-enroll (mint-once-reuse — the lost host re-binds to its customer) + 4. agent config write (identical) + 5. RESTORE: instead of the golden, restore the customer's OWN whole-CT PBS + snapshot (continuity preserved) — agent --selftest=bring-up -mode dr + -archive . Identity/keys come from escrow + the hub recipe. + 6. verify (identical) +EOF + exit 2 +fi +[[ "$MODE" == "provision" ]] || die "Unknown --mode: $MODE (provision|dr)" + +#=============================================================================== +# PROVISION MODE +#=============================================================================== +[[ -n "$CUSTOMER_ID" ]] || die "--customer-id is required (use -h)" + +echo "" +log_info "felhom-host-install v${SCRIPT_VERSION} — mode=provision customer=${CUSTOMER_ID} vmid=${VMID}" +$DRY_RUN && log_warn "DRY-RUN: no mutations will be performed" +echo "" + +#------------------------------------------------------------------------------- +# Read passphrase (no-echo prompt or 0600 file) — never on argv/logs +#------------------------------------------------------------------------------- +read_passphrase() { + if [[ -n "$PASSPHRASE_FILE" ]]; then + [[ -f "$PASSPHRASE_FILE" ]] || die "--passphrase-file not found: $PASSPHRASE_FILE" + local perm; perm=$(stat -c '%a' "$PASSPHRASE_FILE") + [[ "$perm" == "600" || "$perm" == "400" ]] || log_warn "passphrase file $PASSPHRASE_FILE is mode $perm (want 600)" + PASSPHRASE="$(< "$PASSPHRASE_FILE")"; PASSPHRASE="${PASSPHRASE%$'\n'}" + else + read -rsp "Retrieval passphrase for customer '${CUSTOMER_ID}': " PASSPHRASE; echo "" + fi + [[ -n "$PASSPHRASE" ]] || die "empty passphrase" +} + +#------------------------------------------------------------------------------- +# STEP 1 — pre-flight (fail fast before any mutation) +#------------------------------------------------------------------------------- +step_preflight() { + log_step "1/7 pre-flight" + [[ $EUID -eq 0 ]] || die "must run as root" + command -v pveum >/dev/null || die "pveum not found — is this a Proxmox VE host?" + command -v pct >/dev/null || die "pct not found — is this a Proxmox VE host?" + command -v pvesh >/dev/null || die "pvesh not found" + command -v curl >/dev/null || die "curl not found" + command -v python3>/dev/null || die "python3 not found" + + local pvever; pvever=$(pveversion | head -1) + [[ "$pvever" == *"/9."* ]] || log_warn "expected PVE 9.x, got: $pvever" + log_info " $pvever" + + # node + if [[ -z "$NODE" ]]; then + NODE=$(pvesh get /nodes --output-format json 2>/dev/null | python3 -c "import json,sys;print(json.load(sys.stdin)[0]['node'])" 2>/dev/null || hostname) + fi + log_info " node: $NODE" + + # agent config path: honor the existing systemd unit's -config, else default + if systemctl cat felhom-agent >/dev/null 2>&1; then + AGENT_CONFIG=$(systemctl cat felhom-agent 2>/dev/null | grep -oP '(?<=-config )\S+' | head -1) + fi + [[ -n "$AGENT_CONFIG" ]] || AGENT_CONFIG="/etc/felhom-agent/agent.json" + log_info " agent config: $AGENT_CONFIG" + command -v felhom-agent >/dev/null || die "felhom-agent binary not installed (fetch/install it first — see scripts/README.md)" + log_info " agent: $(felhom-agent --version 2>&1 | head -1)" + + # local-lvm free space + local free_gib + free_gib=$(lvs --noheadings --units g -o lv_size,data_percent /dev/pve/data 2>/dev/null | awk '{gsub(/[^0-9.]/,"",$1); used=$2; print int($1*(100-used)/100)}' 2>/dev/null || echo 0) + if [[ "${free_gib:-0}" -gt 0 ]]; then + log_info " local-lvm free: ~${free_gib} GiB" + [[ "$free_gib" -ge "$HARD_MIN_LVM_GIB" ]] || log_warn "local-lvm free ~${free_gib} GiB < hard min ${HARD_MIN_LVM_GIB} GiB" + else + log_warn " could not read local-lvm free space (continuing)" + fi + + # hub reachable + local hc; hc=$(http_code "$HUB_URL/api/v1/config/$CUSTOMER_ID" -H "X-Retrieval-Password: preflight-no-op" || echo 000) + [[ "$hc" != "000" ]] || die "hub unreachable at $HUB_URL" + log_info " hub reachable ($HUB_URL)" + + # customer + passphrase valid (read-only GET /config/{id}) + read_passphrase + local code; code=$(http_code "$HUB_URL/api/v1/config/$CUSTOMER_ID" -H "X-Retrieval-Password: $PASSPHRASE") + case "$code" in + 200) log_success " customer '$CUSTOMER_ID' exists + passphrase valid" ;; + 401) die "passphrase REJECTED (401) for customer '$CUSTOMER_ID'" ;; + 404) die "customer '$CUSTOMER_ID' not found in hub (404) — create it in the hub first" ;; + *) die "unexpected hub status $code on config preflight" ;; + esac + + # golden archive + if [[ -z "$GOLDEN_VOLID" ]]; then + GOLDEN_VOLID=$(pvesm list "$ARCHIVE_STORAGE" --content backup 2>/dev/null | awk -v v="$GOLDEN_VMID" '$0 ~ ("vzdump-lxc-" v "-"){print $1}' | sort | tail -1) + fi + [[ -n "$GOLDEN_VOLID" ]] || die "no golden archive found for vmid $GOLDEN_VMID on $ARCHIVE_STORAGE — run build-golden.sh first" + pvesm list "$ARCHIVE_STORAGE" --content backup 2>/dev/null | grep -q "$(basename "$GOLDEN_VOLID")" || die "golden volid not resolvable: $GOLDEN_VOLID" + log_info " golden: $GOLDEN_VOLID" + + # vmid guard + if pct status "$VMID" >/dev/null 2>&1; then + if $FORCE; then + log_warn " vmid $VMID already exists — --force given, it WILL be destroyed by provision" + else + die "vmid $VMID already exists. Refusing to clobber a live guest. Pass --force to provision over it." + fi + fi + _state_mark preflight + log_success "pre-flight passed" +} + +#------------------------------------------------------------------------------- +# STEP 2 — Proxmox API token (idempotent pveum; reuse-if-working else rotate) +#------------------------------------------------------------------------------- +step_token() { + log_step "2/7 Proxmox API token" + if should_skip token && [[ -n "$PVE_TOKEN" ]]; then return 0; fi + + # role: create or modify to the exact 16 privs + if pveum role list --output-format json 2>/dev/null | python3 -c "import json,sys;sys.exit(0 if any(r['roleid']=='$PVE_ROLE' for r in json.load(sys.stdin)) else 1)"; then + log_info " role $PVE_ROLE exists — ensuring exact privileges" + run pveum role modify "$PVE_ROLE" -privs "$PVE_PRIVS" + else + run pveum role add "$PVE_ROLE" -privs "$PVE_PRIVS" + fi + + # user: tolerate-exists + if pveum user list --output-format json 2>/dev/null | python3 -c "import json,sys;sys.exit(0 if any(u['userid']=='$PVE_USER' for u in json.load(sys.stdin)) else 1)"; then + log_info " user $PVE_USER exists" + else + run pveum user add "$PVE_USER" + fi + + # token: reuse if the existing agent config token still authenticates, else rotate + local reused=false + if [[ -f "$AGENT_CONFIG" ]] && python3 -c "import json,sys;d=json.load(open('$AGENT_CONFIG'));sys.exit(0 if d.get('proxmox',{}).get('token') else 1)" 2>/dev/null; then + log_info " existing agent config has a token — testing it (read-only --selftest)" + if felhom-agent --config "$AGENT_CONFIG" --selftest >/dev/null 2>&1; then + log_success " existing token authenticates — REUSING (no rotation)" + PVE_TOKEN=$(python3 -c "import json;print(json.load(open('$AGENT_CONFIG'))['proxmox']['token'])") + reused=true + else + log_warn " existing token failed selftest — will rotate" + fi + fi + + if ! $reused; then + if $DRY_RUN; then + log_dry "pveum user token remove $PVE_USER $PVE_TOKENID # if present" + log_dry "pveum user token add $PVE_USER $PVE_TOKENID --privsep 1 --output-format json # capture .value" + PVE_TOKEN="${PVE_USER}!${PVE_TOKENID}=" + else + if pveum user token list "$PVE_USER" --output-format json 2>/dev/null | python3 -c "import json,sys;sys.exit(0 if any(t['tokenid']=='$PVE_TOKENID' for t in json.load(sys.stdin)) else 1)"; then + log_info " removing stale token $PVE_TOKENID (secret unrecoverable — rotating)" + pveum user token remove "$PVE_USER" "$PVE_TOKENID" + fi + local secret + secret=$(pveum user token add "$PVE_USER" "$PVE_TOKENID" --privsep 1 --output-format json | python3 -c "import json,sys;print(json.load(sys.stdin)['value'])") + [[ -n "$secret" ]] || die "failed to capture new token secret" + PVE_TOKEN="${PVE_USER}!${PVE_TOKENID}=${secret}" + log_success " token minted (secret captured, not logged)" + fi + fi + + # Both ACL grants — AFTER the token exists (the single most common 403 cause). + # `pveum user token remove` PURGES the token's ACL, so re-applying here (post-rotate) + # is mandatory; `acl modify` is idempotent so this is also safe on the reuse path. + run pveum acl modify / -user "$PVE_USER" -role "$PVE_ROLE" + run pveum acl modify / -token "${PVE_USER}!${PVE_TOKENID}" -role "$PVE_ROLE" + _state_mark token +} + +#------------------------------------------------------------------------------- +# STEP 3 — compute grows (floors) if not passed +#------------------------------------------------------------------------------- +step_grows() { + log_step "3/7 compute volume grows" + # Golden base: rootfs 32G + Docker-data 16G + user-data 8G (build-golden.sh). + if [[ -z "$ROOTFS_GROW$DATAVOL_GROW$SYSDATA_GROW" ]]; then + local free_gib + free_gib=$(lvs --noheadings --units g -o lv_size,data_percent /dev/pve/data 2>/dev/null | awk '{gsub(/[^0-9.]/,"",$1); used=$2; print int($1*(100-used)/100)}' 2>/dev/null || echo 0) + # Reserve headroom; split the rest ~ docker 80% / sysdata 20%; rootfs stays golden. + ROOTFS_GROW=0 + if [[ "${free_gib:-0}" -ge 300 ]]; then + DATAVOL_GROW=184; SYSDATA_GROW=42 # reproduces the standard 200G/50G appliance + elif [[ "${free_gib:-0}" -ge 150 ]]; then + DATAVOL_GROW=84; SYSDATA_GROW=22 + else + DATAVOL_GROW=34; SYSDATA_GROW=12 # minimal floors + fi + log_info " auto-computed from ~${free_gib} GiB free" + fi + ROOTFS_GROW="${ROOTFS_GROW:-0}"; DATAVOL_GROW="${DATAVOL_GROW:-0}"; SYSDATA_GROW="${SYSDATA_GROW:-0}" + log_info " grows: rootfs +${ROOTFS_GROW}G (->$((32+ROOTFS_GROW))G), docker +${DATAVOL_GROW}G (->$((16+DATAVOL_GROW))G), sys_drive +${SYSDATA_GROW}G (->$((8+SYSDATA_GROW))G)" + _state_mark grows +} + +#------------------------------------------------------------------------------- +# STEP 4 — host enroll (option C; single secret, no global key) +#------------------------------------------------------------------------------- +step_enroll() { + log_step "4/7 host enrollment (POST /host-enroll)" + if $DRY_RUN; then + log_dry "curl -fsS -X POST $HUB_URL/api/v1/host-enroll -H 'X-Retrieval-Password: ' -d '{\"customer_id\":\"$CUSTOMER_ID\"}'" + HOST_ID=""; HOST_API_KEY=""; _state_mark enroll; return 0 + fi + local resp code body + resp=$(curl -sS -w $'\n%{http_code}' -X POST "$HUB_URL/api/v1/host-enroll" \ + -H "X-Retrieval-Password: $PASSPHRASE" -H 'Content-Type: application/json' \ + -d "{\"customer_id\":\"$CUSTOMER_ID\"}") + code=$(tail -n1 <<<"$resp"); body=$(sed '$d' <<<"$resp") + case "$code" in + 201) log_success " host MINTED (first enroll)" ;; + 200) log_success " host REUSED (idempotent — existing credential)" ;; + 401) die "host-enroll 401 (passphrase) — should have been caught in preflight" ;; + 404) die "host-enroll 404 (unknown customer)" ;; + *) die "host-enroll unexpected $code: $body" ;; + esac + HOST_ID=$(python3 -c "import json,sys;print(json.loads(sys.argv[1])['host_id'])" "$body") + HOST_API_KEY=$(python3 -c "import json,sys;print(json.loads(sys.argv[1])['api_key'])" "$body") + [[ -n "$HOST_ID" && -n "$HOST_API_KEY" ]] || die "host-enroll: malformed response" + log_info " host_id: $HOST_ID (api_key captured, not logged)" + _state_mark enroll +} + +#------------------------------------------------------------------------------- +# STEP 5 — write agent config + ensure service healthy +#------------------------------------------------------------------------------- +step_agent_config() { + log_step "5/7 agent config + service" + # TLS pin: the SERVED leaf cert fingerprint (not pvesh node info — may differ) + local fp + fp=$(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 2>/dev/null | sed 's/.*=//') + [[ -n "$fp" ]] || log_warn " could not compute TLS fingerprint (leaving empty — agent will use system trust)" + # bridge / local-api addr + if [[ -z "$BRIDGE_ADDR" ]]; then + local ip; ip=$(ip -4 -o addr show vmbr0 2>/dev/null | awk '{print $4}' | cut -d/ -f1 | head -1) + BRIDGE_ADDR="${ip:-127.0.0.1}:8443" + elif [[ "$BRIDGE_ADDR" != *:* ]]; then + BRIDGE_ADDR="${BRIDGE_ADDR}:8443" + fi + log_info " node=$NODE local_api=$BRIDGE_ADDR tls_fp=${fp:0:17}…" + + if $DRY_RUN; then + log_dry "write $AGENT_CONFIG (0600): proxmox{endpoint,node=$NODE,token=,tls.fingerprint=$fp} hub{url=$HUB_URL,host_id=$HOST_ID,api_key=} local_api{$BRIDGE_ADDR}" + log_dry "systemctl restart felhom-agent && felhom-agent --config $AGENT_CONFIG --selftest" + _state_mark agent_config; return 0 + fi + + mkdir -p "$(dirname "$AGENT_CONFIG")" + # Build config: optional preserve base + fresh-host defaults + Day-0 overrides. + # Secrets passed via env (NOT argv) to avoid ps exposure. + PVE_TOKEN="$PVE_TOKEN" HOST_API_KEY="$HOST_API_KEY" \ + NODE="$NODE" FP="$fp" HUB_URL="$HUB_URL" HOST_ID="$HOST_ID" BRIDGE_ADDR="$BRIDGE_ADDR" \ + PRESERVE_FROM="$PRESERVE_FROM" OUT="$AGENT_CONFIG" python3 <<'PY' +import json, os +out = os.environ['OUT'] +base = {} +pf = os.environ.get('PRESERVE_FROM','') +if pf and os.path.exists(pf): + try: base = json.load(open(pf)) + except Exception: base = {} +# fresh-host defaults for any section not preserved +base.setdefault('log_level','info') +base.setdefault('privileged', {"mode":"direct","unit_dir":"/etc/systemd/system","stage_dir":"/var/lib/felhom-agent/units","systemctl":"/usr/bin/systemctl","install":"/usr/bin/install","smartctl":"/usr/sbin/smartctl","lvs":"/usr/sbin/lvs"}) +base.setdefault('storage', {"watchdog_interval_seconds":5,"watchdog_debounce_seconds":15,"known_refresh_seconds":20}) +base.setdefault('backup', {"local_backup_target":"local","restore_storage":"local-lvm","restore_test_cadence_seconds":0,"scratch_vmid_min":990000,"scratch_vmid_max":990009,"pbs_secret_dir":"/etc/pve/priv/storage","backup_cadence_seconds":0}) +base.setdefault('local_api', {}) +base['local_api'].setdefault('enable', True) +base['local_api']['listen_addr'] = os.environ['BRIDGE_ADDR'] +base['local_api'].setdefault('cert_file','/var/lib/felhom-agent/local-api.crt') +base['local_api'].setdefault('key_file','/var/lib/felhom-agent/local-api.key') +base['local_api'].setdefault('token_store','/var/lib/felhom-agent/local-tokens.log') +base.setdefault('lan_resolver', {"enable": True}) +# Day-0 overrides (always authoritative) +base['proxmox'] = { + "endpoint":"https://127.0.0.1:8006", + "node": os.environ['NODE'], + "token": os.environ['PVE_TOKEN'], + "tls": {"fingerprint": os.environ['FP'], "insecure_skip_verify": False}, +} +base['hub'] = { + "url": os.environ['HUB_URL'], + "host_id": os.environ['HOST_ID'], + "api_key": os.environ['HOST_API_KEY'], + "poll_seconds": base.get('hub',{}).get('poll_seconds',900), + "timeout_seconds": base.get('hub',{}).get('timeout_seconds',30), +} +fd = os.open(out, os.O_WRONLY|os.O_CREAT|os.O_TRUNC, 0o600) +with os.fdopen(fd,'w') as f: + json.dump(base, f, indent=2); f.write('\n') +PY + chmod 600 "$AGENT_CONFIG" + log_success " wrote $AGENT_CONFIG (0600)" + + # health: read-only selftest (proxmox) must pass before provisioning + if ! felhom-agent --config "$AGENT_CONFIG" --selftest >/dev/null 2>&1; then + felhom-agent --config "$AGENT_CONFIG" --selftest 2>&1 | tail -20 >&2 + die "agent --selftest FAILED with the new config (token/ACL/TLS problem) — fix before provisioning" + fi + log_success " agent --selftest (read-only) passed" + + # restart the daemon (host-report loop) and confirm a report lands + if systemctl list-unit-files felhom-agent.service >/dev/null 2>&1; then + run systemctl enable felhom-agent >/dev/null 2>&1 || true + run systemctl restart felhom-agent + log_info " felhom-agent service restarted" + else + log_warn " no felhom-agent systemd unit — daemon host-report loop not started (provision one-shot still works)" + fi + _state_mark agent_config +} + +#------------------------------------------------------------------------------- +# STEP 6 — provision (golden restore -> resize -> bootstrap.json -> onboot:1) +#------------------------------------------------------------------------------- +step_provision() { + log_step "6/7 provision guest $VMID" + # NOTE: -hub-password is passed on argv (the agent's only input for it) — briefly + # visible in ps. Tracked as an Observation (candidate: env/stdin in the agent). + if $DRY_RUN; then + log_dry "felhom-agent --config $AGENT_CONFIG --selftest=provision -archive $GOLDEN_VOLID -vmid $VMID -customer-id $CUSTOMER_ID -hub-password -rootfs-grow $ROOTFS_GROW -datavol-grow $DATAVOL_GROW -sysdata-grow $SYSDATA_GROW" + _state_mark provision; return 0 + fi + if ! felhom-agent --config "$AGENT_CONFIG" --selftest=provision \ + -archive "$GOLDEN_VOLID" -vmid "$VMID" \ + -customer-id "$CUSTOMER_ID" -hub-password "$PASSPHRASE" \ + -rootfs-grow "$ROOTFS_GROW" -datavol-grow "$DATAVOL_GROW" -sysdata-grow "$SYSDATA_GROW"; then + die "provision FAILED — see the agent error above. Fix and re-run with --resume." + fi + log_success " provision completed" + _state_mark provision +} + +#------------------------------------------------------------------------------- +# STEP 7 — verify +#------------------------------------------------------------------------------- +step_verify() { + log_step "7/7 verify" + if $DRY_RUN; then log_dry "pct status/config $VMID; docker ps in-guest; host-report includes $VMID"; return 0; fi + local ok=true + local st; st=$(pct status "$VMID" 2>/dev/null | awk '{print $2}') + [[ "$st" == "running" ]] && log_success " pct status: running" || { log_error " pct status: $st"; ok=false; } + if pct config "$VMID" 2>/dev/null | grep -q '^onboot: 1'; then log_success " onboot: 1"; else log_error " onboot NOT 1"; ok=false; fi + pct config "$VMID" 2>/dev/null | grep -E '^(rootfs|mp0|mp1|mp8):' | sed 's/^/ /' + # controller container healthy in-guest + local cstat; cstat=$(pct exec "$VMID" -- docker ps --filter name=felhom-controller --format '{{.Status}}' 2>/dev/null | head -1) + if [[ -n "$cstat" ]]; then log_success " controller: $cstat"; else log_warn " controller container not visible yet (may still be starting)"; fi + local cver; cver=$(pct exec "$VMID" -- docker ps --filter name=felhom-controller --format '{{.Image}}' 2>/dev/null | head -1) + [[ -n "$cver" ]] && log_info " controller image: $cver" + # tunnel + local tun; tun=$(pct exec "$VMID" -- docker ps --filter name=cloudflared --format '{{.Status}}' 2>/dev/null | head -1) + [[ -n "$tun" ]] && log_info " cloudflared: $tun" || log_warn " cloudflared not visible yet" + # host-report includes the guest (best-effort via the agent's hub selftest) + log_info " (confirm in the hub UI that host $HOST_ID reports guest $VMID)" + _state_mark verify + echo "" + if $ok; then + log_success "Day-0 provision SUCCESS — vmid=$VMID host_id=$HOST_ID customer=$CUSTOMER_ID golden=$GOLDEN_VOLID" + else + log_warn "Day-0 provision completed WITH WARNINGS — review the checks above" + fi +} + +#------------------------------------------------------------------------------- +# Main +#------------------------------------------------------------------------------- +trap 'PASSPHRASE=""; PVE_TOKEN=""; HOST_API_KEY=""' EXIT + +if $RESUME && _state_has preflight; then + # still need the passphrase for enroll/provision even on resume + read_passphrase + # re-resolve cheap derived values skipped steps would have set + [[ -n "$NODE" ]] || NODE=$(pvesh get /nodes --output-format json 2>/dev/null | python3 -c "import json,sys;print(json.load(sys.stdin)[0]['node'])" 2>/dev/null || hostname) + if [[ -z "$AGENT_CONFIG" ]] && systemctl cat felhom-agent >/dev/null 2>&1; then + AGENT_CONFIG=$(systemctl cat felhom-agent 2>/dev/null | grep -oP '(?<=-config )\S+' | head -1) + fi + [[ -n "$AGENT_CONFIG" ]] || AGENT_CONFIG="/etc/felhom-agent/agent.json" + # Backfill display values from the already-written config so the summary is complete. + [[ -f "$AGENT_CONFIG" ]] && HOST_ID=$(python3 -c "import json;print(json.load(open('$AGENT_CONFIG')).get('hub',{}).get('host_id',''))" 2>/dev/null || true) + log_skip "pre-flight (resumed)" +else + step_preflight +fi + +should_skip token || step_token +should_skip grows || step_grows +should_skip enroll || step_enroll +should_skip agent_config || step_agent_config +should_skip provision || step_provision +step_verify