diff --git a/CONTEXT.md b/CONTEXT.md index de358b5..109df0d 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -3,6 +3,21 @@ > Created with the REUSE.md rollout (2026-07-03). Authoritative history: `hub/CHANGELOG.md` (hub), > `website/CHANGELOG.md`, `scripts/CHANGELOG.md`; end-of-task detail in `REPORT.md`. +- **2026-07-16 — R-21 SLICE A SHIPPED: bare-metal Felhom ISO pipeline (`scripts/iso/`, scripts + v1.16.0).** A DooPlex pipeline (`build-felhom-iso.sh` + `Dockerfile.assistant`) turns the official + PVE ISO into a Felhom auto-install ISO whose first-boot stub installs a retry-forever + `felhom-bootstrap` unit that unattended-fetches `felhom-host-install.sh` from the public + `felhom.eu/scripts/` channel and runs it (host-install **unmodified**). Validated on VM 310: + build gate + red-proof, disk-filter fail-safe, stub→unit→real-fetch→invoke→retry, resume-decision, + exactly-once, no-net retry+recovery — all GREEN. **Rulings settled:** stub is from-iso (no + cert-rotation risk); ordering fully-up; the retry unit owns all network work. **§4.4 finding + (shapes slice C):** the distribution URL carries no secret, but the unattended install needs the + customer retrieval passphrase → **the slice-A ISO is secret-bearing** (supervised/single-use; the + bootstrap shreds the env on success). OPEN: host-install rc-0 terminal success (hub-enrolled + + guest-provisioned) is operator-gated — a drill customer needs the password-gated create-UI (a + live-`hub.db` insert is unsafe). Slice B = physical N100 RUNBOOK; slice C = claim-pairing to + de-secret the distributed ISO. Detail: `REPORT.md`, `documentation/audits/SPIKE-baremetal-iso-2026-07-16.md`. + - **2026-07-13 eve — ESCROW CEREMONY WIZARD SHIPPED (agent v0.88.0 + controller v0.127.0; this repo: host-install v1.16.0 + RUNBOOK rewrite).** The recovery-code ceremony is now customer-driveable from the controller (`/backup/escrow`): one-shot R reveal, re-stage-first, diff --git a/REPORT.md b/REPORT.md index 58f531e..9a8d7b1 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,126 +2,125 @@ > **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md). -## Capability-map audit — verify every status claim against evidence — 2026-07-16 +## R-21 slice A: bare-metal Felhom ISO (build pipeline + first-boot bootstrap) — 2026-07-16 -**Task:** docs-only audit of `documentation/architecture/00-capability-map.md`. Open every citation, -confirm/correct/demote each of the 55 rows against the actual evidence doc (scenario-specific PASS, -not topic adjacency), prove the MISSING rows absent by code sweep, close ROADMAP R-20, and swap the -draft caveat in the map header. **Zero code changes; read-only against three sibling repos.** +**Repo:** felhom.eu · **Baseline:** `main` @ `914d51a` → scripts **v1.16.0** (ISO pipeline tag; +`felhom-host-install` was *already* v1.16.0 from 2026-07-13 — see Versioning note). +**Validation:** nested VM **310** on felhom-pve; ISO builds on DooPlex. `felhom-host-install.sh` +**not modified** (read-only, invoked only). Hub code untouched. -### 1. Confirmed baselines (re-confirmed at task start) +### Files created (`scripts/iso/`) -| Repo | `main` @ commit | Role | -|------|-----------------|------| -| felhom.eu | `4b18cc5` (local == origin) | audit target — WRITE | -| felhom-controller | `e99c675` (v0.137.0) | read-only reference | -| felhom-agent | `c040c18` (v0.88.0) | read-only reference | -| app-catalog-felhom.eu | `21e8df1` | read-only reference | +`Dockerfile.assistant`, `build-felhom-iso.sh`, `answer.toml.tmpl`, `stub-first-boot.sh`, +`felhom-bootstrap.sh`, `felhom-bootstrap.service`, `profiles/nested-vm.profile`, +`profiles/nested-canary.profile`, `README.md`. Plus `scripts/CHANGELOG.md`, +`documentation/backlog/ROADMAP.md` (R-21 → in-progress), this `REPORT.md`, `CONTEXT.md`. -The map's evidence corpus is read at felhom.eu tip `4b18cc5`; the header caveat now records that. +### Green gates -### 2. Counts +`bash -n` clean on all three scripts; `shellcheck -x` **exit 0, no findings** (run in the assistant +container). Retry-unit behavior de-risked on systemd 257 before wiring (transient unit: `Type=oneshot` ++ `Restart=on-failure` **does** retry — NRestarts climbed). -**55 rows audited = 23 confirmed (18 confirmed + 5 confirmed-absent) + 26 corrected + 6 demoted.** -No row was upgraded (candidate upgrades are Observations). Status distribution after the audit: -**32 PROVEN-LIVE / 15 IMPLEMENTED / 3 PARTIAL / 5 MISSING** (was 38 PROVEN-LIVE → 6 demoted). +### §4.4 finding — does the distribution channel embed a secret? (decides slice C) -### 3. Per-row verdict table +**The channel: no. The unattended install: yes (the passphrase).** The hub install-command generator +(`hub/internal/web/templates/customer_unified.html`, Option 1) emits +`curl -fsSL https://felhom.eu/scripts/felhom-host-install.sh -o … && sudo bash … --customer-id +--mode ` — a **public URL, no token/secret in the command**. But +`felhom-host-install.sh` **requires the customer retrieval passphrase** (`read_passphrase` → +`die "empty passphrase"`; used for config fetch + `POST /host-enroll`). Unattended ⇒ it must be +supplied via `--passphrase-file`, so **the slice-A ISO is secret-bearing** (embeds the passphrase). +**Slice-C implication:** the *distributed* ISO must NOT bake the passphrase — the claim/pairing model +must deliver it (or the config) to the box *after* the customer claims it. Slice-A ISOs are +supervised/single-use; the bootstrap `shred`s `bootstrap.env` on success. -Each verdict names the doc opened and the leg/section carrying (or failing to carry) the scenario. +### 2.3 retry-vs-resume ruling (source-verified, encoded once) -| Row | Scenario | Verdict | Evidence opened → leg / reason | -|-----|----------|---------|--------------------------------| -| A1 | Appliance day-0 zero-click auto-confirm | **CONFIRMED** | `DRILL-day0-take2` §2/§3 (auto-confirm 6m39s, zero clicks; floor self-update 0.120→0.122; claim gate closed) + `DRILL-day0-vm` §5.6 | -| A2 | BYO install (PARTIAL) | **CONFIRMED** | `DRILL-GL6` Phase 2 byo happy-path PASS; coexistence guards (GL6-F3/F6) only fixed v1.12.0, never live-revalidated — PARTIAL right | -| A3 | Customer claim | **CORRECTED** | `DRILL-day0-vm` §10/F-4 carries it; dropped mis-cited `CAMPAIGN-4` F-C (that is the escrow-claim 502, not the password claim) | -| A4 | Escrow ceremony | **CORRECTED** | agent v0.88 REPORT (claim 200→410, R absent). Qualifier fixed demo→**drill VM**; browser-wizard leg flagged not-yet-validated | -| A5 | DR tier by default (IMPLEMENTED) | **CORRECTED** | cited `SPIKE-pbs-tier-provisioning` = slice-0, shipped nothing → `DRILL-day0-take2` §2 (WG both modes, PBS-DR descriptor auto-provisioned live). Upgrade candidate | -| A6 | Uninstall (PARTIAL) | **CORRECTED** | `DRILL-GL6` Phase 1/5 (KEPT-vs-WIPED printed, data intact ×3); secret-purge fixed v1.12.0, `mnt-*.mount` residue open — PARTIAL right | -| B1 | Deploy app from catalog | **CORRECTED** | `CAMPAIGN-2` T-DEPLOY-SET + `RERUN-p1p3` PASS; dropped T-RES-MEMGUARD (guard never *fired* — ample RAM / auth-walled) | -| B2 | App lifecycle | **CORRECTED** | T-LIFECYCLE PASS + remove live in `CAMPAIGN-3`; T-REMOVE-REDEPLOY never cleanly passed (stale dryrun) — noted open edge | -| B3 | Protected infra un-stoppable | **CORRECTED** | cited `CAMPAIGN-2` T-SEC-PROTECTED was a stale-dryrun FAIL → `CAMPAIGN-nomercy` + `RERUN-p1p3` T-SEC-PROTECTED PASS | -| B4 | Catalog sync + validation choke | **CONFIRMED** | `CAMPAIGN-2` T-SYNC-IDEMPOTENT PASS; choke-point is unit-red-proofed (v0.132) — status held | -| B5 | App crash → customer notified (IMPLEMENTED) | **CORRECTED** | citation `CAMPAIGN-3 fix-3 arc` (surfaced the gap) → controller v0.120.0 (impl+red-proof); "delivery proven" softened (6B deferred/6C inconclusive) | -| B6 | Post-deploy config + restart | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | T-PAGE-ALL is a GET-only page-render smoke test; config-save+restart exercised in no campaign (CAMPAIGN-3 skipped it) | -| B7 | Backup classification (13 apps) | **CONFIRMED** | `SPIKE-backup-classification` SQ2 (13 bind apps) + 6D/6E live legs | -| C1 | Nightly DB dumps, atomic writes | **CORRECTED** | cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes + T-RST-DB auth-hollow → `CAMPAIGN-6B` P4 + `6E` B1/B2 (SIGKILL→last-good byte-unchanged) + `6D` P-FAB | -| C2 | Tier-2 copy | **CONFIRMED** | `CAMPAIGN-6E` P-TIER2 deep-4 4/4 PASS + `6C` | -| C3 | Offsite restic | **CORRECTED** | enlargement gate live (`6D` P3-DELIVERY); noted raw-data-quota/retention are `SPIKE-restic-snapshot-shape` **dry-run** only | -| C4 | Offsite restore | **CONFIRMED** | `CAMPAIGN-6D` P-IMMICH/P-PLACE (functional from offsite alone, byte-identical) | -| C5 | Manual `.fab` | **CORRECTED** | `6D` P-FAB (1.7GB circle) + `6B` P2 (chunking at real CF edge via `curl --resolve`); rendered-browser upload leg still deferred; C6B-F1 was the 6B *finding* | -| C6 | Guest-loss PBS DR | **CORRECTED** | cited `VALIDATION-newbox-restore` = offbox **restic** (wrong tier) → `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore`→healthy) + T-PBS-VERIFY (13 snapshots) | -| C7 | PBS-DR self-heal (IMPLEMENTED) | **CORRECTED** | cited spike = findings-only → hub v0.56.0 reconciler + §10 red-proofs. Facts fixed: **scoped to one host** (not fleet-wide); **already fired live** on drill qm300 | -| C8 | Crash mid-backup/migration self-heal | **CORRECTED** | cited `CAMPAIGN-2` T-RBT-* legs empty/auth-hollow → `6D` P5-REST + `6E` B1-B3; mid-*migration* crash still weakest sub-claim | -| C9 | Soft-quota block + notify | **CONFIRMED** | `6D` P3-DELIVERY (block + op+customer email live); usage-bar UI only weakly corroborated | -| C10 | Customer performs a restore (MISSING) | **CONFIRMED-ABSENT** | sweep: only self-reference at map L34 "never executed by a non-Viktor human"; all restores operator-driven | -| D1 | Drive wizard incl. legacy LVM-root | **CORRECTED** | `DISPOSITION-ia` (legacy EFI+LVM, root not offered) + `storage-lifecycle-acceptance` (enroll live) + agent fence self-test; cited CAMPAIGN-2 legs were auth-hollow; UI-wizard leg supervised | -| D2 | Data migration (all/per-app) | **CORRECTED** | `6C` 4P-5 (per-app) + `storage-lifecycle-acceptance` (2× migrate-all via UI, byte-identical); cited CAMPAIGN-2 legs auth-hollow; crash-safe design-level only | -| D3 | NAS client verify-before-commit | **CONFIRMED** | `SPIKE-nas-verify` Q2/Q4 + `SPIKE-nas-dsm` (real DSM 7.2 end-to-end) + `CAMPAIGN-3` | -| D4 | USB enroll/unplug/recommission | **CORRECTED** | `storage-lifecycle-acceptance` E4 (yank→auto-rebind) + E10 (re-enroll) + `4/6A` (3-USB reshuffle); cited RUNBOOK could not complete enroll | -| D5 | Decommission / eject | **CORRECTED** | `storage-lifecycle-acceptance` E9 (decommission-anyway, reboot-safe) + E12; cited CAMPAIGN-2 legs auth-hollow, SPIKE was report-only | -| D6 | Boot ordering survives reboot | **CORRECTED** | **cited `CAMPAIGN-3` F10/F11/F12 are the CRITICAL/HIGH failures** → fixes shipped agent v0.85 + re-validated live `CAMPAIGN-4` (0 ordering cycles) + `6A` 1D | -| E1 | Remote CF Tunnel | **CONFIRMED** | `CAMPAIGN-2` T-FLT-CF PASS (not in the H1-hollow set; recovered 20s) | -| E2 | LAN access when net down (IMPLEMENTED) | **CONFIRMED** | empty citation consistent with IMPLEMENTED; never drilled as customer experience → R-19 | -| E3 | Phone photo backup | **CONFIRMED** | `6D` Accept #1 + SQ3 (immich functional from offsite, byte-identical) | -| E4 | Documents/OCR | **CORRECTED** | tightened vague "6C/6E legs" → `6C` 4P-1 (paperless consume+OCR PASS) | -| E5 | SMB **server** (MISSING) | **CONFIRMED-ABSENT** | sweep `smbd\|samba\|wsdd\|net usershare\|smb.conf`: 10 hits, all client/offsite/docs — no server. Hetzner `SambaEnabled` = a box we write to, not a share we expose | -| E6 | DLNA to TV (MISSING) | **CONFIRMED-ABSENT** | sweep `minidlna\|dlna\|ssdp\|upnp`: 3 hits, all ROADMAP/map docs; zero implementation | -| E7 | File access via browser | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | T-PAGE-ALL renders only controller dashboard pages, not FileBrowser; no browse/download proof anywhere | -| E8 | Forgot-password reset code | **CORRECTED** | tightened "F-15 arc" → `DRILL-day0-take2` F-15 (live re-run, hash applied 1s, code accepted first try) | -| E9 | Multi-user accounts (MISSING) | **CONFIRMED-ABSENT** | sweep of controller `auth.go`: single `effectivePasswordHash()`, no user table / role / permission model | -| E10 | WireGuard OOB (IMPLEMENTED) | **CONFIRMED** | `SPIKE-oob-wg-operator-peer` GO + `SPIKE-felhom-sshd`; unproven on true 100.64/10 — IMPLEMENTED right | -| E11 | Break-glass recovery (IMPLEMENTED) | **CONFIRMED** | `runbooks/break-glass.md`; live incident reproduced in SPIKE-felhom-sshd; no end-to-end recovery drill — IMPLEMENTED right | -| F1 | Health-degradation email | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | cited `6D` P3-DELIVERY is the **enlarge-block** trigger, not health-degradation; `NotifyHealthChange` never fired an email live in any doc | -| F2 | Event catalog | **CORRECTED** | vague "per-arc" → 3 live-delivered families: enlarge-block (`6D`), claim code (`DRILL-day0-vm` F-4), reset code (`take2` F-15); `app_start_failed`/dead-app unit-only | -| F3 | Empty-email wipe guard (IMPLEMENTED) | **CONFIRMED** | controller v0.137.0, red-proofed 07-15 | -| F4 | System/container metrics | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | cited T-RES-CPU/T-SOAK-LOOP are H1/H2 harness artifacts (auth-302); SQLite/Chart.js/30-day downsampling validated nowhere | -| F5 | Debug rings + log-bundle pulls | **CORRECTED** | rings live-exercised `CAMPAIGN-3` fix-6; noted log-bundle-pull TTL/custody half is changelog-only | -| F6 | Operator alerting (Healthchecks) | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | no corpus doc; healthchecks-config app not deployed; per enum "no citation → not PROVEN-LIVE". Re-upgrade candidate if operator cites a live alert | -| G1 | Customer/host management UI | **CORRECTED** | cited "daily operator use" no-doc + `AUDIT-hub-gui` predates features (v0.25) → hub v0.53 (Peti-cluster live) + `CAMPAIGN-4` + `take2` F-16; 8-tab render = render-test-validated (UI password-gated) | -| G2 | Publish train (PARTIAL) | **CONFIRMED** | `runbooks/publish-train-rules.md` (floor-LAST, MinAgent gate); never proven on a real remote customer — PARTIAL right | -| G3 | Agent self-update | **CORRECTED** | cited `SPIKE-agent-selfupdate` is dummy-only, no operator-signed leg → `DRILL-GL6` Phase 3 (armed-key round-trip with the REAL key) | -| G4 | Controller self-update | **CORRECTED** | cited "07-10 arc" CHANGELOG → `DRILL-day0-take2` §2 (credential-empty box self-updated 0.120→0.122 live) | -| G5 | Offsite provisioning | **CONFIRMED** | `VALIDATION-offsite-provisioning-e2e` PASS (real box 611714, end-to-end) + `SPIKE-hetzner-api` + re-issue in `CAMPAIGN-offsite-overnight` | -| G6 | Per-customer offsite fill/staleness/freeze | **DEMOTED** PROVEN-LIVE→IMPLEMENTED | `OffsiteChecker` in code but no live-fired leg; `CAMPAIGN-offsite-overnight` = no fill/staleness emails, freeze write-block inconclusive | -| G7 | Box-level Storage Box aggregate (MISSING) | **CONFIRMED-ABSENT** | hub sweep `storagebox\|aggregate\|oversubscription\|sum.*quota`: only per-customer OffsiteChecker + app-telemetry; no box-level roll-up | -| G8 | Secrets hygiene (IMPLEMENTED) | **CONFIRMED** | hub v0.53.0 closing bundle (bearer → k8s Secret) | -| G9 | Operator login password from UI (IMPLEMENTED) | **CONFIRMED** | hub v0.54.0 (`handleChangePassword`, `effectivePasswordHash()`) | +**Plain first attempt; add `--resume` on any later attempt that finds +`/var/lib/felhom-install/state.json`.** Source landmark: `felhom-host-install.sh` header **v1.11.3** +("--resume correctness — the producer steps (token/enroll/grows) run every pass … a resumed install no +longer writes a config missing hub.host_id/proxmox.token") — so `--resume` is safe, whereas a plain +re-invoke over an existing install state would re-hit the populated-host leaf guard / existing-vmid +refusal. `--mode` is required in both forms. Encoded in `felhom-bootstrap.sh` with the landmark cited +inline; **empirically confirmed** (below). -### 4. R-20 verdict — **CLOSED** +### Per-scenario results (evidence under `180:~/felhom-iso/evidence/`) -Read from current `scripts/felhom-host-install.sh` source (not the changelog). Operator-key pinning -is **fully automatic in the day-0 flow, no manual step**: keys resolve at L1181–1219 (script -constants `OPERATOR_KEY_OPERATIONAL_*`/`_RECOVERY_*`, already populated, optional -`--operator-pubkey-file` override); pinned by `step_agent_config()` — "STEP 6/8 agent config + -service" (L2044; embedded python builds `authz.signers` L2146–2156, and a re-install with no keys -preserves an existing box's signers); verified at the step-8 check L2332–2337 ("authz signers: N -(operator-signed self-update armed)"). No interactive prompt or post-install hand-edit gates it → -ROADMAP R-20 set to **closed**. One loose end (doc-drift, not a reopen): the L193–197 comment still -says the constants are "EMPTY by default" though L198–201 are populated — folded into R-16 hygiene. +- **C — build gate blocks a broken answer (+ red-proof) — PASS.** Answer with both `root-password` + and `root-password-hashed`: the output-parse gate surfaced `` `global.root-password` and + `global.root-password-hashed` cannot be set at the same time / Error: Found issues `` and aborted + (`BUILD-EXIT=1`, **no ISO** — count unchanged), *even though `validate-answer` exits 0*. + **Red-proof:** gutting the gate to trust `$?` made it wrongly report `[OK] answer validated`; the + broken answer was then stopped only by `prepare-iso`'s own error **plus the build's `[[ -f out.iso ]]` + guard** — notably `prepare-iso` *also* exited 0 on the error, so the file-existence check was the + real backstop. Reverted (mutation done on the DooPlex copy; committed tree never touched). +- **D — disk-filter fail-safe through the pipeline — PASS.** Pipeline-built canary ISO + (`filter.ID_SERIAL_SHORT="no-such-disk-zzz"`) → `ERROR: Installation failed: filter did not match any + device`, abort exit 1, no interactive fallback. 8 GB canary disk **byte-identical** after + (`sha256 -c: OK`, host-side; `scenarioD-filter-abort.png`). +- **A — full unattended chain — MECHANISM PASS; terminal success operator-gated.** Zero keypress; + stub ran once (uid 0) and enabled+started `felhom-bootstrap`; the unit **fetched host-install from the + REAL channel** `https://felhom.eu/scripts/felhom-host-install.sh` and invoked it + `--customer-id … --mode appliance --hub-url …`; host-install ran unattended through preflight + (hub reachable) — with the drill customer absent it correctly failed at config preflight (`404`) and + the unit **retried** (`bootstrap-journal.log`; NRestarts ≥3; 12 host-install invocations). + **Not live-proven:** host-install **rc-0** → hub-enrolled + guest-provisioned → done-flag + + unit-disable + no-re-run. Reason: that needs a properly created drill customer, and CC cannot use the + password-gated create-UI while a direct write to the *live, locked* hub `hub.db` (verified open by the + running pod) risks corruption. The done-flag/disable/env-scrub success path is code-reviewed + + shellcheck-clean; empirical proof requires an operator-created drill customer (supervised). +- **B — no network at first boot — PASS.** During a host-side `link_down` window + (13:37:56→13:39:37) the chain failed repeatedly — `host-install FAILED rc=6` (host-install's own hub + fetch, no net) and `ERROR: host-install fetch failed (no network yet?)` — and **the unit kept + retrying (never died)**; on link-up it **resumed automatically** (`13:40:04` fetch+run). No + "single failure → dead unit". The stub-runs-without-network property is spike-proven (S8a); + `bootstrap-journal.log`. +- **2.3 resume decision — PASS.** After creating `/var/lib/felhom-install/state.json`, the next attempt + logged `prior install state present (…) -> adding --resume`; earlier attempts ran plain. +- **Exactly-once stub — PASS.** Host-side reboot → `/var/log/felhom-first-boot.log` hash **unchanged** + (`13cf3fe4…`), pending-flag consumed; the retry unit correctly persisted (still `activating`, not done). -### 5. Commits pushed to `main` +### ISO artifacts (build ≈ 12 s each, source already local) -- (recorded on push) — `docs(map): capability-map evidence audit` (00-capability-map.md header + 26 corrected / 6 demoted rows; ROADMAP R-20 closed; this REPORT). +| ISO | sha256 | size | +|---|---|---| +| `felhom-pve-9.2-1-v1.16.0-nested-vm.iso` | `8cbe5eaeaa0d…82bb` | 1,707,737,088 B | +| `felhom-pve-9.2-1-v1.16.0-nested-canary.iso` | `6b4568eaeac0…0301` | 1,707,737,088 B | +| (validation build, key baked) `…-nested-vm-val.iso` | `0042acfdb70a…98e7` | — | -### 6. Observations (out of scope — NOT acted on) +Source ISO `proxmox-ve_9.2-1.iso` sha256 `4e88fe4…f2c6c` (verified vs official SHA256SUMS). Assistant +`proxmox-installer-common 9.2.7` (trixie no-sub). Artifacts kept under `180:~/felhom-iso/out/`, never +committed (secret-bearing). -- **Candidate upgrades (audit cannot upgrade):** **A5** DR-tier-by-default is shown working *live and - hands-free* in `DRILL-day0-take2` §2 — arguably PROVEN-LIVE(drill), left IMPLEMENTED per the - no-upgrade rule. **C7** self-heal has code + red-proofs + a hands-free live convergence on qm300 — - stronger than a bare IMPLEMENTED, but scoped-to-one-host with no real-customer firing. -- **F6 demotion is a judgment call:** operator Healthchecks→Resend alerting may genuinely run in - production; it simply has no corpus doc, and the strict enum forbids PROVEN-LIVE without a citation. - Re-upgrade the moment a live alert is cited. -- **Recurring "cite-the-finding" inversion:** several rows cited the doc where a defect was *discovered* - as if it were the proof — D6 (F10/F11/F12 failures), C1 (F7 non-atomic finding), C6 - (VALIDATION-newbox = wrong tier), C5/B5 (6B/CAMPAIGN-3 findings). The fixes live in later legs. Worth - a standing authoring rule: cite the fix-validation leg, never the finding ID. -- **`storage-lifecycle-acceptance-2026-06-15` is an uncited, high-value corpus doc** — it live-proves - D2/D4/D5 (and local-drive boot survival). Now cited; consider it first for storage rows. -- **CAMPAIGN-2 is fragile as a citation source for restore/reboot/storage rows:** harness bug H1 - (auth-session expiry → CSRF-403) hollowed every `T-RST-*`, `T-STG-*`, `T-RBT-*`; several cited - evidence dirs are literally empty or a 302. Re-anchor such rows on 6D P5-REST / 6E Phase B / - storage-lifecycle-acceptance, which actually exercised these live. -- **MEMORY.md drift:** the note that `CAMPAIGN-6D` "DEFERRED P3-BROWSER/P3-DELIVERY" is stale — the - committed 6D marks both **PASS**. +### VM 310 final state + +Stopped, kept; SeaBIOS, scsi0 32 GB + scsi1 8 GB (canary intact), ISO detached. Boundaries honored: +VM 300 (drill) running untouched, 9000 stopped, LXC 9201 (demo) running untouched, hub untouched. +Secret-bearing pipeline ISOs removed from felhom-pve (their embedded passphrase was a placeholder, not a +real credential). + +### NOT-live-validated + +- **Physical N100 run (RUNBOOK B):** real DMI/firmware, USB-boot media, vendor Secure Boot key set, + real-NIC naming, prior-LVM `blkdiscard` on real hardware. Not started (out of scope by design). +- **host-install rc-0 terminal success** (hub-enrolled + guest-provisioned + done-flag + disable) and + the **2.3 `--resume` executed against a real partial install** — both need an operator-created drill + customer (password-gated create-UI; live-DB insert unsafe). The *decisions/plumbing* are proven; only + the terminal success is deferred. + +### Versioning note + +`felhom-host-install` reached v1.16.0 on 2026-07-13 (before this task's stated 1.15.x baseline). The +new ISO pipeline is tagged **v1.16.0** to stay coherent with that release (the ISO's baked invocation +targets host-install's flag surface). The `scripts/CHANGELOG.md` entry is a new component under the +same release tag, not a bump of host-install. + +### Observations (recorded, not acted on) + +- `prepare-iso` (not just `validate-answer`) was observed to **exit 0 on a bad answer** — the build's + `[[ -f out.iso ]]` output-existence guard is therefore load-bearing, not just belt-and-suspenders. +- host-install prints `unexpected hub status 404404` (doubled) for an absent customer — cosmetic + host-install quirk; left untouched (read-only this task). +- host-install exits **rc=6** when it cannot reach the hub (no net) — the bootstrap correctly treats + any non-zero rc as retry. diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index 3a4a94c..87423a1 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -49,7 +49,7 @@ | R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | idea | | | R-14 | Headscale/WireGuard spike: Minecraft/gaming port connectivity (CGNAT-proof, sovereign DERP fallback) | M | idea | | | R-15 | Multi-user dashboard accounts (household members, roles) | L | idea | Single password is a stated alpha limitation (R-11) | -| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → claim-code pairing (option C) | XL | **spiked** (2026-07-16) | Spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | +| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → claim-code pairing (option C) | XL | **in-progress** (slice A shipped 2026-07-16 — `scripts/iso/`, scripts v1.16.0) | **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B** = physical N100 RUNBOOK (real DMI/firmware/USB/prior-LVM). **Slice C** = claim-pairing so the *distributed* ISO carries no retrieval passphrase (slice-A ISO is secret-bearing, supervised/single-use). Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | ## Absorbed / superseded notes in this folder diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index 7e9ca29..d445462 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,5 +1,30 @@ # Felhom scripts — Changelog +## felhom bare-metal ISO pipeline v1.16.0 — NEW (R-21 slice A, 2026-07-16) + +New operator tooling under `scripts/iso/`: a reproducible DooPlex pipeline that turns the official +PVE ISO into a Felhom auto-install ISO whose first-boot chain unattended-fetches-and-runs +`felhom-host-install.sh` until the host is enrolled + a guest provisioned. `felhom-host-install.sh` +is **unchanged** (invoked, not modified). Tagged **v1.16.0** to align the ISO with the current +host-install release (they must stay coherent; the ISO's baked invocation targets that flag surface). + +- `Dockerfile.assistant` (assistant paired to the ISO's PVE generation by Debian codename + shellcheck), + `build-felhom-iso.sh` (render answer from `answer.toml.tmpl` + a profile, mint a fresh THROWAWAY root + hash per build, **gate validate-answer on OUTPUT TEXT not `$?`** — spike S1 exit-0-on-failure trap, + `prepare-iso --fetch-from iso --on-first-boot`, emit ISO + sha256 + manifest). +- In-ISO chain: `stub-first-boot.sh` (dumb, exactly-once, from-iso/fully-up) installs + `felhom-bootstrap.sh` + `felhom-bootstrap.service` (retry-forever `Type=oneshot`+`Restart=on-failure`) + + a 0600 `bootstrap.env`. Retry-vs-resume encoded once (plain first; `--resume` when + `/var/lib/felhom-install/state.json` exists — safe per host-install v1.11.3). +- **Secret-bearing:** the unattended install needs the customer retrieval passphrase, so the ISO embeds + it (the distribution URL itself carries no secret). Supervised/single-use; the bootstrap scrubs the + env on success. Slice C removes the secret from the *distributed* ISO via claim-pairing. +- Validated on nested VM 310: build gate + red-proof; disk-filter fail-safe through the pipeline; + stub→unit→real public-channel fetch→invocation→retry; resume decision; exactly-once; no-net retry + + recovery. Terminal host-install success (hub-enrolled + guest-provisioned) is operator-gated (drill + customer needs the password-gated create-UI; live-DB insert unsafe). Full report: root `REPORT.md`, + `documentation/audits/SPIKE-baremetal-iso-2026-07-16.md`. + ## felhom-host-install v1.16.0 — FELHOM_ESCROW rides the canonical sudoers (2026-07-13) Companion to agent v0.88.0 + controller v0.127.0 (the controller-driven escrow ceremony wizard). diff --git a/scripts/iso/Dockerfile.assistant b/scripts/iso/Dockerfile.assistant new file mode 100644 index 0000000..9678347 --- /dev/null +++ b/scripts/iso/Dockerfile.assistant @@ -0,0 +1,24 @@ +# scripts/iso/Dockerfile.assistant — the build host for the Felhom bare-metal ISO pipeline (R-21). +# +# Debian trixie == PVE 9.x codename; the proxmox-auto-install-assistant is pinned to the ISO +# generation by installing it from the PVE no-subscription repo of the ISO's Debian codename +# (spike SPIKE-baremetal-iso-2026-07-16.md §15.6: pair by major.minor). Rebuild this image when the +# target PVE generation changes (e.g. PVE 10 -> Debian forky) and record both versions in the build +# manifest. +# +# Build: docker build -f scripts/iso/Dockerfile.assistant -t felhom-iso-assistant:trixie scripts/iso +FROM debian:trixie + +RUN apt-get update && apt-get install -y --no-install-recommends \ + ca-certificates wget gnupg xorriso shellcheck \ + && rm -rf /var/lib/apt/lists/* + +# PVE 9.x (trixie) no-subscription repo — pairs the assistant to the 9.2 ISO generation. +RUN wget -q https://enterprise.proxmox.com/debian/proxmox-release-trixie.gpg \ + -O /etc/apt/trusted.gpg.d/proxmox-release-trixie.gpg +RUN echo "deb http://download.proxmox.com/debian/pve trixie pve-no-subscription" \ + > /etc/apt/sources.list.d/pve.list + +RUN apt-get update && apt-get install -y --no-install-recommends \ + proxmox-auto-install-assistant \ + && rm -rf /var/lib/apt/lists/* diff --git a/scripts/iso/README.md b/scripts/iso/README.md new file mode 100644 index 0000000..bcfff70 --- /dev/null +++ b/scripts/iso/README.md @@ -0,0 +1,126 @@ +# Felhom bare-metal ISO pipeline (R-21 slice A) + +Turns the official Proxmox VE ISO into a **Felhom auto-install ISO** whose first-boot chain, +unattended, fetches and runs `felhom-host-install.sh` until the host is enrolled and a guest is +provisioned. Boot the ISO on blank hardware → walk away → managed Felhom host. + +Grounding: every install/first-boot/webhook mechanism used here is proven in +`documentation/audits/SPIKE-baremetal-iso-2026-07-16.md` (S1–S8). This pipeline **composes** them; +it does **not** modify `felhom-host-install.sh` (which it only invokes over the public distribution +channel). + +> **Scope:** slice A = the pipeline + the in-ISO bootstrap, validated on nested VM 310. The physical +> N100 run is **RUNBOOK B** (separate, supervised). The customer-facing claim/pairing that removes the +> secret from the ISO is **slice C** (not built here). + +## Files + +| File | Role | +|---|---| +| `Dockerfile.assistant` | build host: Debian trixie + `proxmox-auto-install-assistant` (paired to the ISO's PVE generation by Debian codename) + `xorriso` + `shellcheck` | +| `build-felhom-iso.sh` | the pipeline: render answer, mint throwaway root hash, **validate-answer output-parse gate**, render stub, `prepare-iso`, emit ISO + sha256 + manifest | +| `answer.toml.tmpl` | Proxmox answer template (`__FQDN__` / `__ROOT_HASH__` / `__ROOT_SSH_KEYS__` / `__DISK_SETUP__` placeholders; `[first-boot]` from-iso/fully-up) | +| `profiles/*.profile` | disk-selection + fqdn fragments (sourceable, no secret) | +| `stub-first-boot.sh` | the ONE first-boot executable (skeleton; build injects the bootstrap script/unit/env as base64) | +| `felhom-bootstrap.sh` | per-attempt: fetch host-install from the public channel → run it with the retrieval passphrase → on rc 0 write done-flag + disable | +| `felhom-bootstrap.service` | retry-forever unit (`Type=oneshot`, `Restart=on-failure`, `RestartSec=30`, `StartLimitIntervalSec=0`) | + +## Build + +```bash +# 1. build the assistant image once (rebuild when the target PVE generation changes) +docker build -f scripts/iso/Dockerfile.assistant -t felhom-iso-assistant:trixie scripts/iso + +# 2. build an ISO (on the build host; PVE ISO pre-downloaded + its official sha256) +scripts/iso/build-felhom-iso.sh \ + --pve-iso /path/proxmox-ve_9.2-1.iso \ + --iso-sha256 4e88fe416df9b527624a175f24c9aa07c714d3332afb1ee3dbf3879573ef2c6c \ + --profile scripts/iso/profiles/nested-vm.profile \ + --bootstrap-env /secure/bootstrap.env \ + --out ~/felhom-iso/out +``` + +Output: `felhom-pve--v-.iso` + `.sha256` + `.manifest.txt`. + +The build **gates the answer on validate-answer's OUTPUT TEXT, never `$?`** — `validate-answer` +returns exit 0 even on a broken file (spike S1 trap; also `prepare-iso` was observed to exit 0 on a +bad answer). A broken answer aborts the build with the validator's message and produces **no ISO**. +Note `validate-answer` is *syntactic only*: disk existence is a runtime property (a non-existent disk +passes the build and fails-safe at install — spike S5c/S8b). + +## Profile format + +A sourceable shell fragment: + +```sh +FELHOM_FQDN="felhom-host.local" +FELHOM_DISK_SETUP='[disk-setup] +filesystem = "ext4" +disk-list = ["sda"]' +# optional emergency/validation key baked into root's authorized_keys (blank -> not baked): +# FELHOM_ROOT_SSH_KEY="ssh-ed25519 AAAA... ops@felhom" +``` + +`FELHOM_DISK_SETUP` is any valid Proxmox `[disk-setup]` block — a `disk-list` OR a udev `filter.*` +glob. **A filter matching nothing, or a non-existent disk, fails-safe**: the installer aborts (exit +1, no disk touched) rather than installing on the wrong disk (spike S5c). Never commit a real key in +`FELHOM_ROOT_SSH_KEY` unless it is meant to ship in every ISO built from that profile. + +## bootstrap-env (SECRET-BEARING) + +The `--bootstrap-env` file becomes the in-ISO `/etc/felhom/bootstrap.env`: + +```sh +FELHOM_CUSTOMER_ID= # required +FELHOM_MODE=appliance # required (appliance|byo) +FELHOM_RETRIEVAL_PASSPHRASE= # required — SECRET +# optional: +# FELHOM_HUB_URL=https://hub.felhom.eu +# FELHOM_INSTALL_URL=https://felhom.eu/scripts/felhom-host-install.sh +# FELHOM_EXTRA_ARGS="--cores 4 --memory 8192" # profile-only flags, never secrets +``` + +**Why the ISO is secret-bearing (§4.4 finding):** the hub install-command's *distribution channel* is +the public `https://felhom.eu/scripts/felhom-host-install.sh` — no token, no secret in the command. +But `felhom-host-install.sh` requires the customer **retrieval passphrase** (normally a no-echo +prompt) to fetch the config and enroll the host. An *unattended* install must supply it via +`--passphrase-file`, so the ISO embeds it. **Rules for a secret-bearing ISO:** + +- supervised / single-use only; **never distributed**; **delete after the run**. +- issue the passphrase with the shortest usable lifetime; rotate/retire after the install. +- the bootstrap `shred`s `bootstrap.env` on the box once host-install succeeds (reduces secret-at-rest). +- **slice C** removes this: the claim/pairing flow delivers the passphrase (or config) to the box + *after* the customer claims it, so the *distributed* ISO carries no secret. + +## The first-boot chain + +`stub-first-boot.sh` (exactly-once, `[first-boot]` from-iso/fully-up) is **dumb**: it lays down +`/usr/local/sbin/felhom-bootstrap.sh`, the systemd unit, and `/etc/felhom/bootstrap.env` (0600), then +enables + starts the unit. All fallible/network work lives in `felhom-bootstrap.service`, which +retries forever until `felhom-host-install.sh` exits 0, then writes `/etc/felhom/.bootstrap-done`, +disables itself, and scrubs the env. Retry-vs-resume: the first attempt is plain; any later attempt +that finds `/var/lib/felhom-install/state.json` adds `--resume` (safe per host-install v1.11.3 — its +producer steps re-run each pass). + +## N100 profile (for RUNBOOK B) + +The nested profile uses `disk-list=["sda"]`. **A real single-disk mini-PC should use a unique-serial +udev filter, not `disk-list`:** + +1. Boot once (or use a live env), harvest the target disk's udev id: + `udevadm info --query=property --name=/dev/sda | grep -E 'ID_SERIAL_SHORT|ID_MODEL|ID_WWN'`. +2. Write a profile: `filter.ID_SERIAL_SHORT = ""` + `filter-match = "all"`. A + match-nothing filter fails-safe (won't wipe the wrong disk). +3. **Prior-LVM prerequisite:** the auto-installer does **not** force-clear a pre-existing LVM PV + signature — a disk with a previous Linux/Proxmox install aborts at "cannot init physical volume" + (spike S2b). Wipe first: `blkdiscard -f /dev/` (or `wipefs -a` + zero the PV region). +4. UEFI + Secure Boot: no caveat — installs and boots under SB enforcing, no MOK/keypress (spike S2b). + +## Validation status (nested VM 310, 2026-07-16) + +PASS: build gate + red-proof (C); disk-filter fail-safe through the pipeline (D); stub→unit→**real +public-channel fetch**→correct host-install invocation→retry; 2.3 resume decision; exactly-once stub; +no-network retry + auto-recovery (B). **Operator-gated (not live-proven here):** host-install rc-0 +terminal success → hub-enrolled + guest-provisioned + done-flag + unit-disable — needs a properly +created drill customer (the hub create-UI is password-gated and a live-DB insert is unsafe). Also +out of scope: the physical N100 run (RUNBOOK B). diff --git a/scripts/iso/answer.toml.tmpl b/scripts/iso/answer.toml.tmpl new file mode 100644 index 0000000..5d1186e --- /dev/null +++ b/scripts/iso/answer.toml.tmpl @@ -0,0 +1,34 @@ +# scripts/iso/answer.toml.tmpl — Proxmox auto-install answer template for the Felhom ISO pipeline. +# +# build-felhom-iso.sh renders this into a concrete answer.toml by substituting: +# __FQDN__ <- profile's FELHOM_FQDN +# __ROOT_HASH__ <- a FRESH THROWAWAY crypt hash minted per build (never a real credential) +# __DISK_SETUP__ <- profile's FELHOM_DISK_SETUP block (disk-list OR a udev filter) +# The rendered answer is then run through validate-answer with the OUTPUT-PARSE gate (validate-answer +# returns exit 0 even on failure — spike S1 trap; the build parses the message text, never $?). +# +# This committed template carries ONLY the __ROOT_HASH__ placeholder, never a real hash. + +[global] +keyboard = "en-us" +country = "hu" +fqdn = "__FQDN__" +mailto = "noreply@felhom.eu" +timezone = "Europe/Budapest" +# THROWAWAY-PLACEHOLDER — build-felhom-iso.sh mints a fresh per-build crypt hash here. +root-password-hashed = "__ROOT_HASH__" +# Optional emergency/validation root SSH key (profile FELHOM_ROOT_SSH_KEY); blank -> line removed. +__ROOT_SSH_KEYS__ + +[network] +source = "from-dhcp" + +# Enable the baked first-boot stub (prepare-iso --on-first-boot). from-iso = no network / no +# cert-rotation risk (R-21 ruling); fully-up = the stub runs after pveproxy so pvesh/pct work and +# the retry unit it installs can drive host-install (spike S3/S8a). +[first-boot] +source = "from-iso" +ordering = "fully-up" + +# --- disk selection (from the build profile) --- +__DISK_SETUP__ diff --git a/scripts/iso/build-felhom-iso.sh b/scripts/iso/build-felhom-iso.sh new file mode 100644 index 0000000..4032762 --- /dev/null +++ b/scripts/iso/build-felhom-iso.sh @@ -0,0 +1,218 @@ +#!/bin/bash +#=============================================================================== +# build-felhom-iso.sh — R-21 slice A: turn the official PVE ISO into a Felhom auto-install ISO. +# +# Renders answer.toml (from answer.toml.tmpl + a profile), mints a fresh THROWAWAY root hash, +# gates the answer through validate-answer by PARSING ITS OUTPUT (never $? — validate-answer returns +# exit 0 even on failure, spike S1 trap), renders the first-boot stub (injecting the bootstrap +# script/unit/env), and runs prepare-iso --fetch-from iso --on-first-boot. Emits the ISO + sha256 + +# a build manifest. +# +# SECRET-BEARING: if the bootstrap-env carries a retrieval passphrase (it must, for an unattended +# install — see README "secret-bearing"), the produced ISO embeds it. Supervised/single-use only; +# never distributed; delete after the run. The build log says so loudly. +# +# Runs on DooPlex; delegates validate-answer + prepare-iso to the felhom-iso-assistant container. +#=============================================================================== +set -euo pipefail + +ISO_VERSION="1.16.0" # Felhom release the ISO is tagged to (aligns with felhom-host-install SCRIPT_VERSION). +IMAGE="${FELHOM_ISO_ASSISTANT_IMAGE:-felhom-iso-assistant:trixie}" +HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" + +# --- logging (host-install idiom) ----------------------------------------------------------------- +if [[ -t 1 ]]; then RED=$'\033[0;31m'; GREEN=$'\033[0;32m'; YELLOW=$'\033[1;33m'; BLUE=$'\033[0;34m'; CYAN=$'\033[0;36m'; NC=$'\033[0m' +else RED=""; GREEN=""; YELLOW=""; BLUE=""; CYAN=""; NC=""; fi +log_info() { echo -e "${GREEN}[INFO]${NC} $1"; } +log_warn() { echo -e "${YELLOW}[WARN]${NC} $1"; } +log_error() { echo -e "${RED}[ERROR]${NC} $1" >&2; } +log_step() { echo -e "${BLUE}[STEP]${NC} $1"; } +log_success() { echo -e "${GREEN}[OK]${NC} $1"; } +log_dry() { echo -e "${CYAN}[DRY-RUN]${NC} $1"; } +die() { log_error "$1"; exit 1; } + +PVE_ISO=""; ISO_SHA256=""; PROFILE=""; BOOTSTRAP_ENV=""; OUT_DIR="${HOME}/felhom-iso/out"; PVE_VERSION=""; DRY_RUN=false +usage() { + cat </dev/null || die "docker not found (needed for the assistant container)" +docker image inspect "$IMAGE" >/dev/null 2>&1 || die "assistant image '$IMAGE' not found — build it: docker build -f $HERE/Dockerfile.assistant -t $IMAGE $HERE" + +# --- verify source ISO ---------------------------------------------------------------------------- +log_step "verifying source ISO sha256" +actual_sha=$(sha256sum "$PVE_ISO" | awk '{print $1}') +[[ "$actual_sha" == "$ISO_SHA256" ]] || die "ISO sha256 MISMATCH: expected $ISO_SHA256, got $actual_sha" +log_success "source ISO sha256 OK ($actual_sha)" + +if [[ -z "$PVE_VERSION" ]]; then + PVE_VERSION=$(basename "$PVE_ISO" | sed -E 's/^proxmox-ve_(.+)\.iso$/\1/') + [[ "$PVE_VERSION" != "$(basename "$PVE_ISO")" ]] || die "cannot parse PVE version from '$(basename "$PVE_ISO")' — pass --pve-version" +fi +PROFILE_NAME="$(basename "$PROFILE")"; PROFILE_NAME="${PROFILE_NAME%.profile}" + +# --- load + validate profile ---------------------------------------------------------------------- +log_step "loading profile: $PROFILE" +FELHOM_FQDN=""; FELHOM_DISK_SETUP=""; FELHOM_ROOT_SSH_KEY="" +# shellcheck disable=SC1090 +source "$PROFILE" +[[ -n "$FELHOM_FQDN" ]] || die "profile missing FELHOM_FQDN" +[[ -n "$FELHOM_DISK_SETUP" ]] || die "profile missing FELHOM_DISK_SETUP" +# Optional emergency/validation SSH key baked into the installed root account. +ROOT_SSH_LINE="" +[[ -n "$FELHOM_ROOT_SSH_KEY" ]] && ROOT_SSH_LINE="root-ssh-keys = [\"${FELHOM_ROOT_SSH_KEY}\"]" + +# --- validate bootstrap-env (secret-bearing detection) -------------------------------------------- +log_step "checking bootstrap-env (secret-bearing detection)" +( set +e + FELHOM_CUSTOMER_ID=""; FELHOM_MODE=""; FELHOM_RETRIEVAL_PASSPHRASE="" + # shellcheck disable=SC1090 + source "$BOOTSTRAP_ENV" + [[ -n "$FELHOM_CUSTOMER_ID" ]] || { echo "MISSING FELHOM_CUSTOMER_ID"; exit 3; } + [[ -n "$FELHOM_MODE" ]] || { echo "MISSING FELHOM_MODE"; exit 3; } + [[ -n "$FELHOM_RETRIEVAL_PASSPHRASE" ]] || { echo "MISSING FELHOM_RETRIEVAL_PASSPHRASE"; exit 3; } +) || die "bootstrap-env invalid ($BOOTSTRAP_ENV) — must define FELHOM_CUSTOMER_ID, FELHOM_MODE, FELHOM_RETRIEVAL_PASSPHRASE" +SECRET_BEARING="yes" # a valid bootstrap-env always carries the retrieval passphrase +log_warn "this ISO will be SECRET-BEARING (embeds the customer retrieval passphrase) — supervised/single-use only" + +# --- workspace ------------------------------------------------------------------------------------ +WORK="$(mktemp -d "${TMPDIR:-/tmp}/felhom-iso.XXXXXX")" +cleanup() { rm -rf "$WORK"; } +trap cleanup EXIT +mkdir -p "$OUT_DIR" "$WORK/tmp" +ISO_DIR="$(cd "$(dirname "$PVE_ISO")" && pwd)"; ISO_BASE="$(basename "$PVE_ISO")" + +# --- mint fresh THROWAWAY root hash --------------------------------------------------------------- +log_step "minting fresh throwaway root password hash" +ROOT_PLAIN="felhom-throwaway-$(head -c12 /dev/urandom | base64 | tr -dc 'A-Za-z0-9')" +ROOT_HASH="$(openssl passwd -6 "$ROOT_PLAIN")" +unset ROOT_PLAIN +[[ -n "$ROOT_HASH" ]] || die "failed to mint root hash" +log_info "throwaway root hash written to the answer file (value stored out-of-band, not logged)" + +# --- render answer.toml (pure bash param-expansion; no metachar hazards) --------------------------- +log_step "rendering answer.toml" +ANSWER="$WORK/answer.toml" +: > "$ANSWER" +while IFS= read -r line || [[ -n "$line" ]]; do + if [[ "$line" == "__DISK_SETUP__" ]]; then + printf '%s\n' "$FELHOM_DISK_SETUP" >> "$ANSWER" + elif [[ "$line" == "__ROOT_SSH_KEYS__" ]]; then + [[ -n "$ROOT_SSH_LINE" ]] && printf '%s\n' "$ROOT_SSH_LINE" >> "$ANSWER" # blank -> omit line + else + line="${line//__FQDN__/$FELHOM_FQDN}" + line="${line//__ROOT_HASH__/$ROOT_HASH}" + printf '%s\n' "$line" >> "$ANSWER" + fi +done < "$HERE/answer.toml.tmpl" + +# --- validate-answer OUTPUT-PARSE gate (never $? — spike S1) -------------------------------------- +gate_validate_answer() { + local out + out=$(docker run --rm -v "$WORK":/work "$IMAGE" \ + proxmox-auto-install-assistant validate-answer /work/answer.toml 2>&1) || true + echo "----- validate-answer output -----"; echo "$out"; echo "----------------------------------" + # LOAD-BEARING: validate-answer exits 0 even on failure; decide on the MESSAGE TEXT, not $?. + if echo "$out" | grep -q "parsed successfully" && ! echo "$out" | grep -qi "Found issues"; then + return 0 + fi + return 1 +} +log_step "validating rendered answer (output-parse gate)" +if $DRY_RUN; then + log_dry "docker run … validate-answer /work/answer.toml (output-parse gate)" +else + gate_validate_answer || die "answer validation FAILED — NO ISO produced (fix the answer/profile)" + log_success "answer validated" +fi + +# --- render the first-boot stub (inject bootstrap script/unit/env as base64) ---------------------- +log_step "rendering first-boot stub" +STUB="$WORK/stub-first-boot.sh" +sh_b64="$(base64 -w0 < "$HERE/felhom-bootstrap.sh")" +unit_b64="$(base64 -w0 < "$HERE/felhom-bootstrap.service")" +env_b64="$(base64 -w0 < "$BOOTSTRAP_ENV")" +awk -v sh="$sh_b64" -v unit="$unit_b64" -v env="$env_b64" ' + { gsub(/@@BOOTSTRAP_SH_B64@@/, sh); gsub(/@@BOOTSTRAP_UNIT_B64@@/, unit); gsub(/@@BOOTSTRAP_ENV_B64@@/, env); print } +' "$HERE/stub-first-boot.sh" > "$STUB" +chmod 0755 "$STUB" +grep -q '@@BOOTSTRAP_.*_B64@@' "$STUB" && die "stub still has unfilled markers — injection failed" + +# --- prepare-iso ---------------------------------------------------------------------------------- +OUT_ISO="$OUT_DIR/felhom-pve-${PVE_VERSION}-v${ISO_VERSION}-${PROFILE_NAME}.iso" +log_step "building ISO: $(basename "$OUT_ISO")" +if $DRY_RUN; then + log_dry "docker run … prepare-iso /iso/$ISO_BASE --fetch-from iso --answer-file /work/answer.toml --on-first-boot /work/stub-first-boot.sh --output /work/out.iso" + log_info "DRY-RUN: no ISO produced" + exit 0 +fi +docker run --rm -v "$ISO_DIR":/iso:ro -v "$WORK":/work "$IMAGE" \ + proxmox-auto-install-assistant prepare-iso "/iso/$ISO_BASE" \ + --fetch-from iso --answer-file /work/answer.toml \ + --on-first-boot /work/stub-first-boot.sh \ + --tmp /work/tmp --output /work/out.iso +[[ -f "$WORK/out.iso" ]] || die "prepare-iso produced no output" +cp "$WORK/out.iso" "$OUT_ISO" + +# --- sha256 + manifest ---------------------------------------------------------------------------- +OUT_SHA="$(sha256sum "$OUT_ISO" | awk '{print $1}')" +OUT_SIZE="$(stat -c '%s' "$OUT_ISO")" +ASSISTANT_VER="$(docker run --rm "$IMAGE" proxmox-auto-install-assistant --version 2>&1 | head -1)" +echo "$OUT_SHA $(basename "$OUT_ISO")" > "$OUT_ISO.sha256" +cat > "$OUT_ISO.manifest.txt" </dev/null || echo 'https://felhom.eu/scripts/felhom-host-install.sh (default)') +secret-bearing : ${SECRET_BEARING} (embeds the customer retrieval passphrase — supervised/single-use, delete after the run) +output : $(basename "$OUT_ISO") +output-sha256 : ${OUT_SHA} +output-size-bytes : ${OUT_SIZE} +EOF + +log_success "ISO built: $OUT_ISO" +log_info "sha256 : $OUT_SHA" +log_info "size : $OUT_SIZE bytes" +log_info "manifest : $OUT_ISO.manifest.txt" +log_warn "SECRET-BEARING ISO (embeds the retrieval passphrase). Supervised/single-use; never distribute; delete after the run." diff --git a/scripts/iso/felhom-bootstrap.service b/scripts/iso/felhom-bootstrap.service new file mode 100644 index 0000000..6815628 --- /dev/null +++ b/scripts/iso/felhom-bootstrap.service @@ -0,0 +1,25 @@ +[Unit] +Description=Felhom host bootstrap (fetch + run felhom-host-install.sh unattended, retry until success) +Documentation=https://felhom.eu/documentation/backlog/ROADMAP.md +# Network-dependent work lives HERE (not in the first-boot stub) per spike S8a: the stub is +# exactly-once and network-independent; this unit retries forever until host-install exits 0. +After=network-online.target pve-cluster.service pveproxy.service +Wants=network-online.target +# Second exactly-once lock (the script also writes/checks this flag): once host-install succeeds, +# the flag exists and this unit becomes a no-op even if re-enabled. +ConditionPathExists=!/etc/felhom/.bootstrap-done +# Retry forever: no start-rate limit. +StartLimitIntervalSec=0 + +[Service] +Type=oneshot +ExecStart=/usr/local/sbin/felhom-bootstrap.sh +# Retry every 30s on any non-zero exit (fetch failed, host-install failed mid-run, no network yet). +Restart=on-failure +RestartSec=30 +# Journal-only logging (no secret ever printed; keys-not-values). +StandardOutput=journal +StandardError=journal + +[Install] +WantedBy=multi-user.target diff --git a/scripts/iso/felhom-bootstrap.sh b/scripts/iso/felhom-bootstrap.sh new file mode 100644 index 0000000..8f91484 --- /dev/null +++ b/scripts/iso/felhom-bootstrap.sh @@ -0,0 +1,98 @@ +#!/bin/bash +#=============================================================================== +# felhom-bootstrap.sh — invoked by felhom-bootstrap.service, retried until host-install succeeds. +# +# One attempt: read /etc/felhom/bootstrap.env -> fetch felhom-host-install.sh from the PUBLIC +# distribution channel (hub install-command Option-1 URL) -> run it unattended with the customer's +# retrieval passphrase -> on rc 0 write the done-flag + disable the unit; else exit non-zero so the +# unit retries. Journal-only logging; the passphrase is never echoed and lives only in a 0600 tmpfs +# file for the duration of one host-install invocation. +# +# Retry-vs-resume (source-verified, encoded ONCE): felhom-host-install.sh v1.11.3 makes --resume +# safe — its producer steps (token/enroll/grows) re-run every pass, so a resumed install repopulates +# hub.host_id/proxmox.token and never writes a crash-loop config. A plain re-invoke over an existing +# install state, by contrast, would re-hit the populated-host leaf guard / existing-vmid refusal. +# Therefore: FIRST attempt is plain; any later attempt that finds the install state file adds +# --resume. (--mode is required in both forms.) State file: /var/lib/felhom-install/state.json. +# +# NOT production-generic: this is the R-21 bare-metal first-boot bootstrap. It does NOT modify +# felhom-host-install.sh; it only invokes it. +#=============================================================================== +# Deliberately NOT `set -e`: we must capture host-install's exit code and exit on our own terms. +set -uo pipefail + +ENV_FILE=/etc/felhom/bootstrap.env +DONE_FLAG=/etc/felhom/.bootstrap-done +STATE_FILE=/var/lib/felhom-install/state.json +PASS_FILE=/run/felhom-bootstrap-pass +SCRIPT_TMP=/run/felhom-host-install.sh + +log() { echo "felhom-bootstrap: $*"; } + +cleanup_pass() { [[ -e "$PASS_FILE" ]] && { shred -u "$PASS_FILE" 2>/dev/null || rm -f "$PASS_FILE"; }; return 0; } +trap cleanup_pass EXIT + +# Belt-and-suspenders: the unit already has ConditionPathExists=!done, but guard here too. +if [[ -e "$DONE_FLAG" ]]; then + log "done-flag present ($DONE_FLAG) — nothing to do" + exit 0 +fi + +# --- env ------------------------------------------------------------------------------------------ +if [[ ! -r "$ENV_FILE" ]]; then + log "ERROR: $ENV_FILE missing or unreadable — cannot bootstrap (no guessed defaults)" + exit 1 +fi +# shellcheck disable=SC1090 +source "$ENV_FILE" + +for var in FELHOM_CUSTOMER_ID FELHOM_MODE FELHOM_RETRIEVAL_PASSPHRASE; do + if [[ -z "${!var:-}" ]]; then + log "ERROR: $var is unset/empty in $ENV_FILE — refusing to guess" + exit 1 + fi +done +HUB_URL="${FELHOM_HUB_URL:-https://hub.felhom.eu}" +INSTALL_URL="${FELHOM_INSTALL_URL:-https://felhom.eu/scripts/felhom-host-install.sh}" +EXTRA_ARGS="${FELHOM_EXTRA_ARGS:-}" + +# --- fetch host-install (public channel) ---------------------------------------------------------- +log "fetching host-install: $INSTALL_URL" +if ! curl -fsSL --max-time 60 "$INSTALL_URL" -o "$SCRIPT_TMP"; then + log "ERROR: host-install fetch failed (no network yet?) — unit will retry" + exit 1 +fi +if [[ ! -s "$SCRIPT_TMP" ]]; then + log "ERROR: fetched host-install is empty — unit will retry" + exit 1 +fi + +# --- retrieval passphrase -> 0600 tmpfs file ------------------------------------------------------ +( umask 077; printf '%s' "$FELHOM_RETRIEVAL_PASSPHRASE" > "$PASS_FILE" ) + +# --- retry-vs-resume ruling ----------------------------------------------------------------------- +args=(--customer-id "$FELHOM_CUSTOMER_ID" --mode "$FELHOM_MODE" --hub-url "$HUB_URL" --passphrase-file "$PASS_FILE") +if [[ -f "$STATE_FILE" ]]; then + log "prior install state present ($STATE_FILE) -> adding --resume (host-install v1.11.3: producers re-run, safe)" + args+=(--resume) +fi +# EXTRA_ARGS are profile-only flags (never secrets); intentional word-split. +read -ra extra <<< "$EXTRA_ARGS" + +log "running host-install (customer=${FELHOM_CUSTOMER_ID} mode=${FELHOM_MODE} hub=${HUB_URL})" +bash "$SCRIPT_TMP" "${args[@]}" "${extra[@]}" +rc=$? +cleanup_pass + +if [[ $rc -eq 0 ]]; then + log "host-install SUCCESS — writing done-flag, disabling unit, scrubbing env" + install -d -m 0755 "$(dirname "$DONE_FLAG")" + : > "$DONE_FLAG"; chmod 0644 "$DONE_FLAG" + systemctl disable felhom-bootstrap.service 2>/dev/null || true + # Reduce secret-at-rest: the box is enrolled; the passphrase is no longer needed. + shred -u "$ENV_FILE" 2>/dev/null || rm -f "$ENV_FILE" + exit 0 +fi + +log "host-install FAILED rc=${rc} — unit will retry in 30s" +exit "$rc" diff --git a/scripts/iso/profiles/nested-canary.profile b/scripts/iso/profiles/nested-canary.profile new file mode 100644 index 0000000..0b6fa01 --- /dev/null +++ b/scripts/iso/profiles/nested-canary.profile @@ -0,0 +1,12 @@ +# Felhom ISO build profile — nested-canary (Scenario D: disk-filter fail-safe regression). +# +# A udev filter that matches NOTHING. Proves the pipeline preserves the spike-proven S5c semantics: +# the installer aborts (exit 1, "filter did not match any device") and touches NO disk — verified by +# a host-side canary sha256 that stays byte-identical. NOT for real installs. + +FELHOM_FQDN="felhom-host.local" + +FELHOM_DISK_SETUP='[disk-setup] +filesystem = "ext4" +filter-match = "all" +filter.ID_SERIAL_SHORT = "no-such-disk-zzz"' diff --git a/scripts/iso/profiles/nested-vm.profile b/scripts/iso/profiles/nested-vm.profile new file mode 100644 index 0000000..b2bfba9 --- /dev/null +++ b/scripts/iso/profiles/nested-vm.profile @@ -0,0 +1,12 @@ +# Felhom ISO build profile — nested-vm (VM 310 validation; N100-agnostic). +# +# A profile is a sourceable shell fragment providing the answer's fqdn + [disk-setup] block. +# It carries NO secret. Scenario-A validation uses disk-list=["sda"] (the nested VM's single +# target disk). For the physical N100 (RUNBOOK B) use a unique-serial udev filter harvested from +# the pre-wipe udev dump instead — see README.md "N100 profile". + +FELHOM_FQDN="felhom-host.local" + +FELHOM_DISK_SETUP='[disk-setup] +filesystem = "ext4" +disk-list = ["sda"]' diff --git a/scripts/iso/stub-first-boot.sh b/scripts/iso/stub-first-boot.sh new file mode 100644 index 0000000..af06b92 --- /dev/null +++ b/scripts/iso/stub-first-boot.sh @@ -0,0 +1,46 @@ +#!/bin/bash +#=============================================================================== +# stub-first-boot.sh — the ONE executable the Proxmox auto-installer runs on first boot +# (--on-first-boot, [first-boot] source=from-iso, ordering=fully-up). This file is a SKELETON: +# build-felhom-iso.sh fills the three base64 markers below (bootstrap script, unit, env) and passes +# the RENDERED result to prepare-iso. The rendered stub is secret-bearing (the env carries the +# retrieval passphrase); the committed skeleton is not. +# +# DUMB BY DESIGN: the first-boot hook is exactly-once regardless of success (spike S3 x S8a), so this +# stub does NO network and NO fallible logic — it only lays down the retry unit + env and starts it. +# Everything that can fail lives in felhom-bootstrap.service, which retries forever. +#=============================================================================== +set -euo pipefail + +LOG=/var/log/felhom-first-boot.log +exec >>"$LOG" 2>&1 +echo "=== felhom stub-first-boot $(date -Is) uid=$(id -u) ===" + +install -d -m 0755 /etc/felhom /usr/local/sbin + +# --- bootstrap script (no secret; world-readable ok) ---------------------------------------------- +base64 -d > /usr/local/sbin/felhom-bootstrap.sh <<'__B64_BOOTSTRAP_SH__' +@@BOOTSTRAP_SH_B64@@ +__B64_BOOTSTRAP_SH__ +chmod 0755 /usr/local/sbin/felhom-bootstrap.sh + +# --- systemd retry unit --------------------------------------------------------------------------- +base64 -d > /etc/systemd/system/felhom-bootstrap.service <<'__B64_BOOTSTRAP_UNIT__' +@@BOOTSTRAP_UNIT_B64@@ +__B64_BOOTSTRAP_UNIT__ +chmod 0644 /etc/systemd/system/felhom-bootstrap.service + +# --- bootstrap env (SECRET-BEARING: retrieval passphrase) -> 0600 --------------------------------- +umask 077 +base64 -d > /etc/felhom/bootstrap.env <<'__B64_BOOTSTRAP_ENV__' +@@BOOTSTRAP_ENV_B64@@ +__B64_BOOTSTRAP_ENV__ +chmod 0600 /etc/felhom/bootstrap.env +umask 022 + +systemctl daemon-reload +systemctl enable felhom-bootstrap.service +systemctl start --no-block felhom-bootstrap.service + +echo "=== felhom stub-first-boot done — felhom-bootstrap enabled + started ===" +exit 0