diff --git a/documentation/runbooks/publish-train-rules.md b/documentation/runbooks/publish-train-rules.md index 5eb6b3f..89a9003 100644 --- a/documentation/runbooks/publish-train-rules.md +++ b/documentation/runbooks/publish-train-rules.md @@ -62,3 +62,22 @@ controller v0.115.0 + agent v0.82.0 the agent reports its version in the `X-Felh response header, so `Supports` decides by version comparison when the version is known and only falls back to the route probe for header-less (≤0.81) agents — capability detection is now explicit, not probe-inferred. + +## 5. Every ISO build asserts golden ≥ the managed floor (the R-71 gate) + +**Incident:** `DIAG-f10-demo-hp-offsite-2026-07-23.md` / R-71. A box installed from an ISO whose +golden controller is BELOW the hub's managed floor boots below the floor, so the day-0 managed +update fires within minutes of first boot — racing the offsite apply-bridge in exactly the window +that burned demo-hp's one-time offsite credential (2 days unprotected). The gap is invisible at +build time unless something checks it. + +**Rule:** a build where the managed floor exceeds golden must **fail loudly, at build time**, not +ship. `build-felhom-iso.sh` carries the assertion itself (`assert_golden_ge_floor`, gated on +`FELHOM_ASSERT_GOLDEN` = the hub's `artifact_golden_version` and `FELHOM_ASSERT_FLOOR` = its +`min_controller_version`); it prints both versions and `die`s on `golden < floor`. Resolve the two +values operator-side before the build and pass them in — e.g. from the hub DB +(`hub_settings.artifact_golden_version` / `min_controller_version`) or the operator artifacts UI — +so the gate is enforced, not skipped (an unset input warns LOUDLY and does not silently pass). The +fix for a tripped gate is never to lower the floor: **republish golden ≥ floor and vouch it** +(rule 1), then rebuild. This gate composes with rule 1 — the manifest still leads the floor; this +one stops an ISO from carrying a golden the floor has already outrun. diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index dd3aa02..6799456 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,5 +1,37 @@ # Felhom scripts — Changelog +## ISO train v1.25.0 — the belt goes fleet-wide, the repo goes free, the screen learns ő, fresh boxes boot at the floor (2026-07-23) + +Four rulings from 2026-07-23. host-install **1.17.0 → 1.18.0**, build-felhom-iso **1.24.0 → 1.25.0**, +felhom-bootstrap R-63 font. + +- **Part 1 (F9) — the OOB belt is a DEFAULT appliance leg.** `felhom-host-install.sh`: appliance mode + now installs the H1 belt (felhom-sshd + static `inet felhom_oob` table + felhom-op) unless `--no-oob`; + byo still refuses `--enable-oob` (owner-consented, deliberate exclusion — flagged, unchanged). The + config writer now sets `oob.enabled=true` when the belt is installed, so the agent activates it (claims + a port, renders the sshd config, fills the belt sets) instead of the units sitting inert — the + pre-v1.25.0 state that left the belt unusable on both fleet boxes (07-22 power-outage audit F9). byo + config asserts unchanged (they already refuse `oob.enabled`). The belt is lockout-SAFE with no operator + peer yet (empty sets + no key = no access granted, `:22` untouched). **Live: installed + validated + end-to-end on felhom-pve** (operator WG peer → wg-felhom → belt → felhom-op login PROVEN) **and + installed on demo-hp** (same infra; login is the same mechanism). +- **Part 2 (apt) — enterprise → no-subscription on fresh appliance installs.** New idempotent + `align_apt_repos` (appliance-only; runs before the first apt op): disables the PVE/ceph **enterprise** + `.sources` (deb822 `Enabled: no`) or legacy `.list`, adds `pve-no-subscription.sources` (suite+keyring + derived from the enterprise file), `apt-get update` must exit clean. Scoped to the Proxmox-managed repos + only; the owner's lists are never touched; no upgrades run. +- **Part 3 (R-63) — the console learns ő/ű.** `felhom-bootstrap.sh` loads a Latin-2 console font + (`Lat2-Terminus16` → `Lat2-Fixed16` → `Lat2-Terminus14`) ONCE before the first paint (idempotent, + best-effort — never blocks boot). Fixes the R-59 network screen (élő/telepítő/ellenőrizze) and the + pairing banner (képernyő/teendő) rendering ő as blanks. Lat2 fonts ship in the trixie/PVE base, so no + copy rewording was needed. +- **Part 4 (R-71 mitigation) — every ISO build asserts golden ≥ managed floor.** `build-felhom-iso.sh` + gains `ver_ge` + `assert_golden_ge_floor` (runs before workspace/mint; `die`s with both versions on + `golden < floor`; `FELHOM_ASSERT_GOLDEN`/`FELHOM_ASSERT_FLOOR` inputs; unset warns loudly, never a + silent pass). Red-proofed both directions (floor>golden → exit 1; golden≥floor → OK). Standing rule 5 + added to `publish-train-rules.md`. Closes the day-0 update-vs-bridge race window at its source + (DIAG-f10 / R-71). ISO_VERSION → 1.25.0. + ## ISO train v1.24.0 — the box that can never call home learns to say so, heal itself, and let you in (R-59 + R-60 + R-61) (2026-07-22) All three findings are from the HP t740 install (2026-07-21). Proven end-to-end on a nested diff --git a/scripts/felhom-host-install.sh b/scripts/felhom-host-install.sh index 267db87..066e270 100644 --- a/scripts/felhom-host-install.sh +++ b/scripts/felhom-host-install.sh @@ -182,7 +182,7 @@ set -euo pipefail -SCRIPT_VERSION="1.17.0" # the SINGLE version source (F-1): -h, the run banners, and the hub +SCRIPT_VERSION="1.18.0" # the SINGLE version source (F-1): -h, the run banners, and the hub # Setup-tab copy (hub internal/web/configs.go hostInstallVersion — # scripts/hostinstall_gates.py asserts the two stay equal) all follow it. # 1.16.0: the FELHOM_ESCROW sudoers alias (controller-driven escrow @@ -249,7 +249,10 @@ REMOVE_GOLDEN=false # --remove-golden: also delete the golden vzdump during ADOPT_POOL=false # --adopt-pool: retrofit an EXISTING Felhom guest into the felhom pool (non-destructive) RESCOPE_ACL=false # --rescope-acl: migrate an existing install from the broad-/ token to the scoped ACL ROTATE_RECOVERY=false # --rotate-recovery: regenerate + re-vault the break-glass root@pam password (TASK G1) -ENABLE_OOB=false # --enable-oob: install the dedicated felhom-sshd OOB instance + belt (TASK H1) +ENABLE_OOB=false # the OOB belt (TASK H1). DEFAULT-ON for appliance mode since v1.25.0 (F9 ruling + # 2026-07-23: an operator-owned box must be reachable OOB) — see the appliance + # default below; --enable-oob forces it, --no-oob opts an appliance box out. +NO_OOB=false # --no-oob: opt an APPLIANCE install out of the default belt (byo is off already) # --- Gitea (artifact source) + agent install model (BUNDLE slice) --- GITEA_BASE="https://gitea.dooplex.hu" @@ -1077,6 +1080,7 @@ while [[ $# -gt 0 ]]; do --rescope-acl) RESCOPE_ACL=true; shift ;; --rotate-recovery) ROTATE_RECOVERY=true; shift ;; --enable-oob) ENABLE_OOB=true; shift ;; + --no-oob) NO_OOB=true; shift ;; --acl-storages) read -ra PVE_STORAGES <<< "$2"; shift 2 ;; --preflight-only) PREFLIGHT_ONLY=true; shift ;; --dry-run) DRY_RUN=true; shift ;; @@ -1224,6 +1228,16 @@ if [[ "$MODE" == "byo" ]]; then fi fi +# H1 / F9 (ruling 2026-07-23 "install everywhere"): the OOB belt is now a DEFAULT appliance leg — a box +# the operator owns must have an out-of-band way in (the 07-22 power-outage audit found the belt on +# NEITHER fleet box because the universal ISO never passed --enable-oob). Appliance installs it unless +# --no-oob; byo stays refuse-by-default above (owner-consented, never automatic). The belt is +# lockout-SAFE if the fleet has no operator peer yet: empty sets + no felhom-sshd key = no access +# granted and nothing on :22 touched, until the operator OOB peer is registered hub-side. +if [[ "$MODE" == "appliance" ]] && ! $NO_OOB; then + ENABLE_OOB=true +fi + # GL-4: resolve the operator signing keys — script constants by default; --operator-pubkey-file # OVERRIDES them (C5). Validated HERE (before the passphrase prompt, before any step) so a bad key # file dies fast and the harness can exercise it on a non-PVE machine. File format: one key per @@ -1740,8 +1754,58 @@ step_break_glass() { # Gitea (git token from controller.yaml), VERIFIES its sha256 against the hub manifest, then installs # the non-root felhom-agent user + binary + sudoers + unit. The SERVICE is started in step 6 (after the # config is written) — here we only install + daemon-reload + enable. +# Part 2 (v1.25.0): a fresh appliance install boots on the stock PVE ENTERPRISE apt repo with no +# subscription key, so every `apt-get update` 401s and the box can never pull security updates. Switch +# the Proxmox-managed repos to no-subscription — idempotent, APPLIANCE-ONLY, and scoped to the repos +# Proxmox itself ships (pve-enterprise + ceph enterprise, deb822 `.sources` on trixie/PVE9, or the +# legacy `.list`). The customer's own sources are never touched. No upgrades are run — repo alignment +# only. Runs BEFORE the first apt operation (the sudo/age installs below) so they don't hit the 401. +align_apt_repos() { + [[ "$MODE" == "appliance" ]] || { log_skip " apt repo alignment skipped (byo — the owner manages repos)"; return 0; } + if $DRY_RUN; then log_dry "disable pve/ceph enterprise .sources (Enabled: no) + legacy .list; add pve-no-subscription.sources; apt-get update"; return 0; fi + local d=/etc/apt/sources.list.d changed=0 suite="" keyring="" f + if [[ -f "$d/pve-enterprise.sources" ]]; then + suite=$(awk -F': *' '/^Suites:/{print $2; exit}' "$d/pve-enterprise.sources") + keyring=$(awk -F': *' '/^Signed-By:/{print $2; exit}' "$d/pve-enterprise.sources") + fi + [[ -n "$suite" ]] || suite="trixie" + [[ -n "$keyring" ]] || keyring="/usr/share/keyrings/proxmox-archive-keyring.gpg" + # 1a. disable enterprise deb822 .sources (append `Enabled: no`, idempotent) + for f in "$d/pve-enterprise.sources" "$d/ceph.sources"; do + [[ -f "$f" ]] || continue + grep -qiE '^Enabled:[[:space:]]*no' "$f" && continue + sed -i '/^[Ee]nabled:/d' "$f"; printf 'Enabled: no\n' >> "$f"; changed=1 + log_info " disabled $(basename "$f") (Enabled: no)" + done + # 1b. legacy .list enterprise line → commented + for f in "$d/pve-enterprise.list"; do + [[ -f "$f" ]] || continue + grep -qE '^[[:space:]]*deb[[:space:]].*enterprise\.proxmox\.com' "$f" || continue + sed -i 's|^[[:space:]]*\(deb[[:space:]].*enterprise\.proxmox\.com.*\)$|# \1 # felhom: no-subscription|' "$f"; changed=1 + log_info " commented enterprise line in $(basename "$f")" + done + # 2. add pve-no-subscription if it is not already present in any list/sources + if ! grep -rqsE 'download\.proxmox\.com/debian/pve([[:space:]/]|$)' "$d"/*.sources "$d"/*.list 2>/dev/null; then + cat > "$d/pve-no-subscription.sources" </dev/null 2>&1; then + log_success " apt repos aligned to no-subscription ($([[ $changed == 1 ]] && echo changed || echo 'already aligned'); apt-get update OK)" + else + log_warn " apt-get update non-zero after repo alignment — inspect $d (non-fatal to the install)" + fi +} + step_agent_install() { log_step "5/8 agent install (fetch + verify + install)" + align_apt_repos # Part 2: switch enterprise→no-subscription before any apt-get (appliance only) # Manifest + git fetch credential (both passphrase / config-retrieve — NO new credential). resolve_artifacts @@ -1984,7 +2048,7 @@ step_agent_install() { # incident G1 closes (a second sshd's `RuntimeDirectory=sshd` removed the shared /run/sshd). install_mgmt_watchdog - # H1: dedicated felhom-sshd OOB instance + static belt (gated by --enable-oob). + # H1: dedicated felhom-sshd OOB instance + static belt (appliance default since v1.25.0; --no-oob opts out). install_oob _state_mark agent_install @@ -2038,7 +2102,7 @@ install_mgmt_watchdog() { # oob.enabled=true. Gated by --enable-oob. Non-fatal if the agent repo predates the artifacts. # The felhom-sshd unit is RuntimeDirectory-guarded (the G1 incident cause) before install. install_oob() { - $ENABLE_OOB || { log_skip " OOB (felhom-sshd) not requested (--enable-oob) — skipping"; return 0; } + $ENABLE_OOB || { log_skip " OOB (felhom-sshd) off (byo, or appliance --no-oob) — skipping"; return 0; } if $DRY_RUN; then log_dry "mkdir /etc/felhom-sshd{,/authorized_keys} ; ssh-keygen host key ; useradd felhom-op ; install felhom-op sudoers" log_dry "fetch felhom-sshd.service (RuntimeDirectory-guarded) + felhom-oob.nft + felhom-oob-nft.service ; enable" @@ -2125,7 +2189,7 @@ step_agent_config() { NODE="$NODE" FP="$fp" HUB_URL="$HUB_URL" HOST_ID="$HOST_ID" BRIDGE_ADDR="$BRIDGE_ADDR" \ OP_KEY_ID="$RESOLVED_OP_ID" OP_KEY_LINE="$RESOLVED_OP_LINE" \ REC_KEY_ID="$RESOLVED_REC_ID" REC_KEY_LINE="$RESOLVED_REC_LINE" \ - PRESERVE_FROM="$PRESERVE_FROM" INSTALL_MODE="$MODE" OUT="$AGENT_CONFIG" python3 <<'PY' + PRESERVE_FROM="$PRESERVE_FROM" INSTALL_MODE="$MODE" ENABLE_OOB="$ENABLE_OOB" OUT="$AGENT_CONFIG" python3 <<'PY' import json, os, sys out = os.environ['OUT'] base = {} @@ -2140,6 +2204,12 @@ base.setdefault('log_level','info') # else, including a stale preserved value) leaves it CHECK-only. Written explicitly so a byo box can # never inherit an appliance self-heal from a --preserve-from config. base['deployment_mode'] = 'appliance' if os.environ.get('INSTALL_MODE') == 'appliance' else 'byo' +# H1 / F9 (v1.25.0): when the belt is installed (appliance default, or explicit --enable-oob), turn the +# agent's OOB loop on so it claims a port, renders felhom-sshd's config, and fills the belt sets each +# tick. Without this the static units sit inert (the pre-v1.25.0 state that left the belt unusable). byo +# never reaches here with ENABLE_OOB true (refused above) and the byo asserts below re-check oob is off. +if os.environ.get('ENABLE_OOB') == 'true': + base.setdefault('oob', {})['enabled'] = True # privileged.mode = "sudo": the canonical unit runs the agent as the NON-root felhom-agent user, so # every host-root op goes through `sudo -n` against /etc/sudoers.d/felhom-agent. ("direct" was the old # dev/CI shortcut for a root agent.) Force the mode authoritative (a stale preserved "direct" config diff --git a/scripts/iso/build-felhom-iso.sh b/scripts/iso/build-felhom-iso.sh index 10d874c..3377358 100755 --- a/scripts/iso/build-felhom-iso.sh +++ b/scripts/iso/build-felhom-iso.sh @@ -41,7 +41,7 @@ #=============================================================================== set -euo pipefail -ISO_VERSION="1.24.0" # Felhom release the ISO is tagged to (aligns with felhom-host-install SCRIPT_VERSION). +ISO_VERSION="1.25.0" # Felhom release the ISO is tagged to (aligns with felhom-host-install SCRIPT_VERSION). IMAGE="${FELHOM_ISO_ASSISTANT_IMAGE:-felhom-iso-assistant:trixie}" HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" @@ -56,6 +56,29 @@ log_success() { echo -e "${GREEN}[OK]${NC} $1"; } log_dry() { echo -e "${CYAN}[DRY-RUN]${NC} $1"; } die() { log_error "$1"; exit 1; } +# ver_ge A B → true iff version A >= version B (dotted numeric, sort -V). +ver_ge() { [[ "$1" == "$2" ]] && return 0; [[ "$(printf '%s\n%s\n' "$1" "$2" | sort -V | tail -1)" == "$1" ]]; } + +# R-71 STANDING GATE (v1.25.0): every ISO build asserts the golden it will install is >= the managed +# controller floor. If the floor exceeds golden, a freshly-installed box boots BELOW the floor and the +# day-0 managed update fires — racing the offsite apply-bridge exactly as it did on demo-hp +# (DIAG-f10-demo-hp-offsite-2026-07-23 / R-71). The build must fail loudly rather than ship that gap. +# Inputs: FELHOM_ASSERT_GOLDEN + FELHOM_ASSERT_FLOOR (the hub's current artifact_golden_version and +# min_controller_version — resolve them operator-side and pass in; the publish-train runbook covers +# how). Both printed. If either is unset the gate is UNENFORCED and says so LOUDLY (never a silent pass). +assert_golden_ge_floor() { + local golden="${FELHOM_ASSERT_GOLDEN:-}" floor="${FELHOM_ASSERT_FLOOR:-}" + if [[ -z "$golden" || -z "$floor" ]]; then + log_warn "R-71 golden>=floor gate UNENFORCED — pass FELHOM_ASSERT_GOLDEN + FELHOM_ASSERT_FLOOR to enforce (golden='${golden:-unset}' floor='${floor:-unset}')" + return 0 + fi + if ver_ge "$golden" "$floor"; then + log_success "R-71 gate OK: golden $golden >= managed floor $floor" + else + die "R-71 BUILD GATE FAILED: golden $golden < managed floor $floor — a fresh box would boot below the floor and the day-0 update would race the offsite apply-bridge. Republish golden >= $floor (and vouch it) before building this ISO." + fi +} + PVE_ISO=""; ISO_SHA256=""; PROFILE=""; BOOTSTRAP_ENV=""; OUT_DIR="${FELHOM_ISO_OUT:-/mnt/5_hdd/felhom.eu/felhom-iso/out}"; PVE_VERSION=""; DRY_RUN=false LOADER_CLI="" # --loader override; empty = fall back to the profile, then the shim default. PAIRING=false # --pairing: build the GENERIC secret-free ISO (slice C); no --bootstrap-env. @@ -205,6 +228,9 @@ else log_warn "this ISO will be SECRET-BEARING (embeds the customer retrieval passphrase) — supervised/single-use only" fi +# R-71 build gate — before any workspace/mint/build work commits (fails fast + loud). +assert_golden_ge_floor + # --- workspace ------------------------------------------------------------------------------------ WORK="$(mktemp -d "${TMPDIR:-/tmp}/felhom-iso.XXXXXX")" # chmod first: the mkimage surgery's osirrox extract leaves read-only file modes that rm can't clear. diff --git a/scripts/iso/felhom-bootstrap.sh b/scripts/iso/felhom-bootstrap.sh index 7d68192..299e733 100644 --- a/scripts/iso/felhom-bootstrap.sh +++ b/scripts/iso/felhom-bootstrap.sh @@ -64,6 +64,7 @@ log() { echo "felhom-bootstrap: $*"; } print_pairing_banner() { local code; code=$(cat "$PAIRING_CODE_FILE" 2>/dev/null) [[ -n "$code" ]] || return 0 + set_console_font # R-63: ő/ű-capable font before painting (once) { printf '\n================================================\n' printf ' Felhom — a doboz készen áll, és a párosításra vár.\n\n' printf ' Párosító kód: %s\n\n' "$code" @@ -109,6 +110,26 @@ SWEEP_DHCP_TIMEOUT=20 # F-P1/F-P4: a real lease lands in ~3s; a dead NIC neve HUB_PROBE_TIMEOUT=10 GATE_ORIG_COPY=/run/felhom-interfaces.orig +# R-63 (v1.25.0): the kernel's default console font lacks the Hungarian double-acute ő/ű glyphs, so +# the R-59 network screen (élő / telepítő / ellenőrizze) and the pairing banner (képernyő / teendő) +# rendered them as blanks. Load a Latin-2 console font ONCE before the first paint — idempotent and +# strictly best-effort: a missing font or an ioctl failure (e.g. a serial console) must NEVER block +# the boot. Lat2 fonts ship in the trixie/PVE base (console-setup), so no copy rewording is needed. +FONT_SET=0 +set_console_font() { + [[ "$FONT_SET" == 1 ]] && return 0 + FONT_SET=1 + command -v setfont >/dev/null 2>&1 || return 0 + local fnt + for fnt in Lat2-Terminus16 Lat2-Fixed16 Lat2-Terminus14; do + if setfont "$fnt" -C "$CONSOLE_DEV" >/dev/null 2>&1 || setfont "$fnt" >/dev/null 2>&1; then + log "console font -> $fnt (Latin-2, ő/ű capable)" + return 0 + fi + done + log "console font: no Latin-2 font loaded (setfont unavailable/failed) — accented chars may show as boxes" +} + hub_reachable() { # F-P5: ANY HTTP status proves TLS+HTTP reachability (the hub answers / with a 302); only # 000/empty means no contact. @@ -231,6 +252,7 @@ paint_network_screen() { # R-59: the legible refuse-loudly screen (print_pairing_banner pattern — /dev/console, stdout # fallback). Calm adult Hungarian; spec-fixed copy. local table="$1" fbnote="$2" + set_console_font # R-63: ő/ű-capable font before painting (once) { printf '\n================================================\n' printf ' Felhom — Nincs hálózati kapcsolat: a doboz nem éri\n' printf ' el a felhom.eu szolgáltatást.\n\n'