diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index 092aae4..539d151 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,5 +1,52 @@ # Felhom scripts — Changelog +## build-felhom-iso.sh v1.21.0 — the pairing wait stops looking like a failure (R-33) (2026-07-18) + +**Waiting is not failing, and must not be reported as failure.** Until now the PAIRING poll loop WAS +systemd's `Restart=on-failure`/`RestartSec=30`: each invocation did register-if-needed plus exactly +one poll, then exited **non-zero** so the unit would be restarted 30s later. Functionally fine — but +every one of those exits made systemd print `Failed to start Felhom host bootstrap …` on the physical +console **the customer is watching**. The 2026-07-18 N100 rehearsal measured **52 FAILED lines in +~11 minutes** while nothing whatsoever was wrong: the box had registered correctly and was waiting to +be bound, which is the normal state of a freshly installed appliance. That is the customer's literal +first impression of the product +(`documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md` finding F6, ROADMAP **R-33**). + +- **`felhom-bootstrap.sh` — the wait moved INSIDE the script.** `run_pairing()` is now a + `while true` loop that sleeps `POLL_INTERVAL` (30s — the hub-side rate is **unchanged**) between + polls, so the unit sits in `activating` instead of failing. Registration was split out into + `register_appliance()`, which returns non-zero for a *transient* problem (no network yet, identity + not gatherable, hub returned no token) and is simply retried by the loop rather than taking the + whole unit down. Three cadence constants at the top of the file: `POLL_INTERVAL=30`, + `BANNER_EVERY=10` (console banner every 5 min), `HEARTBEAT_EVERY=20` (journal heartbeat every + 10 min). +- **The journal got quiet without going dark.** A `204 not bound yet` is logged **once** on entry + (worded so nobody reads it as an error) and then only on the 10-minute heartbeat with elapsed + minutes. `404` and unexpected HTTP codes likewise degrade to the heartbeat instead of a line every + 30s. **`410` still exits non-zero on purpose** — the delivery was consumed but no local env exists, + a genuine crash window where handing the box back to systemd for a clean restart is the right move. +- **The console banner got calmer and less repetitive.** Printed every 5 min instead of every cycle + (a customer may walk up at any time, but a banner every 30s is its own noise). Dropped the + belt-and-braces `parositasra var / párosításra vár` double-spelling for a single accented line, and + added the reassurance the rehearsal showed was missing: „Ez a képernyő magától frissül — nincs + teendő a doboznál, és nyugodtan itt hagyhatod bekapcsolva." +- **`felhom-bootstrap.service` — `TimeoutStartSec=infinity` (load-bearing, not cosmetic).** A + `Type=oneshot` `ExecStart` is killed at `DefaultTimeoutStartSec` (90s). Without this line systemd + would kill the new in-script wait after 90 seconds and `Restart=on-failure` would **silently + reinstate the exact console spam this change removes** — a regression that would look like the + feature working for the first three polls. `Restart=on-failure`/`RestartSec=30` are deliberately + kept: they still cover the DIRECT path, a failed host-install, and the 410 crash window. +- **Behaviour verified, not assumed.** Driven in a throwaway Debian container against a stub hub that + answers `204` five times and then delivers: the script logged the wait **once** plus one heartbeat, + **never exited between polls**, then consumed the delivery, wrote the 0600 env, fell through to the + direct install in the same invocation and exited **0**. Under the old design the same sequence + produced five separate unit invocations and five `Failed to start` console lines. + +**Compatibility:** hub-side polling rate, endpoints, payloads and the one-shot delivery semantics are +all unchanged; this is purely how the box waits and what it says while waiting. An older ISO against +any hub keeps working exactly as before. + + ## build-felhom-iso.sh v1.20.0 — console pairing-code banner for customer self-bind (R-27 slice 1) (2026-07-17) Supports the hub's customer self-bind flow (hub v0.66.0). In PAIRING mode, `felhom-bootstrap.sh` now diff --git a/scripts/iso/README.md b/scripts/iso/README.md index 7905e43..83257f0 100644 --- a/scripts/iso/README.md +++ b/scripts/iso/README.md @@ -24,7 +24,7 @@ channel). | `profiles/*.profile` | disk-selection + fqdn fragments (sourceable, no secret) | | `stub-first-boot.sh` | the ONE first-boot executable (skeleton; build injects the bootstrap script/unit/env as base64) | | `felhom-bootstrap.sh` | per-attempt: fetch host-install from the public channel → run it with the retrieval passphrase → on rc 0 write done-flag + disable | -| `felhom-bootstrap.service` | retry-forever unit (`Type=oneshot`, `Restart=on-failure`, `RestartSec=30`, `StartLimitIntervalSec=0`) | +| `felhom-bootstrap.service` | retry-forever unit (`Type=oneshot`, `Restart=on-failure`, `RestartSec=30`, `StartLimitIntervalSec=0`, **`TimeoutStartSec=infinity`** — v1.21.0/R-33: the PAIRING wait loops inside the script, so the unit sits in `activating` instead of failing every 30s; without the lifted timeout systemd would kill it at 90s and restore the console spam) | ## Build diff --git a/scripts/iso/build-felhom-iso.sh b/scripts/iso/build-felhom-iso.sh index 7757694..3085626 100644 --- a/scripts/iso/build-felhom-iso.sh +++ b/scripts/iso/build-felhom-iso.sh @@ -32,7 +32,7 @@ #=============================================================================== set -euo pipefail -ISO_VERSION="1.20.0" # Felhom release the ISO is tagged to (aligns with felhom-host-install SCRIPT_VERSION). +ISO_VERSION="1.21.0" # Felhom release the ISO is tagged to (aligns with felhom-host-install SCRIPT_VERSION). IMAGE="${FELHOM_ISO_ASSISTANT_IMAGE:-felhom-iso-assistant:trixie}" HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" diff --git a/scripts/iso/felhom-bootstrap.service b/scripts/iso/felhom-bootstrap.service index 6815628..e428fa0 100644 --- a/scripts/iso/felhom-bootstrap.service +++ b/scripts/iso/felhom-bootstrap.service @@ -14,7 +14,14 @@ StartLimitIntervalSec=0 [Service] Type=oneshot ExecStart=/usr/local/sbin/felhom-bootstrap.sh -# Retry every 30s on any non-zero exit (fetch failed, host-install failed mid-run, no network yet). +# v1.21.0 (R-33): the PAIRING wait now loops INSIDE the script, so this unit sits in `activating` +# while a box waits to be bound instead of failing every 30s. That wait is unbounded by nature — +# it ends when a human binds the box — so the oneshot start timeout MUST be lifted. Without this, +# systemd kills ExecStart at DefaultTimeoutStartSec (90s) and Restart= silently reinstates exactly +# the `Failed to start` console spam this change exists to remove. +TimeoutStartSec=infinity +# Still retry on a genuine non-zero exit (fetch failed, host-install failed mid-run, crash-window +# 410). The pairing wait no longer exits non-zero just because nobody has bound the box yet. Restart=on-failure RestartSec=30 # Journal-only logging (no secret ever printed; keys-not-values). diff --git a/scripts/iso/felhom-bootstrap.sh b/scripts/iso/felhom-bootstrap.sh index 5783fae..c448fee 100644 --- a/scripts/iso/felhom-bootstrap.sh +++ b/scripts/iso/felhom-bootstrap.sh @@ -13,9 +13,15 @@ # them into the env (0600) and FALLS THROUGH to the DIRECT path — so every later retry is a # plain direct install (the delivery is one-shot; the box must not depend on re-fetching it). # -# The poll loop IS systemd's Restart=on-failure/RestartSec=30: each invocation does register-if-needed -# + exactly ONE poll, exiting non-zero (retry in 30s) until the bind delivers. This keeps every -# invocation short (no long-running-oneshot timeout) and reuses the existing retry machinery. +# The PAIRING wait polls INSIDE this script (v1.21.0, R-33). It used to be systemd's +# Restart=on-failure/RestartSec=30 — one poll per invocation, exiting non-zero until the bind landed. +# That worked, but every 30s systemd printed `Failed to start Felhom host bootstrap` on the physical +# console the CUSTOMER is watching: 52 FAILED lines in ~11 minutes during the 2026-07-18 rehearsal, +# while nothing was wrong (the box was correctly waiting to be bound). Waiting is not failing, so it +# must not be reported as failure. The unit now stays in `activating` and the loop sleeps between +# polls; the unit's Restart= machinery is kept for the DIRECT path and for genuine crashes. +# REQUIRES TimeoutStartSec=infinity in the unit — a Type=oneshot ExecStart is otherwise killed at +# DefaultTimeoutStartSec (90s), which would silently reintroduce the restart spam. # # Retry-vs-resume (source-verified, encoded ONCE): felhom-host-install.sh v1.11.3 makes --resume safe # — its producer steps re-run every pass. FIRST direct attempt is plain; any later attempt that finds @@ -35,6 +41,12 @@ SCRIPT_TMP=/run/felhom-host-install.sh TOKEN_FILE=/etc/felhom/appliance-token # PAIRING: the box's only pre-day-0 credential (0600, persists reboots) PAIRING_CODE_FILE=/etc/felhom/appliance-pairing-code # R-27: non-secret pairing code shown on the console +# PAIRING wait cadence (v1.21.0, R-33). POLL_INTERVAL keeps the hub-side rate identical to the old +# RestartSec=30, so nothing downstream changes; the other two only govern how often we SPEAK. +POLL_INTERVAL=30 # seconds between polls +BANNER_EVERY=10 # re-print the console banner every N cycles (10 x 30s = 5 min) +HEARTBEAT_EVERY=20 # journal heartbeat every N cycles (20 x 30s = 10 min) + log() { echo "felhom-bootstrap: $*"; } # print_pairing_banner (R-27, v0.66.0) — show the pairing code prominently on the physical console while @@ -45,10 +57,12 @@ print_pairing_banner() { local code; code=$(cat "$PAIRING_CODE_FILE" 2>/dev/null) [[ -n "$code" ]] || return 0 { printf '\n================================================\n' - printf ' Felhom — a doboz parositasra var / párosításra vár\n\n' + printf ' Felhom — a doboz készen áll, és a párosításra vár.\n\n' printf ' Párosító kód: %s\n\n' "$code" - printf ' Nyisd meg az e-mailben kapott self-bind linket,\n' - printf ' és add meg ezt a kódot + a jelszavadat.\n' + printf ' Nyisd meg az e-mailben kapott linket, és add meg\n' + printf ' ezt a kódot és a jelszavadat.\n\n' + printf ' Ez a képernyő magától frissül — nincs teendő a\n' + printf ' doboznál, és nyugodtan itt hagyhatod bekapcsolva.\n' printf '================================================\n\n' } > /dev/console 2>/dev/null || printf 'Párosító kód: %s\n' "$code" } @@ -168,48 +182,66 @@ print(json.dumps({ PY } +register_appliance() { + # Register once and persist the token. Returns 0 on success (or if already registered), 1 on a + # transient failure the caller should simply retry — no network yet is the common case on a box + # that has only just booted, and it is not an error worth telling the customer about. + [[ -s "$TOKEN_FILE" ]] && return 0 + + local payload; payload=$(gather_identity_json) + if [[ -z "$payload" || "$payload" != *'"uuid"'* ]]; then + log "could not gather appliance identity yet — retrying" + return 1 + fi + log "registering unclaimed appliance at the hub" + local resp; resp=$(curl -fsS --max-time 30 -X POST \ + -H 'Content-Type: application/json' --data "$payload" \ + "$HUB_URL/api/v1/appliance/register" 2>/dev/null) + if [[ $? -ne 0 || -z "$resp" ]]; then + log "registration did not go through (no network yet?) — retrying" + return 1 + fi + local token; token=$(printf '%s' "$resp" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("appliance_token",""))' 2>/dev/null) + if [[ -z "$token" ]]; then + log "registration returned no appliance token — retrying" + return 1 + fi + ( umask 077; printf '%s' "$token" > "$TOKEN_FILE" ) + # R-27 (v0.66.0): persist the non-secret pairing code (absent on a pre-v0.66.0 hub — tolerated). + local pcode; pcode=$(printf '%s' "$resp" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("pairing_code",""))' 2>/dev/null) + if [[ -n "$pcode" ]]; then + printf '%s' "$pcode" > "$PAIRING_CODE_FILE" + fi + log "registered — appliance token stored (0600); waiting for the operator or a customer self-bind" + return 0 +} + run_pairing() { log "PAIRING mode (generic ISO, no baked customer/passphrase) — hub=$HUB_URL" - # 1. register once (persist the token). A present token means we already registered — go poll. - if [[ ! -s "$TOKEN_FILE" ]]; then - local payload; payload=$(gather_identity_json) - if [[ -z "$payload" || "$payload" != *'"uuid"'* ]]; then - log "ERROR: could not gather appliance identity — unit will retry" - exit 1 + # The wait lives HERE, not in systemd's restart loop (v1.21.0, R-33). Waiting to be bound is the + # NORMAL state of a freshly installed box and must look like it — on the console and in the + # journal alike. Only a genuine crash should ever surface as a unit failure. + local cycle=0 + while true; do + if ! register_appliance; then + sleep "$POLL_INTERVAL"; continue fi - log "registering unclaimed appliance at the hub" - local resp; resp=$(curl -fsS --max-time 30 -X POST \ - -H 'Content-Type: application/json' --data "$payload" \ - "$HUB_URL/api/v1/appliance/register" 2>/dev/null) - if [[ $? -ne 0 || -z "$resp" ]]; then - log "ERROR: registration failed (no network yet?) — unit will retry" - exit 1 - fi - local token; token=$(printf '%s' "$resp" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("appliance_token",""))' 2>/dev/null) - if [[ -z "$token" ]]; then - log "ERROR: registration returned no appliance token — unit will retry" - exit 1 - fi - ( umask 077; printf '%s' "$token" > "$TOKEN_FILE" ) - # R-27 (v0.66.0): persist the non-secret pairing code (absent on a pre-v0.66.0 hub — tolerated). - local pcode; pcode=$(printf '%s' "$resp" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("pairing_code",""))' 2>/dev/null) - if [[ -n "$pcode" ]]; then - printf '%s' "$pcode" > "$PAIRING_CODE_FILE" - fi - log "registered — appliance token stored (0600); waiting for the operator or a customer self-bind" - fi - print_pairing_banner # show the code on the console each pairing cycle - # 2. ONE poll. RestartSec=30 is the poll interval. - local token; token=$(cat "$TOKEN_FILE") - local body code - body=$(curl -sS --max-time 30 -o - -w '\n%{http_code}' \ - -H "Authorization: Bearer $token" "$HUB_URL/api/v1/appliance/poll" 2>/dev/null) - code="${body##*$'\n'}" - body="${body%$'\n'*}" + # Re-show the code periodically, not every cycle: the customer may walk up at any time, but a + # banner every 30s is its own kind of noise. + if (( cycle % BANNER_EVERY == 0 )); then + print_pairing_banner + fi - case "$code" in + local token; token=$(cat "$TOKEN_FILE") + local body code + body=$(curl -sS --max-time 30 -o - -w '\n%{http_code}' \ + -H "Authorization: Bearer $token" "$HUB_URL/api/v1/appliance/poll" 2>/dev/null) + code="${body##*$'\n'}" + body="${body%$'\n'*}" + + case "$code" in 200) log "bind DELIVERED — writing credentials to the env and switching to direct install" # Parse the one-shot delivery into shell-safe env assignments (never echo the passphrase). @@ -242,22 +274,36 @@ emit("FELHOM_EXTRA_ARGS", d.get("extra_args")) run_direct ;; # run_direct exits 204) - log "not bound yet — will poll again in 30s" - exit 1 + # The expected state for as long as nobody has bound the box. Log it once, then only on + # a slow heartbeat — an operator reading the journal still sees liveness without the + # every-30s wall of text that made the real signal hard to find. + if (( cycle == 0 )); then + log "not bound yet — polling every ${POLL_INTERVAL}s until the operator or a customer self-bind lands (this is the normal waiting state, not an error)" + elif (( cycle % HEARTBEAT_EVERY == 0 )); then + log "still waiting to be bound ($(( cycle * POLL_INTERVAL / 60 ))m elapsed; polling continues)" + fi ;; 410) - log "ERROR: delivery already consumed but no local env — unit will retry (rare crash-window)" + # The delivery was consumed but we hold no env — a rare crash window. Exiting hands the + # box back to systemd, whose restart re-runs us from a clean slate. + log "ERROR: delivery already consumed but no local env — exiting so the unit restarts (rare crash-window)" exit 1 ;; 404) - log "appliance token not recognized (discarded, or the hub has no record) — will retry in 30s" - exit 1 + if (( cycle % HEARTBEAT_EVERY == 0 )); then + log "appliance token not recognized (discarded, or the hub has no record) — still retrying" + fi ;; *) - log "poll returned HTTP ${code:-none} — will retry in 30s" - exit 1 + if (( cycle % HEARTBEAT_EVERY == 0 )); then + log "poll returned HTTP ${code:-none} — still retrying" + fi ;; - esac + esac + + sleep "$POLL_INTERVAL" + cycle=$(( cycle + 1 )) + done } # --- mode selection -------------------------------------------------------------------------------