scripts v1.21.0 — the pairing wait stops looking like a failure (R-33)

Waiting to be bound is the NORMAL state of a freshly installed box, and it must
not be reported as failure. The PAIRING poll loop used to BE systemd's
Restart=on-failure/RestartSec=30 — one poll per invocation, exiting non-zero
until the bind landed — so every 30s systemd printed "Failed to start Felhom
host bootstrap" on the physical console the CUSTOMER is watching. The
2026-07-18 N100 rehearsal measured 52 FAILED lines in ~11 minutes while nothing
was wrong (VALIDATION-n100-rehearsal-2026-07-18.md F6).

felhom-bootstrap.sh: run_pairing() is now a while-loop that sleeps
POLL_INTERVAL (30s — the hub-side rate is unchanged) between polls, so the unit
sits in `activating`. Registration split into register_appliance(), which
returns non-zero for a transient problem (no network yet, no identity, no
token) and is retried by the loop instead of taking the unit down. Cadence
constants: POLL_INTERVAL=30, BANNER_EVERY=10 (5 min), HEARTBEAT_EVERY=20
(10 min).

Quiet without going dark: a 204 is logged once on entry (worded so nobody reads
it as an error) and then only on the 10-minute heartbeat with elapsed minutes;
404 and unexpected codes degrade the same way. 410 STILL exits non-zero on
purpose — delivery consumed but no local env is a real crash window, and a
clean systemd restart is the right response.

Console banner: every 5 min instead of every cycle, single accented spelling
instead of the parositasra/párosításra double, and the reassurance the
rehearsal showed was missing ("Ez a képernyő magától frissül — nincs teendő a
doboznál").

felhom-bootstrap.service: TimeoutStartSec=infinity. This is load-bearing, not
cosmetic — a Type=oneshot ExecStart is killed at DefaultTimeoutStartSec (90s),
so without it systemd would kill the new in-script wait after 90 seconds and
Restart=on-failure would silently reinstate the exact spam this removes, after
appearing to work for the first three polls. Restart=/RestartSec= are kept
deliberately: they still cover the DIRECT path, a failed host-install, and 410.

Verified behaviourally, not assumed: driven in a throwaway Debian container
against a stub hub answering 204 five times then delivering — logged the wait
once plus one heartbeat, never exited between polls, then consumed the
delivery, wrote the 0600 env, fell through to the direct install in the same
invocation and exited 0. The old design produced five unit invocations and five
"Failed to start" console lines for that same sequence.

Hub endpoints, payloads, polling rate and one-shot delivery semantics are all
unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
This commit is contained in:
2026-07-18 20:39:42 +02:00
parent 00f26c4f06
commit bcdb04222a
5 changed files with 153 additions and 53 deletions
+1 -1
View File
@@ -24,7 +24,7 @@ channel).
| `profiles/*.profile` | disk-selection + fqdn fragments (sourceable, no secret) |
| `stub-first-boot.sh` | the ONE first-boot executable (skeleton; build injects the bootstrap script/unit/env as base64) |
| `felhom-bootstrap.sh` | per-attempt: fetch host-install from the public channel → run it with the retrieval passphrase → on rc 0 write done-flag + disable |
| `felhom-bootstrap.service` | retry-forever unit (`Type=oneshot`, `Restart=on-failure`, `RestartSec=30`, `StartLimitIntervalSec=0`) |
| `felhom-bootstrap.service` | retry-forever unit (`Type=oneshot`, `Restart=on-failure`, `RestartSec=30`, `StartLimitIntervalSec=0`, **`TimeoutStartSec=infinity`** — v1.21.0/R-33: the PAIRING wait loops inside the script, so the unit sits in `activating` instead of failing every 30s; without the lifted timeout systemd would kill it at 90s and restore the console spam) |
## Build
+1 -1
View File
@@ -32,7 +32,7 @@
#===============================================================================
set -euo pipefail
ISO_VERSION="1.20.0" # Felhom release the ISO is tagged to (aligns with felhom-host-install SCRIPT_VERSION).
ISO_VERSION="1.21.0" # Felhom release the ISO is tagged to (aligns with felhom-host-install SCRIPT_VERSION).
IMAGE="${FELHOM_ISO_ASSISTANT_IMAGE:-felhom-iso-assistant:trixie}"
HERE="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+8 -1
View File
@@ -14,7 +14,14 @@ StartLimitIntervalSec=0
[Service]
Type=oneshot
ExecStart=/usr/local/sbin/felhom-bootstrap.sh
# Retry every 30s on any non-zero exit (fetch failed, host-install failed mid-run, no network yet).
# v1.21.0 (R-33): the PAIRING wait now loops INSIDE the script, so this unit sits in `activating`
# while a box waits to be bound instead of failing every 30s. That wait is unbounded by nature —
# it ends when a human binds the box — so the oneshot start timeout MUST be lifted. Without this,
# systemd kills ExecStart at DefaultTimeoutStartSec (90s) and Restart= silently reinstates exactly
# the `Failed to start` console spam this change exists to remove.
TimeoutStartSec=infinity
# Still retry on a genuine non-zero exit (fetch failed, host-install failed mid-run, crash-window
# 410). The pairing wait no longer exits non-zero just because nobody has bound the box yet.
Restart=on-failure
RestartSec=30
# Journal-only logging (no secret ever printed; keys-not-values).
+96 -50
View File
@@ -13,9 +13,15 @@
# them into the env (0600) and FALLS THROUGH to the DIRECT path — so every later retry is a
# plain direct install (the delivery is one-shot; the box must not depend on re-fetching it).
#
# The poll loop IS systemd's Restart=on-failure/RestartSec=30: each invocation does register-if-needed
# + exactly ONE poll, exiting non-zero (retry in 30s) until the bind delivers. This keeps every
# invocation short (no long-running-oneshot timeout) and reuses the existing retry machinery.
# The PAIRING wait polls INSIDE this script (v1.21.0, R-33). It used to be systemd's
# Restart=on-failure/RestartSec=30 — one poll per invocation, exiting non-zero until the bind landed.
# That worked, but every 30s systemd printed `Failed to start Felhom host bootstrap` on the physical
# console the CUSTOMER is watching: 52 FAILED lines in ~11 minutes during the 2026-07-18 rehearsal,
# while nothing was wrong (the box was correctly waiting to be bound). Waiting is not failing, so it
# must not be reported as failure. The unit now stays in `activating` and the loop sleeps between
# polls; the unit's Restart= machinery is kept for the DIRECT path and for genuine crashes.
# REQUIRES TimeoutStartSec=infinity in the unit — a Type=oneshot ExecStart is otherwise killed at
# DefaultTimeoutStartSec (90s), which would silently reintroduce the restart spam.
#
# Retry-vs-resume (source-verified, encoded ONCE): felhom-host-install.sh v1.11.3 makes --resume safe
# — its producer steps re-run every pass. FIRST direct attempt is plain; any later attempt that finds
@@ -35,6 +41,12 @@ SCRIPT_TMP=/run/felhom-host-install.sh
TOKEN_FILE=/etc/felhom/appliance-token # PAIRING: the box's only pre-day-0 credential (0600, persists reboots)
PAIRING_CODE_FILE=/etc/felhom/appliance-pairing-code # R-27: non-secret pairing code shown on the console
# PAIRING wait cadence (v1.21.0, R-33). POLL_INTERVAL keeps the hub-side rate identical to the old
# RestartSec=30, so nothing downstream changes; the other two only govern how often we SPEAK.
POLL_INTERVAL=30 # seconds between polls
BANNER_EVERY=10 # re-print the console banner every N cycles (10 x 30s = 5 min)
HEARTBEAT_EVERY=20 # journal heartbeat every N cycles (20 x 30s = 10 min)
log() { echo "felhom-bootstrap: $*"; }
# print_pairing_banner (R-27, v0.66.0) — show the pairing code prominently on the physical console while
@@ -45,10 +57,12 @@ print_pairing_banner() {
local code; code=$(cat "$PAIRING_CODE_FILE" 2>/dev/null)
[[ -n "$code" ]] || return 0
{ printf '\n================================================\n'
printf ' Felhom — a doboz parositasra var / párosításra vár\n\n'
printf ' Felhom — a doboz készen áll, és a párosításra vár.\n\n'
printf ' Párosító kód: %s\n\n' "$code"
printf ' Nyisd meg az e-mailben kapott self-bind linket,\n'
printf ' és add meg ezt a kódot + a jelszavadat.\n'
printf ' Nyisd meg az e-mailben kapott linket, és add meg\n'
printf ' ezt a kódot és a jelszavadat.\n\n'
printf ' Ez a képernyő magától frissül — nincs teendő a\n'
printf ' doboznál, és nyugodtan itt hagyhatod bekapcsolva.\n'
printf '================================================\n\n'
} > /dev/console 2>/dev/null || printf 'Párosító kód: %s\n' "$code"
}
@@ -168,48 +182,66 @@ print(json.dumps({
PY
}
register_appliance() {
# Register once and persist the token. Returns 0 on success (or if already registered), 1 on a
# transient failure the caller should simply retry — no network yet is the common case on a box
# that has only just booted, and it is not an error worth telling the customer about.
[[ -s "$TOKEN_FILE" ]] && return 0
local payload; payload=$(gather_identity_json)
if [[ -z "$payload" || "$payload" != *'"uuid"'* ]]; then
log "could not gather appliance identity yet — retrying"
return 1
fi
log "registering unclaimed appliance at the hub"
local resp; resp=$(curl -fsS --max-time 30 -X POST \
-H 'Content-Type: application/json' --data "$payload" \
"$HUB_URL/api/v1/appliance/register" 2>/dev/null)
if [[ $? -ne 0 || -z "$resp" ]]; then
log "registration did not go through (no network yet?) — retrying"
return 1
fi
local token; token=$(printf '%s' "$resp" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("appliance_token",""))' 2>/dev/null)
if [[ -z "$token" ]]; then
log "registration returned no appliance token — retrying"
return 1
fi
( umask 077; printf '%s' "$token" > "$TOKEN_FILE" )
# R-27 (v0.66.0): persist the non-secret pairing code (absent on a pre-v0.66.0 hub — tolerated).
local pcode; pcode=$(printf '%s' "$resp" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("pairing_code",""))' 2>/dev/null)
if [[ -n "$pcode" ]]; then
printf '%s' "$pcode" > "$PAIRING_CODE_FILE"
fi
log "registered — appliance token stored (0600); waiting for the operator or a customer self-bind"
return 0
}
run_pairing() {
log "PAIRING mode (generic ISO, no baked customer/passphrase) — hub=$HUB_URL"
# 1. register once (persist the token). A present token means we already registered — go poll.
if [[ ! -s "$TOKEN_FILE" ]]; then
local payload; payload=$(gather_identity_json)
if [[ -z "$payload" || "$payload" != *'"uuid"'* ]]; then
log "ERROR: could not gather appliance identity — unit will retry"
exit 1
# The wait lives HERE, not in systemd's restart loop (v1.21.0, R-33). Waiting to be bound is the
# NORMAL state of a freshly installed box and must look like it — on the console and in the
# journal alike. Only a genuine crash should ever surface as a unit failure.
local cycle=0
while true; do
if ! register_appliance; then
sleep "$POLL_INTERVAL"; continue
fi
log "registering unclaimed appliance at the hub"
local resp; resp=$(curl -fsS --max-time 30 -X POST \
-H 'Content-Type: application/json' --data "$payload" \
"$HUB_URL/api/v1/appliance/register" 2>/dev/null)
if [[ $? -ne 0 || -z "$resp" ]]; then
log "ERROR: registration failed (no network yet?) — unit will retry"
exit 1
fi
local token; token=$(printf '%s' "$resp" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("appliance_token",""))' 2>/dev/null)
if [[ -z "$token" ]]; then
log "ERROR: registration returned no appliance token — unit will retry"
exit 1
fi
( umask 077; printf '%s' "$token" > "$TOKEN_FILE" )
# R-27 (v0.66.0): persist the non-secret pairing code (absent on a pre-v0.66.0 hub — tolerated).
local pcode; pcode=$(printf '%s' "$resp" | python3 -c 'import json,sys; print(json.load(sys.stdin).get("pairing_code",""))' 2>/dev/null)
if [[ -n "$pcode" ]]; then
printf '%s' "$pcode" > "$PAIRING_CODE_FILE"
fi
log "registered — appliance token stored (0600); waiting for the operator or a customer self-bind"
fi
print_pairing_banner # show the code on the console each pairing cycle
# 2. ONE poll. RestartSec=30 is the poll interval.
local token; token=$(cat "$TOKEN_FILE")
local body code
body=$(curl -sS --max-time 30 -o - -w '\n%{http_code}' \
-H "Authorization: Bearer $token" "$HUB_URL/api/v1/appliance/poll" 2>/dev/null)
code="${body##*$'\n'}"
body="${body%$'\n'*}"
# Re-show the code periodically, not every cycle: the customer may walk up at any time, but a
# banner every 30s is its own kind of noise.
if (( cycle % BANNER_EVERY == 0 )); then
print_pairing_banner
fi
case "$code" in
local token; token=$(cat "$TOKEN_FILE")
local body code
body=$(curl -sS --max-time 30 -o - -w '\n%{http_code}' \
-H "Authorization: Bearer $token" "$HUB_URL/api/v1/appliance/poll" 2>/dev/null)
code="${body##*$'\n'}"
body="${body%$'\n'*}"
case "$code" in
200)
log "bind DELIVERED — writing credentials to the env and switching to direct install"
# Parse the one-shot delivery into shell-safe env assignments (never echo the passphrase).
@@ -242,22 +274,36 @@ emit("FELHOM_EXTRA_ARGS", d.get("extra_args"))
run_direct
;; # run_direct exits
204)
log "not bound yet — will poll again in 30s"
exit 1
# The expected state for as long as nobody has bound the box. Log it once, then only on
# a slow heartbeat — an operator reading the journal still sees liveness without the
# every-30s wall of text that made the real signal hard to find.
if (( cycle == 0 )); then
log "not bound yet — polling every ${POLL_INTERVAL}s until the operator or a customer self-bind lands (this is the normal waiting state, not an error)"
elif (( cycle % HEARTBEAT_EVERY == 0 )); then
log "still waiting to be bound ($(( cycle * POLL_INTERVAL / 60 ))m elapsed; polling continues)"
fi
;;
410)
log "ERROR: delivery already consumed but no local env — unit will retry (rare crash-window)"
# The delivery was consumed but we hold no env — a rare crash window. Exiting hands the
# box back to systemd, whose restart re-runs us from a clean slate.
log "ERROR: delivery already consumed but no local env — exiting so the unit restarts (rare crash-window)"
exit 1
;;
404)
log "appliance token not recognized (discarded, or the hub has no record) — will retry in 30s"
exit 1
if (( cycle % HEARTBEAT_EVERY == 0 )); then
log "appliance token not recognized (discarded, or the hub has no record) — still retrying"
fi
;;
*)
log "poll returned HTTP ${code:-none} — will retry in 30s"
exit 1
if (( cycle % HEARTBEAT_EVERY == 0 )); then
log "poll returned HTTP ${code:-none} — still retrying"
fi
;;
esac
esac
sleep "$POLL_INTERVAL"
cycle=$(( cycle + 1 ))
done
}
# --- mode selection -------------------------------------------------------------------------------