From 485321f694ae81be5a19e0bbc7d949235391708f Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Sat, 25 Jul 2026 14:17:43 +0200 Subject: [PATCH] R-50 Phase A: host-install v1.19.0 island default + hub version sync - felhom-host-install v1.19.0: portless vmbr9 island bridge, appliance binds local_api on 169.254.253.1:8443, writes island_bridge/island_guest_addr, pins lan_resolver.host_ip to the LAN IP (Finding-1). --no-island opt-out. - hub hostInstallVersion 1.16.0 -> 1.19.0 (F-1 sync). hostinstall_gates PASS. - Pairs with agent v0.96.0 (attaches guest net1). byo unchanged. Coupling: island install requires agent >= 0.96.0 (vouch first). --- REPORT.md | 89 ++++++++++---------------------- hub/CHANGELOG.md | 8 +++ hub/internal/web/configs.go | 2 +- scripts/CHANGELOG.md | 19 +++++++ scripts/felhom-host-install.sh | 93 +++++++++++++++++++++++++++++++--- 5 files changed, 143 insertions(+), 68 deletions(-) diff --git a/REPORT.md b/REPORT.md index 3745c94..f0defa4 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,66 +1,33 @@ -# REPORT — R-50 island-bridge drill + empirical spike (RUNBOOK, 2026-07-25 PM) +# REPORT — R-50 island control plane: Phase A artifacts (felhom.eu) (2026-07-25) -## What ran -Provisioned the first **nested-PVE drill appliance on the t740 (`demo-hp`)** and ran the **R-50 -island-bridge empirical spike** end-to-end. Verdict: **GO.** +Phase A of the R-50 implementation (spike GO). Fresh **appliance** installs are now born on the island +(F1-immune). Pairs with agent **v0.96.0** (which attaches the guest's island net1). byo unchanged. -## Part A — drill provisioned through the REAL day-0 -- **VM 300 `drill-r50`** on `demo-hp`: 8 GiB / 4 vCPU (cpu=host) / 32 GiB local-lvm / OVMF (SB off) / one - NIC on `vmbr0` DHCP. Nested-virt already enabled on the host (A2 no-op; no QEMU VM was running at A1). -- Installed from **`felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso`** (current release; the - RUNBOOK's stated *v1.22.0* was both stale **and** the wrong profile — v1.22.0-nested-**canary** is a - deliberate match-nothing installer that aborts touching no disk; corrected to the v1.25.0 **nested-vm** - profile. Friction finding, not a day-0 defect). -- Day-0 drove itself: install → first-boot → **self-register** (appliance #9, pairing `9MB-4QX`) → - **operator bind** to a scratch customer `drill-r50` (minimal config, no DNS/offsite/PBS side-effects) → - **deliver** → nested guest **9201** provisioned, controller stack healthy (agent 0.93.0, golden 0.161.0). -- **R-50 starting condition reached & confirmed:** agent `listen_addr = 192.168.0.176:8443` **and** guest - `bootstrap.json local_api.endpoint = 192.168.0.176:8443` — the F1 LAN literal baked in both places. -- Snapshot **`r50pre`** taken (clean day-0). Break-glass vaulted `host_recovery/drill-r50-0a4f9a`. -- **Day-0 tail not driven:** *claim* is customer-email-gated (scratch customer has no inbox; forcing it - needs a real email + a production Resend send) — left PENDING; *escrow* is N/A (no DR/PBS tier on the - minimal drill). Neither gates the control-plane probes. **Operator decision if the full customer-side - tail should be exercised (needs an email address + DR-tier choice).** +## Changed here +- `scripts/felhom-host-install.sh` → **v1.19.0**: `ensure_island_bridge()` (portless `vmbr9`, + `169.254.253.1/30`, idempotent); appliance default binds `local_api` on the island + (`169.254.253.1:8443`), writes `island_bridge`/`island_guest_addr`, and pins `lan_resolver.host_ip` + to the LAN IP explicitly (Finding-1 — else moving the bind silently kills LAN DNS). `--no-island` + opt-out; explicit `--bridge-ip` overrides. Gate `hostinstall_gates.py` ALL PASS. +- `hub/internal/web/configs.go` → `hostInstallVersion` 1.16.0 → **1.19.0** (F-1 sync; clears prior + drift). Hub CHANGELOG → v0.73.2. Web render test green. -## Part B — probes P1–P8, ALL PASS (verdict GO) -- **P2** `vmbr9` portless island bridge `169.254.253.1/30` — vmbr0 untouched, LAN gw reachable. -- **P3** guest island NIC `eth1 169.254.253.2/30` hot-added — eth0/LAN undisturbed, bidirectional island ping. -- **P4 (dnsmasq trap + fix)** — with `host_ip` unset, moving `listen_addr` to the island rebound dnsmasq - to `169.254.253.1:53` and **killed LAN DNS**; `lan_resolver.host_ip=192.168.0.176` restored it while the - API bind stayed on the island. Finding 1 confirmed + fixed, live. -- **P5 (pin)** — served leaf SHA-256 over the island **identical** (`4ef1d953…`), authenticated - `GET /storage` from the guest over the island → **HTTP 200**. No cert re-issue (Finding 6 live). -- **P6 (F1 replay — money shot)** — LAN `192.168.0.176→.200`, `listen_addr` on the island → agent stays - active + serves (200). LAN-literal contrast reproduced the 2026-07-20 bug verbatim - (`localapi: bind 192.168.0.176:8443` → daemon exit 1). -- **P7 (survival)** — agent restart / guest reboot / **host cold reboot** all return the control plane on - the island with zero intervention (vmbr9, island bind, dnsmasq on LAN IP, guest autostart, controller). -- **Method caveat:** probes drove the runtime chain via manual config edits; the *provisioning* path - (host-install + golden-bake bootstrap template) is the implementation task — now fully de-risked. +## Determination (A0/A3) — recorded +- **A0 = config-only for the bootstrap endpoint** (`main.go` sets it from `listen_addr`); the version + bump lives in the agent's net1-attach. The operator chose **agent-bringup** (v0.96.0) over golden-bake + to eliminate the golden-coupling failure mode. +- **A3 = healer verify-only** — eth0-only (`parseMode` dev-scoped); red-proof test in the agent. +- **net1 deviation from the spike inventory** (it placed net1 in host-install, but host-install doesn't + create the guest — the agent does) resolved by attaching net1 in the agent at bring-up. -## Docs updated -- `documentation/audits/SPIKE-island-bridge-2026-07-25.md` — verdict flipped BLOCKED → **GO**, empirical - P1–P8 results added, old BLOCKED/NOT-RUN sections marked superseded. -- `documentation/backlog/ROADMAP.md` — R-50 → **SPIKED → GO**; next = Phase A/B/C production spec. -- `documentation/operations/nodes.md` — the drill VM 300 now documented on the t740 (access, snapshot, - teardown). +## Coupling / deploy order (important) +An island host-install REQUIRES agent ≥ 0.96.0. **Vouch 0.96.0 before island installs go live**, else a +fresh install writes island config an older agent ignores → guest has no island NIC → broken control +plane. This is the one hard ordering constraint. -## State left behind -Drill VM 300 left **running in the working island configuration** (snapshot `r50pre` preserves clean -day-0). Scratch hub records `drill-r50` (customer + appliance #9) remain — throwaway; teardown noted in -nodes.md. `demo-hp` host networking, guest 9201, and felhom-pve were **untouched**. - -## Part C — agent 0.95.0 deployed to demo-hp (DONE) -`0.93.0 → 0.95.0` via the sanctioned method (green gate build/vet/test pass; `configs/` unchanged since -0.93.0 so binary-only). scp via break-glass → backup `.bak-0.93.0` → `install -m0755` → `systemctl -restart`. Verified: deployed sha256 == local build (`b5fa7c5d…`), version 0.95.0, service active, -**capabilities 68/68 ok, 0 degraded**, local-api listening, all watchdogs up, no ERROR/panic; **guest -9201 untouched/running**; hub confirms demo-hp reporting 0.95.0. (Note: the hub's *vouched* Day-0 -artifact still reads 0.93.0 — the manifest vouch is a separate operator step, not needed for a running -host; flag for follow-up if new installs should get 0.95.0.) Both fleet nodes now run 0.95.0. - -## Forensic — where did qm300 go? (DONE, read-only) -On felhom-pve (demo-felhom, PVE 9.2.2): **no discrete `qmdestroy`.** No `300.conf`, no `vm-300` LVs, no -`:300:` in any task log (incl. archives), `qm list` empty; task history only reaches back to ~2026-07-16. -qm300 (the old drill VM) vanished when the **N100 was fully reprovisioned/reborn** mid-July — the whole -node (task DB + LVM) was recreated from scratch, taking qm300 with it. felhom-pve left untouched. +## Status of the phases +- **Phase A (artifacts): DONE + committed** (agent `dfd5d73`; this repo this commit). Agent build/publish + + hub deploy + golden unchanged (net1 comes from the agent, so no re-bake). +- **Phase B (migration): pending** — `RUNBOOK-island-migration.md` + drill migration (B1) + fresh-install + validation (A4), then demo-hp (B2) and demo-felhom (B3) behind operator go/no-go STOP points. +- **Phase C (Peti cluster): parked** — its own supervised runbook + a new ROADMAP row. diff --git a/hub/CHANGELOG.md b/hub/CHANGELOG.md index 82c029c..6ef92d8 100644 --- a/hub/CHANGELOG.md +++ b/hub/CHANGELOG.md @@ -1,5 +1,13 @@ # Felhom Hub — Changelog +## v0.73.2 — sync `hostInstallVersion` → 1.19.0 (R-50 island host-install) (2026-07-25) + +`hostInstallVersion` (the script version the operator customer page's install-command generator +advertises) bumped 1.16.0 → **1.19.0** to match `felhom-host-install v1.19.0` (R-50 island default). The +`hostinstall_gates.py` F-1 gate requires the two move together; this also clears the pre-existing +1.16.0↔1.18.0 drift. Render test (`render_test.go`) confirms the value reaches the page. No behaviour +change beyond the advertised version string. + ## v0.73.1 — allowlist `disk_health_degraded` (controller v0.169.0 disk-health) (2026-07-24) Adds `disk_health_degraded` to `allowedEventTypes` so the controller's per-disk SMART degradation diff --git a/hub/internal/web/configs.go b/hub/internal/web/configs.go index 6c0b3b8..2ad4db1 100644 --- a/hub/internal/web/configs.go +++ b/hub/internal/web/configs.go @@ -25,7 +25,7 @@ var validCustomerID = regexp.MustCompile(`^[a-zA-Z0-9.\-]+$`) // only ever emits flags this version parses. Display-only (the Option-1 command downloads the served // script, which is always current); bump when the generator's flag surface follows a new script. // Drift is now gated: scripts/hostinstall_gates.py asserts this const == SCRIPT_VERSION (drill F-1). -const hostInstallVersion = "1.16.0" +const hostInstallVersion = "1.19.0" // validSemver matches a bare X.Y.Z controller version (the floor format). Empty is also accepted by // the floor handlers (clears the override). diff --git a/scripts/CHANGELOG.md b/scripts/CHANGELOG.md index 1a1d95c..9079d3d 100644 --- a/scripts/CHANGELOG.md +++ b/scripts/CHANGELOG.md @@ -1,5 +1,24 @@ # Felhom scripts — Changelog +## felhom-host-install v1.19.0 — R-50 island control plane is the appliance default (2026-07-25) + +Implements Phase A of the R-50 island bridge (spike GO, `documentation/audits/SPIKE-island-bridge-2026-07-25.md`). +A fresh **appliance** install is now born immune to F1 (a LAN/DHCP/site move can no longer take the +control plane down). **byo is unchanged**; an explicit `--bridge-ip` or `--no-island` keeps the LAN bind. + +- New `ensure_island_bridge()` creates a portless host-internal bridge `vmbr9` (`169.254.253.1/30`), + idempotently, before the agent binds it. Portless = no physical port, so the /30 cannot collide with + any customer LAN and survives any renumber. vmbr0 untouched. +- `step_agent_config` (appliance default): `local_api.listen_addr` → `169.254.253.1:8443`; writes + `local_api.island_bridge`/`island_guest_addr` (the agent ≥ 0.96.0 attaches the guest's `net1` from + these); and pins `lan_resolver.host_ip` to the **LAN** IP explicitly — **Finding-1**: without it the + agent derives the DNS listen-addr from `listen_addr` and silently moves LAN DNS onto the island. +- `--no-island` opt-out; `--bridge-ip` still overrides. `configs/felhom-localapi-firewall.example` + (agent repo) rewritten — the portless island bind IS the LAN close. +- **Coupling:** this install REQUIRES agent ≥ 0.96.0 (older agents ignore the island fields → no net1 → + broken control plane). Vouch 0.96.0 before island installs go live. Hub `hostInstallVersion` synced to + 1.19.0 (F-1). Existing boxes migrate via `documentation/runbooks/RUNBOOK-island-migration.md`. + ## ISO train v1.25.0 — the belt goes fleet-wide, the repo goes free, the screen learns ő, fresh boxes boot at the floor (2026-07-23) Four rulings from 2026-07-23. host-install **1.17.0 → 1.18.0**, build-felhom-iso **1.24.0 → 1.25.0**, diff --git a/scripts/felhom-host-install.sh b/scripts/felhom-host-install.sh index 066e270..c45ded1 100644 --- a/scripts/felhom-host-install.sh +++ b/scripts/felhom-host-install.sh @@ -111,7 +111,9 @@ # --force-gitea-golden ignore any local golden; fetch+verify the golden from # Gitea (proves the fetch path; used by the live test) # --node NAME PVE node name (default: pvesh /nodes, else hostname) -# --bridge-ip IP[:PORT] local-api listen addr (default: vmbr0 IP : 8443) +# --bridge-ip IP[:PORT] local-api listen addr; an explicit value overrides the island default +# (default: appliance → island 169.254.253.1:8443; byo → vmbr0 IP:8443) +# --no-island appliance only: keep the historical LAN bind instead of the R-50 island # --rootfs-grow N grow OS rootfs by N GiB (default: auto-compute) # --datavol-grow N grow Docker-data vol by N GiB (default: auto-compute) # --sysdata-grow N grow user-data vol by N GiB (default: auto-compute) @@ -182,7 +184,7 @@ set -euo pipefail -SCRIPT_VERSION="1.18.0" # the SINGLE version source (F-1): -h, the run banners, and the hub +SCRIPT_VERSION="1.19.0" # the SINGLE version source (F-1): -h, the run banners, and the hub # Setup-tab copy (hub internal/web/configs.go hostInstallVersion — # scripts/hostinstall_gates.py asserts the two stay equal) all follow it. # 1.16.0: the FELHOM_ESCROW sudoers alias (controller-driven escrow @@ -229,6 +231,19 @@ ARCHIVE_STORAGE="local" NODE="" NODE_EXPLICIT=false # set true when --node is given; gates the multi-node wrong-node guard BRIDGE_ADDR="" +# R-50 island control plane (SPIKE-island-bridge-2026-07-25, GO). The appliance default binds the +# local API on a host-internal, portless bridge with a fixed private /30 so the controller→agent +# channel survives any LAN/DHCP/site move (the F1 fix). These are the spike's VALIDATED parameters — +# do not change them. Set ISLAND_ON=1 only in appliance mode with no explicit --bridge-ip override +# (byo and an explicit --bridge-ip keep the historical LAN bind). --no-island opts an appliance out. +ISLAND_BRIDGE="vmbr9" +ISLAND_HOST_CIDR="169.254.253.1/30" +ISLAND_HOST_IP="169.254.253.1" +ISLAND_GUEST_CIDR="169.254.253.2/30" +ISLAND_PORT="8443" +ISLAND_ON=0 +WANT_ISLAND=true # appliance default; --no-island sets false +LAN_IP="" # vmbr0 IPv4 (computed in step_agent_config; the LAN anchor for lan_resolver) ROOTFS_GROW="" DATAVOL_GROW="" SYSDATA_GROW="" @@ -1061,6 +1076,7 @@ while [[ $# -gt 0 ]]; do --archive-storage) ARCHIVE_STORAGE="$2"; shift 2 ;; --node) NODE="$2"; NODE_EXPLICIT=true; shift 2 ;; --bridge-ip) BRIDGE_ADDR="$2"; shift 2 ;; + --no-island) WANT_ISLAND=false; shift ;; # R-50: keep the historical LAN bind on an appliance --rootfs-grow) ROOTFS_GROW="$2"; shift 2 ;; --datavol-grow) DATAVOL_GROW="$2"; shift 2 ;; --sysdata-grow) SYSDATA_GROW="$2"; shift 2 ;; @@ -2148,6 +2164,47 @@ install_oob() { log_success " installed OOB felhom-sshd instance + static belt (agent renders config + fills sets once oob.enabled)" } +#------------------------------------------------------------------------------- +# ensure_island_bridge — create the R-50 host-internal island bridge (vmbr9, portless, fixed /30) if +# absent, idempotently: append an ifupdown2 stanza to /etc/network/interfaces + `ifreload -a`. +# Portless (bridge-ports none) = no physical NIC, so the /30 cannot collide with any customer LAN and +# survives any LAN renumber; vmbr0 is a separate stanza and is never touched. Safe to re-run (no-ops +# when present). Spike-proven non-disruptive (SPIKE-island-bridge-2026-07-25, probe P2). +#------------------------------------------------------------------------------- +ensure_island_bridge() { + local ifaces=/etc/network/interfaces + if grep -qE "^[[:space:]]*iface[[:space:]]+${ISLAND_BRIDGE}[[:space:]]" "$ifaces" 2>/dev/null; then + log_info " island bridge ${ISLAND_BRIDGE} already present — leaving it" + return 0 + fi + if $DRY_RUN; then + log_dry "append ${ISLAND_BRIDGE} stanza (portless, ${ISLAND_HOST_CIDR}) to $ifaces + ifreload -a" + return 0 + fi + log_info " creating island bridge ${ISLAND_BRIDGE} (portless, ${ISLAND_HOST_CIDR})" + cp -a "$ifaces" "${ifaces}.pre-island.bak" 2>/dev/null || true + cat >> "$ifaces" </dev/null 2>&1; then + run ifreload -a || log_warn " ifreload -a returned non-zero — verify ${ISLAND_BRIDGE} is up" + else + run ifup "${ISLAND_BRIDGE}" || log_warn " ifup ${ISLAND_BRIDGE} returned non-zero" + fi + ip -4 -o addr show "${ISLAND_BRIDGE}" 2>/dev/null | grep -q "${ISLAND_HOST_IP}" \ + && log_success " ${ISLAND_BRIDGE} up: ${ISLAND_HOST_CIDR}" \ + || log_warn " ${ISLAND_BRIDGE} did not come up with ${ISLAND_HOST_IP} — check $ifaces" +} + #------------------------------------------------------------------------------- # STEP 6 — write agent config + ensure service healthy #------------------------------------------------------------------------------- @@ -2157,10 +2214,20 @@ step_agent_config() { local fp fp=$(echo | openssl s_client -connect 127.0.0.1:8006 2>/dev/null | openssl x509 -noout -fingerprint -sha256 2>/dev/null | sed 's/.*=//') [[ -n "$fp" ]] || log_warn " could not compute TLS fingerprint (leaving empty — agent will use system trust)" - # bridge / local-api addr - if [[ -z "$BRIDGE_ADDR" ]]; then - local ip; ip=$(ip -4 -o addr show vmbr0 2>/dev/null | awk '{print $4}' | cut -d/ -f1 | head -1) - BRIDGE_ADDR="${ip:-127.0.0.1}:8443" + # bridge / local-api addr. LAN_IP (vmbr0) is always resolved — it is the lan_resolver anchor even + # on an island install (Finding-1), and the historical bind on a non-island install. + LAN_IP=$(ip -4 -o addr show vmbr0 2>/dev/null | awk '{print $4}' | cut -d/ -f1 | head -1) + if [[ "$MODE" == "appliance" && "$WANT_ISLAND" == true && -z "$BRIDGE_ADDR" ]]; then + # R-50 appliance default: bind the local API on the host-internal island bridge (LAN-move + # immune). Create the bridge first so the agent can bind it; point listen_addr at it; the guest + # gets net1 from the agent (island_bridge/island_guest_addr, written below). lan_resolver.host_ip + # is pinned to the LAN IP in the config write — never let it default off the island listen_addr. + ensure_island_bridge + BRIDGE_ADDR="${ISLAND_HOST_IP}:${ISLAND_PORT}" + ISLAND_ON=1 + log_info " R-50 island ON: local_api=${BRIDGE_ADDR} (${ISLAND_BRIDGE}); guest net1=${ISLAND_GUEST_CIDR}; lan_resolver.host_ip=${LAN_IP:-}" + elif [[ -z "$BRIDGE_ADDR" ]]; then + BRIDGE_ADDR="${LAN_IP:-127.0.0.1}:8443" # byo / --no-island / explicit-LAN: historical bind elif [[ "$BRIDGE_ADDR" != *:* ]]; then BRIDGE_ADDR="${BRIDGE_ADDR}:8443" fi @@ -2187,6 +2254,7 @@ step_agent_config() { # Secrets passed via env (NOT argv) to avoid ps exposure. PVE_TOKEN="$PVE_TOKEN" HOST_API_KEY="$HOST_API_KEY" \ NODE="$NODE" FP="$fp" HUB_URL="$HUB_URL" HOST_ID="$HOST_ID" BRIDGE_ADDR="$BRIDGE_ADDR" \ + ISLAND_ON="$ISLAND_ON" ISLAND_BRIDGE="$ISLAND_BRIDGE" ISLAND_GUEST_CIDR="$ISLAND_GUEST_CIDR" LAN_IP="$LAN_IP" \ OP_KEY_ID="$RESOLVED_OP_ID" OP_KEY_LINE="$RESOLVED_OP_LINE" \ REC_KEY_ID="$RESOLVED_REC_ID" REC_KEY_LINE="$RESOLVED_REC_LINE" \ PRESERVE_FROM="$PRESERVE_FROM" INSTALL_MODE="$MODE" ENABLE_OOB="$ENABLE_OOB" OUT="$AGENT_CONFIG" python3 <<'PY' @@ -2224,6 +2292,12 @@ base.setdefault('backup', {"local_backup_target":"local","local_backup_retention base.setdefault('local_api', {}) base['local_api'].setdefault('enable', True) base['local_api']['listen_addr'] = os.environ['BRIDGE_ADDR'] +# R-50: on an island install, record the island bridge + the guest net1 CIDR so the provisioner +# (bringup) attaches the guest's static island NIC. All-or-nothing — the agent's config.Validate +# rejects a half-set island. On a non-island install these keys are ABSENT (pre-R-50, LAN-only). +if os.environ.get('ISLAND_ON') == '1': + base['local_api']['island_bridge'] = os.environ['ISLAND_BRIDGE'] + base['local_api']['island_guest_addr'] = os.environ['ISLAND_GUEST_CIDR'] base['local_api'].setdefault('cert_file','/var/lib/felhom-agent/local-api.crt') base['local_api'].setdefault('key_file','/var/lib/felhom-agent/local-api.key') base['local_api'].setdefault('token_store','/var/lib/felhom-agent/local-tokens.log') @@ -2234,6 +2308,13 @@ if os.environ.get('INSTALL_MODE') == 'byo': base.setdefault('lan_resolver', {"enable": False}) else: base.setdefault('lan_resolver', {"enable": True}) + # R-50 Finding-1 (the dnsmasq trap): pin the split-horizon DNS to the LAN IP EXPLICITLY. Without + # this, config.go's LANResolverConfig.WithDefaults derives host_ip from listen_addr — which on an + # island install is the island IP (169.254.253.1) — silently moving LAN DNS onto the island bridge + # and killing it (spike-confirmed live). Set whenever the LAN IP is known (island or LAN bind). + _lan = os.environ.get('LAN_IP','').strip() + if _lan: + base['lan_resolver']['host_ip'] = _lan # WG tunnel (drill F-9; operator decision 2026-07-12 #5): WG is BASE infrastructure like # cloudflared — always-on for every install, both modes (it is also the future OOB-management # transport; the PBS-DR tier merely RIDES it). Registration is hands-free (the drill proved it: