R-50 Phase A: host-install v1.19.0 island default + hub version sync
- felhom-host-install v1.19.0: portless vmbr9 island bridge, appliance binds local_api on 169.254.253.1:8443, writes island_bridge/island_guest_addr, pins lan_resolver.host_ip to the LAN IP (Finding-1). --no-island opt-out. - hub hostInstallVersion 1.16.0 -> 1.19.0 (F-1 sync). hostinstall_gates PASS. - Pairs with agent v0.96.0 (attaches guest net1). byo unchanged. Coupling: island install requires agent >= 0.96.0 (vouch first).
This commit is contained in:
@@ -1,66 +1,33 @@
|
||||
# REPORT — R-50 island-bridge drill + empirical spike (RUNBOOK, 2026-07-25 PM)
|
||||
# REPORT — R-50 island control plane: Phase A artifacts (felhom.eu) (2026-07-25)
|
||||
|
||||
## What ran
|
||||
Provisioned the first **nested-PVE drill appliance on the t740 (`demo-hp`)** and ran the **R-50
|
||||
island-bridge empirical spike** end-to-end. Verdict: **GO.**
|
||||
Phase A of the R-50 implementation (spike GO). Fresh **appliance** installs are now born on the island
|
||||
(F1-immune). Pairs with agent **v0.96.0** (which attaches the guest's island net1). byo unchanged.
|
||||
|
||||
## Part A — drill provisioned through the REAL day-0
|
||||
- **VM 300 `drill-r50`** on `demo-hp`: 8 GiB / 4 vCPU (cpu=host) / 32 GiB local-lvm / OVMF (SB off) / one
|
||||
NIC on `vmbr0` DHCP. Nested-virt already enabled on the host (A2 no-op; no QEMU VM was running at A1).
|
||||
- Installed from **`felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso`** (current release; the
|
||||
RUNBOOK's stated *v1.22.0* was both stale **and** the wrong profile — v1.22.0-nested-**canary** is a
|
||||
deliberate match-nothing installer that aborts touching no disk; corrected to the v1.25.0 **nested-vm**
|
||||
profile. Friction finding, not a day-0 defect).
|
||||
- Day-0 drove itself: install → first-boot → **self-register** (appliance #9, pairing `9MB-4QX`) →
|
||||
**operator bind** to a scratch customer `drill-r50` (minimal config, no DNS/offsite/PBS side-effects) →
|
||||
**deliver** → nested guest **9201** provisioned, controller stack healthy (agent 0.93.0, golden 0.161.0).
|
||||
- **R-50 starting condition reached & confirmed:** agent `listen_addr = 192.168.0.176:8443` **and** guest
|
||||
`bootstrap.json local_api.endpoint = 192.168.0.176:8443` — the F1 LAN literal baked in both places.
|
||||
- Snapshot **`r50pre`** taken (clean day-0). Break-glass vaulted `host_recovery/drill-r50-0a4f9a`.
|
||||
- **Day-0 tail not driven:** *claim* is customer-email-gated (scratch customer has no inbox; forcing it
|
||||
needs a real email + a production Resend send) — left PENDING; *escrow* is N/A (no DR/PBS tier on the
|
||||
minimal drill). Neither gates the control-plane probes. **Operator decision if the full customer-side
|
||||
tail should be exercised (needs an email address + DR-tier choice).**
|
||||
## Changed here
|
||||
- `scripts/felhom-host-install.sh` → **v1.19.0**: `ensure_island_bridge()` (portless `vmbr9`,
|
||||
`169.254.253.1/30`, idempotent); appliance default binds `local_api` on the island
|
||||
(`169.254.253.1:8443`), writes `island_bridge`/`island_guest_addr`, and pins `lan_resolver.host_ip`
|
||||
to the LAN IP explicitly (Finding-1 — else moving the bind silently kills LAN DNS). `--no-island`
|
||||
opt-out; explicit `--bridge-ip` overrides. Gate `hostinstall_gates.py` ALL PASS.
|
||||
- `hub/internal/web/configs.go` → `hostInstallVersion` 1.16.0 → **1.19.0** (F-1 sync; clears prior
|
||||
drift). Hub CHANGELOG → v0.73.2. Web render test green.
|
||||
|
||||
## Part B — probes P1–P8, ALL PASS (verdict GO)
|
||||
- **P2** `vmbr9` portless island bridge `169.254.253.1/30` — vmbr0 untouched, LAN gw reachable.
|
||||
- **P3** guest island NIC `eth1 169.254.253.2/30` hot-added — eth0/LAN undisturbed, bidirectional island ping.
|
||||
- **P4 (dnsmasq trap + fix)** — with `host_ip` unset, moving `listen_addr` to the island rebound dnsmasq
|
||||
to `169.254.253.1:53` and **killed LAN DNS**; `lan_resolver.host_ip=192.168.0.176` restored it while the
|
||||
API bind stayed on the island. Finding 1 confirmed + fixed, live.
|
||||
- **P5 (pin)** — served leaf SHA-256 over the island **identical** (`4ef1d953…`), authenticated
|
||||
`GET /storage` from the guest over the island → **HTTP 200**. No cert re-issue (Finding 6 live).
|
||||
- **P6 (F1 replay — money shot)** — LAN `192.168.0.176→.200`, `listen_addr` on the island → agent stays
|
||||
active + serves (200). LAN-literal contrast reproduced the 2026-07-20 bug verbatim
|
||||
(`localapi: bind 192.168.0.176:8443` → daemon exit 1).
|
||||
- **P7 (survival)** — agent restart / guest reboot / **host cold reboot** all return the control plane on
|
||||
the island with zero intervention (vmbr9, island bind, dnsmasq on LAN IP, guest autostart, controller).
|
||||
- **Method caveat:** probes drove the runtime chain via manual config edits; the *provisioning* path
|
||||
(host-install + golden-bake bootstrap template) is the implementation task — now fully de-risked.
|
||||
## Determination (A0/A3) — recorded
|
||||
- **A0 = config-only for the bootstrap endpoint** (`main.go` sets it from `listen_addr`); the version
|
||||
bump lives in the agent's net1-attach. The operator chose **agent-bringup** (v0.96.0) over golden-bake
|
||||
to eliminate the golden-coupling failure mode.
|
||||
- **A3 = healer verify-only** — eth0-only (`parseMode` dev-scoped); red-proof test in the agent.
|
||||
- **net1 deviation from the spike inventory** (it placed net1 in host-install, but host-install doesn't
|
||||
create the guest — the agent does) resolved by attaching net1 in the agent at bring-up.
|
||||
|
||||
## Docs updated
|
||||
- `documentation/audits/SPIKE-island-bridge-2026-07-25.md` — verdict flipped BLOCKED → **GO**, empirical
|
||||
P1–P8 results added, old BLOCKED/NOT-RUN sections marked superseded.
|
||||
- `documentation/backlog/ROADMAP.md` — R-50 → **SPIKED → GO**; next = Phase A/B/C production spec.
|
||||
- `documentation/operations/nodes.md` — the drill VM 300 now documented on the t740 (access, snapshot,
|
||||
teardown).
|
||||
## Coupling / deploy order (important)
|
||||
An island host-install REQUIRES agent ≥ 0.96.0. **Vouch 0.96.0 before island installs go live**, else a
|
||||
fresh install writes island config an older agent ignores → guest has no island NIC → broken control
|
||||
plane. This is the one hard ordering constraint.
|
||||
|
||||
## State left behind
|
||||
Drill VM 300 left **running in the working island configuration** (snapshot `r50pre` preserves clean
|
||||
day-0). Scratch hub records `drill-r50` (customer + appliance #9) remain — throwaway; teardown noted in
|
||||
nodes.md. `demo-hp` host networking, guest 9201, and felhom-pve were **untouched**.
|
||||
|
||||
## Part C — agent 0.95.0 deployed to demo-hp (DONE)
|
||||
`0.93.0 → 0.95.0` via the sanctioned method (green gate build/vet/test pass; `configs/` unchanged since
|
||||
0.93.0 so binary-only). scp via break-glass → backup `.bak-0.93.0` → `install -m0755` → `systemctl
|
||||
restart`. Verified: deployed sha256 == local build (`b5fa7c5d…`), version 0.95.0, service active,
|
||||
**capabilities 68/68 ok, 0 degraded**, local-api listening, all watchdogs up, no ERROR/panic; **guest
|
||||
9201 untouched/running**; hub confirms demo-hp reporting 0.95.0. (Note: the hub's *vouched* Day-0
|
||||
artifact still reads 0.93.0 — the manifest vouch is a separate operator step, not needed for a running
|
||||
host; flag for follow-up if new installs should get 0.95.0.) Both fleet nodes now run 0.95.0.
|
||||
|
||||
## Forensic — where did qm300 go? (DONE, read-only)
|
||||
On felhom-pve (demo-felhom, PVE 9.2.2): **no discrete `qmdestroy`.** No `300.conf`, no `vm-300` LVs, no
|
||||
`:300:` in any task log (incl. archives), `qm list` empty; task history only reaches back to ~2026-07-16.
|
||||
qm300 (the old drill VM) vanished when the **N100 was fully reprovisioned/reborn** mid-July — the whole
|
||||
node (task DB + LVM) was recreated from scratch, taking qm300 with it. felhom-pve left untouched.
|
||||
## Status of the phases
|
||||
- **Phase A (artifacts): DONE + committed** (agent `dfd5d73`; this repo this commit). Agent build/publish
|
||||
+ hub deploy + golden unchanged (net1 comes from the agent, so no re-bake).
|
||||
- **Phase B (migration): pending** — `RUNBOOK-island-migration.md` + drill migration (B1) + fresh-install
|
||||
validation (A4), then demo-hp (B2) and demo-felhom (B3) behind operator go/no-go STOP points.
|
||||
- **Phase C (Peti cluster): parked** — its own supervised runbook + a new ROADMAP row.
|
||||
|
||||
Reference in New Issue
Block a user