admin
515e0c3cc7
Provisioned nested-PVE drill 'drill-r50' (qm300 on demo-hp) via the v1.25.0 nested-vm ISO through the real day-0, then ran the R-50 empirical spike: - vmbr9 portless island bridge + guest island NIC hot-add (LAN undisturbed) - F1 replay money shot: LAN move survives on the island; LAN-literal bind reproduces the 2026-07-20 daemon-exit bug verbatim - dnsmasq trap confirmed live + lan_resolver.host_ip fix proven - pin address-independent (leaf SHA-256 unchanged, HTTP 200 over island) - survival matrix: agent/guest/host-cold-reboot all return on the island Docs: SPIKE verdict BLOCKED->GO, ROADMAP R-50 SPIKED->GO, nodes.md drill VM, REPORT overwrite.
56 lines
4.2 KiB
Markdown
56 lines
4.2 KiB
Markdown
# REPORT — R-50 island-bridge drill + empirical spike (RUNBOOK, 2026-07-25 PM)
|
||
|
||
## What ran
|
||
Provisioned the first **nested-PVE drill appliance on the t740 (`demo-hp`)** and ran the **R-50
|
||
island-bridge empirical spike** end-to-end. Verdict: **GO.**
|
||
|
||
## Part A — drill provisioned through the REAL day-0
|
||
- **VM 300 `drill-r50`** on `demo-hp`: 8 GiB / 4 vCPU (cpu=host) / 32 GiB local-lvm / OVMF (SB off) / one
|
||
NIC on `vmbr0` DHCP. Nested-virt already enabled on the host (A2 no-op; no QEMU VM was running at A1).
|
||
- Installed from **`felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso`** (current release; the
|
||
RUNBOOK's stated *v1.22.0* was both stale **and** the wrong profile — v1.22.0-nested-**canary** is a
|
||
deliberate match-nothing installer that aborts touching no disk; corrected to the v1.25.0 **nested-vm**
|
||
profile. Friction finding, not a day-0 defect).
|
||
- Day-0 drove itself: install → first-boot → **self-register** (appliance #9, pairing `9MB-4QX`) →
|
||
**operator bind** to a scratch customer `drill-r50` (minimal config, no DNS/offsite/PBS side-effects) →
|
||
**deliver** → nested guest **9201** provisioned, controller stack healthy (agent 0.93.0, golden 0.161.0).
|
||
- **R-50 starting condition reached & confirmed:** agent `listen_addr = 192.168.0.176:8443` **and** guest
|
||
`bootstrap.json local_api.endpoint = 192.168.0.176:8443` — the F1 LAN literal baked in both places.
|
||
- Snapshot **`r50pre`** taken (clean day-0). Break-glass vaulted `host_recovery/drill-r50-0a4f9a`.
|
||
- **Day-0 tail not driven:** *claim* is customer-email-gated (scratch customer has no inbox; forcing it
|
||
needs a real email + a production Resend send) — left PENDING; *escrow* is N/A (no DR/PBS tier on the
|
||
minimal drill). Neither gates the control-plane probes. **Operator decision if the full customer-side
|
||
tail should be exercised (needs an email address + DR-tier choice).**
|
||
|
||
## Part B — probes P1–P8, ALL PASS (verdict GO)
|
||
- **P2** `vmbr9` portless island bridge `169.254.253.1/30` — vmbr0 untouched, LAN gw reachable.
|
||
- **P3** guest island NIC `eth1 169.254.253.2/30` hot-added — eth0/LAN undisturbed, bidirectional island ping.
|
||
- **P4 (dnsmasq trap + fix)** — with `host_ip` unset, moving `listen_addr` to the island rebound dnsmasq
|
||
to `169.254.253.1:53` and **killed LAN DNS**; `lan_resolver.host_ip=192.168.0.176` restored it while the
|
||
API bind stayed on the island. Finding 1 confirmed + fixed, live.
|
||
- **P5 (pin)** — served leaf SHA-256 over the island **identical** (`4ef1d953…`), authenticated
|
||
`GET /storage` from the guest over the island → **HTTP 200**. No cert re-issue (Finding 6 live).
|
||
- **P6 (F1 replay — money shot)** — LAN `192.168.0.176→.200`, `listen_addr` on the island → agent stays
|
||
active + serves (200). LAN-literal contrast reproduced the 2026-07-20 bug verbatim
|
||
(`localapi: bind 192.168.0.176:8443` → daemon exit 1).
|
||
- **P7 (survival)** — agent restart / guest reboot / **host cold reboot** all return the control plane on
|
||
the island with zero intervention (vmbr9, island bind, dnsmasq on LAN IP, guest autostart, controller).
|
||
- **Method caveat:** probes drove the runtime chain via manual config edits; the *provisioning* path
|
||
(host-install + golden-bake bootstrap template) is the implementation task — now fully de-risked.
|
||
|
||
## Docs updated
|
||
- `documentation/audits/SPIKE-island-bridge-2026-07-25.md` — verdict flipped BLOCKED → **GO**, empirical
|
||
P1–P8 results added, old BLOCKED/NOT-RUN sections marked superseded.
|
||
- `documentation/backlog/ROADMAP.md` — R-50 → **SPIKED → GO**; next = Phase A/B/C production spec.
|
||
- `documentation/operations/nodes.md` — the drill VM 300 now documented on the t740 (access, snapshot,
|
||
teardown).
|
||
|
||
## State left behind
|
||
Drill VM 300 left **running in the working island configuration** (snapshot `r50pre` preserves clean
|
||
day-0). Scratch hub records `drill-r50` (customer + appliance #9) remain — throwaway; teardown noted in
|
||
nodes.md. `demo-hp` host networking, guest 9201, and felhom-pve were **untouched**.
|
||
|
||
## Not done here (by design / veto)
|
||
- **Part C** (deploy agent 0.95.0 → demo-hp) — veto-able; pending after this evidence.
|
||
- **Forensic** (qm300 disappearance on felhom-pve) — optional, read-only; pending.
|