diff --git a/documentation/runbooks/RUNBOOK-rehersal-v3.md b/documentation/runbooks/RUNBOOK-rehersal-v3.md new file mode 100644 index 0000000..25e39b6 --- /dev/null +++ b/documentation/runbooks/RUNBOOK-rehersal-v3.md @@ -0,0 +1,94 @@ +# RUNBOOK v3 — The rehearsal: RESET → generic ISO → bind → customer zero, round two + +> **What this is:** the complete, final product flow executed end-to-end on real hardware for the +> first time — no per-customer ISO, no manual steps that a future operator wouldn't also do. +> One supervised afternoon closes every "staged for the rehearsal" item in the tree. +> +> **The closes-ledger (what this run flips, collect evidence per item):** +> 1. **F1 physical** — the mkimage loader boots the real AMI firmware (slice B's bet) +> 2. **Slice C physical** — generic secret-free ISO → register → bind → day-0 on metal +> 3. **RESET proven-live, full surface** — operator click + Hetzner sub-account deprovision + +> PBS teardown + escrow-ack, twice (drill + demo) +> 4. **S6b-obsolete live** — no manual host deletion anywhere in the flow +> 5. **F5 auto-sizing live** — the guest gets ~12 GB on the 16 GB host, not the golden 2 GB +> 6. **v0.138.0 awaiting-card live render** — the escrow wizard's waiting state, on a real ceremony +> 7. Fresh-generation claim + re-provisioned offsite from a RESET customer (the re-onboarding +> story, whole) +> +> **Point of no return:** booting the install-armed stick (the auto-installer needs no confirm). +> Recovery unchanged: stock ISO + `felhom-host-install.sh` — and since RESET is the plan, there +> is nothing to restore. Demo-felhom.eu is down for the window (~2 h). + +--- + +## Pre-phase (days before, independent) — drill teardown = the v0.61.0 gap closure + +**P1.** Hosts page: delete `demo-vm-felhom-4846bc` (escrow demotes — v0.60.1) and the ghost +`demo-vm-felhom-2f4b00` stale record. VM 311 can be stopped (it dies with the wipe anyway). +**P2.** **RESET `demo-vm-felhom`** — the live operator click (v0.61.0's first): typed confirm + +escrow ack; watch the journal legs: PBS namespace teardown (tenantsync), **Hetzner sub-account +delete — its first real firing; verify via the Hetzner side that the sub-account is gone**, claim +reset, DB purge last. Evidence: the per-leg journal + events. If the Hetzner leg misbehaves: +that's exactly why it runs here first — journal, report, fix before the main run. + +--- + +## Phase 0 — gates (short now; most of v2.1's gates are proven machinery) + +**G1 — RESET `demo-felhom`:** delete host `demo-felhom-01` first (ruling 3; escrow demotes), +then RESET with the ack (this destroys the retained keys + the current repo's real snapshots — +ruled expendable, stated knowingly). Survives: identity, domain, email, CF tunnel + API tokens, +offsite/DR config. Second live firing of the full external teardown. +**G2 — the ISO:** `build-felhom-iso.sh --pairing --loader mkimage --profile n100` — generic, +secret-free (grep-assert per slice C), serial filter `QDF922W009654S30EX` from the July harvest. +The external HDD and scratch1 stay plugged; the filter is the guard; the boot stick itself is +serial-distinct by construction. Note for later: THIS artifact is reusable — no per-run builds +ever again. +**G3 — media + firmware prep list** (one BIOS visit at S2 covers all three): USB stick via +dd/Rufus-DD; the BIOS items queued: boot order, **Secure Boot OFF** (the mkimage contract — +record prior state), **State-After-G3 → Power On** (the standing sidebar item, finally). +**G4 — window:** no in-flight CC work; ~2 h budget; second stick with the stock ISO as fallback. + +--- + +## Phase 1 — the run + +**S1.** Stop apps, shut down guests + host. Hub shows DOWN — expected. +**S2.** The one BIOS visit: boot order → USB; Secure Boot → OFF; State-After-G3 → Power On. +Photos. +**S3. Boot — the F1 moment.** The mkimage GRUB must load where stock GRUB threw +`relocation 0x0`. Expected: straight into the automated installer; the complete prior install is +simply overwritten (slice-B learning — no LVM abort on a whole disk); the serial filter selects +the internal SSD only. Record wall-clock. +**S4.** Unattended install (~3 min) → reboot → first boot → pairing mode → the box appears under +**Unclaimed appliances** at the hub (uuid/MACs per the July DMI harvest; SSH fingerprints shown). +**S5. The Bind click** — to `demo-felhom`. Then watch the events compose (the four-minute +sequence from the nested run, now on metal): credential delivered → day-0 → **guest provisioned +with ~12 GB RAM (F5 check — record the pct config)** → controller starts → fresh-generation claim +code lands in the inbox → floor train lifts the controller to current. +**S6. Customer zero, round two** (the friction lens ON — notes verbatim, they are R-3's text): +claim with the fresh code → set password → **escrow ceremony** — after the wizard, the remote +page must show the **v0.138.0 awaiting card** ("megerősítésre vár, legfeljebb 15 perc") instead +of the old yellow banner, then flip on the confirming ACK (its first live render — screenshot +both states) → recovery code stored → storage: recommission or reformat the external HDD + +scratch1 (run-time choice) → deploy one app → enable offsite (observe: does the surviving +offsite config re-provision a fresh sub-account automatically, or on the first save? — record +which; either is fine, the answer goes in the RESET docs). +**S7.** "Távoli mentés most" → the first snapshot of the reborn box lands in a brand-new repo +under a freshly escrowed key. No orphan card expected (the repo is new by construction) — if one +appears, that's a finding, capture it. +**S8. Wrap (CC post-run):** evidence to `180:~/n100-rehearsal/`; VALIDATION doc for the run; +capability map per N.5 — the closes-ledger items flip with citations; ROADMAP: R-21 physical +closure noted, rehearsal item collapsed; **R-3's onboarding runbook drafted from this transcript** +(the steps you just walked ARE the friend-alpha script); the S6 friction list routed into R-3/R-11. + +--- + +## Abort & recovery + +- ISO won't boot on the AMI firmware (F1 not closed): fall back to the stock-ISO + host-install + path — the run still completes as a rehearsal of everything from S5 onward, and slice B gets + its finding instead of its closure. Partial-pass, not failure. +- Anything stuck in pairing/day-0: the bootstrap retries by design; diagnose via journal; the + hub's Unclaimed list + events show which leg stalled. +- Nothing to restore at any point — RESET was the plan. \ No newline at end of file