runbook

documentation/runbooks/RUNBOOK-rehersal-v3.md

@@ -0,0 +1,94 @@
+# RUNBOOK v3 — The rehearsal: RESET → generic ISO → bind → customer zero, round two
+
+> **What this is:** the complete, final product flow executed end-to-end on real hardware for the
+> first time — no per-customer ISO, no manual steps that a future operator wouldn't also do.
+> One supervised afternoon closes every "staged for the rehearsal" item in the tree.
+>
+> **The closes-ledger (what this run flips, collect evidence per item):**
+> 1. **F1 physical** — the mkimage loader boots the real AMI firmware (slice B's bet)
+> 2. **Slice C physical** — generic secret-free ISO → register → bind → day-0 on metal
+> 3. **RESET proven-live, full surface** — operator click + Hetzner sub-account deprovision +
+>    PBS teardown + escrow-ack, twice (drill + demo)
+> 4. **S6b-obsolete live** — no manual host deletion anywhere in the flow
+> 5. **F5 auto-sizing live** — the guest gets ~12 GB on the 16 GB host, not the golden 2 GB
+> 6. **v0.138.0 awaiting-card live render** — the escrow wizard's waiting state, on a real ceremony
+> 7. Fresh-generation claim + re-provisioned offsite from a RESET customer (the re-onboarding
+>    story, whole)
+>
+> **Point of no return:** booting the install-armed stick (the auto-installer needs no confirm).
+> Recovery unchanged: stock ISO + `felhom-host-install.sh` — and since RESET is the plan, there
+> is nothing to restore. Demo-felhom.eu is down for the window (~2 h).
+
+---
+
+## Pre-phase (days before, independent) — drill teardown = the v0.61.0 gap closure
+
+**P1.** Hosts page: delete `demo-vm-felhom-4846bc` (escrow demotes — v0.60.1) and the ghost
+`demo-vm-felhom-2f4b00` stale record. VM 311 can be stopped (it dies with the wipe anyway).
+**P2.** **RESET `demo-vm-felhom`** — the live operator click (v0.61.0's first): typed confirm +
+escrow ack; watch the journal legs: PBS namespace teardown (tenantsync), **Hetzner sub-account
+delete — its first real firing; verify via the Hetzner side that the sub-account is gone**, claim
+reset, DB purge last. Evidence: the per-leg journal + events. If the Hetzner leg misbehaves:
+that's exactly why it runs here first — journal, report, fix before the main run.
+
+---
+
+## Phase 0 — gates (short now; most of v2.1's gates are proven machinery)
+
+**G1 — RESET `demo-felhom`:** delete host `demo-felhom-01` first (ruling 3; escrow demotes),
+then RESET with the ack (this destroys the retained keys + the current repo's real snapshots —
+ruled expendable, stated knowingly). Survives: identity, domain, email, CF tunnel + API tokens,
+offsite/DR config. Second live firing of the full external teardown.
+**G2 — the ISO:** `build-felhom-iso.sh --pairing --loader mkimage --profile n100` — generic,
+secret-free (grep-assert per slice C), serial filter `QDF922W009654S30EX` from the July harvest.
+The external HDD and scratch1 stay plugged; the filter is the guard; the boot stick itself is
+serial-distinct by construction. Note for later: THIS artifact is reusable — no per-run builds
+ever again.
+**G3 — media + firmware prep list** (one BIOS visit at S2 covers all three): USB stick via
+dd/Rufus-DD; the BIOS items queued: boot order, **Secure Boot OFF** (the mkimage contract —
+record prior state), **State-After-G3 → Power On** (the standing sidebar item, finally).
+**G4 — window:** no in-flight CC work; ~2 h budget; second stick with the stock ISO as fallback.
+
+---
+
+## Phase 1 — the run
+
+**S1.** Stop apps, shut down guests + host. Hub shows DOWN — expected.
+**S2.** The one BIOS visit: boot order → USB; Secure Boot → OFF; State-After-G3 → Power On.
+Photos.
+**S3. Boot — the F1 moment.** The mkimage GRUB must load where stock GRUB threw
+`relocation 0x0`. Expected: straight into the automated installer; the complete prior install is
+simply overwritten (slice-B learning — no LVM abort on a whole disk); the serial filter selects
+the internal SSD only. Record wall-clock.
+**S4.** Unattended install (~3 min) → reboot → first boot → pairing mode → the box appears under
+**Unclaimed appliances** at the hub (uuid/MACs per the July DMI harvest; SSH fingerprints shown).
+**S5. The Bind click** — to `demo-felhom`. Then watch the events compose (the four-minute
+sequence from the nested run, now on metal): credential delivered → day-0 → **guest provisioned
+with ~12 GB RAM (F5 check — record the pct config)** → controller starts → fresh-generation claim
+code lands in the inbox → floor train lifts the controller to current.
+**S6. Customer zero, round two** (the friction lens ON — notes verbatim, they are R-3's text):
+claim with the fresh code → set password → **escrow ceremony** — after the wizard, the remote
+page must show the **v0.138.0 awaiting card** ("megerősítésre vár, legfeljebb 15 perc") instead
+of the old yellow banner, then flip on the confirming ACK (its first live render — screenshot
+both states) → recovery code stored → storage: recommission or reformat the external HDD +
+scratch1 (run-time choice) → deploy one app → enable offsite (observe: does the surviving
+offsite config re-provision a fresh sub-account automatically, or on the first save? — record
+which; either is fine, the answer goes in the RESET docs).
+**S7.** "Távoli mentés most" → the first snapshot of the reborn box lands in a brand-new repo
+under a freshly escrowed key. No orphan card expected (the repo is new by construction) — if one
+appears, that's a finding, capture it.
+**S8. Wrap (CC post-run):** evidence to `180:~/n100-rehearsal/`; VALIDATION doc for the run;
+capability map per N.5 — the closes-ledger items flip with citations; ROADMAP: R-21 physical
+closure noted, rehearsal item collapsed; **R-3's onboarding runbook drafted from this transcript**
+(the steps you just walked ARE the friend-alpha script); the S6 friction list routed into R-3/R-11.
+
+---
+
+## Abort & recovery
+
+- ISO won't boot on the AMI firmware (F1 not closed): fall back to the stock-ISO + host-install
+  path — the run still completes as a rehearsal of everything from S5 onward, and slice B gets
+  its finding instead of its closure. Partial-pass, not failure.
+- Anything stuck in pairing/day-0: the bootstrap retries by design; diagnose via journal; the
+  hub's Unclaimed list + events show which leg stalled.
+- Nothing to restore at any point — RESET was the plan.