6.2 KiB
RUNBOOK v3 — The rehearsal: RESET → generic ISO → bind → customer zero, round two
What this is: the complete, final product flow executed end-to-end on real hardware for the first time — no per-customer ISO, no manual steps that a future operator wouldn't also do. One supervised afternoon closes every "staged for the rehearsal" item in the tree.
The closes-ledger (what this run flips, collect evidence per item):
- F1 physical — the mkimage loader boots the real AMI firmware (slice B's bet)
- Slice C physical — generic secret-free ISO → register → bind → day-0 on metal
- RESET proven-live, full surface — operator click + Hetzner sub-account deprovision + PBS teardown + escrow-ack, twice (drill + demo)
- S6b-obsolete live — no manual host deletion anywhere in the flow
- F5 auto-sizing live — the guest gets ~12 GB on the 16 GB host, not the golden 2 GB
- v0.138.0 awaiting-card live render — the escrow wizard's waiting state, on a real ceremony
- Fresh-generation claim + re-provisioned offsite from a RESET customer (the re-onboarding story, whole)
Point of no return: booting the install-armed stick (the auto-installer needs no confirm). Recovery unchanged: stock ISO +
felhom-host-install.sh— and since RESET is the plan, there is nothing to restore. Demo-felhom.eu is down for the window (~2 h).
Pre-phase (days before, independent) — drill teardown = the v0.61.0 gap closure
P1. Hosts page: delete demo-vm-felhom-4846bc (escrow demotes — v0.60.1) and the ghost
demo-vm-felhom-2f4b00 stale record. VM 311 can be stopped (it dies with the wipe anyway).
P2. RESET demo-vm-felhom — the live operator click (v0.61.0's first): typed confirm +
escrow ack; watch the journal legs: PBS namespace teardown (tenantsync), Hetzner sub-account
delete — its first real firing; verify via the Hetzner side that the sub-account is gone, claim
reset, DB purge last. Evidence: the per-leg journal + events. If the Hetzner leg misbehaves:
that's exactly why it runs here first — journal, report, fix before the main run.
Phase 0 — gates (short now; most of v2.1's gates are proven machinery)
G1 — RESET demo-felhom: delete host demo-felhom-01 first (ruling 3; escrow demotes),
then RESET with the ack (this destroys the retained keys + the current repo's real snapshots —
ruled expendable, stated knowingly). Survives: identity, domain, email, CF tunnel + API tokens,
offsite/DR config. Second live firing of the full external teardown.
G2 — the ISO: build-felhom-iso.sh --pairing --loader mkimage --profile n100 — generic,
secret-free (grep-assert per slice C), serial filter QDF922W009654S30EX from the July harvest.
The external HDD and scratch1 stay plugged; the filter is the guard; the boot stick itself is
serial-distinct by construction. Note for later: THIS artifact is reusable — no per-run builds
ever again.
G3 — media + firmware prep list (one BIOS visit at S2 covers all three): USB stick via
dd/Rufus-DD; the BIOS items queued: boot order, Secure Boot OFF (the mkimage contract —
record prior state), State-After-G3 → Power On (the standing sidebar item, finally).
G4 — window: no in-flight CC work; ~2 h budget; second stick with the stock ISO as fallback.
Phase 1 — the run
S1. Stop apps, shut down guests + host. Hub shows DOWN — expected.
S2. The one BIOS visit: boot order → USB; Secure Boot → OFF; State-After-G3 → Power On.
Photos.
S3. Boot — the F1 moment. The mkimage GRUB must load where stock GRUB threw
relocation 0x0. Expected: straight into the automated installer; the complete prior install is
simply overwritten (slice-B learning — no LVM abort on a whole disk); the serial filter selects
the internal SSD only. Record wall-clock.
S4. Unattended install (~3 min) → reboot → first boot → pairing mode → the box appears under
Unclaimed appliances at the hub (uuid/MACs per the July DMI harvest; SSH fingerprints shown).
S5. The Bind click — to demo-felhom. Then watch the events compose (the four-minute
sequence from the nested run, now on metal): credential delivered → day-0 → guest provisioned
with ~12 GB RAM (F5 check — record the pct config) → controller starts → fresh-generation claim
code lands in the inbox → floor train lifts the controller to current.
S6. Customer zero, round two (the friction lens ON — notes verbatim, they are R-3's text):
claim with the fresh code → set password → escrow ceremony — after the wizard, the remote
page must show the v0.138.0 awaiting card ("megerősítésre vár, legfeljebb 15 perc") instead
of the old yellow banner, then flip on the confirming ACK (its first live render — screenshot
both states) → recovery code stored → storage: recommission or reformat the external HDD +
scratch1 (run-time choice) → deploy one app → enable offsite (observe: does the surviving
offsite config re-provision a fresh sub-account automatically, or on the first save? — record
which; either is fine, the answer goes in the RESET docs).
S7. "Távoli mentés most" → the first snapshot of the reborn box lands in a brand-new repo
under a freshly escrowed key. No orphan card expected (the repo is new by construction) — if one
appears, that's a finding, capture it.
S8. Wrap (CC post-run): evidence to 180:~/n100-rehearsal/; VALIDATION doc for the run;
capability map per N.5 — the closes-ledger items flip with citations; ROADMAP: R-21 physical
closure noted, rehearsal item collapsed; R-3's onboarding runbook drafted from this transcript
(the steps you just walked ARE the friend-alpha script); the S6 friction list routed into R-3/R-11.
Abort & recovery
- ISO won't boot on the AMI firmware (F1 not closed): fall back to the stock-ISO + host-install path — the run still completes as a rehearsal of everything from S5 onward, and slice B gets its finding instead of its closure. Partial-pass, not failure.
- Anything stuck in pairing/day-0: the bootstrap retries by design; diagnose via journal; the hub's Unclaimed list + events show which leg stalled.
- Nothing to restore at any point — RESET was the plan.