8.8 KiB
RUNBOOK v3 — The rehearsal: RESET → generic ISO → bind → customer zero, round two
What this is: the complete, final product flow executed end-to-end on real hardware for the first time — no per-customer ISO, no manual steps that a future operator wouldn't also do. One supervised afternoon closes every "staged for the rehearsal" item in the tree. (v3.1, 2026-07-17: S5 immediacy proof + S6 resize touchpoint + S7/P2 offsite-gauge observations folded in after hub v0.63–v0.65 + the agent v0.90.0 train shipped; ledger 8–9.)
The closes-ledger (what this run flips, collect evidence per item):
- F1 physical — the mkimage loader boots the real AMI firmware (slice B's bet)
- Slice C physical — generic secret-free ISO → register → bind → day-0 on metal
- RESET proven-live, full surface — operator click + Hetzner sub-account deprovision + PBS teardown + escrow-ack, twice (drill + demo)
- S6b-obsolete live — no manual host deletion anywhere in the flow
- F5 auto-sizing live — the guest gets ~12 GB on the 16 GB host, not the golden 2 GB
- v0.138.0 awaiting-card live render — the escrow wizard's waiting state, on a real ceremony
- Fresh-generation claim + re-provisioned offsite from a RESET customer (the re-onboarding story, whole)
- Immediacy real-onboarding proof — the post-enroll legs compose in seconds on metal: the system-initiated pokes (hub v0.63.0) + the fast-tick over the pre-tunnel window (agent v0.90.0 / R-28's named unfired leg). Evidence: event wall-clocks at S5.
- R-24 successful grow — a customer-driven RAM resize SUCCEEDS through the real UI on a normal-sized box (the v0.90.0 train's honest gap: the 4 GB nested host could only prove the refusals). Evidence at S6.
Point of no return: booting the install-armed stick (the auto-installer needs no confirm). Recovery unchanged: stock ISO +
felhom-host-install.sh— and since RESET is the plan, there is nothing to restore. Demo-felhom.eu is down for the window (~2 h).
Pre-phase (days before, independent) — drill teardown = the v0.61.0 gap closure
P1. Hosts page: delete demo-vm-felhom-4846bc (escrow demotes — v0.60.1) and the ghost
demo-vm-felhom-2f4b00 stale record. VM 311 can be stopped (it dies with the wipe anyway).
P2. RESET demo-vm-felhom — the live operator click (v0.61.0's first): typed confirm +
escrow ack; watch the journal legs: PBS namespace teardown (tenantsync), Hetzner sub-account
delete — its first real firing; verify via the Hetzner side that the sub-account is gone, claim
reset, DB purge last. Evidence: the per-leg journal + events. If the Hetzner leg misbehaves:
that's exactly why it runs here first — journal, report, fix before the main run.
Gauge cross-check (v0.64.0's first teardown observation): after the RESET, the Offsite →
Restic tab drops the drill customer's row and its 50 GB from Σ(shared quotas); the box-fill
number itself follows on the next ≤15-min refresh. Screenshot before/after.
Phase 0 — gates (short now; most of v2.1's gates are proven machinery)
G1 — RESET demo-felhom: delete host demo-felhom-01 first (ruling 3; escrow demotes),
then RESET with the ack (this destroys the retained keys + the current repo's real snapshots —
ruled expendable, stated knowingly). Survives: identity, domain, email, CF tunnel + API tokens,
offsite/DR config. Second live firing of the full external teardown.
G2 — the ISO: build-felhom-iso.sh --pairing --loader mkimage --profile n100 — generic,
secret-free (grep-assert per slice C), serial filter QDF922W009654S30EX from the July harvest.
The external HDD and scratch1 stay plugged; the filter is the guard; the boot stick itself is
serial-distinct by construction. Note for later: THIS artifact is reusable — no per-run builds
ever again.
G3 — media + firmware prep list (one BIOS visit at S2 covers all three): USB stick via
dd/Rufus-DD; the BIOS items queued: boot order, Secure Boot OFF (the mkimage contract —
record prior state), State-After-G3 → Power On (the standing sidebar item, finally).
G4 — window: no in-flight CC work; ~2 h budget; second stick with the stock ISO as fallback.
Phase 1 — the run
S1. Stop apps, shut down guests + host. Hub shows DOWN — expected.
S2. The one BIOS visit: boot order → USB; Secure Boot → OFF; State-After-G3 → Power On.
Photos.
S3. Boot — the F1 moment. The mkimage GRUB must load where stock GRUB threw
relocation 0x0. Expected: straight into the automated installer; the complete prior install is
simply overwritten (slice-B learning — no LVM abort on a whole disk); the serial filter selects
the internal SSD only. Record wall-clock.
S4. Unattended install (~3 min) → reboot → first boot → pairing mode → the box appears under
Unclaimed appliances at the hub (uuid/MACs per the July DMI harvest; SSH fingerprints shown).
S5. The Bind click — to demo-felhom. Then watch the events compose (the four-minute
sequence from the nested run, now on metal): credential delivered → day-0 → guest provisioned
with ~12 GB RAM (F5 check — record the pct config) → then the immediacy legs, their first
live firing (ledger 8 — record the event wall-clocks, don't assume): the agent enrolls → the
fresh-generation claim code re-issue fires (the re-enroll intent bump) → WG registers → the
pre-tunnel window rides the fast-tick (~30 s pulses — the leg no hub poke can reach) → tunnel
up → the auto-provision poke lands the PBS-DR descriptor in seconds → controller starts →
claim code lands in the inbox → floor train lifts the controller to current. Expected: seconds
between the post-enroll legs; any ~15-minute stall between two legs = a finding, capture which
leg and the timestamps (that IS the evidence, pass or fail).
S6. Customer zero, round two (the friction lens ON — notes verbatim, they are R-3's text):
claim with the fresh code → set password → escrow ceremony — after the wizard, the remote
page must show the v0.138.0 awaiting card ("megerősítésre vár, legfeljebb 15 perc") instead
of the old yellow banner, then flip on the confirming ACK (its first live render — screenshot
both states) → recovery code stored → storage: recommission or reformat the external HDD +
scratch1 (run-time choice) → Rendszer page, the memory card (R-24, ledger 9): it shows the
F5 ~12 GB (cross-check with the S5 pct config); customer-drive a grow (e.g. +1024 MB) —
success flash, the new total visible, /proc/meminfo ripple on the deploy page's memory math;
then shrink back to the F5 value (the shrink-warning confirm renders on the way). The nested demo
proved the refusals; THIS proves the apply → deploy one app → enable offsite (observe: does the
surviving offsite config re-provision a fresh sub-account automatically, or on the first save? —
record which; either is fine, the answer goes in the RESET docs).
S7. "Távoli mentés most" → the first snapshot of the reborn box lands in a brand-new repo
under a freshly escrowed key. No orphan card expected (the repo is new by construction) — if one
appears, that's a finding, capture it. Operator-side (v0.64/v0.65 gauges): the Offsite →
Restic tab shows demo-felhom's row with the fresh repo's first bytes within the next
report/refresh cycle (≤15 min — the box-fill gauge follows on its own throttle); the PBS DR
gauge stays nominal (the reborn box's first PBS backup runs on schedule, likely outside this
window — that's fine, note it, don't wait for it). Screenshot both gauges.
S8. Wrap (CC post-run): evidence to 180:~/n100-rehearsal/; VALIDATION doc for the run;
capability map per N.5 — the closes-ledger items flip with citations (8 flips the immediacy row's
"real-onboarding proof pending" note; 9 completes R-24's live story); ROADMAP: R-21 physical
closure noted, rehearsal item collapsed; R-3's onboarding runbook drafted from this transcript
(the steps you just walked ARE the friend-alpha script); the S6 friction list routed into R-3/R-11.
Abort & recovery
- ISO won't boot on the AMI firmware (F1 not closed): fall back to the stock-ISO + host-install path — the run still completes as a rehearsal of everything from S5 onward, and slice B gets its finding instead of its closure. Partial-pass, not failure.
- Anything stuck in pairing/day-0: the bootstrap retries by design; diagnose via journal; the hub's Unclaimed list + events show which leg stalled.
- An immediacy leg stalls (ledger 8's fail-shape): do NOT wait it out silently — note the leg + timestamps, then let the normal cycle carry it (everything converges within one agent tick by construction); the run continues, the finding is the deliverable.
- Nothing to restore at any point — RESET was the plan.