123 lines
8.8 KiB
Markdown
123 lines
8.8 KiB
Markdown
# RUNBOOK v3 — The rehearsal: RESET → generic ISO → bind → customer zero, round two
|
||
|
||
> **What this is:** the complete, final product flow executed end-to-end on real hardware for the
|
||
> first time — no per-customer ISO, no manual steps that a future operator wouldn't also do.
|
||
> One supervised afternoon closes every "staged for the rehearsal" item in the tree.
|
||
> *(v3.1, 2026-07-17: S5 immediacy proof + S6 resize touchpoint + S7/P2 offsite-gauge
|
||
> observations folded in after hub v0.63–v0.65 + the agent v0.90.0 train shipped; ledger 8–9.)*
|
||
>
|
||
> **The closes-ledger (what this run flips, collect evidence per item):**
|
||
> 1. **F1 physical** — the mkimage loader boots the real AMI firmware (slice B's bet)
|
||
> 2. **Slice C physical** — generic secret-free ISO → register → bind → day-0 on metal
|
||
> 3. **RESET proven-live, full surface** — operator click + Hetzner sub-account deprovision +
|
||
> PBS teardown + escrow-ack, twice (drill + demo)
|
||
> 4. **S6b-obsolete live** — no manual host deletion anywhere in the flow
|
||
> 5. **F5 auto-sizing live** — the guest gets ~12 GB on the 16 GB host, not the golden 2 GB
|
||
> 6. **v0.138.0 awaiting-card live render** — the escrow wizard's waiting state, on a real ceremony
|
||
> 7. Fresh-generation claim + re-provisioned offsite from a RESET customer (the re-onboarding
|
||
> story, whole)
|
||
> 8. **Immediacy real-onboarding proof** — the post-enroll legs compose in seconds on metal:
|
||
> the system-initiated pokes (hub v0.63.0) + the fast-tick over the pre-tunnel window
|
||
> (agent v0.90.0 / R-28's named unfired leg). Evidence: event wall-clocks at S5.
|
||
> 9. **R-24 successful grow** — a customer-driven RAM resize SUCCEEDS through the real UI on a
|
||
> normal-sized box (the v0.90.0 train's honest gap: the 4 GB nested host could only prove
|
||
> the refusals). Evidence at S6.
|
||
>
|
||
> **Point of no return:** booting the install-armed stick (the auto-installer needs no confirm).
|
||
> Recovery unchanged: stock ISO + `felhom-host-install.sh` — and since RESET is the plan, there
|
||
> is nothing to restore. Demo-felhom.eu is down for the window (~2 h).
|
||
|
||
---
|
||
|
||
## Pre-phase (days before, independent) — drill teardown = the v0.61.0 gap closure
|
||
|
||
**P1.** Hosts page: delete `demo-vm-felhom-4846bc` (escrow demotes — v0.60.1) and the ghost
|
||
`demo-vm-felhom-2f4b00` stale record. VM 311 can be stopped (it dies with the wipe anyway).
|
||
**P2.** **RESET `demo-vm-felhom`** — the live operator click (v0.61.0's first): typed confirm +
|
||
escrow ack; watch the journal legs: PBS namespace teardown (tenantsync), **Hetzner sub-account
|
||
delete — its first real firing; verify via the Hetzner side that the sub-account is gone**, claim
|
||
reset, DB purge last. Evidence: the per-leg journal + events. If the Hetzner leg misbehaves:
|
||
that's exactly why it runs here first — journal, report, fix before the main run.
|
||
**Gauge cross-check (v0.64.0's first teardown observation):** after the RESET, the Offsite →
|
||
Restic tab drops the drill customer's row and its 50 GB from Σ(shared quotas); the box-fill
|
||
number itself follows on the next ≤15-min refresh. Screenshot before/after.
|
||
|
||
---
|
||
|
||
## Phase 0 — gates (short now; most of v2.1's gates are proven machinery)
|
||
|
||
**G1 — RESET `demo-felhom`:** delete host `demo-felhom-01` first (ruling 3; escrow demotes),
|
||
then RESET with the ack (this destroys the retained keys + the current repo's real snapshots —
|
||
ruled expendable, stated knowingly). Survives: identity, domain, email, CF tunnel + API tokens,
|
||
offsite/DR config. Second live firing of the full external teardown.
|
||
**G2 — the ISO:** `build-felhom-iso.sh --pairing --loader mkimage --profile n100` — generic,
|
||
secret-free (grep-assert per slice C), serial filter `QDF922W009654S30EX` from the July harvest.
|
||
The external HDD and scratch1 stay plugged; the filter is the guard; the boot stick itself is
|
||
serial-distinct by construction. Note for later: THIS artifact is reusable — no per-run builds
|
||
ever again.
|
||
**G3 — media + firmware prep list** (one BIOS visit at S2 covers all three): USB stick via
|
||
dd/Rufus-DD; the BIOS items queued: boot order, **Secure Boot OFF** (the mkimage contract —
|
||
record prior state), **State-After-G3 → Power On** (the standing sidebar item, finally).
|
||
**G4 — window:** no in-flight CC work; ~2 h budget; second stick with the stock ISO as fallback.
|
||
|
||
---
|
||
|
||
## Phase 1 — the run
|
||
|
||
**S1.** Stop apps, shut down guests + host. Hub shows DOWN — expected.
|
||
**S2.** The one BIOS visit: boot order → USB; Secure Boot → OFF; State-After-G3 → Power On.
|
||
Photos.
|
||
**S3. Boot — the F1 moment.** The mkimage GRUB must load where stock GRUB threw
|
||
`relocation 0x0`. Expected: straight into the automated installer; the complete prior install is
|
||
simply overwritten (slice-B learning — no LVM abort on a whole disk); the serial filter selects
|
||
the internal SSD only. Record wall-clock.
|
||
**S4.** Unattended install (~3 min) → reboot → first boot → pairing mode → the box appears under
|
||
**Unclaimed appliances** at the hub (uuid/MACs per the July DMI harvest; SSH fingerprints shown).
|
||
**S5. The Bind click** — to `demo-felhom`. Then watch the events compose (the four-minute
|
||
sequence from the nested run, now on metal): credential delivered → day-0 → **guest provisioned
|
||
with ~12 GB RAM (F5 check — record the pct config)** → then the **immediacy legs, their first
|
||
live firing (ledger 8 — record the event wall-clocks, don't assume):** the agent enrolls → the
|
||
fresh-generation claim code re-issue fires (the re-enroll intent bump) → WG registers → the
|
||
**pre-tunnel window rides the fast-tick (~30 s pulses — the leg no hub poke can reach)** → tunnel
|
||
up → the **auto-provision poke lands the PBS-DR descriptor in seconds** → controller starts →
|
||
claim code lands in the inbox → floor train lifts the controller to current. **Expected: seconds
|
||
between the post-enroll legs; any ~15-minute stall between two legs = a finding, capture which
|
||
leg and the timestamps** (that IS the evidence, pass or fail).
|
||
**S6. Customer zero, round two** (the friction lens ON — notes verbatim, they are R-3's text):
|
||
claim with the fresh code → set password → **escrow ceremony** — after the wizard, the remote
|
||
page must show the **v0.138.0 awaiting card** ("megerősítésre vár, legfeljebb 15 perc") instead
|
||
of the old yellow banner, then flip on the confirming ACK (its first live render — screenshot
|
||
both states) → recovery code stored → storage: recommission or reformat the external HDD +
|
||
scratch1 (run-time choice) → **Rendszer page, the memory card (R-24, ledger 9):** it shows the
|
||
F5 ~12 GB (cross-check with the S5 pct config); customer-drive a **grow** (e.g. +1024 MB) —
|
||
success flash, the new total visible, `/proc/meminfo` ripple on the deploy page's memory math;
|
||
then shrink back to the F5 value (the shrink-warning confirm renders on the way). The nested demo
|
||
proved the refusals; THIS proves the apply → deploy one app → enable offsite (observe: does the
|
||
surviving offsite config re-provision a fresh sub-account automatically, or on the first save? —
|
||
record which; either is fine, the answer goes in the RESET docs).
|
||
**S7.** "Távoli mentés most" → the first snapshot of the reborn box lands in a brand-new repo
|
||
under a freshly escrowed key. No orphan card expected (the repo is new by construction) — if one
|
||
appears, that's a finding, capture it. **Operator-side (v0.64/v0.65 gauges):** the Offsite →
|
||
Restic tab shows demo-felhom's row with the fresh repo's first bytes within the next
|
||
report/refresh cycle (≤15 min — the box-fill gauge follows on its own throttle); the PBS DR
|
||
gauge stays nominal (the reborn box's first PBS backup runs on schedule, likely outside this
|
||
window — that's fine, note it, don't wait for it). Screenshot both gauges.
|
||
**S8. Wrap (CC post-run):** evidence to `180:~/n100-rehearsal/`; VALIDATION doc for the run;
|
||
capability map per N.5 — the closes-ledger items flip with citations (8 flips the immediacy row's
|
||
"real-onboarding proof pending" note; 9 completes R-24's live story); ROADMAP: R-21 physical
|
||
closure noted, rehearsal item collapsed; **R-3's onboarding runbook drafted from this transcript**
|
||
(the steps you just walked ARE the friend-alpha script); the S6 friction list routed into R-3/R-11.
|
||
|
||
---
|
||
|
||
## Abort & recovery
|
||
|
||
- ISO won't boot on the AMI firmware (F1 not closed): fall back to the stock-ISO + host-install
|
||
path — the run still completes as a rehearsal of everything from S5 onward, and slice B gets
|
||
its finding instead of its closure. Partial-pass, not failure.
|
||
- Anything stuck in pairing/day-0: the bootstrap retries by design; diagnose via journal; the
|
||
hub's Unclaimed list + events show which leg stalled.
|
||
- An immediacy leg stalls (ledger 8's fail-shape): do NOT wait it out silently — note the leg +
|
||
timestamps, then let the normal cycle carry it (everything converges within one agent tick by
|
||
construction); the run continues, the finding is the deliverable.
|
||
- Nothing to restore at any point — RESET was the plan. |