felhom.eu/documentation/runbooks/RUNBOOK-onboarding-draft-v4.md

# DRAFT R-3 — Friend-alpha onboarding runbook (operator)

> **Status: DRAFT, v4 — refreshed 2026-07-22.** Generalized from
> `runbooks/RUNBOOK-rehearsal-v3.md` (S1–S8) and the shipped claim/escrow/self-bind flows, refined
> against the N100 rehearsal that executed it (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`),
> and now carrying a **second full onboarding datapoint on virgin hardware** — the HP t740
> (`demo-hp`, 2026-07-21; `operations/nodes.md`, `CONTEXT.md`). This refresh deleted the R-36 and
> R-39 interim workarounds (both shipped and proven) and updated the freemail/loader/golden notes.
> Every step here has been walked on real hardware **except C7**, the customer-performed restore.
>
> **It graduates to `RUNBOOK-onboarding.md` when C7 has evidence** — that step is simultaneously the
> last unwalked step of this script and the capability map's last MISSING customer-facing row, so
> one execution closes both. Until then this stays a DRAFT.
>
> Scope: **appliance hardware only** (BYO track is R-1/Peti, explicitly out of alpha scope).
> One step below still carries an **interim workaround for a known defect** (R-31) — it is marked
> inline and should be deleted from this runbook the moment that item ships.

## A. Before the visit (operator, ~30 min, all remote-doable)

1. **Hub:** create the customer (name, e-mail, domain `<név>.felhom.eu`) — verify the CF tunnel +
   zone token provisioning completes. Confirm the Day-0 manifest is current (agent, golden,
   MinAgent) and the managed-update floor is at the intended release — a stale golden is fine
   (the floor lifts the box post-install), a stale FLOOR is not. *The current golden (0.153.0)
   carries all four infra images, and the floor-lift is now proven on TWO shapes:* the rehearsal
   (0.143.0 → 0.145.0 in **5 s**, unattended — `tests/VALIDATION-n100-rehearsal-2026-07-18.md`)
   AND a virgin board on first contact (0.153.0 → 0.156.0 during day-0, HP t740, 2026-07-21 —
   `CONTEXT.md`). Golden freshness before the first real tester is the standing rule in **F** —
   apply it there, don't re-derive it here.

   **Verify the self-bind link went out.** Since hub v0.67.0 (R-36) the link is **auto-minted at
   customer creation and at RESET completion**, so the console banner's „e-mailben kapott link" is
   already true by construction; "Send self-bind link" on the Setup tab is the manual re-mint if
   the customer lost it or the 7-day token expired. See B3.
2. **Hardware:** the standard appliance + its external HDD. One-time bench prep list:
   memtest pass, SMART clean, BIOS queue for the single visit in B2.
3. **Install media:** the **reusable generic pairing ISO** (secret-free, slice C). Only rebuild it
   when `scripts/` ships a new version — never per-customer. Flash via dd/Rufus-DD.
   **Console credential for bench work (R-61, since v1.24.0):** every build emits the baked root
   password into the 0600 sibling **`<iso>.rootpw.txt`** next to the ISO in the build output — that
   file is how you log into the console of any box installed from that build (bench checks, dead-
   network diagnosis). Operator-only; never commit or paste it anywhere; the G1 break-glass vault
   remains the mechanism for a *lost* password on an enrolled box.
   Loader per firmware — record the board model + loader choice in the customer record; full
   fleet inventory in `operations/nodes.md`:

   | Board / firmware | Loader | Secure Boot |
   |---|---|---|
   | N100 (AMI AN3PLUS) | `mkimage` | **OFF** — that firmware can't relocate the signed GRUB (F1, `tests/VALIDATION-n100-baremetal-2026-07-16.md`) |
   | **HP t740 (AMI M42)** | **`shim`** | **ENABLED** (proven 2026-07-21, `operations/nodes.md`) |

   **SB-off is a per-firmware workaround, not a Felhom requirement** — the shim loader boots with
   Secure Boot enforcing on firmware that handles it (the HP proved it).
4. **Paper:** the tester one-pager (R-11, printed or PDF), the tester agreement
   (PETI-agreement sibling), and the recovery-code envelope step rehearsed in your head — the
   escrow ceremony is the one moment that must not be rushed.
5. **Agent train ≥ v0.93.0 on the customer's box before the FIRST real tester's escrow ceremony.**
   v0.93.0 removes the four hyphenated EFF words from the recovery-code generator
   (`felhom-agent/CHANGELOG.md` v0.93.0, 2026-07-21) — ceremonies run **box-side**, so the fix
   protects a ceremony only once the train is **BUILT + DEPLOYED to that box**; source-only does
   not count. Already-issued codes stay valid.
   **DISCHARGED 2026-07-22** — agent 0.93.0 published (sha `a68b2ff7…`), Day-0-manifest-vouched,
   and deployed to both fleet boxes (`demo-felhom-8363b5` + `demo-hp-bb76ea`), clean-restart
   verified; any future manifest-driven install also lands ≥ 0.93.0. Record:
   `pilot/RUNBOOK-publish-agent-0.93-2026-07-22.md`.

## B. Install + bind (~15 min wall-clock)

1. Boot the stick. ONE BIOS visit: boot order → USB, Secure Boot per loader, **restore-after-AC-loss
   → Power On** (a power blip must never strand a household).
2. Unattended install (~3 min) → reboot → pairing mode: the console shows the Hungarian
   **pairing-code banner**; the box appears under **Unclaimed appliances** at the hub
   (verify uuid/MAC/SSH fingerprints match the physical box before binding — never bind a
   fingerprint you didn't read off the console).

   > **Multi-NIC gotcha: verify the CABLED port got the DHCP lease during the install.** On the
   > HP t740 the cable was on the 4-port expansion card, which gets **no lease** — and instead of
   > aborting, the installer baked its `192.168.100.2` fallback as a **static** config and
   > completed: a box that looked installed and could never call home
   > (`operations/nodes.md` — the NIC map and the trap). **Since ISO v1.24.0 the box handles this
   > itself (R-59+R-60):** if the hub is unreachable on first boot it sweeps every NIC (bounded
   > DHCP + hub probe) and keeps the first that works — a cable in the wrong port just costs the
   > sweep a minute — and when nothing works it paints a legible Hungarian screen with the NIC
   > table (név/MAC/kábel/sebesség) and retries every minute. Nested-drill proven
   > (`audits/SPIKE-firstboot-nic-sweep-2026-07-22.md`); on a pre-v1.24.0 stick the old manual
   > check still applies: confirm at the installer's network step that the leasing interface is
   > the cabled one.
3. **Bind** to the customer — **the rehearsal used the SELF-BIND path, and it is now the default for
   this runbook.** The link is already in the customer's inbox (auto-minted at creation, A1); the
   customer opens it and completes the public two-factor `/bind/<token>` page (console pairing code
   + retrieval passphrase). The 2026-07-18 run bound in **60 s from mint to consume, with zero failed
   attempts**, and the credential was delivered **26 s later** with no operator action. **Second
   datapoint, virgin hardware:** the 2026-07-21 HP t740 onboarding walked the full chain — register
   → pairing banner → unclaimed list → self-bind (code + passphrase) → day-0 → dashboard — on
   hardware the system had never seen (`CONTEXT.md` 2026-07-21 entry, `operations/nodes.md`).

   > **The self-bind link exists BEFORE the customer sees the console.** The console banner tells
   > them to use „az e-mailben kapott link" — since hub v0.67.0 (R-36) the link is auto-minted at
   > customer creation and RESET completion, so the banner is true by construction (in the
   > pre-auto-mint rehearsal the box sat in pairing mode for **~11.7 minutes** waiting on the
   > operator, printing failures the whole time — R-33). Verify the mint in A1; the token is valid
   > for **7 days**.

   Operator-bind on the Hosts page remains the fallback when the customer is not present or the
   token has expired.
4. Watch the event sequence compose (expected: **seconds** between post-enroll legs — enroll →
   claim-code issue → WG → fast-tick window → tunnel → PBS-DR descriptor poke → controller up →
   claim e-mail lands → floor lift to current). **Any ~15-minute stall between two legs is a
   finding** — capture which leg + wall-clocks.

   **Reference wall-clocks — measured on the N100, 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`).
   Compare against these; a leg that takes minutes where this table says seconds is the finding:

   | Leg | Expect |
   |---|---|
   | Blank-metal boot → registered as an unclaimed appliance | **< 1 s** (same second) |
   | Bind → credential delivered | **26 s** |
   | Bind → agent enrolled + WG tunnel up | **56 s** |
   | Bind → controller running the CURRENT version | **2 min 44 s** |
   | Credential delivered → day-0 provision SUCCESS | **2 min 11 s** |
   | Managed-floor lift (golden version → current) | **5 s**, unattended |
   | Drive enrol: destructive-confirm → usable (incl. mkfs) | **~30 s** |
   | Escrow ceremony started → offsite runs enabled | **12 s** |

   The **PBS-DR descriptor** leg no longer needs special distrust: the DR tier now self-detects a
   dead credential (a loud `auth_failed`) and self-heals via damped re-issue, and the hub gauge is
   trustworthy (**R-39 CLOSED, proven live 2026-07-21** — hub 0.68.1 + agent 0.91.2, the 13 s
   re-issue chain `applied → auth_failed → applied`; `backlog/ROADMAP.md` R-39).

## C. Customer zero script (the customer drives, operator narrates — ~30 min)

This is the friction lens: the customer's hands on the keyboard, the operator's hands OFF.

1. **Claim:** open the claim e-mail → claim the dashboard → set the dashboard password.
   (R-4 gate: the e-mail must land in inbox, not spam. **gmail PROVEN 2026-07-18** — the rehearsal's
   claim mail was the first sent under the tightened DMARC `p=quarantine` and landed in the **Inbox**.
   **freemail.hu PROVEN 2026-07-21** — an operator test-send was received at a freemail.hu address;
   R-4 is COMPLETE (`backlog/ROADMAP.md` R-4). Keep the habit anyway: if a mail seems missing, have
   them check spam before you conclude it was lost.)
2. **Escrow ceremony:** run the recovery-code wizard → the awaiting card
   („megerősítésre vár, legfeljebb 15 perc") → flips on ACK. The code goes on paper, into the
   envelope, into the customer's drawer — **state plainly: without this code, a dead box means
   the remote backup is unrecoverable; we cannot read it for them.** That sentence is the product.
3. **Storage:** enroll/format the external HDD via the wizard (customer clicks, operator explains
   the destructive-confirm). Expect **~30 s** from the confirm to a usable drive — the wizard runs
   mkfs detached and polls, so a slow format is not a hang.

4. **Offsite:** enable „Távoli mentés". One operator behaviour here is a **workaround for a known
   defect** — follow it until the item ships, and do not improvise:

   > **Click Save exactly ONCE, then wait, then verify.** Provisioning runs synchronously inside the
   > request, so on a slow Hetzner leg the request can return an nginx **504 while succeeding
   > server-side**. A second click races the first attempt. Wait, then confirm on the hub's
   > Offsite → Restic tab rather than believing the browser. (**R-31**, still open)

   The old re-onboarding manual-Save check is gone: since hub v0.67.0 (R-36 SHIPPED,
   `backlog/ROADMAP.md`) an enabled-but-unprovisioned offsite target raises an **amber banner on
   the customer page** naming the state and the fix — trust the banner.

   > **Expect the first run to be clean on a NEW customer.** If an **orphan card** appears
   > („a távoli mentési tároló elárvult"), the repo is not new — that is the R-32 case, which fires
   > when a customer's pool directory outlived a RESET. It is safe (the guard moves the old repo
   > aside, never deletes it) but it means the box is writing into a reused directory. Record it.
5. **First app:** deploy one app the household actually wants (paperless or immich recommended) —
   through the catalog UI, no shortcuts.
6. **Sharing:** enable „Megosztás", set the household SMB password, create one share; open
   `\\NÉV` from their own laptop. (Set expectations per the one-pager: device list may lag on
   some routers; the address always works.)
7. **Protection proof — the customer performs a restore, unassisted** (this step IS the
   capability-map evidence, per customer): drop a file on the share or into the app → „Távoli
   mentés most" → wait for the snapshot → delete the file → the customer restores it via
   **Visszaállítás** with the operator silent. If they can't without help, that's an R-11/UX
   finding, not a customer failure — write it down verbatim.

   > **⚠ THIS IS THE ONE STEP THE REHEARSAL DID NOT EXECUTE.** Every other step in this document was
   > walked on real hardware on 2026-07-18; C7 was not, so it carries **no friction notes and no
   > evidence** — the capability-map row *"a customer (not the operator) performs a restore via UI
   > alone"* is still **MISSING**. This runbook stays a **DRAFT** until C7 has been performed once.
   > It is remote-doable on the reborn demo box (the dashboard is remote) and does not need the
   > hardware in hand.
8. **Alerts sanity:** show them what an alert e-mail looks like (the health-degradation mail);
   confirm their address receives it.

## D. Handover (10 min)

- Walk the one-pager together: what it does, what it doesn't do yet, how to report.
- Reporting channel: **support@felhom.eu** (catch-all already delivers it) + the personal channel
  for the friend circle. „Ha bármi furcsa: e-mail vagy üzenet, screenshot ha van."
- Sign the tester agreement. Leave the one-pager with them.
- Operator record: board model + loader, drive serials, claim date, escrow ACK timestamp,
  friction notes.
- If this relationship ever ends, the levers are **RULED and shipped** (R-25b, hub v0.69.0/v0.70.0,
  TASK-I): full teardown = the guided **DELETE cascade**; identity-preserving re-onboarding =
  **RESET** (the Friday flow). Which lever and why: **§G below**.

## E. Operator post-checks (same day, remote)

1. Hub: customer row green; Offsite → Restic tab shows the first bytes within a report cycle;
   PBS-DR gauge nominal (first PBS run on its own schedule — note, don't wait). The gauge is
   trustworthy: since R-39 closed (hub 0.68.1 + agent 0.91.2, proven live 2026-07-21 —
   `backlog/ROADMAP.md` R-39), a dead DR credential surfaces as a loud `auth_failed` and self-heals
   via damped re-issue; no manual `pvesm status` check is needed.
2. Floor convergence: controller at current, agent at MinAgent+.
3. Route friction notes: UX items → ROADMAP; one-pager corrections → R-11; runbook corrections →
   this doc.

## F. Standing rules

- Never perform steps C1–C7 *for* the customer. The script's value is watching them do it.
- Never handle the recovery code — see it into the envelope, never into your hands or a photo.
- One onboarding per day maximum during alpha — the friction notes are the point, and they
  degrade after the first.
- Post-rehearsal checklist item: rebuild the golden to the current controller before the FIRST
  real tester (fresh boxes shouldn't sit in a stale window longer than needed), and re-verify the
  Day-0 manifest after every controller release during alpha.

## G. Offboarding — which lever to pull (hub v0.69.0, R-25b)

Two levers, and they are not interchangeable:

- **Ügyfél-visszaállítás (RESET)** — *identity-preserving re-onboarding*. Every operational trace
  dies (offsite repository, PBS namespace, DR recipe, one-time password, claim state); the customer
  record, the basic config and the audit history survive. This is Peti's Friday flow: the same
  customer coming back on new hardware. RESET still **refuses while any host row exists** — delete
  the host(s) first, on purpose.
- **Delete customer… (the DELETE cascade)** — *full teardown, the customer is leaving*. One guided
  dialog shows the live inventory, takes **three acknowledgements** (1: the host(s) will be deleted,
  custody demoted not destroyed; 2: the customer will be RESET incl. external teardown and offsite
  data destruction; 3: the record and ALL escrow ciphertext are purged) plus the **typed
  customer-id**, then runs `hosts → RESET → purge` in that order.

Notes for the operator:

- An **ONLINE host refuses the whole cascade** — decommission the box first (a live agent would get
  permanent 401s). The dialog says so and disables the button.
- If a leg fails (a slow Hetzner call), **nothing is left half-forgotten**: the journal is retained,
  the error names the leg, and re-opening the dialog offers **Resume**. Completed legs are no-ops.
- The **audit event stream survives** the delete — the customer_deleted event and the host-deletion
  provenance outlive the record, by design.
- **A deleted customer disappears from the Customers list immediately** (hub v0.70.0). If one lingers
  with a `MANUAL` config badge, it is a **ghost** left by a pre-v0.70.0 delete: the record is gone but
  its report stream survives, which both keeps it on the list and keeps the staleness/offsite checkers
  emailing you about it. Open **Delete customer…** on it and confirm — the dialog labels it as a ghost
  and the cascade clears the residue. (With no config row the Hetzner leg cannot be determined and is
  recorded as `skipped_no_config` — verify the Hetzner side by hand for that customer.)