felhom.eu/documentation/runbooks/RUNBOOK-onboarding-draft-v3.md

DRAFT R-3 — Friend-alpha onboarding runbook (operator)

Status: DRAFT, post-rehearsal — all [REFINE] slots filled 2026-07-18. Generalized from runbooks/RUNBOOK-rehearsal-v3.md (S1–S8) and the shipped claim/escrow/self-bind flows, then refined against the N100 rehearsal that executed it (tests/VALIDATION-n100-rehearsal-2026-07-18.md). Every step here has been walked on real hardware except C7, the customer-performed restore.

It graduates to RUNBOOK-onboarding.md when C7 has evidence — that step is simultaneously the last unwalked step of this script and the capability map's last MISSING customer-facing row, so one execution closes both. Until then this stays a DRAFT.

Scope: appliance hardware only (BYO track is R-1/Peti, explicitly out of alpha scope). Three steps below carry interim workarounds for known defects (R-31, R-36, R-39) — they are marked inline and should be deleted from this runbook the moment those items ship.

A. Before the visit (operator, ~30 min, all remote-doable)

1. Hub: create the customer (name, e-mail, domain <név>.felhom.eu) — verify the CF tunnel + zone token provisioning completes. Confirm the Day-0 manifest is current (agent, golden, MinAgent) and the managed-update floor is at the intended release — a stale golden is fine (the floor lifts the box post-install), a stale FLOOR is not. The rehearsal proved the lift: a 0.143.0 golden reached 0.145.0 in 5 s, unattended, inside the first minute of controller life. Even so, rebuild the golden before the first real tester — proven-recoverable is not a reason to ship stale.

   Mint the self-bind link now ("Send self-bind link" on the customer's Setup tab). It is valid for 7 days, and doing it here means the console banner's „e-mailben kapott link" is already true when the customer first looks at the screen. See B3.

2. Hardware: the standard appliance + its external HDD. One-time bench prep list: memtest pass, SMART clean, BIOS queue for the single visit in B2.

3. Install media: the reusable generic pairing ISO (secret-free, slice C). Only rebuild it when scripts/ ships a new version — never per-customer. Flash via dd/Rufus-DD. Loader note per firmware: mkimage requires Secure Boot OFF (F1); record the board model + loader choice in the customer record.

4. Paper: the tester one-pager (R-11, printed or PDF), the tester agreement (PETI-agreement sibling), and the recovery-code envelope step rehearsed in your head — the escrow ceremony is the one moment that must not be rushed.

B. Install + bind (~15 min wall-clock)

1. Boot the stick. ONE BIOS visit: boot order → USB, Secure Boot per loader, restore-after-AC-loss → Power On (a power blip must never strand a household).

2. Unattended install (~3 min) → reboot → pairing mode: the console shows the Hungarian pairing-code banner; the box appears under Unclaimed appliances at the hub (verify uuid/MAC/SSH fingerprints match the physical box before binding — never bind a fingerprint you didn't read off the console).

3. Bind to the customer — the rehearsal used the SELF-BIND path, and it is now the default for this runbook. Mint the link with "Send self-bind link" on the customer's Setup tab; the customer opens it and completes the public two-factor /bind/<token> page (console pairing code

   • retrieval passphrase). The 2026-07-18 run bound in 60 s from mint to consume, with zero failed attempts, and the credential was delivered 26 s later with no operator action.

   Send the self-bind link BEFORE the customer sees the console. The console banner tells them to use „az e-mailben kapott link" — if the link has not been minted yet, the box is instructing the customer to look for something that does not exist. In the rehearsal the box sat in pairing mode for ~11.7 minutes waiting on the operator, printing failures the whole time (R-33). Mint it during step A1 if you can; the token is valid for 7 days, so there is no reason to wait. Auto-minting on customer create/RESET is R-36.

   Operator-bind on the Hosts page remains the fallback when the customer is not present or the token has expired.

4. Watch the event sequence compose (expected: seconds between post-enroll legs — enroll → claim-code issue → WG → fast-tick window → tunnel → PBS-DR descriptor poke → controller up → claim e-mail lands → floor lift to current). Any ~15-minute stall between two legs is a finding — capture which leg + wall-clocks.

   Reference wall-clocks — measured on the N100, 2026-07-18 (tests/VALIDATION-n100-rehearsal-2026-07-18.md). Compare against these; a leg that takes minutes where this table says seconds is the finding:

   Leg	Expect
   Blank-metal boot → registered as an unclaimed appliance	< 1 s (same second)
   Bind → credential delivered	26 s
   Bind → agent enrolled + WG tunnel up	56 s
   Bind → controller running the CURRENT version	2 min 44 s
   Credential delivered → day-0 provision SUCCESS	2 min 11 s
   Managed-floor lift (golden version → current)	5 s, unattended
   Drive enrol: destructive-confirm → usable (incl. mkfs)	~30 s
   Escrow ceremony started → offsite runs enabled	12 s

   Note the PBS-DR descriptor leg specifically: in the rehearsal it reported converged while the credential behind it was already stale, leaving the DR tier authenticating 401 with nothing surfacing the fault (R-39). Until that is fixed, treat "descriptor provisioned" as unproven and check pvesm status on the box during step E1.

C. Customer zero script (the customer drives, operator narrates — ~30 min)

This is the friction lens: the customer's hands on the keyboard, the operator's hands OFF.

1. Claim: open the claim e-mail → claim the dashboard → set the dashboard password. (R-4 gate: the e-mail must land in inbox, not spam. gmail PROVEN 2026-07-18 — the rehearsal's claim mail was the first sent under the tightened DMARC p=quarantine and landed in the Inbox. freemail.hu is still untested — if your tester is on freemail.hu, treat step C1 as unproven and have them check spam before you conclude the mail was lost.)

2. Escrow ceremony: run the recovery-code wizard → the awaiting card („megerősítésre vár, legfeljebb 15 perc") → flips on ACK. The code goes on paper, into the envelope, into the customer's drawer — state plainly: without this code, a dead box means the remote backup is unrecoverable; we cannot read it for them. That sentence is the product.

3. Storage: enroll/format the external HDD via the wizard (customer clicks, operator explains the destructive-confirm). Expect ~30 s from the confirm to a usable drive — the wizard runs mkfs detached and polls, so a slow format is not a hang.

4. Offsite: enable „Távoli mentés". Two operator behaviours here are workarounds for known defects — follow them until the items ship, and do not improvise:

   Click Save exactly ONCE, then wait, then verify. Provisioning runs synchronously inside the request, so on a slow Hetzner leg the request can return an nginx 504 while succeeding server-side. A second click races the first attempt. Wait, then confirm on the hub's Offsite → Restic tab rather than believing the browser. (R-31)

   Re-onboarding an existing customer? Check offsite provisioning by hand. After a RESET + re-enroll the hub can sit in "enabled but unprovisioned" and say nothing — provisioning is Save-triggered, and the re-enroll auto-re-issue deliberately skips unprovisioned targets. So an offsite target that looks configured may never have been created. Open the offsite settings and press Save once to force provisioning, then verify. (R-36)

   Expect the first run to be clean on a NEW customer. If an orphan card appears („a távoli mentési tároló elárvult"), the repo is not new — that is the R-32 case, which fires when a customer's pool directory outlived a RESET. It is safe (the guard moves the old repo aside, never deletes it) but it means the box is writing into a reused directory. Record it.

5. First app: deploy one app the household actually wants (paperless or immich recommended) — through the catalog UI, no shortcuts.

6. Sharing: enable „Megosztás", set the household SMB password, create one share; open \\NÉV from their own laptop. (Set expectations per the one-pager: device list may lag on some routers; the address always works.)

7. Protection proof — the customer performs a restore, unassisted (this step IS the capability-map evidence, per customer): drop a file on the share or into the app → „Távoli mentés most" → wait for the snapshot → delete the file → the customer restores it via Visszaállítás with the operator silent. If they can't without help, that's an R-11/UX finding, not a customer failure — write it down verbatim.

   ⚠ THIS IS THE ONE STEP THE REHEARSAL DID NOT EXECUTE. Every other step in this document was walked on real hardware on 2026-07-18; C7 was not, so it carries no friction notes and no evidence — the capability-map row "a customer (not the operator) performs a restore via UI alone" is still MISSING. This runbook stays a DRAFT until C7 has been performed once. It is remote-doable on the reborn demo box (the dashboard is remote) and does not need the hardware in hand.

8. Alerts sanity: show them what an alert e-mail looks like (the health-degradation mail); confirm their address receives it.

D. Handover (10 min)

• Walk the one-pager together: what it does, what it doesn't do yet, how to report.
• Reporting channel: support@felhom.eu (catch-all already delivers it) + the personal channel for the friend circle. „Ha bármi furcsa: e-mail vagy üzenet, screenshot ha van."
• Sign the tester agreement. Leave the one-pager with them.
• Operator record: board model + loader, drive serials, claim date, escrow ACK timestamp, friction notes.

E. Operator post-checks (same day, remote)

1. Hub: customer row green; Offsite → Restic tab shows the first bytes within a report cycle; PBS-DR gauge nominal (first PBS run on its own schedule — note, don't wait).

   Verify the DR tier on the box, not from the hub (R-39, interim). The agent can report the PBS descriptor converged while the credential behind it is stale, and its own verify loop cannot read the credential to notice. Until R-39 ships, SSH to the host and run pvesm status — the felhom-pbs line must be active. A 401 Unauthorized / inactive there means the DR tier is dead despite everything upstream looking green, and it will stay dead silently until someone needs a restore.

2. Floor convergence: controller at current, agent at MinAgent+.

3. Route friction notes: UX items → ROADMAP; one-pager corrections → R-11; runbook corrections → this doc.

F. Standing rules

• Never perform steps C1–C7 for the customer. The script's value is watching them do it.
• Never handle the recovery code — see it into the envelope, never into your hands or a photo.
• One onboarding per day maximum during alpha — the friction notes are the point, and they degrade after the first.
• Post-rehearsal checklist item: rebuild the golden to the current controller before the FIRST real tester (fresh boxes shouldn't sit in a stale window longer than needed), and re-verify the Day-0 manifest after every controller release during alpha.

G. Offboarding — which lever to pull (hub v0.69.0, R-25b)

Two levers, and they are not interchangeable:

• Ügyfél-visszaállítás (RESET) — identity-preserving re-onboarding. Every operational trace dies (offsite repository, PBS namespace, DR recipe, one-time password, claim state); the customer record, the basic config and the audit history survive. This is Peti's Friday flow: the same customer coming back on new hardware. RESET still refuses while any host row exists — delete the host(s) first, on purpose.
• Delete customer… (the DELETE cascade) — full teardown, the customer is leaving. One guided dialog shows the live inventory, takes three acknowledgements (1: the host(s) will be deleted, custody demoted not destroyed; 2: the customer will be RESET incl. external teardown and offsite data destruction; 3: the record and ALL escrow ciphertext are purged) plus the typed customer-id, then runs hosts → RESET → purge in that order.

Notes for the operator:

• An ONLINE host refuses the whole cascade — decommission the box first (a live agent would get permanent 401s). The dialog says so and disables the button.
• If a leg fails (a slow Hetzner call), nothing is left half-forgotten: the journal is retained, the error names the leg, and re-opening the dialog offers Resume. Completed legs are no-ops.
• The audit event stream survives the delete — the customer_deleted event and the host-deletion provenance outlive the record, by design.
• A deleted customer disappears from the Customers list immediately (hub v0.70.0). If one lingers with a MANUAL config badge, it is a ghost left by a pre-v0.70.0 delete: the record is gone but its report stream survives, which both keeps it on the list and keeps the staleness/offsite checkers emailing you about it. Open Delete customer… on it and confirm — the dialog labels it as a ghost and the cascade clears the residue. (With no config row the Hetzner leg cannot be determined and is recorded as skipped_no_config — verify the Hetzner side by hand for that customer.)