Files
felhom.eu/documentation/runbooks/RUNBOOK-onboarding-draft-v3.md
T
admin aa9c08f0dd docs(rehearsal): N100 S8 wrap — VALIDATION, map flips, ROADMAP R-30..R-39, R-3 draft
The 2026-07-18 N100 rehearsal ran the complete final-product flow on real metal
in one pass: RESET -> generic pairing ISO v1.20.0 -> customer self-bind -> day-0
-> managed-floor lift -> escrow ceremony -> offsite snapshots. No code changed;
every finding is recorded and ranked, none fixed.

VALIDATION-n100-rehearsal-2026-07-18.md — run context, a UTC-normalised timeline
built from the hub events stream / hub DB / controller log / bootstrap + agent
journals, per-ledger verdicts for S1-S8 + ledgers 8 and 9, 12 findings, the
not-exercised list, and 7 discrepancies against the brief.

Headline wall-clocks: bind -> credential 26 s; bind -> controller running the
current version 2 min 44 s; managed floor 0.143.0 -> 0.145.0 in 5 s unattended
(initiated_by: auto-floor); escrow ceremony -> offsite enabled 12 s; drive enrol
30.3 s. No post-bind leg stalled, which is the immediacy row's real-onboarding
proof.

Capability map (10 citations added):
- Bare-metal Felhom ISO           PARTIAL     -> PROVEN-LIVE (F1 closed on metal)
- Customer self-bind (slice 1)    IMPLEMENTED -> PROVEN-LIVE (customer_selfbind)
- Guest RAM resize (R-24)         IMPLEMENTED -> PROVEN-LIVE (shrink AND grow)
- Customer RESET                  two real firings + verified external teardown
- Escrow ceremony                 first live wizard firing
- Immediacy row                   "real-onboarding proof pending" cleared
- Publish train                   box-side floor lift proven on a fresh install
- Customer claim                  R-4 gmail half (Inbox under p=quarantine)
- Offsite orphan guard            staged live leg fired on its own
- DR tier by default              candidate PROVEN-LIVE upgrade WITHDRAWN (R-39)
Not flipped, as instructed: customer-performs-restore, BYO, DLNA, multi-user.

ROADMAP — collapsed R-1 (appliance half done, Peti half survives), R-21
(physically closed), R-24, R-27 slice 1, R-4. New ranked items:
  P2-HIGH  R-39 PBS DR applied-but-dead   R-30 liveness from the wait channel
           R-31 async offsite + status    R-32 RESET base-dir purge
           R-33 bootstrap quiet-poll
  P2       R-34 backup lifecycle          R-35 config-apply session survival
           R-36 post-RESET offsite prompt R-27c console-passphrase bind
  P3       R-37 post-RESET health card    R-38 installer GRUB slice
Plus a pre-invite checklist (golden 0.145.x rebuild, freemail.hu, C6, R-11).

R-39 is NEW and was not on the brief: the PBS DR descriptor auto-provisions and
the agent converges state=applied, but pvesm reports 401 Unauthorized/inactive
and a direct probe 401s on every endpoint including /version while WG is healthy.
The hub minted a second token secret two minutes after the agent applied the
first and consumed_at is still NULL; the converged state machine will not
re-apply, and the agent's verify loop cannot read the credential to notice it
(non-root read of a file it writes through a root wrapper). Rank is provisional
pending Viktor.

R-3 draft: all four [REFINE] slots filled, self-bind made the default path with
"send the link BEFORE the customer sees the console", the measured wall-clock
table added, and interim operator workarounds for R-31/R-36/R-39. C6 (renumbered
C7) is marked as the single unexecuted step and keeps the doc a DRAFT.

Evidence bundle: 180:~/n100-rehearsal/ (10 files + MANIFEST.md), collected before
the box was unplugged for travel. Secrets read only to run probes; recorded as
lengths and metadata, never values.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
2026-07-18 19:56:17 +02:00

12 KiB
Raw Blame History

DRAFT R-3 — Friend-alpha onboarding runbook (operator)

Status: DRAFT, post-rehearsal — all [REFINE] slots filled 2026-07-18. Generalized from runbooks/RUNBOOK-rehearsal-v3.md (S1S8) and the shipped claim/escrow/self-bind flows, then refined against the N100 rehearsal that executed it (tests/VALIDATION-n100-rehearsal-2026-07-18.md). Every step here has been walked on real hardware except C7, the customer-performed restore.

It graduates to RUNBOOK-onboarding.md when C7 has evidence — that step is simultaneously the last unwalked step of this script and the capability map's last MISSING customer-facing row, so one execution closes both. Until then this stays a DRAFT.

Scope: appliance hardware only (BYO track is R-1/Peti, explicitly out of alpha scope). Three steps below carry interim workarounds for known defects (R-31, R-36, R-39) — they are marked inline and should be deleted from this runbook the moment those items ship.

A. Before the visit (operator, ~30 min, all remote-doable)

  1. Hub: create the customer (name, e-mail, domain <név>.felhom.eu) — verify the CF tunnel + zone token provisioning completes. Confirm the Day-0 manifest is current (agent, golden, MinAgent) and the managed-update floor is at the intended release — a stale golden is fine (the floor lifts the box post-install), a stale FLOOR is not. The rehearsal proved the lift: a 0.143.0 golden reached 0.145.0 in 5 s, unattended, inside the first minute of controller life. Even so, rebuild the golden before the first real tester — proven-recoverable is not a reason to ship stale.

    Mint the self-bind link now ("Send self-bind link" on the customer's Setup tab). It is valid for 7 days, and doing it here means the console banner's „e-mailben kapott link" is already true when the customer first looks at the screen. See B3.

  2. Hardware: the standard appliance + its external HDD. One-time bench prep list: memtest pass, SMART clean, BIOS queue for the single visit in B2.

  3. Install media: the reusable generic pairing ISO (secret-free, slice C). Only rebuild it when scripts/ ships a new version — never per-customer. Flash via dd/Rufus-DD. Loader note per firmware: mkimage requires Secure Boot OFF (F1); record the board model + loader choice in the customer record.

  4. Paper: the tester one-pager (R-11, printed or PDF), the tester agreement (PETI-agreement sibling), and the recovery-code envelope step rehearsed in your head — the escrow ceremony is the one moment that must not be rushed.

B. Install + bind (~15 min wall-clock)

  1. Boot the stick. ONE BIOS visit: boot order → USB, Secure Boot per loader, restore-after-AC-loss → Power On (a power blip must never strand a household).

  2. Unattended install (~3 min) → reboot → pairing mode: the console shows the Hungarian pairing-code banner; the box appears under Unclaimed appliances at the hub (verify uuid/MAC/SSH fingerprints match the physical box before binding — never bind a fingerprint you didn't read off the console).

  3. Bind to the customer — the rehearsal used the SELF-BIND path, and it is now the default for this runbook. Mint the link with "Send self-bind link" on the customer's Setup tab; the customer opens it and completes the public two-factor /bind/<token> page (console pairing code

    • retrieval passphrase). The 2026-07-18 run bound in 60 s from mint to consume, with zero failed attempts, and the credential was delivered 26 s later with no operator action.

    Send the self-bind link BEFORE the customer sees the console. The console banner tells them to use „az e-mailben kapott link" — if the link has not been minted yet, the box is instructing the customer to look for something that does not exist. In the rehearsal the box sat in pairing mode for ~11.7 minutes waiting on the operator, printing failures the whole time (R-33). Mint it during step A1 if you can; the token is valid for 7 days, so there is no reason to wait. Auto-minting on customer create/RESET is R-36.

    Operator-bind on the Hosts page remains the fallback when the customer is not present or the token has expired.

  4. Watch the event sequence compose (expected: seconds between post-enroll legs — enroll → claim-code issue → WG → fast-tick window → tunnel → PBS-DR descriptor poke → controller up → claim e-mail lands → floor lift to current). Any ~15-minute stall between two legs is a finding — capture which leg + wall-clocks.

    Reference wall-clocks — measured on the N100, 2026-07-18 (tests/VALIDATION-n100-rehearsal-2026-07-18.md). Compare against these; a leg that takes minutes where this table says seconds is the finding:

    Leg Expect
    Blank-metal boot → registered as an unclaimed appliance < 1 s (same second)
    Bind → credential delivered 26 s
    Bind → agent enrolled + WG tunnel up 56 s
    Bind → controller running the CURRENT version 2 min 44 s
    Credential delivered → day-0 provision SUCCESS 2 min 11 s
    Managed-floor lift (golden version → current) 5 s, unattended
    Drive enrol: destructive-confirm → usable (incl. mkfs) ~30 s
    Escrow ceremony started → offsite runs enabled 12 s

    Note the PBS-DR descriptor leg specifically: in the rehearsal it reported converged while the credential behind it was already stale, leaving the DR tier authenticating 401 with nothing surfacing the fault (R-39). Until that is fixed, treat "descriptor provisioned" as unproven and check pvesm status on the box during step E1.

C. Customer zero script (the customer drives, operator narrates — ~30 min)

This is the friction lens: the customer's hands on the keyboard, the operator's hands OFF.

  1. Claim: open the claim e-mail → claim the dashboard → set the dashboard password. (R-4 gate: the e-mail must land in inbox, not spam. gmail PROVEN 2026-07-18 — the rehearsal's claim mail was the first sent under the tightened DMARC p=quarantine and landed in the Inbox. freemail.hu is still untested — if your tester is on freemail.hu, treat step C1 as unproven and have them check spam before you conclude the mail was lost.)

  2. Escrow ceremony: run the recovery-code wizard → the awaiting card („megerősítésre vár, legfeljebb 15 perc") → flips on ACK. The code goes on paper, into the envelope, into the customer's drawer — state plainly: without this code, a dead box means the remote backup is unrecoverable; we cannot read it for them. That sentence is the product.

  3. Storage: enroll/format the external HDD via the wizard (customer clicks, operator explains the destructive-confirm). Expect ~30 s from the confirm to a usable drive — the wizard runs mkfs detached and polls, so a slow format is not a hang.

  4. Offsite: enable „Távoli mentés". Two operator behaviours here are workarounds for known defects — follow them until the items ship, and do not improvise:

    Click Save exactly ONCE, then wait, then verify. Provisioning runs synchronously inside the request, so on a slow Hetzner leg the request can return an nginx 504 while succeeding server-side. A second click races the first attempt. Wait, then confirm on the hub's Offsite → Restic tab rather than believing the browser. (R-31)

    Re-onboarding an existing customer? Check offsite provisioning by hand. After a RESET + re-enroll the hub can sit in "enabled but unprovisioned" and say nothing — provisioning is Save-triggered, and the re-enroll auto-re-issue deliberately skips unprovisioned targets. So an offsite target that looks configured may never have been created. Open the offsite settings and press Save once to force provisioning, then verify. (R-36)

    Expect the first run to be clean on a NEW customer. If an orphan card appears („a távoli mentési tároló elárvult"), the repo is not new — that is the R-32 case, which fires when a customer's pool directory outlived a RESET. It is safe (the guard moves the old repo aside, never deletes it) but it means the box is writing into a reused directory. Record it.

  5. First app: deploy one app the household actually wants (paperless or immich recommended) — through the catalog UI, no shortcuts.

  6. Sharing: enable „Megosztás", set the household SMB password, create one share; open \\NÉV from their own laptop. (Set expectations per the one-pager: device list may lag on some routers; the address always works.)

  7. Protection proof — the customer performs a restore, unassisted (this step IS the capability-map evidence, per customer): drop a file on the share or into the app → „Távoli mentés most" → wait for the snapshot → delete the file → the customer restores it via Visszaállítás with the operator silent. If they can't without help, that's an R-11/UX finding, not a customer failure — write it down verbatim.

    ⚠ THIS IS THE ONE STEP THE REHEARSAL DID NOT EXECUTE. Every other step in this document was walked on real hardware on 2026-07-18; C7 was not, so it carries no friction notes and no evidence — the capability-map row "a customer (not the operator) performs a restore via UI alone" is still MISSING. This runbook stays a DRAFT until C7 has been performed once. It is remote-doable on the reborn demo box (the dashboard is remote) and does not need the hardware in hand.

  8. Alerts sanity: show them what an alert e-mail looks like (the health-degradation mail); confirm their address receives it.

D. Handover (10 min)

  • Walk the one-pager together: what it does, what it doesn't do yet, how to report.
  • Reporting channel: support@felhom.eu (catch-all already delivers it) + the personal channel for the friend circle. „Ha bármi furcsa: e-mail vagy üzenet, screenshot ha van."
  • Sign the tester agreement. Leave the one-pager with them.
  • Operator record: board model + loader, drive serials, claim date, escrow ACK timestamp, friction notes.

E. Operator post-checks (same day, remote)

  1. Hub: customer row green; Offsite → Restic tab shows the first bytes within a report cycle; PBS-DR gauge nominal (first PBS run on its own schedule — note, don't wait).

    Verify the DR tier on the box, not from the hub (R-39, interim). The agent can report the PBS descriptor converged while the credential behind it is stale, and its own verify loop cannot read the credential to notice. Until R-39 ships, SSH to the host and run pvesm status — the felhom-pbs line must be active. A 401 Unauthorized / inactive there means the DR tier is dead despite everything upstream looking green, and it will stay dead silently until someone needs a restore.

  2. Floor convergence: controller at current, agent at MinAgent+.

  3. Route friction notes: UX items → ROADMAP; one-pager corrections → R-11; runbook corrections → this doc.

F. Standing rules

  • Never perform steps C1C7 for the customer. The script's value is watching them do it.
  • Never handle the recovery code — see it into the envelope, never into your hands or a photo.
  • One onboarding per day maximum during alpha — the friction notes are the point, and they degrade after the first.
  • Post-rehearsal checklist item: rebuild the golden to the current controller before the FIRST real tester (fresh boxes shouldn't sit in a stale window longer than needed), and re-verify the Day-0 manifest after every controller release during alpha.