admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(rehearsal): N100 S8 wrap — VALIDATION, map flips, ROADMAP R-30..R-39, R-3 draft

Browse Source 
The 2026-07-18 N100 rehearsal ran the complete final-product flow on real metal
in one pass: RESET -> generic pairing ISO v1.20.0 -> customer self-bind -> day-0
-> managed-floor lift -> escrow ceremony -> offsite snapshots. No code changed;
every finding is recorded and ranked, none fixed.

VALIDATION-n100-rehearsal-2026-07-18.md — run context, a UTC-normalised timeline
built from the hub events stream / hub DB / controller log / bootstrap + agent
journals, per-ledger verdicts for S1-S8 + ledgers 8 and 9, 12 findings, the
not-exercised list, and 7 discrepancies against the brief.

Headline wall-clocks: bind -> credential 26 s; bind -> controller running the
current version 2 min 44 s; managed floor 0.143.0 -> 0.145.0 in 5 s unattended
(initiated_by: auto-floor); escrow ceremony -> offsite enabled 12 s; drive enrol
30.3 s. No post-bind leg stalled, which is the immediacy row's real-onboarding
proof.

Capability map (10 citations added):
- Bare-metal Felhom ISO           PARTIAL     -> PROVEN-LIVE (F1 closed on metal)
- Customer self-bind (slice 1)    IMPLEMENTED -> PROVEN-LIVE (customer_selfbind)
- Guest RAM resize (R-24)         IMPLEMENTED -> PROVEN-LIVE (shrink AND grow)
- Customer RESET                  two real firings + verified external teardown
- Escrow ceremony                 first live wizard firing
- Immediacy row                   "real-onboarding proof pending" cleared
- Publish train                   box-side floor lift proven on a fresh install
- Customer claim                  R-4 gmail half (Inbox under p=quarantine)
- Offsite orphan guard            staged live leg fired on its own
- DR tier by default              candidate PROVEN-LIVE upgrade WITHDRAWN (R-39)
Not flipped, as instructed: customer-performs-restore, BYO, DLNA, multi-user.

ROADMAP — collapsed R-1 (appliance half done, Peti half survives), R-21
(physically closed), R-24, R-27 slice 1, R-4. New ranked items:
  P2-HIGH  R-39 PBS DR applied-but-dead   R-30 liveness from the wait channel
           R-31 async offsite + status    R-32 RESET base-dir purge
           R-33 bootstrap quiet-poll
  P2       R-34 backup lifecycle          R-35 config-apply session survival
           R-36 post-RESET offsite prompt R-27c console-passphrase bind
  P3       R-37 post-RESET health card    R-38 installer GRUB slice
Plus a pre-invite checklist (golden 0.145.x rebuild, freemail.hu, C6, R-11).

R-39 is NEW and was not on the brief: the PBS DR descriptor auto-provisions and
the agent converges state=applied, but pvesm reports 401 Unauthorized/inactive
and a direct probe 401s on every endpoint including /version while WG is healthy.
The hub minted a second token secret two minutes after the agent applied the
first and consumed_at is still NULL; the converged state machine will not
re-apply, and the agent's verify loop cannot read the credential to notice it
(non-root read of a file it writes through a root wrapper). Rank is provisional
pending Viktor.

R-3 draft: all four [REFINE] slots filled, self-bind made the default path with
"send the link BEFORE the customer sees the console", the measured wall-clock
table added, and interim operator workarounds for R-31/R-36/R-39. C6 (renumbered
C7) is marked as the single unexecuted step and keeps the doc a DRAFT.

Evidence bundle: 180:~/n100-rehearsal/ (10 files + MANIFEST.md), collected before
the box was unplugged for travel. Secrets read only to run probes; recorded as
lengths and metadata, never values.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
This commit is contained in:
authentik Default Admin
2026-07-18 19:56:17 +02:00
parent bbaa844f8c
commit aa9c08f0dd
4 changed files with 447 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/runbooks/RUNBOOK-onboarding-draft-v3.md
+100 -19
View File
@@ -1,17 +1,32 @@
# DRAFT R-3 — Friend-alpha onboarding runbook (operator)
> **Status: DRAFT, pre-rehearsal.** Generalized from `runbooks/RUNBOOK-rehearsal-v3.md` (S1–S8) and
> the shipped claim/escrow/self-bind flows. The N100 rehearsal EXECUTES this document; every
> `[REFINE]` slot gets filled from its friction notes, then this graduates to
> `documentation/runbooks/RUNBOOK-onboarding.md` and the rehearsal item collapses.
> **Status: DRAFT, post-rehearsal — all `[REFINE]` slots filled 2026-07-18.** Generalized from
> `runbooks/RUNBOOK-rehearsal-v3.md` (S1–S8) and the shipped claim/escrow/self-bind flows, then
> refined against the N100 rehearsal that executed it
> (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). Every step here has been walked on real
> hardware **except C7**, the customer-performed restore.
>
> **It graduates to `RUNBOOK-onboarding.md` when C7 has evidence** — that step is simultaneously the
> last unwalked step of this script and the capability map's last MISSING customer-facing row, so
> one execution closes both. Until then this stays a DRAFT.
>
> Scope: **appliance hardware only** (BYO track is R-1/Peti, explicitly out of alpha scope).
> Three steps below carry **interim workarounds for known defects** (R-31, R-36, R-39) — they are
> marked inline and should be deleted from this runbook the moment those items ship.
## A. Before the visit (operator, ~30 min, all remote-doable)
1. **Hub:** create the customer (name, e-mail, domain `<név>.felhom.eu`) — verify the CF tunnel +
zone token provisioning completes. Confirm the Day-0 manifest is current (agent, golden,
MinAgent) and the managed-update floor is at the intended release — a stale golden is fine
(the floor lifts the box post-install), a stale FLOOR is not.
(the floor lifts the box post-install), a stale FLOOR is not. *The rehearsal proved the lift:
a 0.143.0 golden reached 0.145.0 in **5 s**, unattended, inside the first minute of controller
life.* Even so, **rebuild the golden before the first real tester** — proven-recoverable is not
a reason to ship stale.
**Mint the self-bind link now** ("Send self-bind link" on the customer's Setup tab). It is valid
for **7 days**, and doing it here means the console banner's „e-mailben kapott link" is already
true when the customer first looks at the screen. See B3.
2. **Hardware:** the standard appliance + its external HDD. One-time bench prep list:
memtest pass, SMART clean, BIOS queue for the single visit in B2.
3. **Install media:** the **reusable generic pairing ISO** (secret-free, slice C). Only rebuild it
@@ -30,39 +45,98 @@
**pairing-code banner**; the box appears under **Unclaimed appliances** at the hub
(verify uuid/MAC/SSH fingerprints match the physical box before binding — never bind a
fingerprint you didn't read off the console).
3. **Bind** to the customer. Self-bind path (customer enters the code themselves) preferred when
the customer is present — it is the flow real remote onboarding will use. `[REFINE: which path
the rehearsal used + friction]`
3. **Bind** to the customer — **the rehearsal used the SELF-BIND path, and it is now the default for
this runbook.** Mint the link with **"Send self-bind link"** on the customer's Setup tab; the
customer opens it and completes the public two-factor `/bind/<token>` page (console pairing code
+ retrieval passphrase). The 2026-07-18 run bound in **60 s from mint to consume, with zero failed
attempts**, and the credential was delivered **26 s later** with no operator action.
> **Send the self-bind link BEFORE the customer sees the console.** The console banner tells them
> to use „az e-mailben kapott link" — if the link has not been minted yet, the box is instructing
> the customer to look for something that does not exist. In the rehearsal the box sat in pairing
> mode for **~11.7 minutes** waiting on the operator, printing failures the whole time (R-33).
> Mint it during step A1 if you can; the token is valid for **7 days**, so there is no reason to
> wait. Auto-minting on customer create/RESET is R-36.
Operator-bind on the Hosts page remains the fallback when the customer is not present or the
token has expired.
4. Watch the event sequence compose (expected: **seconds** between post-enroll legs — enroll →
claim-code issue → WG → fast-tick window → tunnel → PBS-DR descriptor poke → controller up →
claim e-mail lands → floor lift to current). **Any ~15-minute stall between two legs is a
finding** — capture which leg + wall-clocks. `[REFINE: reference wall-clock table from the
rehearsal]`
finding** — capture which leg + wall-clocks.
**Reference wall-clocks — measured on the N100, 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`).
Compare against these; a leg that takes minutes where this table says seconds is the finding:
| Leg | Expect |
|---|---|
| Blank-metal boot → registered as an unclaimed appliance | **< 1 s** (same second) |
| Bind → credential delivered | **26 s** |
| Bind → agent enrolled + WG tunnel up | **56 s** |
| Bind → controller running the CURRENT version | **2 min 44 s** |
| Credential delivered → day-0 provision SUCCESS | **2 min 11 s** |
| Managed-floor lift (golden version → current) | **5 s**, unattended |
| Drive enrol: destructive-confirm → usable (incl. mkfs) | **~30 s** |
| Escrow ceremony started → offsite runs enabled | **12 s** |
Note the **PBS-DR descriptor** leg specifically: in the rehearsal it reported `converged` while
the credential behind it was already stale, leaving the DR tier authenticating `401` with nothing
surfacing the fault (**R-39**). Until that is fixed, treat "descriptor provisioned" as unproven
and check `pvesm status` on the box during step E1.
## C. Customer zero script (the customer drives, operator narrates — ~30 min)
This is the friction lens: the customer's hands on the keyboard, the operator's hands OFF.
1. **Claim:** open the claim e-mail → claim the dashboard → set the dashboard password.
(R-4 gate: the e-mail must land in inbox, not spam — gmail proven; `[REFINE: freemail.hu
result]`.)
(R-4 gate: the e-mail must land in inbox, not spam. **gmail PROVEN 2026-07-18** — the rehearsal's
claim mail was the first sent under the tightened DMARC `p=quarantine` and landed in the **Inbox**.
**freemail.hu is still untested** — if your tester is on freemail.hu, treat step C1 as unproven and
have them check spam before you conclude the mail was lost.)
2. **Escrow ceremony:** run the recovery-code wizard → the awaiting card
(„megerősítésre vár, legfeljebb 15 perc") → flips on ACK. The code goes on paper, into the
envelope, into the customer's drawer — **state plainly: without this code, a dead box means
the remote backup is unrecoverable; we cannot read it for them.** That sentence is the product.
3. **Storage:** enroll/format the external HDD via the wizard (customer clicks, operator explains
the destructive-confirm).
4. **First app:** deploy one app the household actually wants (paperless or immich recommended) —
the destructive-confirm). Expect **~30 s** from the confirm to a usable drive — the wizard runs
mkfs detached and polls, so a slow format is not a hang.
4. **Offsite:** enable „Távoli mentés". Two operator behaviours here are **workarounds for known
defects** — follow them until the items ship, and do not improvise:
> **Click Save exactly ONCE, then wait, then verify.** Provisioning runs synchronously inside the
> request, so on a slow Hetzner leg the request can return an nginx **504 while succeeding
> server-side**. A second click races the first attempt. Wait, then confirm on the hub's
> Offsite → Restic tab rather than believing the browser. (**R-31**)
> **Re-onboarding an existing customer? Check offsite provisioning by hand.** After a RESET +
> re-enroll the hub can sit in "enabled but unprovisioned" and **say nothing** — provisioning is
> *Save*-triggered, and the re-enroll auto-re-issue deliberately skips unprovisioned targets. So
> an offsite target that looks configured may never have been created. Open the offsite settings
> and press Save once to force provisioning, then verify. (**R-36**)
> **Expect the first run to be clean on a NEW customer.** If an **orphan card** appears
> („a távoli mentési tároló elárvult"), the repo is not new — that is the R-32 case, which fires
> when a customer's pool directory outlived a RESET. It is safe (the guard moves the old repo
> aside, never deletes it) but it means the box is writing into a reused directory. Record it.
5. **First app:** deploy one app the household actually wants (paperless or immich recommended) —
through the catalog UI, no shortcuts.
5. **Sharing:** enable „Megosztás", set the household SMB password, create one share; open
6. **Sharing:** enable „Megosztás", set the household SMB password, create one share; open
`\\NÉV` from their own laptop. (Set expectations per the one-pager: device list may lag on
some routers; the address always works.)
6. **Protection proof — the customer performs a restore, unassisted** (this step IS the
7. **Protection proof — the customer performs a restore, unassisted** (this step IS the
capability-map evidence, per customer): drop a file on the share or into the app → „Távoli
mentés most" → wait for the snapshot → delete the file → the customer restores it via
**Visszaállítás** with the operator silent. If they can't without help, that's an R-11/UX
finding, not a customer failure — write it down verbatim. `[REFINE: rehearsal friction notes]`
7. **Alerts sanity:** show them what an alert e-mail looks like (the health-degradation mail);
finding, not a customer failure — write it down verbatim.
> **⚠ THIS IS THE ONE STEP THE REHEARSAL DID NOT EXECUTE.** Every other step in this document was
> walked on real hardware on 2026-07-18; C7 was not, so it carries **no friction notes and no
> evidence** — the capability-map row *"a customer (not the operator) performs a restore via UI
> alone"* is still **MISSING**. This runbook stays a **DRAFT** until C7 has been performed once.
> It is remote-doable on the reborn demo box (the dashboard is remote) and does not need the
> hardware in hand.
8. **Alerts sanity:** show them what an alert e-mail looks like (the health-degradation mail);
confirm their address receives it.
## D. Handover (10 min)
@@ -78,13 +152,20 @@ This is the friction lens: the customer's hands on the keyboard, the operator's
1. Hub: customer row green; Offsite → Restic tab shows the first bytes within a report cycle;
PBS-DR gauge nominal (first PBS run on its own schedule — note, don't wait).
> **Verify the DR tier on the box, not from the hub (R-39, interim).** The agent can report the
> PBS descriptor `converged` while the credential behind it is stale, and its own verify loop
> cannot read the credential to notice. Until R-39 ships, SSH to the host and run
> `pvesm status` — the `felhom-pbs` line must be **active**. A `401 Unauthorized` / `inactive`
> there means the DR tier is dead despite everything upstream looking green, and it will stay
> dead silently until someone needs a restore.
2. Floor convergence: controller at current, agent at MinAgent+.
3. Route friction notes: UX items → ROADMAP; one-pager corrections → R-11; runbook corrections →
this doc.
## F. Standing rules
- Never perform steps C1–C6 *for* the customer. The script's value is watching them do it.
- Never perform steps C1–C7 *for* the customer. The script's value is watching them do it.
- Never handle the recovery code — see it into the envelope, never into your hands or a photo.
- One onboarding per day maximum during alpha — the friction notes are the point, and they
degrade after the first.