diff --git a/REPORT.md b/REPORT.md index 08421ba..fe4b9c8 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,187 +1,56 @@ -# REPORT — TASK-I: R-25b, customer DELETE becomes the guided full-teardown cascade +# REPORT — Runbook refresh: RUNBOOK-onboarding-draft v3 → v4 (docs-only, XS) -**Two releases: v0.69.0 (the cascade) and v0.70.0 (the residue leg + ghost cleanup, found validating v0.69.0 against the live hub).** - -**Date:** 2026-07-21 · **Repo:** `felhom.eu` · **Baseline:** `f59aa97` (clean, HEAD == origin/main) -**Scope:** hub only — **v0.68.1 → v0.69.0**. No agent / controller / catalog change. - -## What shipped - -`POST /configs/{id}/delete` (same route, new behaviour) is now the guided full-teardown cascade. -`GET` on the same path returns the dialog's live inventory. The shallow `handleConfigDelete` is -**gone**. - -Three legs, fixed order: - -1. **hosts** — every host row via `store.DeleteHost(hostID, true)`; escrow **DEMOTED** to retained - custody, never destroyed. Host-delete's own ONLINE rule is kept: an ONLINE host refuses the whole - cascade, checked for every host up front so it never half-runs. -2. **reset** — the committed RESET sequence verbatim (Hetzner → PBS → claim → descriptor → DB purge) - through the newly extracted `commitCustomerReset`, called with `purgeEscrow=false`. -3. **purge** — `store.DeleteCustomerConfig`: the customer record **and all escrow ciphertext**. - -Two invariants are asserted, not merely commented: - -- **Ruling 3 by construction** — leg 2 can only run after leg 1, so the RESET sequence never sees a - host row. The standalone RESET handler's 409 gate is untouched. -- **Custody purged exactly ONCE, in leg 3** — leg 1 demotes; leg 2 runs with `purgeEscrow=false`; - leg 3 is the one true purge point (v0.60.1). - -**Gates, all before any write** (a refusal has zero side effects): three acknowledgements -(`ack_hosts` / `ack_reset` / `ack_purge`, each exactly `1`), the typed customer-id, a **stale-preview** -check (the acknowledged host count must still match live → else 409), and the ONLINE-host refusal. -No force flag, no skip flag, no partial-run downgrade. - -**Resume:** a failed leg retains the `customer_resets` journal row and the HTTP error names the leg. -The dialog renders the incomplete journal and offers **Resume**; a re-run is idempotent and must pass -every gate again (acknowledgements are not cached across attempts). - -**UI:** Danger zone → **Delete customer…** → guided dialog (inventory panel: hosts by name + status, -offsite repository identifier, PBS namespace, custody state; three consequence checkboxes; typed -customer-id; one submit). Client-side checks are convenience only. - -## Refactor — standalone RESET behaviour unchanged - -`handleCustomerReset`'s committed half became -`commitCustomerReset(ctx, cfg, resetID int64, purgeEscrow bool) *resetLegError`. The standalone path -is byte-identical to v0.68.1: same leg order, same leg names, same operator-facing messages, same -status codes. Its existing suite is untouched and green. - -## Files - -| File | Change | -|---|---| -| `hub/internal/web/customer_delete.go` | NEW — the cascade + preview | -| `hub/internal/web/customer_delete_test.go` | NEW — scenarios A–E | -| `hub/internal/web/customer_reset.go` | `commitCustomerReset` + `resetLegError` extracted | -| `hub/internal/web/configs.go` | `handleConfigDelete` removed (replaced by a do-not-reintroduce note) | -| `hub/internal/web/server.go` | route: GET → preview, POST → cascade | -| `hub/internal/web/templates/customer_unified.html` | guided dialog replaces the one-click Delete | -| `hub/internal/web/customer_edit_tab_test.go` | the delete redirect case now posts the full acks | -| `hub/CHANGELOG.md`, `REUSE.md`, `documentation/backlog/ROADMAP.md`, `documentation/architecture/00-capability-map.md`, `documentation/runbooks/RUNBOOK-onboarding-draft-v3.md` | docs | - -## Tests + red-proofs - -Green gate in `hub/`: `go build ./... && go vet ./... && go test ./...` — **all green**, no flakes. - -New coverage (`customer_delete_test.go`): - -- **A — leg ORDER**, observed from *inside* leg 2 via a `tenancyProvisioner` fake whose `Deprovision` - snapshots store state: at that instant hosts = 0 (leg 1 done), the customer row is still present - (leg 3 not started), retained custody still present (leg 1 demoted, did not purge). Plus final - state, the four journal legs stamped `ok`, completion stamped, and the `customer_deleted` audit - event surviving the record. -- **B — 9 fail-closed gate cases** (each missing ack, an ack sent as `yes`, id mismatch, id absent, - host count moved, host count absent, ONLINE host). Each asserts the status code **and** that the - host, the customer row, the current escrow and the retained custody are untouched, **and** that - zero external calls fired, **and** that no journal row was opened. -- **C — resume**: an injected PBS failure → 502 naming the leg; hosts gone, customer + custody - SURVIVE, journal retained with `hosts=ok pbs=failed`; re-run converges and completes. Plus: a - resume without ack #3 is still refused. -- **E — custody**: `commitCustomerReset(..., purgeEscrow=false)` leaves the retained blobs; leg 3 - purges them. -- **Preview**: names the real host/offsite/PBS/custody facts and leaks no secret (one-time password, - API keys, escrow blobs all asserted absent). - -**Five red-proofs run, each failed red with the wrong value visible, then restored (`git diff` clean):** - -| # | Pre-fix shape restored | Failure observed | -|---|---|---| -| 1 | ack gate disabled | `status = 303, want 400` + host deleted, customer deleted, custody destroyed, PBS deprovision fired, journal row opened | -| 2 | stale-preview gate weakened to always pass | `status = 303, want 409` + the same six non-effect assertions | -| 3 | ONLINE-host gate removed | `status = 303, want 409` + live host deleted | -| 4 | leg order inverted (RESET leg before the host leg) | `at the RESET leg the customer still had 1 host(s) — leg 1 must complete FIRST` | -| 5 | cascade's RESET leg called with `purgeEscrow=true` | `retained blobs after the RESET leg = 0, want 2` | - -## Method / limits - -Unit-land only so far — **the STOP-gated live leg has not been run** (see below). Validation method: -Go tests against a real SQLite store on `t.TempDir()` with fakes at the existing `tenancyProvisioner` -seam. No browser is available on DooPlex; the dialog's markup is covered by the existing -customer-page render tests (exactly one `/configs/{id}/delete` form on the page) — a strict -click-through remains a manual operator pass. - -**Not covered by unit tests:** the Hetzner offsite `Deprovision` leg (`offsite.Provisioner` is a -concrete type, no interface seam — same as the standalone RESET suite). That leg is exactly what the -live run is for. - -## STOP — operator-present live leg (NOT yet run) - -Create a scratch customer on a throwaway domain, provision **offsite only** (no host — cheap), run -the cascade end-to-end, then verify from **outside** the hub that the Hetzner repository is gone and -the customer row is purged. **Never run against Demo Ügyfél, Demo HP, or Peti.** Failure paths are -unit-proven; the live leg proves the happy path + external teardown only. - -## Deploy status - -**LIVE.** Code `61dbd87` → `main`; image `gitea.dooplex.hu/admin/felhom-hub:0.69.0` built + pushed; -`manifests/hub.yaml` bumped in `5dcb72a`; ArgoCD hard-refresh + deliberate sync → app `felhom` -**Synced / Healthy**; `deploy/hub` rolled out on image `:0.69.0`; startup log -`[INFO] felhom-hub 0.69.0 starting` … `Listening on :8080`, no errors. - -The STOP-gated live cascade run above is still **outstanding** — the code is deployed, the scratch -customer teardown has not been performed. - ---- - -# Addendum — v0.70.0: a deleted customer actually disappears - -## How it was found - -The operator reported that `demo-vm-felhom` "was deleted but is still here". It had NOT failed: -config row gone, both hosts deleted (07-16, 07-18, escrow acked), `host_escrow` and -`host_escrow_superseded` empty, the 07-18 RESET journal complete with every leg `ok`. - -The customer was still listed because **`store.GetCustomers()` derives the customer list purely from -the REPORT stream**, and no lifecycle tier has ever deleted a report. 502 report rows kept the ghost -alive. - -**This was not cosmetic.** The staleness and offsite checkers iterate the same report-derived list, -so the hub kept raising `offsite_stale` for a customer that no longer exists — **10 events, the most -recent 2026-07-21 17:34, three days after deletion, with an operator email sent at 19:34** (after the -v0.69.0 deploy). Verified by streaming the live `hub.db` out read-only and querying it. - -Two further residue rows are **credential-bearing**, not telemetry: -`appliance_registrations` (a `token_hash` with `status='delivered'`, still bound to the dead customer -— confirmed present for `demo-vm-felhom`) and `selfbind_tokens` (an unconsumed bind token would be a -working path to bind a box to a nonexistent customer). +**Date:** 2026-07-22 · **Repo:** `felhom.eu` · **Scope:** documentation only — no code, no version +bump, no deploy. The onboarding runbook was refreshed against what shipped 2026-07-18..21 and +against the second full onboarding on virgin hardware (HP t740, 2026-07-21). ## What changed -- **New leg 3, `residue`** (`store.PurgeCustomerResidue`, one transaction): `reports`, - `app_telemetry`, `app_log_tails`, `log_tail_requests`, `customer_notifications`, - `selfbind_tokens`, `appliance_registrations`. Runs BEFORE the record purge — `customer_configs` is - the identifying descriptor and goes last. The cascade is now `hosts → RESET → residue → purge`. - `events`, `notification_log`, `host_deletions`, `customer_resets` still survive. - The counter and the purge walk **one shared `residueQueries` list**, so a table cannot be - counted-but-not-purged. -- **Ghost customers are deletable.** Both the cascade and its preview used to 404 whenever the config - row was missing — meaning no operator surface could clear a customer deleted by any earlier path. - **404 now means "there is nothing here"** (no config, no host, no residue). With no config row the - offsite descriptor is unknowable, so `commitCustomerReset` records **`skipped_no_config`** for the - Hetzner and descriptor legs — never a bare `skipped`, which would read as "nothing to do". PBS is - id-keyed and idempotent, so it still runs. The dialog labels the ghost case and names the row count. +`documentation/runbooks/RUNBOOK-onboarding-draft-v3.md` → **`RUNBOOK-onboarding-draft-v4.md`** +(git mv, one commit), per the project-Claude refresh brief of 2026-07-21: -## Tests + red-proofs (v0.70.0) +1. **R-39 interim blocks DELETED** (B4 descriptor caveat + E1 manual `pvesm status` check) — + replaced with one sentence each: the DR tier self-detects a dead credential (loud `auth_failed`) + and self-heals via damped re-issue; the hub gauge is trustworthy. Cited: `backlog/ROADMAP.md` + R-39 (CLOSED, proven live 2026-07-21, hub 0.68.1 + agent 0.91.2, 13 s chain + `applied → auth_failed → applied`). +2. **R-36 workaround block in C4 DELETED** (re-onboarding manual-Save) — replaced with the shipped + behaviour: enabled-but-unprovisioned raises an amber banner (hub v0.67.0). R-31 (Save-once/504) + and the R-32 orphan-card note KEPT. Consistency follow-through: A1 and the B3 note now say the + self-bind link is **auto-minted at create/RESET** (also R-36) — the manual mint is the re-mint + fallback. +3. **C1 freemail gate** — freemail.hu PROVEN 2026-07-21 (operator test-send received; R-4 COMPLETE + per `backlog/ROADMAP.md`); the check-spam habit line kept. +4. **A1 golden note** — golden 0.153.0 carries all four infra images; floor-lift proven on TWO + shapes (rehearsal 0.143→0.145 in 5 s; virgin HP t740 0.153.0→0.156.0 during day-0, `CONTEXT.md`). + "Rebuild before first tester" softened to a pointer at the standing rule in F. +5. **A3 loader note** — the single F1 line replaced with the loader table. **Deviation from the + brief, on the record's side:** the brief said "N100/AMI = shim"; the record + (`operations/nodes.md` + `tests/VALIDATION-n100-baremetal-2026-07-16.md` F1) says the N100 + installed with **mkimage, SB off** — the AMI AN3PLUS firmware cannot relocate the signed GRUB, + which is exactly why shim was not usable there. Table written per the record, as the brief's + "(SB state as recorded)" directs. HP t740 = shim with Secure Boot ENABLED (2026-07-21). +6. **B2/B3 second datapoint** — the HP onboarding walked register → pairing banner → unclaimed + list → self-bind (code + passphrase) → day-0 → dashboard on never-seen hardware (cited + `CONTEXT.md` + `operations/nodes.md`). New B2 gotcha: on multi-NIC boards verify the CABLED + port got the lease during install (the t740 4-port trap; R-59/R-60 pending). +7. **New A5 checklist item** — agent train ≥ v0.93.0 (hyphen-free recovery wordlist, + `felhom-agent/CHANGELOG.md`) must be BUILT + DEPLOYED to the customer's box before the FIRST + real tester's escrow ceremony; ceremonies run box-side, source-only does not count. +8. **D offboarding pointer** — R-25b ruling noted: full teardown = DELETE cascade (TASK-I), + RESET = identity-preserving re-onboarding; pointer to §G. +9. **KEPT unchanged:** DRAFT status, the C7 graduation-gate paragraph (still the last unwalked + step), all reference wall-clocks, all F rules, all of §G. -- `TestDeleteCascade_PurgesResidueAndUnlistsCustomer` — residue zeroed, customer absent from - `GetCustomers()`, appliance registration and self-bind token gone **by name**, audit + F-14 - provenance intact, journal `residue=ok customer_delete=ok`. -- `TestDeleteCascade_GhostCustomerIsDeletable` — the exact `demo-vm-felhom` shape (hosts deleted, - config dropped, residue alive): preview 200 with `has_config:false`, cascade completes, journal - records `skipped_no_config` for hetzner + descriptor. -- `TestDeleteCascade_404WhenNothingRemains`. +Also: `backlog/ROADMAP.md` R-25b's live pointer updated `…draft-v3.md` → `…draft-v4.md` so the +citation resolves (historical references in the rehearsal validation doc and past REPORTs left +as-is — they describe the file as it was named then). -| # | Pre-fix shape restored | Failure observed | -|---|---|---| -| 6 | residue leg removed (the v0.69.0 shape) | `residue after cascade = {Reports:1 AppTelemetry:1 NotificationPrefs:1 SelfBindTokens:1 ApplianceRegistrations:1}` + "customer is STILL on the Customers list" + appliance/self-bind rows outlived their customer | -| 7 | `cfg == nil` 404 restored | `ghost preview = 404, want 200` | +## Not done / unchanged -Full suite green (`go build ./... && go vet ./... && go test ./...`), hub confirm gate OK. +- C7 not flipped; the capability-map row untouched; no other runbook touched. +- No CHANGELOG entry: this repo's changelogs are per-area (`hub/`, `scripts/`, `website/`) and none + of those areas changed; prior docs-only commits (e.g. TASK-H nodes.md) followed the same pattern. -## Still outstanding - -The STOP-gated live leg. `demo-vm-felhom` is now the natural subject — it is a real ghost, the -operator has authorised it, and clearing it proves the v0.70.0 path end to end on production data. -It cannot prove the **Hetzner** teardown (no config row → `skipped_no_config`), so a scratch customer -with offsite provisioned is still needed for that half. +*(Previous REPORT — TASK-I, hub v0.69.0/v0.70.0 DELETE cascade — preserved in git history at +`9b3381b` and summarized in `backlog/ROADMAP.md` R-25b.)* diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index a2cdebb..f7efcb2 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -84,7 +84,7 @@ | R-26 | **Guided old-history recovery via a retained superseded escrow + the recovery code.** Enabled by hub v0.60.0 (Part B) which now RETAINS superseded escrow blobs (`host_escrow_superseded`, `ListSupersededEscrow`). Build the flow that, given the customer's recovery code, unwraps a retained old blob → recovers the old repo passphrase → mounts/reads the moved-aside `.orphaned-` repo for restore. | M | idea (enabled by v0.60.0) | Turns "history recoverable in principle" into a real customer-drivable path; pairs with the controller v0.142.0 orphaned-repo move-aside. Origin `DIAGNOSE-offbox-repo-orphaned-2026-07-17` | | R-27 | **Customer-facing self-bind page (R-21 slice C follow-on).** Today an unclaimed appliance is bound by the OPERATOR on the Hosts page (hub v0.62.0). Build the customer-facing flow so a customer can claim/bind their own freshly-installed box. | M | **SHIPPED (slice 1, hub v0.66.0, 2026-07-17)** | **Slice 1 = the FIRST-box flow, DONE:** operator "Send self-bind link" → 7-day tokenized capability link over Hungarian email → public two-factor `/bind/` (console pairing code shown on the box screen + retrieval passphrase) → hub stages the bind via the same `BindAppliance`, no operator. Viktor's 3 rulings verbatim (console code not a list; operator-sent link; 5-attempt lockout→call support). No oracle; expiry falls back to operator-bind; THE TRAP `/bind/` exemption tight (single predicate, 4 red-proofs). GC verdict §3 (no appliance GC → the 7-day TTL stands alone). **Slice 1 PROVEN-LIVE 2026-07-18:** in the rehearsal the operator minted + emailed the link at 16:28:55 and **the customer bound their own box at 16:29:55 with zero failed attempts**, `appliance_bound` carrying source `customer_selfbind`, credential delivered 26 s later with no operator action (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). The new-ISO console banner rendered on the real box. **Slice 2 = console-passphrase bind → R-27c** (below). Reused the appliance_registrations + one-shot delivery machinery; the capability link is the whole customer-auth surface (no login system built). Controller/agent untouched. See hub v0.66.0 REPORT. **Multi-box per link is out of scope by design** (single-active, one-shot → binding several boxes = repeated operator sends) | | R-27b | **Customer self-bind, second-box flow (controller side).** For a customer who ALREADY has a bound box and installs another, the controller shows a dismissable "bind another box" prompt (and a bind-later entry under settings) that walks to the hub `/bind/` page — so a returning customer isn't emailed a fresh operator-sent link for every box. Mechanism sketched in the hub v0.66.0 REPORT; NOT built (R-27 slice 1 deliberately did not touch the controller). | M | idea (minted by hub v0.66.0) | Origin: hub v0.66.0 slice-1 ship (first-box only). Reuses the same `/bind/` public page + tokenized-link machinery; adds a controller-side entry point + the operator "mint a link for an existing customer" affordance | -| R-25b | **RULED: customer DELETE becomes a guided full-teardown cascade.** The middle-tier Customer RESET (hub v0.61.0) runs the full external teardown (Hetzner sub-account/box + PBS namespace/groups/token) and refuses while any host row exists. The Danger-zone DELETE still (a) leaves host rows and (b) does NOT run that teardown. | M (was S) | **SHIPPED hub v0.69.0 (2026-07-21)** | **operator ruling 2026-07-21**: **DELETE subsumes the whole cascade, behind explicit consent.** Three separate acknowledgements, each its own checkbox — (1) the host(s) will be deleted, (2) the customer will be RESET including external teardown and offsite data destruction, (3) the customer record and escrow will be purged — plus a **typed customer-name confirmation** before the button arms. Internal order is **host-delete → RESET → delete**, which preserves every existing invariant rather than relaxing any: RESET keeps its no-hosts precondition (hosts are already gone by then), and escrow keeps its demote-then-purge custody rule (host delete DEMOTES to retained custody, the final delete PURGES — the one true purge point). **Re-sized S → M: this is a multi-step destructive wizard with three acks and a typed confirmation, not a checkbox.** Implementation is explicitly NOT part of TASK-E; the row carries the ruling and awaits its own spec. **It no longer blocks R-3** — the model is decided, so the friend-alpha runbook can be written against it. **IMPLEMENTED per the ruling (TASK-I, hub v0.69.0):** `POST /configs/{id}/delete` now runs `hosts → RESET → purge`; three acks + typed customer-id + a stale-preview check + the ONLINE-host refusal, all gates before any write (zero side effects on refusal); custody purged exactly ONCE in leg 3 (leg 2 runs with `purgeEscrow=false`); ruling-3 preserved BY CONSTRUCTION and asserted from inside leg 2; failed legs retain the journal and the dialog offers Resume. Standalone RESET byte-identical. 5 red-proofs. Offboarding guidance: `runbooks/RUNBOOK-onboarding-draft-v3.md` §G. **v0.70.0 follow-up (same day, found validating against the live hub):** a completed delete still left the customer on the Customers list and still ALERTING, because `GetCustomers()` is report-derived and no tier ever deleted a report — new **residue** leg (reports/telemetry/log-tails/notif-prefs + the credential-bearing `appliance_registrations`/`selfbind_tokens`), and **ghost customers are now deletable** (404 = nothing here, not no-config-row) | +| R-25b | **RULED: customer DELETE becomes a guided full-teardown cascade.** The middle-tier Customer RESET (hub v0.61.0) runs the full external teardown (Hetzner sub-account/box + PBS namespace/groups/token) and refuses while any host row exists. The Danger-zone DELETE still (a) leaves host rows and (b) does NOT run that teardown. | M (was S) | **SHIPPED hub v0.69.0 (2026-07-21)** | **operator ruling 2026-07-21**: **DELETE subsumes the whole cascade, behind explicit consent.** Three separate acknowledgements, each its own checkbox — (1) the host(s) will be deleted, (2) the customer will be RESET including external teardown and offsite data destruction, (3) the customer record and escrow will be purged — plus a **typed customer-name confirmation** before the button arms. Internal order is **host-delete → RESET → delete**, which preserves every existing invariant rather than relaxing any: RESET keeps its no-hosts precondition (hosts are already gone by then), and escrow keeps its demote-then-purge custody rule (host delete DEMOTES to retained custody, the final delete PURGES — the one true purge point). **Re-sized S → M: this is a multi-step destructive wizard with three acks and a typed confirmation, not a checkbox.** Implementation is explicitly NOT part of TASK-E; the row carries the ruling and awaits its own spec. **It no longer blocks R-3** — the model is decided, so the friend-alpha runbook can be written against it. **IMPLEMENTED per the ruling (TASK-I, hub v0.69.0):** `POST /configs/{id}/delete` now runs `hosts → RESET → purge`; three acks + typed customer-id + a stale-preview check + the ONLINE-host refusal, all gates before any write (zero side effects on refusal); custody purged exactly ONCE in leg 3 (leg 2 runs with `purgeEscrow=false`); ruling-3 preserved BY CONSTRUCTION and asserted from inside leg 2; failed legs retain the journal and the dialog offers Resume. Standalone RESET byte-identical. 5 red-proofs. Offboarding guidance: `runbooks/RUNBOOK-onboarding-draft-v4.md` §G. **v0.70.0 follow-up (same day, found validating against the live hub):** a completed delete still left the customer on the Customers list and still ALERTING, because `GetCustomers()` is report-derived and no tier ever deleted a report — new **residue** leg (reports/telemetry/log-tails/notif-prefs + the credential-bearing `appliance_registrations`/`selfbind_tokens`), and **ghost customers are now deletable** (404 = nothing here, not no-config-row) | | R-25 | **Device-node TOCTOU hardening (drive init).** Graduate the controller v0.141.0 Observation: the `format → resolveEnrollUUID(path) → AssignDisk(uuid)` sequence has a narrow /dev-re-enumeration window (agent-guarded on the destructive format via anti-retarget durable-id; benign fs-UUID mount). Bind resolve+assign to the format's durable-id so the mount can't target a moved node. | S | idea | From the v0.141.0 F6 commit's security-review finding (`felhom-controller` REPORT). Low real risk (single-operator, agent-guarded), but cheap to close | | R-24 | **Guest RAM resize (live) — SHIPPED (agent v0.90.0 + controller v0.143.0, 2026-07-17).** The customer right-sizes the guest's memory from the controller's Rendszer page; the agent enforces every bound fresh + applies via PVE `SetConfig` (live cgroup, no reboot, Phase-0 proven). **Framing note:** the original hub-desired-state framing is SUPERSEDED by Viktor's controller-direct ruling (2026-07-17) — the resize is controller→agent local-API, never through the hub. Memory only (cores stay observation). **COMPLETE — the apply is proven in BOTH directions on metal (2026-07-18):** customer zero shrank the guest **11675 → 8192 MB (16:50:22)** and grew it **8192 → 12288 MB (17:02:17)**, each a live cgroup apply with no reboot, the new total rippling into the deploy page's memory math. The nested demo had proven the refusals; the N100 proved the applies. `tests/VALIDATION-n100-rehearsal-2026-07-18.md` ledger 9. **Cores/live-resize as hub desired-state is NOT built** (deferred, low demand). | M | **SHIPPED + PROVEN-LIVE** | See felhom-agent + felhom-controller REPORTs; capability-map row "customer right-sizes guest RAM". | | R-12 | Cluster mode: agent-follows-guest, bind-mount reconciliation on HA migration | XL | idea | Scoped 07-15; interim = HA-group pin to one node. Driven by Peti's two-node cluster | diff --git a/documentation/runbooks/RUNBOOK-onboarding-draft-v3.md b/documentation/runbooks/RUNBOOK-onboarding-draft-v4.md similarity index 64% rename from documentation/runbooks/RUNBOOK-onboarding-draft-v3.md rename to documentation/runbooks/RUNBOOK-onboarding-draft-v4.md index 732a1e8..0d75e5a 100644 --- a/documentation/runbooks/RUNBOOK-onboarding-draft-v3.md +++ b/documentation/runbooks/RUNBOOK-onboarding-draft-v4.md @@ -1,41 +1,59 @@ # DRAFT R-3 — Friend-alpha onboarding runbook (operator) -> **Status: DRAFT, post-rehearsal — all `[REFINE]` slots filled 2026-07-18.** Generalized from -> `runbooks/RUNBOOK-rehearsal-v3.md` (S1–S8) and the shipped claim/escrow/self-bind flows, then -> refined against the N100 rehearsal that executed it -> (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). Every step here has been walked on real -> hardware **except C7**, the customer-performed restore. +> **Status: DRAFT, v4 — refreshed 2026-07-22.** Generalized from +> `runbooks/RUNBOOK-rehearsal-v3.md` (S1–S8) and the shipped claim/escrow/self-bind flows, refined +> against the N100 rehearsal that executed it (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`), +> and now carrying a **second full onboarding datapoint on virgin hardware** — the HP t740 +> (`demo-hp`, 2026-07-21; `operations/nodes.md`, `CONTEXT.md`). This refresh deleted the R-36 and +> R-39 interim workarounds (both shipped and proven) and updated the freemail/loader/golden notes. +> Every step here has been walked on real hardware **except C7**, the customer-performed restore. > > **It graduates to `RUNBOOK-onboarding.md` when C7 has evidence** — that step is simultaneously the > last unwalked step of this script and the capability map's last MISSING customer-facing row, so > one execution closes both. Until then this stays a DRAFT. > > Scope: **appliance hardware only** (BYO track is R-1/Peti, explicitly out of alpha scope). -> Three steps below carry **interim workarounds for known defects** (R-31, R-36, R-39) — they are -> marked inline and should be deleted from this runbook the moment those items ship. +> One step below still carries an **interim workaround for a known defect** (R-31) — it is marked +> inline and should be deleted from this runbook the moment that item ships. ## A. Before the visit (operator, ~30 min, all remote-doable) 1. **Hub:** create the customer (name, e-mail, domain `.felhom.eu`) — verify the CF tunnel + zone token provisioning completes. Confirm the Day-0 manifest is current (agent, golden, MinAgent) and the managed-update floor is at the intended release — a stale golden is fine - (the floor lifts the box post-install), a stale FLOOR is not. *The rehearsal proved the lift: - a 0.143.0 golden reached 0.145.0 in **5 s**, unattended, inside the first minute of controller - life.* Even so, **rebuild the golden before the first real tester** — proven-recoverable is not - a reason to ship stale. + (the floor lifts the box post-install), a stale FLOOR is not. *The current golden (0.153.0) + carries all four infra images, and the floor-lift is now proven on TWO shapes:* the rehearsal + (0.143.0 → 0.145.0 in **5 s**, unattended — `tests/VALIDATION-n100-rehearsal-2026-07-18.md`) + AND a virgin board on first contact (0.153.0 → 0.156.0 during day-0, HP t740, 2026-07-21 — + `CONTEXT.md`). Golden freshness before the first real tester is the standing rule in **F** — + apply it there, don't re-derive it here. - **Mint the self-bind link now** ("Send self-bind link" on the customer's Setup tab). It is valid - for **7 days**, and doing it here means the console banner's „e-mailben kapott link" is already - true when the customer first looks at the screen. See B3. + **Verify the self-bind link went out.** Since hub v0.67.0 (R-36) the link is **auto-minted at + customer creation and at RESET completion**, so the console banner's „e-mailben kapott link" is + already true by construction; "Send self-bind link" on the Setup tab is the manual re-mint if + the customer lost it or the 7-day token expired. See B3. 2. **Hardware:** the standard appliance + its external HDD. One-time bench prep list: memtest pass, SMART clean, BIOS queue for the single visit in B2. 3. **Install media:** the **reusable generic pairing ISO** (secret-free, slice C). Only rebuild it when `scripts/` ships a new version — never per-customer. Flash via dd/Rufus-DD. - Loader note per firmware: mkimage requires **Secure Boot OFF** (F1); record the board model + - loader choice in the customer record. + Loader per firmware — record the board model + loader choice in the customer record; full + fleet inventory in `operations/nodes.md`: + + | Board / firmware | Loader | Secure Boot | + |---|---|---| + | N100 (AMI AN3PLUS) | `mkimage` | **OFF** — that firmware can't relocate the signed GRUB (F1, `tests/VALIDATION-n100-baremetal-2026-07-16.md`) | + | **HP t740 (AMI M42)** | **`shim`** | **ENABLED** (proven 2026-07-21, `operations/nodes.md`) | + + **SB-off is a per-firmware workaround, not a Felhom requirement** — the shim loader boots with + Secure Boot enforcing on firmware that handles it (the HP proved it). 4. **Paper:** the tester one-pager (R-11, printed or PDF), the tester agreement (PETI-agreement sibling), and the recovery-code envelope step rehearsed in your head — the escrow ceremony is the one moment that must not be rushed. +5. **Agent train ≥ v0.93.0 on the customer's box before the FIRST real tester's escrow ceremony.** + v0.93.0 removes the four hyphenated EFF words from the recovery-code generator + (`felhom-agent/CHANGELOG.md` v0.93.0, 2026-07-21) — ceremonies run **box-side**, so the fix + protects a ceremony only once the train is **BUILT + DEPLOYED to that box**; source-only does + not count. Already-issued codes stay valid. ## B. Install + bind (~15 min wall-clock) @@ -45,18 +63,29 @@ **pairing-code banner**; the box appears under **Unclaimed appliances** at the hub (verify uuid/MAC/SSH fingerprints match the physical box before binding — never bind a fingerprint you didn't read off the console). + + > **Multi-NIC gotcha: verify the CABLED port got the DHCP lease during the install.** On the + > HP t740 the cable was on the 4-port expansion card, which gets **no lease** — and instead of + > aborting, the installer baked its `192.168.100.2` fallback as a **static** config and + > completed: a box that looked installed and could never call home + > (`operations/nodes.md` — the NIC map and the trap; filed as **R-59** hard-abort and **R-60** + > first-boot NIC sweep, both pending). Until those ship, on any multi-NIC board confirm at the + > installer's network step that the interface holding the lease is the one with the cable. 3. **Bind** to the customer — **the rehearsal used the SELF-BIND path, and it is now the default for - this runbook.** Mint the link with **"Send self-bind link"** on the customer's Setup tab; the + this runbook.** The link is already in the customer's inbox (auto-minted at creation, A1); the customer opens it and completes the public two-factor `/bind/` page (console pairing code + retrieval passphrase). The 2026-07-18 run bound in **60 s from mint to consume, with zero failed - attempts**, and the credential was delivered **26 s later** with no operator action. + attempts**, and the credential was delivered **26 s later** with no operator action. **Second + datapoint, virgin hardware:** the 2026-07-21 HP t740 onboarding walked the full chain — register + → pairing banner → unclaimed list → self-bind (code + passphrase) → day-0 → dashboard — on + hardware the system had never seen (`CONTEXT.md` 2026-07-21 entry, `operations/nodes.md`). - > **Send the self-bind link BEFORE the customer sees the console.** The console banner tells them - > to use „az e-mailben kapott link" — if the link has not been minted yet, the box is instructing - > the customer to look for something that does not exist. In the rehearsal the box sat in pairing - > mode for **~11.7 minutes** waiting on the operator, printing failures the whole time (R-33). - > Mint it during step A1 if you can; the token is valid for **7 days**, so there is no reason to - > wait. Auto-minting on customer create/RESET is R-36. + > **The self-bind link exists BEFORE the customer sees the console.** The console banner tells + > them to use „az e-mailben kapott link" — since hub v0.67.0 (R-36) the link is auto-minted at + > customer creation and RESET completion, so the banner is true by construction (in the + > pre-auto-mint rehearsal the box sat in pairing mode for **~11.7 minutes** waiting on the + > operator, printing failures the whole time — R-33). Verify the mint in A1; the token is valid + > for **7 days**. Operator-bind on the Hosts page remains the fallback when the customer is not present or the token has expired. @@ -79,10 +108,10 @@ | Drive enrol: destructive-confirm → usable (incl. mkfs) | **~30 s** | | Escrow ceremony started → offsite runs enabled | **12 s** | - Note the **PBS-DR descriptor** leg specifically: in the rehearsal it reported `converged` while - the credential behind it was already stale, leaving the DR tier authenticating `401` with nothing - surfacing the fault (**R-39**). Until that is fixed, treat "descriptor provisioned" as unproven - and check `pvesm status` on the box during step E1. + The **PBS-DR descriptor** leg no longer needs special distrust: the DR tier now self-detects a + dead credential (a loud `auth_failed`) and self-heals via damped re-issue, and the hub gauge is + trustworthy (**R-39 CLOSED, proven live 2026-07-21** — hub 0.68.1 + agent 0.91.2, the 13 s + re-issue chain `applied → auth_failed → applied`; `backlog/ROADMAP.md` R-39). ## C. Customer zero script (the customer drives, operator narrates — ~30 min) @@ -91,8 +120,9 @@ This is the friction lens: the customer's hands on the keyboard, the operator's 1. **Claim:** open the claim e-mail → claim the dashboard → set the dashboard password. (R-4 gate: the e-mail must land in inbox, not spam. **gmail PROVEN 2026-07-18** — the rehearsal's claim mail was the first sent under the tightened DMARC `p=quarantine` and landed in the **Inbox**. - **freemail.hu is still untested** — if your tester is on freemail.hu, treat step C1 as unproven and - have them check spam before you conclude the mail was lost.) + **freemail.hu PROVEN 2026-07-21** — an operator test-send was received at a freemail.hu address; + R-4 is COMPLETE (`backlog/ROADMAP.md` R-4). Keep the habit anyway: if a mail seems missing, have + them check spam before you conclude it was lost.) 2. **Escrow ceremony:** run the recovery-code wizard → the awaiting card („megerősítésre vár, legfeljebb 15 perc") → flips on ACK. The code goes on paper, into the envelope, into the customer's drawer — **state plainly: without this code, a dead box means @@ -101,19 +131,17 @@ This is the friction lens: the customer's hands on the keyboard, the operator's the destructive-confirm). Expect **~30 s** from the confirm to a usable drive — the wizard runs mkfs detached and polls, so a slow format is not a hang. -4. **Offsite:** enable „Távoli mentés". Two operator behaviours here are **workarounds for known - defects** — follow them until the items ship, and do not improvise: +4. **Offsite:** enable „Távoli mentés". One operator behaviour here is a **workaround for a known + defect** — follow it until the item ships, and do not improvise: > **Click Save exactly ONCE, then wait, then verify.** Provisioning runs synchronously inside the > request, so on a slow Hetzner leg the request can return an nginx **504 while succeeding > server-side**. A second click races the first attempt. Wait, then confirm on the hub's - > Offsite → Restic tab rather than believing the browser. (**R-31**) + > Offsite → Restic tab rather than believing the browser. (**R-31**, still open) - > **Re-onboarding an existing customer? Check offsite provisioning by hand.** After a RESET + - > re-enroll the hub can sit in "enabled but unprovisioned" and **say nothing** — provisioning is - > *Save*-triggered, and the re-enroll auto-re-issue deliberately skips unprovisioned targets. So - > an offsite target that looks configured may never have been created. Open the offsite settings - > and press Save once to force provisioning, then verify. (**R-36**) + The old re-onboarding manual-Save check is gone: since hub v0.67.0 (R-36 SHIPPED, + `backlog/ROADMAP.md`) an enabled-but-unprovisioned offsite target raises an **amber banner on + the customer page** naming the state and the fix — trust the banner. > **Expect the first run to be clean on a NEW customer.** If an **orphan card** appears > („a távoli mentési tároló elárvult"), the repo is not new — that is the R-32 case, which fires @@ -147,18 +175,17 @@ This is the friction lens: the customer's hands on the keyboard, the operator's - Sign the tester agreement. Leave the one-pager with them. - Operator record: board model + loader, drive serials, claim date, escrow ACK timestamp, friction notes. +- If this relationship ever ends, the levers are **RULED and shipped** (R-25b, hub v0.69.0/v0.70.0, + TASK-I): full teardown = the guided **DELETE cascade**; identity-preserving re-onboarding = + **RESET** (the Friday flow). Which lever and why: **§G below**. ## E. Operator post-checks (same day, remote) 1. Hub: customer row green; Offsite → Restic tab shows the first bytes within a report cycle; - PBS-DR gauge nominal (first PBS run on its own schedule — note, don't wait). - - > **Verify the DR tier on the box, not from the hub (R-39, interim).** The agent can report the - > PBS descriptor `converged` while the credential behind it is stale, and its own verify loop - > cannot read the credential to notice. Until R-39 ships, SSH to the host and run - > `pvesm status` — the `felhom-pbs` line must be **active**. A `401 Unauthorized` / `inactive` - > there means the DR tier is dead despite everything upstream looking green, and it will stay - > dead silently until someone needs a restore. + PBS-DR gauge nominal (first PBS run on its own schedule — note, don't wait). The gauge is + trustworthy: since R-39 closed (hub 0.68.1 + agent 0.91.2, proven live 2026-07-21 — + `backlog/ROADMAP.md` R-39), a dead DR credential surfaces as a loud `auth_failed` and self-heals + via damped re-issue; no manual `pvesm status` check is needed. 2. Floor convergence: controller at current, agent at MinAgent+. 3. Route friction notes: UX items → ROADMAP; one-pager corrections → R-11; runbook corrections → this doc. @@ -172,6 +199,7 @@ This is the friction lens: the customer's hands on the keyboard, the operator's - Post-rehearsal checklist item: rebuild the golden to the current controller before the FIRST real tester (fresh boxes shouldn't sit in a stale window longer than needed), and re-verify the Day-0 manifest after every controller release during alpha. + ## G. Offboarding — which lever to pull (hub v0.69.0, R-25b) Two levers, and they are not interchangeable: