diff --git a/documentation/architecture/00-capability-map.md b/documentation/architecture/00-capability-map.md index a9b8b2b..7f7ed31 100644 --- a/documentation/architecture/00-capability-map.md +++ b/documentation/architecture/00-capability-map.md @@ -31,12 +31,12 @@ |---|---|---|---|---| | Appliance day-0 install: golden image → first boot → auto-confirm (zero clicks) → claimable box | installer, agent, hub, golden | **PROVEN-LIVE** (nested VM) | `DRILL-day0-vm-2026-07-12`, `DRILL-day0-take2-2026-07-12` | First firing on real customer hardware pending → R-1 | | BYO install: `--mode byo`, mandatory caps, host-mutation disclosure, coexistence guards | installer v1.15+, agent | **PARTIAL** | `DRILL-GL6-2026-07-08` (demo box); GL-8 coexistence fixes | Peti clean-slate reinstall on proxmox2 is the first real BYO run of the current path → R-1 | -| Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`); selectable UEFI loader; **universal secret-free / operator-bind** mode | scripts v1.19.0 (`scripts/iso/`) + hub v0.62.0 + assistant container | **PARTIAL** (nested VM 310/311; live endpoints) | slice A `SPIKE-baremetal-iso-2026-07-16` (build gate, disk-filter fail-safe, stub→host-install fetch); slice B RUNBOOK-B (shim boots+installs OVMF SB-enforcing + SeaBIOS; `--loader mkimage` boots+installs SB-off; mkimage SB-enforcing **FAILS** `Access Denied`; surgery byte-identical); **slice C (2026-07-17): the GENERIC secret-free ISO** — box self-registers as an unclaimed appliance (`POST /api/v1/appliance/register`, one-shot poll delivery, 404-no-oracle — all live-verified through the public ingress), operator binds on the Hosts page, hub delivers credentials once; bootstrap harness proves direct(zero-appliance-calls)/pairing/delivery; artifact proven secret-free (baked env = hub URL only) | **F1 loader caveat:** `--loader mkimage` fixes cheap AMI firmware that can't USB-boot the stock GRUB — UNSIGNED → **Secure Boot must be OFF**; default `shim` keeps SB. **Slice C bind is operator-password-gated** (CC stages, Viktor binds) → the live boot→register→bind→day-0 composition + physical N100 boot fold into the supervised rehearsal (R-1). Customer-facing **self-bind page = R-27 slice 1 SHIPPED (hub v0.66.0, 2026-07-17)** — see the dedicated self-bind row | -| Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) | -| Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **IMPLEMENTED** | hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run + new-ISO console banner pending** (folds into the supervised rehearsal R-1). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends | -| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | Endpoints driven on the drill VM; customer-facing **browser wizard** leg not yet live-validated. First supervised ceremony with a real customer pending → R-1. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody` | -| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **IMPLEMENTED** | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected. Candidate upgrade to PROVEN-LIVE — see REPORT.) **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant | -| Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown)** | hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Not live-clicked:** the web POST is password-gated (CC verifies via render + httptest orchestration tests); the Hetzner sub-account delete is unit-tested + a faithful mirror of the live-proven `ReissueCredentials`. **Consistency gap → R-25b:** the Danger-zone DELETE leaves host rows and doesn't run this teardown | +| Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`); selectable UEFI loader; **universal secret-free / operator-bind** mode | scripts v1.19.0 (`scripts/iso/`) + hub v0.62.0 + assistant container | **PROVEN-LIVE** (physical N100, one pass, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — the full chain on real metal in a single pass:** the generic reusable pairing ISO (v1.20.0, `--loader mkimage`, SB off) booted the cheap AMI board that F1 had blocked, installed unattended, and the box **self-registered as an unclaimed appliance at 16:17:14 — the same second it first booted** (`appliance_registrations` id=3), then bound → credential-delivered → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware.** Prior nested legs: slice A `SPIKE-baremetal-iso-2026-07-16` (build gate, disk-filter fail-safe, stub→host-install fetch); slice B RUNBOOK-B (shim boots+installs OVMF SB-enforcing + SeaBIOS; `--loader mkimage` boots+installs SB-off; mkimage SB-enforcing **FAILS** `Access Denied`; surgery byte-identical); **slice C (2026-07-17): the GENERIC secret-free ISO** — box self-registers as an unclaimed appliance (`POST /api/v1/appliance/register`, one-shot poll delivery, 404-no-oracle — all live-verified through the public ingress), operator binds on the Hosts page, hub delivers credentials once; bootstrap harness proves direct(zero-appliance-calls)/pairing/delivery; artifact proven secret-free (baked env = hub URL only) | **F1 loader caveat:** `--loader mkimage` fixes cheap AMI firmware that can't USB-boot the stock GRUB — UNSIGNED → **Secure Boot must be OFF**; default `shim` keeps SB. **Slice C bind is operator-password-gated** (CC stages, Viktor binds) → the live boot→register→bind→day-0 composition + physical N100 boot fold into the supervised rehearsal (R-1). Customer-facing **self-bind page = R-27 slice 1 SHIPPED (hub v0.66.0, 2026-07-17)** — see the dedicated self-bind row | +| Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. **Deliverability (R-4), gmail half DONE 2026-07-18:** the rehearsal's claim email was the first sent under the tightened DMARC `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu remains Viktor's open half.** (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) | +| Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** (real customer-zero bind on metal, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md`:** operator minted + emailed the link 16:28:55 (7-day TTL, expiry 2026-07-25 recorded); **the customer bound their own box at 16:29:55 with `attempts=0`, `locked=0`** — `appliance_bound` carries source **`customer_selfbind`**, and the credential was delivered **26 s later** with no operator action. Hub-side lifecycle in `hub-state.txt` (`selfbind_tokens` mint→email→consume). Prior unit evidence: hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run DONE 2026-07-18** (rehearsal; the console banner rendered on the real ISO). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends | +| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody` | +| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **IMPLEMENTED** | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant | +| Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **Consistency gap → R-25b:** the Danger-zone DELETE leaves host rows and doesn't run this teardown | | Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 | ## B. Apps & catalog @@ -57,7 +57,7 @@ |---|---|---|---|---| | Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | `CAMPAIGN-2` T-BAK-FULL (pg+mariadb autodiscovered); atomicity `CAMPAIGN-6B` P4 + `CAMPAIGN-6E` B1/B2 (SIGKILL mid-write → only `.tar.tmp` touched, last-good byte-unchanged); DB restore `CAMPAIGN-6D` P-FAB | (Cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes, and T-RST-DB was auth-hollow — corrected to the 6B/6E fix-proofs.) T-6E-1 dir-fsync asymmetry (LOW) → R-10 | | Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C` | | -| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs; live leg staged for the delete/re-create rehearsal (a scratch-target swap disturbs the live escrow state). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` | +| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs. **The live leg FIRED on its own during the 2026-07-18 rehearsal** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S7): after a RESET + re-enable, the first offsite run hit the previous lifecycle's ciphertext and the guard **classified it, pushed `offbox_repo_orphaned`, skipped the run and showed the card (16:58:14)** rather than nightly-spamming a raw restic error; the operator-confirmed reset then **moved the repo aside (never deleted) to `.orphaned-20260718` and re-initialised (16:59:26→16:59:32)**, and the next run produced 2 snapshots / 48.717 MiB. The guard behaved exactly as designed — **the finding is that it had to fire at all** (R-32: RESET destroys custody, so the ciphertext it leaves behind is dead by design and should be purged, while the move-aside guard stays correct for reinstall-WITHOUT-RESET). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` | | Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone) | | | Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D | | Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill | @@ -109,9 +109,9 @@ | Scenario | Components | Status | Evidence | Gap / roadmap | |---|---|---|---|---| | Customer/host management: 8-tab detail, scoped auto-refresh, safe stale-host deletion, capability chips | hub v0.47–0.53 | **PROVEN-LIVE** | hub v0.53.0 dead-host roll-up live on the Peti cluster (proxmox1 down 23h); `CAMPAIGN-4-2026-07-13` (operator UI driven live); `DRILL-day0-take2` F-16 (offsite/freeze buttons live) | 8-tab render + capability chips are **render-test-validated** (hub UI is password-gated; CC cannot log in). (Cited "daily operator use" was a no-doc citation; `AUDIT-hub-gui-2026-06-30` predates these features at hub v0.25) | -| Config/state change round-trips in **seconds** (hub↔box immediacy; 15-min cycle stays the backbone): box→hub out-of-cycle report (Dir 1) + hub→box `GET /api/v1/wait` long-poll wake (Dir 2) | controller v0.139/140, hub v0.58/0.63 | **PARTIAL** | Transport proven live through the real DNS-only ingress: `SPIKE-immediate-sync-transport-2026-07-16` + hub v0.58.0 / controller v0.140.0 REPORTs — 240 s no-annotation hold (25 s heartbeat defeats nginx's 60 s `proxy_read_timeout`, no ingress change), 0.047 s wake-on-change, hub `rollout restart` = 1 WARN + 0-storm reconnect; Dir-1 2 s box→hub round-trip live in controller v0.139.0 | The operator-UI **save→apply** round-trip is not fired end-to-end live (hub UI password-gated; CC can't log in) → R-23; the wake transport and the ACK→config_version→`ConfigRefresher` delivery chain are each proven, only the UI-triggered bump leg is unexercised. Agent-plane (host-domain desired-state) poke **first slice PROVEN-LIVE** (Direction-2a, agent v0.89.0 + hub v0.59.0, 2026-07-17): contentless ep0-relayed UDP poke → agent immediate desired-state cycle, per `SPIKE-immediate-sync-transport-2026-07-16` P4. Full path live-proven: a real operator manifest save fired `poke: sync-poke delivered to 10.77.0.2`; the box (0.89.0) received it and logged `poke received → triggering an immediate desired-state cycle` → `out-of-band report triggered` — **~31 ms ep0→box, sub-ms to the report cycle** (WG-confined, from 10.77.0.1 to the 10.77.0.2-bound socket); save→tick ≈ ~0.45 s (SSH-dominated), well under ≤2–3 s. R-13 first slice (listener+sender only; the rest of the mutual-repair arc stays open). **System-initiated immediacy wired (hub v0.63.0, this REPORT):** the mutation sites that only OPERATOR actions used to notify now fire the correct plane's notifier when the hub itself mints state — agent-plane pokes at `PBSDRAutoProvision` (the observed slice-C lag), `ReissuePBSDR` (also the pbsdrheal escalation), `handlePBSDRReissue`, and the two admin desired-state api writers; controller-plane bump at `reissueOnReenroll`. Unit-tested + red-proofed, not yet fired on a real system event (folds into the rehearsal bind sequence). Still PARTIAL: the R-23 operator-UI save→apply leg and the agent **fast-tick-until-first-convergence** SECONDARY (the WG-registration leg a poke can't reach pre-tunnel) remain unfired live. **Fast-tick SHIPPED (agent v0.90.0, R-28):** while any desired-state item is unapplied — incl. the pre-tunnel window a poke can't reach — the agent pulses the out-of-band trigger every 30 s and self-disarms on convergence (state-based; four cached sources; LOUD states excluded). LIVE on both demo agents (the `fast-tick armed: 30s …` startup line verified); its real-onboarding value (the pre-tunnel window shrinking ≤15 min → ~30 s) is unfired live — folds into the rehearsal S5 bind sequence | -| Customer right-sizes guest RAM from the controller (agent-enforced bounds, live cgroup apply, no reboot) | agent v0.90.0 + controller v0.143.0 (R-24) | **IMPLEMENTED** (→ PROVEN-LIVE pending a real grow) | Controller-direct (R-24's hub-desired-state framing SUPERSEDED, Viktor 2026-07-17). Agent `GET`/`POST /guest/memory` enforces every bound FRESH (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)) + verify-after-apply; PVE `SetConfig` hot-applies (**Phase-0 PROVEN** on the nested box: maxmem moves with the guest running, /proc/meminfo ripples via lxcfs, no reboot). Controller "Szerver memória (RAM)" card + code→Hungarian map, gated on `FeatureGuestMemoryResize` (MinAgent 0.90.0). LIVE-validated end-to-end through the real endpoint on the demo (above_max + below_min refusals render the Hungarian, agent English never leaks; SupportYes via the version header) | A **successful** grow/shrink through the bounded endpoint is unfired live — the 4 GB nested demo host is too small (max **Sub-rank `P2-HIGH` = close before the first REMOTE tester.** These are the 2026-07-18 N100 +> rehearsal's findings (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). They are not P1 — the +> rehearsal proved the product flow works — but each one either misleads the operator, misleads the +> customer, or hides a failure, and all of that gets materially worse the moment the box is somewhere +> you cannot walk over to. + | ID | Item | Size | Status | Notes | |----|------|------|--------|-------| +| R-39 | **[P2-HIGH] The PBS DR tier can be `applied` and dead at the same time — and nothing notices.** On the reborn N100 the descriptor auto-provisioned and the agent converged `state=applied`, yet `pvesm status` reports `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive` and a direct probe with the stored credential 401s on **every** endpoint including `/version` (WG transport healthy: handshake 9 s, 27.9 ms RTT — so authentication, not ACL scope). Three compounding defects: **(a)** a **mint/consume race** — the hub minted a SECOND token secret at 16:47:52, two minutes *after* the agent applied the first, and `consumed_at` is still NULL; **(b)** the converged state machine will not re-apply, so the box is pinned to a stale secret; **(c)** the agent's 15-minute PBS verify loop **cannot read the credential to detect any of it** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — the non-root agent writes that file through a root sudo wrapper, then reads it directly). | M | idea | **NEW — discovered by CC while collecting Phase-A evidence; not on the brief's finding list, so its P2-HIGH rank is provisional pending Viktor.** The severity case: this is the DR tier, the failure is silent, and it would surface first at a real restore. Suggested shape: make `consumed_at` authoritative (a fresh unconsumed secret must un-converge the reconciler), fix the verify loop's read path (read via the same root wrapper that writes it), and make a failing `pvesm status` a LOUD state rather than a skipped datastore. **Blocks the DR-tier map row's candidate upgrade to PROVEN-LIVE — that upgrade is now explicitly WITHDRAWN.** Evidence `pbs-dr-state.txt`, `hub-state.txt` | +| R-30 | **[P2-HIGH] Liveness presence should come from the wait channel, not the report clock.** The box was powered off at the start of the rehearsal, yet the hub carried it as healthy until the staleness threshold expired ~30 min later (`host_stale` 16:05:24 "no report for 30m"; cleared 16:33:24 "was stale for 27m"). The host-delete guard compounds it: RESET refuses while any host row exists, so a stale-but-"Online" host stalls a forced teardown. | M | idea | Direction: derive presence from **Dir-2 long-poll connectedness (~90 s grace)**, decoupled from notification hysteresis (the hysteresis is right for *alerting*, wrong for *presence*); an agent/ep0 analog can follow. Pairs with R-13/R-23 — the transport already exists, this is about believing it. *(Discussed in-session as "R-29"; that number was already taken by the gate-rot item earlier the same day, so it is R-30.)* | +| R-31 | **[P2-HIGH] Offsite provisioning is synchronous with no status affordance.** Save runs the Hetzner sync in-request, so the request can hit the nginx 504 **while succeeding server-side**: the operator cannot tell failed from slow, and a retry races the first attempt. | M | idea | Direction: make it async + a status card, reusing the proven **awaiting-card/poll idiom** (v0.138.0 escrow card). **Interim mitigation belongs in R-3 as an operator note: click once, wait, verify — do not re-click.** | +| R-32 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. | M | idea | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` | +| R-33 | **[P2-HIGH] Bootstrap pairing-poll spams the customer-visible console.** Each 30 s pairing poll is a separate `felhom-bootstrap.service` run that exits 1, so systemd prints `Failed to start …` on the screen the customer is watching — **52 FAILED lines in ~11 minutes** while nothing was wrong (the box was correctly waiting to be bound). | S | idea | Fix in **scripts v1.21**: poll *inside* the script rather than via unit restarts, and reword the first-boot noise. Cheap, and it is the customer's literal first impression of the product. Evidence `bootstrap-journal.txt` | | R-5 | Hub: offsite storage visibility — RESTIC box aggregate (v0.64.0) + PBS DR datastore (v0.65.0), each with fill + operator alert; Offsite tabs + dual dashboard gauges | M | **SHIPPED (hub v0.64.0 + v0.65.0 + tenantsync v1.2.0, 2026-07-17)** | Restic: `monitor.OffsiteBoxChecker` (throttled Hetzner GET; fill 80/90% + oversub 2.0×; scope `"pool-box"`). PBS DR (v0.65.0, Option A): read-only `usage` op on the ep0 `felhom-tenantsync` forced command (`df`, no mutation) → `monitor.PBSDRBoxChecker` (fill 80/90%, states ok/unavailable/degraded, scope `"pbsdr-box"`); `/offsite` Restic/PBS-DR tabs + two gauges; hub deploy ⟂ ep0 update (graceful "n/a"). **Both data pipelines PROVEN-LIVE** (restic `0.2% full`; PBS `19.1% full, 7.1 GB of 37.2 GB` after ep0 → v1.2.0). Thresholds pending Viktor (named keys; separate restic/PBS pairs). OPEN: UI render unit-only (password-gated), alert emails unit + red-proof only (both nominal). See hub v0.64.0 + v0.65.0 REPORTs | | R-6 | **Spike: LAN service discovery from the guest** — SSDP multicast (UDP 1900, DLNA), WSD (Windows discovery), mDNS; host-network vs macvlan; is the customer LXC LAN-bridged in appliance deployments? | M | **spiked (2026-07-18)** | **VERDICT: appliance guest IS LAN-bridged (own DHCP lease on the household /24); multicast discovery works ONLY in the guest netns — guest-direct or Docker `--network host` (SSDP/mDNS/WSD all PASS both ways); the default docker bridge is categorically DEAF to LAN multicast (WSD/mDNS RX FAIL, unicast-publish PASS). Real samba+wsdd on host-net → Windows 11 ProbeMatch + FELHOM-SPIKE renders in Explorer + 445 + authenticated SMB round-trip all PASS; real SSDP `MediaServer:1` advert reaches both LAN clients. → R-7 SMB stack MUST be host-network LAN-bound; R-8 Jellyfin-DLNA plausible if host-network. Caveat: `vmbr0 multicast_snooping=1` worked only because the household router is a live querier — customer LANs w/ snooping+no-querier, and Peti's BYO bridge, are UNTESTED gaps.** **S4b (human leg, the sharpest finding): wsdd makes the box VISIBLE but the Explorer double-click FAILS `0x80070035` — WSD gives no name resolution; the flat `\\FELHOM-SPIKE` resolved by no path. Adding `nmbd` (NetBIOS) fixed it live (flat name resolves + mounts). → R-7 needs smbd+wsdd+nmbd (+avahi/.local for modern clients), not wsdd alone.** Doc: `audits/SPIKE-lan-discovery-2026-07-18.md`. | | R-7 | SMB **server** share — LAN file sharing as an embedded controller feature („Megosztás") | L | **SHIPPED slice 1 (controller v0.144.0 + `felhom-samba:1.0.0`, 2026-07-18)** | Flips map row *Files from Windows Explorer / Mac Finder (SMB server)* MISSING → **IMPLEMENTED**. Delivered as the FOURTH protected infra stack (host-net, smbd+**nmbd**+wsdd, `bind interfaces only = lo eth0`), NOT a catalog app. „Megosztás" → „Hálózati megosztás": enable + ONE household password (STDIN→smbpasswd, never persisted — only `user_set`), shares table, create flow (new folder under `/shares/` or an existing folder via a guarded picker). Picker guard = absolute → EvalSymlinks → containment in a registered LIVE root → deny-listed system subtree (derived from `ProtectedHDDPaths`, provably a subset) → is-a-dir, with UNIFORM refusals (no filesystem oracle). Per-share read-only enforced in smb.conf AND as a `:ro` bind. `force user = felhom` → SMB writes land as uid 1000. Live-validated on demo through the real endpoints + a Win11 workstation (445 True, NetBIOS `FELHOM` resolves, write/read byte-compare PASS, **write to a read-only share REFUSED**, WSD ProbeMatch). **Explorer leg PASSED 2026-07-18 (Viktor):** both shares open from the Network view; an Explorer save into the writable share landed as uid 1000 and a write into the read-only share was refused, folder untouched. Slice 1 is fully PROVEN-LIVE. Docs: `controller/sharing.md`; ship report `felhom-controller/REPORT.md`. **Slice 2 remainder:** avahi/`.local` for modern+Apple clients, curated app-folder presets (one-click paperless `consume/`), per-share users + guest access, recycle bin. **Backup gap split out → R-7b.** | @@ -43,24 +54,43 @@ | R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE | | R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) | | R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, shipped 2026-07-16): (a) live-validate the operator-UI **save→apply** round-trip end-to-end — needs an operator login, so fold into a Peti/alpha supervised session (one config save → box wakes in seconds → self-restart → startup report; also demonstrates the restart *single-fire* once a bump has advanced the generation past 0); (b) cosmetic: the controller Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect — surface recovery at connect time | S | idea | Flips the new map row "config/state change round-trips in seconds" PARTIAL→PROVEN-LIVE. Transport + mechanism already proven live (`SPIKE-immediate-sync-transport-2026-07-16`, v0.58/v0.140 REPORTs: 240 s no-annotation hold, 0.047 s wake, restart 1-WARN/0-storm); only the login-gated UI-triggered bump leg is unexercised | +| R-34 | **Backup data lifecycle management.** An "inactive backups" section on „Távoli mentés": apps that have snapshots but no active backup — **disabled OR uninstalled** — listed with name / size / last snapshot / restorable, plus an explicit **double-confirmed per-app delete** via `restic forget --tag` + nightly prune. | M | idea | **RULING: the offsite toggle NEVER offers deletion — policy and destruction stay decoupled.** Turning backups off must never be a data-destroying act, and deletion must never hide behind a toggle. Origin: 2026-07-18 rehearsal. Pairs with R-32 (that one is the operator's view of dead bytes; this one is the customer's) | +| R-35 | **Config-apply should not end the customer's session.** The offsite config push bumped `config_version` 10→11 at 16:54:58 and the controller self-restarted (container `StartedAt` 16:54:59Z, back up 16:55:02); in-memory sessions died with it and **customer zero was force-logged-out mid-flow**. | S | idea | Direction: **hot-apply the offbox target** (no restart for a config the running process can adopt), or **persist sessions** across restart. The restart itself is by design — the collateral is not. Evidence `controller-log-full.txt` | +| R-36 | **Post-RESET re-enroll leaves offsite "enabled but unprovisioned" — silently.** The hub knows the state and says nothing on the customer page. | S | idea | Source-cited behaviour, confirmed live in the rehearsal: **provisioning is Save-triggered** (`configs.go` `applyOffsite`) — which also answers S6's open question — and the re-enroll auto-re-issue **correctly** skips unprovisioned targets (`handler.go`). So nothing is broken; the gap is that nobody is told. Direction: flash it on the customer page. **Interim: an R-3 step.** **Related sub-item:** auto-mint the **self-bind link on customer create/RESET**, so the console banner's „e-mailben kapott link" is always already true instead of true-once-the-operator-remembers | +| R-27c | **Customer self-bind, slice 2 — console-passphrase bind.** Viktor's direction: bind using a passphrase shown on the box console, alongside (not instead of) the emailed capability link. | M | idea | **Security constraints from the session ruling, all load-bearing:** passphrase **issued at customer creation**; the global-lookup endpoint must be **spray-hardened** — per-appliance **and** per-IP caps, constant-time comparison, a **single generic failure** (no oracle), alerting on abuse; an **accent-free wordlist** (console keymaps are not Hungarian); the **web capability-link path is RETAINED**; **claim-by-email is RETAINED** as the delivery-channel proof. **Also under this item:** the self-bind email gains the **public universal-ISO download link + two-line instructions** (the DIY case). **Secret-bearing per-customer ISOs are ruled OUT.** Sibling of R-27b (second-box flow) — different axis, both build on the same `/bind/` page | ## P3 — post-alpha | ID | Item | Size | Status | Notes | |----|------|------|--------|-------| | R-26 | **Guided old-history recovery via a retained superseded escrow + the recovery code.** Enabled by hub v0.60.0 (Part B) which now RETAINS superseded escrow blobs (`host_escrow_superseded`, `ListSupersededEscrow`). Build the flow that, given the customer's recovery code, unwraps a retained old blob → recovers the old repo passphrase → mounts/reads the moved-aside `.orphaned-` repo for restore. | M | idea (enabled by v0.60.0) | Turns "history recoverable in principle" into a real customer-drivable path; pairs with the controller v0.142.0 orphaned-repo move-aside. Origin `DIAGNOSE-offbox-repo-orphaned-2026-07-17` | -| R-27 | **Customer-facing self-bind page (R-21 slice C follow-on).** Today an unclaimed appliance is bound by the OPERATOR on the Hosts page (hub v0.62.0). Build the customer-facing flow so a customer can claim/bind their own freshly-installed box. | M | **SHIPPED (slice 1, hub v0.66.0, 2026-07-17)** | **Slice 1 = the FIRST-box flow, DONE:** operator "Send self-bind link" → 7-day tokenized capability link over Hungarian email → public two-factor `/bind/` (console pairing code shown on the box screen + retrieval passphrase) → hub stages the bind via the same `BindAppliance`, no operator. Viktor's 3 rulings verbatim (console code not a list; operator-sent link; 5-attempt lockout→call support). No oracle; expiry falls back to operator-bind; THE TRAP `/bind/` exemption tight (single predicate, 4 red-proofs). GC verdict §3 (no appliance GC → the 7-day TTL stands alone). **Live first-run + new-ISO console banner fold into the supervised rehearsal (R-1).** Reused the appliance_registrations + one-shot delivery machinery; the capability link is the whole customer-auth surface (no login system built). Controller/agent untouched. See hub v0.66.0 REPORT. **Multi-box per link is out of scope by design** (single-active, one-shot → binding several boxes = repeated operator sends) | +| R-27 | **Customer-facing self-bind page (R-21 slice C follow-on).** Today an unclaimed appliance is bound by the OPERATOR on the Hosts page (hub v0.62.0). Build the customer-facing flow so a customer can claim/bind their own freshly-installed box. | M | **SHIPPED (slice 1, hub v0.66.0, 2026-07-17)** | **Slice 1 = the FIRST-box flow, DONE:** operator "Send self-bind link" → 7-day tokenized capability link over Hungarian email → public two-factor `/bind/` (console pairing code shown on the box screen + retrieval passphrase) → hub stages the bind via the same `BindAppliance`, no operator. Viktor's 3 rulings verbatim (console code not a list; operator-sent link; 5-attempt lockout→call support). No oracle; expiry falls back to operator-bind; THE TRAP `/bind/` exemption tight (single predicate, 4 red-proofs). GC verdict §3 (no appliance GC → the 7-day TTL stands alone). **Slice 1 PROVEN-LIVE 2026-07-18:** in the rehearsal the operator minted + emailed the link at 16:28:55 and **the customer bound their own box at 16:29:55 with zero failed attempts**, `appliance_bound` carrying source `customer_selfbind`, credential delivered 26 s later with no operator action (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). The new-ISO console banner rendered on the real box. **Slice 2 = console-passphrase bind → R-27c** (below). Reused the appliance_registrations + one-shot delivery machinery; the capability link is the whole customer-auth surface (no login system built). Controller/agent untouched. See hub v0.66.0 REPORT. **Multi-box per link is out of scope by design** (single-active, one-shot → binding several boxes = repeated operator sends) | | R-27b | **Customer self-bind, second-box flow (controller side).** For a customer who ALREADY has a bound box and installs another, the controller shows a dismissable "bind another box" prompt (and a bind-later entry under settings) that walks to the hub `/bind/` page — so a returning customer isn't emailed a fresh operator-sent link for every box. Mechanism sketched in the hub v0.66.0 REPORT; NOT built (R-27 slice 1 deliberately did not touch the controller). | M | idea (minted by hub v0.66.0) | Origin: hub v0.66.0 slice-1 ship (first-box only). Reuses the same `/bind/` public page + tokenized-link machinery; adds a controller-side entry point + the operator "mint a link for an existing customer" affordance | | R-25b | **Customer DELETE ↔ RESET consistency.** The middle-tier Customer RESET (hub v0.61.0) runs the full external teardown (Hetzner sub-account/box + PBS namespace/groups/token) and refuses while any host row exists. The Danger-zone DELETE still (a) leaves host rows and (b) does NOT run that teardown — it purges escrow custody + drops the config only. Decide the model: DELETE requires a prior RESET, or DELETE subsumes RESET's teardown, or they stay orthogonal (RESET = recycle-in-place, DELETE = escrow-purge). | S | idea | Origin: hub v0.61.0 RESET ship. Flips a future "customer fully offboarded (external resources released)" map row. Cheap once the model is chosen | | R-25 | **Device-node TOCTOU hardening (drive init).** Graduate the controller v0.141.0 Observation: the `format → resolveEnrollUUID(path) → AssignDisk(uuid)` sequence has a narrow /dev-re-enumeration window (agent-guarded on the destructive format via anti-retarget durable-id; benign fs-UUID mount). Bind resolve+assign to the format's durable-id so the mount can't target a moved node. | S | idea | From the v0.141.0 F6 commit's security-review finding (`felhom-controller` REPORT). Low real risk (single-operator, agent-guarded), but cheap to close | -| R-24 | **Guest RAM resize (live) — SHIPPED (agent v0.90.0 + controller v0.143.0, 2026-07-17).** The customer right-sizes the guest's memory from the controller's Rendszer page; the agent enforces every bound fresh + applies via PVE `SetConfig` (live cgroup, no reboot, Phase-0 proven). **Framing note:** the original hub-desired-state framing is SUPERSEDED by Viktor's controller-direct ruling (2026-07-17) — the resize is controller→agent local-API, never through the hub. Memory only (cores stay observation). Deployed + live-validated on the demo (refusals render Hungarian end-to-end); a successful grow on a normal-sized box folds into the rehearsal. **Cores/live-resize as hub desired-state is NOT built** (deferred, low demand). | M | **SHIPPED** | See felhom-agent + felhom-controller REPORTs; capability-map row "customer right-sizes guest RAM". | +| R-24 | **Guest RAM resize (live) — SHIPPED (agent v0.90.0 + controller v0.143.0, 2026-07-17).** The customer right-sizes the guest's memory from the controller's Rendszer page; the agent enforces every bound fresh + applies via PVE `SetConfig` (live cgroup, no reboot, Phase-0 proven). **Framing note:** the original hub-desired-state framing is SUPERSEDED by Viktor's controller-direct ruling (2026-07-17) — the resize is controller→agent local-API, never through the hub. Memory only (cores stay observation). **COMPLETE — the apply is proven in BOTH directions on metal (2026-07-18):** customer zero shrank the guest **11675 → 8192 MB (16:50:22)** and grew it **8192 → 12288 MB (17:02:17)**, each a live cgroup apply with no reboot, the new total rippling into the deploy page's memory math. The nested demo had proven the refusals; the N100 proved the applies. `tests/VALIDATION-n100-rehearsal-2026-07-18.md` ledger 9. **Cores/live-resize as hub desired-state is NOT built** (deferred, low demand). | M | **SHIPPED + PROVEN-LIVE** | See felhom-agent + felhom-controller REPORTs; capability-map row "customer right-sizes guest RAM". | | R-12 | Cluster mode: agent-follows-guest, bind-mount reconciliation on HA migration | XL | idea | Scoped 07-15; interim = HA-group pin to one node. Driven by Peti's two-node cluster | | R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | **first slice PROVEN-LIVE (poke channel)** | **FIRST SLICE PROVEN-LIVE — the agent-plane poke channel (Direction-2a), agent v0.89.0 + hub v0.59.0 (2026-07-17):** the ep0-relayed contentless poke (hub→ep0 `felhom-poke` forced-cmd→UDP→box WG /32:51822, peer-confined, zero ep0/box infra change) reaches the **agent** and fires an immediate desired-state cycle. Full path live: real operator manifest save → `sync-poke delivered to 10.77.0.2`; box → `poke received → immediate desired-state cycle` (~31 ms ep0→box, save→tick ≈ ~0.45 s). This is ONLY the listener+sender; the rest of the mutual-repair arc (self-heal actions over the channel) stays open. Per `SPIKE-immediate-sync-transport-2026-07-16` P4. The controller-plane Direction-2 wait channel (hub v0.58 / controller v0.140) shipped the config-puller leg separately | | R-28 | **Agent fast-tick-until-first-convergence — SHIPPED (agent v0.90.0, 2026-07-17).** Closes the immediacy leg a hub poke can't reach (the pre-tunnel WG-registration window): while ANY desired-state item is unapplied the agent pulses the out-of-band trigger every 30 s and **self-disarms** on convergence (state-based; four cached sources — desired-gen==0, reconcile Planned−Pending>0, pbsdr waiting_secret ONLY, wgtunnel desired-not-operational; LOUD states + pending_signature excluded). LIVE on both demo agents (startup line verified). **Guests-0/0 passenger:** diagnosed live and REFUTED as a bug — the 0/0 was the legitimate pre-provision reporting window; the fast-tick is exactly its mitigation (≤15 min → ~30 s). **Unfired live:** the real-onboarding proof (a fresh box converging in seconds without a poke) → the rehearsal S5. | M | **SHIPPED** | See felhom-agent REPORT (v0.90.0). Third immediacy leg alongside R-13 (poke) + R-23 (UI). | | R-14 | Headscale/WireGuard spike: Minecraft/gaming port connectivity (CGNAT-proof, sovereign DERP fallback) | M | idea | | | R-15 | Multi-user dashboard accounts (household members, roles) | L | idea | Single password is a stated alpha limitation (R-11) | -| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED (slices A+B+C, 2026-07-17)** — physical N100 boot + the live boot→bind→day-0 composition fold into the supervised rehearsal (R-1) | **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | -| R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v :/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff | +| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* | +| R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v :/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** | + +| R-37 | **Post-RESET health card shows stale pre-RESET warnings.** After a RESET the card should read **„RESET óta nincs adat"** instead of carrying warnings about a lifecycle that no longer exists. | XS | idea | Origin: 2026-07-18 rehearsal. Same family as R-36 — the hub knows the state changed and the UI has not caught up | +| R-38 | **Installer GRUB slice.** A single default „Felhom telepítés" entry; the **interactive installers REMOVED** (safety: an interactive entry is how a wrong-disk manual install happens); felhom background. | S | idea | Origin: 2026-07-18 rehearsal, alongside R-21's physical closure. **Squashfs/theme rebranding explicitly DEFERRED** — this item is the menu and the safety, not a skin | + +## Pre-invite checklist — what stands between here and the first remote tester + +Not roadmap items in their own right; the short list the 2026-07-18 rehearsal leaves behind. +Everything here is **remote-doable** — the N100 is packed, and none of it needs hands on the box. + +| Action | Owner | Note | +|---|---|---| +| **Rebuild the golden → 0.145.x** | Viktor + CC train | The rehearsal banked the floor-lift evidence (0.143.0→0.145.0 unattended in 5 s), so the mechanism is proven — but a fresh box should not *start* two versions stale and then self-update in front of the customer. Do this before anyone is invited | +| **freemail.hu test-send** | Viktor | The open half of R-4; the gmail half closed on 2026-07-18 under `p=quarantine` | +| **C6 — customer performs a restore, unassisted** | Viktor as customer zero | The one open script step in R-3 and still MISSING as capability evidence. Remote-doable on the reborn box — the dashboard is remote | +| **R-11 rulings** | Viktor | Contact channel, tester agreement, alert thresholds (the R-5 gauge thresholds are still pending a ruling) | ## Absorbed / superseded notes in this folder diff --git a/documentation/runbooks/RUNBOOK-onboarding-draft-v3.md b/documentation/runbooks/RUNBOOK-onboarding-draft-v3.md index 6de6025..dcdb1fa 100644 --- a/documentation/runbooks/RUNBOOK-onboarding-draft-v3.md +++ b/documentation/runbooks/RUNBOOK-onboarding-draft-v3.md @@ -1,17 +1,32 @@ # DRAFT R-3 — Friend-alpha onboarding runbook (operator) -> **Status: DRAFT, pre-rehearsal.** Generalized from `runbooks/RUNBOOK-rehearsal-v3.md` (S1–S8) and -> the shipped claim/escrow/self-bind flows. The N100 rehearsal EXECUTES this document; every -> `[REFINE]` slot gets filled from its friction notes, then this graduates to -> `documentation/runbooks/RUNBOOK-onboarding.md` and the rehearsal item collapses. +> **Status: DRAFT, post-rehearsal — all `[REFINE]` slots filled 2026-07-18.** Generalized from +> `runbooks/RUNBOOK-rehearsal-v3.md` (S1–S8) and the shipped claim/escrow/self-bind flows, then +> refined against the N100 rehearsal that executed it +> (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). Every step here has been walked on real +> hardware **except C7**, the customer-performed restore. +> +> **It graduates to `RUNBOOK-onboarding.md` when C7 has evidence** — that step is simultaneously the +> last unwalked step of this script and the capability map's last MISSING customer-facing row, so +> one execution closes both. Until then this stays a DRAFT. +> > Scope: **appliance hardware only** (BYO track is R-1/Peti, explicitly out of alpha scope). +> Three steps below carry **interim workarounds for known defects** (R-31, R-36, R-39) — they are +> marked inline and should be deleted from this runbook the moment those items ship. ## A. Before the visit (operator, ~30 min, all remote-doable) 1. **Hub:** create the customer (name, e-mail, domain `.felhom.eu`) — verify the CF tunnel + zone token provisioning completes. Confirm the Day-0 manifest is current (agent, golden, MinAgent) and the managed-update floor is at the intended release — a stale golden is fine - (the floor lifts the box post-install), a stale FLOOR is not. + (the floor lifts the box post-install), a stale FLOOR is not. *The rehearsal proved the lift: + a 0.143.0 golden reached 0.145.0 in **5 s**, unattended, inside the first minute of controller + life.* Even so, **rebuild the golden before the first real tester** — proven-recoverable is not + a reason to ship stale. + + **Mint the self-bind link now** ("Send self-bind link" on the customer's Setup tab). It is valid + for **7 days**, and doing it here means the console banner's „e-mailben kapott link" is already + true when the customer first looks at the screen. See B3. 2. **Hardware:** the standard appliance + its external HDD. One-time bench prep list: memtest pass, SMART clean, BIOS queue for the single visit in B2. 3. **Install media:** the **reusable generic pairing ISO** (secret-free, slice C). Only rebuild it @@ -30,39 +45,98 @@ **pairing-code banner**; the box appears under **Unclaimed appliances** at the hub (verify uuid/MAC/SSH fingerprints match the physical box before binding — never bind a fingerprint you didn't read off the console). -3. **Bind** to the customer. Self-bind path (customer enters the code themselves) preferred when - the customer is present — it is the flow real remote onboarding will use. `[REFINE: which path - the rehearsal used + friction]` +3. **Bind** to the customer — **the rehearsal used the SELF-BIND path, and it is now the default for + this runbook.** Mint the link with **"Send self-bind link"** on the customer's Setup tab; the + customer opens it and completes the public two-factor `/bind/` page (console pairing code + + retrieval passphrase). The 2026-07-18 run bound in **60 s from mint to consume, with zero failed + attempts**, and the credential was delivered **26 s later** with no operator action. + + > **Send the self-bind link BEFORE the customer sees the console.** The console banner tells them + > to use „az e-mailben kapott link" — if the link has not been minted yet, the box is instructing + > the customer to look for something that does not exist. In the rehearsal the box sat in pairing + > mode for **~11.7 minutes** waiting on the operator, printing failures the whole time (R-33). + > Mint it during step A1 if you can; the token is valid for **7 days**, so there is no reason to + > wait. Auto-minting on customer create/RESET is R-36. + + Operator-bind on the Hosts page remains the fallback when the customer is not present or the + token has expired. 4. Watch the event sequence compose (expected: **seconds** between post-enroll legs — enroll → claim-code issue → WG → fast-tick window → tunnel → PBS-DR descriptor poke → controller up → claim e-mail lands → floor lift to current). **Any ~15-minute stall between two legs is a - finding** — capture which leg + wall-clocks. `[REFINE: reference wall-clock table from the - rehearsal]` + finding** — capture which leg + wall-clocks. + + **Reference wall-clocks — measured on the N100, 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). + Compare against these; a leg that takes minutes where this table says seconds is the finding: + + | Leg | Expect | + |---|---| + | Blank-metal boot → registered as an unclaimed appliance | **< 1 s** (same second) | + | Bind → credential delivered | **26 s** | + | Bind → agent enrolled + WG tunnel up | **56 s** | + | Bind → controller running the CURRENT version | **2 min 44 s** | + | Credential delivered → day-0 provision SUCCESS | **2 min 11 s** | + | Managed-floor lift (golden version → current) | **5 s**, unattended | + | Drive enrol: destructive-confirm → usable (incl. mkfs) | **~30 s** | + | Escrow ceremony started → offsite runs enabled | **12 s** | + + Note the **PBS-DR descriptor** leg specifically: in the rehearsal it reported `converged` while + the credential behind it was already stale, leaving the DR tier authenticating `401` with nothing + surfacing the fault (**R-39**). Until that is fixed, treat "descriptor provisioned" as unproven + and check `pvesm status` on the box during step E1. ## C. Customer zero script (the customer drives, operator narrates — ~30 min) This is the friction lens: the customer's hands on the keyboard, the operator's hands OFF. 1. **Claim:** open the claim e-mail → claim the dashboard → set the dashboard password. - (R-4 gate: the e-mail must land in inbox, not spam — gmail proven; `[REFINE: freemail.hu - result]`.) + (R-4 gate: the e-mail must land in inbox, not spam. **gmail PROVEN 2026-07-18** — the rehearsal's + claim mail was the first sent under the tightened DMARC `p=quarantine` and landed in the **Inbox**. + **freemail.hu is still untested** — if your tester is on freemail.hu, treat step C1 as unproven and + have them check spam before you conclude the mail was lost.) 2. **Escrow ceremony:** run the recovery-code wizard → the awaiting card („megerősítésre vár, legfeljebb 15 perc") → flips on ACK. The code goes on paper, into the envelope, into the customer's drawer — **state plainly: without this code, a dead box means the remote backup is unrecoverable; we cannot read it for them.** That sentence is the product. 3. **Storage:** enroll/format the external HDD via the wizard (customer clicks, operator explains - the destructive-confirm). -4. **First app:** deploy one app the household actually wants (paperless or immich recommended) — + the destructive-confirm). Expect **~30 s** from the confirm to a usable drive — the wizard runs + mkfs detached and polls, so a slow format is not a hang. + +4. **Offsite:** enable „Távoli mentés". Two operator behaviours here are **workarounds for known + defects** — follow them until the items ship, and do not improvise: + + > **Click Save exactly ONCE, then wait, then verify.** Provisioning runs synchronously inside the + > request, so on a slow Hetzner leg the request can return an nginx **504 while succeeding + > server-side**. A second click races the first attempt. Wait, then confirm on the hub's + > Offsite → Restic tab rather than believing the browser. (**R-31**) + + > **Re-onboarding an existing customer? Check offsite provisioning by hand.** After a RESET + + > re-enroll the hub can sit in "enabled but unprovisioned" and **say nothing** — provisioning is + > *Save*-triggered, and the re-enroll auto-re-issue deliberately skips unprovisioned targets. So + > an offsite target that looks configured may never have been created. Open the offsite settings + > and press Save once to force provisioning, then verify. (**R-36**) + + > **Expect the first run to be clean on a NEW customer.** If an **orphan card** appears + > („a távoli mentési tároló elárvult"), the repo is not new — that is the R-32 case, which fires + > when a customer's pool directory outlived a RESET. It is safe (the guard moves the old repo + > aside, never deletes it) but it means the box is writing into a reused directory. Record it. +5. **First app:** deploy one app the household actually wants (paperless or immich recommended) — through the catalog UI, no shortcuts. -5. **Sharing:** enable „Megosztás", set the household SMB password, create one share; open +6. **Sharing:** enable „Megosztás", set the household SMB password, create one share; open `\\NÉV` from their own laptop. (Set expectations per the one-pager: device list may lag on some routers; the address always works.) -6. **Protection proof — the customer performs a restore, unassisted** (this step IS the +7. **Protection proof — the customer performs a restore, unassisted** (this step IS the capability-map evidence, per customer): drop a file on the share or into the app → „Távoli mentés most" → wait for the snapshot → delete the file → the customer restores it via **Visszaállítás** with the operator silent. If they can't without help, that's an R-11/UX - finding, not a customer failure — write it down verbatim. `[REFINE: rehearsal friction notes]` -7. **Alerts sanity:** show them what an alert e-mail looks like (the health-degradation mail); + finding, not a customer failure — write it down verbatim. + + > **⚠ THIS IS THE ONE STEP THE REHEARSAL DID NOT EXECUTE.** Every other step in this document was + > walked on real hardware on 2026-07-18; C7 was not, so it carries **no friction notes and no + > evidence** — the capability-map row *"a customer (not the operator) performs a restore via UI + > alone"* is still **MISSING**. This runbook stays a **DRAFT** until C7 has been performed once. + > It is remote-doable on the reborn demo box (the dashboard is remote) and does not need the + > hardware in hand. +8. **Alerts sanity:** show them what an alert e-mail looks like (the health-degradation mail); confirm their address receives it. ## D. Handover (10 min) @@ -78,13 +152,20 @@ This is the friction lens: the customer's hands on the keyboard, the operator's 1. Hub: customer row green; Offsite → Restic tab shows the first bytes within a report cycle; PBS-DR gauge nominal (first PBS run on its own schedule — note, don't wait). + + > **Verify the DR tier on the box, not from the hub (R-39, interim).** The agent can report the + > PBS descriptor `converged` while the credential behind it is stale, and its own verify loop + > cannot read the credential to notice. Until R-39 ships, SSH to the host and run + > `pvesm status` — the `felhom-pbs` line must be **active**. A `401 Unauthorized` / `inactive` + > there means the DR tier is dead despite everything upstream looking green, and it will stay + > dead silently until someone needs a restore. 2. Floor convergence: controller at current, agent at MinAgent+. 3. Route friction notes: UX items → ROADMAP; one-pager corrections → R-11; runbook corrections → this doc. ## F. Standing rules -- Never perform steps C1–C6 *for* the customer. The script's value is watching them do it. +- Never perform steps C1–C7 *for* the customer. The script's value is watching them do it. - Never handle the recovery code — see it into the envelope, never into your hands or a photo. - One onboarding per day maximum during alpha — the friction notes are the point, and they degrade after the first. diff --git a/documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md b/documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md new file mode 100644 index 0000000..b2269e6 --- /dev/null +++ b/documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md @@ -0,0 +1,301 @@ +# VALIDATION — N100 rehearsal: RESET → generic ISO → self-bind → customer zero, 2026-07-18 + +> Supervised run of `runbooks/RUNBOOK-rehearsal-v3.md` (S1–S8). Viktor at the box as customer zero; +> CC collected the evidence afterwards (Phase A) before the N100 was unplugged for travel. **No +> production code changed by this run** — this document, the capability-map flips, and the ROADMAP +> items are its outputs. Every finding below is **recorded and ranked, never fixed here.** +> +> **Headline — read this first.** The complete final-product flow ran **end to end on real metal in +> one pass**: blank hardware → zero-touch install → self-registration → **customer self-bind** → +> credential delivery → day-0 → managed-floor lift → drive enrolment → escrow ceremony → offsite +> snapshots. The chain from **bind to a running current-version controller took 2 min 44 s**, and the +> managed floor lifted the controller **0.143.0 → 0.145.0 in 5 seconds, unattended**. Three things +> did not go to plan and are the run's real value: **(1)** the offsite repo came up **ORPHANED**, +> which S7 defines in advance as a finding; **(2)** the **PBS DR tier is silently non-functional** — +> descriptor applied, token rejected `401`, and the agent's own verify loop cannot read the +> credential to notice; **(3)** pushing the offsite config **restarted the controller and logged the +> customer out mid-flow**. Net: **core objectives green, 11 ranked items, one of them new and +> unforecast.** + +--- + +## Run context + +| | | +|---|---| +| Hardware | Intel **N100** (AlderLake-N), 16 GB; the same physical box as `VALIDATION-n100-baremetal-2026-07-16` | +| Host | `demo-felhom`, PVE **9.2.2** (kernel 7.0.2-6-pve), guest **9201** (`demo-felhom`) | +| Customer | `demo-felhom`, domain demo-felhom.eu — reborn clean-slate on this run | +| Versions | controller **0.145.0** · agent **0.90.0** · hub **v0.66.0** · ISO scripts **v1.20.0** · host-install **1.17.0** | +| Preparation | **RESET ×2** (host-delete-first): `demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51 — both with all five legs `ok` and escrow acked | +| Media | safety ISO first, then the real **universal pairing ISO v1.20.0**, `--loader mkimage`, Secure Boot **OFF** (mkimage is unsigned — the documented slice-B constraint) | +| Evidence | `180:~/n100-rehearsal/` (10 files + `MANIFEST.md`) | + +> **⚠ Timezone.** The PVE host journal is **CEST (+0200)**; the guest, the controller and the hub are +> **UTC**. **Every timestamp in this document is UTC.** Subtract 2 h from any `journalctl` line +> before comparing it with a controller or hub timestamp. This split is live-captured in +> `timezones.txt` and is itself worth a ruling (see *Observations*). + +--- + +## Timeline (wall-clocks, all UTC) + +Sourced from the hub events stream, the hub DB, the controller log, the bootstrap journal and the +agent journal — **not** from expectation. Where the brief and the evidence disagreed, the evidence +is recorded and the disagreement is called out under *Discrepancies*. + +| UTC | What happened | Evidence | +|---|---|---| +| 15:49:57 → 15:50:06 | **RESET #1** — customer `demo-vm-felhom`; legs `claim/db_purge/descriptor/hetzner/pbs` all `ok`, escrow acked | `hub-state.txt` `customer_resets` id=1 | +| 16:05:24 | `host_stale` + `node_stale` — "no report for 30m" | events | +| 16:08:51 → 16:08:59 | **RESET #2** — customer `demo-felhom`; all five legs `ok`, escrow acked (8 s) | `customer_resets` id=2 | +| **16:17:14** | **Blank metal first boot.** `felhom-bootstrap` in **PAIRING** mode; registers as an unclaimed appliance the same second | `bootstrap-journal.txt`; `appliance_registrations.first_seen` | +| 16:17:14 → 16:28 | 30 s pairing-poll loop; **every iteration exits 1** → **52** systemd `FAILED`/`Failed to start` lines on the customer-visible console | `bootstrap-journal.txt` | +| 16:28:55 | Operator mints + emails the **self-bind link**; 7-day TTL (expires 2026-07-25 16:28:55) | `selfbind_tokens` `created_at`/`emailed_at`/`expires_at` | +| **16:29:55** | **Customer self-binds** — token consumed, `attempts=0`, `locked=0`; `appliance_bound` source **`customer_selfbind`** | `selfbind_tokens.consumed_at`; events | +| 16:30:21 | `appliance_credential_delivered` (**26 s** after bind) | events; `appliance_registrations.delivered_at` | +| 16:30:40 | Hub `hosts` row created — `demo-felhom-8363b5` | `hosts.created_at` | +| 16:30:49 | Agent **0.90.0** daemon start; fast-tick armed | `agent-journal.txt` (18:30:49 CEST) | +| 16:30:51 | WG keypair generated → **registered with hub** → `10.77.0.2/32` → tunnel applied (**~2 s** end to end) | agent journal | +| 16:30:54 | Poke listener up, WG-confined `10.77.0.2:51822` | agent journal | +| 16:32:28 | `controller_started` **0.143.0** (the golden's version) | events | +| 16:32:29 | **Managed-floor update initiated — `initiated_by: auto-floor`** | `update-state.json` | +| 16:32:32 | **Day-0 provision SUCCESS** — done-flag written, unit disabled, bootstrap secrets scrubbed | `bootstrap-journal.txt` | +| **16:32:34** | **Floor update complete: 0.143.0 → 0.145.0 in 5 s**; new container created | `update-state.json`; `docker inspect .Created` | +| 16:32:39 | `controller_started` **0.145.0** | events | +| 16:33:24 | `node_recovered` — "Reports resumed (was stale for 27m)" | events | +| 16:41:41 | Drive init starts — `/mnt/hdd_1`, `/dev/sdb`, ext4 | controller log | +| 16:41:42 | Agent wipe gate: `storage_wipe` → `customer_confirmable`, **allowed=false**, `pending_confirmation` | agent journal | +| 16:41:51 | Wipe gate: **allowed=true**, `customer_confirmed` (9 s of customer thinking time) | agent journal | +| 16:42:19 | Device formatted ext4 | agent journal | +| 16:42:21 | Mount ensured → bound into guest → storage path registered → FileBrowser mounts synced. **Init 30.3 s** (F6 working) | controller log + agent journal | +| **16:50:22** | **RAM shrink 11675 → 8192 MB**, live cgroup, no reboot | controller log + agent `local-api: guest-memory resized` | +| 16:45:51 | Agent consumes a one-time **PBS** token secret (`secret_len=36`) and applies the descriptor | agent journal | +| 16:45:53 | `pbsdr: converged state=applied` | agent journal | +| **16:47:52** | **Hub mints a NEW PBS token secret — `consumed_at` is still NULL** | `host_pbs_secrets` | +| 16:47:55 → | Agent PBS verify loop: `cannot read token secret … permission denied` — **repeats every ~15 min, indefinitely** | agent journal | +| 16:54:58 | Config-refresh: hub `config_version=11 != applied=10` → **controller self-restarts** | controller log | +| 16:55:02 | Controller 0.145.0 restarts (container `StartedAt` 16:54:59Z — a **real restart**, not a reload) | controller log; `docker inspect` | +| 16:55:05 | `offsite-apply`: Storage Box target configured, *pending key escrow* | controller log | +| 16:56:29 | **Escrow ceremony started** via the wizard | controller log | +| 16:56:39 | Recovery code claimed (one-shot, not logged) | controller log | +| **16:56:41** | Hub-verified → `EscrowState` auto-confirmed → offsite runs enabled (**12 s** ceremony → confirmed) | controller log | +| 16:58:12 | First offsite run starts | controller log | +| **16:58:14** | **ORPHAN GUARD FIRES** — `offbox_repo_orphaned`, run skipped, card shown | controller log + events | +| 16:59:26 | Operator-confirmed reset: move-aside `/home/felhom-repo` → `.orphaned-20260718` | controller log | +| 16:59:32 | Fresh repo initialised; `offbox_repo_reset` | controller log + events | +| 16:59:38 → 16:59:50 | Backup run: 0 apps toggled, 0 snapshots, 7 s | controller log | +| **17:02:17** | **RAM grow 8192 → 12288 MB**, live cgroup, no reboot | controller log + agent journal | +| 17:05:15 | Calibre-Web deployed; deploy-page memory math now reads `total=12288MB` (**the grow rippled**) | events + controller log | +| 17:06:40 | Immich deployed | events | +| 17:11:16 | BookStack deployed (first start attempt failed `exit 1` 17:11:36; healthy by 17:12:03) | events + controller log | +| 17:13:38 → 17:14:22 | **Offsite run: 2 apps backed up, 1 skipped, 2 snapshots, 38 s** | controller log | +| 17:13:42 | Snapshot **`243ad08c`** — calibre-web | `restic-and-pool.txt` | +| 17:13:46 | Snapshot **`6df12205`** — immich | `restic-and-pool.txt` | + +### Derived wall-clocks (the numbers R-3 needs) + +| Leg | Wall-clock | +|---|---| +| Blank-metal boot → registered as unclaimed appliance | **< 1 s** (same second) | +| Self-bind link minted → customer consumed it | 1 min 0 s | +| **Bind → credential delivered** | **26 s** | +| Bind → agent enrolled + WG tunnel up | 56 s | +| **Bind → controller running the CURRENT version** | **2 min 44 s** | +| Credential delivered → day-0 provision SUCCESS | 2 min 11 s | +| **Managed-floor lift 0.143.0 → 0.145.0** | **5 s**, unattended | +| Drive enrol (confirm → usable, incl. mkfs) | 30.3 s | +| **Escrow ceremony → offsite runs enabled** | **12 s** | +| Boot → day-0 SUCCESS (incl. ~11.7 min waiting for the operator to send the link) | 15 min 18 s | + +--- + +## Per-ledger verdicts + +| Ledger | Verdict | Evidence | +|---|---|---| +| **S1** — stop apps, shut down, hub DOWN | **PASS** | `host_stale`/`node_stale` 16:05:24 | +| **S2** — the one BIOS visit (boot order, SB off, State-After-G3) | **PASS** (human leg) | Viktor at the box; SB-off is required by `--loader mkimage` | +| **S3** — the F1 moment: mkimage GRUB boots where stock GRUB threw `relocation 0x0` | **PASS** | The box reached first boot at 16:17:14 on the real board — **F1 is closed on physical hardware** | +| **S4** — unattended install → pairing mode → appears under Unclaimed appliances | **PASS** | `bootstrap-journal.txt` "PAIRING mode … registered — appliance token stored (0600)"; `appliance_registrations` id=3 `status=registered` | +| **S5** — the bind + the four-minute event composition, **no ~15-min stall between legs** | **PASS — and this is the immediacy proof** | Every post-bind leg is seconds apart: bind 16:29:55 → credential 16:30:21 → agent 16:30:49 → WG 16:30:51 → controller 16:32:28 → floor-lifted 16:32:39. **No leg stalled.** F5 auto-sizing landed the guest at **11675 MB** | +| **S6** — customer zero: claim → password → escrow ceremony → storage → **RAM grow *and* shrink** → deploy → enable offsite | **PASS on every leg, with two findings attached** | Ceremony 16:56:29→16:56:41; wipe gate pending→confirmed 16:41:42→16:41:51; **shrink 11675→8192 at 16:50:22 and grow 8192→12288 at 17:02:17, both live with no reboot**; 3 apps deployed. Findings: the config push logged the customer out (F3); offsite provisioning is **Save-triggered**, answering S6's open question | +| **S7** — first snapshot into a brand-new repo; **"if an orphan card appears, that's a finding"** | **FINDING — the orphan card appeared** | `offbox_repo_orphaned` 16:58:14. Recovered by the operator-confirmed move-aside 16:59:26→16:59:32; the run then produced **2 snapshots / 48.717 MiB** at 17:13:42–46. The *guard* worked exactly as designed — the finding is that it had to | +| **S7 (operator gauges)** | **PARTIAL** | Restic tab: Viktor's screenshots. **PBS DR tab was not captured, and the reason now matters — see F2** | +| **S8** — wrap: evidence, VALIDATION doc, map flips, ROADMAP, R-3 | **DONE** — this document | `180:~/n100-rehearsal/MANIFEST.md` | +| **Ledger 8** — immediacy legs, first live firing on a real onboarding | **PASS** | See S5. The "real-onboarding proof pending" note on the immediacy row can be cleared | +| **Ledger 9** — R-24 guest RAM resize, the *apply* (not just the refusals) | **PASS, both directions** | Shrink **and** grow, live cgroup, no reboot, and the new total rippled into the deploy page's memory math at 17:05:15 | + +--- + +## Findings (recorded and ranked — not fixed) + +Ranked as proposed for the ROADMAP; IDs assigned in `backlog/ROADMAP.md`. + +### F1 — Liveness presence lags reality by ~30 min (→ R-30, P2-HIGH) +The box was powered off at the start of the run, yet the hub carried it as healthy until the +staleness threshold expired at **16:05:24** ("no report for 30m"), and cleared only at **16:33:24** +("was stale for 27m"). The host-delete guard compounds this: RESET refuses while any host row +exists, so a stale-but-"online" host stalls a forced teardown. **Direction:** derive presence from +Dir-2 long-poll connectedness (~90 s grace), decoupled from notification hysteresis; agent/ep0 +analog later. + +### F2 — PBS DR tier is provisioned-but-dead, and cannot self-report it (→ R-39, P2-HIGH, **NEW — not in the brief**) +This was not forecast by the brief and is the run's most serious finding. Three facts: +1. The descriptor **was** auto-provisioned: the agent consumed a one-time token secret at 16:45:51 + and reached `converged state=applied` at 16:45:53. `storage.cfg` holds a complete `pbs: + felhom-pbs` stanza (namespace `demo-felhom`, token `felhom@pbs!demo-felhom`, encryption key, + fingerprint). +2. **The storage is non-functional.** `pvesm status` → `felhom-pbs: error fetching datastores - 401 + Unauthorized`, `inactive`. A direct probe with the stored credential returns **401 on every + endpoint including `/version`** — so this is authentication failure, not ACL scope. The transport + is healthy (WG handshake 9 s old, 27.9 ms RTT, `/version` reachable), so the token itself is + rejected server-side. +3. **The hub minted a second secret at 16:47:52 — two minutes after the agent applied the first — + and `consumed_at` is still NULL** ~55 min later. The agent will not pick it up: it has already + converged to `state=applied`, and its 15-minute verify loop cannot even read the credential to + detect the failure (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — the agent + runs non-root, writes the file through a root sudo wrapper, then tries to read it directly). + +Net: a mint/consume race leaves the box on a stale secret, a converged state machine prevents +re-application, and a permission bug blinds the only loop that could notice. **The DR tier would +have failed silently on the first real restore.** This also explains why the S7 PBS DR gauge was +worth capturing — it would have shown the fault. + +### F3 — Config apply restarts the controller and drops customer sessions (→ R-35, P2) +The offsite config push bumped `config_version` 10→11 at 16:54:58; the controller self-restarted +(container `StartedAt` 16:54:59Z) and came back at 16:55:02. In-memory sessions died with it, so +customer zero was force-logged-out mid-flow. **Direction:** hot-apply the offbox target, or persist +sessions across restart. + +### F4 — Offsite provisioning is synchronous, with no status affordance (→ R-31, P2-HIGH) +Save runs the Hetzner sync in-request; the request can hit the nginx 504 while succeeding +server-side, so the operator cannot distinguish failed from slow, and a retry races the first +attempt. **Direction:** reuse the awaiting-card/poll idiom. Interim mitigation belongs in R-3: +**click once, wait, verify.** + +### F5 — RESET destroys custody but not the ciphertext; the pool accumulates unattributed bytes (→ R-32, P2-HIGH) +The RESET ledger records `"hetzner":"ok"` — the sub-account **was** destroyed. But a Hetzner +sub-account is an access-control object, not a data object: deleting it leaves its directory intact. +Re-enabling offsite recreated a sub-account over the **same** `/home`, which still held the previous +lifecycle's repo — under a key destroyed by that same RESET. Hence S7's unexpected orphan card. +Measured on the pool box tonight: + +| Path | Size | Status | +|---|---|---| +| `/home/felhom-repo` | **49 M** | live, attributed (2 snapshots, 48.717 MiB) | +| `/home/felhom-repo.orphaned-20260717` | **1.4 G** | unattributed, key destroyed — from the 07-17 incident | +| `/home/felhom-repo.orphaned-20260718` | **3.0 M** | unattributed, key destroyed — from tonight | + +**Ruling from the run:** because RESET destroys custody, leftover ciphertext is unrecoverable **by +design**, so RESET gains a main-account purge of the customer base dir (the operator ack already +covers it). The move-aside guard **stays** for reinstall-without-RESET, where custody survives and +the card's promise is true. The operator Restic tab should show per-customer directory bytes against +attributed snapshot bytes so dead data cannot hide. + +### F6 — Bootstrap pairing-poll spams the customer-visible console (→ R-33, P2-HIGH) +Each 30 s poll is a separate `felhom-bootstrap.service` run that exits 1, so systemd prints +`Failed to start …` on the console the customer is watching. **52 FAILED lines in ~11 minutes**, +while nothing was actually wrong — the box was correctly waiting to be bound. **Direction:** poll +*inside* the script (scripts v1.21) and reword the first-boot noise. + +### F7 — Design-v2 gates are unenforced; one has been red for 16 releases +**Already ranked as R-29** (committed `2eccda4`, 2026-07-18) — the brief asked for a new ID, but the +finding already has one, including the `docker_run_volume_path_gate` / +`internal/appexport/estimate.go` attribution. **No new ID minted.** + +### F8 — Backup data lifecycle is unmanaged (→ R-34, P2) +Apps with snapshots but no active backup (disabled or uninstalled) are invisible and undeletable. +**Direction:** an "inactive backups" section on Távoli mentés — name / size / last snapshot / +restorable, with an explicit double-confirmed per-app delete via `restic forget --tag` + nightly +prune. **RULING: the offsite toggle NEVER offers deletion — policy and destruction stay decoupled.** + +### F9 — Post-RESET re-enroll leaves offsite "enabled but unprovisioned", silently (→ R-36, P2) +The hub knows the state and says nothing. Provisioning is **Save-triggered** (`configs.go` +`applyOffsite`) — confirmed live tonight, which answers S6's open question — and the re-enroll +auto-re-issue correctly skips unprovisioned targets (`handler.go`). **Direction:** flash it on the +customer page. Interim: an R-3 step. **Related:** auto-mint the self-bind link on customer +create/RESET so the console banner's "e-mailben kapott link" is always already true. + +### F10 — Console-passphrase bind (→ R-27c, P2) +Viktor's direction, with the session's security constraints: passphrase issued at customer creation; +spray-hardened global-lookup endpoint (per-appliance + per-IP caps, constant-time, single generic +failure, alerting); accent-free wordlist for console keymaps; the web capability-link path +**RETAINED**; claim-by-email **RETAINED** as the delivery-channel proof. The self-bind email gains +the public universal-ISO download link + two-line instructions (the DIY case). **Secret-bearing +per-customer ISOs are ruled OUT.** + +### F11 — Post-RESET health card shows stale pre-RESET warnings (→ R-37, P3) +Should read „RESET óta nincs adat". + +### F12 — Installer GRUB menu offers interactive installers (→ R-38, P3) +Single default „Felhom telepítés" entry; interactive installers **REMOVED** (safety: wrong-disk +manual installs); felhom background. Squashfs/theme rebranding explicitly deferred. + +--- + +## Observations (not findings — no ID minted) + +- **Host/guest timezone split.** The PVE host runs CEST while the guest, controller and hub run UTC. + Nothing is broken, but every cross-tier correlation needs a manual 2 h shift, which is exactly the + kind of friction that produces wrong incident timelines under pressure. Flagged for a ruling + rather than ranked unilaterally. +- **The escrow awaiting-card window was 12 seconds.** Ceremony → hub-confirmed took 16:56:29 → + 16:56:41, so the „megerősítésre vár, legfeljebb 15 perc" card rendered only briefly. The card is + correct and its ≤15-minute promise is honest, but capturing *both* states on screen is a matter of + luck at this speed. +- **BookStack skipped its first offsite run** — `no recovery unit found on any connected drive` + (17:13:42). This is the expected pre-first-nightly-dump state; the nightly job is scheduled + 2026-07-19 04:15 CEST. **Watch item, not a finding** — it becomes one only if it still skips + afterwards. +- **BookStack's first start attempt failed** (`exit 1`, 17:11:36) and succeeded on retry by + 17:12:03. Self-corrected; noted for pattern-watching only. +- **`felhom-samba` was not running** on the reborn box — sharing was not exercised this run + (consistent with *What was NOT exercised*). + +--- + +## What was NOT exercised (explicit) + +- **C6 — a customer performs a restore unassisted.** The single open script step in R-3, and still + **MISSING as evidence**. Remains remote-doable on the reborn box from vacation (the dashboard is + remote). +- **freemail.hu deliverability.** The gmail half is evidenced (claim email → Inbox under + `p=quarantine`); freemail.hu is Viktor's open half. +- **Sharing smoke on the reborn box** — R-7/R-7b were proven on the pre-RESET box; `felhom-samba` was + not deployed here. +- **R-19 WAN-outage drill.** +- **The PBS DR operator gauge** — not captured, and F2 means it would have shown a fault. +- **BookStack's offsite unit** — watch item, see *Observations*. +- **SMB positive auth with the real household password** — carried over from R-7b, unchanged. + +--- + +## Discrepancies vs the Phase-A brief (live evidence wins) + +1. **"expect ~11675 MB ballpark" — the final `pct config` reads 12288 MB.** Both are right at + different times: F5 auto-sized the guest to **11675 MB**, then customer zero **shrank it to 8192** + (16:50:22) and **grew it to 12288** (17:02:17). The brief described the F5 value; the box ended + above it because the grow was the last resize. Not a fault — and it is what completes ledger 9. +2. **"2.6 GB pool vs ~0 attributed" — measured 1.4 G + 3.0 M unattributed against 49 M attributed, + across TWO orphaned directories, not one.** The F5 ruling is unaffected; the numbers are. +3. **"felhom-pbs entry present" — present but `401 Unauthorized` / `inactive`.** The brief treated + the PBS descriptor as done; it is provisioned but dead. See F2. +4. **"RESET ×2 … second + third live firings" — the ledger holds exactly two rows, both tonight** + (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51), on **two different customers**. Any earlier + firing is not in `customer_resets`. Described here as tonight's two firings rather than as + ordinals. +5. **"keep the name R-29 if the number is free" — it is not free.** R-29 was committed earlier the + same day (`2eccda4`) for the design-v2 gate rot, which is also exactly the brief's item 5. The + liveness item therefore takes **R-30**, and no new ID was minted for the red-gate item. +6. **The brief's tip `a03e142` is two commits behind** `bbaa844`; Viktor's R-3 draft landed as + `runbooks/RUNBOOK-onboarding-draft-v3.md`, not `DRAFT-R3-onboarding-runbook.md`. +7. **F2 has no counterpart in the brief at all** — it was found by probing the box rather than by + following the finding list. + +--- + +## Evidence bundle + +`180:~/n100-rehearsal/` — see `MANIFEST.md` there for the per-file map. Secrets were read only to +run probes and are recorded as lengths and metadata, never as values.