docs(rehearsal): N100 S8 wrap — VALIDATION, map flips, ROADMAP R-30..R-39, R-3 draft
The 2026-07-18 N100 rehearsal ran the complete final-product flow on real metal
in one pass: RESET -> generic pairing ISO v1.20.0 -> customer self-bind -> day-0
-> managed-floor lift -> escrow ceremony -> offsite snapshots. No code changed;
every finding is recorded and ranked, none fixed.
VALIDATION-n100-rehearsal-2026-07-18.md — run context, a UTC-normalised timeline
built from the hub events stream / hub DB / controller log / bootstrap + agent
journals, per-ledger verdicts for S1-S8 + ledgers 8 and 9, 12 findings, the
not-exercised list, and 7 discrepancies against the brief.
Headline wall-clocks: bind -> credential 26 s; bind -> controller running the
current version 2 min 44 s; managed floor 0.143.0 -> 0.145.0 in 5 s unattended
(initiated_by: auto-floor); escrow ceremony -> offsite enabled 12 s; drive enrol
30.3 s. No post-bind leg stalled, which is the immediacy row's real-onboarding
proof.
Capability map (10 citations added):
- Bare-metal Felhom ISO PARTIAL -> PROVEN-LIVE (F1 closed on metal)
- Customer self-bind (slice 1) IMPLEMENTED -> PROVEN-LIVE (customer_selfbind)
- Guest RAM resize (R-24) IMPLEMENTED -> PROVEN-LIVE (shrink AND grow)
- Customer RESET two real firings + verified external teardown
- Escrow ceremony first live wizard firing
- Immediacy row "real-onboarding proof pending" cleared
- Publish train box-side floor lift proven on a fresh install
- Customer claim R-4 gmail half (Inbox under p=quarantine)
- Offsite orphan guard staged live leg fired on its own
- DR tier by default candidate PROVEN-LIVE upgrade WITHDRAWN (R-39)
Not flipped, as instructed: customer-performs-restore, BYO, DLNA, multi-user.
ROADMAP — collapsed R-1 (appliance half done, Peti half survives), R-21
(physically closed), R-24, R-27 slice 1, R-4. New ranked items:
P2-HIGH R-39 PBS DR applied-but-dead R-30 liveness from the wait channel
R-31 async offsite + status R-32 RESET base-dir purge
R-33 bootstrap quiet-poll
P2 R-34 backup lifecycle R-35 config-apply session survival
R-36 post-RESET offsite prompt R-27c console-passphrase bind
P3 R-37 post-RESET health card R-38 installer GRUB slice
Plus a pre-invite checklist (golden 0.145.x rebuild, freemail.hu, C6, R-11).
R-39 is NEW and was not on the brief: the PBS DR descriptor auto-provisions and
the agent converges state=applied, but pvesm reports 401 Unauthorized/inactive
and a direct probe 401s on every endpoint including /version while WG is healthy.
The hub minted a second token secret two minutes after the agent applied the
first and consumed_at is still NULL; the converged state machine will not
re-apply, and the agent's verify loop cannot read the credential to notice it
(non-root read of a file it writes through a root wrapper). Rank is provisional
pending Viktor.
R-3 draft: all four [REFINE] slots filled, self-bind made the default path with
"send the link BEFORE the customer sees the console", the measured wall-clock
table added, and interim operator workarounds for R-31/R-36/R-39. C6 (renumbered
C7) is marked as the single unexecuted step and keeps the doc a DRAFT.
Evidence bundle: 180:~/n100-rehearsal/ (10 files + MANIFEST.md), collected before
the box was unplugged for travel. Secrets read only to run probes; recorded as
lengths and metadata, never values.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Nn3VgQk9iwEGgyx6QJ2NvE
This commit is contained in:
1 parent
bbaa844f8c
commit
aa9c08f0dd
4 files changed
+447
-35
No files matched your search
@@ -31,12 +31,12 @@
|
||||
|---|---|---|---|---|
|
||||
| Appliance day-0 install: golden image → first boot → auto-confirm (zero clicks) → claimable box | installer, agent, hub, golden | **PROVEN-LIVE** (nested VM) | `DRILL-day0-vm-2026-07-12`, `DRILL-day0-take2-2026-07-12` | First firing on real customer hardware pending → R-1 |
|
||||
| BYO install: `--mode byo`, mandatory caps, host-mutation disclosure, coexistence guards | installer v1.15+, agent | **PARTIAL** | `DRILL-GL6-2026-07-08` (demo box); GL-8 coexistence fixes | Peti clean-slate reinstall on proxmox2 is the first real BYO run of the current path → R-1 |
|
||||
| Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`); selectable UEFI loader; **universal secret-free / operator-bind** mode | scripts v1.19.0 (`scripts/iso/`) + hub v0.62.0 + assistant container | **PARTIAL** (nested VM 310/311; live endpoints) | slice A `SPIKE-baremetal-iso-2026-07-16` (build gate, disk-filter fail-safe, stub→host-install fetch); slice B RUNBOOK-B (shim boots+installs OVMF SB-enforcing + SeaBIOS; `--loader mkimage` boots+installs SB-off; mkimage SB-enforcing **FAILS** `Access Denied`; surgery byte-identical); **slice C (2026-07-17): the GENERIC secret-free ISO** — box self-registers as an unclaimed appliance (`POST /api/v1/appliance/register`, one-shot poll delivery, 404-no-oracle — all live-verified through the public ingress), operator binds on the Hosts page, hub delivers credentials once; bootstrap harness proves direct(zero-appliance-calls)/pairing/delivery; artifact proven secret-free (baked env = hub URL only) | **F1 loader caveat:** `--loader mkimage` fixes cheap AMI firmware that can't USB-boot the stock GRUB — UNSIGNED → **Secure Boot must be OFF**; default `shim` keeps SB. **Slice C bind is operator-password-gated** (CC stages, Viktor binds) → the live boot→register→bind→day-0 composition + physical N100 boot fold into the supervised rehearsal (R-1). Customer-facing **self-bind page = R-27 slice 1 SHIPPED (hub v0.66.0, 2026-07-17)** — see the dedicated self-bind row |
|
||||
| Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) |
|
||||
| Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **IMPLEMENTED** | hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run + new-ISO console banner pending** (folds into the supervised rehearsal R-1). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends |
|
||||
| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | Endpoints driven on the drill VM; customer-facing **browser wizard** leg not yet live-validated. First supervised ceremony with a real customer pending → R-1. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody` |
|
||||
| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **IMPLEMENTED** | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected. Candidate upgrade to PROVEN-LIVE — see REPORT.) **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant |
|
||||
| Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown)** | hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Not live-clicked:** the web POST is password-gated (CC verifies via render + httptest orchestration tests); the Hetzner sub-account delete is unit-tested + a faithful mirror of the live-proven `ReissueCredentials`. **Consistency gap → R-25b:** the Danger-zone DELETE leaves host rows and doesn't run this teardown |
|
||||
| Bare-metal Felhom ISO (blank hardware → zero-touch auto-install → first-boot `host-install`); selectable UEFI loader; **universal secret-free / operator-bind** mode | scripts v1.19.0 (`scripts/iso/`) + hub v0.62.0 + assistant container | **PROVEN-LIVE** (physical N100, one pass, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — the full chain on real metal in a single pass:** the generic reusable pairing ISO (v1.20.0, `--loader mkimage`, SB off) booted the cheap AMI board that F1 had blocked, installed unattended, and the box **self-registered as an unclaimed appliance at 16:17:14 — the same second it first booted** (`appliance_registrations` id=3), then bound → credential-delivered → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware.** Prior nested legs: slice A `SPIKE-baremetal-iso-2026-07-16` (build gate, disk-filter fail-safe, stub→host-install fetch); slice B RUNBOOK-B (shim boots+installs OVMF SB-enforcing + SeaBIOS; `--loader mkimage` boots+installs SB-off; mkimage SB-enforcing **FAILS** `Access Denied`; surgery byte-identical); **slice C (2026-07-17): the GENERIC secret-free ISO** — box self-registers as an unclaimed appliance (`POST /api/v1/appliance/register`, one-shot poll delivery, 404-no-oracle — all live-verified through the public ingress), operator binds on the Hosts page, hub delivers credentials once; bootstrap harness proves direct(zero-appliance-calls)/pairing/delivery; artifact proven secret-free (baked env = hub URL only) | **F1 loader caveat:** `--loader mkimage` fixes cheap AMI firmware that can't USB-boot the stock GRUB — UNSIGNED → **Secure Boot must be OFF**; default `shim` keeps SB. **Slice C bind is operator-password-gated** (CC stages, Viktor binds) → the live boot→register→bind→day-0 composition + physical N100 boot fold into the supervised rehearsal (R-1). Customer-facing **self-bind page = R-27 slice 1 SHIPPED (hub v0.66.0, 2026-07-17)** — see the dedicated self-bind row |
|
||||
| Customer claim: one-time emailed code → customer sets own password (bcrypt, operator never sees it) | controller v0.122, hub v0.50 | **PROVEN-LIVE** (drill VM) | `DRILL-day0-vm-2026-07-12` §10/F-4 (gate ON via real edge; claimed, code consumed) | Never executed by a non-Viktor human → R-3. **Deliverability (R-4), gmail half DONE 2026-07-18:** the rehearsal's claim email was the first sent under the tightened DMARC `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **freemail.hu remains Viktor's open half.** (Dropped mis-cited `CAMPAIGN-4` F-C — that is the escrow-claim 502, not password claim) |
|
||||
| Customer binds their own appliance (self-service): operator-sent 7-day tokenized capability link → public two-factor `/bind/<token>` (console pairing code + retrieval passphrase) → hub stages the bind, no operator | hub v0.66.0 + ISO scripts v1.20.0 | **PROVEN-LIVE** (real customer-zero bind on metal, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md`:** operator minted + emailed the link 16:28:55 (7-day TTL, expiry 2026-07-25 recorded); **the customer bound their own box at 16:29:55 with `attempts=0`, `locked=0`** — `appliance_bound` carries source **`customer_selfbind`**, and the credential was delivered **26 s later** with no operator action. Hub-side lifecycle in `hub-state.txt` (`selfbind_tokens` mint→email→consume). Prior unit evidence: hub v0.66.0 (`web/selfbind.go`, `store/selfbind.go`; Scenarios A–F + F1/F2; 4 red-proofs verified red — THE TRAP `/bind/` exemption, no-oracle, lockout, single-active); GC verdict §3 (no appliance GC → TTL stands alone) | R-27 **slice 1**. No appliance list ever rendered; wrong code == wrong passphrase (one generic failure); 5-attempt lockout → call support; expiry falls back to operator-bind. **Live first-run DONE 2026-07-18** (rehearsal; the console banner rendered on the real ISO). **R-27b** (controller second-box dismissable prompt) deferred; **multi-box-per-link** = repeated operator sends |
|
||||
| Escrow ceremony: customer-facing wizard, one-shot R claim, operator zero-knowledge | controller v0.127, agent v0.88/0.89 | **PROVEN-LIVE** (drill VM, endpoint-exact) | agent v0.88.0 REPORT (ceremony ~4s, one-shot claim 200→410, R absent from every payload); `SPIKE-controller-escrow-2026-07-13` | **Customer-facing browser wizard FIRST LIVE FIRING 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S6): customer zero drove the wizard on the reborn box — ceremony started 16:56:29, recovery code claimed one-shot 16:56:39 (absent from logs by design), hub-verified and `EscrowState` auto-confirmed 16:56:41, **offsite runs enabled 12 s after the ceremony began**; the v0.138.0 „megerősítésre vár, legfeljebb 15 perc" awaiting card rendered and flipped on the ACK (operator screenshots: Viktor's set). Honest caveat: at a 12-second confirm the awaiting window is so short that catching *both* states on screen is luck, not procedure. Prior: endpoints driven on the drill VM. **agent v0.89.0:** `/escrow/preflight` `pbs_storage_id` row now live-reloads (reads current agent.json) — a pbsdr convergence that seeds the id flips it green with NO service restart. **hub v0.60.0 (data-first retention):** a re-escrow with a DIFFERENT sealed passphrase no longer destroys the old blob — the hub RETAINS it (`host_escrow_superseded`), so a previous passphrase stays recoverable with its recovery code (turns the reinstall-orphan incident from "history destroyed" into "history recoverable"). Guided-recovery flow = R-26. Red-proof `TestSaveHostEscrow_RetainsSuperseded`. **hub v0.60.1 — custody survives the host lifecycle:** host deletion (with the escrow ack) DEMOTES the current blob to retained custody (moved into `host_escrow_superseded`, never destroyed; existing superseded rows spared); the customer Danger-zone Delete is the one true purge point (cascades both escrow tables incl. already-deleted hosts). No operator path through host lifecycle can lose a blob. Red-proofs `TestDeleteHost_DemotesEscrowNeverDestroys` + `TestDeleteCustomer_PurgesEscrowCustody` |
|
||||
| DR tier by default: PBS + WireGuard base infra on every install, hub-controlled activation | installer v1.15, agent v0.86, hub v0.51 | **IMPLEMENTED** | `DRILL-day0-take2-2026-07-12` §2 (WG enabled both modes, PBS-DR descriptor auto-provisioned ~1s after WG registration, zero operator steps); ships installer v1.15/agent v0.86/hub v0.51 | Live only on demo/drill fleet. (Cited spike was slice-0 mechanics — shipped nothing; corrected.) **⚠ The candidate upgrade to PROVEN-LIVE is WITHDRAWN — the 2026-07-18 rehearsal produced a live counter-example (R-39).** On the reborn N100 the descriptor auto-provisioned and the agent reported `converged state=applied` (16:45:53), yet **the storage is dead**: `pvesm status` → `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive`, and a direct probe with the stored credential returns **401 on every endpoint including `/version`** while the WG transport is healthy (handshake 9 s, 27.9 ms RTT) — i.e. authentication failure, not ACL scope. Root cause in the evidence: **the hub minted a SECOND token secret at 16:47:52, two minutes after the agent had applied the first, and `consumed_at` is still NULL**; the converged state machine will not re-apply, and the agent's 15-minute verify loop **cannot even read the credential to notice** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — non-root agent reading a file it writes through a root wrapper). A tier that reports `applied` while silently unable to authenticate is exactly the shape that must not carry a PROVEN-LIVE badge. See `tests/VALIDATION-n100-rehearsal-2026-07-18.md` F2 and `pbs-dr-state.txt`. **agent v0.89.0 closes the F4 non-default-storage-id gap (R-22) — PROVEN-LIVE 2026-07-17:** the reconcile self-grants the ACL through the root wrapper on a pre-check 403 instead of dead-locking. Reproduced F4 on the demo (marker moved aside = reinstall fresh-state + felhom-offsite ACLs revoked) → next reconcile tick `pbsdr: pre-check 403 … self-granting … (R-22)` → `converged state=adopted` in ~3 s, ACLs self-restored, `pvesm status felhom-offsite`=active, zero operator action. No more one-shot `pveum` grant |
|
||||
| Customer RESET (middle lifecycle tier: host delete < RESET < customer Delete): one operator action → pre-first-install; all operational state destroyed, identity + basic config survive | hub v0.61.0, felhom-tenantsync v1.1.0 | **PROVEN-LIVE (external teardown, incl. two real firings)** | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` — two live firings, both host-delete-first, on two different customers** (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51): every leg `ok` (`claim`, `db_purge`, `descriptor`, `hetzner`, `pbs`), escrow acked separately, each completing in 8–9 s (`hub-state.txt` `customer_resets`). The **Hetzner sub-account destruction is now verified against the live pool box** — and produced the run's sharpest lesson: **a sub-account is an access-control object, not a data object.** Deleting it left its `/home` intact, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext under a key this same RESET had destroyed — which is why the orphan guard fired at 16:58:14 (**a finding by S7's own criterion**) and why RESET now needs a base-dir purge → **R-32**. Prior: hub v0.61.0 REPORT; **ep0 live drill 2026-07-17** (throwaway `drill-reset-01` with a real backup: deprovision `deleted:true` destroyed the namespace + backup group + token, idempotent re-run `deleted:false`, all 3 real tenants + shared user survived); red-proofs (ack-gate, partial-failure resumability) + orchestration/store/offsite/render tests | External teardown FIRST, DB purge LAST, every leg idempotent; refuses while any host row exists; separate escrow-custody ack; clears claim (fresh code next onboarding); keeps the offsite tier CHOICE, drops provisioned fields. **Live-clicked 2026-07-18** (twice, by Viktor) — this supersedes the earlier "not live-clicked / Hetzner delete unit-tested only" note. **Consistency gap → R-25b:** the Danger-zone DELETE leaves host rows and doesn't run this teardown |
|
||||
| Uninstall: KEPT-vs-WIPED statement, secret purge, enrolled-drive handling | installer | **PARTIAL** | `DRILL-GL6-2026-07-08` Phase 1/5 (KEPT-vs-WIPED printed verbatim; drive data intact ×3); GL-4 code | Secret purge (GL6-F1 `.bak` residue) fixed v1.12.0; enrolled-drive `mnt-*.mount` units survive (GL6-F2, open); cluster-aware `felhom_guests` guard + saferemove cost warning missing → R-9 |
|
||||
|
||||
## B. Apps & catalog
|
||||
@@ -57,7 +57,7 @@
|
||||
|---|---|---|---|---|
|
||||
| Nightly DB dumps (postgres/mariadb autodiscovery), atomic writes | controller v0.118 | **PROVEN-LIVE** | `CAMPAIGN-2` T-BAK-FULL (pg+mariadb autodiscovered); atomicity `CAMPAIGN-6B` P4 + `CAMPAIGN-6E` B1/B2 (SIGKILL mid-write → only `.tar.tmp` touched, last-good byte-unchanged); DB restore `CAMPAIGN-6D` P-FAB | (Cited `CAMPAIGN-3` F7 is the *finding* of non-atomic writes, and T-RST-DB was auth-hollow — corrected to the 6B/6E fix-proofs.) T-6E-1 dir-fsync asymmetry (LOW) → R-10 |
|
||||
| Tier-2 secondary-drive copy: class-driven legs, v2 relpath layout, NAS-target exclusion, safe-remove boundary | controller v0.135 | **PROVEN-LIVE** | `CAMPAIGN-6E-2026-07-15` (P-TIER2 deep-4 PASS), `CAMPAIGN-6C` | |
|
||||
| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs; live leg staged for the delete/re-create rehearsal (a scratch-target swap disturbs the live escrow state). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` |
|
||||
| Offsite (restic → Hetzner Storage Box): mandatory class only, raw-data quota, enlargement gate, retention regrouping | controller v0.134, agent, hub | **PROVEN-LIVE** | `CAMPAIGN-6D-2026-07-15` (mandatory-only P-IMMICH; enlargement gate fired at real 50GiB quota P3-DELIVERY); `VALIDATION-offbox-storagebox-2026-07-09` (byte-perfect round-trip) | Raw-data quota (SP-1) + retention regrouping (SP-2) are `SPIKE-restic-snapshot-shape` **dry-run** verdicts — mechanism validated, not fired in a live product run; only the enlargement gate is live-fired. **Reinstall-continuity (controller v0.142.0, 2026-07-17):** a recreated data volume that orphaned the repo (new passphrase can't open the old keys) is now CLASSIFIED (`wrong password or no key found`) → explicit ORPHANED card + event (not nightly-spam) + a move-aside (never-delete) reset (unclaimed auto / claimed confirm), instead of a raw nightly restic error. Fake-based scenarios + red-proofs. **The live leg FIRED on its own during the 2026-07-18 rehearsal** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`, S7): after a RESET + re-enable, the first offsite run hit the previous lifecycle's ciphertext and the guard **classified it, pushed `offbox_repo_orphaned`, skipped the run and showed the card (16:58:14)** rather than nightly-spamming a raw restic error; the operator-confirmed reset then **moved the repo aside (never deleted) to `.orphaned-20260718` and re-initialised (16:59:26→16:59:32)**, and the next run produced 2 snapshots / 48.717 MiB. The guard behaved exactly as designed — **the finding is that it had to fire at all** (R-32: RESET destroys custody, so the ciphertext it leaves behind is dead by design and should be purged, while the move-aside guard stays correct for reinstall-WITHOUT-RESET). `DIAGNOSE-offbox-repo-orphaned-2026-07-17` |
|
||||
| Offsite restore: local-preferred scratch, unit-only default, full two-step, missing-only place-to-live | controller v0.134/134.1/135 | **PROVEN-LIVE** | `CAMPAIGN-6D` accept legs (immich end-to-end from offsite alone) | |
|
||||
| Manual `.fab` export/import: class-scoped capture, browser up/download, tunnel-proof chunking | controller v0.125/128/130/136 | **PROVEN-LIVE** | `CAMPAIGN-6D` P-FAB / Accept #1 (1.7 GB full circle, byte-identical, app boots); chunking `CAMPAIGN-6B` P2 (100 MiB via real CF edge, 120 MiB→413) | Chunking proven at the real CF edge via `curl --resolve`; the **rendered browser file-picker** upload leg is still Viktor's open full-circle test (6C ran it NOT-RUN). C6B-F1 was the 6B *finding*; fix verified in 6D |
|
||||
| Guest-loss DR: PBS restore with full-fidelity layout from archive, restore-test verification | agent v0.75/0.76, PBS | **PROVEN-LIVE** | `CAMPAIGN-2` T-P9-DESTROY-RESTORE (whole-guest `pct restore` of 9201 → running+healthy) + T-PBS-VERIFY (`verify_state: ok`, 13 snapshots); `DRILL-GL6-2026-07-08` Phase 0d (restore-test `mount_parity: ok`) | (Cited `VALIDATION-newbox-restore` is offbox **restic** file-restore, wrong tier — corrected.) Real **offsite** guest-loss round-trip still R1-blocked → S5 DR drill |
|
||||
@@ -109,9 +109,9 @@
|
||||
| Scenario | Components | Status | Evidence | Gap / roadmap |
|
||||
|---|---|---|---|---|
|
||||
| Customer/host management: 8-tab detail, scoped auto-refresh, safe stale-host deletion, capability chips | hub v0.47–0.53 | **PROVEN-LIVE** | hub v0.53.0 dead-host roll-up live on the Peti cluster (proxmox1 down 23h); `CAMPAIGN-4-2026-07-13` (operator UI driven live); `DRILL-day0-take2` F-16 (offsite/freeze buttons live) | 8-tab render + capability chips are **render-test-validated** (hub UI is password-gated; CC cannot log in). (Cited "daily operator use" was a no-doc citation; `AUDIT-hub-gui-2026-06-30` predates these features at hub v0.25) |
|
||||
| Config/state change round-trips in **seconds** (hub↔box immediacy; 15-min cycle stays the backbone): box→hub out-of-cycle report (Dir 1) + hub→box `GET /api/v1/wait` long-poll wake (Dir 2) | controller v0.139/140, hub v0.58/0.63 | **PARTIAL** | Transport proven live through the real DNS-only ingress: `SPIKE-immediate-sync-transport-2026-07-16` + hub v0.58.0 / controller v0.140.0 REPORTs — 240 s no-annotation hold (25 s heartbeat defeats nginx's 60 s `proxy_read_timeout`, no ingress change), 0.047 s wake-on-change, hub `rollout restart` = 1 WARN + 0-storm reconnect; Dir-1 2 s box→hub round-trip live in controller v0.139.0 | The operator-UI **save→apply** round-trip is not fired end-to-end live (hub UI password-gated; CC can't log in) → R-23; the wake transport and the ACK→config_version→`ConfigRefresher` delivery chain are each proven, only the UI-triggered bump leg is unexercised. Agent-plane (host-domain desired-state) poke **first slice PROVEN-LIVE** (Direction-2a, agent v0.89.0 + hub v0.59.0, 2026-07-17): contentless ep0-relayed UDP poke → agent immediate desired-state cycle, per `SPIKE-immediate-sync-transport-2026-07-16` P4. Full path live-proven: a real operator manifest save fired `poke: sync-poke delivered to 10.77.0.2`; the box (0.89.0) received it and logged `poke received → triggering an immediate desired-state cycle` → `out-of-band report triggered` — **~31 ms ep0→box, sub-ms to the report cycle** (WG-confined, from 10.77.0.1 to the 10.77.0.2-bound socket); save→tick ≈ ~0.45 s (SSH-dominated), well under ≤2–3 s. R-13 first slice (listener+sender only; the rest of the mutual-repair arc stays open). **System-initiated immediacy wired (hub v0.63.0, this REPORT):** the mutation sites that only OPERATOR actions used to notify now fire the correct plane's notifier when the hub itself mints state — agent-plane pokes at `PBSDRAutoProvision` (the observed slice-C lag), `ReissuePBSDR` (also the pbsdrheal escalation), `handlePBSDRReissue`, and the two admin desired-state api writers; controller-plane bump at `reissueOnReenroll`. Unit-tested + red-proofed, not yet fired on a real system event (folds into the rehearsal bind sequence). Still PARTIAL: the R-23 operator-UI save→apply leg and the agent **fast-tick-until-first-convergence** SECONDARY (the WG-registration leg a poke can't reach pre-tunnel) remain unfired live. **Fast-tick SHIPPED (agent v0.90.0, R-28):** while any desired-state item is unapplied — incl. the pre-tunnel window a poke can't reach — the agent pulses the out-of-band trigger every 30 s and self-disarms on convergence (state-based; four cached sources; LOUD states excluded). LIVE on both demo agents (the `fast-tick armed: 30s …` startup line verified); its real-onboarding value (the pre-tunnel window shrinking ≤15 min → ~30 s) is unfired live — folds into the rehearsal S5 bind sequence |
|
||||
| Customer right-sizes guest RAM from the controller (agent-enforced bounds, live cgroup apply, no reboot) | agent v0.90.0 + controller v0.143.0 (R-24) | **IMPLEMENTED** (→ PROVEN-LIVE pending a real grow) | Controller-direct (R-24's hub-desired-state framing SUPERSEDED, Viktor 2026-07-17). Agent `GET`/`POST /guest/memory` enforces every bound FRESH (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)) + verify-after-apply; PVE `SetConfig` hot-applies (**Phase-0 PROVEN** on the nested box: maxmem moves with the guest running, /proc/meminfo ripples via lxcfs, no reboot). Controller "Szerver memória (RAM)" card + code→Hungarian map, gated on `FeatureGuestMemoryResize` (MinAgent 0.90.0). LIVE-validated end-to-end through the real endpoint on the demo (above_max + below_min refusals render the Hungarian, agent English never leaks; SupportYes via the version header) | A **successful** grow/shrink through the bounded endpoint is unfired live — the 4 GB nested demo host is too small (max<current, all correctly refused); a normal-sized box folds into the rehearsal S6. Cores stay observation |
|
||||
| Publish train: MinAgent floors, gated auto-Reissue, version channels, floor-field-LAST rules | hub v0.45/0.53, agent | **PARTIAL** | `runbooks/publish-train-rules.md`; demo-fleet updates proven | **Never proven on a real remote customer** — parked trains `RUNBOOK-publish-0.79/0.81/0.85-*` await Peti → R-1 |
|
||||
| Config/state change round-trips in **seconds** (hub↔box immediacy; 15-min cycle stays the backbone): box→hub out-of-cycle report (Dir 1) + hub→box `GET /api/v1/wait` long-poll wake (Dir 2) | controller v0.139/140, hub v0.58/0.63 | **PARTIAL** | Transport proven live through the real DNS-only ingress: `SPIKE-immediate-sync-transport-2026-07-16` + hub v0.58.0 / controller v0.140.0 REPORTs — 240 s no-annotation hold (25 s heartbeat defeats nginx's 60 s `proxy_read_timeout`, no ingress change), 0.047 s wake-on-change, hub `rollout restart` = 1 WARN + 0-storm reconnect; Dir-1 2 s box→hub round-trip live in controller v0.139.0 | The operator-UI **save→apply** round-trip is not fired end-to-end live (hub UI password-gated; CC can't log in) → R-23; the wake transport and the ACK→config_version→`ConfigRefresher` delivery chain are each proven, only the UI-triggered bump leg is unexercised. Agent-plane (host-domain desired-state) poke **first slice PROVEN-LIVE** (Direction-2a, agent v0.89.0 + hub v0.59.0, 2026-07-17): contentless ep0-relayed UDP poke → agent immediate desired-state cycle, per `SPIKE-immediate-sync-transport-2026-07-16` P4. Full path live-proven: a real operator manifest save fired `poke: sync-poke delivered to 10.77.0.2`; the box (0.89.0) received it and logged `poke received → triggering an immediate desired-state cycle` → `out-of-band report triggered` — **~31 ms ep0→box, sub-ms to the report cycle** (WG-confined, from 10.77.0.1 to the 10.77.0.2-bound socket); save→tick ≈ ~0.45 s (SSH-dominated), well under ≤2–3 s. R-13 first slice (listener+sender only; the rest of the mutual-repair arc stays open). **System-initiated immediacy wired (hub v0.63.0, this REPORT):** the mutation sites that only OPERATOR actions used to notify now fire the correct plane's notifier when the hub itself mints state — agent-plane pokes at `PBSDRAutoProvision` (the observed slice-C lag), `ReissuePBSDR` (also the pbsdrheal escalation), `handlePBSDRReissue`, and the two admin desired-state api writers; controller-plane bump at `reissueOnReenroll`. Unit-tested + red-proofed, not yet fired on a real system event (folds into the rehearsal bind sequence). Still PARTIAL: the R-23 operator-UI save→apply leg and the agent **fast-tick-until-first-convergence** SECONDARY (the WG-registration leg a poke can't reach pre-tunnel) remain unfired live. **Fast-tick SHIPPED (agent v0.90.0, R-28):** while any desired-state item is unapplied — incl. the pre-tunnel window a poke can't reach — the agent pulses the out-of-band trigger every 30 s and self-disarms on convergence (state-based; four cached sources; LOUD states excluded). LIVE on both demo agents (the `fast-tick armed: 30s …` startup line verified). **REAL-ONBOARDING PROOF DONE — `tests/VALIDATION-n100-rehearsal-2026-07-18.md` (ledger 8, S5):** on a genuine first onboarding on metal, **every post-bind leg landed seconds apart with no ~15-minute stall anywhere** — bind 16:29:55 → credential delivered 16:30:21 (26 s) → agent 0.90.0 up 16:30:49 → **WG registered + tunnel applied 16:30:51 (~2 s)** → poke listener 16:30:54 → controller 16:32:28 → floor-lifted and running current 16:32:39. **Bind → running-current = 2 min 44 s.** The PBS-DR descriptor auto-provisioned on the same cadence (agent `converged state=applied` 16:45:53) — though see the DR-tier row: the descriptor converged while the credential behind it was already stale (R-39). The pre-tunnel fast-tick window is therefore proven in its real setting; the remaining PARTIAL is the R-23 operator-UI save→apply leg alone |
|
||||
| Customer right-sizes guest RAM from the controller (agent-enforced bounds, live cgroup apply, no reboot) | agent v0.90.0 + controller v0.143.0 (R-24) | **PROVEN-LIVE** (grow **and** shrink on metal, 2026-07-18) | **`tests/VALIDATION-n100-rehearsal-2026-07-18.md` (ledger 9) — the apply is now proven in both directions on a normal-sized box:** customer zero **shrank 11675 → 8192 MB at 16:50:22** and **grew 8192 → 12288 MB at 17:02:17**, each a live cgroup apply with **no reboot** (`local-api: guest-memory resized` in the agent journal, `[web] memory resized` in the controller log), and the new total rippled into the deploy page's memory math at 17:05:15 (`total=12288MB`). F5 auto-sizing had landed the guest at 11675 MB. Controller-direct (R-24's hub-desired-state framing SUPERSEDED, Viktor 2026-07-17). Agent `GET`/`POST /guest/memory` enforces every bound FRESH (min 2048 / max host_total−2048 / shrink floor max(2048, usage+512)) + verify-after-apply; PVE `SetConfig` hot-applies (**Phase-0 PROVEN** on the nested box: maxmem moves with the guest running, /proc/meminfo ripples via lxcfs, no reboot). Controller "Szerver memória (RAM)" card + code→Hungarian map, gated on `FeatureGuestMemoryResize` (MinAgent 0.90.0). LIVE-validated end-to-end through the real endpoint on the demo (above_max + below_min refusals render the Hungarian, agent English never leaks; SupportYes via the version header) | Row complete as of the 2026-07-18 rehearsal — the refusals were proven on the nested demo, the **applies** on the N100. Cores stay observation |
|
||||
| Publish train: MinAgent floors, gated auto-Reissue, version channels, floor-field-LAST rules | hub v0.45/0.53, agent | **PARTIAL** | `runbooks/publish-train-rules.md`; demo-fleet updates proven | **Box-side floor lift PROVEN-LIVE on a fresh install** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the day-0 golden deployed controller **0.143.0** at 16:32:28 and the managed floor lifted it to **0.145.0 by 16:32:34 — a 5-second, fully unattended update inside the first minute of controller life**, `update-state.json` recording `initiated_by: auto-floor` with `controller_updated` pushed to the hub. So the *mechanism* is no longer nested-only. **Still never proven on a real REMOTE customer** — parked trains `RUNBOOK-publish-0.79/0.81/0.85-*` await Peti → R-1. **Action before first invite: rebuild the golden to 0.145.x** now that this evidence is banked, so fresh boxes don't sit two versions stale |
|
||||
| Agent self-update: A/B slots, crash-loop auto-rollback, operator-signed | agent v0.70+ | **PROVEN-LIVE** (demo) | `SPIKE-agent-selfupdate-2026-07-05` | Remote-customer proof pending → R-1 |
|
||||
| Controller self-update: anonymous registry, no credentials in guest | controller v0.112 | **PROVEN-LIVE** (demo) | 07-10 arc | |
|
||||
| Offsite provisioning: Hetzner API, sub-account per customer, host-key pinning, credential re-issue | hub v0.37–0.39 | **PROVEN-LIVE** | `VALIDATION-offsite-provisioning-e2e-2026-07-09`, `SPIKE-hetzner-api-provisioning-2026-07-09` | |
|
||||
|
||||
@@ -20,15 +20,26 @@
|
||||
|
||||
| ID | Item | Size | Status | Notes / map rows flipped |
|
||||
|----|------|------|--------|--------------------------|
|
||||
| R-1 | **Peti convergence:** clean-slate proxmox2 reinstall (spec'd 07-15), first live auto-confirm, supervised escrow ceremony, execute parked publish trains (agent 0.81→0.88, controller → 0.137) | L | spec'd | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path |
|
||||
| R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path |
|
||||
| R-2 | ~~Resolve ~215 lines of foreign WIP in felhom.eu clone (`hub/internal/notify/`, `store.go`, `hub/internal/claim/`)~~ | S | **killed** (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — `notify/`+`claim/engine.go` in `6b40eb8` (v0.50.0), `store.go` in `a1d0450` (v0.54.0), plus follow-up `e205a2d`; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) |
|
||||
| R-3 | Friend-alpha onboarding runbook (generalized from `pilot/RUNBOOK-peti-return-2026-07-13`): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step | M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of `PETI-tester-agreement.md`. **Next from-scratch rehearsal to include customer DELETE + re-create** — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). **S6b (manual stale-host delete before re-enroll) is OBSOLETE** — re-enrollment upserts the existing host row cleanly (`store.UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. **NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE):** one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). **Remaining consistency gap:** the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b |
|
||||
| R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu / citromail.hu / t-online.hu; tighten DMARC `p=none` → `p=quarantine` (pending since email.md 02-04) | S | **DNS half DONE (2026-07-18); test-send open** | A claim code in spam bricks onboarding at step 1. **DMARC now `p=quarantine`** (`RUNBOOK-publish-0.90-0.143-2026-07-18` Phase 4): `_dmarc.felhom.eu` TXT `v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu` set by Viktor (CF, Path B); `dig` gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM `d=felhom.eu` + relaxed `send.` SPF). **Remaining: Viktor's deliverability test-send** (gmail + freemail.hu, `dmarc=pass` check) |
|
||||
| R-4 | Claim-code deliverability: test-send to gmail.com / freemail.hu / citromail.hu / t-online.hu; tighten DMARC `p=none` → `p=quarantine` (pending since email.md 02-04) | S | **DNS half DONE + gmail half DONE (2026-07-18); freemail.hu open** | A claim code in spam bricks onboarding at step 1. **DMARC now `p=quarantine`** (`RUNBOOK-publish-0.90-0.143-2026-07-18` Phase 4): `_dmarc.felhom.eu` TXT `v=DMARC1; p=quarantine; rua=mailto:admin@felhom.eu` set by Viktor (CF, Path B); `dig` gate green (1.1.1.1 + 8.8.8.8). Alignment safe (SES/Resend DKIM `d=felhom.eu` + relaxed `send.` SPF). **gmail half CLOSED by the rehearsal (2026-07-18):** the run's claim email was the first sent under `p=quarantine` and **landed in the gmail Inbox, not spam** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). **Remaining: the freemail.hu test-send** (Viktor, remote-doable, `dmarc=pass` check) |
|
||||
|
||||
## P2 — during alpha
|
||||
|
||||
> **Sub-rank `P2-HIGH` = close before the first REMOTE tester.** These are the 2026-07-18 N100
|
||||
> rehearsal's findings (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). They are not P1 — the
|
||||
> rehearsal proved the product flow works — but each one either misleads the operator, misleads the
|
||||
> customer, or hides a failure, and all of that gets materially worse the moment the box is somewhere
|
||||
> you cannot walk over to.
|
||||
|
||||
| ID | Item | Size | Status | Notes |
|
||||
|----|------|------|--------|-------|
|
||||
| R-39 | **[P2-HIGH] The PBS DR tier can be `applied` and dead at the same time — and nothing notices.** On the reborn N100 the descriptor auto-provisioned and the agent converged `state=applied`, yet `pvesm status` reports `felhom-pbs: error fetching datastores - 401 Unauthorized` / `inactive` and a direct probe with the stored credential 401s on **every** endpoint including `/version` (WG transport healthy: handshake 9 s, 27.9 ms RTT — so authentication, not ACL scope). Three compounding defects: **(a)** a **mint/consume race** — the hub minted a SECOND token secret at 16:47:52, two minutes *after* the agent applied the first, and `consumed_at` is still NULL; **(b)** the converged state machine will not re-apply, so the box is pinned to a stale secret; **(c)** the agent's 15-minute PBS verify loop **cannot read the credential to detect any of it** (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — the non-root agent writes that file through a root sudo wrapper, then reads it directly). | M | idea | **NEW — discovered by CC while collecting Phase-A evidence; not on the brief's finding list, so its P2-HIGH rank is provisional pending Viktor.** The severity case: this is the DR tier, the failure is silent, and it would surface first at a real restore. Suggested shape: make `consumed_at` authoritative (a fresh unconsumed secret must un-converge the reconciler), fix the verify loop's read path (read via the same root wrapper that writes it), and make a failing `pvesm status` a LOUD state rather than a skipped datastore. **Blocks the DR-tier map row's candidate upgrade to PROVEN-LIVE — that upgrade is now explicitly WITHDRAWN.** Evidence `pbs-dr-state.txt`, `hub-state.txt` |
|
||||
| R-30 | **[P2-HIGH] Liveness presence should come from the wait channel, not the report clock.** The box was powered off at the start of the rehearsal, yet the hub carried it as healthy until the staleness threshold expired ~30 min later (`host_stale` 16:05:24 "no report for 30m"; cleared 16:33:24 "was stale for 27m"). The host-delete guard compounds it: RESET refuses while any host row exists, so a stale-but-"Online" host stalls a forced teardown. | M | idea | Direction: derive presence from **Dir-2 long-poll connectedness (~90 s grace)**, decoupled from notification hysteresis (the hysteresis is right for *alerting*, wrong for *presence*); an agent/ep0 analog can follow. Pairs with R-13/R-23 — the transport already exists, this is about believing it. *(Discussed in-session as "R-29"; that number was already taken by the gate-rot item earlier the same day, so it is R-30.)* |
|
||||
| R-31 | **[P2-HIGH] Offsite provisioning is synchronous with no status affordance.** Save runs the Hetzner sync in-request, so the request can hit the nginx 504 **while succeeding server-side**: the operator cannot tell failed from slow, and a retry races the first attempt. | M | idea | Direction: make it async + a status card, reusing the proven **awaiting-card/poll idiom** (v0.138.0 escrow card). **Interim mitigation belongs in R-3 as an operator note: click once, wait, verify — do not re-click.** |
|
||||
| R-32 | **[P2-HIGH] RESET must purge the customer base dir; the orphan card must stay honest; unattributed bytes must be visible.** The rehearsal's S7 said in advance that an orphan card would BE a finding — and one appeared (16:58:14). Cause: RESET's `"hetzner":"ok"` leg destroys the sub-account, but **a Hetzner sub-account is an access-control object, not a data object** — its directory survives, so re-enabling offsite recreated an account over the previous lifecycle's ciphertext, encrypted under a key that same RESET had destroyed. | M | idea | **Ruling from the run (three parts, deliberately separate):** (1) because RESET destroys custody, the ciphertext it leaves behind is unrecoverable **BY DESIGN** → RESET gains a **main-account purge of the customer base dir** (the existing operator ack already covers it); (2) the **move-aside guard STAYS** for reinstall-*without*-RESET — there custody survives and the card's "history recoverable" promise is true (R-26 depends on exactly that); (3) the operator **Restic tab shows per-customer directory bytes vs attributed snapshot bytes**, so dead data cannot hide. Measured on the pool box that night: **49 M attributed** (2 snapshots, 48.717 MiB) against **1.4 G + 3.0 M unattributed** across TWO `.orphaned-*` dirs. Evidence `restic-and-pool.txt` |
|
||||
| R-33 | **[P2-HIGH] Bootstrap pairing-poll spams the customer-visible console.** Each 30 s pairing poll is a separate `felhom-bootstrap.service` run that exits 1, so systemd prints `Failed to start …` on the screen the customer is watching — **52 FAILED lines in ~11 minutes** while nothing was wrong (the box was correctly waiting to be bound). | S | idea | Fix in **scripts v1.21**: poll *inside* the script rather than via unit restarts, and reword the first-boot noise. Cheap, and it is the customer's literal first impression of the product. Evidence `bootstrap-journal.txt` |
|
||||
| R-5 | Hub: offsite storage visibility — RESTIC box aggregate (v0.64.0) + PBS DR datastore (v0.65.0), each with fill + operator alert; Offsite tabs + dual dashboard gauges | M | **SHIPPED (hub v0.64.0 + v0.65.0 + tenantsync v1.2.0, 2026-07-17)** | Restic: `monitor.OffsiteBoxChecker` (throttled Hetzner GET; fill 80/90% + oversub 2.0×; scope `"pool-box"`). PBS DR (v0.65.0, Option A): read-only `usage` op on the ep0 `felhom-tenantsync` forced command (`df`, no mutation) → `monitor.PBSDRBoxChecker` (fill 80/90%, states ok/unavailable/degraded, scope `"pbsdr-box"`); `/offsite` Restic/PBS-DR tabs + two gauges; hub deploy ⟂ ep0 update (graceful "n/a"). **Both data pipelines PROVEN-LIVE** (restic `0.2% full`; PBS `19.1% full, 7.1 GB of 37.2 GB` after ep0 → v1.2.0). Thresholds pending Viktor (named keys; separate restic/PBS pairs). OPEN: UI render unit-only (password-gated), alert emails unit + red-proof only (both nominal). See hub v0.64.0 + v0.65.0 REPORTs |
|
||||
| R-6 | **Spike: LAN service discovery from the guest** — SSDP multicast (UDP 1900, DLNA), WSD (Windows discovery), mDNS; host-network vs macvlan; is the customer LXC LAN-bridged in appliance deployments? | M | **spiked (2026-07-18)** | **VERDICT: appliance guest IS LAN-bridged (own DHCP lease on the household /24); multicast discovery works ONLY in the guest netns — guest-direct or Docker `--network host` (SSDP/mDNS/WSD all PASS both ways); the default docker bridge is categorically DEAF to LAN multicast (WSD/mDNS RX FAIL, unicast-publish PASS). Real samba+wsdd on host-net → Windows 11 ProbeMatch + FELHOM-SPIKE renders in Explorer + 445 + authenticated SMB round-trip all PASS; real SSDP `MediaServer:1` advert reaches both LAN clients. → R-7 SMB stack MUST be host-network LAN-bound; R-8 Jellyfin-DLNA plausible if host-network. Caveat: `vmbr0 multicast_snooping=1` worked only because the household router is a live querier — customer LANs w/ snooping+no-querier, and Peti's BYO bridge, are UNTESTED gaps.** **S4b (human leg, the sharpest finding): wsdd makes the box VISIBLE but the Explorer double-click FAILS `0x80070035` — WSD gives no name resolution; the flat `\\FELHOM-SPIKE` resolved by no path. Adding `nmbd` (NetBIOS) fixed it live (flat name resolves + mounts). → R-7 needs smbd+wsdd+nmbd (+avahi/.local for modern clients), not wsdd alone.** Doc: `audits/SPIKE-lan-discovery-2026-07-18.md`. |
|
||||
| R-7 | SMB **server** share — LAN file sharing as an embedded controller feature („Megosztás") | L | **SHIPPED slice 1 (controller v0.144.0 + `felhom-samba:1.0.0`, 2026-07-18)** | Flips map row *Files from Windows Explorer / Mac Finder (SMB server)* MISSING → **IMPLEMENTED**. Delivered as the FOURTH protected infra stack (host-net, smbd+**nmbd**+wsdd, `bind interfaces only = lo eth0`), NOT a catalog app. „Megosztás" → „Hálózati megosztás": enable + ONE household password (STDIN→smbpasswd, never persisted — only `user_set`), shares table, create flow (new folder under `<storage>/shares/` or an existing folder via a guarded picker). Picker guard = absolute → EvalSymlinks → containment in a registered LIVE root → deny-listed system subtree (derived from `ProtectedHDDPaths`, provably a subset) → is-a-dir, with UNIFORM refusals (no filesystem oracle). Per-share read-only enforced in smb.conf AND as a `:ro` bind. `force user = felhom` → SMB writes land as uid 1000. Live-validated on demo through the real endpoints + a Win11 workstation (445 True, NetBIOS `FELHOM` resolves, write/read byte-compare PASS, **write to a read-only share REFUSED**, WSD ProbeMatch). **Explorer leg PASSED 2026-07-18 (Viktor):** both shares open from the Network view; an Explorer save into the writable share landed as uid 1000 and a write into the read-only share was refused, folder untouched. Slice 1 is fully PROVEN-LIVE. Docs: `controller/sharing.md`; ship report `felhom-controller/REPORT.md`. **Slice 2 remainder:** avahi/`.local` for modern+Apple clients, curated app-folder presets (one-click paperless `consume/`), per-share users + guest access, recycle bin. **Backup gap split out → R-7b.** |
|
||||
@@ -43,24 +54,43 @@
|
||||
| R-19 | Internet-outage customer-experience drill: pull WAN on demo, verify lan_resolver path, document what the customer actually sees/does | S | idea | Flips map row E "LAN access" IMPLEMENTED→PROVEN-LIVE |
|
||||
| R-20 | ~~Verify operator-key pinning is fully in the day-0 install flow~~ | XS | **closed** (2026-07-16) | Confirmed against `scripts/felhom-host-install.sh` source (not changelog): keys resolve at L1181–1219 (script constants `OPERATOR_KEY_*`, populated, `--operator-pubkey-file` override), pinned automatically by `step_agent_config()` "STEP 6/8" (L2044; python builds `authz.signers` L2146–2156, reinstall preserves existing), verified at L2332–2337 ("authz signers: N … operator-signed self-update armed"). No interactive prompt or post-install hand-edit — fully automatic. Doc-drift note: the L193–197 "EMPTY by default" comment is stale vs the now-populated constants (→ R-16 hygiene) |
|
||||
| R-23 | **Immediate-sync Direction-2 follow-ups** (hub v0.58 / controller v0.140, shipped 2026-07-16): (a) live-validate the operator-UI **save→apply** round-trip end-to-end — needs an operator login, so fold into a Peti/alpha supervised session (one config save → box wakes in seconds → self-restart → startup report; also demonstrates the restart *single-fire* once a bump has advanced the generation past 0); (b) cosmetic: the controller Waiter's "recovered" INFO logs on the next hold completion (`pollOnce` blocks ~240 s), not at reconnect — surface recovery at connect time | S | idea | Flips the new map row "config/state change round-trips in seconds" PARTIAL→PROVEN-LIVE. Transport + mechanism already proven live (`SPIKE-immediate-sync-transport-2026-07-16`, v0.58/v0.140 REPORTs: 240 s no-annotation hold, 0.047 s wake, restart 1-WARN/0-storm); only the login-gated UI-triggered bump leg is unexercised |
|
||||
| R-34 | **Backup data lifecycle management.** An "inactive backups" section on „Távoli mentés": apps that have snapshots but no active backup — **disabled OR uninstalled** — listed with name / size / last snapshot / restorable, plus an explicit **double-confirmed per-app delete** via `restic forget --tag` + nightly prune. | M | idea | **RULING: the offsite toggle NEVER offers deletion — policy and destruction stay decoupled.** Turning backups off must never be a data-destroying act, and deletion must never hide behind a toggle. Origin: 2026-07-18 rehearsal. Pairs with R-32 (that one is the operator's view of dead bytes; this one is the customer's) |
|
||||
| R-35 | **Config-apply should not end the customer's session.** The offsite config push bumped `config_version` 10→11 at 16:54:58 and the controller self-restarted (container `StartedAt` 16:54:59Z, back up 16:55:02); in-memory sessions died with it and **customer zero was force-logged-out mid-flow**. | S | idea | Direction: **hot-apply the offbox target** (no restart for a config the running process can adopt), or **persist sessions** across restart. The restart itself is by design — the collateral is not. Evidence `controller-log-full.txt` |
|
||||
| R-36 | **Post-RESET re-enroll leaves offsite "enabled but unprovisioned" — silently.** The hub knows the state and says nothing on the customer page. | S | idea | Source-cited behaviour, confirmed live in the rehearsal: **provisioning is Save-triggered** (`configs.go` `applyOffsite`) — which also answers S6's open question — and the re-enroll auto-re-issue **correctly** skips unprovisioned targets (`handler.go`). So nothing is broken; the gap is that nobody is told. Direction: flash it on the customer page. **Interim: an R-3 step.** **Related sub-item:** auto-mint the **self-bind link on customer create/RESET**, so the console banner's „e-mailben kapott link" is always already true instead of true-once-the-operator-remembers |
|
||||
| R-27c | **Customer self-bind, slice 2 — console-passphrase bind.** Viktor's direction: bind using a passphrase shown on the box console, alongside (not instead of) the emailed capability link. | M | idea | **Security constraints from the session ruling, all load-bearing:** passphrase **issued at customer creation**; the global-lookup endpoint must be **spray-hardened** — per-appliance **and** per-IP caps, constant-time comparison, a **single generic failure** (no oracle), alerting on abuse; an **accent-free wordlist** (console keymaps are not Hungarian); the **web capability-link path is RETAINED**; **claim-by-email is RETAINED** as the delivery-channel proof. **Also under this item:** the self-bind email gains the **public universal-ISO download link + two-line instructions** (the DIY case). **Secret-bearing per-customer ISOs are ruled OUT.** Sibling of R-27b (second-box flow) — different axis, both build on the same `/bind/` page |
|
||||
|
||||
## P3 — post-alpha
|
||||
|
||||
| ID | Item | Size | Status | Notes |
|
||||
|----|------|------|--------|-------|
|
||||
| R-26 | **Guided old-history recovery via a retained superseded escrow + the recovery code.** Enabled by hub v0.60.0 (Part B) which now RETAINS superseded escrow blobs (`host_escrow_superseded`, `ListSupersededEscrow`). Build the flow that, given the customer's recovery code, unwraps a retained old blob → recovers the old repo passphrase → mounts/reads the moved-aside `.orphaned-<date>` repo for restore. | M | idea (enabled by v0.60.0) | Turns "history recoverable in principle" into a real customer-drivable path; pairs with the controller v0.142.0 orphaned-repo move-aside. Origin `DIAGNOSE-offbox-repo-orphaned-2026-07-17` |
|
||||
| R-27 | **Customer-facing self-bind page (R-21 slice C follow-on).** Today an unclaimed appliance is bound by the OPERATOR on the Hosts page (hub v0.62.0). Build the customer-facing flow so a customer can claim/bind their own freshly-installed box. | M | **SHIPPED (slice 1, hub v0.66.0, 2026-07-17)** | **Slice 1 = the FIRST-box flow, DONE:** operator "Send self-bind link" → 7-day tokenized capability link over Hungarian email → public two-factor `/bind/<token>` (console pairing code shown on the box screen + retrieval passphrase) → hub stages the bind via the same `BindAppliance`, no operator. Viktor's 3 rulings verbatim (console code not a list; operator-sent link; 5-attempt lockout→call support). No oracle; expiry falls back to operator-bind; THE TRAP `/bind/` exemption tight (single predicate, 4 red-proofs). GC verdict §3 (no appliance GC → the 7-day TTL stands alone). **Live first-run + new-ISO console banner fold into the supervised rehearsal (R-1).** Reused the appliance_registrations + one-shot delivery machinery; the capability link is the whole customer-auth surface (no login system built). Controller/agent untouched. See hub v0.66.0 REPORT. **Multi-box per link is out of scope by design** (single-active, one-shot → binding several boxes = repeated operator sends) |
|
||||
| R-27 | **Customer-facing self-bind page (R-21 slice C follow-on).** Today an unclaimed appliance is bound by the OPERATOR on the Hosts page (hub v0.62.0). Build the customer-facing flow so a customer can claim/bind their own freshly-installed box. | M | **SHIPPED (slice 1, hub v0.66.0, 2026-07-17)** | **Slice 1 = the FIRST-box flow, DONE:** operator "Send self-bind link" → 7-day tokenized capability link over Hungarian email → public two-factor `/bind/<token>` (console pairing code shown on the box screen + retrieval passphrase) → hub stages the bind via the same `BindAppliance`, no operator. Viktor's 3 rulings verbatim (console code not a list; operator-sent link; 5-attempt lockout→call support). No oracle; expiry falls back to operator-bind; THE TRAP `/bind/` exemption tight (single predicate, 4 red-proofs). GC verdict §3 (no appliance GC → the 7-day TTL stands alone). **Slice 1 PROVEN-LIVE 2026-07-18:** in the rehearsal the operator minted + emailed the link at 16:28:55 and **the customer bound their own box at 16:29:55 with zero failed attempts**, `appliance_bound` carrying source `customer_selfbind`, credential delivered 26 s later with no operator action (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). The new-ISO console banner rendered on the real box. **Slice 2 = console-passphrase bind → R-27c** (below). Reused the appliance_registrations + one-shot delivery machinery; the capability link is the whole customer-auth surface (no login system built). Controller/agent untouched. See hub v0.66.0 REPORT. **Multi-box per link is out of scope by design** (single-active, one-shot → binding several boxes = repeated operator sends) |
|
||||
| R-27b | **Customer self-bind, second-box flow (controller side).** For a customer who ALREADY has a bound box and installs another, the controller shows a dismissable "bind another box" prompt (and a bind-later entry under settings) that walks to the hub `/bind/` page — so a returning customer isn't emailed a fresh operator-sent link for every box. Mechanism sketched in the hub v0.66.0 REPORT; NOT built (R-27 slice 1 deliberately did not touch the controller). | M | idea (minted by hub v0.66.0) | Origin: hub v0.66.0 slice-1 ship (first-box only). Reuses the same `/bind/` public page + tokenized-link machinery; adds a controller-side entry point + the operator "mint a link for an existing customer" affordance |
|
||||
| R-25b | **Customer DELETE ↔ RESET consistency.** The middle-tier Customer RESET (hub v0.61.0) runs the full external teardown (Hetzner sub-account/box + PBS namespace/groups/token) and refuses while any host row exists. The Danger-zone DELETE still (a) leaves host rows and (b) does NOT run that teardown — it purges escrow custody + drops the config only. Decide the model: DELETE requires a prior RESET, or DELETE subsumes RESET's teardown, or they stay orthogonal (RESET = recycle-in-place, DELETE = escrow-purge). | S | idea | Origin: hub v0.61.0 RESET ship. Flips a future "customer fully offboarded (external resources released)" map row. Cheap once the model is chosen |
|
||||
| R-25 | **Device-node TOCTOU hardening (drive init).** Graduate the controller v0.141.0 Observation: the `format → resolveEnrollUUID(path) → AssignDisk(uuid)` sequence has a narrow /dev-re-enumeration window (agent-guarded on the destructive format via anti-retarget durable-id; benign fs-UUID mount). Bind resolve+assign to the format's durable-id so the mount can't target a moved node. | S | idea | From the v0.141.0 F6 commit's security-review finding (`felhom-controller` REPORT). Low real risk (single-operator, agent-guarded), but cheap to close |
|
||||
| R-24 | **Guest RAM resize (live) — SHIPPED (agent v0.90.0 + controller v0.143.0, 2026-07-17).** The customer right-sizes the guest's memory from the controller's Rendszer page; the agent enforces every bound fresh + applies via PVE `SetConfig` (live cgroup, no reboot, Phase-0 proven). **Framing note:** the original hub-desired-state framing is SUPERSEDED by Viktor's controller-direct ruling (2026-07-17) — the resize is controller→agent local-API, never through the hub. Memory only (cores stay observation). Deployed + live-validated on the demo (refusals render Hungarian end-to-end); a successful grow on a normal-sized box folds into the rehearsal. **Cores/live-resize as hub desired-state is NOT built** (deferred, low demand). | M | **SHIPPED** | See felhom-agent + felhom-controller REPORTs; capability-map row "customer right-sizes guest RAM". |
|
||||
| R-24 | **Guest RAM resize (live) — SHIPPED (agent v0.90.0 + controller v0.143.0, 2026-07-17).** The customer right-sizes the guest's memory from the controller's Rendszer page; the agent enforces every bound fresh + applies via PVE `SetConfig` (live cgroup, no reboot, Phase-0 proven). **Framing note:** the original hub-desired-state framing is SUPERSEDED by Viktor's controller-direct ruling (2026-07-17) — the resize is controller→agent local-API, never through the hub. Memory only (cores stay observation). **COMPLETE — the apply is proven in BOTH directions on metal (2026-07-18):** customer zero shrank the guest **11675 → 8192 MB (16:50:22)** and grew it **8192 → 12288 MB (17:02:17)**, each a live cgroup apply with no reboot, the new total rippling into the deploy page's memory math. The nested demo had proven the refusals; the N100 proved the applies. `tests/VALIDATION-n100-rehearsal-2026-07-18.md` ledger 9. **Cores/live-resize as hub desired-state is NOT built** (deferred, low demand). | M | **SHIPPED + PROVEN-LIVE** | See felhom-agent + felhom-controller REPORTs; capability-map row "customer right-sizes guest RAM". |
|
||||
| R-12 | Cluster mode: agent-follows-guest, bind-mount reconciliation on HA migration | XL | idea | Scoped 07-15; interim = HA-group pin to one node. Driven by Peti's two-node cluster |
|
||||
| R-13 | OOB management arc: dual-use existing WireGuard + hub desired-state channel as mutual-repair | L | **first slice PROVEN-LIVE (poke channel)** | **FIRST SLICE PROVEN-LIVE — the agent-plane poke channel (Direction-2a), agent v0.89.0 + hub v0.59.0 (2026-07-17):** the ep0-relayed contentless poke (hub→ep0 `felhom-poke` forced-cmd→UDP→box WG /32:51822, peer-confined, zero ep0/box infra change) reaches the **agent** and fires an immediate desired-state cycle. Full path live: real operator manifest save → `sync-poke delivered to 10.77.0.2`; box → `poke received → immediate desired-state cycle` (~31 ms ep0→box, save→tick ≈ ~0.45 s). This is ONLY the listener+sender; the rest of the mutual-repair arc (self-heal actions over the channel) stays open. Per `SPIKE-immediate-sync-transport-2026-07-16` P4. The controller-plane Direction-2 wait channel (hub v0.58 / controller v0.140) shipped the config-puller leg separately |
|
||||
| R-28 | **Agent fast-tick-until-first-convergence — SHIPPED (agent v0.90.0, 2026-07-17).** Closes the immediacy leg a hub poke can't reach (the pre-tunnel WG-registration window): while ANY desired-state item is unapplied the agent pulses the out-of-band trigger every 30 s and **self-disarms** on convergence (state-based; four cached sources — desired-gen==0, reconcile Planned−Pending>0, pbsdr waiting_secret ONLY, wgtunnel desired-not-operational; LOUD states + pending_signature excluded). LIVE on both demo agents (startup line verified). **Guests-0/0 passenger:** diagnosed live and REFUTED as a bug — the 0/0 was the legitimate pre-provision reporting window; the fast-tick is exactly its mitigation (≤15 min → ~30 s). **Unfired live:** the real-onboarding proof (a fresh box converging in seconds without a poke) → the rehearsal S5. | M | **SHIPPED** | See felhom-agent REPORT (v0.90.0). Third immediacy leg alongside R-13 (poke) + R-23 (UI). |
|
||||
| R-14 | Headscale/WireGuard spike: Minecraft/gaming port connectivity (CGNAT-proof, sovereign DERP fallback) | M | idea | |
|
||||
| R-15 | Multi-user dashboard accounts (household members, roles) | L | idea | Single password is a stated alpha limitation (R-11) |
|
||||
| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED (slices A+B+C, 2026-07-17)** — physical N100 boot + the live boot→bind→day-0 composition fold into the supervised rehearsal (R-1) | **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needs ONE real bare-metal run): vendor DMI serials (empty on virt), real firmware/NIC quirks, a pre-existing-LVM wipe step. Pipeline notes: assistant pairs to the ISO by Debian codename, ~11.5 s/ISO on DooPlex, **gate on `validate-answer` output not `$?`** (exit 0 on failure). Would flip a new capability-map MISSING row "customer self-installs on bare hardware" once spec'd. *(brief called this R-22)* |
|
||||
| R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v <volumeName>:/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff |
|
||||
| R-21 | **Bare-metal Felhom ISO** — per-PVE-release auto-install ISO for blank customer hardware → first-boot wrapper (invokes `felhom-host-install.sh`) → universal secret-free / operator-bind (option C) | XL | **SHIPPED + PHYSICALLY CLOSED (slices A+B+C; rehearsal executed 2026-07-18)** | **PHYSICAL CLOSURE 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the generic pairing ISO v1.20.0 (`--loader mkimage`, SB off) **booted the very AMI board that F1 blocked**, installed unattended, and the box self-registered as an unclaimed appliance the same second it first booted (16:17:14) → self-bind → credential → day-0 SUCCESS 16:32:32 → floor-lifted to current. **F1 is closed on physical hardware and the rehearsal dependency on this item is discharged.** Two residual notes stay open, neither blocking: PXE/network-boot is still unbuilt (the third F1 option, for boards where even USB-mkimage fails), and mkimage remains unsigned → **SB must be OFF**. The installer's GRUB menu still offers interactive installers → new item **R-38**. — Prior: **PHYSICAL RUN 2026-07-16 (`tests/VALIDATION-n100-baremetal-2026-07-16.md`):** demo N100 reinstalled clean-slate from a pipeline ISO → chain reached **rc-0 first try on real hardware** (closes slice A's operator-gated boundary), serial-filter safety proven on metal, PBS-DR reconciler self-healed on the reused peer, DMI verdict = key on MAC+UUID. **F1 (HIGH, slice-B input):** this cheap AMI `AN3PLUS 0.01` firmware won't UEFI-boot the ISO's GRUB from USB (`relocation 0x0`) — SB-off/shim-bypass don't help; worked around live with a `grub-mkimage` loader built from the box's own GRUB. Pipeline must ship a firmware-compatible loader / PXE path. Reused-customer edges (F2 claim re-issue, F3 offsite re-issue, F4 non-default-storage-id ACL 403) feed R-1/Peti. UX: F6 drive-init doesn't mount+attach, F5 guest-RAM not configurable, F7 back-route. — **Slice A (build pipeline + first-boot bootstrap) DONE + validated on VM 310:** build gate/red-proof, disk-filter fail-safe, stub→retry-unit→real public-channel host-install fetch+invoke→retry, resume-decision, exactly-once, no-net retry+recovery all GREEN. Operator-gated remainder: host-install rc-0 terminal success (drill customer needs the password-gated create-UI). **Slice B — SHIPPED (scripts v1.18.0, 2026-07-17):** the F1 firmware fix is now a first-class pipeline mode `build-felhom-iso.sh --loader shim|mkimage` (default shim; `mkimage` = monolithic grub-mkimage loader from the ISO's own GRUB, recipe from the run evidence). RUNBOOK-B legs on nested VM 311 proved it: shim boots+installs under OVMF SB-enforcing + SeaBIOS; mkimage boots+installs under OVMF SB-off; mkimage under SB-enforcing FAILS `Access Denied` (unsigned → **SB must be OFF**, documented); surgery byte-identical payload. **Physical N100 boot on the real board still pending** → folds into the supervised rehearsal (R-1; an `n100-safety` match-nothing ISO is built + sha-recorded for a zero-risk pre-flight). **PXE/network-boot** (the third F1 option, for boards where even USB-mkimage fails) stays a deferred note under this item — not built. **Slice C — SHIPPED (hub v0.62.0 + scripts v1.19.0, 2026-07-17):** the GENERIC secret-free universal ISO (`build-felhom-iso.sh --pairing`). The box self-registers as an unclaimed appliance (keyed by SMBIOS-uuid + MAC set — the DMI-verdict tiebreaker), the operator BINDS it to a customer on the Hosts page, and the hub delivers customer-id + retrieval passphrase ONCE (`/api/v1/appliance/register` + one-shot poll, 404-no-oracle — all live-verified through the public ingress); the bootstrap then falls through to the slice-A direct path. **One unit, two modes** (direct = byte-identical, regression-proven zero-appliance-calls). Artifact proven secret-free (baked env = hub URL only; manifest `secret-bearing: no`). The **bind is operator-password-gated** → the live boot→register→bind→day-0 composition (with a Viktor-created drill customer) + the physical N100 boot fold into the supervised rehearsal — **which now runs the COMPLETE final product flow in one pass: RESET the demo → boot the generic ISO → bind → day-0**. SSH-host-key pinning: stored + fingerprints displayed (attaching to the host on bind = future, no clean hand-off surface today). **Customer-facing self-bind page = R-27 (future).** Origin spike `audits/SPIKE-baremetal-iso-2026-07-16.md`: every mechanism GREEN on nested virt (VM 310 on felhom-pve). Zero-touch install BIOS **and** UEFI incl. **Secure Boot enforcing** (no MOK/keypress); first-boot hook `fully-up` = root + working pvesh/pct, exactly-once via `pending-first-boot-setup` flag; post-install **webhook** carries SMBIOS-UUID + management-MAC + host SSH keys → the unclaimed-appliance record; disk-filter installs only the target (canary byte-identical) and **fails-safe** on match-nothing / bad disk; `from-url` + `cert-fingerprint` **fails CLOSED**. OPEN (needLine truncated
|
||||
| R-29 | **The design-v2 green gates are not enforced anywhere — one has been RED for 16 releases.** `controller/scripts/docker_run_volume_path_gate.py` has failed continuously since **2026-07-14 (v0.129.0)** and nobody noticed until R-7b's close-out ran it by hand at v0.145.0. Two separable parts. **(a) The finding itself is benign and the fix is 3 lines.** The flagged call is `internal/appexport/estimate.go:179` `docker run --rm -v <volumeName>:/vol:ro alpine du` — a **NAMED-VOLUME** mount, i.e. daemon-side with no host path, which is the *safe* shape and byte-for-byte the same pattern as three entries already on the gate's ALLOWLIST (`export.go` `volName+":/vol"`, `backup.go` `volName+":/vol:ro"`, `restore.go` `volName+":/vol"`). It is NOT the v0.124.0 path-strand class the gate exists to catch — the author of the v0.129.0 F-A fix explicitly avoided that class (see the function's own comment) and simply never added the allowlist entry. So the fix is an ALLOWLIST addition WITH ITS WHY, **not** a docker-cp rewrite; anyone who 'fixes' this by rewriting the call has misread the gate. **(b) The systemic half is the real item:** the gates run only when a human remembers to run them, so a gate can sit red across 16 releases while every REPORT says 'green'. This is the SECOND instance of the class — cf. the v0.123.0 note *'Windows green gate silently red (read-only fsync)'*. Decide where they run (pre-push hook, `build.sh` step, or a CI job) and make a red gate block the train the way the Go green gate does. | S (a) / M (b) | idea | Origin: R-7b close-out, `felhom-controller` REPORT §4(f) — CC correctly left it alone as out-of-scope and pre-existing, and verified by stashing that it fails identically on the unmodified tree. Flips no capability-map row (engineering hygiene, no customer-visible behaviour). Affected gates to audit for the same rot: controller `template_id_gate` / `emoji_gate` / `native_confirm_gate` / `offbox_rename_gate` / `mojibake_gate` / `app_row_dedup_gate` / `docker_run_volume_path_gate`, hub `hub_confirm_gate`, manifests `manifest_bearer_gate`, website `site_gates`. **Do not bundle (a) into an unrelated feature commit** — it is a one-line behavioural claim about a mount's safety and deserves its own reviewed diff. **2026-07-18 rehearsal note:** the run's finding list independently re-raised "assign the pre-existing `docker_run_volume_path_gate` failure its ID so red stops normalizing" — **that is this item; no second ID was minted.** |
|
||||
|
||||
| R-37 | **Post-RESET health card shows stale pre-RESET warnings.** After a RESET the card should read **„RESET óta nincs adat"** instead of carrying warnings about a lifecycle that no longer exists. | XS | idea | Origin: 2026-07-18 rehearsal. Same family as R-36 — the hub knows the state changed and the UI has not caught up |
|
||||
| R-38 | **Installer GRUB slice.** A single default „Felhom telepítés" entry; the **interactive installers REMOVED** (safety: an interactive entry is how a wrong-disk manual install happens); felhom background. | S | idea | Origin: 2026-07-18 rehearsal, alongside R-21's physical closure. **Squashfs/theme rebranding explicitly DEFERRED** — this item is the menu and the safety, not a skin |
|
||||
|
||||
## Pre-invite checklist — what stands between here and the first remote tester
|
||||
|
||||
Not roadmap items in their own right; the short list the 2026-07-18 rehearsal leaves behind.
|
||||
Everything here is **remote-doable** — the N100 is packed, and none of it needs hands on the box.
|
||||
|
||||
| Action | Owner | Note |
|
||||
|---|---|---|
|
||||
| **Rebuild the golden → 0.145.x** | Viktor + CC train | The rehearsal banked the floor-lift evidence (0.143.0→0.145.0 unattended in 5 s), so the mechanism is proven — but a fresh box should not *start* two versions stale and then self-update in front of the customer. Do this before anyone is invited |
|
||||
| **freemail.hu test-send** | Viktor | The open half of R-4; the gmail half closed on 2026-07-18 under `p=quarantine` |
|
||||
| **C6 — customer performs a restore, unassisted** | Viktor as customer zero | The one open script step in R-3 and still MISSING as capability evidence. Remote-doable on the reborn box — the dashboard is remote |
|
||||
| **R-11 rulings** | Viktor | Contact channel, tester agreement, alert thresholds (the R-5 gauge thresholds are still pending a ruling) |
|
||||
|
||||
## Absorbed / superseded notes in this folder
|
||||
|
||||
|
||||
@@ -1,17 +1,32 @@
|
||||
# DRAFT R-3 — Friend-alpha onboarding runbook (operator)
|
||||
|
||||
> **Status: DRAFT, pre-rehearsal.** Generalized from `runbooks/RUNBOOK-rehearsal-v3.md` (S1–S8) and
|
||||
> the shipped claim/escrow/self-bind flows. The N100 rehearsal EXECUTES this document; every
|
||||
> `[REFINE]` slot gets filled from its friction notes, then this graduates to
|
||||
> `documentation/runbooks/RUNBOOK-onboarding.md` and the rehearsal item collapses.
|
||||
> **Status: DRAFT, post-rehearsal — all `[REFINE]` slots filled 2026-07-18.** Generalized from
|
||||
> `runbooks/RUNBOOK-rehearsal-v3.md` (S1–S8) and the shipped claim/escrow/self-bind flows, then
|
||||
> refined against the N100 rehearsal that executed it
|
||||
> (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`). Every step here has been walked on real
|
||||
> hardware **except C7**, the customer-performed restore.
|
||||
>
|
||||
> **It graduates to `RUNBOOK-onboarding.md` when C7 has evidence** — that step is simultaneously the
|
||||
> last unwalked step of this script and the capability map's last MISSING customer-facing row, so
|
||||
> one execution closes both. Until then this stays a DRAFT.
|
||||
>
|
||||
> Scope: **appliance hardware only** (BYO track is R-1/Peti, explicitly out of alpha scope).
|
||||
> Three steps below carry **interim workarounds for known defects** (R-31, R-36, R-39) — they are
|
||||
> marked inline and should be deleted from this runbook the moment those items ship.
|
||||
|
||||
## A. Before the visit (operator, ~30 min, all remote-doable)
|
||||
|
||||
1. **Hub:** create the customer (name, e-mail, domain `<név>.felhom.eu`) — verify the CF tunnel +
|
||||
zone token provisioning completes. Confirm the Day-0 manifest is current (agent, golden,
|
||||
MinAgent) and the managed-update floor is at the intended release — a stale golden is fine
|
||||
(the floor lifts the box post-install), a stale FLOOR is not.
|
||||
(the floor lifts the box post-install), a stale FLOOR is not. *The rehearsal proved the lift:
|
||||
a 0.143.0 golden reached 0.145.0 in **5 s**, unattended, inside the first minute of controller
|
||||
life.* Even so, **rebuild the golden before the first real tester** — proven-recoverable is not
|
||||
a reason to ship stale.
|
||||
|
||||
**Mint the self-bind link now** ("Send self-bind link" on the customer's Setup tab). It is valid
|
||||
for **7 days**, and doing it here means the console banner's „e-mailben kapott link" is already
|
||||
true when the customer first looks at the screen. See B3.
|
||||
2. **Hardware:** the standard appliance + its external HDD. One-time bench prep list:
|
||||
memtest pass, SMART clean, BIOS queue for the single visit in B2.
|
||||
3. **Install media:** the **reusable generic pairing ISO** (secret-free, slice C). Only rebuild it
|
||||
@@ -30,39 +45,98 @@
|
||||
**pairing-code banner**; the box appears under **Unclaimed appliances** at the hub
|
||||
(verify uuid/MAC/SSH fingerprints match the physical box before binding — never bind a
|
||||
fingerprint you didn't read off the console).
|
||||
3. **Bind** to the customer. Self-bind path (customer enters the code themselves) preferred when
|
||||
the customer is present — it is the flow real remote onboarding will use. `[REFINE: which path
|
||||
the rehearsal used + friction]`
|
||||
3. **Bind** to the customer — **the rehearsal used the SELF-BIND path, and it is now the default for
|
||||
this runbook.** Mint the link with **"Send self-bind link"** on the customer's Setup tab; the
|
||||
customer opens it and completes the public two-factor `/bind/<token>` page (console pairing code
|
||||
+ retrieval passphrase). The 2026-07-18 run bound in **60 s from mint to consume, with zero failed
|
||||
attempts**, and the credential was delivered **26 s later** with no operator action.
|
||||
|
||||
> **Send the self-bind link BEFORE the customer sees the console.** The console banner tells them
|
||||
> to use „az e-mailben kapott link" — if the link has not been minted yet, the box is instructing
|
||||
> the customer to look for something that does not exist. In the rehearsal the box sat in pairing
|
||||
> mode for **~11.7 minutes** waiting on the operator, printing failures the whole time (R-33).
|
||||
> Mint it during step A1 if you can; the token is valid for **7 days**, so there is no reason to
|
||||
> wait. Auto-minting on customer create/RESET is R-36.
|
||||
|
||||
Operator-bind on the Hosts page remains the fallback when the customer is not present or the
|
||||
token has expired.
|
||||
4. Watch the event sequence compose (expected: **seconds** between post-enroll legs — enroll →
|
||||
claim-code issue → WG → fast-tick window → tunnel → PBS-DR descriptor poke → controller up →
|
||||
claim e-mail lands → floor lift to current). **Any ~15-minute stall between two legs is a
|
||||
finding** — capture which leg + wall-clocks. `[REFINE: reference wall-clock table from the
|
||||
rehearsal]`
|
||||
finding** — capture which leg + wall-clocks.
|
||||
|
||||
**Reference wall-clocks — measured on the N100, 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`).
|
||||
Compare against these; a leg that takes minutes where this table says seconds is the finding:
|
||||
|
||||
| Leg | Expect |
|
||||
|---|---|
|
||||
| Blank-metal boot → registered as an unclaimed appliance | **< 1 s** (same second) |
|
||||
| Bind → credential delivered | **26 s** |
|
||||
| Bind → agent enrolled + WG tunnel up | **56 s** |
|
||||
| Bind → controller running the CURRENT version | **2 min 44 s** |
|
||||
| Credential delivered → day-0 provision SUCCESS | **2 min 11 s** |
|
||||
| Managed-floor lift (golden version → current) | **5 s**, unattended |
|
||||
| Drive enrol: destructive-confirm → usable (incl. mkfs) | **~30 s** |
|
||||
| Escrow ceremony started → offsite runs enabled | **12 s** |
|
||||
|
||||
Note the **PBS-DR descriptor** leg specifically: in the rehearsal it reported `converged` while
|
||||
the credential behind it was already stale, leaving the DR tier authenticating `401` with nothing
|
||||
surfacing the fault (**R-39**). Until that is fixed, treat "descriptor provisioned" as unproven
|
||||
and check `pvesm status` on the box during step E1.
|
||||
|
||||
## C. Customer zero script (the customer drives, operator narrates — ~30 min)
|
||||
|
||||
This is the friction lens: the customer's hands on the keyboard, the operator's hands OFF.
|
||||
|
||||
1. **Claim:** open the claim e-mail → claim the dashboard → set the dashboard password.
|
||||
(R-4 gate: the e-mail must land in inbox, not spam — gmail proven; `[REFINE: freemail.hu
|
||||
result]`.)
|
||||
(R-4 gate: the e-mail must land in inbox, not spam. **gmail PROVEN 2026-07-18** — the rehearsal's
|
||||
claim mail was the first sent under the tightened DMARC `p=quarantine` and landed in the **Inbox**.
|
||||
**freemail.hu is still untested** — if your tester is on freemail.hu, treat step C1 as unproven and
|
||||
have them check spam before you conclude the mail was lost.)
|
||||
2. **Escrow ceremony:** run the recovery-code wizard → the awaiting card
|
||||
(„megerősítésre vár, legfeljebb 15 perc") → flips on ACK. The code goes on paper, into the
|
||||
envelope, into the customer's drawer — **state plainly: without this code, a dead box means
|
||||
the remote backup is unrecoverable; we cannot read it for them.** That sentence is the product.
|
||||
3. **Storage:** enroll/format the external HDD via the wizard (customer clicks, operator explains
|
||||
the destructive-confirm).
|
||||
4. **First app:** deploy one app the household actually wants (paperless or immich recommended) —
|
||||
the destructive-confirm). Expect **~30 s** from the confirm to a usable drive — the wizard runs
|
||||
mkfs detached and polls, so a slow format is not a hang.
|
||||
|
||||
4. **Offsite:** enable „Távoli mentés". Two operator behaviours here are **workarounds for known
|
||||
defects** — follow them until the items ship, and do not improvise:
|
||||
|
||||
> **Click Save exactly ONCE, then wait, then verify.** Provisioning runs synchronously inside the
|
||||
> request, so on a slow Hetzner leg the request can return an nginx **504 while succeeding
|
||||
> server-side**. A second click races the first attempt. Wait, then confirm on the hub's
|
||||
> Offsite → Restic tab rather than believing the browser. (**R-31**)
|
||||
|
||||
> **Re-onboarding an existing customer? Check offsite provisioning by hand.** After a RESET +
|
||||
> re-enroll the hub can sit in "enabled but unprovisioned" and **say nothing** — provisioning is
|
||||
> *Save*-triggered, and the re-enroll auto-re-issue deliberately skips unprovisioned targets. So
|
||||
> an offsite target that looks configured may never have been created. Open the offsite settings
|
||||
> and press Save once to force provisioning, then verify. (**R-36**)
|
||||
|
||||
> **Expect the first run to be clean on a NEW customer.** If an **orphan card** appears
|
||||
> („a távoli mentési tároló elárvult"), the repo is not new — that is the R-32 case, which fires
|
||||
> when a customer's pool directory outlived a RESET. It is safe (the guard moves the old repo
|
||||
> aside, never deletes it) but it means the box is writing into a reused directory. Record it.
|
||||
5. **First app:** deploy one app the household actually wants (paperless or immich recommended) —
|
||||
through the catalog UI, no shortcuts.
|
||||
5. **Sharing:** enable „Megosztás", set the household SMB password, create one share; open
|
||||
6. **Sharing:** enable „Megosztás", set the household SMB password, create one share; open
|
||||
`\\NÉV` from their own laptop. (Set expectations per the one-pager: device list may lag on
|
||||
some routers; the address always works.)
|
||||
6. **Protection proof — the customer performs a restore, unassisted** (this step IS the
|
||||
7. **Protection proof — the customer performs a restore, unassisted** (this step IS the
|
||||
capability-map evidence, per customer): drop a file on the share or into the app → „Távoli
|
||||
mentés most" → wait for the snapshot → delete the file → the customer restores it via
|
||||
**Visszaállítás** with the operator silent. If they can't without help, that's an R-11/UX
|
||||
finding, not a customer failure — write it down verbatim. `[REFINE: rehearsal friction notes]`
|
||||
7. **Alerts sanity:** show them what an alert e-mail looks like (the health-degradation mail);
|
||||
finding, not a customer failure — write it down verbatim.
|
||||
|
||||
> **⚠ THIS IS THE ONE STEP THE REHEARSAL DID NOT EXECUTE.** Every other step in this document was
|
||||
> walked on real hardware on 2026-07-18; C7 was not, so it carries **no friction notes and no
|
||||
> evidence** — the capability-map row *"a customer (not the operator) performs a restore via UI
|
||||
> alone"* is still **MISSING**. This runbook stays a **DRAFT** until C7 has been performed once.
|
||||
> It is remote-doable on the reborn demo box (the dashboard is remote) and does not need the
|
||||
> hardware in hand.
|
||||
8. **Alerts sanity:** show them what an alert e-mail looks like (the health-degradation mail);
|
||||
confirm their address receives it.
|
||||
|
||||
## D. Handover (10 min)
|
||||
@@ -78,13 +152,20 @@ This is the friction lens: the customer's hands on the keyboard, the operator's
|
||||
|
||||
1. Hub: customer row green; Offsite → Restic tab shows the first bytes within a report cycle;
|
||||
PBS-DR gauge nominal (first PBS run on its own schedule — note, don't wait).
|
||||
|
||||
> **Verify the DR tier on the box, not from the hub (R-39, interim).** The agent can report the
|
||||
> PBS descriptor `converged` while the credential behind it is stale, and its own verify loop
|
||||
> cannot read the credential to notice. Until R-39 ships, SSH to the host and run
|
||||
> `pvesm status` — the `felhom-pbs` line must be **active**. A `401 Unauthorized` / `inactive`
|
||||
> there means the DR tier is dead despite everything upstream looking green, and it will stay
|
||||
> dead silently until someone needs a restore.
|
||||
2. Floor convergence: controller at current, agent at MinAgent+.
|
||||
3. Route friction notes: UX items → ROADMAP; one-pager corrections → R-11; runbook corrections →
|
||||
this doc.
|
||||
|
||||
## F. Standing rules
|
||||
|
||||
- Never perform steps C1–C6 *for* the customer. The script's value is watching them do it.
|
||||
- Never perform steps C1–C7 *for* the customer. The script's value is watching them do it.
|
||||
- Never handle the recovery code — see it into the envelope, never into your hands or a photo.
|
||||
- One onboarding per day maximum during alpha — the friction notes are the point, and they
|
||||
degrade after the first.
|
||||
|
||||
@@ -0,0 +1,301 @@
|
||||
# VALIDATION — N100 rehearsal: RESET → generic ISO → self-bind → customer zero, 2026-07-18
|
||||
|
||||
> Supervised run of `runbooks/RUNBOOK-rehearsal-v3.md` (S1–S8). Viktor at the box as customer zero;
|
||||
> CC collected the evidence afterwards (Phase A) before the N100 was unplugged for travel. **No
|
||||
> production code changed by this run** — this document, the capability-map flips, and the ROADMAP
|
||||
> items are its outputs. Every finding below is **recorded and ranked, never fixed here.**
|
||||
>
|
||||
> **Headline — read this first.** The complete final-product flow ran **end to end on real metal in
|
||||
> one pass**: blank hardware → zero-touch install → self-registration → **customer self-bind** →
|
||||
> credential delivery → day-0 → managed-floor lift → drive enrolment → escrow ceremony → offsite
|
||||
> snapshots. The chain from **bind to a running current-version controller took 2 min 44 s**, and the
|
||||
> managed floor lifted the controller **0.143.0 → 0.145.0 in 5 seconds, unattended**. Three things
|
||||
> did not go to plan and are the run's real value: **(1)** the offsite repo came up **ORPHANED**,
|
||||
> which S7 defines in advance as a finding; **(2)** the **PBS DR tier is silently non-functional** —
|
||||
> descriptor applied, token rejected `401`, and the agent's own verify loop cannot read the
|
||||
> credential to notice; **(3)** pushing the offsite config **restarted the controller and logged the
|
||||
> customer out mid-flow**. Net: **core objectives green, 11 ranked items, one of them new and
|
||||
> unforecast.**
|
||||
|
||||
---
|
||||
|
||||
## Run context
|
||||
|
||||
| | |
|
||||
|---|---|
|
||||
| Hardware | Intel **N100** (AlderLake-N), 16 GB; the same physical box as `VALIDATION-n100-baremetal-2026-07-16` |
|
||||
| Host | `demo-felhom`, PVE **9.2.2** (kernel 7.0.2-6-pve), guest **9201** (`demo-felhom`) |
|
||||
| Customer | `demo-felhom`, domain demo-felhom.eu — reborn clean-slate on this run |
|
||||
| Versions | controller **0.145.0** · agent **0.90.0** · hub **v0.66.0** · ISO scripts **v1.20.0** · host-install **1.17.0** |
|
||||
| Preparation | **RESET ×2** (host-delete-first): `demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51 — both with all five legs `ok` and escrow acked |
|
||||
| Media | safety ISO first, then the real **universal pairing ISO v1.20.0**, `--loader mkimage`, Secure Boot **OFF** (mkimage is unsigned — the documented slice-B constraint) |
|
||||
| Evidence | `180:~/n100-rehearsal/` (10 files + `MANIFEST.md`) |
|
||||
|
||||
> **⚠ Timezone.** The PVE host journal is **CEST (+0200)**; the guest, the controller and the hub are
|
||||
> **UTC**. **Every timestamp in this document is UTC.** Subtract 2 h from any `journalctl` line
|
||||
> before comparing it with a controller or hub timestamp. This split is live-captured in
|
||||
> `timezones.txt` and is itself worth a ruling (see *Observations*).
|
||||
|
||||
---
|
||||
|
||||
## Timeline (wall-clocks, all UTC)
|
||||
|
||||
Sourced from the hub events stream, the hub DB, the controller log, the bootstrap journal and the
|
||||
agent journal — **not** from expectation. Where the brief and the evidence disagreed, the evidence
|
||||
is recorded and the disagreement is called out under *Discrepancies*.
|
||||
|
||||
| UTC | What happened | Evidence |
|
||||
|---|---|---|
|
||||
| 15:49:57 → 15:50:06 | **RESET #1** — customer `demo-vm-felhom`; legs `claim/db_purge/descriptor/hetzner/pbs` all `ok`, escrow acked | `hub-state.txt` `customer_resets` id=1 |
|
||||
| 16:05:24 | `host_stale` + `node_stale` — "no report for 30m" | events |
|
||||
| 16:08:51 → 16:08:59 | **RESET #2** — customer `demo-felhom`; all five legs `ok`, escrow acked (8 s) | `customer_resets` id=2 |
|
||||
| **16:17:14** | **Blank metal first boot.** `felhom-bootstrap` in **PAIRING** mode; registers as an unclaimed appliance the same second | `bootstrap-journal.txt`; `appliance_registrations.first_seen` |
|
||||
| 16:17:14 → 16:28 | 30 s pairing-poll loop; **every iteration exits 1** → **52** systemd `FAILED`/`Failed to start` lines on the customer-visible console | `bootstrap-journal.txt` |
|
||||
| 16:28:55 | Operator mints + emails the **self-bind link**; 7-day TTL (expires 2026-07-25 16:28:55) | `selfbind_tokens` `created_at`/`emailed_at`/`expires_at` |
|
||||
| **16:29:55** | **Customer self-binds** — token consumed, `attempts=0`, `locked=0`; `appliance_bound` source **`customer_selfbind`** | `selfbind_tokens.consumed_at`; events |
|
||||
| 16:30:21 | `appliance_credential_delivered` (**26 s** after bind) | events; `appliance_registrations.delivered_at` |
|
||||
| 16:30:40 | Hub `hosts` row created — `demo-felhom-8363b5` | `hosts.created_at` |
|
||||
| 16:30:49 | Agent **0.90.0** daemon start; fast-tick armed | `agent-journal.txt` (18:30:49 CEST) |
|
||||
| 16:30:51 | WG keypair generated → **registered with hub** → `10.77.0.2/32` → tunnel applied (**~2 s** end to end) | agent journal |
|
||||
| 16:30:54 | Poke listener up, WG-confined `10.77.0.2:51822` | agent journal |
|
||||
| 16:32:28 | `controller_started` **0.143.0** (the golden's version) | events |
|
||||
| 16:32:29 | **Managed-floor update initiated — `initiated_by: auto-floor`** | `update-state.json` |
|
||||
| 16:32:32 | **Day-0 provision SUCCESS** — done-flag written, unit disabled, bootstrap secrets scrubbed | `bootstrap-journal.txt` |
|
||||
| **16:32:34** | **Floor update complete: 0.143.0 → 0.145.0 in 5 s**; new container created | `update-state.json`; `docker inspect .Created` |
|
||||
| 16:32:39 | `controller_started` **0.145.0** | events |
|
||||
| 16:33:24 | `node_recovered` — "Reports resumed (was stale for 27m)" | events |
|
||||
| 16:41:41 | Drive init starts — `/mnt/hdd_1`, `/dev/sdb`, ext4 | controller log |
|
||||
| 16:41:42 | Agent wipe gate: `storage_wipe` → `customer_confirmable`, **allowed=false**, `pending_confirmation` | agent journal |
|
||||
| 16:41:51 | Wipe gate: **allowed=true**, `customer_confirmed` (9 s of customer thinking time) | agent journal |
|
||||
| 16:42:19 | Device formatted ext4 | agent journal |
|
||||
| 16:42:21 | Mount ensured → bound into guest → storage path registered → FileBrowser mounts synced. **Init 30.3 s** (F6 working) | controller log + agent journal |
|
||||
| **16:50:22** | **RAM shrink 11675 → 8192 MB**, live cgroup, no reboot | controller log + agent `local-api: guest-memory resized` |
|
||||
| 16:45:51 | Agent consumes a one-time **PBS** token secret (`secret_len=36`) and applies the descriptor | agent journal |
|
||||
| 16:45:53 | `pbsdr: converged state=applied` | agent journal |
|
||||
| **16:47:52** | **Hub mints a NEW PBS token secret — `consumed_at` is still NULL** | `host_pbs_secrets` |
|
||||
| 16:47:55 → | Agent PBS verify loop: `cannot read token secret … permission denied` — **repeats every ~15 min, indefinitely** | agent journal |
|
||||
| 16:54:58 | Config-refresh: hub `config_version=11 != applied=10` → **controller self-restarts** | controller log |
|
||||
| 16:55:02 | Controller 0.145.0 restarts (container `StartedAt` 16:54:59Z — a **real restart**, not a reload) | controller log; `docker inspect` |
|
||||
| 16:55:05 | `offsite-apply`: Storage Box target configured, *pending key escrow* | controller log |
|
||||
| 16:56:29 | **Escrow ceremony started** via the wizard | controller log |
|
||||
| 16:56:39 | Recovery code claimed (one-shot, not logged) | controller log |
|
||||
| **16:56:41** | Hub-verified → `EscrowState` auto-confirmed → offsite runs enabled (**12 s** ceremony → confirmed) | controller log |
|
||||
| 16:58:12 | First offsite run starts | controller log |
|
||||
| **16:58:14** | **ORPHAN GUARD FIRES** — `offbox_repo_orphaned`, run skipped, card shown | controller log + events |
|
||||
| 16:59:26 | Operator-confirmed reset: move-aside `/home/felhom-repo` → `.orphaned-20260718` | controller log |
|
||||
| 16:59:32 | Fresh repo initialised; `offbox_repo_reset` | controller log + events |
|
||||
| 16:59:38 → 16:59:50 | Backup run: 0 apps toggled, 0 snapshots, 7 s | controller log |
|
||||
| **17:02:17** | **RAM grow 8192 → 12288 MB**, live cgroup, no reboot | controller log + agent journal |
|
||||
| 17:05:15 | Calibre-Web deployed; deploy-page memory math now reads `total=12288MB` (**the grow rippled**) | events + controller log |
|
||||
| 17:06:40 | Immich deployed | events |
|
||||
| 17:11:16 | BookStack deployed (first start attempt failed `exit 1` 17:11:36; healthy by 17:12:03) | events + controller log |
|
||||
| 17:13:38 → 17:14:22 | **Offsite run: 2 apps backed up, 1 skipped, 2 snapshots, 38 s** | controller log |
|
||||
| 17:13:42 | Snapshot **`243ad08c`** — calibre-web | `restic-and-pool.txt` |
|
||||
| 17:13:46 | Snapshot **`6df12205`** — immich | `restic-and-pool.txt` |
|
||||
|
||||
### Derived wall-clocks (the numbers R-3 needs)
|
||||
|
||||
| Leg | Wall-clock |
|
||||
|---|---|
|
||||
| Blank-metal boot → registered as unclaimed appliance | **< 1 s** (same second) |
|
||||
| Self-bind link minted → customer consumed it | 1 min 0 s |
|
||||
| **Bind → credential delivered** | **26 s** |
|
||||
| Bind → agent enrolled + WG tunnel up | 56 s |
|
||||
| **Bind → controller running the CURRENT version** | **2 min 44 s** |
|
||||
| Credential delivered → day-0 provision SUCCESS | 2 min 11 s |
|
||||
| **Managed-floor lift 0.143.0 → 0.145.0** | **5 s**, unattended |
|
||||
| Drive enrol (confirm → usable, incl. mkfs) | 30.3 s |
|
||||
| **Escrow ceremony → offsite runs enabled** | **12 s** |
|
||||
| Boot → day-0 SUCCESS (incl. ~11.7 min waiting for the operator to send the link) | 15 min 18 s |
|
||||
|
||||
---
|
||||
|
||||
## Per-ledger verdicts
|
||||
|
||||
| Ledger | Verdict | Evidence |
|
||||
|---|---|---|
|
||||
| **S1** — stop apps, shut down, hub DOWN | **PASS** | `host_stale`/`node_stale` 16:05:24 |
|
||||
| **S2** — the one BIOS visit (boot order, SB off, State-After-G3) | **PASS** (human leg) | Viktor at the box; SB-off is required by `--loader mkimage` |
|
||||
| **S3** — the F1 moment: mkimage GRUB boots where stock GRUB threw `relocation 0x0` | **PASS** | The box reached first boot at 16:17:14 on the real board — **F1 is closed on physical hardware** |
|
||||
| **S4** — unattended install → pairing mode → appears under Unclaimed appliances | **PASS** | `bootstrap-journal.txt` "PAIRING mode … registered — appliance token stored (0600)"; `appliance_registrations` id=3 `status=registered` |
|
||||
| **S5** — the bind + the four-minute event composition, **no ~15-min stall between legs** | **PASS — and this is the immediacy proof** | Every post-bind leg is seconds apart: bind 16:29:55 → credential 16:30:21 → agent 16:30:49 → WG 16:30:51 → controller 16:32:28 → floor-lifted 16:32:39. **No leg stalled.** F5 auto-sizing landed the guest at **11675 MB** |
|
||||
| **S6** — customer zero: claim → password → escrow ceremony → storage → **RAM grow *and* shrink** → deploy → enable offsite | **PASS on every leg, with two findings attached** | Ceremony 16:56:29→16:56:41; wipe gate pending→confirmed 16:41:42→16:41:51; **shrink 11675→8192 at 16:50:22 and grow 8192→12288 at 17:02:17, both live with no reboot**; 3 apps deployed. Findings: the config push logged the customer out (F3); offsite provisioning is **Save-triggered**, answering S6's open question |
|
||||
| **S7** — first snapshot into a brand-new repo; **"if an orphan card appears, that's a finding"** | **FINDING — the orphan card appeared** | `offbox_repo_orphaned` 16:58:14. Recovered by the operator-confirmed move-aside 16:59:26→16:59:32; the run then produced **2 snapshots / 48.717 MiB** at 17:13:42–46. The *guard* worked exactly as designed — the finding is that it had to |
|
||||
| **S7 (operator gauges)** | **PARTIAL** | Restic tab: Viktor's screenshots. **PBS DR tab was not captured, and the reason now matters — see F2** |
|
||||
| **S8** — wrap: evidence, VALIDATION doc, map flips, ROADMAP, R-3 | **DONE** — this document | `180:~/n100-rehearsal/MANIFEST.md` |
|
||||
| **Ledger 8** — immediacy legs, first live firing on a real onboarding | **PASS** | See S5. The "real-onboarding proof pending" note on the immediacy row can be cleared |
|
||||
| **Ledger 9** — R-24 guest RAM resize, the *apply* (not just the refusals) | **PASS, both directions** | Shrink **and** grow, live cgroup, no reboot, and the new total rippled into the deploy page's memory math at 17:05:15 |
|
||||
|
||||
---
|
||||
|
||||
## Findings (recorded and ranked — not fixed)
|
||||
|
||||
Ranked as proposed for the ROADMAP; IDs assigned in `backlog/ROADMAP.md`.
|
||||
|
||||
### F1 — Liveness presence lags reality by ~30 min (→ R-30, P2-HIGH)
|
||||
The box was powered off at the start of the run, yet the hub carried it as healthy until the
|
||||
staleness threshold expired at **16:05:24** ("no report for 30m"), and cleared only at **16:33:24**
|
||||
("was stale for 27m"). The host-delete guard compounds this: RESET refuses while any host row
|
||||
exists, so a stale-but-"online" host stalls a forced teardown. **Direction:** derive presence from
|
||||
Dir-2 long-poll connectedness (~90 s grace), decoupled from notification hysteresis; agent/ep0
|
||||
analog later.
|
||||
|
||||
### F2 — PBS DR tier is provisioned-but-dead, and cannot self-report it (→ R-39, P2-HIGH, **NEW — not in the brief**)
|
||||
This was not forecast by the brief and is the run's most serious finding. Three facts:
|
||||
1. The descriptor **was** auto-provisioned: the agent consumed a one-time token secret at 16:45:51
|
||||
and reached `converged state=applied` at 16:45:53. `storage.cfg` holds a complete `pbs:
|
||||
felhom-pbs` stanza (namespace `demo-felhom`, token `felhom@pbs!demo-felhom`, encryption key,
|
||||
fingerprint).
|
||||
2. **The storage is non-functional.** `pvesm status` → `felhom-pbs: error fetching datastores - 401
|
||||
Unauthorized`, `inactive`. A direct probe with the stored credential returns **401 on every
|
||||
endpoint including `/version`** — so this is authentication failure, not ACL scope. The transport
|
||||
is healthy (WG handshake 9 s old, 27.9 ms RTT, `/version` reachable), so the token itself is
|
||||
rejected server-side.
|
||||
3. **The hub minted a second secret at 16:47:52 — two minutes after the agent applied the first —
|
||||
and `consumed_at` is still NULL** ~55 min later. The agent will not pick it up: it has already
|
||||
converged to `state=applied`, and its 15-minute verify loop cannot even read the credential to
|
||||
detect the failure (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — the agent
|
||||
runs non-root, writes the file through a root sudo wrapper, then tries to read it directly).
|
||||
|
||||
Net: a mint/consume race leaves the box on a stale secret, a converged state machine prevents
|
||||
re-application, and a permission bug blinds the only loop that could notice. **The DR tier would
|
||||
have failed silently on the first real restore.** This also explains why the S7 PBS DR gauge was
|
||||
worth capturing — it would have shown the fault.
|
||||
|
||||
### F3 — Config apply restarts the controller and drops customer sessions (→ R-35, P2)
|
||||
The offsite config push bumped `config_version` 10→11 at 16:54:58; the controller self-restarted
|
||||
(container `StartedAt` 16:54:59Z) and came back at 16:55:02. In-memory sessions died with it, so
|
||||
customer zero was force-logged-out mid-flow. **Direction:** hot-apply the offbox target, or persist
|
||||
sessions across restart.
|
||||
|
||||
### F4 — Offsite provisioning is synchronous, with no status affordance (→ R-31, P2-HIGH)
|
||||
Save runs the Hetzner sync in-request; the request can hit the nginx 504 while succeeding
|
||||
server-side, so the operator cannot distinguish failed from slow, and a retry races the first
|
||||
attempt. **Direction:** reuse the awaiting-card/poll idiom. Interim mitigation belongs in R-3:
|
||||
**click once, wait, verify.**
|
||||
|
||||
### F5 — RESET destroys custody but not the ciphertext; the pool accumulates unattributed bytes (→ R-32, P2-HIGH)
|
||||
The RESET ledger records `"hetzner":"ok"` — the sub-account **was** destroyed. But a Hetzner
|
||||
sub-account is an access-control object, not a data object: deleting it leaves its directory intact.
|
||||
Re-enabling offsite recreated a sub-account over the **same** `/home`, which still held the previous
|
||||
lifecycle's repo — under a key destroyed by that same RESET. Hence S7's unexpected orphan card.
|
||||
Measured on the pool box tonight:
|
||||
|
||||
| Path | Size | Status |
|
||||
|---|---|---|
|
||||
| `/home/felhom-repo` | **49 M** | live, attributed (2 snapshots, 48.717 MiB) |
|
||||
| `/home/felhom-repo.orphaned-20260717` | **1.4 G** | unattributed, key destroyed — from the 07-17 incident |
|
||||
| `/home/felhom-repo.orphaned-20260718` | **3.0 M** | unattributed, key destroyed — from tonight |
|
||||
|
||||
**Ruling from the run:** because RESET destroys custody, leftover ciphertext is unrecoverable **by
|
||||
design**, so RESET gains a main-account purge of the customer base dir (the operator ack already
|
||||
covers it). The move-aside guard **stays** for reinstall-without-RESET, where custody survives and
|
||||
the card's promise is true. The operator Restic tab should show per-customer directory bytes against
|
||||
attributed snapshot bytes so dead data cannot hide.
|
||||
|
||||
### F6 — Bootstrap pairing-poll spams the customer-visible console (→ R-33, P2-HIGH)
|
||||
Each 30 s poll is a separate `felhom-bootstrap.service` run that exits 1, so systemd prints
|
||||
`Failed to start …` on the console the customer is watching. **52 FAILED lines in ~11 minutes**,
|
||||
while nothing was actually wrong — the box was correctly waiting to be bound. **Direction:** poll
|
||||
*inside* the script (scripts v1.21) and reword the first-boot noise.
|
||||
|
||||
### F7 — Design-v2 gates are unenforced; one has been red for 16 releases
|
||||
**Already ranked as R-29** (committed `2eccda4`, 2026-07-18) — the brief asked for a new ID, but the
|
||||
finding already has one, including the `docker_run_volume_path_gate` /
|
||||
`internal/appexport/estimate.go` attribution. **No new ID minted.**
|
||||
|
||||
### F8 — Backup data lifecycle is unmanaged (→ R-34, P2)
|
||||
Apps with snapshots but no active backup (disabled or uninstalled) are invisible and undeletable.
|
||||
**Direction:** an "inactive backups" section on Távoli mentés — name / size / last snapshot /
|
||||
restorable, with an explicit double-confirmed per-app delete via `restic forget --tag` + nightly
|
||||
prune. **RULING: the offsite toggle NEVER offers deletion — policy and destruction stay decoupled.**
|
||||
|
||||
### F9 — Post-RESET re-enroll leaves offsite "enabled but unprovisioned", silently (→ R-36, P2)
|
||||
The hub knows the state and says nothing. Provisioning is **Save-triggered** (`configs.go`
|
||||
`applyOffsite`) — confirmed live tonight, which answers S6's open question — and the re-enroll
|
||||
auto-re-issue correctly skips unprovisioned targets (`handler.go`). **Direction:** flash it on the
|
||||
customer page. Interim: an R-3 step. **Related:** auto-mint the self-bind link on customer
|
||||
create/RESET so the console banner's "e-mailben kapott link" is always already true.
|
||||
|
||||
### F10 — Console-passphrase bind (→ R-27c, P2)
|
||||
Viktor's direction, with the session's security constraints: passphrase issued at customer creation;
|
||||
spray-hardened global-lookup endpoint (per-appliance + per-IP caps, constant-time, single generic
|
||||
failure, alerting); accent-free wordlist for console keymaps; the web capability-link path
|
||||
**RETAINED**; claim-by-email **RETAINED** as the delivery-channel proof. The self-bind email gains
|
||||
the public universal-ISO download link + two-line instructions (the DIY case). **Secret-bearing
|
||||
per-customer ISOs are ruled OUT.**
|
||||
|
||||
### F11 — Post-RESET health card shows stale pre-RESET warnings (→ R-37, P3)
|
||||
Should read „RESET óta nincs adat".
|
||||
|
||||
### F12 — Installer GRUB menu offers interactive installers (→ R-38, P3)
|
||||
Single default „Felhom telepítés" entry; interactive installers **REMOVED** (safety: wrong-disk
|
||||
manual installs); felhom background. Squashfs/theme rebranding explicitly deferred.
|
||||
|
||||
---
|
||||
|
||||
## Observations (not findings — no ID minted)
|
||||
|
||||
- **Host/guest timezone split.** The PVE host runs CEST while the guest, controller and hub run UTC.
|
||||
Nothing is broken, but every cross-tier correlation needs a manual 2 h shift, which is exactly the
|
||||
kind of friction that produces wrong incident timelines under pressure. Flagged for a ruling
|
||||
rather than ranked unilaterally.
|
||||
- **The escrow awaiting-card window was 12 seconds.** Ceremony → hub-confirmed took 16:56:29 →
|
||||
16:56:41, so the „megerősítésre vár, legfeljebb 15 perc" card rendered only briefly. The card is
|
||||
correct and its ≤15-minute promise is honest, but capturing *both* states on screen is a matter of
|
||||
luck at this speed.
|
||||
- **BookStack skipped its first offsite run** — `no recovery unit found on any connected drive`
|
||||
(17:13:42). This is the expected pre-first-nightly-dump state; the nightly job is scheduled
|
||||
2026-07-19 04:15 CEST. **Watch item, not a finding** — it becomes one only if it still skips
|
||||
afterwards.
|
||||
- **BookStack's first start attempt failed** (`exit 1`, 17:11:36) and succeeded on retry by
|
||||
17:12:03. Self-corrected; noted for pattern-watching only.
|
||||
- **`felhom-samba` was not running** on the reborn box — sharing was not exercised this run
|
||||
(consistent with *What was NOT exercised*).
|
||||
|
||||
---
|
||||
|
||||
## What was NOT exercised (explicit)
|
||||
|
||||
- **C6 — a customer performs a restore unassisted.** The single open script step in R-3, and still
|
||||
**MISSING as evidence**. Remains remote-doable on the reborn box from vacation (the dashboard is
|
||||
remote).
|
||||
- **freemail.hu deliverability.** The gmail half is evidenced (claim email → Inbox under
|
||||
`p=quarantine`); freemail.hu is Viktor's open half.
|
||||
- **Sharing smoke on the reborn box** — R-7/R-7b were proven on the pre-RESET box; `felhom-samba` was
|
||||
not deployed here.
|
||||
- **R-19 WAN-outage drill.**
|
||||
- **The PBS DR operator gauge** — not captured, and F2 means it would have shown a fault.
|
||||
- **BookStack's offsite unit** — watch item, see *Observations*.
|
||||
- **SMB positive auth with the real household password** — carried over from R-7b, unchanged.
|
||||
|
||||
---
|
||||
|
||||
## Discrepancies vs the Phase-A brief (live evidence wins)
|
||||
|
||||
1. **"expect ~11675 MB ballpark" — the final `pct config` reads 12288 MB.** Both are right at
|
||||
different times: F5 auto-sized the guest to **11675 MB**, then customer zero **shrank it to 8192**
|
||||
(16:50:22) and **grew it to 12288** (17:02:17). The brief described the F5 value; the box ended
|
||||
above it because the grow was the last resize. Not a fault — and it is what completes ledger 9.
|
||||
2. **"2.6 GB pool vs ~0 attributed" — measured 1.4 G + 3.0 M unattributed against 49 M attributed,
|
||||
across TWO orphaned directories, not one.** The F5 ruling is unaffected; the numbers are.
|
||||
3. **"felhom-pbs entry present" — present but `401 Unauthorized` / `inactive`.** The brief treated
|
||||
the PBS descriptor as done; it is provisioned but dead. See F2.
|
||||
4. **"RESET ×2 … second + third live firings" — the ledger holds exactly two rows, both tonight**
|
||||
(`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51), on **two different customers**. Any earlier
|
||||
firing is not in `customer_resets`. Described here as tonight's two firings rather than as
|
||||
ordinals.
|
||||
5. **"keep the name R-29 if the number is free" — it is not free.** R-29 was committed earlier the
|
||||
same day (`2eccda4`) for the design-v2 gate rot, which is also exactly the brief's item 5. The
|
||||
liveness item therefore takes **R-30**, and no new ID was minted for the red-gate item.
|
||||
6. **The brief's tip `a03e142` is two commits behind** `bbaa844`; Viktor's R-3 draft landed as
|
||||
`runbooks/RUNBOOK-onboarding-draft-v3.md`, not `DRAFT-R3-onboarding-runbook.md`.
|
||||
7. **F2 has no counterpart in the brief at all** — it was found by probing the box rather than by
|
||||
following the finding list.
|
||||
|
||||
---
|
||||
|
||||
## Evidence bundle
|
||||
|
||||
`180:~/n100-rehearsal/` — see `MANIFEST.md` there for the per-file map. Secrets were read only to
|
||||
run probes and are recorded as lengths and metadata, never as values.
|
||||
Reference in new issue
Block a user