felhom.eu/documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md

# VALIDATION — N100 rehearsal: RESET → generic ISO → self-bind → customer zero, 2026-07-18

> Supervised run of `runbooks/RUNBOOK-rehearsal-v3.md` (S1–S8). Viktor at the box as customer zero;
> CC collected the evidence afterwards (Phase A) before the N100 was unplugged for travel. **No
> production code changed by this run** — this document, the capability-map flips, and the ROADMAP
> items are its outputs. Every finding below is **recorded and ranked, never fixed here.**
>
> **Headline — read this first.** The complete final-product flow ran **end to end on real metal in
> one pass**: blank hardware → zero-touch install → self-registration → **customer self-bind** →
> credential delivery → day-0 → managed-floor lift → drive enrolment → escrow ceremony → offsite
> snapshots. The chain from **bind to a running current-version controller took 2 min 44 s**, and the
> managed floor lifted the controller **0.143.0 → 0.145.0 in 5 seconds, unattended**. Three things
> did not go to plan and are the run's real value: **(1)** the offsite repo came up **ORPHANED**,
> which S7 defines in advance as a finding; **(2)** the **PBS DR tier is silently non-functional** —
> descriptor applied, token rejected `401`, and the agent's own verify loop cannot read the
> credential to notice; **(3)** pushing the offsite config **restarted the controller and logged the
> customer out mid-flow**. Net: **core objectives green, 11 ranked items, one of them new and
> unforecast.**

---

## Run context

| | |
|---|---|
| Hardware | Intel **N100** (AlderLake-N), 16 GB; the same physical box as `VALIDATION-n100-baremetal-2026-07-16` |
| Host | `demo-felhom`, PVE **9.2.2** (kernel 7.0.2-6-pve), guest **9201** (`demo-felhom`) |
| Customer | `demo-felhom`, domain demo-felhom.eu — reborn clean-slate on this run |
| Versions | controller **0.145.0** · agent **0.90.0** · hub **v0.66.0** · ISO scripts **v1.20.0** · host-install **1.17.0** |
| Preparation | **RESET ×2** (host-delete-first): `demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51 — both with all five legs `ok` and escrow acked |
| Media | safety ISO first, then the real **universal pairing ISO v1.20.0**, `--loader mkimage`, Secure Boot **OFF** (mkimage is unsigned — the documented slice-B constraint) |
| Evidence | `180:~/n100-rehearsal/` (10 files + `MANIFEST.md`) |

> **⚠ Timezone.** The PVE host journal is **CEST (+0200)**; the guest, the controller and the hub are
> **UTC**. **Every timestamp in this document is UTC.** Subtract 2 h from any `journalctl` line
> before comparing it with a controller or hub timestamp. This split is live-captured in
> `timezones.txt` and is itself worth a ruling (see *Observations*).

---

## Timeline (wall-clocks, all UTC)

Sourced from the hub events stream, the hub DB, the controller log, the bootstrap journal and the
agent journal — **not** from expectation. Where the brief and the evidence disagreed, the evidence
is recorded and the disagreement is called out under *Discrepancies*.

| UTC | What happened | Evidence |
|---|---|---|
| 15:49:57 → 15:50:06 | **RESET #1** — customer `demo-vm-felhom`; legs `claim/db_purge/descriptor/hetzner/pbs` all `ok`, escrow acked | `hub-state.txt` `customer_resets` id=1 |
| 16:05:24 | `host_stale` + `node_stale` — "no report for 30m" | events |
| 16:08:51 → 16:08:59 | **RESET #2** — customer `demo-felhom`; all five legs `ok`, escrow acked (8 s) | `customer_resets` id=2 |
| **16:17:14** | **Blank metal first boot.** `felhom-bootstrap` in **PAIRING** mode; registers as an unclaimed appliance the same second | `bootstrap-journal.txt`; `appliance_registrations.first_seen` |
| 16:17:14 → 16:28 | 30 s pairing-poll loop; **every iteration exits 1** → **52** systemd `FAILED`/`Failed to start` lines on the customer-visible console | `bootstrap-journal.txt` |
| 16:28:55 | Operator mints + emails the **self-bind link**; 7-day TTL (expires 2026-07-25 16:28:55) | `selfbind_tokens` `created_at`/`emailed_at`/`expires_at` |
| **16:29:55** | **Customer self-binds** — token consumed, `attempts=0`, `locked=0`; `appliance_bound` source **`customer_selfbind`** | `selfbind_tokens.consumed_at`; events |
| 16:30:21 | `appliance_credential_delivered` (**26 s** after bind) | events; `appliance_registrations.delivered_at` |
| 16:30:40 | Hub `hosts` row created — `demo-felhom-8363b5` | `hosts.created_at` |
| 16:30:49 | Agent **0.90.0** daemon start; fast-tick armed | `agent-journal.txt` (18:30:49 CEST) |
| 16:30:51 | WG keypair generated → **registered with hub** → `10.77.0.2/32` → tunnel applied (**~2 s** end to end) | agent journal |
| 16:30:54 | Poke listener up, WG-confined `10.77.0.2:51822` | agent journal |
| 16:32:28 | `controller_started` **0.143.0** (the golden's version) | events |
| 16:32:29 | **Managed-floor update initiated — `initiated_by: auto-floor`** | `update-state.json` |
| 16:32:32 | **Day-0 provision SUCCESS** — done-flag written, unit disabled, bootstrap secrets scrubbed | `bootstrap-journal.txt` |
| **16:32:34** | **Floor update complete: 0.143.0 → 0.145.0 in 5 s**; new container created | `update-state.json`; `docker inspect .Created` |
| 16:32:39 | `controller_started` **0.145.0** | events |
| 16:33:24 | `node_recovered` — "Reports resumed (was stale for 27m)" | events |
| 16:41:41 | Drive init starts — `/mnt/hdd_1`, `/dev/sdb`, ext4 | controller log |
| 16:41:42 | Agent wipe gate: `storage_wipe` → `customer_confirmable`, **allowed=false**, `pending_confirmation` | agent journal |
| 16:41:51 | Wipe gate: **allowed=true**, `customer_confirmed` (9 s of customer thinking time) | agent journal |
| 16:42:19 | Device formatted ext4 | agent journal |
| 16:42:21 | Mount ensured → bound into guest → storage path registered → FileBrowser mounts synced. **Init 30.3 s** (F6 working) | controller log + agent journal |
| **16:50:22** | **RAM shrink 11675 → 8192 MB**, live cgroup, no reboot | controller log + agent `local-api: guest-memory resized` |
| 16:45:51 | Agent consumes a one-time **PBS** token secret (`secret_len=36`) and applies the descriptor | agent journal |
| 16:45:53 | `pbsdr: converged state=applied` | agent journal |
| **16:47:52** | **Hub mints a NEW PBS token secret — `consumed_at` is still NULL** | `host_pbs_secrets` |
| 16:47:55 → | Agent PBS verify loop: `cannot read token secret … permission denied` — **repeats every ~15 min, indefinitely** | agent journal |
| 16:54:58 | Config-refresh: hub `config_version=11 != applied=10` → **controller self-restarts** | controller log |
| 16:55:02 | Controller 0.145.0 restarts (container `StartedAt` 16:54:59Z — a **real restart**, not a reload) | controller log; `docker inspect` |
| 16:55:05 | `offsite-apply`: Storage Box target configured, *pending key escrow* | controller log |
| 16:56:29 | **Escrow ceremony started** via the wizard | controller log |
| 16:56:39 | Recovery code claimed (one-shot, not logged) | controller log |
| **16:56:41** | Hub-verified → `EscrowState` auto-confirmed → offsite runs enabled (**12 s** ceremony → confirmed) | controller log |
| 16:58:12 | First offsite run starts | controller log |
| **16:58:14** | **ORPHAN GUARD FIRES** — `offbox_repo_orphaned`, run skipped, card shown | controller log + events |
| 16:59:26 | Operator-confirmed reset: move-aside `/home/felhom-repo` → `.orphaned-20260718` | controller log |
| 16:59:32 | Fresh repo initialised; `offbox_repo_reset` | controller log + events |
| 16:59:38 → 16:59:50 | Backup run: 0 apps toggled, 0 snapshots, 7 s | controller log |
| **17:02:17** | **RAM grow 8192 → 12288 MB**, live cgroup, no reboot | controller log + agent journal |
| 17:05:15 | Calibre-Web deployed; deploy-page memory math now reads `total=12288MB` (**the grow rippled**) | events + controller log |
| 17:06:40 | Immich deployed | events |
| 17:11:16 | BookStack deployed (first start attempt failed `exit 1` 17:11:36; healthy by 17:12:03) | events + controller log |
| 17:13:38 → 17:14:22 | **Offsite run: 2 apps backed up, 1 skipped, 2 snapshots, 38 s** | controller log |
| 17:13:42 | Snapshot **`243ad08c`** — calibre-web | `restic-and-pool.txt` |
| 17:13:46 | Snapshot **`6df12205`** — immich | `restic-and-pool.txt` |

### Derived wall-clocks (the numbers R-3 needs)

| Leg | Wall-clock |
|---|---|
| Blank-metal boot → registered as unclaimed appliance | **< 1 s** (same second) |
| Self-bind link minted → customer consumed it | 1 min 0 s |
| **Bind → credential delivered** | **26 s** |
| Bind → agent enrolled + WG tunnel up | 56 s |
| **Bind → controller running the CURRENT version** | **2 min 44 s** |
| Credential delivered → day-0 provision SUCCESS | 2 min 11 s |
| **Managed-floor lift 0.143.0 → 0.145.0** | **5 s**, unattended |
| Drive enrol (confirm → usable, incl. mkfs) | 30.3 s |
| **Escrow ceremony → offsite runs enabled** | **12 s** |
| Boot → day-0 SUCCESS (incl. ~11.7 min waiting for the operator to send the link) | 15 min 18 s |

---

## Per-ledger verdicts

| Ledger | Verdict | Evidence |
|---|---|---|
| **S1** — stop apps, shut down, hub DOWN | **PASS** | `host_stale`/`node_stale` 16:05:24 |
| **S2** — the one BIOS visit (boot order, SB off, State-After-G3) | **PASS** (human leg) | Viktor at the box; SB-off is required by `--loader mkimage` |
| **S3** — the F1 moment: mkimage GRUB boots where stock GRUB threw `relocation 0x0` | **PASS** | The box reached first boot at 16:17:14 on the real board — **F1 is closed on physical hardware** |
| **S4** — unattended install → pairing mode → appears under Unclaimed appliances | **PASS** | `bootstrap-journal.txt` "PAIRING mode … registered — appliance token stored (0600)"; `appliance_registrations` id=3 `status=registered` |
| **S5** — the bind + the four-minute event composition, **no ~15-min stall between legs** | **PASS — and this is the immediacy proof** | Every post-bind leg is seconds apart: bind 16:29:55 → credential 16:30:21 → agent 16:30:49 → WG 16:30:51 → controller 16:32:28 → floor-lifted 16:32:39. **No leg stalled.** F5 auto-sizing landed the guest at **11675 MB** |
| **S6** — customer zero: claim → password → escrow ceremony → storage → **RAM grow *and* shrink** → deploy → enable offsite | **PASS on every leg, with two findings attached** | Ceremony 16:56:29→16:56:41; wipe gate pending→confirmed 16:41:42→16:41:51; **shrink 11675→8192 at 16:50:22 and grow 8192→12288 at 17:02:17, both live with no reboot**; 3 apps deployed. Findings: the config push logged the customer out (F3); offsite provisioning is **Save-triggered**, answering S6's open question |
| **S7** — first snapshot into a brand-new repo; **"if an orphan card appears, that's a finding"** | **FINDING — the orphan card appeared** | `offbox_repo_orphaned` 16:58:14. Recovered by the operator-confirmed move-aside 16:59:26→16:59:32; the run then produced **2 snapshots / 48.717 MiB** at 17:13:42–46. The *guard* worked exactly as designed — the finding is that it had to |
| **S7 (operator gauges)** | **PARTIAL** | Restic tab: Viktor's screenshots. **PBS DR tab was not captured, and the reason now matters — see F2** |
| **S8** — wrap: evidence, VALIDATION doc, map flips, ROADMAP, R-3 | **DONE** — this document | `180:~/n100-rehearsal/MANIFEST.md` |
| **Ledger 8** — immediacy legs, first live firing on a real onboarding | **PASS** | See S5. The "real-onboarding proof pending" note on the immediacy row can be cleared |
| **Ledger 9** — R-24 guest RAM resize, the *apply* (not just the refusals) | **PASS, both directions** | Shrink **and** grow, live cgroup, no reboot, and the new total rippled into the deploy page's memory math at 17:05:15 |

---

## Findings (recorded and ranked — not fixed)

Ranked as proposed for the ROADMAP; IDs assigned in `backlog/ROADMAP.md`.

### F1 — Liveness presence lags reality by ~30 min (→ R-30, P2-HIGH)
The box was powered off at the start of the run, yet the hub carried it as healthy until the
staleness threshold expired at **16:05:24** ("no report for 30m"), and cleared only at **16:33:24**
("was stale for 27m"). The host-delete guard compounds this: RESET refuses while any host row
exists, so a stale-but-"online" host stalls a forced teardown. **Direction:** derive presence from
Dir-2 long-poll connectedness (~90 s grace), decoupled from notification hysteresis; agent/ep0
analog later.

### F2 — PBS DR tier is provisioned-but-dead, and cannot self-report it (→ R-39, P2-HIGH, **NEW — not in the brief**)
This was not forecast by the brief and is the run's most serious finding. Three facts:
1. The descriptor **was** auto-provisioned: the agent consumed a one-time token secret at 16:45:51
   and reached `converged state=applied` at 16:45:53. `storage.cfg` holds a complete `pbs:
   felhom-pbs` stanza (namespace `demo-felhom`, token `felhom@pbs!demo-felhom`, encryption key,
   fingerprint).
2. **The storage is non-functional.** `pvesm status` → `felhom-pbs: error fetching datastores - 401
   Unauthorized`, `inactive`. A direct probe with the stored credential returns **401 on every
   endpoint including `/version`** — so this is authentication failure, not ACL scope. The transport
   is healthy (WG handshake 9 s old, 27.9 ms RTT, `/version` reachable), so the token itself is
   rejected server-side.
3. **The hub minted a second secret at 16:47:52 — two minutes after the agent applied the first —
   and `consumed_at` is still NULL** ~55 min later. The agent will not pick it up: it has already
   converged to `state=applied`, and its 15-minute verify loop cannot even read the credential to
   detect the failure (`open /etc/pve/priv/storage/felhom-pbs.pw: permission denied` — the agent
   runs non-root, writes the file through a root sudo wrapper, then tries to read it directly).

Net: a mint/consume race leaves the box on a stale secret, a converged state machine prevents
re-application, and a permission bug blinds the only loop that could notice. **The DR tier would
have failed silently on the first real restore.** This also explains why the S7 PBS DR gauge was
worth capturing — it would have shown the fault.

### F3 — Config apply restarts the controller and drops customer sessions (→ R-35, P2)
The offsite config push bumped `config_version` 10→11 at 16:54:58; the controller self-restarted
(container `StartedAt` 16:54:59Z) and came back at 16:55:02. In-memory sessions died with it, so
customer zero was force-logged-out mid-flow. **Direction:** hot-apply the offbox target, or persist
sessions across restart.

### F4 — Offsite provisioning is synchronous, with no status affordance (→ R-31, P2-HIGH)
Save runs the Hetzner sync in-request; the request can hit the nginx 504 while succeeding
server-side, so the operator cannot distinguish failed from slow, and a retry races the first
attempt. **Direction:** reuse the awaiting-card/poll idiom. Interim mitigation belongs in R-3:
**click once, wait, verify.**

### F5 — RESET destroys custody but not the ciphertext; the pool accumulates unattributed bytes (→ R-32, P2-HIGH)
The RESET ledger records `"hetzner":"ok"` — the sub-account **was** destroyed. But a Hetzner
sub-account is an access-control object, not a data object: deleting it leaves its directory intact.
Re-enabling offsite recreated a sub-account over the **same** `/home`, which still held the previous
lifecycle's repo — under a key destroyed by that same RESET. Hence S7's unexpected orphan card.
Measured on the pool box tonight:

| Path | Size | Status |
|---|---|---|
| `/home/felhom-repo` | **49 M** | live, attributed (2 snapshots, 48.717 MiB) |
| `/home/felhom-repo.orphaned-20260717` | **1.4 G** | unattributed, key destroyed — from the 07-17 incident |
| `/home/felhom-repo.orphaned-20260718` | **3.0 M** | unattributed, key destroyed — from tonight |

**Ruling from the run:** because RESET destroys custody, leftover ciphertext is unrecoverable **by
design**, so RESET gains a main-account purge of the customer base dir (the operator ack already
covers it). The move-aside guard **stays** for reinstall-without-RESET, where custody survives and
the card's promise is true. The operator Restic tab should show per-customer directory bytes against
attributed snapshot bytes so dead data cannot hide.

### F6 — Bootstrap pairing-poll spams the customer-visible console (→ R-33, P2-HIGH)
Each 30 s poll is a separate `felhom-bootstrap.service` run that exits 1, so systemd prints
`Failed to start …` on the console the customer is watching. **52 FAILED lines in ~11 minutes**,
while nothing was actually wrong — the box was correctly waiting to be bound. **Direction:** poll
*inside* the script (scripts v1.21) and reword the first-boot noise.

### F7 — Design-v2 gates are unenforced; one has been red for 16 releases
**Already ranked as R-29** (committed `2eccda4`, 2026-07-18) — the brief asked for a new ID, but the
finding already has one, including the `docker_run_volume_path_gate` /
`internal/appexport/estimate.go` attribution. **No new ID minted.**

### F8 — Backup data lifecycle is unmanaged (→ R-34, P2)
Apps with snapshots but no active backup (disabled or uninstalled) are invisible and undeletable.
**Direction:** an "inactive backups" section on Távoli mentés — name / size / last snapshot /
restorable, with an explicit double-confirmed per-app delete via `restic forget --tag` + nightly
prune. **RULING: the offsite toggle NEVER offers deletion — policy and destruction stay decoupled.**

### F9 — Post-RESET re-enroll leaves offsite "enabled but unprovisioned", silently (→ R-36, P2)
The hub knows the state and says nothing. Provisioning is **Save-triggered** (`configs.go`
`applyOffsite`) — confirmed live tonight, which answers S6's open question — and the re-enroll
auto-re-issue correctly skips unprovisioned targets (`handler.go`). **Direction:** flash it on the
customer page. Interim: an R-3 step. **Related:** auto-mint the self-bind link on customer
create/RESET so the console banner's "e-mailben kapott link" is always already true.

### F10 — Console-passphrase bind (→ R-27c, P2)
Viktor's direction, with the session's security constraints: passphrase issued at customer creation;
spray-hardened global-lookup endpoint (per-appliance + per-IP caps, constant-time, single generic
failure, alerting); accent-free wordlist for console keymaps; the web capability-link path
**RETAINED**; claim-by-email **RETAINED** as the delivery-channel proof. The self-bind email gains
the public universal-ISO download link + two-line instructions (the DIY case). **Secret-bearing
per-customer ISOs are ruled OUT.**

### F11 — Post-RESET health card shows stale pre-RESET warnings (→ R-37, P3)
Should read „RESET óta nincs adat".

### F12 — Installer GRUB menu offers interactive installers (→ R-38, P3)
Single default „Felhom telepítés" entry; interactive installers **REMOVED** (safety: wrong-disk
manual installs); felhom background. Squashfs/theme rebranding explicitly deferred.

---

## Observations (not findings — no ID minted)

- **Host/guest timezone split.** The PVE host runs CEST while the guest, controller and hub run UTC.
  Nothing is broken, but every cross-tier correlation needs a manual 2 h shift, which is exactly the
  kind of friction that produces wrong incident timelines under pressure. Flagged for a ruling
  rather than ranked unilaterally.
- **The escrow awaiting-card window was 12 seconds.** Ceremony → hub-confirmed took 16:56:29 →
  16:56:41, so the „megerősítésre vár, legfeljebb 15 perc" card rendered only briefly. The card is
  correct and its ≤15-minute promise is honest, but capturing *both* states on screen is a matter of
  luck at this speed.
- **BookStack skipped its first offsite run** — `no recovery unit found on any connected drive`
  (17:13:42). This is the expected pre-first-nightly-dump state; the nightly job is scheduled
  2026-07-19 04:15 CEST. **Watch item, not a finding** — it becomes one only if it still skips
  afterwards.
- **BookStack's first start attempt failed** (`exit 1`, 17:11:36) and succeeded on retry by
  17:12:03. Self-corrected; noted for pattern-watching only.
- **`felhom-samba` was not running** on the reborn box — sharing was not exercised this run
  (consistent with *What was NOT exercised*).

---

## What was NOT exercised (explicit)

- **C6 — a customer performs a restore unassisted.** The single open script step in R-3, and still
  **MISSING as evidence**. Remains remote-doable on the reborn box from vacation (the dashboard is
  remote).
- **freemail.hu deliverability.** The gmail half is evidenced (claim email → Inbox under
  `p=quarantine`); freemail.hu is Viktor's open half.
- **Sharing smoke on the reborn box** — R-7/R-7b were proven on the pre-RESET box; `felhom-samba` was
  not deployed here.
- **R-19 WAN-outage drill.**
- **The PBS DR operator gauge** — not captured, and F2 means it would have shown a fault.
- **BookStack's offsite unit** — watch item, see *Observations*.
- **SMB positive auth with the real household password** — carried over from R-7b, unchanged.

---

## Discrepancies vs the Phase-A brief (live evidence wins)

1. **"expect ~11675 MB ballpark" — the final `pct config` reads 12288 MB.** Both are right at
   different times: F5 auto-sized the guest to **11675 MB**, then customer zero **shrank it to 8192**
   (16:50:22) and **grew it to 12288** (17:02:17). The brief described the F5 value; the box ended
   above it because the grow was the last resize. Not a fault — and it is what completes ledger 9.
2. **"2.6 GB pool vs ~0 attributed" — measured 1.4 G + 3.0 M unattributed against 49 M attributed,
   across TWO orphaned directories, not one.** The F5 ruling is unaffected; the numbers are.
3. **"felhom-pbs entry present" — present but `401 Unauthorized` / `inactive`.** The brief treated
   the PBS descriptor as done; it is provisioned but dead. See F2.
4. **"RESET ×2 … second + third live firings" — the ledger holds exactly two rows, both tonight**
   (`demo-vm-felhom` 15:49:57, `demo-felhom` 16:08:51), on **two different customers**. Any earlier
   firing is not in `customer_resets`. Described here as tonight's two firings rather than as
   ordinals.
5. **"keep the name R-29 if the number is free" — it is not free.** R-29 was committed earlier the
   same day (`2eccda4`) for the design-v2 gate rot, which is also exactly the brief's item 5. The
   liveness item therefore takes **R-30**, and no new ID was minted for the red-gate item.
6. **The brief's tip `a03e142` is two commits behind** `bbaa844`; Viktor's R-3 draft landed as
   `runbooks/RUNBOOK-onboarding-draft-v3.md`, not `DRAFT-R3-onboarding-runbook.md`.
7. **F2 has no counterpart in the brief at all** — it was found by probing the box rather than by
   following the finding list.

---

## Evidence bundle

`180:~/n100-rehearsal/` — see `MANIFEST.md` there for the per-file map. Secrets were read only to
run probes and are recorded as lengths and metadata, never as values.