felhom.eu/documentation/tests/VALIDATION-n100-rehearsal-2026-07-18.md

VALIDATION — N100 rehearsal: RESET → generic ISO → self-bind → customer zero, 2026-07-18

Supervised run of runbooks/RUNBOOK-rehearsal-v3.md (S1–S8). Viktor at the box as customer zero; CC collected the evidence afterwards (Phase A) before the N100 was unplugged for travel. No production code changed by this run — this document, the capability-map flips, and the ROADMAP items are its outputs. Every finding below is recorded and ranked, never fixed here.

Headline — read this first. The complete final-product flow ran end to end on real metal in one pass: blank hardware → zero-touch install → self-registration → customer self-bind → credential delivery → day-0 → managed-floor lift → drive enrolment → escrow ceremony → offsite snapshots. The chain from bind to a running current-version controller took 2 min 44 s, and the managed floor lifted the controller 0.143.0 → 0.145.0 in 5 seconds, unattended. Three things did not go to plan and are the run's real value: (1) the offsite repo came up ORPHANED, which S7 defines in advance as a finding; (2) the PBS DR tier is silently non-functional — descriptor applied, token rejected 401, and the agent's own verify loop cannot read the credential to notice; (3) pushing the offsite config restarted the controller and logged the customer out mid-flow. Net: core objectives green, 11 ranked items, one of them new and unforecast.

---

Run context

Hardware	Intel N100 (AlderLake-N), 16 GB; the same physical box as VALIDATION-n100-baremetal-2026-07-16
Host	demo-felhom, PVE 9.2.2 (kernel 7.0.2-6-pve), guest 9201 (demo-felhom)
Customer	demo-felhom, domain demo-felhom.eu — reborn clean-slate on this run
Versions	controller 0.145.0 · agent 0.90.0 · hub v0.66.0 · ISO scripts v1.20.0 · host-install 1.17.0
Preparation	RESET ×2 (host-delete-first): demo-vm-felhom 15:49:57, demo-felhom 16:08:51 — both with all five legs ok and escrow acked
Media	safety ISO first, then the real universal pairing ISO v1.20.0, --loader mkimage, Secure Boot OFF (mkimage is unsigned — the documented slice-B constraint)
Evidence	180:~/n100-rehearsal/ (10 files + MANIFEST.md)

⚠ Timezone. The PVE host journal is CEST (+0200); the guest, the controller and the hub are UTC. Every timestamp in this document is UTC. Subtract 2 h from any journalctl line before comparing it with a controller or hub timestamp. This split is live-captured in timezones.txt and is itself worth a ruling (see Observations).

---

Timeline (wall-clocks, all UTC)

Sourced from the hub events stream, the hub DB, the controller log, the bootstrap journal and the agent journal — not from expectation. Where the brief and the evidence disagreed, the evidence is recorded and the disagreement is called out under Discrepancies.

UTC	What happened	Evidence
15:49:57 → 15:50:06	RESET #1 — customer demo-vm-felhom; legs claim/db_purge/descriptor/hetzner/pbs all ok, escrow acked	hub-state.txt customer_resets id=1
16:05:24	host_stale + node_stale — "no report for 30m"	events
16:08:51 → 16:08:59	RESET #2 — customer demo-felhom; all five legs ok, escrow acked (8 s)	customer_resets id=2
16:17:14	Blank metal first boot. felhom-bootstrap in PAIRING mode; registers as an unclaimed appliance the same second	bootstrap-journal.txt; appliance_registrations.first_seen
16:17:14 → 16:28	30 s pairing-poll loop; every iteration exits 1 → 52 systemd FAILED/Failed to start lines on the customer-visible console	bootstrap-journal.txt
16:28:55	Operator mints + emails the self-bind link; 7-day TTL (expires 2026-07-25 16:28:55)	selfbind_tokens created_at/emailed_at/expires_at
16:29:55	Customer self-binds — token consumed, attempts=0, locked=0; appliance_bound source customer_selfbind	selfbind_tokens.consumed_at; events
16:30:21	appliance_credential_delivered (26 s after bind)	events; appliance_registrations.delivered_at
16:30:40	Hub hosts row created — demo-felhom-8363b5	hosts.created_at
16:30:49	Agent 0.90.0 daemon start; fast-tick armed	agent-journal.txt (18:30:49 CEST)
16:30:51	WG keypair generated → registered with hub → 10.77.0.2/32 → tunnel applied (~2 s end to end)	agent journal
16:30:54	Poke listener up, WG-confined 10.77.0.2:51822	agent journal
16:32:28	controller_started 0.143.0 (the golden's version)	events
16:32:29	Managed-floor update initiated — initiated_by: auto-floor	update-state.json
16:32:32	Day-0 provision SUCCESS — done-flag written, unit disabled, bootstrap secrets scrubbed	bootstrap-journal.txt
16:32:34	Floor update complete: 0.143.0 → 0.145.0 in 5 s; new container created	update-state.json; docker inspect .Created
16:32:39	controller_started 0.145.0	events
16:33:24	node_recovered — "Reports resumed (was stale for 27m)"	events
16:41:41	Drive init starts — /mnt/hdd_1, /dev/sdb, ext4	controller log
16:41:42	Agent wipe gate: storage_wipe → customer_confirmable, allowed=false, pending_confirmation	agent journal
16:41:51	Wipe gate: allowed=true, customer_confirmed (9 s of customer thinking time)	agent journal
16:42:19	Device formatted ext4	agent journal
16:42:21	Mount ensured → bound into guest → storage path registered → FileBrowser mounts synced. Init 30.3 s (F6 working)	controller log + agent journal
16:50:22	RAM shrink 11675 → 8192 MB, live cgroup, no reboot	controller log + agent local-api: guest-memory resized
16:45:51	Agent consumes a one-time PBS token secret (secret_len=36) and applies the descriptor	agent journal
16:45:53	pbsdr: converged state=applied	agent journal
16:47:52	Hub mints a NEW PBS token secret — consumed_at is still NULL	host_pbs_secrets
16:47:55 →	Agent PBS verify loop: cannot read token secret … permission denied — repeats every ~15 min, indefinitely	agent journal
16:54:58	Config-refresh: hub config_version=11 != applied=10 → controller self-restarts	controller log
16:55:02	Controller 0.145.0 restarts (container StartedAt 16:54:59Z — a real restart, not a reload)	controller log; docker inspect
16:55:05	offsite-apply: Storage Box target configured, pending key escrow	controller log
16:56:29	Escrow ceremony started via the wizard	controller log
16:56:39	Recovery code claimed (one-shot, not logged)	controller log
16:56:41	Hub-verified → EscrowState auto-confirmed → offsite runs enabled (12 s ceremony → confirmed)	controller log
16:58:12	First offsite run starts	controller log
16:58:14	ORPHAN GUARD FIRES — offbox_repo_orphaned, run skipped, card shown	controller log + events
16:59:26	Operator-confirmed reset: move-aside /home/felhom-repo → .orphaned-20260718	controller log
16:59:32	Fresh repo initialised; offbox_repo_reset	controller log + events
16:59:38 → 16:59:50	Backup run: 0 apps toggled, 0 snapshots, 7 s	controller log
17:02:17	RAM grow 8192 → 12288 MB, live cgroup, no reboot	controller log + agent journal
17:05:15	Calibre-Web deployed; deploy-page memory math now reads total=12288MB (the grow rippled)	events + controller log
17:06:40	Immich deployed	events
17:11:16	BookStack deployed (first start attempt failed exit 1 17:11:36; healthy by 17:12:03)	events + controller log
17:13:38 → 17:14:22	Offsite run: 2 apps backed up, 1 skipped, 2 snapshots, 38 s	controller log
17:13:42	Snapshot 243ad08c — calibre-web	restic-and-pool.txt
17:13:46	Snapshot 6df12205 — immich	restic-and-pool.txt

Derived wall-clocks (the numbers R-3 needs)

Leg	Wall-clock
Blank-metal boot → registered as unclaimed appliance	< 1 s (same second)
Self-bind link minted → customer consumed it	1 min 0 s
Bind → credential delivered	26 s
Bind → agent enrolled + WG tunnel up	56 s
Bind → controller running the CURRENT version	2 min 44 s
Credential delivered → day-0 provision SUCCESS	2 min 11 s
Managed-floor lift 0.143.0 → 0.145.0	5 s, unattended
Drive enrol (confirm → usable, incl. mkfs)	30.3 s
Escrow ceremony → offsite runs enabled	12 s
Boot → day-0 SUCCESS (incl. ~11.7 min waiting for the operator to send the link)	15 min 18 s

---

Per-ledger verdicts

Ledger	Verdict	Evidence
S1 — stop apps, shut down, hub DOWN	PASS	host_stale/node_stale 16:05:24
S2 — the one BIOS visit (boot order, SB off, State-After-G3)	PASS (human leg)	Viktor at the box; SB-off is required by --loader mkimage
S3 — the F1 moment: mkimage GRUB boots where stock GRUB threw relocation 0x0	PASS	The box reached first boot at 16:17:14 on the real board — F1 is closed on physical hardware
S4 — unattended install → pairing mode → appears under Unclaimed appliances	PASS	bootstrap-journal.txt "PAIRING mode … registered — appliance token stored (0600)"; appliance_registrations id=3 status=registered
S5 — the bind + the four-minute event composition, no ~15-min stall between legs	PASS — and this is the immediacy proof	Every post-bind leg is seconds apart: bind 16:29:55 → credential 16:30:21 → agent 16:30:49 → WG 16:30:51 → controller 16:32:28 → floor-lifted 16:32:39. No leg stalled. F5 auto-sizing landed the guest at 11675 MB
S6 — customer zero: claim → password → escrow ceremony → storage → RAM grow and shrink → deploy → enable offsite	PASS on every leg, with two findings attached	Ceremony 16:56:29→16:56:41; wipe gate pending→confirmed 16:41:42→16:41:51; shrink 11675→8192 at 16:50:22 and grow 8192→12288 at 17:02:17, both live with no reboot; 3 apps deployed. Findings: the config push logged the customer out (F3); offsite provisioning is Save-triggered, answering S6's open question
S7 — first snapshot into a brand-new repo; "if an orphan card appears, that's a finding"	FINDING — the orphan card appeared	offbox_repo_orphaned 16:58:14. Recovered by the operator-confirmed move-aside 16:59:26→16:59:32; the run then produced 2 snapshots / 48.717 MiB at 17:13:42–46. The guard worked exactly as designed — the finding is that it had to
S7 (operator gauges)	PARTIAL	Restic tab: Viktor's screenshots. PBS DR tab was not captured, and the reason now matters — see F2
S8 — wrap: evidence, VALIDATION doc, map flips, ROADMAP, R-3	DONE — this document	180:~/n100-rehearsal/MANIFEST.md
Ledger 8 — immediacy legs, first live firing on a real onboarding	PASS	See S5. The "real-onboarding proof pending" note on the immediacy row can be cleared
Ledger 9 — R-24 guest RAM resize, the apply (not just the refusals)	PASS, both directions	Shrink and grow, live cgroup, no reboot, and the new total rippled into the deploy page's memory math at 17:05:15

---

Findings (recorded and ranked — not fixed)

Ranked as proposed for the ROADMAP; IDs assigned in backlog/ROADMAP.md.

F1 — Liveness presence lags reality by ~30 min (→ R-30, P2-HIGH)

The box was powered off at the start of the run, yet the hub carried it as healthy until the staleness threshold expired at 16:05:24 ("no report for 30m"), and cleared only at 16:33:24 ("was stale for 27m"). The host-delete guard compounds this: RESET refuses while any host row exists, so a stale-but-"online" host stalls a forced teardown. Direction: derive presence from Dir-2 long-poll connectedness (~90 s grace), decoupled from notification hysteresis; agent/ep0 analog later.

F2 — PBS DR tier is provisioned-but-dead, and cannot self-report it (→ R-39, P2-HIGH, NEW — not in the brief)

This was not forecast by the brief and is the run's most serious finding. Three facts:

1. The descriptor was auto-provisioned: the agent consumed a one-time token secret at 16:45:51 and reached converged state=applied at 16:45:53. storage.cfg holds a complete pbs: felhom-pbs stanza (namespace demo-felhom, token felhom@pbs!demo-felhom, encryption key, fingerprint).
2. The storage is non-functional. pvesm status → felhom-pbs: error fetching datastores - 401 Unauthorized, inactive. A direct probe with the stored credential returns 401 on every endpoint including /version — so this is authentication failure, not ACL scope. The transport is healthy (WG handshake 9 s old, 27.9 ms RTT, /version reachable), so the token itself is rejected server-side.
3. The hub minted a second secret at 16:47:52 — two minutes after the agent applied the first — and consumed_at is still NULL ~55 min later. The agent will not pick it up: it has already converged to state=applied, and its 15-minute verify loop cannot even read the credential to detect the failure (open /etc/pve/priv/storage/felhom-pbs.pw: permission denied — the agent runs non-root, writes the file through a root sudo wrapper, then tries to read it directly).

Net: a mint/consume race leaves the box on a stale secret, a converged state machine prevents re-application, and a permission bug blinds the only loop that could notice. The DR tier would have failed silently on the first real restore. This also explains why the S7 PBS DR gauge was worth capturing — it would have shown the fault.

F3 — Config apply restarts the controller and drops customer sessions (→ R-35, P2)

The offsite config push bumped config_version 10→11 at 16:54:58; the controller self-restarted (container StartedAt 16:54:59Z) and came back at 16:55:02. In-memory sessions died with it, so customer zero was force-logged-out mid-flow. Direction: hot-apply the offbox target, or persist sessions across restart.

F4 — Offsite provisioning is synchronous, with no status affordance (→ R-31, P2-HIGH)

Save runs the Hetzner sync in-request; the request can hit the nginx 504 while succeeding server-side, so the operator cannot distinguish failed from slow, and a retry races the first attempt. Direction: reuse the awaiting-card/poll idiom. Interim mitigation belongs in R-3: click once, wait, verify.

F5 — RESET destroys custody but not the ciphertext; the pool accumulates unattributed bytes (→ R-32, P2-HIGH)

The RESET ledger records "hetzner":"ok" — the sub-account was destroyed. But a Hetzner sub-account is an access-control object, not a data object: deleting it leaves its directory intact. Re-enabling offsite recreated a sub-account over the same /home, which still held the previous lifecycle's repo — under a key destroyed by that same RESET. Hence S7's unexpected orphan card. Measured on the pool box tonight:

Path	Size	Status
/home/felhom-repo	49 M	live, attributed (2 snapshots, 48.717 MiB)
/home/felhom-repo.orphaned-20260717	1.4 G	unattributed, key destroyed — from the 07-17 incident
/home/felhom-repo.orphaned-20260718	3.0 M	unattributed, key destroyed — from tonight

Ruling from the run: because RESET destroys custody, leftover ciphertext is unrecoverable by design, so RESET gains a main-account purge of the customer base dir (the operator ack already covers it). The move-aside guard stays for reinstall-without-RESET, where custody survives and the card's promise is true. The operator Restic tab should show per-customer directory bytes against attributed snapshot bytes so dead data cannot hide.

F6 — Bootstrap pairing-poll spams the customer-visible console (→ R-33, P2-HIGH)

Each 30 s poll is a separate felhom-bootstrap.service run that exits 1, so systemd prints Failed to start … on the console the customer is watching. 52 FAILED lines in ~11 minutes, while nothing was actually wrong — the box was correctly waiting to be bound. Direction: poll inside the script (scripts v1.21) and reword the first-boot noise.

F7 — Design-v2 gates are unenforced; one has been red for 16 releases

Already ranked as R-29 (committed 2eccda4, 2026-07-18) — the brief asked for a new ID, but the finding already has one, including the docker_run_volume_path_gate / internal/appexport/estimate.go attribution. No new ID minted.

F8 — Backup data lifecycle is unmanaged (→ R-34, P2)

Apps with snapshots but no active backup (disabled or uninstalled) are invisible and undeletable. Direction: an "inactive backups" section on Távoli mentés — name / size / last snapshot / restorable, with an explicit double-confirmed per-app delete via restic forget --tag + nightly prune. RULING: the offsite toggle NEVER offers deletion — policy and destruction stay decoupled.

F9 — Post-RESET re-enroll leaves offsite "enabled but unprovisioned", silently (→ R-36, P2)

The hub knows the state and says nothing. Provisioning is Save-triggered (configs.go applyOffsite) — confirmed live tonight, which answers S6's open question — and the re-enroll auto-re-issue correctly skips unprovisioned targets (handler.go). Direction: flash it on the customer page. Interim: an R-3 step. Related: auto-mint the self-bind link on customer create/RESET so the console banner's "e-mailben kapott link" is always already true.

F10 — Console-passphrase bind (→ R-27c, P2)

Viktor's direction, with the session's security constraints: passphrase issued at customer creation; spray-hardened global-lookup endpoint (per-appliance + per-IP caps, constant-time, single generic failure, alerting); accent-free wordlist for console keymaps; the web capability-link path RETAINED; claim-by-email RETAINED as the delivery-channel proof. The self-bind email gains the public universal-ISO download link + two-line instructions (the DIY case). Secret-bearing per-customer ISOs are ruled OUT.

F11 — Post-RESET health card shows stale pre-RESET warnings (→ R-37, P3)

Should read „RESET óta nincs adat".

F12 — Installer GRUB menu offers interactive installers (→ R-38, P3)

Single default „Felhom telepítés" entry; interactive installers REMOVED (safety: wrong-disk manual installs); felhom background. Squashfs/theme rebranding explicitly deferred.

---

Observations (not findings — no ID minted)

• Host/guest timezone split. The PVE host runs CEST while the guest, controller and hub run UTC. Nothing is broken, but every cross-tier correlation needs a manual 2 h shift, which is exactly the kind of friction that produces wrong incident timelines under pressure. Flagged for a ruling rather than ranked unilaterally.
• The escrow awaiting-card window was 12 seconds. Ceremony → hub-confirmed took 16:56:29 → 16:56:41, so the „megerősítésre vár, legfeljebb 15 perc" card rendered only briefly. The card is correct and its ≤15-minute promise is honest, but capturing both states on screen is a matter of luck at this speed.
• BookStack skipped its first offsite run — no recovery unit found on any connected drive (17:13:42). This is the expected pre-first-nightly-dump state; the nightly job is scheduled 2026-07-19 04:15 CEST. Watch item, not a finding — it becomes one only if it still skips afterwards.
• BookStack's first start attempt failed (exit 1, 17:11:36) and succeeded on retry by 17:12:03. Self-corrected; noted for pattern-watching only.
• felhom-samba was not running on the reborn box — sharing was not exercised this run (consistent with What was NOT exercised).

---

What was NOT exercised (explicit)

• C6 — a customer performs a restore unassisted. The single open script step in R-3, and still MISSING as evidence. Remains remote-doable on the reborn box from vacation (the dashboard is remote).
• freemail.hu deliverability. The gmail half is evidenced (claim email → Inbox under p=quarantine); freemail.hu is Viktor's open half.
• Sharing smoke on the reborn box — R-7/R-7b were proven on the pre-RESET box; felhom-samba was not deployed here.
• R-19 WAN-outage drill.
• The PBS DR operator gauge — not captured, and F2 means it would have shown a fault.
• BookStack's offsite unit — watch item, see Observations.
• SMB positive auth with the real household password — carried over from R-7b, unchanged.

---

Discrepancies vs the Phase-A brief (live evidence wins)

1. "expect ~11675 MB ballpark" — the final pct config reads 12288 MB. Both are right at different times: F5 auto-sized the guest to 11675 MB, then customer zero shrank it to 8192 (16:50:22) and grew it to 12288 (17:02:17). The brief described the F5 value; the box ended above it because the grow was the last resize. Not a fault — and it is what completes ledger 9.
2. "2.6 GB pool vs ~0 attributed" — measured 1.4 G + 3.0 M unattributed against 49 M attributed, across TWO orphaned directories, not one. The F5 ruling is unaffected; the numbers are.
3. "felhom-pbs entry present" — present but 401 Unauthorized / inactive. The brief treated the PBS descriptor as done; it is provisioned but dead. See F2.
4. "RESET ×2 … second + third live firings" — the ledger holds exactly two rows, both tonight (demo-vm-felhom 15:49:57, demo-felhom 16:08:51), on two different customers. Any earlier firing is not in customer_resets. Described here as tonight's two firings rather than as ordinals.
5. "keep the name R-29 if the number is free" — it is not free. R-29 was committed earlier the same day (2eccda4) for the design-v2 gate rot, which is also exactly the brief's item 5. The liveness item therefore takes R-30, and no new ID was minted for the red-gate item.
6. The brief's tip a03e142 is two commits behind bbaa844; Viktor's R-3 draft landed as runbooks/RUNBOOK-onboarding-draft-v3.md, not DRAFT-R3-onboarding-runbook.md.
7. F2 has no counterpart in the brief at all — it was found by probing the box rather than by following the finding list.

---

Evidence bundle

180:~/n100-rehearsal/ — see MANIFEST.md there for the per-file map. Secrets were read only to run probes and are recorded as lengths and metadata, never as values.