felhom.eu/documentation/audits/DRILL-day0-vm-2026-07-12.md

DRILL — Day-0 on the Demo-VM (nested PVE on felhom-pve), 2026-07-12

Executed per RUNBOOK (Day-0 drill on the Demo-VM). Actors: Viktor (console install, dry-run go/no-go, fork decisions, the ceremony/R-moment, dialog clicks) + CC (everything scriptable over SSH + the hub/dashboard browser tracks). Outcome: the full arc ran to completion — Day-0 appliance install, first live escrow ceremony + AUTO-CONFIRM FIRST FIRING, offsite backup + restore round-trip — at the cost of one mid-drill fork (PBS DR tier attached) forced by the drill's headline findings (F-4, F-6).

1. Baselines recorded (§1 of the runbook, verified LIVE before Phase 1)

Item	Recorded
Day-0 artifact manifest	Agent 0.85.0 sha256 31babb2c4f8fa5a1f961428519a0dfbb3a19b6e536fba01c2664afca339da93d; Golden 0.120.0 sha256 f7d7d02c76b49891719df9cf68624c9ae193e438e2a3a63b948a2a9d12887596; MinAgent 0.81.0 — matches the 0.85/0.120 publish train, no bump needed
Global controller floor	Effective v0.120.0, source DB (hub_settings); env fallback also v0.120.0
Customer demo-vm-felhom	exists, domain enkisfelhom.hu, CF tunnel + API tokens saved (API token perms incl. Zone WAF:Edit); git credentials EMPTY (correct per G3; hub edit-form already labels Git Sync "Opcionális" — only day0-install.md A.2 was stale, fixed in this commit); DEBUG MÓD was ON
Wildcard DNS	*.enkisfelhom.hu → CF proxied edge (104.21.3.175 / 172.67.130.252 + AAAA), verified via 1.1.1.1 on a real and a random subdomain
felhom-pve capacity	local-lvm thin pool 348.82g @ 7.47%; RAM available 12.9 GiB; nested virt Y
Installer	felhom-host-install.sh v1.14.0 per -h (but see F-1)
Golden-baked controller	0.120.0 (= floor; see 4.3 note)

2. Per-phase gates

Phase	Gate	Result	Evidence (abridged)
0 — build VM	P0	PASS	qm 300 drill-day0 (8G/4c/host/250G thin); PVE 9.2.2 ISO 9.2-1; key SSH proven; snapshot pre-day0-clean 15:12:41. ISO detached + boot→scsi0 BEFORE the snapshot (clean config)
1 — hub verify	—	PASS	§1 table above; nothing created/changed in the hub
2 — box prereqs	P2	PASS	single node; nested local-lvm 149.88 GiB free (≥120 ✓ — note: a 250G VM disk yields ~150G nested pool after the installer's root/swap split); vmid 9201 absent; hub 302 / gitea 200 / felhom.eu 200
3 — install	P3	PASS	dry-run reviewed by Viktor → GO → real run 15:25–15:28: "Day-0 provision SUCCESS — vmid=9201 host_id=demo-vm-felhom-2482b0"; both artifacts sha-verified vs the hub manifest; both expected ANONYMOUS-fetch warns; 4b break-glass vaulted; passphrase file 0600, shredded after (never in CC's transcript — Viktor wrote it himself)
4 — post-install	D.1–D.4	PASS (1 finding)	see §3
5 — hub tracks	G9/G10	PREMISE COLLAPSED — F-4/F-5; geo exercised via the shipped path	see §4
6 — escrow/auto-confirm/offsite	fork-4	PASS after fork (PBS DR attached) — F-6/F-7/F-10/F-11	see §5

3. Phase 4 detail (post-install verification)

• D.1: agent active as non-root felhom-agent; full read-only selftest ALL-OK incl. pool read (pool felhom, member 9201); guest running, onboot=1, mounts mp0 docker 50G / mp1 sys 20G / mp8 felhom-drives / rootfs 32G; in-guest containers controller 0.120.0 (healthy) + filebrowser + cloudflared + traefik; dashboard 200 in-guest (traefik https + Host header; :80 → 301).
• 4.2 appliance gating (first live appliance box): agent.json deployment_mode="appliance"; journal selfheal: node watchdog starting mode=appliance interval_s=60 + storage watchdog armed; felhom-mgmt-watchdog.timer enabled+active. NOT byo-defaulted.
• 4.3 floor self-update (tester-recruitment gate): guest landed 0.120.0 = floor at its FIRST report, zero manual steps, elapsed ≈ 0. LIMITATION: baked == floor on this train, so the floor-driven update path was not stressed — re-prove on the next train where golden < floor.
• 4.4 hub: customer PENDING→ok ~2 min after provision; host ONLINE agent 0.85.0; guest 9201 appears at the next agent report (900 s cadence — the empty Guests panel in between is timing, not a bug); only-degraded capabilities = the 3 pbsdr-* ("binary not found") — expected no-PBS shape (and see F-7).
• 4.5 customer-visible via the REAL edge (method: curl --resolve on both CF anycast IPs — the split-horizon-proof variant): felhom.enkisfelhom.hu → 200 "Vezérlőpult", OPEN, no auth (G10 "before", ~15:30).
• 4.6 first-app smoke: ActualBudget via the open dashboard → "Telepítés sikeres"; real-edge 200 + <title>Actual</title> on budget.enkisfelhom.hu. (The old POST-via-public-URL no-op gotcha did NOT reproduce on 0.120.0 for normal forms.)
• Snapshot post-install 15:35:37 (live, no fs-freeze — no qemu-guest-agent in the drill VM).

4. Phase 5 detail — the G9/G10 premise vs shipped code

• 5.1 operator password-set: IMPOSSIBLE (F-4). No hub UI/API sets per-customer web.password_hash; the controller's security page in the open state says "Kérd az üzemeltetőt"; settingsPasswordHandler requires a current-password match (no initial-set); the hub-preseeded Day-0 setup path skips the only wizard form having a password field.
• 5.2 geo-restriction: apply lives in the CUSTOMER dashboard (controller settings_security.html + api/geo.go), not the hub (hub has only handleGeoDisable) — F-5. Exercised via the shipped path: HU-only enabled → controller created WAF rule "[felhom-geo] Global" ((not ip.src.country in {"HU"}), action block) on the zone, sync reported 1 active rule; HU access still 200 through the real edge. Non-HU block not testable from an HU vantage.
• 5.3 G10 statement: BEFORE — dashboard OPEN (proven in-guest + real-edge, ~15:30). AFTER — STILL OPEN; closure impossible until F-4 ships. The tester-agreement onboarding gate that points at this line CANNOT currently be satisfied.

5. Phase 6 detail — offsite → ceremony → auto-confirm → tier proof

Pre-ceremony state (all verified): offbox pre-provisioned at customer-create (u629488-sub3@…your-storagebox.de:23, /home/felhom-repo, pinned host fingerprint, 0/50 GB); controller EscrowState=pending — the /backups banner "a mentés addig nem fut" (the F6 no-single-copy guard, LIVE); staged escrow-stage/restic_repo_password on the agent (0600, staged at provision — fork-4 stage-FIRST ordering held); ActualBudget toggled for NAS.

The ceremony blocked → the drill's second headline (F-6): --selftest=escrow-create --upload refuses without a PBS storage/key; identity-only mode does not exist in any shipped agent (the ≥0.80.0 claim in RUNBOOK-escrow-ceremony.md was false — fixed in this commit). On a no-PBS appliance box the offsite arc can NEVER complete. Fork decision (Viktor): attach the PBS DR tier (ep0) — which converted the drill into a full rehearsal of Peti's pending sequence:

1. Ship configs/felhom-pbs-apply → /usr/local/sbin (the binary is missing from host-install — F-7; the FELHOM_PBSDR sudoers alias DOES ship). Agent restart → zero capability-DEGRADED.
2. Hub PBS-DR enable failed correctly: "host has not reported a WG key yet — the tunnel peer must exist before the PBS DR tier" — the dependency is hub-enforced (good), and the drill runbook's "no WG" scope was inconsistent with any PBS path.
3. WG: wg_tunnel.enabled=true in agent.json (installer never sets it — decide the appliance default together with the F-6 spec) → keygen → hands-free hub registration (10.77.0.3/32, gen 1, no operator vouch) → conf applied → handshake + ping 10.77.0.1 (33 ms).
4. Hub PBS-DR enable → "Configuration updated" → descriptor gen 2; apply-bridge first 403'd (Datastore.Allocate on /storage/felhom-pbs) — consequence of the drill's narrowed --acl-storages "local local-lvm"; the installer DEFAULT includes felhom-pbs exactly for this. Retrofit dual-grant → next tick: token consumed (single-use) → entry + K created → escrow.pbs_storage_id seeded → state=applied (16:17). pvesm ACTIVE; felhom-pbs.{enc,pw} present.
5. Ceremony (Viktor, R on paper, nothing in CC's transcript): attempt 1 failed — age missing (F-10, apt-get install -y age → 1.2.1); attempt 2 SUCCESS ~16:35; the bundle auto-captured +wg_private_key +restic_repo_password; ceremony wiped the staged secret; hub host page flipped to DR RECIPE: present / KEY ESCROW: present.
6. AUTO-CONFIRM FIRST LIVE FIRING: hands off, "Letét megerősítése" untouched → 16:42:36 controller log: "hub-verified: the escrow covers the current repo password (hash 99c16e8d84e7…) — EscrowState auto-confirmed escrowed; offsite runs enabled." Elapsed ≈ 7.5 min, zero clicks. /backups gate banner gone.
7. Tier proof: "NAS-mentés most" → repo initialized on the Storage Box → 1 snapshot, 14 s, "✓ Rendben". Restore-to-verify (native confirm() blocked automation twice and Viktor missed the popup — F-11; also live operator confusion: he first launched the FULL local tier-1 restore, which itself completed healthy in 8.7 s) → offbox verification restore → "restored actualbudget → …/offbox-restore/actualbudget", 48K payload (app.yaml, .felhom.yml, docker-compose.yml, manifest.json) decrypted from the box — the offsite round-trip is proven. Bonus: the full local tier-1 restore was ALSO proven the same afternoon.

6. Headline metrics

Metric	Value
Wall-clock Phase 2 → 6 complete	~15:15 → ~17:09 (~1 h 55 m, including the mid-drill fork, 3 fix-and-continue stops, and two supervised STOPs)
Install run itself (Phase 3)	~3 min
Floor self-update (4.3)	at-floor at FIRST report, 0 manual steps (baked == floor; update path not stressed)
Auto-confirm (6.4)	~7.5 min ceremony→escrowed, zero manual clicks (FIRST LIVE FIRING)
Offsite backup / restore	14 s backup (1 snapshot) / verification restore round-trip proven

7. Findings

#	Sev	Phase	Finding	Disposition
F-1	LOW	3	Version-string mismatches: -h v1.14.0 vs run banner v1.13.0 vs hub Setup-tab copy "1.12.0"	SHIPPED 2026-07-12 (installer v1.15.0 + hub v0.51.0): single SCRIPT_VERSION source (header carries no literal; -h/banners read it; hub hostInstallVersion=1.15.0) — drift now gated by scripts/hostinstall_gates.py (asserts installer == hub const)
F-2	COSMETIC	3	dry-run prints curl -u <git> on the anonymous-fetch branch	SHIPPED (installer v1.15.0): dry-run fetch lines print the real auth mode (curl (anonymous — no git credential) vs -u <git>)
F-3	MEDIUM	4	Root-run provision leaves /var/lib/felhom-agent/guests{,/9201} root:root 0700 inside the agent-owned state dir → non-root agent lanresolver "permission denied". LIVE-FIXED (chown the two parent dirs; the guest-root-owned bootstrap subtree untouched)	SHIPPED (agent v0.86.0): a root-run provision chowns the guests/+guests/<vmid>/ PARENTS to the state-dir owner (chown --reference, non-recursive; bootstrap leaf stays guest-root) — red-proofed. felhom-pve latent state CONFIRMED + FIXED live 2026-07-12 (same chown); Peti's host unreachable — his check deferred to the D/E/G visit
F-4	HIGH	5	No operator-set dashboard password path exists anywhere (hub has no UI/API for per-customer password_hash; controller open-state page defers to the operator; Day-0 preseeded path skips the wizard's password form) → G10 unclosable; every fresh box's dashboard stays OPEN on the internet	RESOLVED 2026-07-12 — customer-claim arc (controller v0.122.0 + hub v0.50.0). The customer OWNS the password, set via a one-time claim code the hub emails to the registered address; an unclaimed box serves only the claim page (no open window). The operator-set premise was correctly abandoned. Live-proven on the drill box (gate ON via the real edge).
F-5	MEDIUM	5	Geo-restriction APPLY is customer-dashboard-side; hub only disables. Runbook premise stale; also: the open dashboard (F-4) exposes the geo toggle unauthenticated	RESOLVED with F-4 — the geo toggle (and every other route) is now behind the claim gate on an unclaimed box. The apply-is-customer-side design is accepted (documented).
F-6	HIGH	6	Identity-only escrow ceremony was never implemented (escrow-create hard-requires PBS storage + key; the ≥0.80.0 runbook claim was false) → on a no-PBS box (the documented appliance standard!) the offsite escrow chain can never complete	CLOSED BY POLICY 2026-07-12 (DR-tier-by-default decisions 3/4): identity-only escrow is formally PARKED; every install is DR-capable (installer v1.15.0), activation = the hub per-customer dr_tier flag (default ON, hub v0.51.0), and offsite REQUIRES the tier (guard: "Offsite backup requires the DR tier — enable it first (the escrow ceremony depends on the PBS key)"). The no-PBS shape can no longer reach the offsite arc. Ceremony runbook carries the decision block
F-7	MEDIUM	6	host-install ships the FELHOM_PBSDR sudoers alias but NOT the felhom-pbs-apply binary → pbsdr capabilities born DEGRADED on every fresh box	SHIPPED (installer v1.15.0): wrapper fetched + bash -n + installed 0755 (the mkfs/selfupdate pattern); uninstall removes it; gate-asserted. Companion: agent v0.86.0 adds the inactive capability state (DR off = neutral chip, NEVER degraded; missing binary STAYS degraded + hub migration hint)
F-8	LOW/UX	3	Step 4b rotates root@pam + vaults silently — operator surprised by 401 at the PVE GUI (live: Viktor)	SHIPPED (installer v1.15.0): loud WARN at 4b ("the root@pam password just CHANGED… retrieve at hub → host page") + repeated in the final summary; pointer only, never the credential
F-9	NOTE	6	Installer never sets wg_tunnel.enabled; WG registration itself is hands-free once enabled	DECIDED + SHIPPED (decision 5; installer v1.15.0): WG is BASE infrastructure — wg_tunnel.enabled: true rendered in BOTH modes (setdefault; deliberate preserved-off respected). byo wg-off assert retired; consent = typed disclosure ack + tester-agreement line. Hub v0.51.0 auto-provisions the PBS-DR descriptor on the peer registration (hands-free cascade)
F-10	MEDIUM	6	age (ceremony identity-wrap dependency) not installed by host-install — fresh-box ceremony dies; demo host masked it (spike-era install)	SHIPPED (installer v1.15.0): age installed idempotently in step 5 (both modes); gate-asserted; runbook prereq updated
F-11	LOW	6	Native confirm() on the offbox restore-verify form: blocks browser automation, easy to miss (operator missed it twice live; meanwhile launched the full tier-1 restore from the adjacent form — the two restore controls invite confusion)	convert to the design-system inline confirm pattern; consider renaming

Observations (not F-numbered): hub "REGISTRY LATEST v0.120.0 — up to date" vs the dashboard's own "Új controller verzió elérhető: 0.121.0" banner disagree on "latest"; the hub dashboard row for Peti shows "OK · minutes ago" (controller-derived, via his proxmox2 migration) while his agent HOST is DOWN 23h — the roll-up masks a dead host; retrofit-ACL note: adding the PBS tier to a box installed with narrowed --acl-storages needs the /storage/ dual-grant (documented default avoids it).

8. What this proved for Peti — and what it did not

Proven live on a fresh box (his exact pending sequence): wrapper+age prep → WG enable → hands-free peer registration → hub PBS-DR enable (dependency guard works) → apply-bridge fresh path (consume → K → grant → escrow seed) → ceremony (with K) → auto-confirm → gated offsite run → restore round-trip. Prep list for his box: felhom-pbs-apply + age + wg_tunnel.enabled

• (if his ACL set was narrowed) the /storage dual-grant.

Deliberately NOT covered: WG OOB operator peer (separate arc); S5 DR restore from total loss (identity-consume — queued as its own drill); byo-mode re-run from pre-day0-clean (queued); non-HU geo-block verification (needs a non-HU vantage).

9. Snapshot inventory (qm 300 on felhom-pve, at drill end)

Snapshot	When	State
pre-day0-clean	15:12	PVE 9.2.2 + keys, nothing Felhom — the universal re-drill zero
post-install	15:35	Day-0 SUCCESS + ActualBudget + guests-dir chown
post-drill	17:11	full drill end state (WG + PBS-DR + escrowed + geo HU + dashboard OPEN per F-4)
post-claim-arc	19:08	customer-claim arc: ctrl 0.122.0, demo-vm-felhom CLAIMED (customer-set password), gate proven

10. F-4/F-5 resolution — customer-claim arc (2026-07-12, same day)

Shipped hub v0.50.0 + controller v0.122.0 (both LIVE): the dashboard password is customer-owned, set via a one-time claim code the hub emails to the registered address. An unclaimed box serves only the claim page — F-4's open-dashboard exposure and F-5's unauthenticated geo toggle are both closed. Live-proven on this drill box (gate ON via the real edge; Viktor claimed it, code generation consumed, box now password-gated) and on the live demo after the floor raise (0.121→0.122 → claim-gated). Two follow-up observations for the backlog: (a) the hub issues a claim code to EVERY reporting customer on first report (peti + demo also emailed) — benign for pre-0.122 boxes (no gate, ACK ignored), but a real customer on an old controller gets an unusable "beállító kód" email; consider gating issuance on reported controller version ≥ 0.122. (b) the drill's reset flow (Scenario C) was deferred by the operator — worth one supervised pass before Peti onboarding.

Blast radius honored: guest demo 9201 on felhom-pve, Peti's hub entry, and the demo offbox were never touched. Drill leftovers to tidy at teardown (deliberately kept for re-drills now): hub customer demo-vm-felhom (its api_key + CF tokens surfaced in the operator UI during the drill — delete/rotate at teardown), DEBUG MÓD on, geo HU rule on the drill zone, VM 300 running.