Go-live package: Day-0 install runbook + clean-room drill + host-install v1.9.1

- documentation/runbooks/day0-install.md (NEW): operator Day-0 guide, every
  command drill-executed verbatim (Parts A-F incl. the mandatory one-time
  controller update D.1b and the OQ-3 prereq checklist).
- documentation/audits/DRILL-day0-cleanroom-2026-07-03.md (NEW): nested-PVE
  clean-room drill evidence D0-D6, residue tables, OQ-1/2/3 answers, findings
  ledger (B1-B5), go/no-go for Peti (GO).
- scripts/felhom-host-install.sh v1.9.1: residue-free uninstall (agent config,
  shared-parent unit/script/mount, mkfs wrapper, hook snippet, dnsmasq
  snippets; zero-residue diff proven) + post-provision guest reboot (R6) +
  header/version sync.
- CONTEXT.md, REPORT.md, scripts/CHANGELOG.md updated.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6

documentation/audits/DRILL-day0-cleanroom-2026-07-03.md

@@ -0,0 +1,197 @@
+# DRILL — Day-0 clean-room install / uninstall / reinstall (2026-07-03)
+
+**Class:** Clean-room drill (empirical validation of `documentation/runbooks/day0-install.md` +
+residue-proof of `--uninstall`). Companion deliverables: the runbook itself and
+`scripts/felhom-host-install.sh` **v1.9.1**.
+
+**Verdict (short):** the Day-0 guide is **executable verbatim end-to-end** on a virgin PVE 9.2 box:
+install → verify → app deploy → uninstall → **zero-Felhom-residue diff** → reinstall all pass.
+The drill found **one installer gap that mattered** (the controller never deployed without a guest
+reboot — fixed in v1.9.1), **five uninstall residue items** (all fixed in v1.9.1, re-proven by a
+second residue-diff), and produced a decisive **OQ-2 answer** (a fresh box does NOT self-update its
+controller — a one-time manual update is a mandatory install-day step). **Go for Peti** with the
+go/no-go list in §8.
+
+No secrets appear in this document. Drill credentials lived only on the build server (0600) and in
+the drill VM; the drill VM was reverted to its virgin snapshot afterwards.
+
+---
+
+## 1. Environment (Phase 0 — primary mechanism, validated)
+
+| Item | Value |
+|---|---|
+| Host | build server 192.168.0.180 (Debian 13, /dev/kvm, QEMU 10.0.8 installed for the drill) |
+| VM | nested **PVE 9.2.2** (unattended install: `proxmox-auto-install-assistant` 9.2.7 answer file, ext4/LVM-thin layout), 4 vCPU / 8 GiB RAM / 150 GiB sparse qcow2 |
+| Resulting storage | `local` 46 GiB dir + `local-lvm` **76 GiB** thin — deliberately BELOW the 120 GiB floor, so the warn path is exercised |
+| Network | QEMU user-mode NAT (slirp); outbound to hub.felhom.eu / gitea.dooplex.hu / felhom.eu confirmed from host AND guest; SSH via hostfwd |
+| Restore point | qcow2 internal snapshot `virgin` taken post-install, pre-first-boot |
+| Hub side | live hub v0.31.0; throwaway customer **drill-1** (domain `drill.demo-felhom.eu`, fake CF tunnel token, real Gitea read credential copied from the demo customer's row, hub-generated-format passphrase) inserted directly into `customer_configs` (the web create form is operator-password-gated — CC cannot drive it; the row is byte-identical to `SaveCustomerConfig`, spike-day0 precedent) |
+| Versions under test | script v1.9.0 → v1.9.1, agent v0.62.0, golden v0.85.1 (hub artifact manifest), controller floor 0.87.0, latest 0.98.3 |
+
+**Environment quirk (drill-only, not product):** slirp's DHCP pool starts at 10.0.2.15 — the same
+address the PVE host holds statically — so the freshly-provisioned guest's DHCP grabbed the host's
+IP, intermittently breaking the drill SSH (and truncating the D2 transcript). Worked around by
+pinning the guest to 10.0.2.40. A real LAN with a real DHCP server does not have this failure mode.
+
+## 2. D0 — baseline
+
+Captured off-box before any Felhom artifact touched the box: sorted file list of
+`/etc /usr/local/bin /usr/local/sbin /var/lib` (5 076 entries), `systemctl list-unit-files`
+(379 units), `pveum user/role/acl/pool list`, `pvesm status`, `pct list; qm list` (empty).
+Stored at `~/drill/D0/` on the build server.
+
+## 3. D1 — hub onboarding (guide Part A)
+
+- Customer-create field set verified against `hub/internal/web/configs.go`
+  (`handleConfigCreate` + `buildConfigJSON`): customer_id/name/domain/email +
+  `infrastructure.cf_tunnel_token`, `infrastructure.cf_api_token`, `git.username`, `git.token`;
+  passphrase (5 words) + api-key hub-generated. **`git.token` is load-bearing for Day-0** — the
+  installer dies at step 5/8 without it (guide Part A marks it required).
+- Box-facing contracts probed for drill-1: `GET /api/v1/config/drill-1` → **200**;
+  `GET /api/v1/artifacts/drill-1` → agent 0.62.0 + golden 0.85.1; wrong passphrase → **401**.
+- **UI caveat:** the operator-password-gated create form itself was not click-driven (CC has no
+  operator password); the row was inserted store-identically. Part A's field list is code-verified
+  and its outputs are live-verified.
+
+## 4. D2 — install (guide Parts B/C, script v1.9.0)
+
+Part B checklist ran verbatim (two guide corrections folded in: hub root answers **302**, and a
+cleaner node-count one-liner). Script fetched from `https://felhom.eu/scripts/felhom-host-install.sh`
+(text/plain, the git-synced repo file). `--dry-run` previewed all 8 steps correctly. Real run:
+
+```
+./felhom-host-install.sh --customer-id drill-1 --vmid 120 --cores 2 --memory 4096 \
+    --force-gitea-golden --acl-storages "local local-lvm" --passphrase-file /root/.drill-pass
+```
+
+- pre-flight: PVE 9.2.2, node auto (single), **`local-lvm ~75 GiB < 120 GiB` warn fired as designed**,
+  passphrase validated, grows auto-computed for the small box (docker +34G, sys +12G).
+- token + pool + scoped 3-role ACL applied (Base@/, Guest@/pool/felhom, Store@[local local-lvm]).
+- host-enroll: **201 MINTED** (`drill-1-c867f3`).
+- agent 0.62.0 fetched + **sha256-verified** against the hub manifest; sudo installed; non-root
+  user/sudoers/unit installed; selftest passed; service active.
+- golden 0.85.1 fetched + sha256-verified from Gitea (`--force-gitea-golden` — the real fresh-box path).
+- provision: guest 120 up in 1m26s, cores=2/memory=4096 applied, 32/50/20G volumes, mp9 bootstrap
+  mount, onboot=1, pool member. State file recorded `provisioned_vmid=120`
+  (**confirms** the v0.62.0-era open question: a fresh install DOES record it).
+
+## 5. D3/D4 — verify + app deploy
+
+- Agent: `--selftest` all-OK **including `pool read pool "felhom", 1 member`** (the v0.62.0
+  `pve:pool-read` capability); service active as `felhom-agent`.
+- Hub: host row reporting (agent 0.62.0), guest 120 `running`; events pipeline live from the fresh
+  box (`controller_started`, `app_deployed`, `controller_updated`, 2× `host_leaf_changed` warnings
+  from the reinstalls — correct behavior).
+- **Finding R6 (the big one):** after provision the controller was NOT running —
+  `felhom-controller-bootstrap.service` was skipped
+  (`ConditionPathExists=/etc/felhom-bootstrap/bootstrap.json` unmet at boot): the agent back-half
+  **hot-plugs** the bootstrap mount into the already-running guest and its own output says
+  *"next: reboot the guest"*, but the v1.9.0 script never rebooted. Fast hardware (demo) usually
+  won this race; the slow clean-room box lost it deterministically. **Fixed in v1.9.1**
+  (post-provision `pct reboot` + bounded verify wait); re-proven in D6b (§7).
+- Dashboard: the controller sits behind traefik — correct probe is
+  `curl -k -H "Host: felhom.<domain>" https://127.0.0.1/` → **200** (Hungarian dashboard). The
+  guide's draft `:8080` probe was wrong and was corrected.
+- **D4 app deploy:** `bentopdf` deployed through the exact UI pipeline (page fetch → the same
+  `POST /api/stacks/bentopdf/deploy` + empty-CSRF headers the dashboard JS sends — endpoint-proxy
+  method per the live-validation rule; no browser bridge was available in this session). Container
+  healthy; app answers **200** at `pdf.drill.demo-felhom.eu` via traefik. Public Cloudflare leg not
+  drill-testable (fake tunnel token) — that leg is live-proven daily on the demo box.
+- **OQ-2 evidence:** fresh box landed controller **0.85.1** (golden-baked) and did NOT
+  self-update (floor 0.87.0 ignored — 0.85.1 predates the v0.86.0 floor code; its one report ACK
+  carried no consequence). Manual update (`/api/selfupdate/check` + `/update`, the settings-page
+  button flow) first **failed with the known fresh-install local-API 401** (`POST /controller/swap:
+  HTTP 401`, stale token map); after `systemctl restart felhom-agent` the same trigger pulled
+  0.98.3 and the agent swap landed it: **0.85.1 → 0.98.3, healthy**. Both the failure and the
+  recovery are now guide steps (D.1b).
+
+## 6. D5 — uninstall + residue diff (v1.9.0 → the gap list)
+
+`--uninstall --vmid 120 --remove-golden` ran with the typed-vmid confirmation; guest/agent/pveum/
+pool/state/golden all removed. Full re-capture + diff vs D0 classified every delta:
+
+| Delta (v1.9.0 uninstall) | Class |
+|---|---|
+| `/etc/felhom-agent/agent.json` (+dir) — **holds the live hub api_key** | **RESIDUE R1** → fixed v1.9.1 |
+| `felhom-shared-parent.service` (+wants links) + `/usr/local/sbin/felhom-shared-parent.sh` + `/mnt/felhom-drives` self-bind still mounted | **RESIDUE R2** → fixed v1.9.1 |
+| `/usr/local/sbin/felhom-mkfs-guarded` | **RESIDUE R3** (suspected by the spec — confirmed) → fixed v1.9.1 |
+| `/var/lib/vz/snippets/felhom-guest-hook.sh` | **RESIDUE R4** (would remain where the snippet install succeeded; on this box it never installed — see B2) → fixed v1.9.1 |
+| `/etc/dnsmasq.d/felhom-resolver-base.conf` | **RESIDUE R5** → fixed v1.9.1 |
+| `sudo` + `dnsmasq` packages (+ their files/units) | expected remnant (documented in the guide; dnsmasq newly noted) |
+| hub host record + report/guest/event history | expected remnant (no hub delete path — B4) |
+| LVM archive/backup metadata, dpkg lists, PVE rrd + `/var/lib/lxc/120` scratch, `.pve-staged-mounts` | system noise (PVE/dpkg-owned) |
+
+## 7. D6 / D6b — reinstall proofs
+
+- **D6 (v1.9.0, post-uninstall box):** full reinstall succeeded — host-enroll **200 REUSED**
+  (mint-once-reuse proven), fresh leaf minted (box had no guests → the populated-host guard
+  correctly stayed silent), golden re-fetched, guest provisioned. Uninstall leaves a reinstallable
+  box. (Controller again required the reboot — the R6 evidence repeated deterministically, incl.
+  the agent's own "next: reboot the guest" line and the guesthook warn, see B2.)
+- **v1.9.1 re-verification (all fixes):**
+  - `bash -n` + `-h` (header now v1.9.1) + full `--dry-run` previews of the new removal steps.
+  - **Uninstall re-drill:** v1.9.1 `--uninstall --remove-golden` → re-diff vs D0:
+    **zero `felhom`-named entries** in the file diff, zero pveum objects, no mounts, both dirs gone.
+  - **D6b install re-drill:** full v1.9.1 install on the clean box → the script rebooted the guest
+    itself → verify reported `controller: Up … (after ~0s)`, cloudflared up — **no manual
+    intervention**. R6 closed at the installer.
+
+## 8. OQ answers + go/no-go for Peti (sajatfelhom.hu)
+
+**OQ-1 (offsite backup): recommendation stands — go live with local backups only**
+(`--acl-storages "local local-lvm"`, exactly the drilled canonical command). The PBS box is
+LAN-only until Headscale/WireGuard lands; the ACL/`--rescope-acl` path retrofits the offsite grant
+later without reinstalling. The guide documents this as the current standard. (Operator holds the
+pen — if offsite-first is wanted instead, install day waits on the VPN spike.)
+
+**OQ-2 (fresh-install controller version): answered with evidence** — lands 0.85.1, never
+self-updates (pre-floor code). Raising the floor does NOT fix a fresh box. Install-day step D.1b
+(restart agent → trigger update → verify 0.98.3) is mandatory and drill-proven. Structural fix =
+rebuild + re-vouch the golden with a ≥ 0.86.0 controller (operator follow-up, not a Peti blocker).
+
+**OQ-3 (pre-install facts):** the guide's Part B checklist is the collection instrument (PVE 9.x,
+single-node vs cluster → `--node`, ≥120 GiB local-lvm, free vmid from `pct list`+`qm list`, host
+cores/RAM for the cap, root SSH, outbound reach incl. the three exact URLs+expected codes, python3).
+
+**Go/no-go list:**
+
+| # | Item | State |
+|---|---|---|
+| 1 | Install path proven verbatim on a virgin box (twice + reinstall) | ✅ GO |
+| 2 | Uninstall residue-clean (v1.9.1) — safe on a shared box | ✅ GO (deploy/serve v1.9.1 BEFORE install day; drilled from the served URL at v1.9.0 + locally at v1.9.1) |
+| 3 | D.1b one-time controller update executed on install day | ⚠️ REQUIRED STEP (guide covers it; 2 minutes) |
+| 4 | Day-0 manifest vouches agent **0.62.0** + golden **0.85.1** | ✅ current; **but hub Day-0 manifest bump to agent 0.62.0 was already flagged as operator follow-up — CONFIRMED DONE (drill fetched 0.62.0)** |
+| 5 | Real CF tunnel token created + entered in Part A (drill used a fake) | ⚠️ operator does A.1 for sajatfelhom.hu; tunnel leg is demo-proven, not drill-proven |
+| 6 | `git.username`/`git.token` set in the customer config | ⚠️ REQUIRED (installer dies without it) |
+| 7 | Multi-node? If Peti's box is a cluster, add `--node` | per Part B answer |
+| 8 | Known non-blockers on the box: B2 (no pre-start hook on fresh boxes), B3 (401-until-agent-restart) | documented + workarounds in the guide |
+
+**Verdict: GO** — with items 3/5/6 executed on/before install day.
+
+## 9. Findings ledger (blockers + follow-ups, per the §9 gap policy)
+
+| ID | Where | Severity | What | Status |
+|---|---|---|---|---|
+| R1–R5 | installer | — | uninstall residue (config w/ secrets, shared-parent unit+script+mount, mkfs wrapper, hook snippet, dnsmasq snippets) | **FIXED v1.9.1**, residue-diff-proven |
+| R6 | installer | — | controller-bootstrap unit skipped: no post-provision reboot (agent hot-plugs the mount; unit condition is boot-time) | **FIXED v1.9.1** (reboot + bounded verify wait), D6b-proven |
+| B1 | agent/golden | LOW | structural fix for R6 belongs in the product too (path unit in the golden, or agent-side restart after back-half) — installer reboot is a correct but external crutch | RECORDED (agent/golden follow-up) |
+| B2 | agent | LOW | `guesthook.InstallSnippet` fails on a fresh box: `/var/lib/vz/snippets` doesn't exist and `install` won't create it → no pre-start self-heal hook, warn-only (`install: cannot create regular file … No such file or directory`, agent journal 14:58:40) | RECORDED (fix: mkdir -p SnippetDir; demo box unaffected — dir pre-exists) |
+| B3 | agent | MED | fresh-install local-API **401 until `systemctl restart felhom-agent`** (stale token map, pre-existing open issue) — NEW consequence found: it blocks the controller self-update agent-swap | RECORDED (workaround = guide D.1b step 1) |
+| B4 | hub | LOW | **no host-delete path** (UI or API) — drill host rows removed via direct SQL (spike precedent); customer-delete exists but leaves hosts/reports/guests/events orphaned | RECORDED (candidate small hub follow-up) |
+| B5 | golden/fleet | MED | golden bakes a pre-floor controller (0.85.1) → every fresh install needs D.1b manually | RECORDED (operator follow-up: rebuild golden ≥ 0.86.0 + re-vouch in the Day-0 manifest UI) |
+
+## 10. Cleanup confirmation
+
+- **Hub:** all `drill-1` rows deleted (customer_configs / hosts / host_reports / reports / guests /
+  events / notification_log / customer_notifications → remaining **0**); `demo-felhom` confirmed
+  intact. (The web customer-delete would have left the host/report rows — B4 — hence SQL, spike
+  precedent.)
+- **Drill VM:** powered off; disk reverted to the `virgin` snapshot and kept
+  (**virgin-snapshotted-and-stopped**, not removed — reusable for future drills). Footprint
+  ~6.7 GiB at `~/drill/` on the build server (ISO + qcow2 + evidence logs + keys, keys 0600);
+  the baked auto-install ISO variant was deleted.
+- Evidence archived off-box at `~/drill/{D0,D5,D5b}/` + `D2-install.log`, `D5-uninstall.log`,
+  `D5b-uninstall.log`, `D6-install.log`, `D6b-install.log` on the build server.
+- QEMU/`proxmox-auto-install-assistant` packages remain installed on the build server (useful for
+  re-drills; noted).