felhom.eu/REPORT.md

# felhom.eu — task reports

> **Overwrite** this file with a summary of the most recent task only (uniform with the other repos; not cumulative). The cumulative hub history lives in [hub/CHANGELOG.md](hub/CHANGELOG.md); the scripts history lives in [scripts/CHANGELOG.md](scripts/CHANGELOG.md).

---

# Go-live package — Day-0 install guide, clean-room drill, uninstall proof (2026-07-03)

## Baselines

| Repo | `main` before | After |
|---|---|---|
| felhom.eu | `996526273a` (script v1.9.0) | this commit — script **v1.9.1** + `documentation/runbooks/day0-install.md` (NEW) + `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md` (NEW) |
| felhom-agent | `84f3f7ddb1` v0.62.0 | untouched (read-only; 3 findings recorded, not patched) |

## What shipped

1. **`documentation/runbooks/day0-install.md`** — the complete first-time deployment guide
   (Parts A–F: hub onboarding / box-prereq checklist (OQ-3) / the canonical shared-box install
   command / post-install verification incl. the mandatory one-time controller update D.1b /
   uninstall + expected-remnants / troubleshooting from real drill hiccups). Header states the
   drill date/environment/versions. Every command was executed verbatim during the drill.
2. **Clean-room drill (Phase 0 + D0–D6)** — throwaway nested PVE 9.2.2 under QEMU/KVM on the build
   server (unattended install via `proxmox-auto-install-assistant`, virgin qcow2 snapshot, slirp
   NAT; the primary Phase-0 mechanism worked — no fallback needed). Full cycle proven: baseline
   manifest → hub onboarding of throwaway customer `drill-1` → install (real
   `--force-gitea-golden` + `--cores 2 --memory 4096` + `--acl-storages "local local-lvm"`,
   sha256-verified agent v0.62.0 + golden v0.85.1) → verify (selftest incl. `pve:pool-read`,
   hub reporting, dashboard 200 behind traefik) → app deploy via the exact UI endpoint
   (`bentopdf`, 200 at its own hostname) → uninstall → **residue diff** → reinstall
   (host-enroll REUSED; fresh leaf on an empty box). Evidence doc:
   `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md`.
3. **`scripts/felhom-host-install.sh` v1.9.1** — drill-justified fixes, each re-verified live:
   - header/version sync (v1.8.0 header vs 1.9.0 var) + keep-in-sync note;
   - uninstall removes the five drill-found residue items (agent **config with the live hub
     api_key**, shared-parent unit+script+`/mnt/felhom-drives`, guarded-mkfs wrapper, guest-hook
     snippet, dnsmasq snippets) — v1.9.1 re-drill diff vs the virgin baseline shows **zero
     Felhom-named leftovers**;
   - **post-provision guest reboot + bounded verify wait (R6)** — without it the golden's
     controller-bootstrap unit (boot-time `ConditionPathExists`) loses the race with the agent's
     hot-plugged bootstrap mount on slower hardware and the controller never deploys; v1.9.1
     reinstall brought the controller up with no manual intervention.
   Gates: `bash -n`, `--dry-run` previews, live re-drill of both paths.

## OQ answers (detail in the drill doc §8)

- **OQ-1:** go live **local-backups-only** (`--acl-storages "local local-lvm"`) — PBS is LAN-only
  until Headscale; retrofit later via `--rescope-acl`. Documented as the guide's standard.
- **OQ-2 (evidence-based):** a fresh install lands controller **0.85.1** (golden-baked, pre-floor)
  and **never self-updates** — raising the floor does not help a fresh box. The guide's D.1b
  (restart `felhom-agent` → trigger the settings-page update → verify) is the mandatory
  install-day step; drill-proven 0.85.1 → 0.98.3. Structural fix = rebuild+re-vouch the golden
  (operator follow-up).
- **OQ-3:** Part B of the guide is the customer-confirmed pre-install checklist.

## Go/no-go for Peti: **GO**

With three install-day items: serve v1.9.1 (this push), execute D.1b, create the real CF tunnel +
git credentials in Part A. Full list + blockers ledger: drill doc §8–§9.

## Recorded findings NOT fixed here (per the gap policy)

- **B1** (agent/golden, LOW): product-side fix for the bootstrap-race (golden path unit or
  agent-side restart) — the installer reboot is a correct external fix.
- **B2** (agent, LOW): `guesthook.InstallSnippet` fails on fresh boxes (`/var/lib/vz/snippets`
  missing; `install` can't create parents) → no pre-start self-heal hook, warn-only.
- **B3** (agent, MED, pre-existing): fresh-install local-API 401 until agent restart — new
  consequence found: blocks the controller self-update swap.
- **B4** (hub, LOW): no host-delete path (UI or API); drill rows cleaned via direct SQL.
- **B5** (fleet, MED): golden bakes a pre-floor controller → D.1b needed on every fresh install
  until the golden is rebuilt ≥ 0.86.0 and re-vouched.

## Cleanup

Hub `drill-1` rows fully deleted (all tables → 0; `demo-felhom` intact). Drill VM
**virgin-snapshotted and stopped** (kept at `~/drill/` on 192.168.0.180, ~6.7 GiB, for future
drills); evidence logs archived there. Nothing touched felhom-pve, guest 9201, or any real
customer/host record. No secrets in any committed file.