docs: golden 0.98.3 live — D.1b retired, drill B1/B5 fixed, backlog note resolved + evidence doc

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PSK5g6qYLknKj8u3QAFEr6

REPORT.md

@@ -4,75 +4,75 @@
 
 ---
 
-# Go-live package — Day-0 install guide, clean-room drill, uninstall proof (2026-07-03)
+# REPORT — Golden rebuild 0.98.3 (drill B5 + B1) — docs half (2026-07-03)
+
+Implementation half in `felhom-agent/REPORT.md` (build-golden.sh v2.0.0 @ `ceca355`). Full drill
+evidence: **`documentation/audits/DRILL-golden-098-2026-07-03.md`** (A–D transcripts, unit states,
+resolution-order + fetch/sha proofs, cleanup, observations). This repo's changes are docs-only.
 
 ## Baselines
 
-| Repo | `main` before | After |
-|---|---|---|
-| felhom.eu | `996526273a` (script v1.9.0) | this commit — script **v1.9.1** + `documentation/runbooks/day0-install.md` (NEW) + `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md` (NEW) |
-| felhom-agent | `84f3f7ddb1` v0.62.0 | untouched (read-only; 3 findings recorded, not patched) |
+| Repo | Base → head |
+|---|---|
+| felhom.eu | `2e33a8b` → this push (docs) |
+| felhom-agent | `c9f963d` → `ceca355` (script + CHANGELOG) |
+| felhom-controller | untouched; **0.98.3** reconfirmed current + pullable, and is the baked tag |
 
-## What shipped
+## What shipped (system-level)
 
-1. **`documentation/runbooks/day0-install.md`** — the complete first-time deployment guide
-   (Parts A–F: hub onboarding / box-prereq checklist (OQ-3) / the canonical shared-box install
-   command / post-install verification incl. the mandatory one-time controller update D.1b /
-   uninstall + expected-remnants / troubleshooting from real drill hiccups). Header states the
-   drill date/environment/versions. Every command was executed verbatim during the drill.
-2. **Clean-room drill (Phase 0 + D0–D6)** — throwaway nested PVE 9.2.2 under QEMU/KVM on the build
-   server (unattended install via `proxmox-auto-install-assistant`, virgin qcow2 snapshot, slirp
-   NAT; the primary Phase-0 mechanism worked — no fallback needed). Full cycle proven: baseline
-   manifest → hub onboarding of throwaway customer `drill-1` → install (real
-   `--force-gitea-golden` + `--cores 2 --memory 4096` + `--acl-storages "local local-lvm"`,
-   sha256-verified agent v0.62.0 + golden v0.85.1) → verify (selftest incl. `pve:pool-read`,
-   hub reporting, dashboard 200 behind traefik) → app deploy via the exact UI endpoint
-   (`bentopdf`, 200 at its own hostname) → uninstall → **residue diff** → reinstall
-   (host-enroll REUSED; fresh leaf on an empty box). Evidence doc:
-   `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md`.
-3. **`scripts/felhom-host-install.sh` v1.9.1** — drill-justified fixes, each re-verified live:
-   - header/version sync (v1.8.0 header vs 1.9.0 var) + keep-in-sync note;
-   - uninstall removes the five drill-found residue items (agent **config with the live hub
-     api_key**, shared-parent unit+script+`/mnt/felhom-drives`, guarded-mkfs wrapper, guest-hook
-     snippet, dnsmasq snippets) — v1.9.1 re-drill diff vs the virgin baseline shows **zero
-     Felhom-named leftovers**;
-   - **post-provision guest reboot + bounded verify wait (R6)** — without it the golden's
-     controller-bootstrap unit (boot-time `ConditionPathExists`) loses the race with the agent's
-     hot-plugged bootstrap mount on slower hardware and the controller never deploys; v1.9.1
-     reinstall brought the controller up with no manual intervention.
-   Gates: `bash -n`, `--dry-run` previews, live re-drill of both paths.
+- **Golden 0.98.3** — bakes controller 0.98.3 + the `felhom-controller-bootstrap.path` unit;
+  published to Gitea (`felhom-golden/0.98.3/golden.tar.zst`, HTTP 201, round-trip sha
+  `b9a02ef1b6f02b9b58babc4c6aad9cf6c053ebdfba116c78c8e7830de757fd01`) and **operator-vouched** in
+  the Day-0 manifest (now: agent 0.63.0 + golden 0.98.3 — verified via `/api/v1/artifacts`).
+- **B5 dead:** clean-room Day-0 install (Scenario C, local golden; Scenario D, vouched Gitea
+  fetch + sha verify) lands controller **0.98.3 on first boot**; `selfupdate/check` reports
+  up-to-date → the box self-manages; agent selftest clean; bentopdf deploys + answers 200.
+- **B1 dead:** isolated proof — service condition-failed + path `active (waiting)` on a mount-less
+  boot; `pct set -mp9 …` against the RUNNING guest started the controller in ~1 s, no reboot
+  (`uptime -s` unchanged). Installer v1.9.1 reboot retained as belt (removal = recorded cleanup).
 
-## OQ answers (detail in the drill doc §8)
+## Docs changed (this repo)
 
-- **OQ-1:** go live **local-backups-only** (`--acl-storages "local local-lvm"`) — PBS is LAN-only
-  until Headscale; retrofit later via `--rescope-acl`. Documented as the guide's standard.
-- **OQ-2 (evidence-based):** a fresh install lands controller **0.85.1** (golden-baked, pre-floor)
-  and **never self-updates** — raising the floor does not help a fresh box. The guide's D.1b
-  (restart `felhom-agent` → trigger the settings-page update → verify) is the mandatory
-  install-day step; drill-proven 0.85.1 → 0.98.3. Structural fix = rebuild+re-vouch the golden
-  (operator follow-up).
-- **OQ-3:** Part B of the guide is the customer-confirmed pre-install checklist.
+- `documentation/runbooks/day0-install.md` — **D.1b retired** to a one-line `selfupdate/check`
+  verification; old procedure → Part F troubleshooting row keyed on "golden older than 0.86.0";
+  header versions line (script v1.9.1 / agent v0.63.0 / golden v0.98.3); A.3 drilled-known-good
+  pair + vouch-≥0.98.3 note; A.4 floor text rewritten + raise-floor recommendation.
+- `documentation/audits/DRILL-day0-cleanroom-2026-07-03.md` — ledger **B1, B5 → FIXED**; R6
+  belt-note.
+- `documentation/backlog/FOLLOWUP-golden-default-controller-tag.md` + `backlog/README.md` —
+  **RESOLVED** (M18/M19 convention: file kept + annotated, README entry marked FIXED; the note's
+  `:0.43.0` numbers were history — the live default had already rotted to `:0.85.1`, which is the
+  form of the problem the mandatory arg kills).
+- NEW `documentation/audits/DRILL-golden-098-2026-07-03.md` — the evidence doc.
 
-## Go/no-go for Peti: **GO**
+## Key proofs (short form; transcripts in the evidence doc)
 
-With three install-day items: serve v1.9.1 (this push), execute D.1b, create the real CF tunnel +
-git credentials in Part A. Full list + blockers ledger: drill doc §8–§9.
+| Gate | Evidence |
+|---|---|
+| B5 red-proof | no-arg `build-golden.sh` dies with usage, exit 1, before any `pct` op (run on Windows + in the drill VM) |
+| Scenario A | `[golden] build-golden.sh v2.0.0 — baking controller …0.98.3`; vzdump log: mp0 AND mp1 **included**; guest 9100 destroyed |
+| Scenario B | before: `ConditionPathExists … not met` + path `active (waiting)`; after mp9 hot-plug: service SUCCESS @ +1 s, container `Up (healthy)` 0.98.3, boot time unchanged |
+| Scenario C | `[SKIP] using local golden: …18_01_21.tar.zst` (resolution order); first boot 0.98.3; `update_available:false`; hub rows agent 0.63.0 / controller 0.98.3; bentopdf 200 |
+| Publish | pre-delete 404 → PUT **201** → round-trip GET sha **matches** |
+| Scenario D | `fetching golden v0.98.3 from Gitea` → `verified sha256 b9a02ef1… matches the hub manifest` → SUCCESS; first boot 0.98.3; up-to-date |
+| Cleanup | all 8 drill-1 hub tables at count **0**, demo-felhom + peti-felhom intact; drill VM reverted to `virgin` (kept); bake cred file removed |
 
-## Recorded findings NOT fixed here (per the gap policy)
+Secrets: registry read-cred via 0600 env file only; the bake script's in-guest
+`docker logout + rm /root/.docker/config.json` line is present and ran before archiving; publish
+used the build server's out-of-band Gitea admin credential; nothing committed.
 
-- **B1** (agent/golden, LOW): product-side fix for the bootstrap-race (golden path unit or
-  agent-side restart) — the installer reboot is a correct external fix.
-- **B2** (agent, LOW): `guesthook.InstallSnippet` fails on fresh boxes (`/var/lib/vz/snippets`
-  missing; `install` can't create parents) → no pre-start self-heal hook, warn-only.
-- **B3** (agent, MED, pre-existing): fresh-install local-API 401 until agent restart — new
-  consequence found: blocks the controller self-update swap.
-- **B4** (hub, LOW): no host-delete path (UI or API); drill rows cleaned via direct SQL.
-- **B5** (fleet, MED): golden bakes a pre-floor controller → D.1b needed on every fresh install
-  until the golden is rebuilt ≥ 0.86.0 and re-vouched.
+## Observations / operator follow-ups
 
-## Cleanup
-
-Hub `drill-1` rows fully deleted (all tables → 0; `demo-felhom` intact). Drill VM
-**virgin-snapshotted and stopped** (kept at `~/drill/` on 192.168.0.180, ~6.7 GiB, for future
-drills); evidence logs archived there. Nothing touched felhom-pve, guest 9201, or any real
-customer/host record. No secrets in any committed file.
+1. **SECURITY:** the customer-config `git.token` (held by every customer box) is a Gitea **admin**
+   token with **package-WRITE** — the bake proved it by successfully publishing with it. The
+   manifest-sha chain protects installs from tampered artifacts, but the capability shouldn't exist
+   customer-side: issue a scoped read-only account/token + rotate.
+2. `build-golden.sh`'s publish block auto-fires whenever `REGISTRY_*` is set (needed for the pull
+   too) → it published BEFORE Scenario C; deleted (204) and re-published after the gate. Candidate
+   cleanup: a `GOLDEN_PUBLISH=1` opt-in flag.
+3. The installer's post-provision reboot is now redundant (path unit wins first) — candidate
+   removal in a future installer version; kept per the task rules.
+4. Recommended: raise the global controller floor to 0.98.3 (UI, 1 min) for drift protection.
+5. Drill-environment note: launching the drill VM with `dhcpstart=10.0.2.30` (+ explicit
+   `hostfwd…-10.0.2.15:22`) eliminates the prior drill's slirp DHCP/IP-collision quirk — worth
+   using in every future drill.