# DRILL — Golden rebuild 0.98.3: current-controller baseline (B5) + hot-plug path unit (B1) (2026-07-03)

**Class:** implementation + clean-room drill. Companion deliverables: `felhom-agent/configs/build-golden.sh`
**v2.0.0** (commit `ceca355`) and the golden archive **0.98.3** (published + operator-vouched).
Source findings: `DRILL-day0-cleanroom-2026-07-03.md` §9 **B5** (golden bakes a pre-floor controller →
mandatory manual D.1b on every fresh install) and **B1** (controller-bootstrap only fires at boot; the
hot-plugged config mount needed the installer's reboot crutch). Also closes the stale backlog note
`documentation/backlog/FOLLOWUP-golden-default-controller-tag.md`.

**Verdict (short):** both findings are FIXED in the product. A fresh Day-0 install now lands controller
**0.98.3** on first boot and self-manages from there (no D.1b), and the baked
`felhom-controller-bootstrap.path` unit starts the controller the moment the agent hot-plugs the
bootstrap mount — **no reboot needed** (the installer's v1.9.1 reboot is retained as a redundant belt).
Everything was proven clean-room on the drill VM BEFORE publish; the vouch is the only production change.

No secrets appear in this document. The bake's registry credential was passed via env from a 0600 file
on the build server; the script logs out + removes `/root/.docker/config.json` in the build guest before
archiving (`build-golden.sh` — verified present in the bake transcript path).

---

## 1. Environment & baselines

| Item | Value |
|---|---|
| Drill VM | the DRILL-day0-cleanroom nested PVE 9.2.2 on the build server (192.168.0.180), restored from the `virgin` qcow2 snapshot before the bake and again before each install scenario |
| VM launch quirk fixed | slirp DHCP pool moved with `dhcpstart=10.0.2.30` (+ explicit `hostfwd=…-10.0.2.15:22`) so nested guests can't grab the PVE host's static 10.0.2.15 — the prior drill's IP-collision quirk is fully avoided, no guest IP pinning needed |
| felhom-agent base | `c9f963d` → script commit `ceca355` (build-golden.sh v2.0.0; no Go change, no agent version bump) |
| felhom-controller | **v0.98.3** reconfirmed as the current released tag (CHANGELOG top entry; manifest fetch HTTP 200 with the read credential) — the tag baked |
| Hub manifest at start | agent **0.63.0** (the v0.63.0 REPORT's operator follow-up was already done) + golden **0.85.1** |
| Debian template | `debian-13-standard_13.1-2_amd64.tar.zst` (pveam-downloaded into the virgin VM — not present on a virgin box) |
| Throwaway hub customer | `drill-1` re-created store-identically (SQL insert matching `SaveCustomerConfig`: plaintext retrieval passphrase reused from the prior drill's 0600 file, `RandomHex(32)`-format api_key, demo customer's config_json with **fake** CF tunnel + CF API tokens, real Gitea read credential); probes: config 200 with passphrase / 401 wrong; deleted after (§7) |

## 2. Script changes (Phase 1) + red-proof

`build-golden.sh` v2.0.0 (see felhom-agent CHANGELOG for the full rationale):

1. **CONTROLLER_IMAGE (arg 6) mandatory** — no default; die-with-usage names the convention.
2. **`felhom-controller-bootstrap.path` baked + enabled** (`PathExists=/etc/felhom-bootstrap/bootstrap.json`,
   `WantedBy=multi-user.target`); the service (oneshot/RemainAfterExit + ConditionPathExists) unchanged.
3. `GOLDEN_SCRIPT_VERSION=2.0.0` + a `[golden]` provenance line (script version + baked tag) in every transcript.

Gates: `bash -n` clean (Windows + in-VM after CRLF strip). **B5 red-proof:** the no-arg invocation dies
with usage, exit 1, before any `pct` op:

```text
[golden] FATAL: CONTROLLER_IMAGE (argument 6) is required — pass the released controller tag explicitly.
Usage: build-golden.sh [VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] CONTROLLER_IMAGE
  e.g.: build-golden.sh 9100 local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst local-lvm local vmbr0 gitea.dooplex.hu/admin/felhom-controller:0.98.3
exit=1
```

(Re-run in the drill VM: rc=1 as well.)

## 3. Scenario A — bake integrity (PASS)

Bake on the virgin-restored drill VM as root@pam:
`build-golden-new.sh 9100 local:vztmpl/debian-13-standard_13.1-2_amd64.tar.zst local-lvm local vmbr0 gitea.dooplex.hu/admin/felhom-controller:0.98.3`
with `REGISTRY_USER/REGISTRY_TOKEN` from env (0600 file). Transcript (`bake-A.log`, archived at
`~/drill/` on the build server):

```text
[golden] build-golden.sh v2.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.98.3
…
[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …
…
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/var/lib/docker') in backup
INFO: including mount point mp1 ('/mnt/sys_drive') in backup
INFO: archive file size: 583MB
[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_07_03-18_01_21.tar.zst
```

- **mp0 + mp1 both INCLUDED** (the load-bearing vzdump guard confirmed; no exclusion lines).
- Build guest 9100 torn down (`pct destroy 9100 --purge`, volumes removed).
- Archive sha256 `b9a02ef1b6f02b9b58babc4c6aad9cf6c053ebdfba116c78c8e7830de757fd01` (612 002 238 bytes),
  identical after copy to the build server.
- **Deviation caught + corrected:** the script's opt-in Gitea publish block keys on the same
  `REGISTRY_*` vars the pull needs, so the bake ALSO published immediately — before Scenario C. The
  premature package was DELETED (HTTP 204, anon GET 404 verified) and re-published only after
  Scenario C passed (§6). Recorded as an observation (§8-O2), script not changed (no-"while-here" rule).

## 4. Scenario B — isolated hot-plug proof (PASS — the B1 STOP gate)

Scratch guest 9300 `pct restore`d from the NEW archive (the bring-up mechanism, minus the bootstrap
mount — the exact provision race window), booted 16:05:58 UTC.

**Before attach** (the R6 failure state, reproduced):

```text
felhom-controller-bootstrap.service  Active: inactive (dead)
  Condition: start condition unmet … ConditionPathExists=/etc/felhom-bootstrap/bootstrap.json was not met
  TriggeredBy: ● felhom-controller-bootstrap.path
felhom-controller-bootstrap.path     Active: active (waiting)
docker ps: (no containers) ; /etc/felhom-bootstrap: No such file or directory
```

**Attach to the RUNNING guest** — the back-half's exact op (host dir + 0600 bootstrap.json +
`chown -R 100000:100000` + `pct set 9300 -mp9 <dir>,mp=/etc/felhom-bootstrap,ro=1`) at 16:07:04.

**After (~1 s later, NO reboot — `uptime -s` unchanged at 16:05:58):**

```text
Jul 03 16:07:05 systemd[1]: Starting felhom-controller-bootstrap.service …
Jul 03 16:07:05 felhom-controller-bootstrap.sh[521]: [ctrl-bootstrap] deploying gitea.dooplex.hu/admin/felhom-controller:0.98.3 …
Jul 03 16:07:05 systemd[1]: Finished felhom-controller-bootstrap.service
felhom-controller-bootstrap.service  Active: active (exited) … status=0/SUCCESS
docker ps: felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.98.3 Up (healthy)
```

The systemd-semantics assumption holds empirically: a `.path` unit DOES see a hot-plugged LXC bind
mount and DOES start a Condition-gated oneshot service; `RemainAfterExit=yes` showed no re-trigger.
(Note: the hand-crafted minimal bootstrap.json's `"id"` didn't survive shell quoting for the
`--hostname` sed parse → `hostname=<unset>` fallback, by design fail-safe; Scenario C exercises the
real agent-rendered document.) Guest 9300 + its host dir removed after.

## 5. Scenario C — full Day-0 proof, LOCAL golden (PASS — the publish gate)

Virgin-restored VM; the new archive placed as the only local vzdump of GOLDEN_VMID 9100; installer
fetched from `https://felhom.eu/scripts/felhom-host-install.sh` (served **v1.9.1**); the guide's
canonical command WITHOUT `--force-gitea-golden`:

```text
./felhom-host-install.sh --customer-id drill-1 --vmid 120 --cores 2 --memory 4096 \
    --acl-storages "local local-lvm" --passphrase-file /root/.drill-pass
```

- **Resolution order proof:** `7/8 golden archive → [SKIP] using local golden: local:backup/vzdump-lxc-9100-2026_07_03-18_01_21.tar.zst`;
  final verdict `Day-0 provision SUCCESS — vmid=120 host_id=drill-1-20e7a3 … golden=local:backup/…`.
- **First boot = 0.98.3, NO D.1b performed:** in-guest
  `felhom-controller gitea.dooplex.hu/admin/felhom-controller:0.98.3 Up (healthy)`; hub `reports`
  row shows `drill-1 | 0.98.3`; host row `drill-1-20e7a3 | agent 0.63.0`.
- **Self-manage proof:** `POST /api/selfupdate/check` →
  `{"current_version":"0.98.3","latest_version":"0.98.3","update_available":false}` — the box is
  current and the floor mechanism owns it from here (0.98.3 ≥ the v0.86.0 floor-honoring code).
- Agent `--selftest` all-OK including `pool read pool "felhom", 1 member`.
- Base infra: traefik + filebrowser healthy; cloudflared restarting on the FAKE tunnel token
  (expected drill behavior, public leg demo-proven — same as the prior drill).
- **D4 smoke:** `bentopdf` deployed via the dashboard's own endpoint
  (`POST /api/stacks/bentopdf/deploy` with `{"values":{}}` — endpoint-proxy method, no browser
  bridge in this session) → `Up (healthy)`, and `pdf.drill.demo-felhom.eu` answers **200** through
  traefik in-guest.
- In-flow B1 note: the installer's v1.9.1 belt-reboot still ran (kept by design), so the container
  journal of the pre-reboot boot was not retrievable (volatile journal); the no-reboot claim rests
  on the §4 isolated proof.

## 6. Publish + vouch + Scenario D — published-artifact proof

- **Publish (after Scenario C):** `PUT …/api/packages/admin/generic/felhom-golden/0.98.3/golden.tar.zst`
  (exactly the installer's `step_golden` URL convention) → **HTTP 201**; round-trip GET sha256
  **matches** `b9a02ef1…fd01`. Publish creds = the build server's Gitea admin credential (out-of-band);
  version string = the baked controller tag per the 0.85.1 precedent.
- **Vouch (operator, password-gated UI):** hub Day-0 manifest → golden **0.98.3** / sha `b9a02ef1…fd01`
  (agent 0.63.0 was already vouched). Confirmed via `GET /api/v1/artifacts/drill-1`:
  `{"agent":{"version":"0.63.0",…},"golden":{"version":"0.98.3","sha256":"b9a02ef1…fd01"}}`.
- **Scenario D (PASS):** virgin-restored VM, NO local golden (dump dir + `pvesm list` both empty,
  verified), same canonical command + `--force-gitea-golden`:

```text
[INFO]   golden: none local — will fetch + verify from Gitea in step 7/8
[INFO]   manifest: agent v0.63.0 (sha b4a89c81c33d51ea…), golden v0.98.3
[OK]     verified sha256 b4a89c81c33d51ea… matches the hub manifest      (agent)
[INFO]   fetching golden v0.98.3 from Gitea → /var/lib/vz/dump/vzdump-lxc-9100-2026_07_03-18_31_49.tar.zst
[OK]     verified sha256 b9a02ef1b6f02b9b… matches the hub manifest      (golden)
[OK]     golden imported + verified: local:backup/vzdump-lxc-9100-2026_07_03-18_31_49.tar.zst
[OK]  Day-0 provision SUCCESS — vmid=120 host_id=drill-1-20e7a3 customer=drill-1
```

  First boot: `felhom-controller:0.98.3 Up (healthy)`; `selfupdate/check` →
  `{"current_version":"0.98.3","latest_version":"0.98.3","update_available":false}`. Host-enroll
  REUSED the Scenario C host id (mint-once-reuse held). This is the exact fetch+verify path a real
  customer box (Peti's) takes.

## 7. Cleanup confirmation

- Hub: all `drill-1` rows deleted AFTER Scenario D (customer_configs / hosts / host_reports /
  reports / guests / events / notification_log / customer_notifications — every count verified
  **0**); `demo-felhom` + `peti-felhom` intact. The vouch (golden 0.98.3 in the Day-0 manifest) is
  the one intended production change and stays.
- Drill VM: powered off; disk reverted to the `virgin` snapshot and kept (reusable). Local archives
  and staged files on the VM gone with the revert.
- Build server: `~/drill/golden-0.98.3.tar.zst` + `bake-A.log` retained as evidence;
  `.bake-cred.env` (0600) removed after the run.

## 8. Observations (recorded, not acted on)

- **O1 (SECURITY, operator follow-up):** the demo customer's `git.token` — the credential every
  customer box holds for artifact fetch — is a Gitea **admin** token with **package-WRITE** rights
  (the Scenario A bake proved it by successfully publishing with it). A customer-side compromise
  could overwrite published artifacts. The manifest-sha trust chain means installs would reject a
  tampered artifact, but the write capability should not exist on customer boxes at all → issue a
  scoped read-only Gitea account/token for customer configs and rotate the current one.
- **O2:** `build-golden.sh`'s publish block auto-fires when `REGISTRY_USER/REGISTRY_TOKEN` are set
  (they are also needed for the pull) — a validate-before-publish flow must either use a read-only
  cred (blocked on O1) or expect the early publish and delete it. Candidate cleanup: a separate
  `GOLDEN_PUBLISH=1` opt-in flag.
- **O3:** the installer's post-provision reboot (v1.9.1, R6 fix) is now a redundant belt — the path
  unit deploys the controller before the reboot even happens. Removal is a candidate cleanup for a
  future installer version; NOT done here (rule: keep the belt).
- **O4:** `journalctl -b -1` inside the LXC guest returns nothing (volatile journal) — pre-reboot
  boot evidence must be captured before a reboot, or the journal made persistent, in future drills.