felhom.eu/documentation/audits/R120-golden-rebake-2026-07-30.md

R120-golden-rebake-2026-07-30 — the golden is current again; the class still needs a mechanism

Run: R-120, CC on DooPlex, 2026-07-30. Golden 0.186.0 baked, published, vouched, and proven on a fresh box. The mechanism half (Part 3) awaits an operator ruling — see §6.

1. Baselines, established separately

Thing	Value	Established from
controller on main	0.186.0	felhom-controller/CHANGELOG.md:1
controller the golden baked before this task	0.185.1	the golden's own record: drill/bake-0.185.1.log:1 (baking controller …:0.185.1) and :330 (GOLDEN_VERSION=0.185.1)
controller running, felhom-pve	0.186.0	pct exec 9201 -- docker ps
controller running, demo-hp	0.185.1	same
agent published + vouched	0.116.0, sha b47c5c4d…	registry GET + manifest read-back
golden vouched before	0.185.1, sha dba00f3e…	manifest read-back
felhom.eu HEAD at start	772956d	git rev-parse

R-120 confirmed at source, not inferred: build-golden.sh:345 derives GOLDEN_VERSION="${GOLDEN_VERSION:-${CONTROLLER_IMAGE##*:}}" — the golden version is the controller tag it bakes. So golden 0.185.1 ⇒ every fresh box installed controller 0.185.1, which predates R-114 + R-112 (shipped in 0.186.0), while a deployed box was already on 0.186.0. One release behind, and the missing release is the one that tells the truth about an absent backup target.

2. The bake procedure — it IS written down

Contrary to the risk the task flagged, the procedure exists and was followed, not reconstructed:

• documentation/runbooks/RUNBOOK-manual-build.md:101-115 §4 "Golden image (fresh Day-0 installs)" — the five steps: revert the drill VM to virgin, boot it, run build-golden.sh <controller VER>, teardown per GL-1, then vouch in the hub.
• felhom-agent/configs/build-golden.sh — the tool, with its usage in its own header ([VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] CONTROLLER_IMAGE), and it publishes to Gitea itself (:335-371) when GITEA_USER+GITEA_TOKEN are set. There is no separate publish-golden.sh — the bake script is the publisher.
• documentation/runbooks/publish-train-rules.md — rules 1–4. Rule 2 (save the floor field LAST, separate save) and rule 3 (MinAgent) were consulted; see §4 for how they applied.

One documentation-integrity finding, small but worth recording. RUNBOOK-manual-build.md:103-105 says to use the "RECORDED qemu launch line; do not reconstruct from memory" and points at RUNBOOK-publish-0.79-0.110-2026-07-10.md Phase C — but that line is itself labelled PASS (reconstructed — DEVIATION) at :29, with :77-79 stating the canonical one-liner was never saved. So the runbook forbids reconstruction while citing a reconstruction. The line at :29 is nonetheless correct — it booted first try here, and independently earlier today. A trap learned today and worth adding to it: adding any -device at launch shifts PCI slot assignment and renames the guest NIC, so vmbr0 loses its port and SSH never comes up while the console shows a healthy PVE login. Boot the recorded line unchanged and hot-plug extras over QMP.

3. The bake — controller 0.186.0

Run in the DooPlex nested drill VM (its documented purpose — target-selection.md names it a bake fixture, and this is a bake), reverted to virgin first, build-golden.sh v2.1.0 as a transient unit:

[golden] build-golden.sh v2.1.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.186.0
  docker OK (overlay2; data-root /var/lib/docker)
INFO: including mount point rootfs ('/')
INFO: including mount point mp0 ('/var/lib/docker')
INFO: including mount point mp1 ('/mnt/sys_drive')
INFO: archive file size: 618MB
[golden]   upload OK (HTTP 201)
GOLDEN_VERSION=0.186.0
GOLDEN_SHA256=b760ac6a33e70700f427b4207afeebc8694bd6bb9478973c30901d9f059bde17

Pass markers, counted on the saved log: overlay2 present, FATAL 0, exclusions 0, 3 mounts included, upload 201. Template: debian-13-standard_13.6-1_amd64.tar.zst (the pinned version in the script's default has rotted away; the current one was resolved with pveam available). Log retained out-of-VM at drill/bake-0.186.0.log (330 lines); token-leak grep against the literal token value on the saved log = 0.

GL-1 teardown: build guest 9100 pct destroy --purge; .gitea-token, the script and the in-VM log shred -u'd (0 residual); VM powered off; drill.qcow2 reverted to virgin (snapshot intact, ID 1, 2026-07-03) — the fixture is exactly as found.

4. The three publish/vouch observables, quoted as returned

(1) PUBLISHED — anonymous GET, i.e. what the installer actually does, sha over the bytes returned:
    HTTP 200  bytes=648930639
    b760ac6a33e70700f427b4207afeebc8694bd6bb9478973c30901d9f059bde17  /tmp/golden-0.186.0.tar.zst
    (identical to the bake's GOLDEN_SHA256)

(2) VOUCHED — manifest read BACK after the POST, never the 303:
    agent_version:  vouched=['0.116.0']      agent_sha256  = b47c5c4dab641ee5…
    golden_version: vouched=['0.186.0']      golden_sha256 = b760ac6a33e70700…
    min_agent = 0.113.0     min_controller_version = 0.156.0
    wrapper_sha256 = 104db0a4401f65bb…   (re-checked against configs/felhom-pbs-apply — NO drift)

(3) A CONSUMER RESOLVES IT — the hub telling a real fresh box what to install:
    [INFO] Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)

Publish-train rules as applied: rule 1 (manifest before floor) — satisfied trivially, no floor moved. Rule 2 (floor field LAST, separate save) — the floor was not touched at all; min_controller_version read back 0.156.0, unchanged, and the floor is a separate form (/configuration/global-floor) which was never posted. Rule 3 (MinAgent) — left at 0.113.0, because controller 0.186.0's CHANGELOG header declares MinAgent 0.113.0 (unchanged); raising it would hold demo-hp (agent 0.113.0) for no reason. The agent vouch was carried through untouched.

5. §2.1 — the fresh-box proof

Machine: demo-hp (Tier 0, the designated drill+build VM host). A real day-0, not the fixture — per the rule committed in Part 1, this is a path claim, so a fixture could not prove it. VM 9402 r120-drill, disks on a dir storage r120-images at /mnt/nvme-1tb root (never local-lvm), installed from the v1.25.0 ISO: self-register (appliance 14) → operator bind to sess-f → credentials delivered once → guest provisioned from the new golden.

R-120's fix, proven at the install path:

hub  → Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)
hub  → Event from sess-f: controller_started (info) — Controller elindult (0.186.0)
box  → felhom-agent --version                → felhom-agent 0.116.0
box  → pct exec 9201 -- docker ps            → gitea.dooplex.hu/admin/felhom-controller:0.186.0

A freshly installed box now runs 0.186.0. Before this task it ran 0.185.1. That is the defect closed.

Target drive enrolled through the real endpoints and detached (real hot-detach), so the box sits in R-120's exact state — and agent 0.116.0's R-116 fix is visible on it:

/disks felhom-backup row: state=disconnected role=system mount_path='' 
                          guest_path='/mnt/felhom-drives/cel' backup_target=True
pvesm: felhom-backup dir inactive 0 0 0

The endpoint proof is NOT yet captured, and is not claimed: GET /api/storage/backup-target returns {"ok":false,"error":"dashboard not yet claimed"}. The claim gate runs before auth (controller/internal/web/auth.go:52-67) and exempts only /claim, /claim/request-new-code, /api/health, /static/ (claim.go:221-228) — no Bearer escape, and regen-password (hub/internal/web/configs.go:965-980) regenerates the retrieval passphrase, not the dashboard password. So an operator-relayed code is the only route. This is R-119's fourth instance.

6. Part 3 — the ruling, and the recommendation with its wiring

Recommendation: the gate, and it must REFUSE, not warn. The record is the argument: this gap has opened three times (R-111, R-115, R-120) and the first two were closed by re-baking and remembering. Remembering has now failed twice. And R-29 is the standing proof that a check nobody runs is worse than none, because it reads as coverage — hostinstall_gates.py sat red and uninvoked across three version bumps and hub_confirm_gate.py has never run at all. So the distinguishing property is not does a check exist but does it block.

Where it wires — established, not proposed loosely. hub/internal/web/configs.go:1116 handleSetArtifacts is the vouch path: the only UI writer of store.SetArtifactManifest (:1142), reached from server.go:484. It already normalises the versions (:1121-1125) and resolves the SHAs (:1128-1131), so a refusal belongs immediately before :1142. Nothing can vouch a golden without passing through it — which is precisely what makes it a mechanism rather than a fourth orphan in scripts/.

The comparison signal is already local. The hub stores controller_version per guest (hub/internal/store/store.go:294), populated from reports. A gate comparing the submitted golden_version against the newest controller version any box has reported would have caught this exact case: felhom-pve was reporting 0.186.0 while the manifest vouched a 0.185.1 golden. No outbound registry call, no new credential.

Its honest blind spot, stated up front: if no box has ever run the newer controller, the hub cannot know it exists, so a golden baked behind an unreleased-to-the-fleet controller would still pass. That is a real limit — and it does not touch the failure that actually happened three times, which is a golden left behind a controller already running in the field.

The test must go through the production wiring, not an injected seam: drive handleSetArtifacts over httptest with a guest row reporting 0.186.0 and a submitted golden of 0.185.1, then assert both that the response is the refusal and that the manifest was not written. Three shipped defects in this project were fully green with the seam disconnected; a gate that can be inert is the thing this gate exists to prevent.

This is the operator's call because a refusing gate can block a release. Not implemented pending the ruling. If it is declined, the weaker form goes in and the reason is recorded here rather than absorbed silently.

If the gate is deferred, R-120 does not close — it is re-opened, or a successor is filed naming the class: "a published artifact and its baked copy drift and nothing checks". Three instances closed individually is how a class survives.

7. Teardown

layer	item	disposition
1 — machine	DooPlex bake fixture: build guest 9100 purged, drill.qcow2 reverted to virgin	DONE
1 — machine	demo-hp VM 9402 r120-drill	RETAINED pending the claim code (§5's endpoint read)
2 — host	r120-images dir storage at /mnt/nvme-1tb	RETAINED with 9402
3 — hub	customer sess-f, host sess-f-2670b5, appliance 14	RETAINED with 9402

pvesm status on demo-hp before this run, for the layer-2 comparison at teardown:

felhom-backup  dir     active  983379700  4566012  928787076  0.46%
local          dir     active   40516856 14980024   23446440 36.97%
local-lvm  lvmthin     active   56545280 21962186   34583093 38.84%   ← the fence figure

Teardown commands, recorded now:

ssh demo-hp 'qm stop 9402; qm destroy 9402 --purge; pvesm remove r120-images; pvesm status'
POST /configs/sess-f/delete  ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-f expect_hosts=1

Previous session's layer 3 is now DISCHARGED. sess-e was gate-blocked on ONLINE last run with the command recorded; it aged to STALE and the cascade completed this run — /customers/sess-e → 404, 0 hosts, and the hub logged customer DELETE cascade COMPLETE for sess-e (journal #7) — full teardown including residue purged (reports=4 app_telemetry=4 … appliance_registrations=1). That is the §13 layer-3 discipline working as intended: recorded, then executed.

Fences held: drill-r50 untouched · guest 9201 on both demo boxes untouched · neither demo box re-targeted · nothing on local-lvm · Peti untouched · no existing box received the new golden · felhom-agent/felhom-controller source untouched · the hub DB copy taken for the break-glass credential was shred -u'd immediately.