felhom.eu/documentation/audits/R120-golden-rebake-2026-07-30.md

R120-golden-rebake-2026-07-30 — the golden is current again, and the class now has a gate that refuses

Run: R-120, CC on DooPlex, 2026-07-30. CLOSED on both halves. Golden 0.186.0 baked, published, vouched and proven on a real day-0; and the class now has a mechanism — hub v0.82.0's vouch-path gate, which refuses a golden the fleet has already outrun, deployed and proven live by re-attempting the original mistake.

1. Baselines, established separately

Thing	Value	Established from
controller on main	0.186.0	felhom-controller/CHANGELOG.md:1
controller the golden baked before this task	0.185.1	the golden's own record: drill/bake-0.185.1.log:1 (baking controller …:0.185.1) and :330 (GOLDEN_VERSION=0.185.1)
controller running, felhom-pve	0.186.0	pct exec 9201 -- docker ps
controller running, demo-hp	0.185.1	same
agent published + vouched	0.116.0, sha b47c5c4d…	registry GET + manifest read-back
golden vouched before	0.185.1, sha dba00f3e…	manifest read-back
felhom.eu HEAD at start	772956d	git rev-parse

R-120 confirmed at source, not inferred: build-golden.sh:345 derives GOLDEN_VERSION="${GOLDEN_VERSION:-${CONTROLLER_IMAGE##*:}}" — the golden version is the controller tag it bakes. So golden 0.185.1 ⇒ every fresh box installed controller 0.185.1, which predates R-114 + R-112 (shipped in 0.186.0), while a deployed box was already on 0.186.0. One release behind, and the missing release is the one that tells the truth about an absent backup target.

2. The bake procedure — it IS written down

Contrary to the risk the task flagged, the procedure exists and was followed, not reconstructed:

• documentation/runbooks/RUNBOOK-manual-build.md:101-115 §4 "Golden image (fresh Day-0 installs)" — the five steps: revert the drill VM to virgin, boot it, run build-golden.sh <controller VER>, teardown per GL-1, then vouch in the hub.
• felhom-agent/configs/build-golden.sh — the tool, with its usage in its own header ([VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] CONTROLLER_IMAGE), and it publishes to Gitea itself (:335-371) when GITEA_USER+GITEA_TOKEN are set. There is no separate publish-golden.sh — the bake script is the publisher.
• documentation/runbooks/publish-train-rules.md — rules 1–4. Rule 2 (save the floor field LAST, separate save) and rule 3 (MinAgent) were consulted; see §4 for how they applied.

One documentation-integrity finding, small but worth recording. RUNBOOK-manual-build.md:103-105 says to use the "RECORDED qemu launch line; do not reconstruct from memory" and points at RUNBOOK-publish-0.79-0.110-2026-07-10.md Phase C — but that line is itself labelled PASS (reconstructed — DEVIATION) at :29, with :77-79 stating the canonical one-liner was never saved. So the runbook forbids reconstruction while citing a reconstruction. The line at :29 is nonetheless correct — it booted first try here, and independently earlier today. A trap learned today and worth adding to it: adding any -device at launch shifts PCI slot assignment and renames the guest NIC, so vmbr0 loses its port and SSH never comes up while the console shows a healthy PVE login. Boot the recorded line unchanged and hot-plug extras over QMP.

3. The bake — controller 0.186.0

Run in the DooPlex nested drill VM (its documented purpose — target-selection.md names it a bake fixture, and this is a bake), reverted to virgin first, build-golden.sh v2.1.0 as a transient unit:

[golden] build-golden.sh v2.1.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.186.0
  docker OK (overlay2; data-root /var/lib/docker)
INFO: including mount point rootfs ('/')
INFO: including mount point mp0 ('/var/lib/docker')
INFO: including mount point mp1 ('/mnt/sys_drive')
INFO: archive file size: 618MB
[golden]   upload OK (HTTP 201)
GOLDEN_VERSION=0.186.0
GOLDEN_SHA256=b760ac6a33e70700f427b4207afeebc8694bd6bb9478973c30901d9f059bde17

Pass markers, counted on the saved log: overlay2 present, FATAL 0, exclusions 0, 3 mounts included, upload 201. Template: debian-13-standard_13.6-1_amd64.tar.zst (the pinned version in the script's default has rotted away; the current one was resolved with pveam available). Log retained out-of-VM at drill/bake-0.186.0.log (330 lines); token-leak grep against the literal token value on the saved log = 0.

GL-1 teardown: build guest 9100 pct destroy --purge; .gitea-token, the script and the in-VM log shred -u'd (0 residual); VM powered off; drill.qcow2 reverted to virgin (snapshot intact, ID 1, 2026-07-03) — the fixture is exactly as found.

4. The three publish/vouch observables, quoted as returned

(1) PUBLISHED — anonymous GET, i.e. what the installer actually does, sha over the bytes returned:
    HTTP 200  bytes=648930639
    b760ac6a33e70700f427b4207afeebc8694bd6bb9478973c30901d9f059bde17  /tmp/golden-0.186.0.tar.zst
    (identical to the bake's GOLDEN_SHA256)

(2) VOUCHED — manifest read BACK after the POST, never the 303:
    agent_version:  vouched=['0.116.0']      agent_sha256  = b47c5c4dab641ee5…
    golden_version: vouched=['0.186.0']      golden_sha256 = b760ac6a33e70700…
    min_agent = 0.113.0     min_controller_version = 0.156.0
    wrapper_sha256 = 104db0a4401f65bb…   (re-checked against configs/felhom-pbs-apply — NO drift)

(3) A CONSUMER RESOLVES IT — the hub telling a real fresh box what to install:
    [INFO] Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)

Publish-train rules as applied: rule 1 (manifest before floor) — satisfied trivially, no floor moved. Rule 2 (floor field LAST, separate save) — the floor was not touched at all; min_controller_version read back 0.156.0, unchanged, and the floor is a separate form (/configuration/global-floor) which was never posted. Rule 3 (MinAgent) — left at 0.113.0, because controller 0.186.0's CHANGELOG header declares MinAgent 0.113.0 (unchanged); raising it would hold demo-hp (agent 0.113.0) for no reason. The agent vouch was carried through untouched.

5. §2.1 — the fresh-box proof

Machine: demo-hp (Tier 0, the designated drill+build VM host). A real day-0, not the fixture — per the rule committed in Part 1, this is a path claim, so a fixture could not prove it. VM 9402 r120-drill, disks on a dir storage r120-images at /mnt/nvme-1tb root (never local-lvm), installed from the v1.25.0 ISO: self-register (appliance 14) → operator bind to sess-f → credentials delivered once → guest provisioned from the new golden.

R-120's fix, proven at the install path:

hub  → Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)
hub  → Event from sess-f: controller_started (info) — Controller elindult (0.186.0)
box  → felhom-agent --version                → felhom-agent 0.116.0
box  → pct exec 9201 -- docker ps            → gitea.dooplex.hu/admin/felhom-controller:0.186.0

A freshly installed box now runs 0.186.0. Before this task it ran 0.185.1. That is the defect closed.

Target drive enrolled through the real endpoints and detached (real hot-detach), so the box sits in R-120's exact state — and agent 0.116.0's R-116 fix is visible on it:

/disks felhom-backup row: state=disconnected role=system mount_path='' 
                          guest_path='/mnt/felhom-drives/cel' backup_target=True
pvesm: felhom-backup dir inactive 0 0 0

The capture — R-120's symptom, inverted

Operator-relayed claim code, then GET /api/storage/backup-target with the target still detached, on controller 0.186.0 from the new golden. Verbatim:

{"data":{"degraded":true,"known":true,"label":"",
  "message":"A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod, a teljes rendszermentés nem készül el.",
  "target":"felhom-backup"},"ok":true}

• ✅ the TargetAbsent copy — the backup drive is unreachable; until you reconnect it the full system backup will not be made. Reconnect that drive.
• ✅ offer_path is absent from the payload entirely — no drive is offered as a remedy.
• ✅ not the generic "A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer".

For contrast, the same read on the OLD 0.185.1 golden one day earlier (audits/R116-v0116-2026-07-30.md §6a caveat) returned the generic system-disk message plus offer_label: "Adat meghajto" and offer_path: "/mnt/felhom-drives/adat". The false message and the wrong offer are both gone. That is R-120 closed at the customer-visible layer.

The claim gate had to be cleared by an operator-relayed code — the endpoint first returned {"ok":false,"error":"dashboard not yet claimed"}. The claim gate runs before auth (controller/internal/web/auth.go:52-67) and exempts only /claim, /claim/request-new-code, /api/health, /static/ (claim.go:221-228) — no Bearer escape, and regen-password (hub/internal/web/configs.go:965-980) regenerates the retrieval passphrase, not the dashboard password. So an operator-relayed code is the only route. This is R-119's fourth instance.

6. Part 3 — the ruling, and the recommendation with its wiring

Recommendation: the gate, and it must REFUSE, not warn. The record is the argument: this gap has opened three times (R-111, R-115, R-120) and the first two were closed by re-baking and remembering. Remembering has now failed twice. And R-29 is the standing proof that a check nobody runs is worse than none, because it reads as coverage — hostinstall_gates.py sat red and uninvoked across three version bumps and hub_confirm_gate.py has never run at all. So the distinguishing property is not does a check exist but does it block.

Where it wires — established, not proposed loosely. hub/internal/web/configs.go:1116 handleSetArtifacts is the vouch path: the only UI writer of store.SetArtifactManifest (:1142), reached from server.go:484. It already normalises the versions (:1121-1125) and resolves the SHAs (:1128-1131), so a refusal belongs immediately before :1142. Nothing can vouch a golden without passing through it — which is precisely what makes it a mechanism rather than a fourth orphan in scripts/.

The comparison signal is already local. The hub stores controller_version per guest (hub/internal/store/store.go:294), populated from reports. A gate comparing the submitted golden_version against the newest controller version any box has reported would have caught this exact case: felhom-pve was reporting 0.186.0 while the manifest vouched a 0.185.1 golden. No outbound registry call, no new credential.

Its honest blind spot, stated up front: if no box has ever run the newer controller, the hub cannot know it exists, so a golden baked behind an unreleased-to-the-fleet controller would still pass. That is a real limit — and it does not touch the failure that actually happened three times, which is a golden left behind a controller already running in the field.

The test must go through the production wiring, not an injected seam: drive handleSetArtifacts over httptest with a guest row reporting 0.186.0 and a submitted golden of 0.185.1, then assert both that the response is the refusal and that the manifest was not written. Three shipped defects in this project were fully green with the seam disconnected; a gate that can be inert is the thing this gate exists to prevent.

RULING: REFUSE — approved by the operator, 2026-07-30. Built, deployed, proven live.

Shipped as hub v0.82.0: the gate in handleSetArtifacts immediately before the only write; store.NewestReportedControllerVersion() over reports.controller_version (the column SaveReport denormalises, store.go:903), semver-compared in Go; an operator-legible refusal flash naming the remedy; fail-open in exactly two deliberate cases — an empty golden field, and an unknown fleet version.

A near-miss worth keeping. The first draft read guests.controller_version — a column that exists in the schema (store.go:294) and that nothing writes. That gate would always have seen "" and failed open: inert, i.e. this gate's own failure shape, and one grep from shipping. Caught by looking for a writer before trusting the column.

Tests: 4, through the production handler over httptest, never an injected seam. The refusal asserts both the flash and that the manifest was not written — a gate that redirects and saves anyway reads as enforcement while providing none. Plus the allow case, both fail-open cases, and the semver-ordering case (0.99.0 vs 0.186.0, a pair this fleet has shipped). Red-proof: deleting the block makes the stale golden vouchable and both refusal assertions fail.

PROVEN LIVE on the deployed hub — the original mistake, re-attempted and blocked

Hub 0.82.0 built, pushed, manifest pinned, ArgoCD hard-refreshed + synced → Synced / Healthy, pod on felhom-hub:0.82.0. Then the exact R-120 mistake was re-attempted — vouch the stale 0.185.1 golden:

POST /configuration/artifacts  →  HTTP 303  …/configuration?flash=golden_behind_fleet
hub log: [WARN] artifact vouch REFUSED: golden 0.185.1 is older than the newest controller the fleet
         reports (0.186.0) — a fresh install would land on stale application code (R-120)
manifest read back afterwards: golden still 0.186.0, sha still b760ac6a… — the refused POST did NOT write

Refused and unwritten, on the live hub against the real fleet signal rather than a unit fixture.

R-29's audit list now records it (ROADMAP.md) as the first enforced gate beside its three orphans, so the contrast is kept rather than lost. The orphans are unchanged — this proves the pattern is available, not that the backlog moved.

7. Teardown

layer	item	disposition
1 — machine	DooPlex bake fixture: build guest 9100 purged, drill.qcow2 reverted to virgin	DONE
1 — machine	demo-hp VM 9402 r120-drill + all volumes	DESTROYED qm destroy 9402 --purge; qm list shows only drill-r50
2 — host	r120-images dir storage at /mnt/nvme-1tb	REMOVED; felhom-backup available 928787076 KiB after == before, local-lvm 38.84 % unchanged — space returned
3 — hub	customer sess-f, host sess-f-2670b5, appliance 14	GATE-BLOCKED — command recorded below. Cascade attempted, correctly refused HTTP 409 (host still ONLINE, customer_delete.go:220-228). Deletable once it ages ONLINE→DOWN, exactly as sess-e was this run

pvesm status on demo-hp before this run, for the layer-2 comparison at teardown:

felhom-backup  dir     active  983379700  4566012  928787076  0.46%
local          dir     active   40516856 14980024   23446440 36.97%
local-lvm  lvmthin     active   56545280 21962186   34583093 38.84%   ← the fence figure

Teardown commands, recorded now:

ssh demo-hp 'qm stop 9402; qm destroy 9402 --purge; pvesm remove r120-images; pvesm status'
POST /configs/sess-f/delete  ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-f expect_hosts=1

Previous session's layer 3 is now DISCHARGED. sess-e was gate-blocked on ONLINE last run with the command recorded; it aged to STALE and the cascade completed this run — /customers/sess-e → 404, 0 hosts, and the hub logged customer DELETE cascade COMPLETE for sess-e (journal #7) — full teardown including residue purged (reports=4 app_telemetry=4 … appliance_registrations=1). That is the §13 layer-3 discipline working as intended: recorded, then executed.

Fences held: drill-r50 untouched · guest 9201 on both demo boxes untouched · neither demo box re-targeted · nothing on local-lvm · Peti untouched · no existing box received the new golden · felhom-agent/felhom-controller source untouched · the hub DB copy taken for the break-glass credential was shred -u'd immediately.