docs(R-120): golden rebaked to 0.186.0, published, vouched, proven on a real day-0
The golden baked controller 0.185.1 -- confirmed from the golden's OWN record (drill/bake-0.185.1.log:1 and :330) and from build-golden.sh:345, which derives GOLDEN_VERSION from the controller tag. 0.185.1 predates R-114 + R-112, so every freshly installed box told a customer whose backup drive had fallen out that the backup was on the same disk as the system (false) and offered a different drive as the remedy. Baked golden 0.186.0 from main's controller in the DooPlex bake fixture: overlay2 OK, 3 mounts included, FATAL 0, exclusions 0, 618 MB, upload HTTP 201, GOLDEN_SHA256 b760ac6a33e70700..., token-leak grep 0, GL-1 teardown with drill.qcow2 back to virgin. Three observables, quoted as returned: PUBLISHED (anonymous GET -- what the installer does -- 200 / 648930639 bytes / sha identical to the bake); VOUCHED (manifest read BACK, not the 303); RESOLVED BY A CONSUMER (Artifact manifest served for customer sess-f, golden=0.186.0). Floor NOT touched per publish-train rule 2 -- it is a separate form and min_controller_version still reads 0.156.0. MinAgent left 0.113.0 because 0.186.0 declares it unchanged. Proven on a REAL day-0 on demo-hp, not the fixture, per the rule committed in Part 1: VM 9402 from the v1.25.0 ISO -> Controller elindult (0.186.0), box confirms felhom-controller:0.186.0 + agent 0.116.0. A fresh box now runs 0.186.0 where it ran 0.185.1. The procedure was NOT unwritten: RUNBOOK-manual-build.md:101-115 documents it and build-golden.sh carries its own usage and publishes to Gitea itself. One documentation-integrity finding: that runbook says to use the RECORDED qemu line and not reconstruct, while the line it cites is itself labelled reconstructed, the canonical one never having been saved. NOT done and not claimed: the TargetAbsent/empty-offer_path endpoint capture (the claim gate runs before auth with no Bearer escape -- R-119's fourth instance), and the Part 3 mechanism, which awaits the operator ruling. Recommendation and exact wiring recorded in the audit rather than built. VM 9402 + r120-images + customer sess-f retained pending that read, with teardown commands recorded. Previous session's sess-e layer-3 is now DISCHARGED -- it aged to STALE and the cascade completed, full residue purge logged.
This commit is contained in:
@@ -0,0 +1,209 @@
|
||||
# R120-golden-rebake-2026-07-30 — the golden is current again; the class still needs a mechanism
|
||||
|
||||
**Run:** R-120, CC on DooPlex, 2026-07-30. **Golden 0.186.0 baked, published, vouched, and proven on a
|
||||
fresh box.** The mechanism half (Part 3) awaits an operator ruling — see §6.
|
||||
|
||||
## 1. Baselines, established separately
|
||||
|
||||
| Thing | Value | Established from |
|
||||
|---|---|---|
|
||||
| controller on `main` | **0.186.0** | `felhom-controller/CHANGELOG.md:1` |
|
||||
| controller the golden baked **before** this task | **0.185.1** | the golden's **own record**: `drill/bake-0.185.1.log:1` (`baking controller …:0.185.1`) and `:330` (`GOLDEN_VERSION=0.185.1`) |
|
||||
| controller running, felhom-pve | **0.186.0** | `pct exec 9201 -- docker ps` |
|
||||
| controller running, demo-hp | **0.185.1** | same |
|
||||
| agent published + vouched | **0.116.0**, sha `b47c5c4d…` | registry GET + manifest read-back |
|
||||
| golden vouched **before** | **0.185.1**, sha `dba00f3e…` | manifest read-back |
|
||||
| `felhom.eu` HEAD at start | `772956d` | `git rev-parse` |
|
||||
|
||||
**R-120 confirmed at source, not inferred:** `build-golden.sh:345` derives
|
||||
`GOLDEN_VERSION="${GOLDEN_VERSION:-${CONTROLLER_IMAGE##*:}}"` — the golden version **is** the controller
|
||||
tag it bakes. So golden 0.185.1 ⇒ every fresh box installed controller 0.185.1, which predates R-114 +
|
||||
R-112 (shipped in 0.186.0), while a deployed box was already on 0.186.0. One release behind, and the
|
||||
missing release is the one that tells the truth about an absent backup target.
|
||||
|
||||
## 2. The bake procedure — it IS written down
|
||||
|
||||
Contrary to the risk the task flagged, the procedure exists and was followed, not reconstructed:
|
||||
|
||||
- **`documentation/runbooks/RUNBOOK-manual-build.md:101-115`** §4 "Golden image (fresh Day-0 installs)" —
|
||||
the five steps: revert the drill VM to `virgin`, boot it, run `build-golden.sh <controller VER>`,
|
||||
teardown per GL-1, then vouch in the hub.
|
||||
- **`felhom-agent/configs/build-golden.sh`** — the tool, with its usage in its own header
|
||||
(`[VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] CONTROLLER_IMAGE`), and it
|
||||
publishes to Gitea itself (`:335-371`) when `GITEA_USER`+`GITEA_TOKEN` are set. There is no separate
|
||||
`publish-golden.sh` — the bake script is the publisher.
|
||||
- **`documentation/runbooks/publish-train-rules.md`** — rules 1–4. Rule 2 (**save the floor field
|
||||
LAST**, separate save) and rule 3 (MinAgent) were consulted; see §4 for how they applied.
|
||||
|
||||
**One documentation-integrity finding, small but worth recording.** `RUNBOOK-manual-build.md:103-105`
|
||||
says to use the *"RECORDED* qemu launch line; do not reconstruct from memory" and points at
|
||||
`RUNBOOK-publish-0.79-0.110-2026-07-10.md` Phase C — but that line is *itself* labelled
|
||||
`PASS (reconstructed — DEVIATION)` at `:29`, with `:77-79` stating the canonical one-liner **was never
|
||||
saved**. So the runbook forbids reconstruction while citing a reconstruction. The line at `:29` is
|
||||
nonetheless correct — it booted first try here, and independently earlier today. **A trap learned today
|
||||
and worth adding to it:** adding any `-device` at launch shifts PCI slot assignment and renames the
|
||||
guest NIC, so `vmbr0` loses its port and SSH never comes up while the console shows a healthy PVE login.
|
||||
Boot the recorded line unchanged and hot-plug extras over QMP.
|
||||
|
||||
## 3. The bake — controller 0.186.0
|
||||
|
||||
Run in the DooPlex nested drill VM (its documented purpose — `target-selection.md` names it a **bake**
|
||||
fixture, and this is a bake), reverted to `virgin` first, `build-golden.sh v2.1.0` as a transient unit:
|
||||
|
||||
```
|
||||
[golden] build-golden.sh v2.1.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.186.0
|
||||
docker OK (overlay2; data-root /var/lib/docker)
|
||||
INFO: including mount point rootfs ('/')
|
||||
INFO: including mount point mp0 ('/var/lib/docker')
|
||||
INFO: including mount point mp1 ('/mnt/sys_drive')
|
||||
INFO: archive file size: 618MB
|
||||
[golden] upload OK (HTTP 201)
|
||||
GOLDEN_VERSION=0.186.0
|
||||
GOLDEN_SHA256=b760ac6a33e70700f427b4207afeebc8694bd6bb9478973c30901d9f059bde17
|
||||
```
|
||||
|
||||
Pass markers, counted on the saved log: `overlay2` present, **FATAL 0**, **exclusions 0**, **3 mounts
|
||||
included**, upload **201**. Template: `debian-13-standard_13.6-1_amd64.tar.zst` (the pinned version in
|
||||
the script's default has rotted away; the current one was resolved with `pveam available`).
|
||||
Log retained out-of-VM at `drill/bake-0.186.0.log` (330 lines); **token-leak grep against the literal
|
||||
token value on the saved log = 0**.
|
||||
|
||||
**GL-1 teardown:** build guest 9100 `pct destroy --purge`; `.gitea-token`, the script and the in-VM log
|
||||
`shred -u`'d (0 residual); VM powered off; **`drill.qcow2` reverted to `virgin`** (snapshot intact, ID 1,
|
||||
2026-07-03) — the fixture is exactly as found.
|
||||
|
||||
## 4. The three publish/vouch observables, quoted as returned
|
||||
|
||||
```
|
||||
(1) PUBLISHED — anonymous GET, i.e. what the installer actually does, sha over the bytes returned:
|
||||
HTTP 200 bytes=648930639
|
||||
b760ac6a33e70700f427b4207afeebc8694bd6bb9478973c30901d9f059bde17 /tmp/golden-0.186.0.tar.zst
|
||||
(identical to the bake's GOLDEN_SHA256)
|
||||
|
||||
(2) VOUCHED — manifest read BACK after the POST, never the 303:
|
||||
agent_version: vouched=['0.116.0'] agent_sha256 = b47c5c4dab641ee5…
|
||||
golden_version: vouched=['0.186.0'] golden_sha256 = b760ac6a33e70700…
|
||||
min_agent = 0.113.0 min_controller_version = 0.156.0
|
||||
wrapper_sha256 = 104db0a4401f65bb… (re-checked against configs/felhom-pbs-apply — NO drift)
|
||||
|
||||
(3) A CONSUMER RESOLVES IT — the hub telling a real fresh box what to install:
|
||||
[INFO] Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)
|
||||
```
|
||||
|
||||
**Publish-train rules as applied:** rule 1 (manifest before floor) — satisfied trivially, no floor moved.
|
||||
**Rule 2 (floor field LAST, separate save)** — the floor was **not touched at all**; `min_controller_version`
|
||||
read back **0.156.0**, unchanged, and the floor is a separate form (`/configuration/global-floor`) which
|
||||
was never posted. **Rule 3 (MinAgent)** — left at **0.113.0**, because controller 0.186.0's CHANGELOG
|
||||
header declares `MinAgent 0.113.0 (unchanged)`; raising it would hold demo-hp (agent 0.113.0) for no
|
||||
reason. The agent vouch was carried through untouched.
|
||||
|
||||
## 5. §2.1 — the fresh-box proof
|
||||
|
||||
**Machine: `demo-hp`** (Tier 0, the designated drill+build VM host). **A real day-0, not the fixture** —
|
||||
per the rule committed in Part 1, this is a *path* claim, so a fixture could not prove it. VM **9402**
|
||||
`r120-drill`, disks on a dir storage `r120-images` at **`/mnt/nvme-1tb` root** (never `local-lvm`),
|
||||
installed from the v1.25.0 ISO: self-register (appliance 14) → operator bind to `sess-f` → credentials
|
||||
delivered once → guest provisioned **from the new golden**.
|
||||
|
||||
**R-120's fix, proven at the install path:**
|
||||
|
||||
```
|
||||
hub → Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)
|
||||
hub → Event from sess-f: controller_started (info) — Controller elindult (0.186.0)
|
||||
box → felhom-agent --version → felhom-agent 0.116.0
|
||||
box → pct exec 9201 -- docker ps → gitea.dooplex.hu/admin/felhom-controller:0.186.0
|
||||
```
|
||||
|
||||
**A freshly installed box now runs 0.186.0. Before this task it ran 0.185.1.** That is the defect closed.
|
||||
|
||||
Target drive enrolled through the real endpoints and detached (real hot-detach), so the box sits in
|
||||
R-120's exact state — and agent 0.116.0's R-116 fix is visible on it:
|
||||
|
||||
```
|
||||
/disks felhom-backup row: state=disconnected role=system mount_path=''
|
||||
guest_path='/mnt/felhom-drives/cel' backup_target=True
|
||||
pvesm: felhom-backup dir inactive 0 0 0
|
||||
```
|
||||
|
||||
**The endpoint proof is NOT yet captured**, and is not claimed: `GET /api/storage/backup-target` returns
|
||||
`{"ok":false,"error":"dashboard not yet claimed"}`. The claim gate runs **before** auth
|
||||
(`controller/internal/web/auth.go:52-67`) and exempts only `/claim`, `/claim/request-new-code`,
|
||||
`/api/health`, `/static/` (`claim.go:221-228`) — **no Bearer escape**, and `regen-password`
|
||||
(`hub/internal/web/configs.go:965-980`) regenerates the *retrieval passphrase*, not the dashboard
|
||||
password. So an operator-relayed code is the only route. **This is R-119's fourth instance.**
|
||||
|
||||
## 6. Part 3 — the ruling, and the recommendation with its wiring
|
||||
|
||||
**Recommendation: the gate, and it must REFUSE, not warn.** The record is the argument: this gap has
|
||||
opened three times (**R-111**, **R-115**, **R-120**) and the first two were closed by re-baking and
|
||||
remembering. Remembering has now failed twice. And **R-29** is the standing proof that a check nobody
|
||||
runs is worse than none, because it reads as coverage — `hostinstall_gates.py` sat red and uninvoked
|
||||
across three version bumps and `hub_confirm_gate.py` has never run at all. So the distinguishing
|
||||
property is not *does a check exist* but **does it block**.
|
||||
|
||||
**Where it wires — established, not proposed loosely.** `hub/internal/web/configs.go:1116`
|
||||
`handleSetArtifacts` is **the** vouch path: the only UI writer of `store.SetArtifactManifest`
|
||||
(`:1142`), reached from `server.go:484`. It already normalises the versions (`:1121-1125`) and resolves
|
||||
the SHAs (`:1128-1131`), so a refusal belongs immediately before `:1142`. Nothing can vouch a golden
|
||||
without passing through it — which is precisely what makes it a mechanism rather than a fourth orphan
|
||||
in `scripts/`.
|
||||
|
||||
**The comparison signal is already local.** The hub stores `controller_version` per guest
|
||||
(`hub/internal/store/store.go:294`), populated from reports. A gate comparing the submitted
|
||||
`golden_version` against **the newest controller version any box has reported** would have caught this
|
||||
exact case: felhom-pve was reporting **0.186.0** while the manifest vouched a **0.185.1** golden. No
|
||||
outbound registry call, no new credential.
|
||||
|
||||
**Its honest blind spot, stated up front:** if no box has ever run the newer controller, the hub cannot
|
||||
know it exists, so a golden baked behind an *unreleased-to-the-fleet* controller would still pass. That
|
||||
is a real limit — and it does not touch the failure that actually happened three times, which is a
|
||||
golden left behind a controller already running in the field.
|
||||
|
||||
**The test must go through the production wiring**, not an injected seam: drive `handleSetArtifacts` over
|
||||
`httptest` with a guest row reporting 0.186.0 and a submitted golden of 0.185.1, then assert **both**
|
||||
that the response is the refusal **and** that the manifest was not written. Three shipped defects in
|
||||
this project were fully green with the seam disconnected; a gate that can be inert is the thing this
|
||||
gate exists to prevent.
|
||||
|
||||
**This is the operator's call because a refusing gate can block a release.** Not implemented pending the
|
||||
ruling. If it is declined, the weaker form goes in and the reason is recorded here rather than absorbed
|
||||
silently.
|
||||
|
||||
**If the gate is deferred, R-120 does not close** — it is re-opened, or a successor is filed naming the
|
||||
class: *"a published artifact and its baked copy drift and nothing checks"*. Three instances closed
|
||||
individually is how a class survives.
|
||||
|
||||
## 7. Teardown
|
||||
|
||||
| layer | item | disposition |
|
||||
|---|---|---|
|
||||
| 1 — machine | DooPlex bake fixture: build guest 9100 purged, `drill.qcow2` reverted to **`virgin`** | **DONE** |
|
||||
| 1 — machine | demo-hp VM **9402** `r120-drill` | **RETAINED pending the claim code** (§5's endpoint read) |
|
||||
| 2 — host | `r120-images` dir storage at `/mnt/nvme-1tb` | **RETAINED** with 9402 |
|
||||
| 3 — hub | customer **`sess-f`**, host **`sess-f-2670b5`**, appliance **14** | **RETAINED** with 9402 |
|
||||
|
||||
`pvesm status` on demo-hp before this run, for the layer-2 comparison at teardown:
|
||||
|
||||
```
|
||||
felhom-backup dir active 983379700 4566012 928787076 0.46%
|
||||
local dir active 40516856 14980024 23446440 36.97%
|
||||
local-lvm lvmthin active 56545280 21962186 34583093 38.84% ← the fence figure
|
||||
```
|
||||
|
||||
Teardown commands, recorded now:
|
||||
|
||||
```bash
|
||||
ssh demo-hp 'qm stop 9402; qm destroy 9402 --purge; pvesm remove r120-images; pvesm status'
|
||||
POST /configs/sess-f/delete ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-f expect_hosts=1
|
||||
```
|
||||
|
||||
**Previous session's layer 3 is now DISCHARGED.** `sess-e` was gate-blocked on ONLINE last run with the
|
||||
command recorded; it aged to STALE and the cascade completed this run — `/customers/sess-e` → **404**, 0
|
||||
hosts, and the hub logged `customer DELETE cascade COMPLETE for sess-e (journal #7) — full teardown`
|
||||
including `residue purged (reports=4 app_telemetry=4 … appliance_registrations=1)`. That is the §13
|
||||
layer-3 discipline working as intended: recorded, then executed.
|
||||
|
||||
**Fences held:** `drill-r50` untouched · guest 9201 on both demo boxes untouched · neither demo box
|
||||
re-targeted · nothing on `local-lvm` · Peti untouched · no existing box received the new golden ·
|
||||
`felhom-agent`/`felhom-controller` **source untouched** · the hub DB copy taken for the break-glass
|
||||
credential was `shred -u`'d immediately.
|
||||
Reference in New Issue
Block a user