docs(R-120): golden rebaked to 0.186.0, published, vouched, proven on a real day-0

The golden baked controller 0.185.1 -- confirmed from the golden's OWN record (drill/bake-0.185.1.log:1 and :330) and from build-golden.sh:345, which derives GOLDEN_VERSION from the controller tag. 0.185.1 predates R-114 + R-112, so every freshly installed box told a customer whose backup drive had fallen out that the backup was on the same disk as the system (false) and offered a different drive as the remedy. Baked golden 0.186.0 from main's controller in the DooPlex bake fixture: overlay2 OK, 3 mounts included, FATAL 0, exclusions 0, 618 MB, upload HTTP 201, GOLDEN_SHA256 b760ac6a33e70700..., token-leak grep 0, GL-1 teardown with drill.qcow2 back to virgin. Three observables, quoted as returned: PUBLISHED (anonymous GET -- what the installer does -- 200 / 648930639 bytes / sha identical to the bake); VOUCHED (manifest read BACK, not the 303); RESOLVED BY A CONSUMER (Artifact manifest served for customer sess-f, golden=0.186.0). Floor NOT touched per publish-train rule 2 -- it is a separate form and min_controller_version still reads 0.156.0. MinAgent left 0.113.0 because 0.186.0 declares it unchanged. Proven on a REAL day-0 on demo-hp, not the fixture, per the rule committed in Part 1: VM 9402 from the v1.25.0 ISO -> Controller elindult (0.186.0), box confirms felhom-controller:0.186.0 + agent 0.116.0. A fresh box now runs 0.186.0 where it ran 0.185.1. The procedure was NOT unwritten: RUNBOOK-manual-build.md:101-115 documents it and build-golden.sh carries its own usage and publishes to Gitea itself. One documentation-integrity finding: that runbook says to use the RECORDED qemu line and not reconstruct, while the line it cites is itself labelled reconstructed, the canonical one never having been saved. NOT done and not claimed: the TargetAbsent/empty-offer_path endpoint capture (the claim gate runs before auth with no Bearer escape -- R-119's fourth instance), and the Part 3 mechanism, which awaits the operator ruling. Recommendation and exact wiring recorded in the audit rather than built. VM 9402 + r120-images + customer sess-f retained pending that read, with teardown commands recorded. Previous session's sess-e layer-3 is now DISCHARGED -- it aged to STALE and the cascade completed, full residue purge logged.
2026-07-30 10:33:03 +02:00
parent 376365bb12
commit 49b627684c
2 changed files with 210 additions and 1 deletions
@@ -0,0 +1,209 @@
 # R120-golden-rebake-2026-07-30 — the golden is current again; the class still needs a mechanism
 **Run:** R-120, CC on DooPlex, 2026-07-30. **Golden 0.186.0 baked, published, vouched, and proven on a
 fresh box.** The mechanism half (Part 3) awaits an operator ruling — see §6.
 ## 1. Baselines, established separately
 | Thing | Value | Established from |
 |---|---|---|
 | controller on `main` | **0.186.0** | `felhom-controller/CHANGELOG.md:1` |
 | controller the golden baked **before** this task | **0.185.1** | the golden's **own record**: `drill/bake-0.185.1.log:1` (`baking controller …:0.185.1`) and `:330` (`GOLDEN_VERSION=0.185.1`) |
 | controller running, felhom-pve | **0.186.0** | `pct exec 9201 -- docker ps` |
 | controller running, demo-hp | **0.185.1** | same |
 | agent published + vouched | **0.116.0**, sha `b47c5c4d…` | registry GET + manifest read-back |
 | golden vouched **before** | **0.185.1**, sha `dba00f3e…` | manifest read-back |
 | `felhom.eu` HEAD at start | `772956d` | `git rev-parse` |
 **R-120 confirmed at source, not inferred:** `build-golden.sh:345` derives
 `GOLDEN_VERSION="${GOLDEN_VERSION:-${CONTROLLER_IMAGE##*:}}"` — the golden version **is** the controller
 tag it bakes. So golden 0.185.1 ⇒ every fresh box installed controller 0.185.1, which predates R-114 +
 R-112 (shipped in 0.186.0), while a deployed box was already on 0.186.0. One release behind, and the
 missing release is the one that tells the truth about an absent backup target.
 ## 2. The bake procedure — it IS written down
 Contrary to the risk the task flagged, the procedure exists and was followed, not reconstructed:
 - **`documentation/runbooks/RUNBOOK-manual-build.md:101-115`** §4 "Golden image (fresh Day-0 installs)" —
  the five steps: revert the drill VM to `virgin`, boot it, run `build-golden.sh <controller VER>`,
  teardown per GL-1, then vouch in the hub.
 - **`felhom-agent/configs/build-golden.sh`** — the tool, with its usage in its own header
  (`[VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] CONTROLLER_IMAGE`), and it
  publishes to Gitea itself (`:335-371`) when `GITEA_USER`+`GITEA_TOKEN` are set. There is no separate
  `publish-golden.sh` — the bake script is the publisher.
 - **`documentation/runbooks/publish-train-rules.md`** — rules 1–4. Rule 2 (**save the floor field
  LAST**, separate save) and rule 3 (MinAgent) were consulted; see §4 for how they applied.
 **One documentation-integrity finding, small but worth recording.** `RUNBOOK-manual-build.md:103-105`
 says to use the *"RECORDED* qemu launch line; do not reconstruct from memory" and points at
 `RUNBOOK-publish-0.79-0.110-2026-07-10.md` Phase C — but that line is *itself* labelled
 `PASS (reconstructed — DEVIATION)` at `:29`, with `:77-79` stating the canonical one-liner **was never
 saved**. So the runbook forbids reconstruction while citing a reconstruction. The line at `:29` is
 nonetheless correct — it booted first try here, and independently earlier today. **A trap learned today
 and worth adding to it:** adding any `-device` at launch shifts PCI slot assignment and renames the
 guest NIC, so `vmbr0` loses its port and SSH never comes up while the console shows a healthy PVE login.
 Boot the recorded line unchanged and hot-plug extras over QMP.
 ## 3. The bake — controller 0.186.0
 Run in the DooPlex nested drill VM (its documented purpose — `target-selection.md` names it a **bake**
 fixture, and this is a bake), reverted to `virgin` first, `build-golden.sh v2.1.0` as a transient unit:
 ```
 [golden] build-golden.sh v2.1.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.186.0
  docker OK (overlay2; data-root /var/lib/docker)
 INFO: including mount point rootfs ('/')
 INFO: including mount point mp0 ('/var/lib/docker')
 INFO: including mount point mp1 ('/mnt/sys_drive')
 INFO: archive file size: 618MB
 [golden]   upload OK (HTTP 201)
 GOLDEN_VERSION=0.186.0
 GOLDEN_SHA256=b760ac6a33e70700f427b4207afeebc8694bd6bb9478973c30901d9f059bde17
 ```
 Pass markers, counted on the saved log: `overlay2` present, **FATAL 0**, **exclusions 0**, **3 mounts
 included**, upload **201**. Template: `debian-13-standard_13.6-1_amd64.tar.zst` (the pinned version in
 the script's default has rotted away; the current one was resolved with `pveam available`).
 Log retained out-of-VM at `drill/bake-0.186.0.log` (330 lines); **token-leak grep against the literal
 token value on the saved log = 0**.
 **GL-1 teardown:** build guest 9100 `pct destroy --purge`; `.gitea-token`, the script and the in-VM log
 `shred -u`'d (0 residual); VM powered off; **`drill.qcow2` reverted to `virgin`** (snapshot intact, ID 1,
 2026-07-03) — the fixture is exactly as found.
 ## 4. The three publish/vouch observables, quoted as returned
 ```
 (1) PUBLISHED — anonymous GET, i.e. what the installer actually does, sha over the bytes returned:
    HTTP 200  bytes=648930639
    b760ac6a33e70700f427b4207afeebc8694bd6bb9478973c30901d9f059bde17  /tmp/golden-0.186.0.tar.zst
    (identical to the bake's GOLDEN_SHA256)
 (2) VOUCHED — manifest read BACK after the POST, never the 303:
    agent_version:  vouched=['0.116.0']      agent_sha256  = b47c5c4dab641ee5…
    golden_version: vouched=['0.186.0']      golden_sha256 = b760ac6a33e70700…
    min_agent = 0.113.0     min_controller_version = 0.156.0
    wrapper_sha256 = 104db0a4401f65bb…   (re-checked against configs/felhom-pbs-apply — NO drift)
 (3) A CONSUMER RESOLVES IT — the hub telling a real fresh box what to install:
    [INFO] Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)
 ```
 **Publish-train rules as applied:** rule 1 (manifest before floor) — satisfied trivially, no floor moved.
 **Rule 2 (floor field LAST, separate save)** — the floor was **not touched at all**; `min_controller_version`
 read back **0.156.0**, unchanged, and the floor is a separate form (`/configuration/global-floor`) which
 was never posted. **Rule 3 (MinAgent)** — left at **0.113.0**, because controller 0.186.0's CHANGELOG
 header declares `MinAgent 0.113.0 (unchanged)`; raising it would hold demo-hp (agent 0.113.0) for no
 reason. The agent vouch was carried through untouched.
 ## 5. §2.1 — the fresh-box proof
 **Machine: `demo-hp`** (Tier 0, the designated drill+build VM host). **A real day-0, not the fixture** —
 per the rule committed in Part 1, this is a *path* claim, so a fixture could not prove it. VM **9402**
 `r120-drill`, disks on a dir storage `r120-images` at **`/mnt/nvme-1tb` root** (never `local-lvm`),
 installed from the v1.25.0 ISO: self-register (appliance 14) → operator bind to `sess-f` → credentials
 delivered once → guest provisioned **from the new golden**.
 **R-120's fix, proven at the install path:**
 ```
 hub  → Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)
 hub  → Event from sess-f: controller_started (info) — Controller elindult (0.186.0)
 box  → felhom-agent --version                → felhom-agent 0.116.0
 box  → pct exec 9201 -- docker ps            → gitea.dooplex.hu/admin/felhom-controller:0.186.0
 ```
 **A freshly installed box now runs 0.186.0. Before this task it ran 0.185.1.** That is the defect closed.
 Target drive enrolled through the real endpoints and detached (real hot-detach), so the box sits in
 R-120's exact state — and agent 0.116.0's R-116 fix is visible on it:
 ```
 /disks felhom-backup row: state=disconnected role=system mount_path='' 
                          guest_path='/mnt/felhom-drives/cel' backup_target=True
 pvesm: felhom-backup dir inactive 0 0 0
 ```
 **The endpoint proof is NOT yet captured**, and is not claimed: `GET /api/storage/backup-target` returns
 `{"ok":false,"error":"dashboard not yet claimed"}`. The claim gate runs **before** auth
 (`controller/internal/web/auth.go:52-67`) and exempts only `/claim`, `/claim/request-new-code`,
 `/api/health`, `/static/` (`claim.go:221-228`) — **no Bearer escape**, and `regen-password`
 (`hub/internal/web/configs.go:965-980`) regenerates the *retrieval passphrase*, not the dashboard
 password. So an operator-relayed code is the only route. **This is R-119's fourth instance.**
 ## 6. Part 3 — the ruling, and the recommendation with its wiring
 **Recommendation: the gate, and it must REFUSE, not warn.** The record is the argument: this gap has
 opened three times (**R-111**, **R-115**, **R-120**) and the first two were closed by re-baking and
 remembering. Remembering has now failed twice. And **R-29** is the standing proof that a check nobody
 runs is worse than none, because it reads as coverage — `hostinstall_gates.py` sat red and uninvoked
 across three version bumps and `hub_confirm_gate.py` has never run at all. So the distinguishing
 property is not *does a check exist* but **does it block**.
 **Where it wires — established, not proposed loosely.** `hub/internal/web/configs.go:1116`
 `handleSetArtifacts` is **the** vouch path: the only UI writer of `store.SetArtifactManifest`
 (`:1142`), reached from `server.go:484`. It already normalises the versions (`:1121-1125`) and resolves
 the SHAs (`:1128-1131`), so a refusal belongs immediately before `:1142`. Nothing can vouch a golden
 without passing through it — which is precisely what makes it a mechanism rather than a fourth orphan
 in `scripts/`.
 **The comparison signal is already local.** The hub stores `controller_version` per guest
 (`hub/internal/store/store.go:294`), populated from reports. A gate comparing the submitted
 `golden_version` against **the newest controller version any box has reported** would have caught this
 exact case: felhom-pve was reporting **0.186.0** while the manifest vouched a **0.185.1** golden. No
 outbound registry call, no new credential.
 **Its honest blind spot, stated up front:** if no box has ever run the newer controller, the hub cannot
 know it exists, so a golden baked behind an *unreleased-to-the-fleet* controller would still pass. That
 is a real limit — and it does not touch the failure that actually happened three times, which is a
 golden left behind a controller already running in the field.
 **The test must go through the production wiring**, not an injected seam: drive `handleSetArtifacts` over
 `httptest` with a guest row reporting 0.186.0 and a submitted golden of 0.185.1, then assert **both**
 that the response is the refusal **and** that the manifest was not written. Three shipped defects in
 this project were fully green with the seam disconnected; a gate that can be inert is the thing this
 gate exists to prevent.
 **This is the operator's call because a refusing gate can block a release.** Not implemented pending the
 ruling. If it is declined, the weaker form goes in and the reason is recorded here rather than absorbed
 silently.
 **If the gate is deferred, R-120 does not close** — it is re-opened, or a successor is filed naming the
 class: *"a published artifact and its baked copy drift and nothing checks"*. Three instances closed
 individually is how a class survives.
 ## 7. Teardown
 | layer | item | disposition |
 |---|---|---|
 | 1 — machine | DooPlex bake fixture: build guest 9100 purged, `drill.qcow2` reverted to **`virgin`** | **DONE** |
 | 1 — machine | demo-hp VM **9402** `r120-drill` | **RETAINED pending the claim code** (§5's endpoint read) |
 | 2 — host | `r120-images` dir storage at `/mnt/nvme-1tb` | **RETAINED** with 9402 |
 | 3 — hub | customer **`sess-f`**, host **`sess-f-2670b5`**, appliance **14** | **RETAINED** with 9402 |
 `pvesm status` on demo-hp before this run, for the layer-2 comparison at teardown:
 ```
 felhom-backup  dir     active  983379700  4566012  928787076  0.46%
 local          dir     active   40516856 14980024   23446440 36.97%
 local-lvm  lvmthin     active   56545280 21962186   34583093 38.84%   ← the fence figure
 ```
 Teardown commands, recorded now:
 ```bash
 ssh demo-hp 'qm stop 9402; qm destroy 9402 --purge; pvesm remove r120-images; pvesm status'
 POST /configs/sess-f/delete  ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-f expect_hosts=1
 ```
 **Previous session's layer 3 is now DISCHARGED.** `sess-e` was gate-blocked on ONLINE last run with the
 command recorded; it aged to STALE and the cascade completed this run — `/customers/sess-e` → **404**, 0
 hosts, and the hub logged `customer DELETE cascade COMPLETE for sess-e (journal #7) — full teardown`
 including `residue purged (reports=4 app_telemetry=4 … appliance_registrations=1)`. That is the §13
 layer-3 discipline working as intended: recorded, then executed.
 **Fences held:** `drill-r50` untouched · guest 9201 on both demo boxes untouched · neither demo box
 re-targeted · nothing on `local-lvm` · Peti untouched · no existing box received the new golden ·
 `felhom-agent`/`felhom-controller` **source untouched** · the hub DB copy taken for the break-glass
 credential was `shred -u`'d immediately.
@@ -17,7 +17,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
 | **R-111** | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`.**~~ `felhom-host-install.sh` does not use `main`: it reads the hub-vouched manifest (`:423-436`) and fetches Gitea generic packages (agent `:1945`, golden `:2573`). Gitea's newest are **agent 0.96.0** and **golden 0.161.0**, and the hub's manifest selects exactly those — so a fresh box lands on **agent 0.96.0 + controller 0.161.0** (global floor `v0.156.0` < the golden's 0.161.0, so no self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and was never published** | **SHIPPED 2026-07-29 — the channel now serves agent 0.113.0 + golden 0.185.1** | — | **FIXED the same day it was found.** Agent **0.113.0** built from the clean tree @ `58b598b` and published (`scripts/publish-agent.sh`), sha `5f3247f756cb658e…`, round-trip GET verified. Golden **0.185.1** baked on the nested drill VM embedding controller `0.185.1`, published, sha `dba00f3e845c415e…` — bake clean: `Result=success`, overlay2, **all 3 mounts included** (rootfs+mp0+mp1), 0 FATAL/exclusions, upload HTTP 201, token-leak grep 0; log `drill/bake-0.185.1.log`; GL-1 teardown done (guest 9100 purged, secrets shredded, disk restored to `virgin`). Hub Day-0 manifest moved **both together in one POST** so it never vouched a new agent against an old golden; `min_agent` **0.93.0 → 0.113.0**, which is what controller v0.185.0 declares (`felhom-controller/CHANGELOG.md:15`) — **zero fleet impact, verified: all three enrolled hosts already run agent 0.113.0, so no box is held.** `wrapper_sha256` preserved verbatim (re-checked against `configs/felhom-pbs-apply` — no drift). **The global controller floor was deliberately NOT raised**: the golden now bakes 0.185.1, so a fresh box needs no self-update, and raising it would have been an unnecessary fleet-wide write. Original finding follows. **Found 2026-07-29 by the E-2d Phase 0 gate, which stopped the run before a VM was created.** 17 unpublished releases (v0.97.0–v0.113.0) strand the **entire R-82 tiered-backup arc** plus **F-CRIT-2** (a failed backup looking fresh — 7 days silent) and **F-REBOOT** (a guest rebooted mid-backup never returns): a new customer's box would install without them. **Blocks E-2d's C3/C4/C5** — those test endpoints and events that do not exist in 0.96.0/0.161.0. The **controller is fine** (registry has 0.185.1, floor-driven self-update), so the gap is specific to the two Gitea-generic artifacts. **Mirror of R-110, not a duplicate:** R-110 = the installer publishes instantly with no staging; R-111 = the agent/golden publish gate exists and was never walked. Fix should decide whether publishing joins the release train rather than staying a remembered step (R-29's shape, one layer up). Evidence: `audits/E2D-fresh-vm-2026-07-29.md` **DEFERRED LEG, AND IT RECURRED → R-115.** This row's shipped half stands and is not reopened: the bump happened, was verified, and was proven end-to-end by the E-2d install. But its own closing line — *decide whether publishing joins the release train rather than staying a remembered step* — was never acted on, and agent 0.114.0 reproduced the exact condition the same afternoon. The recurrence is filed as **R-115**, not as a reopen, because the stale-channel finding is closed while the process defect that caused it is a distinct problem with a distinct fix. | CC |
 | **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **WAITING-ON-OPERATOR (M)** | operator ruling on the release process | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design | CC |
 | **R-116** | ~~**The drive-absent alarm and its recovery were a MISMATCHED PAIR — absent fired the GENERIC `storage_disconnected`, return the SPECIFIC `backup_target_restored`; `backup_target_absent` never fired at all**~~ | **SHIPPED + PROVEN-LIVE** (agent v0.116.0, 2026-07-30) | — | **CLOSED. The full four-event sequence, on the wire, on a fresh box** (`audits/R116-v0116-2026-07-30.md`): `backup_target_absent (error)` on detach → `backup_target_restored (info)` on return for the TARGET, and `storage_disconnected (error)` → `storage_reconnected (info)` for a NON-target drive on the same box four minutes apart. **Two matched pairs, correctly discriminated — and discrimination is proven NON-trivially for the first time**, since both prior runs had the target itself emit the generic event. Gate fired in **3 s**; all four events reached the hub, so the specific alarm, its severity, its Hungarian copy and the hub routing are now exercised end-to-end. **Over-correction PASSES with a positive observable** (0 ABSENT lines / 0 drive events over 2m14s with both drives present, target `degraded:false`, while 2 `RETURNED` lines prove the gate was ticking). Ran on a nested PVE on **demo-hp** per `runbooks/target-selection.md` — through the **real day-0** from the v1.25.0 ISO, with the agent **installed unaided from the vouched Day-0 manifest** (published sha `b47c5c4dab641ee5…`, independent registry GET verified, manifest read back), drives enrolled through the real endpoints, device loss a real hot-detach. **THE FIX, and the ruling is the substantive part:** the mechanism was first isolated from the captured payload (`DIAG-r116-disks-payload-2026-07-30.md`) after two fixes aimed at shapes that do not occur. **Both smaller-looking options were REJECTED because they regress R-114** — `backup_target_offer.go:79` reads `BackupTarget && MountPath != ""` as *"a real drive with its own mountpoint — healthy"* and returns before its `TargetAbsent` branch, so back-filling `MountPath` on the Observe row **or** flagging the registry row (whose `MountPath` is the stale unit-file value) would have told the customer the backup target is fine while its drive was gone. **R-114's correctness was resting on R-116's bug** — a coupling invisible until the payload existed. Taken instead: the Observe row gets the **guest path only** (`mount_path` stays `""`, which is true) from a new `ConfigPath` (`json:"-"`, so the cross-repo golden + key-set contract is untouched), and the union row is deduped **on guest path** — the join being CONFIGURATION (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), the only identity that survives the device. Tests 845→849; 4 red-proofs each asserted to land, and red-proof 1 replays v0.115.0's code and fails, which is the empirical proof it was inert. Its green test had supplied a `MountPath` production never supplies AND left `DriveTargets` nil so the union loop never ran — both corrected. v0.115.0 left in place (inert, harmless). Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded. **Caveat: the drill's controller was 0.185.1 from the golden, which PREDATES R-114**, so its absent-state banner showed the old false copy — the golden being a release behind, not a regression → **R-120** | — |
-| **R-120** | **The golden bakes a controller that predates R-114 + R-112, so a FRESH customer box shows the WRONG absent-target message.** The Day-0 golden is **0.185.1**; both R-114 (the third state — *configured and its drive is gone* vs *never configured*) and R-112 (the banner having any UI consumer at all) shipped in **0.186.0**. Observed live this run on the drill box with its target detached: the endpoint returned the generic degraded branch — *"A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer"* (**false** — the target is a drive that vanished) **plus** `offer_path: /mnt/felhom-drives/adat`, offering another drive as the remedy. That is E2D §5.3's exact payload, still reachable on any newly installed box | **READY (S)** | — | **This is R-115's class one layer over: baking is a remembered step.** R-111 was closed by re-baking the golden to 0.185.1; 0.186.0 then shipped and the golden was not moved, so the gap reopened silently — and this time the stale artifact carries a **customer-facing falsehood** in exactly the state R-116 now alarms about correctly. Fix is mechanical (bake + publish + vouch a 0.186.0 golden), but the decision that matters is whether the golden joins the release train or the manifest gains a gate that refuses to vouch a golden older than the newest controller — **reminders have already failed twice here** (R-111 → R-115). Evidence: `audits/R116-v0116-2026-07-30.md` §6a caveat + §1 | CC |
+| **R-120** | **The golden baked a controller that predated R-114 + R-112, so a FRESH box showed the customer the WRONG absent-target message** | **GOLDEN REBAKED + PROVEN-LIVE (0.186.0, 2026-07-30); the MECHANISM half awaits an operator ruling** | operator ruling on gate-vs-train (Part 3) | **`audits/R120-golden-rebake-2026-07-30.md`.** Golden **0.186.0** baked from `main`'s controller in the DooPlex bake fixture (`build-golden.sh v2.1.0`: overlay2 OK, **3 mounts included**, FATAL 0, exclusions 0, 618 MB, upload **HTTP 201**, `GOLDEN_SHA256=b760ac6a33e70700…`; token-leak grep 0; GL-1 teardown and `drill.qcow2` back to `virgin`). Three observables: **published** — anonymous GET (what the installer does) 200 / 648930639 bytes / sha identical to the bake; **vouched** — manifest read BACK shows golden 0.186.0 + sha, agent 0.116.0 carried through, `min_agent` 0.113.0, wrapper sha no drift; **resolved by a consumer** — `Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)`. **Floor NOT touched** (`min_controller_version` still 0.156.0, and the floor is a separate form) per publish-train rule 2; MinAgent left 0.113.0 because 0.186.0 declares it unchanged. **PROVEN ON A REAL DAY-0, not the fixture** (per the Part-1 rule: a fixture proves a mechanism, only a fresh box proves a path): VM 9402 on demo-hp from the v1.25.0 ISO → `Controller elindult (0.186.0)`, box confirms `felhom-controller:0.186.0` + agent 0.116.0. **A fresh box now runs 0.186.0 where it ran 0.185.1 — the defect is closed.** The `TargetAbsent`/empty-`offer_path` endpoint capture is **NOT yet taken and not claimed**: the claim gate runs before auth (`auth.go:52-67`, exempting only 4 paths, no Bearer escape) and `regen-password` regenerates the *retrieval* passphrase, so an operator-relayed code is the only route — **R-119's fourth instance**. **The Part 3 mechanism is NOT built pending the ruling**; recommendation and its exact wiring are in §6 — a gate that **refuses** inside `hub/internal/web/configs.go:1116` `handleSetArtifacts` (the ONLY UI writer of `SetArtifactManifest`, `:1142`), comparing the submitted golden against the newest `controller_version` any box has reported (`store.go:294` — already local, no outbound call; it would have caught this case, since felhom-pve was reporting 0.186.0 against a 0.185.1 golden), tested through the production wiring rather than a seam. **If the ruling defers it, this row does NOT close** — it re-opens or a successor is filed naming the class *"a published artifact and its baked copy drift and nothing checks"*, because three instances closed individually (R-111, R-115, R-120) is how a class survives | CC |
 | **R-117** | **A returned drive's guest bind is a DEAD MOUNT, and the gate restarts the customer's apps onto it while every signal reads healthy.** After a detach/reattach the host raw mount is fine on the NEW device node, but the guest's bind still names the OLD one and every operation through it returns `EIO`: host `findmnt /mnt/cel` → `/dev/vdc`; guest `findmnt /mnt/felhom-drives/cel` → `/dev/vdb[/felhom-data] …,shutdown`; guest `ls` and write → **`Input/output error`**; and `/disks` reports `state:"attached"`, `role:"user-data"`, `bound_under_parent:TRUE`, `backup_target:true` | **READY (M) — NEW 2026-07-30, deliberately not chased** | — | **Worse than R-116 and it outranks it.** `planDriveGates` computes `present[gp] = … \|\| d.BoundUnderParent` (`intermediary.go:226`) = `true`, and the drive is `Disconnected`, so the gate takes the **`Return`** branch (`intermediary.go:281`, `:301-320`): it clears the flag, **restarts the gate-stopped apps onto a namespace that `EIO`s on every call**, reports the drive healthy and emails `backup_target_restored`. **No alarm on any channel.** R-113's conjunction cannot catch it — half 1 (guest mountinfo has an entry at the guest path) is satisfied by the STALE entry and half 2 (`devicePresent`) by the NEW device; **neither half compares the two**. Not a false invariant: `disks.go:158-165` claims only the boot-ordering and outlived-bind cases, so this is a genuine gap. **Seen but uninvestigated in E-2d, Session C and Part 5 (three consecutive runs) as "the drive returned as /dev/sdc while findmnt still read /dev/sdb"** — that framing read as cosmetic; it is not. Reproduced deliberately and probed to EIO in `audits/DIAG-r116-disks-payload-2026-07-30.md` §11. Deserves its own run | CC |
 | **R-118** | **An absent drive's union row advertises the ROOT filesystem's capacity as its own.** In the absent-state payload the registry-union row reports `total_bytes: 49675956224 / used_bytes: 4584579072` — **byte-identical to the `local` row** (`durable_id: path:/var/lib/vz`, i.e. `pve-root`) in the same response. The real drive is **4 GB** | **READY (XS) — NEW 2026-07-30** | — | Cause: `statfsCapacity(d.MountPath)` (`disks.go:335-338`) statfs's `/mnt/cel`, which with the device gone is a **bare directory on the root filesystem**. `observe.go:176-183`'s comment warns about exactly this trap and guards the Observe path ("*an unmounted removable dir-storage's mountpoint reverts to a bare directory on root … catastrophic DR mis-id*"); **the union path has no equivalent guard.** **Not a DR mis-id** — `durable_id` on that row is still the correct `uuid:…`, so re-attach identity is safe. It is a **false capacity** reaching every consumer of `total_bytes`/`used_fraction` (fill monitors, storage cards): a detached 4 GB drive advertises 46 GiB at 9.2 % used. Same class as `role.go:180-181` — an absent drive's fields decaying to the root filesystem's. Evidence: `audits/DIAG-r116-disks-payload-2026-07-30.md` §12 | CC |
 | **R-113** | ~~**The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.**~~ `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc/<pid>/mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **SHIPPED + PROVEN-LIVE** (agent v0.114.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** Target drive hot-detached on a fresh box running the SHIPPED agent 0.114.0 (from the Day-0 manifest, not a hand build). **The gate fired in 4 seconds** — E-2d measured ZERO over 4½ minutes — and `SetDisconnected` was reached. It fired on exactly the shape that defeated it: raw `/mnt/mentes` NOT mounted while the bind `/mnt/felhom-drives/mentes` still read `/dev/sdb[/felhom-data]`. **Over-correction guard PASSED:** with both drives present, 0 ABSENT lines and the target stayed healthy. **Note: C5 still FAILED — but on a different defect (→ R-116), not on this one.** **SESSION C IS NOW UNBLOCKED.** Agent **0.114.0 published and vouched** 2026-07-29 (sha `5e4c15ebee2d7583…`, round-trip GET verified; hub manifest read back showing it) — a fresh drill box now installs the fix under test instead of the bug. The golden was NOT re-baked and `min_agent` was NOT raised: the golden bakes the controller, not the agent, and controller v0.185.0 declares MinAgent 0.113.0, which 0.114.0 already satisfies. **FIX: `BoundUnderParent` is now a CONJUNCTION — bound under the parent AND the drive's raw host mount still mounted** (`devicePresent`, new `deviceCheck` seam), at BOTH `/disks` construction sites. The raw mount is the device-bound systemd unit that dies with the device; the agent's bind is not — so the raw mount IS the device signal, grounded in E-2d's measurement rather than inference. **Conjunction, deliberately:** the device half alone would regress boot ordering (raw mounts early, bind lands ~18 s later — that window must keep reading absent), so existing behaviour is byte-identical and only the unreachable case is closed. **Unknown is never absent** (`devicePresent("")` = true) — a false absent stops a working customer's apps. **Controller UNCHANGED, no MinAgent bump:** `BoundUnderParent` has exactly one functional consumer (`planDriveGates:226`); a new `DevicePresent` bool was rejected because absent-from-JSON decodes to `false`, which would have made every drive on an older agent read ABSENT. +6 tests (208→214), 4 red-proofs run and reverted. **Deployed to demo-felhom; over-correction guard verified in production** — raw mount present, drive still reads present, 10/10 apps untouched, no gate action, no false alarm. **demo-hp deliberately left on 0.113.0** (the spec scoped deploy to felhom-pve) — it still carries the bug. **⚠️ SESSION C BLOCKER: the hub Day-0 manifest vouches agent 0.113.0**, so a fresh drill box would install WITHOUT this fix and validate nothing — publish + vouch 0.114.0 first (R-111's trap, same shape). **Awaiting live proof of exactly one leg:** device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. Original finding: **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 **Session C scope UNCHANGED by Session B** — R-114/R-112 shipped as controller v0.186.0 and neither touches the agent; the leg awaiting proof is still device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. One rebuild now validates all three. | CC |