diff --git a/documentation/audits/R120-golden-rebake-2026-07-30.md b/documentation/audits/R120-golden-rebake-2026-07-30.md new file mode 100644 index 0000000..29a55d0 --- /dev/null +++ b/documentation/audits/R120-golden-rebake-2026-07-30.md @@ -0,0 +1,209 @@ +# R120-golden-rebake-2026-07-30 — the golden is current again; the class still needs a mechanism + +**Run:** R-120, CC on DooPlex, 2026-07-30. **Golden 0.186.0 baked, published, vouched, and proven on a +fresh box.** The mechanism half (Part 3) awaits an operator ruling — see §6. + +## 1. Baselines, established separately + +| Thing | Value | Established from | +|---|---|---| +| controller on `main` | **0.186.0** | `felhom-controller/CHANGELOG.md:1` | +| controller the golden baked **before** this task | **0.185.1** | the golden's **own record**: `drill/bake-0.185.1.log:1` (`baking controller …:0.185.1`) and `:330` (`GOLDEN_VERSION=0.185.1`) | +| controller running, felhom-pve | **0.186.0** | `pct exec 9201 -- docker ps` | +| controller running, demo-hp | **0.185.1** | same | +| agent published + vouched | **0.116.0**, sha `b47c5c4d…` | registry GET + manifest read-back | +| golden vouched **before** | **0.185.1**, sha `dba00f3e…` | manifest read-back | +| `felhom.eu` HEAD at start | `772956d` | `git rev-parse` | + +**R-120 confirmed at source, not inferred:** `build-golden.sh:345` derives +`GOLDEN_VERSION="${GOLDEN_VERSION:-${CONTROLLER_IMAGE##*:}}"` — the golden version **is** the controller +tag it bakes. So golden 0.185.1 ⇒ every fresh box installed controller 0.185.1, which predates R-114 + +R-112 (shipped in 0.186.0), while a deployed box was already on 0.186.0. One release behind, and the +missing release is the one that tells the truth about an absent backup target. + +## 2. The bake procedure — it IS written down + +Contrary to the risk the task flagged, the procedure exists and was followed, not reconstructed: + +- **`documentation/runbooks/RUNBOOK-manual-build.md:101-115`** §4 "Golden image (fresh Day-0 installs)" — + the five steps: revert the drill VM to `virgin`, boot it, run `build-golden.sh `, + teardown per GL-1, then vouch in the hub. +- **`felhom-agent/configs/build-golden.sh`** — the tool, with its usage in its own header + (`[VMID] [TEMPLATE_VOLID] [ROOTFS_STORAGE] [ARCHIVE_STORAGE] [BRIDGE] CONTROLLER_IMAGE`), and it + publishes to Gitea itself (`:335-371`) when `GITEA_USER`+`GITEA_TOKEN` are set. There is no separate + `publish-golden.sh` — the bake script is the publisher. +- **`documentation/runbooks/publish-train-rules.md`** — rules 1–4. Rule 2 (**save the floor field + LAST**, separate save) and rule 3 (MinAgent) were consulted; see §4 for how they applied. + +**One documentation-integrity finding, small but worth recording.** `RUNBOOK-manual-build.md:103-105` +says to use the *"RECORDED* qemu launch line; do not reconstruct from memory" and points at +`RUNBOOK-publish-0.79-0.110-2026-07-10.md` Phase C — but that line is *itself* labelled +`PASS (reconstructed — DEVIATION)` at `:29`, with `:77-79` stating the canonical one-liner **was never +saved**. So the runbook forbids reconstruction while citing a reconstruction. The line at `:29` is +nonetheless correct — it booted first try here, and independently earlier today. **A trap learned today +and worth adding to it:** adding any `-device` at launch shifts PCI slot assignment and renames the +guest NIC, so `vmbr0` loses its port and SSH never comes up while the console shows a healthy PVE login. +Boot the recorded line unchanged and hot-plug extras over QMP. + +## 3. The bake — controller 0.186.0 + +Run in the DooPlex nested drill VM (its documented purpose — `target-selection.md` names it a **bake** +fixture, and this is a bake), reverted to `virgin` first, `build-golden.sh v2.1.0` as a transient unit: + +``` +[golden] build-golden.sh v2.1.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.186.0 + docker OK (overlay2; data-root /var/lib/docker) +INFO: including mount point rootfs ('/') +INFO: including mount point mp0 ('/var/lib/docker') +INFO: including mount point mp1 ('/mnt/sys_drive') +INFO: archive file size: 618MB +[golden] upload OK (HTTP 201) +GOLDEN_VERSION=0.186.0 +GOLDEN_SHA256=b760ac6a33e70700f427b4207afeebc8694bd6bb9478973c30901d9f059bde17 +``` + +Pass markers, counted on the saved log: `overlay2` present, **FATAL 0**, **exclusions 0**, **3 mounts +included**, upload **201**. Template: `debian-13-standard_13.6-1_amd64.tar.zst` (the pinned version in +the script's default has rotted away; the current one was resolved with `pveam available`). +Log retained out-of-VM at `drill/bake-0.186.0.log` (330 lines); **token-leak grep against the literal +token value on the saved log = 0**. + +**GL-1 teardown:** build guest 9100 `pct destroy --purge`; `.gitea-token`, the script and the in-VM log +`shred -u`'d (0 residual); VM powered off; **`drill.qcow2` reverted to `virgin`** (snapshot intact, ID 1, +2026-07-03) — the fixture is exactly as found. + +## 4. The three publish/vouch observables, quoted as returned + +``` +(1) PUBLISHED — anonymous GET, i.e. what the installer actually does, sha over the bytes returned: + HTTP 200 bytes=648930639 + b760ac6a33e70700f427b4207afeebc8694bd6bb9478973c30901d9f059bde17 /tmp/golden-0.186.0.tar.zst + (identical to the bake's GOLDEN_SHA256) + +(2) VOUCHED — manifest read BACK after the POST, never the 303: + agent_version: vouched=['0.116.0'] agent_sha256 = b47c5c4dab641ee5… + golden_version: vouched=['0.186.0'] golden_sha256 = b760ac6a33e70700… + min_agent = 0.113.0 min_controller_version = 0.156.0 + wrapper_sha256 = 104db0a4401f65bb… (re-checked against configs/felhom-pbs-apply — NO drift) + +(3) A CONSUMER RESOLVES IT — the hub telling a real fresh box what to install: + [INFO] Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0) +``` + +**Publish-train rules as applied:** rule 1 (manifest before floor) — satisfied trivially, no floor moved. +**Rule 2 (floor field LAST, separate save)** — the floor was **not touched at all**; `min_controller_version` +read back **0.156.0**, unchanged, and the floor is a separate form (`/configuration/global-floor`) which +was never posted. **Rule 3 (MinAgent)** — left at **0.113.0**, because controller 0.186.0's CHANGELOG +header declares `MinAgent 0.113.0 (unchanged)`; raising it would hold demo-hp (agent 0.113.0) for no +reason. The agent vouch was carried through untouched. + +## 5. §2.1 — the fresh-box proof + +**Machine: `demo-hp`** (Tier 0, the designated drill+build VM host). **A real day-0, not the fixture** — +per the rule committed in Part 1, this is a *path* claim, so a fixture could not prove it. VM **9402** +`r120-drill`, disks on a dir storage `r120-images` at **`/mnt/nvme-1tb` root** (never `local-lvm`), +installed from the v1.25.0 ISO: self-register (appliance 14) → operator bind to `sess-f` → credentials +delivered once → guest provisioned **from the new golden**. + +**R-120's fix, proven at the install path:** + +``` +hub → Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0) +hub → Event from sess-f: controller_started (info) — Controller elindult (0.186.0) +box → felhom-agent --version → felhom-agent 0.116.0 +box → pct exec 9201 -- docker ps → gitea.dooplex.hu/admin/felhom-controller:0.186.0 +``` + +**A freshly installed box now runs 0.186.0. Before this task it ran 0.185.1.** That is the defect closed. + +Target drive enrolled through the real endpoints and detached (real hot-detach), so the box sits in +R-120's exact state — and agent 0.116.0's R-116 fix is visible on it: + +``` +/disks felhom-backup row: state=disconnected role=system mount_path='' + guest_path='/mnt/felhom-drives/cel' backup_target=True +pvesm: felhom-backup dir inactive 0 0 0 +``` + +**The endpoint proof is NOT yet captured**, and is not claimed: `GET /api/storage/backup-target` returns +`{"ok":false,"error":"dashboard not yet claimed"}`. The claim gate runs **before** auth +(`controller/internal/web/auth.go:52-67`) and exempts only `/claim`, `/claim/request-new-code`, +`/api/health`, `/static/` (`claim.go:221-228`) — **no Bearer escape**, and `regen-password` +(`hub/internal/web/configs.go:965-980`) regenerates the *retrieval passphrase*, not the dashboard +password. So an operator-relayed code is the only route. **This is R-119's fourth instance.** + +## 6. Part 3 — the ruling, and the recommendation with its wiring + +**Recommendation: the gate, and it must REFUSE, not warn.** The record is the argument: this gap has +opened three times (**R-111**, **R-115**, **R-120**) and the first two were closed by re-baking and +remembering. Remembering has now failed twice. And **R-29** is the standing proof that a check nobody +runs is worse than none, because it reads as coverage — `hostinstall_gates.py` sat red and uninvoked +across three version bumps and `hub_confirm_gate.py` has never run at all. So the distinguishing +property is not *does a check exist* but **does it block**. + +**Where it wires — established, not proposed loosely.** `hub/internal/web/configs.go:1116` +`handleSetArtifacts` is **the** vouch path: the only UI writer of `store.SetArtifactManifest` +(`:1142`), reached from `server.go:484`. It already normalises the versions (`:1121-1125`) and resolves +the SHAs (`:1128-1131`), so a refusal belongs immediately before `:1142`. Nothing can vouch a golden +without passing through it — which is precisely what makes it a mechanism rather than a fourth orphan +in `scripts/`. + +**The comparison signal is already local.** The hub stores `controller_version` per guest +(`hub/internal/store/store.go:294`), populated from reports. A gate comparing the submitted +`golden_version` against **the newest controller version any box has reported** would have caught this +exact case: felhom-pve was reporting **0.186.0** while the manifest vouched a **0.185.1** golden. No +outbound registry call, no new credential. + +**Its honest blind spot, stated up front:** if no box has ever run the newer controller, the hub cannot +know it exists, so a golden baked behind an *unreleased-to-the-fleet* controller would still pass. That +is a real limit — and it does not touch the failure that actually happened three times, which is a +golden left behind a controller already running in the field. + +**The test must go through the production wiring**, not an injected seam: drive `handleSetArtifacts` over +`httptest` with a guest row reporting 0.186.0 and a submitted golden of 0.185.1, then assert **both** +that the response is the refusal **and** that the manifest was not written. Three shipped defects in +this project were fully green with the seam disconnected; a gate that can be inert is the thing this +gate exists to prevent. + +**This is the operator's call because a refusing gate can block a release.** Not implemented pending the +ruling. If it is declined, the weaker form goes in and the reason is recorded here rather than absorbed +silently. + +**If the gate is deferred, R-120 does not close** — it is re-opened, or a successor is filed naming the +class: *"a published artifact and its baked copy drift and nothing checks"*. Three instances closed +individually is how a class survives. + +## 7. Teardown + +| layer | item | disposition | +|---|---|---| +| 1 — machine | DooPlex bake fixture: build guest 9100 purged, `drill.qcow2` reverted to **`virgin`** | **DONE** | +| 1 — machine | demo-hp VM **9402** `r120-drill` | **RETAINED pending the claim code** (§5's endpoint read) | +| 2 — host | `r120-images` dir storage at `/mnt/nvme-1tb` | **RETAINED** with 9402 | +| 3 — hub | customer **`sess-f`**, host **`sess-f-2670b5`**, appliance **14** | **RETAINED** with 9402 | + +`pvesm status` on demo-hp before this run, for the layer-2 comparison at teardown: + +``` +felhom-backup dir active 983379700 4566012 928787076 0.46% +local dir active 40516856 14980024 23446440 36.97% +local-lvm lvmthin active 56545280 21962186 34583093 38.84% ← the fence figure +``` + +Teardown commands, recorded now: + +```bash +ssh demo-hp 'qm stop 9402; qm destroy 9402 --purge; pvesm remove r120-images; pvesm status' +POST /configs/sess-f/delete ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-f expect_hosts=1 +``` + +**Previous session's layer 3 is now DISCHARGED.** `sess-e` was gate-blocked on ONLINE last run with the +command recorded; it aged to STALE and the cascade completed this run — `/customers/sess-e` → **404**, 0 +hosts, and the hub logged `customer DELETE cascade COMPLETE for sess-e (journal #7) — full teardown` +including `residue purged (reports=4 app_telemetry=4 … appliance_registrations=1)`. That is the §13 +layer-3 discipline working as intended: recorded, then executed. + +**Fences held:** `drill-r50` untouched · guest 9201 on both demo boxes untouched · neither demo box +re-targeted · nothing on `local-lvm` · Peti untouched · no existing box received the new golden · +`felhom-agent`/`felhom-controller` **source untouched** · the hub DB copy taken for the break-glass +credential was `shred -u`'d immediately. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 4f0eed2..1473ee2 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -17,7 +17,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-111** | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`.**~~ `felhom-host-install.sh` does not use `main`: it reads the hub-vouched manifest (`:423-436`) and fetches Gitea generic packages (agent `:1945`, golden `:2573`). Gitea's newest are **agent 0.96.0** and **golden 0.161.0**, and the hub's manifest selects exactly those — so a fresh box lands on **agent 0.96.0 + controller 0.161.0** (global floor `v0.156.0` < the golden's 0.161.0, so no self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and was never published** | **SHIPPED 2026-07-29 — the channel now serves agent 0.113.0 + golden 0.185.1** | — | **FIXED the same day it was found.** Agent **0.113.0** built from the clean tree @ `58b598b` and published (`scripts/publish-agent.sh`), sha `5f3247f756cb658e…`, round-trip GET verified. Golden **0.185.1** baked on the nested drill VM embedding controller `0.185.1`, published, sha `dba00f3e845c415e…` — bake clean: `Result=success`, overlay2, **all 3 mounts included** (rootfs+mp0+mp1), 0 FATAL/exclusions, upload HTTP 201, token-leak grep 0; log `drill/bake-0.185.1.log`; GL-1 teardown done (guest 9100 purged, secrets shredded, disk restored to `virgin`). Hub Day-0 manifest moved **both together in one POST** so it never vouched a new agent against an old golden; `min_agent` **0.93.0 → 0.113.0**, which is what controller v0.185.0 declares (`felhom-controller/CHANGELOG.md:15`) — **zero fleet impact, verified: all three enrolled hosts already run agent 0.113.0, so no box is held.** `wrapper_sha256` preserved verbatim (re-checked against `configs/felhom-pbs-apply` — no drift). **The global controller floor was deliberately NOT raised**: the golden now bakes 0.185.1, so a fresh box needs no self-update, and raising it would have been an unnecessary fleet-wide write. Original finding follows. **Found 2026-07-29 by the E-2d Phase 0 gate, which stopped the run before a VM was created.** 17 unpublished releases (v0.97.0–v0.113.0) strand the **entire R-82 tiered-backup arc** plus **F-CRIT-2** (a failed backup looking fresh — 7 days silent) and **F-REBOOT** (a guest rebooted mid-backup never returns): a new customer's box would install without them. **Blocks E-2d's C3/C4/C5** — those test endpoints and events that do not exist in 0.96.0/0.161.0. The **controller is fine** (registry has 0.185.1, floor-driven self-update), so the gap is specific to the two Gitea-generic artifacts. **Mirror of R-110, not a duplicate:** R-110 = the installer publishes instantly with no staging; R-111 = the agent/golden publish gate exists and was never walked. Fix should decide whether publishing joins the release train rather than staying a remembered step (R-29's shape, one layer up). Evidence: `audits/E2D-fresh-vm-2026-07-29.md` **DEFERRED LEG, AND IT RECURRED → R-115.** This row's shipped half stands and is not reopened: the bump happened, was verified, and was proven end-to-end by the E-2d install. But its own closing line — *decide whether publishing joins the release train rather than staying a remembered step* — was never acted on, and agent 0.114.0 reproduced the exact condition the same afternoon. The recurrence is filed as **R-115**, not as a reopen, because the stale-channel finding is closed while the process defect that caused it is a distinct problem with a distinct fix. | CC | | **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **WAITING-ON-OPERATOR (M)** | operator ruling on the release process | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design | CC | | **R-116** | ~~**The drive-absent alarm and its recovery were a MISMATCHED PAIR — absent fired the GENERIC `storage_disconnected`, return the SPECIFIC `backup_target_restored`; `backup_target_absent` never fired at all**~~ | **SHIPPED + PROVEN-LIVE** (agent v0.116.0, 2026-07-30) | — | **CLOSED. The full four-event sequence, on the wire, on a fresh box** (`audits/R116-v0116-2026-07-30.md`): `backup_target_absent (error)` on detach → `backup_target_restored (info)` on return for the TARGET, and `storage_disconnected (error)` → `storage_reconnected (info)` for a NON-target drive on the same box four minutes apart. **Two matched pairs, correctly discriminated — and discrimination is proven NON-trivially for the first time**, since both prior runs had the target itself emit the generic event. Gate fired in **3 s**; all four events reached the hub, so the specific alarm, its severity, its Hungarian copy and the hub routing are now exercised end-to-end. **Over-correction PASSES with a positive observable** (0 ABSENT lines / 0 drive events over 2m14s with both drives present, target `degraded:false`, while 2 `RETURNED` lines prove the gate was ticking). Ran on a nested PVE on **demo-hp** per `runbooks/target-selection.md` — through the **real day-0** from the v1.25.0 ISO, with the agent **installed unaided from the vouched Day-0 manifest** (published sha `b47c5c4dab641ee5…`, independent registry GET verified, manifest read back), drives enrolled through the real endpoints, device loss a real hot-detach. **THE FIX, and the ruling is the substantive part:** the mechanism was first isolated from the captured payload (`DIAG-r116-disks-payload-2026-07-30.md`) after two fixes aimed at shapes that do not occur. **Both smaller-looking options were REJECTED because they regress R-114** — `backup_target_offer.go:79` reads `BackupTarget && MountPath != ""` as *"a real drive with its own mountpoint — healthy"* and returns before its `TargetAbsent` branch, so back-filling `MountPath` on the Observe row **or** flagging the registry row (whose `MountPath` is the stale unit-file value) would have told the customer the backup target is fine while its drive was gone. **R-114's correctness was resting on R-116's bug** — a coupling invisible until the payload existed. Taken instead: the Observe row gets the **guest path only** (`mount_path` stays `""`, which is true) from a new `ConfigPath` (`json:"-"`, so the cross-repo golden + key-set contract is untouched), and the union row is deduped **on guest path** — the join being CONFIGURATION (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), the only identity that survives the device. Tests 845→849; 4 red-proofs each asserted to land, and red-proof 1 replays v0.115.0's code and fails, which is the empirical proof it was inert. Its green test had supplied a `MountPath` production never supplies AND left `DriveTargets` nil so the union loop never ran — both corrected. v0.115.0 left in place (inert, harmless). Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded. **Caveat: the drill's controller was 0.185.1 from the golden, which PREDATES R-114**, so its absent-state banner showed the old false copy — the golden being a release behind, not a regression → **R-120** | — | -| **R-120** | **The golden bakes a controller that predates R-114 + R-112, so a FRESH customer box shows the WRONG absent-target message.** The Day-0 golden is **0.185.1**; both R-114 (the third state — *configured and its drive is gone* vs *never configured*) and R-112 (the banner having any UI consumer at all) shipped in **0.186.0**. Observed live this run on the drill box with its target detached: the endpoint returned the generic degraded branch — *"A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer"* (**false** — the target is a drive that vanished) **plus** `offer_path: /mnt/felhom-drives/adat`, offering another drive as the remedy. That is E2D §5.3's exact payload, still reachable on any newly installed box | **READY (S)** | — | **This is R-115's class one layer over: baking is a remembered step.** R-111 was closed by re-baking the golden to 0.185.1; 0.186.0 then shipped and the golden was not moved, so the gap reopened silently — and this time the stale artifact carries a **customer-facing falsehood** in exactly the state R-116 now alarms about correctly. Fix is mechanical (bake + publish + vouch a 0.186.0 golden), but the decision that matters is whether the golden joins the release train or the manifest gains a gate that refuses to vouch a golden older than the newest controller — **reminders have already failed twice here** (R-111 → R-115). Evidence: `audits/R116-v0116-2026-07-30.md` §6a caveat + §1 | CC | +| **R-120** | **The golden baked a controller that predated R-114 + R-112, so a FRESH box showed the customer the WRONG absent-target message** | **GOLDEN REBAKED + PROVEN-LIVE (0.186.0, 2026-07-30); the MECHANISM half awaits an operator ruling** | operator ruling on gate-vs-train (Part 3) | **`audits/R120-golden-rebake-2026-07-30.md`.** Golden **0.186.0** baked from `main`'s controller in the DooPlex bake fixture (`build-golden.sh v2.1.0`: overlay2 OK, **3 mounts included**, FATAL 0, exclusions 0, 618 MB, upload **HTTP 201**, `GOLDEN_SHA256=b760ac6a33e70700…`; token-leak grep 0; GL-1 teardown and `drill.qcow2` back to `virgin`). Three observables: **published** — anonymous GET (what the installer does) 200 / 648930639 bytes / sha identical to the bake; **vouched** — manifest read BACK shows golden 0.186.0 + sha, agent 0.116.0 carried through, `min_agent` 0.113.0, wrapper sha no drift; **resolved by a consumer** — `Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)`. **Floor NOT touched** (`min_controller_version` still 0.156.0, and the floor is a separate form) per publish-train rule 2; MinAgent left 0.113.0 because 0.186.0 declares it unchanged. **PROVEN ON A REAL DAY-0, not the fixture** (per the Part-1 rule: a fixture proves a mechanism, only a fresh box proves a path): VM 9402 on demo-hp from the v1.25.0 ISO → `Controller elindult (0.186.0)`, box confirms `felhom-controller:0.186.0` + agent 0.116.0. **A fresh box now runs 0.186.0 where it ran 0.185.1 — the defect is closed.** The `TargetAbsent`/empty-`offer_path` endpoint capture is **NOT yet taken and not claimed**: the claim gate runs before auth (`auth.go:52-67`, exempting only 4 paths, no Bearer escape) and `regen-password` regenerates the *retrieval* passphrase, so an operator-relayed code is the only route — **R-119's fourth instance**. **The Part 3 mechanism is NOT built pending the ruling**; recommendation and its exact wiring are in §6 — a gate that **refuses** inside `hub/internal/web/configs.go:1116` `handleSetArtifacts` (the ONLY UI writer of `SetArtifactManifest`, `:1142`), comparing the submitted golden against the newest `controller_version` any box has reported (`store.go:294` — already local, no outbound call; it would have caught this case, since felhom-pve was reporting 0.186.0 against a 0.185.1 golden), tested through the production wiring rather than a seam. **If the ruling defers it, this row does NOT close** — it re-opens or a successor is filed naming the class *"a published artifact and its baked copy drift and nothing checks"*, because three instances closed individually (R-111, R-115, R-120) is how a class survives | CC | | **R-117** | **A returned drive's guest bind is a DEAD MOUNT, and the gate restarts the customer's apps onto it while every signal reads healthy.** After a detach/reattach the host raw mount is fine on the NEW device node, but the guest's bind still names the OLD one and every operation through it returns `EIO`: host `findmnt /mnt/cel` → `/dev/vdc`; guest `findmnt /mnt/felhom-drives/cel` → `/dev/vdb[/felhom-data] …,shutdown`; guest `ls` and write → **`Input/output error`**; and `/disks` reports `state:"attached"`, `role:"user-data"`, `bound_under_parent:TRUE`, `backup_target:true` | **READY (M) — NEW 2026-07-30, deliberately not chased** | — | **Worse than R-116 and it outranks it.** `planDriveGates` computes `present[gp] = … \|\| d.BoundUnderParent` (`intermediary.go:226`) = `true`, and the drive is `Disconnected`, so the gate takes the **`Return`** branch (`intermediary.go:281`, `:301-320`): it clears the flag, **restarts the gate-stopped apps onto a namespace that `EIO`s on every call**, reports the drive healthy and emails `backup_target_restored`. **No alarm on any channel.** R-113's conjunction cannot catch it — half 1 (guest mountinfo has an entry at the guest path) is satisfied by the STALE entry and half 2 (`devicePresent`) by the NEW device; **neither half compares the two**. Not a false invariant: `disks.go:158-165` claims only the boot-ordering and outlived-bind cases, so this is a genuine gap. **Seen but uninvestigated in E-2d, Session C and Part 5 (three consecutive runs) as "the drive returned as /dev/sdc while findmnt still read /dev/sdb"** — that framing read as cosmetic; it is not. Reproduced deliberately and probed to EIO in `audits/DIAG-r116-disks-payload-2026-07-30.md` §11. Deserves its own run | CC | | **R-118** | **An absent drive's union row advertises the ROOT filesystem's capacity as its own.** In the absent-state payload the registry-union row reports `total_bytes: 49675956224 / used_bytes: 4584579072` — **byte-identical to the `local` row** (`durable_id: path:/var/lib/vz`, i.e. `pve-root`) in the same response. The real drive is **4 GB** | **READY (XS) — NEW 2026-07-30** | — | Cause: `statfsCapacity(d.MountPath)` (`disks.go:335-338`) statfs's `/mnt/cel`, which with the device gone is a **bare directory on the root filesystem**. `observe.go:176-183`'s comment warns about exactly this trap and guards the Observe path ("*an unmounted removable dir-storage's mountpoint reverts to a bare directory on root … catastrophic DR mis-id*"); **the union path has no equivalent guard.** **Not a DR mis-id** — `durable_id` on that row is still the correct `uuid:…`, so re-attach identity is safe. It is a **false capacity** reaching every consumer of `total_bytes`/`used_fraction` (fill monitors, storage cards): a detached 4 GB drive advertises 46 GiB at 9.2 % used. Same class as `role.go:180-181` — an absent drive's fields decaying to the root filesystem's. Evidence: `audits/DIAG-r116-disks-payload-2026-07-30.md` §12 | CC | | **R-113** | ~~**The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.**~~ `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc//mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **SHIPPED + PROVEN-LIVE** (agent v0.114.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** Target drive hot-detached on a fresh box running the SHIPPED agent 0.114.0 (from the Day-0 manifest, not a hand build). **The gate fired in 4 seconds** — E-2d measured ZERO over 4½ minutes — and `SetDisconnected` was reached. It fired on exactly the shape that defeated it: raw `/mnt/mentes` NOT mounted while the bind `/mnt/felhom-drives/mentes` still read `/dev/sdb[/felhom-data]`. **Over-correction guard PASSED:** with both drives present, 0 ABSENT lines and the target stayed healthy. **Note: C5 still FAILED — but on a different defect (→ R-116), not on this one.** **SESSION C IS NOW UNBLOCKED.** Agent **0.114.0 published and vouched** 2026-07-29 (sha `5e4c15ebee2d7583…`, round-trip GET verified; hub manifest read back showing it) — a fresh drill box now installs the fix under test instead of the bug. The golden was NOT re-baked and `min_agent` was NOT raised: the golden bakes the controller, not the agent, and controller v0.185.0 declares MinAgent 0.113.0, which 0.114.0 already satisfies. **FIX: `BoundUnderParent` is now a CONJUNCTION — bound under the parent AND the drive's raw host mount still mounted** (`devicePresent`, new `deviceCheck` seam), at BOTH `/disks` construction sites. The raw mount is the device-bound systemd unit that dies with the device; the agent's bind is not — so the raw mount IS the device signal, grounded in E-2d's measurement rather than inference. **Conjunction, deliberately:** the device half alone would regress boot ordering (raw mounts early, bind lands ~18 s later — that window must keep reading absent), so existing behaviour is byte-identical and only the unreachable case is closed. **Unknown is never absent** (`devicePresent("")` = true) — a false absent stops a working customer's apps. **Controller UNCHANGED, no MinAgent bump:** `BoundUnderParent` has exactly one functional consumer (`planDriveGates:226`); a new `DevicePresent` bool was rejected because absent-from-JSON decodes to `false`, which would have made every drive on an older agent read ABSENT. +6 tests (208→214), 4 red-proofs run and reverted. **Deployed to demo-felhom; over-correction guard verified in production** — raw mount present, drive still reads present, 10/10 apps untouched, no gate action, no false alarm. **demo-hp deliberately left on 0.113.0** (the spec scoped deploy to felhom-pve) — it still carries the bug. **⚠️ SESSION C BLOCKER: the hub Day-0 manifest vouches agent 0.113.0**, so a fresh drill box would install WITHOUT this fix and validate nothing — publish + vouch 0.114.0 first (R-111's trap, same shape). **Awaiting live proof of exactly one leg:** device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. Original finding: **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 **Session C scope UNCHANGED by Session B** — R-114/R-112 shipped as controller v0.186.0 and neither touches the agent; the leg awaiting proof is still device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. One rebuild now validates all three. | CC |