diff --git a/documentation/audits/R116-v0116-2026-07-30.md b/documentation/audits/R116-v0116-2026-07-30.md new file mode 100644 index 0000000..0105ac6 --- /dev/null +++ b/documentation/audits/R116-v0116-2026-07-30.md @@ -0,0 +1,229 @@ +# R116-v0116-2026-07-30 — the fix works at the payload layer, live; the events leg is blocked on an emailed code + +**Run:** R-116 join task, CC on DooPlex, 2026-07-30. Agent **v0.116.0** built, published, vouched, and +**installed by a fresh box from the Day-0 manifest**. + +| Claim | Verdict | +|---|---| +| **The fix is effective live** — the absent target row carries the flag **and** the key the gate uses | ✅ **PASS**, on real hardware. `isTarget["/mnt/felhom-drives/cel"] = true` (was `false` through v0.115.0) | +| **R-114 not regressed** — no absent-state row combines the flag with a mount path | ✅ **PASS** | +| **No over-correction** — the absent row does not register presence | ✅ **PASS** (`bound_under_parent: false`) | +| **Discrimination at the payload layer** — non-target gets the flag on no row | ✅ **PASS** | +| **C5 — `backup_target_absent` / `_restored` events on the wire, paired** | ⛔ **NOT REACHED.** Blocked on the controller claim code (emailed only). See §6 | +| **Discrimination as events** | ⛔ **NOT REACHED**, same blocker. **Not reported as a pass** | + +## 1. Baselines as actually running + +| | `main` | running | a FRESH box gets | +|---|---|---|---| +| agent | **0.116.0** (`21b0164`) | felhom-pve 0.115.0 · demo-hp 0.113.0 · **drill `sess-e-5d4427` 0.116.0** | **0.116.0** (vouched this run) | +| controller | 0.186.0 (`b331f18`) | felhom-pve **0.186.0** · demo-hp **0.185.1** · drill **0.185.1** | golden bakes **0.185.1** | +| hub | 0.81.0 | 0.81.0 | `min_agent` 0.113.0 · `min_controller` 0.156.0 | +| `felhom.eu` | `1aa1bd1` | — | — | + +**The golden bakes controller 0.185.1** — which is why the fix had to land in the agent: a controller-side +fix would not reach a fresh box without a re-bake, whereas the agent channel serves the newest vouched +version immediately. Confirmed by this run: the drill box installed **0.116.0** unaided. + +## 2. §5's three publish observables, as returned + +``` +(1) registry, independent GET of the published bytes: + HTTP 200 bytes=14022347 + b47c5c4dab641ee57e2cf5893c2aebf1806a41c376ab06322342b02c86940a98 /tmp/rt-0.116.0 + /tmp/rt-0.116.0 --version → felhom-agent 0.116.0 +(2) hub Day-0 manifest, read BACK after the POST (not the 303): + agent_version: SELECTED=['0.116.0'] agent_sha256 = b47c5c4dab641ee5… + golden_version: SELECTED=['0.185.1'] min_agent = 0.113.0 + wrapper_sha256 = 104db0a4401f65bb… (re-checked against configs/felhom-pbs-apply — NO drift) +(3) the box under test reports it running: + hub log → Artifact manifest served for customer sess-e (agent=0.116.0 golden=0.185.1) + host row → sess-e-5d4427 … 0.116.0 ONLINE + on the box → felhom-agent --version → felhom-agent 0.116.0 +``` + +`min_agent` deliberately **not** raised: controller v0.185.0 declares MinAgent 0.113.0, and raising it +would hold demo-hp (0.113.0) for no reason. The golden was not moved — the controller is unchanged. + +## 3. The rig — real pipeline, no bypass + +**Machine: `demo-hp`**, per `runbooks/target-selection.md` (Tier 0, the designated drill+build VM host). +**This differs from the previous R-116 run, which used the DooPlex fixture** — the document says DooPlex is +Tier 2 and its `drill.qcow2` is a bake fixture, not a drill target, so this run followed the document. + +VM **9401** `r116-drill`: nested PVE, 8 GiB/4 vCPU cpu=host, OVMF, disks on a **dir storage +`r116-images` at `/mnt/nvme-1tb` root** — **not** `local-lvm` (over-subscribed thin pool hosting a live +guest). Installed from the **real v1.25.0 felhom ISO** already on demo-hp, through the **real day-0**: +self-register (appliance 13, pairing `D54-DG5`) → operator bind to `sess-e` → credentials delivered once → +guest 9201 provisioned from the vouched golden → `controller_started (0.185.1)` → reports flowing. + +Drives, both enrolled through the **real endpoints** (`/disks/format` → `/disks/assign` → +`/disks/guest-attach` → `/backup/target`), no hand-set state: + +| | device | host mount | guest bind | role | +|---|---|---|---|---| +| target | `/dev/sdb` (scsi1) | `/mnt/cel` | `/mnt/felhom-drives/cel` | primary backup target (`felhom-backup`, `is_mountpoint 1`) | +| non-target | `/dev/sdc` (scsi2) | `/mnt/adat` | `/mnt/felhom-drives/adat` | user-data only | + +`backup tier armed target=felhom-backup … primary=true` after the required restart. + +Device loss is a **real hot-detach** (`qm set 9401 --delete scsi1`); the volume survived as `unused0` and +was reattached, so the present-state control is re-runnable. + +## 4. The payloads — PASS, and this is what v0.115.0 could not do + +Host state at the absent capture, identical to every prior run: + +``` +ls /dev/sdb → No such file or directory +findmnt /mnt/cel → rc=1 (not mounted) +pvesm status → unable to activate storage 'felhom-backup' - directory is expected to be a + mount point but is not mounted: '/mnt/cel' + felhom-backup dir inactive 0 0 0 +``` + +### The target drive's row, all three states + +| state | rows | rows for the target | `mount_path` | `guest_path` | `backup_target` | `bound_under_parent` | `state` | `role` | +|---|---|---|---|---|---|---|---|---| +| PRESENT | 4 | **1** | `/mnt/cel` | `/mnt/felhom-drives/cel` | **`true`** | `true` | attached | user-data | +| **ABSENT** | 4 | **1** | **`""`** | **`/mnt/felhom-drives/cel`** | **`true`** | `false` | disconnected | system | +| RETURNED | 4 | **1** | `/mnt/cel` | `/mnt/felhom-drives/cel` | **`true`** | `true` | attached | user-data | + +**Exactly one row in every state** — the registry row for the target is deduped by guest path in the +absent state (`registry-row-for-target = 0` in all three captures). Before this fix the absent state +carried it **twice**. + +### `driveTargetByPath` computed from the captured absent payload + +``` +driveTargetByPath = {'/mnt/felhom-drives/cel': True, + '/mnt/felhom-drives/adat': False, '/mnt/adat': False} +isTarget[/mnt/felhom-drives/cel] = True ← was FALSE through v0.115.0 +``` + +`notifyDriveAbsent` therefore takes the **specific** branch. RETURNED gives `True` as well, so the pair +matches. + +### The three guards, from the same payload + +- **R-114 preserved.** No row has `backup_target: true` **and** `mount_path != ""`. The absent target row + reports `mount_path: ""`, so `backup_target_offer.go:79` does **not** match and R-114's `TargetAbsent` + branch stays reachable. This is why option (a) was rejected — see `felhom-agent/REPORT.md`. +- **No over-correction.** `bound_under_parent: false` on the absent row, so + `present[gp] = … || d.BoundUnderParent` stays false and the Stop branch is still reachable. +- **Discrimination, payload layer.** The non-target `adat` carries `backup_target` **absent ⇒ false** on + both its keys, in both states. So the two are now genuinely distinguishable — which is the thing both + prior runs could not show. + +### Raw payloads + +Absent (all 4 rows, verbatim): + +```json +{"ok":true,"data":{"disks":[{"name":"felhom-backup","type":"local-dir","state":"disconnected","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":0,"used_bytes":0,"used_fraction":0,"durable_id":"path:/mnt/cel","guest_attached":false,"backup_target":true,"guest_path":"/mnt/felhom-drives/cel","bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}},{"name":"local","type":"local","state":"attached","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":14484905984,"used_bytes":5096075264,"used_fraction":0.351819699045967,"durable_id":"path:/var/lib/vz","guest_attached":false,"bound_under_parent":false,"smart":{"health":"UNKNOWN","model_name":"QEMU QEMU HARDDISK","temperature_c":0,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}},{"name":"local-lvm","type":"lvmthin","state":"attached","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":12675186688,"used_bytes":4224639723,"used_fraction":0.33329999999129,"durable_id":"pve/data","guest_attached":false,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}},{"name":"d248508a-0c4f-466f-9c43-0dae988efc62","type":"usb","state":"attached","backing_device":"/dev/sdc","mount_path":"/mnt/adat","class":"","role":"user-data","data_bearing":false,"total_bytes":8350298112,"used_bytes":2125824,"used_fraction":0.00025458061155266215,"durable_id":"uuid:d248508a-0c4f-466f-9c43-0dae988efc62","guest_attached":false,"guest_path":"/mnt/felhom-drives/adat","bound_under_parent":true,"smart":{"health":"UNKNOWN","model_name":"QEMU QEMU HARDDISK","temperature_c":0,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}}],"guest_boot_id":"1785394603-28275","vmid":9201}} +``` + +Present and returned payloads: `d-PRESENT.json` / `d-RETURNED.json`, on the box at `/root/` and in the +session scratchpad. The target row differs from the absent one only in the fields tabulated above. + +## 5. Where R-117 interfered — and it did + +**Reproduced again, fourth consecutive run, and on real hardware this time:** + +``` +after reattach: host findmnt /mnt/cel → /dev/sdd + guest findmnt /mnt/felhom-drives/cel → /dev/sdb[/felhom-data] rw,relatime,shutdown + guest read through the bind → GUEST-READ-EIO + /disks says → state attached, bound_under_parent TRUE +``` + +**How it interferes with reading this run:** the reattach leg's `isTarget = true` and +`bound_under_parent = true` are *correct as answers to the questions this fix asks*, but they are read off +a drive whose guest namespace is **dead**. So "the drive came back healthy" cannot be concluded from the +reattach capture — only "the flag and the key rejoined on one row", which is what was under test. When the +events leg runs, the `Return` branch will fire against a dead bind, so a `backup_target_restored` there +proves pairing, **not** recovery. Recorded, not fixed (R-117 has its own row). + +## 6. Why C5 was not reached — the one genuine human step, again + +`planDriveGates` iterates `s.settings.GetStoragePaths()`. The drill controller has **none**: + +``` +controller log → [WARN] Storage paths: no storage paths registered +``` + +Registering one requires the controller's storage API, and **every** route is behind the claim gate: + +``` +GET /api/storage/backup-target (Host: felhom.sess-e.test) → 401 {"ok":false,"error":"dashboard not yet claimed"} +GET /api/storage/paths → 401 +GET /api/storage/list → 401 +``` + +The claim code is **bcrypt-hashed in the hub and only ever emailed** (`claim/engine.go`), and the +self-bind link is likewise mint-and-email — `handleSelfBindLinkSend` (`selfbind_mint.go:139-161`) only +ever renders a flash, never the token. **There is no operator-side route to either secret**, which is the +same wall E2D hit and named "the one genuine human step". + +A fresh claim code (**generation 2**) was emailed to the registered address at 09:07:20 by +`POST /configs/sess-e/claim-resend`, so the code is waiting; the run stops here rather than reporting a +third trivial pass. + +**Explicitly NOT done, and not claimed:** no `backup_target_absent` or `backup_target_restored` event was +observed on the wire this run, and the non-target mirror was not exercised as events. The payload evidence +in §4 determines what the gate *will* decide, but it is not the event on the wire, and after two runs that +conflated the two, that distinction is kept. + +## 7. Teardown — layers 1–3, per the §13 paragraph this task added + +**DELIBERATELY RETAINED, pending the claim code**, so the events leg can finish without rebuilding: + +| layer | item | disposition | +|---|---|---| +| 1 — machine | VM **9401** `r116-drill` on demo-hp (+ `unused0`) | **retained**, running | +| 2 — host | `r116-images` dir storage at `/mnt/nvme-1tb` | **retained** | +| 3 — hub | customer **`sess-e`**, host **`sess-e-5d4427`**, appliance **13** | **retained** | + +`pvesm status` on demo-hp **before** the run, for the layer-2 comparison at teardown: + +``` +felhom-backup dir active 983379700 4566008 928787080 0.46% +felhom-pbs pbs active 0 0 0 0.00% +local dir active 40516856 14961808 23464656 36.93% +local-lvm lvmthin active 56545280 21956532 34588747 38.83% ← the fence figure, unchanged +``` + +Teardown commands, recorded now so they cannot be forgotten: + +```bash +ssh demo-hp 'qm stop 9401; qm destroy 9401 --purge; pvesm remove r116-images; pvesm status' +# hub, once the host ages ONLINE→DOWN (customer_delete.go:220-228 refuses only on ONLINE): +POST /configs/sess-e/delete ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-e expect_hosts=1 +``` + +**Already clear, and not by this session:** `sess-c` and `sess-d` — both `/customers/` → **HTTP 404**, +absent from `/configs` and `/hosts`. The operator removed them using the commands the previous report +recorded. Verified directly rather than inferred. + +**Fences held:** guest 9201 on both demo boxes untouched · `drill-r50` (VM 300) untouched, `stopped` · +neither demo box re-targeted · nothing on `local-lvm` · Peti untouched · v0.115.0 not reverted · +the hub DB copy taken for the break-glass credential was `shred -u`'d immediately. + +## 8. Findings — filed, none fixed + +- **R-117** — reproduced on real hardware with the read/write probe (`EIO` both directions) while `/disks` + reports `attached` + `bound_under_parent: true`. Already filed; this run adds the metal-adjacent + reproduction and the note in §5 about how it colours the reattach leg. +- **R-118** — the union row's root-filesystem capacity. Its **symptom** is now absent in the target's + absent state because that row is suppressed; **R-118 is not fixed** and remains open for every other + registry-only drive (the non-target `adat` row still carries statfs values from its own live mount, so + the class is unchanged). +- **R-119 (new) — the claim gate makes the drive-gate legs unreachable to CC, by design, every time.** + Three sessions have now stopped at the same wall: the controller's storage API is claim-gated, the claim + code is emailed-only, and `planDriveGates` cannot act until a storage path is registered. This is not a + defect in the gate — it is correct customer-ownership — but it means **every** future validation of a + drive-gate behaviour costs an operator email round-trip. Worth a ruling: either a documented + operator-side test affordance (an operator-scoped claim, explicitly audit-logged), or accept the + round-trip and put it in the runbook as a **prerequisite step** rather than a mid-run surprise. Filed, + not designed. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index d57048b..bcd1219 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -16,7 +16,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-110** | **`main` is the installer's publish channel — there is no staging.** `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a 30 s period and nginx serves that working tree directly (`location /scripts/`, `root …/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as **root on a virgin box**, the single most privileged thing Felhom ships | **WAITING-ON-OPERATOR (S)** | operator ruling | **Two consequences worth stating:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" **was never available to take**. **Open question for the operator, not a defect to fix blind:** whether `/scripts/` should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether `main`-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now. **SECOND INSTANCE, found 2026-07-29 by the E-2d run and filed here rather than as a new ID:** `felhom-host-install.sh` fetches **nine** files from `raw/branch/main` (`:2072`–`:2206`) and the hub manifest vouches a sha for exactly **one** (`wrapper_sha256` → `felhom-pbs-apply`; re-checked this run, no drift). E-2a's `felhom-backup-target-apply` (`:2116`) is installed **0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n` — a root-executed artifact taken from `main` with no pinned integrity, which is this row's class exactly | CC | | **R-111** | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`.**~~ `felhom-host-install.sh` does not use `main`: it reads the hub-vouched manifest (`:423-436`) and fetches Gitea generic packages (agent `:1945`, golden `:2573`). Gitea's newest are **agent 0.96.0** and **golden 0.161.0**, and the hub's manifest selects exactly those — so a fresh box lands on **agent 0.96.0 + controller 0.161.0** (global floor `v0.156.0` < the golden's 0.161.0, so no self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and was never published** | **SHIPPED 2026-07-29 — the channel now serves agent 0.113.0 + golden 0.185.1** | — | **FIXED the same day it was found.** Agent **0.113.0** built from the clean tree @ `58b598b` and published (`scripts/publish-agent.sh`), sha `5f3247f756cb658e…`, round-trip GET verified. Golden **0.185.1** baked on the nested drill VM embedding controller `0.185.1`, published, sha `dba00f3e845c415e…` — bake clean: `Result=success`, overlay2, **all 3 mounts included** (rootfs+mp0+mp1), 0 FATAL/exclusions, upload HTTP 201, token-leak grep 0; log `drill/bake-0.185.1.log`; GL-1 teardown done (guest 9100 purged, secrets shredded, disk restored to `virgin`). Hub Day-0 manifest moved **both together in one POST** so it never vouched a new agent against an old golden; `min_agent` **0.93.0 → 0.113.0**, which is what controller v0.185.0 declares (`felhom-controller/CHANGELOG.md:15`) — **zero fleet impact, verified: all three enrolled hosts already run agent 0.113.0, so no box is held.** `wrapper_sha256` preserved verbatim (re-checked against `configs/felhom-pbs-apply` — no drift). **The global controller floor was deliberately NOT raised**: the golden now bakes 0.185.1, so a fresh box needs no self-update, and raising it would have been an unnecessary fleet-wide write. Original finding follows. **Found 2026-07-29 by the E-2d Phase 0 gate, which stopped the run before a VM was created.** 17 unpublished releases (v0.97.0–v0.113.0) strand the **entire R-82 tiered-backup arc** plus **F-CRIT-2** (a failed backup looking fresh — 7 days silent) and **F-REBOOT** (a guest rebooted mid-backup never returns): a new customer's box would install without them. **Blocks E-2d's C3/C4/C5** — those test endpoints and events that do not exist in 0.96.0/0.161.0. The **controller is fine** (registry has 0.185.1, floor-driven self-update), so the gap is specific to the two Gitea-generic artifacts. **Mirror of R-110, not a duplicate:** R-110 = the installer publishes instantly with no staging; R-111 = the agent/golden publish gate exists and was never walked. Fix should decide whether publishing joins the release train rather than staying a remembered step (R-29's shape, one layer up). Evidence: `audits/E2D-fresh-vm-2026-07-29.md` **DEFERRED LEG, AND IT RECURRED → R-115.** This row's shipped half stands and is not reopened: the bump happened, was verified, and was proven end-to-end by the E-2d install. But its own closing line — *decide whether publishing joins the release train rather than staying a remembered step* — was never acted on, and agent 0.114.0 reproduced the exact condition the same afternoon. The recurrence is filed as **R-115**, not as a reopen, because the stale-channel finding is closed while the process defect that caused it is a distinct problem with a distinct fix. | CC | | **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **WAITING-ON-OPERATOR (M)** | operator ruling on the release process | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design | CC | -| **R-116** | **The drive-absent alarm and its recovery are a MISMATCHED PAIR — absent fires the GENERIC `storage_disconnected`, return fires the SPECIFIC `backup_target_restored`.** `backup_target_absent` never fires at all (count 0 across a full Session-C run). An operator gets an alarm they cannot match to its recovery, which is exactly what `notifyDriveReturned`'s own comment forbids | **OPEN — MECHANISM ISOLATED 2026-07-30, fix NOT yet written** | nothing — READY (S) | **MECHANISM ISOLATED FROM THE REAL PAYLOAD (`audits/DIAG-r116-disks-payload-2026-07-30.md`).** The `/disks` read path is solved and written down verbatim: the token's plaintext lives ONLY in `/var/lib/felhom-agent/guests//bootstrap/bootstrap.json` **on the Proxmox host** (the agent's token store keeps hashes only, `tokenstore.go:26-32` — which is what defeated three prior extraction attempts); curl the island endpoint from the host itself. **Present-drive control run PASSED first** (felhom-pve, HTTP 200, 4 rows) so a 0-row query could not be mistaken for an empty payload. Absent state then staged on the existing DooPlex nested-PVE drill fixture with the **byte-identical live 0.115.0 binary** (`sha256 f48544ad…`) and every state-producing step through the real endpoints (format→assign→guest-attach→backup/target); device loss by QMP hot-unplug. **THE ANSWER IS THEORY #1 — "the registry-union row writes `false`" — the theory that was raised, declared wrong and RETRACTED. The retraction was the error.** In the absent state `/disks` returns **4 rows, not 3**: the drive appears TWICE and the two facts the controller needs sit on DIFFERENT rows. The Observe row (`felhom-backup`) has `backup_target:true` but `mount_path:""` **and** `guest_path:""` (role flipped to `system`; v0.115.0's fallback calls `StablePathForRaw("")` which returns `""` — `intermediary.go:69-75` — so **it assigns nothing, which is exactly why it is inert**), so it contributes NO key to `driveTargetByPath` at all. The registry-union row (`disks.go:297-339`) is **NOT deduped** — `seen` is keyed on `MountPath` (`:290-295`) and the Observe row's is now empty — and it owns both keys while **`BackupTarget` is absent from its struct literal (`:301-306`) ⇒ `false`**. Its `MountPath` survives the device because the union source is the **systemd `.mount` unit FILE** (`registry_known.go:40-75` via `main.go:605`→`:764`), which never consults the mount table. So `isTarget["/mnt/felhom-drives/cel"] = false` → generic; on return the two rows re-merge into one carrying both facts → specific. Verified by applying `intermediary.go:602-618` to the captured payload: PRESENT `True` / ABSENT `False` / RETURNED `True` — **the live asymmetry reproduced from payload alone.** **Theory #2 (the basis of v0.115.0) is FALSE on both halves** — a row DOES carry the guest path, and the union row is not deduped *because of* the very premise #2 was built on; **project-Claude's #3 is FALSE too** (`isTarget["/mnt/cel"]` is `false` as well — same row writes both keys). **FIX CONSTRAINTS, not a design** (see §10 of the diag): the necessary+sufficient condition is that the row owning `/mnt/felhom-drives/` reports `backup_target:true`; the site is the union struct literal; **trap — `d.Name` on a union row is the drive UUID, not the storage id, so a naive `d.Name == primaryTargetID` silently never matches**; do NOT extend v0.115.0 (there is no path value on that row to derive from — the unit file is the only source that survives, which argues for fixing the union row); keep `BoundUnderParent` false there so the gate cannot read absent as present; and **red-proof against §6.2's real shape, not Session C's fixture** (v0.115.0 was red-proofed against a shape that does not occur). v0.115.0 is provably INERT and harmless — keep-or-delete is the fix spec's call. **Original Part-5 record follows.** **PART 5 RE-VALIDATION FAILED 2026-07-30 (`audits/R116-PART5-2026-07-30.md`).** A fresh box running the FULLY SHIPPED stack — agent **0.115.0 from the Day-0 manifest** + controller 0.185.1 from the vouched golden — still fired the **generic** `storage_disconnected` on detach and the **specific** `backup_target_restored` on return. `backup_target_absent` count 0. Identical to Session C. **Discrimination FAILS** (the target itself produced the generic event, so the non-target leg was pointless to stage — reported as a fail, not as Session C's trivial pass). **Over-correction guard PASSES** (0 ABSENT lines with the drive present). **The v0.115.0 fix targets a shape that does not occur live, and WHICH shape does occur is NOT ISOLATED.** With the drive detached PVE reports the storage `inactive` with zeroed fields — a shape the unit fixture did not model. Three attempts to read the real `/disks` payload failed on token extraction, and **a present-drive control also returned 0 rows, proving the QUERY was broken, not the payload** — without that control this would have been a third false mechanism, after "the union row writes false" (wrong, corrected) and "no row carries the guest path" (unverified). **NEXT SESSION'S FIRST JOB: a working `/disks` read with a present-drive control run FIRST, before any code.** v0.115.0 is published, vouched and **inert**; not reverted (reverting is itself a change, and the runbook forbids fixing mid-run) | CC | +| **R-116** | **The drive-absent alarm and its recovery are a MISMATCHED PAIR — absent fired the GENERIC `storage_disconnected`, return the SPECIFIC `backup_target_restored`** | **FIX SHIPPED + PROVEN-LIVE AT THE PAYLOAD LAYER (agent v0.116.0, 2026-07-30); the EVENTS leg is the one open item** | an emailed controller claim code (→ R-119) | **v0.116.0 works live where v0.115.0 was inert** (`audits/R116-v0116-2026-07-30.md`). Built, published (sha `b47c5c4dab641ee5…`, independent registry GET verified), vouched in the Day-0 manifest, and **installed unaided by a fresh box** — `Artifact manifest served for customer sess-e (agent=0.116.0 golden=0.185.1)`, host `sess-e-5d4427 … 0.116.0 ONLINE`. Real day-0 on a nested PVE on **demo-hp** (per `runbooks/target-selection.md`, which sent this run to demo-hp rather than the DooPlex fixture the prior run used), drives enrolled through the real endpoints, device loss a real hot-detach. **CAPTURED LIVE: the absent target is now ONE row carrying `backup_target:true` AND `guest_path:/mnt/felhom-drives/cel` with `mount_path:""`, so `isTarget["/mnt/felhom-drives/cel"] = TRUE`** (it was `false` through v0.115.0) — the gate will take the specific branch, and RETURNED gives `true` too so the pair matches. All three guards pass from the same payload: **R-114 preserved** (no row combines the flag with a non-empty `mount_path`, so `backup_target_offer.go:79` does not match and the `TargetAbsent` branch stays reachable), **no over-correction** (`bound_under_parent:false`), **discrimination at the payload layer** (the non-target carries the flag on no row — the thing both prior runs could not show). **THE RULING, and it is the substantive output:** the spec's preferred option (back-fill `MountPath` on the Observe row) and its alternative (flag the registry row, whose `MountPath` comes from the stale unit file) **both regress R-114**, which shipped 2026-07-29 — either manufactures a row that `:79` reads as *healthy* while the drive is missing. R-114's correctness rested on the absent rows NOT combining those two fields, a coupling invisible until the payload existed. Taken instead: give the Observe row the **guest path only** (from a new `ConfigPath`, `json:"-"` so the cross-repo golden/key-set contract is untouched) and dedup the registry row **on guest path** — the join being CONFIGURATION (storage.cfg `path` vs the `.mount` unit's `Where`), the only identity that survives the device. Tests 845→849, 4 red-proofs each verified to land; red-proof 1 replays v0.115.0's code and fails, which is the empirical proof it was inert. Its green test had supplied a `MountPath` production never supplies AND left `DriveTargets` nil so the union loop never ran — both corrected. **STILL OPEN — the events on the wire.** `planDriveGates` iterates registered StoragePaths; the drill controller has none (`[WARN] Storage paths: no storage paths registered`) and every storage route answers `401 dashboard not yet claimed`. The claim code is bcrypt-hashed and emailed-only, and `handleSelfBindLinkSend` (`selfbind_mint.go:139-161`) renders a flash, never the token — **no operator-side route exists** (→ **R-119**). A gen-2 code was re-sent 09:07:20Z; the drill VM 9401, `r116-images` and customer `sess-e` are **deliberately RETAINED** with teardown commands recorded, so the leg finishes without a rebuild. **No `backup_target_absent` was seen on the wire this run and none is claimed.** v0.115.0 left in place (inert, harmless). R-118's symptom vanishes incidentally on this one row; R-118 NOT fixed | CC | | **R-117** | **A returned drive's guest bind is a DEAD MOUNT, and the gate restarts the customer's apps onto it while every signal reads healthy.** After a detach/reattach the host raw mount is fine on the NEW device node, but the guest's bind still names the OLD one and every operation through it returns `EIO`: host `findmnt /mnt/cel` → `/dev/vdc`; guest `findmnt /mnt/felhom-drives/cel` → `/dev/vdb[/felhom-data] …,shutdown`; guest `ls` and write → **`Input/output error`**; and `/disks` reports `state:"attached"`, `role:"user-data"`, `bound_under_parent:TRUE`, `backup_target:true` | **READY (M) — NEW 2026-07-30, deliberately not chased** | — | **Worse than R-116 and it outranks it.** `planDriveGates` computes `present[gp] = … \|\| d.BoundUnderParent` (`intermediary.go:226`) = `true`, and the drive is `Disconnected`, so the gate takes the **`Return`** branch (`intermediary.go:281`, `:301-320`): it clears the flag, **restarts the gate-stopped apps onto a namespace that `EIO`s on every call**, reports the drive healthy and emails `backup_target_restored`. **No alarm on any channel.** R-113's conjunction cannot catch it — half 1 (guest mountinfo has an entry at the guest path) is satisfied by the STALE entry and half 2 (`devicePresent`) by the NEW device; **neither half compares the two**. Not a false invariant: `disks.go:158-165` claims only the boot-ordering and outlived-bind cases, so this is a genuine gap. **Seen but uninvestigated in E-2d, Session C and Part 5 (three consecutive runs) as "the drive returned as /dev/sdc while findmnt still read /dev/sdb"** — that framing read as cosmetic; it is not. Reproduced deliberately and probed to EIO in `audits/DIAG-r116-disks-payload-2026-07-30.md` §11. Deserves its own run | CC | | **R-118** | **An absent drive's union row advertises the ROOT filesystem's capacity as its own.** In the absent-state payload the registry-union row reports `total_bytes: 49675956224 / used_bytes: 4584579072` — **byte-identical to the `local` row** (`durable_id: path:/var/lib/vz`, i.e. `pve-root`) in the same response. The real drive is **4 GB** | **READY (XS) — NEW 2026-07-30** | — | Cause: `statfsCapacity(d.MountPath)` (`disks.go:335-338`) statfs's `/mnt/cel`, which with the device gone is a **bare directory on the root filesystem**. `observe.go:176-183`'s comment warns about exactly this trap and guards the Observe path ("*an unmounted removable dir-storage's mountpoint reverts to a bare directory on root … catastrophic DR mis-id*"); **the union path has no equivalent guard.** **Not a DR mis-id** — `durable_id` on that row is still the correct `uuid:…`, so re-attach identity is safe. It is a **false capacity** reaching every consumer of `total_bytes`/`used_fraction` (fill monitors, storage cards): a detached 4 GB drive advertises 46 GiB at 9.2 % used. Same class as `role.go:180-181` — an absent drive's fields decaying to the root filesystem's. Evidence: `audits/DIAG-r116-disks-payload-2026-07-30.md` §12 | CC | | **R-113** | ~~**The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.**~~ `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc//mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **SHIPPED + PROVEN-LIVE** (agent v0.114.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** Target drive hot-detached on a fresh box running the SHIPPED agent 0.114.0 (from the Day-0 manifest, not a hand build). **The gate fired in 4 seconds** — E-2d measured ZERO over 4½ minutes — and `SetDisconnected` was reached. It fired on exactly the shape that defeated it: raw `/mnt/mentes` NOT mounted while the bind `/mnt/felhom-drives/mentes` still read `/dev/sdb[/felhom-data]`. **Over-correction guard PASSED:** with both drives present, 0 ABSENT lines and the target stayed healthy. **Note: C5 still FAILED — but on a different defect (→ R-116), not on this one.** **SESSION C IS NOW UNBLOCKED.** Agent **0.114.0 published and vouched** 2026-07-29 (sha `5e4c15ebee2d7583…`, round-trip GET verified; hub manifest read back showing it) — a fresh drill box now installs the fix under test instead of the bug. The golden was NOT re-baked and `min_agent` was NOT raised: the golden bakes the controller, not the agent, and controller v0.185.0 declares MinAgent 0.113.0, which 0.114.0 already satisfies. **FIX: `BoundUnderParent` is now a CONJUNCTION — bound under the parent AND the drive's raw host mount still mounted** (`devicePresent`, new `deviceCheck` seam), at BOTH `/disks` construction sites. The raw mount is the device-bound systemd unit that dies with the device; the agent's bind is not — so the raw mount IS the device signal, grounded in E-2d's measurement rather than inference. **Conjunction, deliberately:** the device half alone would regress boot ordering (raw mounts early, bind lands ~18 s later — that window must keep reading absent), so existing behaviour is byte-identical and only the unreachable case is closed. **Unknown is never absent** (`devicePresent("")` = true) — a false absent stops a working customer's apps. **Controller UNCHANGED, no MinAgent bump:** `BoundUnderParent` has exactly one functional consumer (`planDriveGates:226`); a new `DevicePresent` bool was rejected because absent-from-JSON decodes to `false`, which would have made every drive on an older agent read ABSENT. +6 tests (208→214), 4 red-proofs run and reverted. **Deployed to demo-felhom; over-correction guard verified in production** — raw mount present, drive still reads present, 10/10 apps untouched, no gate action, no false alarm. **demo-hp deliberately left on 0.113.0** (the spec scoped deploy to felhom-pve) — it still carries the bug. **⚠️ SESSION C BLOCKER: the hub Day-0 manifest vouches agent 0.113.0**, so a fresh drill box would install WITHOUT this fix and validate nothing — publish + vouch 0.114.0 first (R-111's trap, same shape). **Awaiting live proof of exactly one leg:** device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. Original finding: **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 **Session C scope UNCHANGED by Session B** — R-114/R-112 shipped as controller v0.186.0 and neither touches the agent; the leg awaiting proof is still device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. One rebuild now validates all three. | CC |