From e70b5feebeaa23aed855dda9a296fa14583078a6 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 30 Jul 2026 11:49:59 +0200 Subject: [PATCH] =?UTF-8?q?docs(R-117):=20the=20hang=20case=20measured=20?= =?UTF-8?q?=E2=80=94=20an=20I/O=20probe=20turns=20a=20wedged=20drive=20int?= =?UTF-8?q?o=20an=20unkillable=20agent?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Completes the spike once the venue came back. Q4's hang case and teardown are now measurements, not plans. Against a dmsetup-suspended device (I/O queues instead of returning EIO): - P1 (devno compare) and P2 (ext4 abort flags) completed in 364us / 206us. They read /proc, so no block device is involved. - statfs and getdents completed and reported HEALTHY — on a wedged device they do not even hang. R-117b confirmed in a second failure mode. - EVERY probe that touches the device blocked, including a buffered write with no fsync: the O_CREAT metadata path needs journal access (wchan=do_get_write_access). There is no cheap-and-safe write probe. - The blocked process survived SIGTERM AND SIGKILL (stat=D, wchan=folio_wait_bit_common, still alive 3m50s after kill -9) and died only when the device was resumed. So `systemctl restart felhom-agent` would hang, leaving the agent unrecoverable until the device returns or the host reboots. The thread count does not reveal the leak (5->5, 5->6). Filed as R-117f. A timeout protects the caller's control flow and nothing else, so "the fix must issue no block I/O" is now a fence rather than a preference — the thread-leak hypothesis the probes were built to test turned out to be the weaker half of the result. Teardown done, all three layers: guest 9301 destroyed, r117scratch removed, both dm and both loop devices gone, scsi_debug unloaded, local back to 37.02% against a 37.00% session start. Fences re-verified AFTER teardown: 9201 running, drill-r50 stopped, local-lvm 38.84% byte-identical, felhom-backup content unchanged, live /mnt/felhom-drives intact with both submounts, agent active. Layer 3 genuinely empty — 9301 had no NIC and ran no controller. Trap recorded: a suspended dm device must be resumed BEFORE any umount, or the teardown blocks on the same uninterruptible sleep. --- REPORT-r117-spike.md | 56 ++++-- .../SPIKE-r117-bind-liveness-2026-07-30.md | 180 ++++++++++++------ documentation/backlog/OPEN-ITEMS.md | 2 +- 3 files changed, 161 insertions(+), 77 deletions(-) diff --git a/REPORT-r117-spike.md b/REPORT-r117-spike.md index ad2f9dd..f57ab01 100644 --- a/REPORT-r117-spike.md +++ b/REPORT-r117-spike.md @@ -27,7 +27,7 @@ the source of a behavioural claim — every predicate result came from a probe b ## Results -All seven questions answered empirically except the hang case (below). +**All seven questions answered empirically.** - **Q1 — reproduced 3/3**, two device classes, on a purpose-built scratch LXC (9301) on demo-hp. **The device-node change is a consequence of the defect, not a precondition** — control test: with @@ -39,8 +39,9 @@ All seven questions answered empirically except the hang case (below). - **Q3 — confirmed at source and live.** Both halves of the R-113 conjunction compare **field 5** of a mountinfo line and **never read field 3 (`major:minor`)**, so neither can see that the bind and the raw mount name different devices. Measured `BoundUnderParent = TRUE` over an `EIO` namespace. -- **Q4 — a pure-`/proc` check costs 0.17–0.45 ms**, cannot hang, spins up no disk, writes nothing. - **`statfs` and `getdents` both SUCCEED on a dead namespace** — probes built on either are hollow. +- **Q4 — a pure-`/proc` check costs 0.16–0.45 ms**, cannot hang, spins up no disk, writes nothing. + **`statfs` and `getdents` both SUCCEED on a dead namespace** — probes built on either are hollow. The + hang case is below; it is the sharpest result in the run. - **Q5 — the agent**, and not on balance: the controller runs inside the guest and cannot see the host mount tables the check needs. - **Q6 — recovery works in place, guest never restarted** (init PID identical). **The repair code @@ -49,26 +50,47 @@ All seven questions answered empirically except the hang case (below). - **Q7 — a bind can die in steady state, no cycle at all.** The gate produces no action and **nothing is emitted on any channel.** A `Return`-branch fix cannot reach this half. -## Not established +## Q4's hang case — measured, and it is the sharpest result -**The hang case (findings §6.3).** The venue (`dmsetup suspend --noflush`) and the two blocking probes -(write+`fsync`, cache-dropped read, both with timeout-fires and thread-leak reporting) were built; the -run did not happen — the venue lost internet mid-transfer of the rebuilt binary. **The associated -thread-leak hypothesis is explicitly not claimed as a result.** It does not gate the `/proc` -recommendation but does gate adding any I/O confirmation probe. +Against a `dmsetup suspend`ed device (I/O queues instead of returning `EIO`): -## Teardown — OWED +- **P1 and P2 completed in 364 µs / 206 µs.** They read `/proc`, so no block device is involved. +- **`statfs` and `getdents` completed and reported HEALTHY** — on a wedged device they do not even hang. +- **Every probe that touches the device blocked, including a buffered write with no `fsync`** — the + `O_CREAT` metadata path needs journal access (`wchan=do_get_write_access`). There is no cheap-and-safe + write probe. +- **The blocked process survived `SIGTERM` and `SIGKILL`** (`stat=D`, still alive 3m50s after `kill -9`) + and died only when the device was resumed. So **`systemctl restart felhom-agent` would hang**, leaving + the agent unrecoverable until the device returns or the host reboots. The thread count does not reveal + the leak (5→5, 5→6). -**Not performed** — the second site outage began first. Findings §14 lists the exact commands. Nothing -fenced was touched at any point (9201 `running`, `drill-r50` `stopped`, `local-lvm` untouched, -`felhom-backup` not re-targeted, `/mnt/felhom-drives` untouched, ep0 read-only). **Layer 3 is -genuinely empty** — guest 9301 had no network interface and ran no controller, so no hub-side record -was created. +**A timeout protects the caller's control flow and nothing else.** This turns "prefer a cheap probe" into +a fence: **the fix must issue no block I/O.** + +## Teardown — done, all three layers + +Guest 9301 destroyed; `r117scratch` removed; both dm devices and both loop devices gone; `scsi_debug` +unloaded (`/dev/sd*` back to `sda1..3`); no `r117` mounts, `/mnt` and `/root` exactly as found; `local` +**37.02 %** against a session-start **37.00 %**. Fences re-verified *after* teardown: 9201 `running`, +`drill-r50` `stopped`, `local-lvm` **38.84 % byte-identical**, `felhom-backup` `content backup` +unchanged, live `/mnt/felhom-drives` intact with both submounts, agent service `active`. **Layer 3 is +genuinely empty** — 9301 had no network interface and ran no controller, so no hub-side record was ever +created. + +**Ordering trap worth keeping:** a suspended dm device must be `dmsetup resume`d *before* any `umount`, +or the teardown itself blocks on the same uninterruptible sleep. + +## Not measured + +No load or duration testing of the recommended check — P1/P2 were single calls, not a sustained +reconcile loop on a many-drive box. Nothing suggests a problem (they are two `/proc` reads the code +already performs), but it was not measured. ## Findings filed, none fixed R-117 (mechanism + recipe), **R-117a** steady-state death with no event (HIGH, larger than R-117 as filed), **R-117b** `statfs`/`getdents` are hollow liveness probes, **R-117c** three untested comments promising "live + usable in the guest", **R-117d** the self-heal that already exists is short-circuited -(HIGH), **R-117e** both demo boxes share one failure domain — no route survives the site losing -internet, including the WireGuard OOB path. +(HIGH), **R-117e** both demo boxes share one failure domain — no route survives the site losing internet, +including the WireGuard OOB path, **R-117f** an I/O liveness probe turns a wedged drive into an +unkillable agent (HIGH — disqualifies a whole probe class). diff --git a/documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md b/documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md index 35f07c4..e7f4826 100644 --- a/documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md +++ b/documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md @@ -12,7 +12,7 @@ Output is this document. Findings are filed as `R-n`, none fixed. | Q1 | Reproduce deliberately | **YES — 3/3 attempts, two independent device classes.** A device-node change is **NOT a precondition — it is a CONSEQUENCE** of the stale bind, proven by a control test | | Q2 | What is dead, at which layer | The **guest bind and the host bind** (same mount, one peer group); the **raw host mount is healthy on the new device**. **Two distinct death states** with different errnos — `EIO`(5) both directions, and `EROFS`(30) on write only | | Q3 | Why `boundUnderParent` returns true | It is a **string comparison of field 5 of a mountinfo line**. It never touches the filesystem, and **neither half reads field 3 (the device number)** — so neither can see that the two entries name different devices | -| Q4 | Cost + failure modes of a real check | A **pure-`/proc` check costs 0.19–0.45 ms, cannot hang, cannot spin up a disk and writes nothing.** The two cheapest *I/O* probes — `statfs` and `getdents` — **succeed over a fully dead filesystem** and are worthless. The hang case was **NOT measured** (§6.3) | +| Q4 | Cost + failure modes of a real check | A **pure-`/proc` check costs 0.16–0.45 ms, cannot hang, cannot spin up a disk and writes nothing.** The two cheapest *I/O* probes — `statfs` and `getdents` — **succeed over a fully dead filesystem** and are worthless. **Every probe that reaches the device blocks on a wedged one, and the process then cannot be killed even with `SIGKILL`** (§6.3) | | Q5 | Where the check belongs | **The agent**, and not on balance — the controller **cannot** perform it: the inputs are host mount tables it has no access to | | Q6 | In-place recovery possible | **YES — proven, guest never restarted** (init PID identical across the repair). And **the repair code already exists, is already invoked at exactly the right moment, and is short-circuited by the same false predicate** | | Q7 | Scope — steady state too? | **YES, and this is the worse half.** A drive can die **with no detach/return cycle at all**; the gate then never runs, the `Return` branch never fires, and **no event of any kind is emitted** | @@ -358,9 +358,9 @@ All timings from the R-117 state on drive B, on the venue, one run each, `-timeo | **P2 ext4 abort options** | `shutdown` **or** `emergency_ro` in the super options — **pure `/proc`** | **DEAD=true** ✅ | **DEAD=true** ✅ | false ✅ | **0.17–0.24 ms** | **Catches both.** ext4-specific | | P3 `statfs` | one syscall | **OK — MISSES IT** ❌ | OK — misses ❌ | OK | 0.042 ms | **Worthless** | | P4 `getdents` | read one dir entry | **OK — MISSES IT** ❌ | OK — misses ❌ | OK | 0.045–0.061 ms | **Worthless** | -| P5 write + unlink | create, write 1 byte, remove | `EIO`(5) ✅ | `EROFS`(30) ✅ | OK | 0.042–0.152 ms | Catches both, **but writes**, and cannot distinguish a legitimately-`ro` mount from a dead one | -| P6 write + **fsync** | forces the byte to the device | **not measured** (§6.3) | — | — | — | **The probe that can hang** | -| P7 read with page cache dropped (`POSIX_FADV_DONTNEED`) | forces a read to reach the device | **not measured** (§6.3) | — | — | — | **The probe that can hang** | +| P5 write + unlink | create, write 1 byte, remove | `EIO`(5) ✅ | `EROFS`(30) ✅ | OK | 0.042–0.152 ms | Catches both, **but writes**, cannot distinguish a legitimately-`ro` mount from a dead one, **and BLOCKS on a wedged device** (§6.3) | +| P6 write + **fsync** | forces the byte to the device | — | — | OK, **11.4 ms** | **11.4 ms healthy** | **BLOCKS on a wedged device.** 25–70× the `/proc` probes even when healthy, on a loop device with no seek cost | +| P7 read with page cache dropped (`POSIX_FADV_DONTNEED`) | forces a read to reach the device | — | — | OK, 0.18 ms | 0.18 ms | **BLOCKS on a wedged device** | `GuestSeesMount` itself costs **3.85–5.03 ms**, an order of magnitude more than any probe above, because it forks `lxc-info`. **Any of P1/P2 is free relative to the check it would strengthen.** @@ -380,26 +380,75 @@ because it forks `lxc-info`. **Any of P1/P2 is free relative to the check it wou This is the strongest argument for the pure-`/proc` pair over anything else, independent of the hang question. -### 6.3 The hang case — NOT ESTABLISHED +### 6.3 The hang case — MEASURED, and it is worse than "a probe that hangs" -**This is the one question in the brief that this run does not answer, and the gap is real.** +Venue: the `dm-linear` drive suspended with `dmsetup suspend --noflush --nolockfs`, which **queues all +I/O indefinitely instead of returning `EIO`** — the controlled "half-dead or spun-down device" the brief +asks about. Verified suspended (`dmsetup info -o attr` → `L-sw`, the `s`), then the full probe set with +`-timeout 3s`, then `dmsetup resume`. -Designed and built, not run: a `dm-linear` device suspended with `dmsetup suspend --noflush`, which -queues all I/O indefinitely instead of returning `EIO` — the controlled "half-dead or spun-down device" -the brief asks about. `r117probe` was extended with **P6 (write + `fsync`)** and **P7 (read with the page -cache dropped)** — the two probes that must reach the platter and therefore can block — plus reporting of -whether the blocked goroutine **ever returns** and the process's OS-thread count before and after. +``` +=== PROBES against a WEDGED (not erroring) device, timeout 3s === +HALF1 GuestSeesMount = true [3.983519ms] +HALF2 isHostMountpoint = true [234.181µs] +=====> BoundUnderParent = true +P1 devno-compare raw="252:12" bind="252:12" -> STALE=false [363.878µs] <-- COMPLETED +P2 ext4-abort-opts superopts="rw" -> DEAD=false [206.104µs] <-- COMPLETED +P3 statfs COMPLETED in 40µs err= <-- reports HEALTHY +P4 getdents COMPLETED in 64µs err= <-- reports HEALTHY +P6 write+fsync TIMED OUT after 3.002s — caller proceeds; goroutine still blocked +P6 ...still blocked 8s in; threads 5 -> 5 (LEAKED — the syscall is uninterruptible) +P7 read-uncached TIMED OUT after 3.002s — caller proceeds; goroutine still blocked +P7 ...still blocked 8s in; threads 5 -> 5 (LEAKED) +P5 write+unlink TIMED OUT after 3.002s — caller proceeds; goroutine still blocked +P5 ...still blocked 8s in; threads 5 -> 6 (LEAKED) +``` -**It was not run: the second house-internet outage (§2.5) began while the rebuilt 3.6 MB binary was being -copied to the venue.** The build succeeded locally; the transfer did not complete. +**Four results, in ascending order of severity.** -Not measured, therefore not claimed. What is *known* from what did run: +1. **P1 and P2 completed normally, in microseconds.** They read `/proc/self/mountinfo`, which involves + no block device. **This is the whole argument for the §11 recommendation**: on a wedged device the + pure-`/proc` check is not merely faster, it is the only one that returns at all. +2. **P3 and P4 completed *and reported healthy*.** On a wedged device they do not even have the decency + to hang — they lie, exactly as they do on a dead one (§6.1). Confirms **R-117b** in a second failure + mode. +3. **Every probe that touches the device blocked — including P5.** P5 is a buffered `write` with no + `fsync`, which was *expected* to complete out of page cache. It did not: the `O_CREAT` metadata path + needs journal access (`wchan=do_get_write_access`). **There is no such thing as a cheap write probe + that is safe on a wedged device.** +4. **The process could not be killed — not by `SIGTERM`, not by `SIGKILL`.** This is the finding that + changes the recommendation from a preference into a fence: -- **P1 and P2 cannot hang** — they read `/proc/self/mountinfo`, which involves no block device. This is - a property of the files they read, and it is why the recommendation in §8 rests on them. -- The reasoning that a timeout around a blocked uninterruptible syscall leaves a **leaked thread per - poll** — and that at the reconcile ticker's 20-second cadence (`main.go:973`) that is 180 per hour — - is the hypothesis P6/P7 were built to test. **It is untested. It must not be quoted as a result.** +``` +# after `timeout 200` had already sent SIGTERM: + PID STAT WCHAN ELAPSED COMMAND +2493045 Dl folio_wait_bit_common 03:20 r117probe + tid 2493045 stat=D wchan=folio_wait_bit_common + tid 2493049 stat=D wchan=do_get_write_access + +# explicit SIGKILL: +kill -9 2493045 +after SIGKILL: 2493045 Dl folio_wait_bit_common 03:50 <-- STILL ALIVE + +# only the device recovering released it: +dmsetup resume r117cel2 -> probe still alive? no — it died once I/O completed +``` + +**A timeout protects the caller's control flow and nothing else.** The goroutine and its OS thread are +consumed permanently, and the *process* enters uninterruptible sleep. Applied to the agent, whose +reconcile ticker runs every 20 s (`main.go:973`): + +- the report loop's caller proceeds, so the timeout "works" — and the evidence of the problem is a + thread you cannot see and cannot reclaim; +- **`systemctl restart felhom-agent` would hang**, because `SIGKILL` does not reach a `D`-state thread. + The agent becomes unrecoverable **until the device recovers or the host reboots** — on a box whose + whole purpose is to be recovered remotely; +- the thread count is **not** a reliable tell (5→5, 5→5, 5→6): the Go runtime reuses threads, so the + leak does not announce itself in the one metric an operator might watch. + +> **This disqualifies I/O probing on this path outright.** Not "use a timeout", not "use it carefully" — +> a liveness probe that issues block I/O converts a dead drive into an unkillable agent. §11's +> recommendation issues none. --- @@ -623,9 +672,12 @@ no write, and no possibility of blocking the report loop.** Effectively free. promising a property nothing currently tests; the fix owes one that pins it. 6. **Q7 needs its own scenario.** A test that only exercises detach→return will pass while the silent half stays broken. The in-place recipe in §10 is the second scenario. -7. **The hang question is open** (§6.3). It does not gate this recommendation — P1 and P2 read `/proc` - and cannot block — but it **does** gate any proposal to add an I/O confirmation probe on top. Settle - §6.3 before accepting one. +7. **Do not add an I/O confirmation probe. Measured, not preferred** (§6.3). On a wedged device every + probe that touches it blocks — including a buffered write with no `fsync` — and the process then + **cannot be killed by `SIGKILL`**, so the agent could not be restarted until the device recovered or + the host rebooted. A timeout protects the caller's control flow and nothing else. **P1 and P2 both + returned in microseconds on that same wedged device**; the recommendation issues no block I/O, and + that is now a fence rather than a preference. --- @@ -662,7 +714,8 @@ no write, and no possibility of blocking the report loop.** Effectively free. |---|---|---|---| | **R-117** | *(existing row, now with a mechanism and a recipe)* The bind-liveness signal is a path-presence test; the R-117 state is `BoundUnderParent = true` over an `EIO` namespace | **HIGH** | §3, §5 | | **R-117a** | **A drive's bind can die in STEADY STATE, with no detach/return cycle, and no event is emitted on any channel.** The gate produces neither action, so a `Return`-branch fix cannot reach it. **P1 does not detect it; only the filesystem's abort flag does** | **HIGH — larger than R-117 as filed** | §9 | -| **R-117b** | **`statfs` and `getdents` both SUCCEED on a namespace returning `EIO` for all content.** Any liveness probe built on either is hollow by construction — it reports healthy in exactly the state it exists to detect | **MEDIUM (design trap)** | §4, §6.1 | +| **R-117b** | **`statfs` and `getdents` both SUCCEED on a namespace returning `EIO` for all content** — and also on a **wedged** one, where they do not even hang. Any liveness probe built on either is hollow by construction — it reports healthy in exactly the state it exists to detect | **MEDIUM (design trap)** | §4, §6.1, §6.3 | +| **R-117f** | **A liveness probe that issues block I/O turns a wedged drive into an UNKILLABLE AGENT.** On a suspended device every I/O probe blocked — including a buffered write with no `fsync` — and the process survived `SIGTERM` **and `SIGKILL`**, dying only when the device recovered. `systemctl restart felhom-agent` would therefore hang, leaving the agent unrecoverable until the device returns or the host reboots. **A timeout protects the caller's control flow and nothing else**, and the thread count does not reveal the leak (5→5, 5→6) | **HIGH (constraint on the fix — it disqualifies an entire probe class)** | §6.3 | | **R-117c** | **Three comments** (`disks.go:83`, `:154-155`, `:155-165`) state that `BoundUnderParent` means "live + usable in the guest". **Nothing tests it, and it is false in the measured state.** Not counted as a shipped false invariant because `:158-165` scopes its own claim narrowly — but the summary lines do not | **MEDIUM** | §5.3 | | **R-117d** | **`AttachDrive` logs and returns `"fully live"` for a namespace that is `EIO` on every call** (`intermediary.go:235`), disabling all three wired repair paths including the controller's pre-restart re-attach | **HIGH — this is the self-heal that already exists** | §8.2 | | **R-117e** | **The two demo boxes share one uplink and one failure domain.** Both went unreachable together, twice in one session, on all four known routes including the WireGuard OOB path via ep0. There is no route to either box that survives the site losing internet — the "out-of-band" path is not out-of-band with respect to this failure | **LOW (operational, not a code defect)** | §2.5 | @@ -674,50 +727,57 @@ about.** Whoever fixes R-117 will need that box on a current agent to validate l --- -## 14. Teardown — **OWED, NOT YET DONE** +## 14. Teardown — DONE, all three layers -**The second internet outage (§2.5) began before teardown. This section is a liability, not a record.** -Everything below is inert scratch on a Tier 0 box; nothing fenced was touched at any point (§14.2). +Run after the venue came back; the site outage delayed it, it did not skip it. +**Ordering note, recorded because it is a real trap:** any suspended dm device must be `dmsetup resume`d +**before** any `umount`, or every unmount in the teardown blocks on the wedged device — the same +uninterruptible-sleep mechanism as §6.3. Both devices were verified `attr=L--w` (not suspended) first. -### 14.1 Outstanding on demo-hp — the exact commands +### 14.1 Layer 1 — the machine -```bash -# layer 1 — the machine -pct stop 9301 && pct destroy 9301 --purge -# layer 2 — the host -umount /mnt/r117-drives/sd /mnt/r117-drives/cel 2>/dev/null -systemctl stop mnt-r117sd.mount mnt-r117cel.mount -umount /mnt/r117-drives # the scratch shared parent self-bind -dmsetup remove r117cel r117cel2 # now unheld once the binds are gone -losetup -d /dev/loop0 -modprobe -r scsi_debug # removes /dev/sdb (the fake disk) entirely -pvesm remove r117scratch && rm -rf /var/lib/vz/r117-scratch -rmdir /mnt/r117-drives/sd /mnt/r117-drives/cel /mnt/r117-drives /mnt/r117cel /mnt/r117sd -rm -rf /root/r117 /root/setup1.sh /root/setup2.sh /root/setup3.sh \ - /root/capture.sh /root/capture2.sh /root/r117probe /root/r117-pveam.log -# optional: rm /var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst (124 MB, downloaded -# this run; harmless to keep and useful for the next scratch guest) -# then verify: pct list | pvesm status | dmsetup ls | losetup -a | lsmod | grep scsi_debug | findmnt /mnt/r117-drives +``` +BEFORE pct list -> 9201 running demo-hp / 9301 running r117 + pct stop 9301 ; pct destroy 9301 --purge ("purging CT 9301 from related configurations..") +AFTER pct list -> 9201 running demo-hp <-- only the pre-existing guest ``` -### 14.2 Layer 3 — the hub: **nothing to dispose of** +### 14.2 Layer 2 — the host, with the space actually returned + +| | Before | After | +|---|---|---| +| `pvesm status` | `r117scratch` **active**, `local` **38.97 %** | `r117scratch` **removed**, `local` **37.02 %** *(session start: 37.00 %)* | +| `df -h /var/lib/vz` | 16G used, **42 %** | 15G used, **40 %** *(session start: 40 %)* | +| `dmsetup ls` | `r117cel (252:11)`, `r117cel2 (252:12)` + 11 pre-existing | **both gone**; the 11 pre-existing `pve-*` unchanged | +| `losetup -a` | `/dev/loop0` (cel.img), `/dev/loop1` (9301 rootfs) | **empty — no loop devices** | +| `scsi_debug` | loaded (1) | **unloaded (0)**; `/dev/sd*` back to `sda1..3` only | +| r117 mounts | raw + bind + scratch shared parent | `findmnt \| grep r117` → **(none)** | +| `/mnt` | `+ r117-drives`, `r117cel`, `r117sd` | **`felhom-drives`, `nvme-1tb`, `vzsnap0`** — exactly as found | +| `/root` | 7 scratch files + `r117/` | `ls /root \| grep -i r117` → **(none)** | + +The transient `mnt-r117cel.mount` / `mnt-r117sd.mount` units died with the unmounts +(`Unit mnt-r117sd.mount not loaded` — created by `systemd-mount`, so nothing was left in `/etc`). The +124 MB `debian-13-standard` template was also removed. + +### 14.3 Layer 3 — the hub: nothing to dispose of **This run created no hub-side record.** No customer, no appliance, no pairing, no claim: guest 9301 ran no controller and had **no network interface at all**, and no agent was enrolled against it. Stated -explicitly per §13's rule that "teardown clean" without layer 3 is the `sess-c` failure — here layer 3 is -genuinely empty, and that is a different claim from silence. +explicitly per §13 of the template — "teardown clean" without layer 3 is the `sess-c` failure. Here layer +3 is genuinely empty, which is a different claim from silence. -### 14.3 Fences — verified untouched throughout +### 14.4 Fences — re-verified after teardown, not assumed | Fence | State | |---|---| -| demo-hp guest **9201** | `running` throughout; never a probe target | -| demo-hp **`drill-r50`** (VM 300) | `stopped`; never started, never touched | -| demo-hp **`local-lvm`** | untouched — scratch rootfs deliberately placed on a new dir storage on `pve-root` | -| demo-hp **`felhom-backup`** / `/mnt/nvme-1tb` | not re-targeted, not written to, `content backup` unchanged | +| demo-hp guest **9201** | **`running`** — before, during and after; never a probe target | +| demo-hp **`drill-r50`** (VM 300) | **`stopped`** — never started, never touched | +| demo-hp **`local-lvm`** | **38.84 %** — byte-identical before and after, and to the session-start reading | +| demo-hp **`felhom-backup`** | **`active`, 0.46 %**, `content backup` **unchanged** — not re-targeted, not written to | +| live shared parent `/mnt/felhom-drives` | intact with **both** submounts (`nvme-1tb` on `/dev/nvme0n1[/felhom-data]`, `Felhom-Share` autofs) — a **separate** scratch parent was used throughout | +| `felhom-agent` service on demo-hp | **`active`** | | demo-felhom, Peti's cluster | never contacted | | **ep0** | **read-only** — one `ssh`, `wg show`, and TCP reachability probes toward `10.77.0.2/.3` during outage 1. Nothing written | -| `/mnt/felhom-drives` (live shared parent) | untouched; a **separate** scratch parent was used | | **Code** | **no `.go` file in either repo modified**; nothing built for deployment, nothing published, no version bumped | | Repos | the probe lives outside both, in the session scratchpad; the agent tree copy verified byte-identical to `d4eb259` | @@ -725,12 +785,14 @@ genuinely empty, and that is a different claim from silence. ## 15. What did not happen, and why -- **Q4's hang measurement was not taken** (§6.3) — the venue went offline mid-transfer. The venue - (`dmsetup suspend --noflush`) and the two probes (P6 write+`fsync`, P7 cache-dropped read, both with - timeout-fires reporting and thread-leak counting) are built and specified; only the run is missing. - **The associated hypothesis — that a timeout around a blocked syscall leaks a thread per poll — is - explicitly NOT claimed as a result.** -- **Teardown was not performed** (§14). Commands recorded; nothing fenced was touched. +- **Everything the brief asked for was measured.** The two items the first internet outage deferred — + Q4's hang case and teardown — were completed once the site came back: §6.3 and §14 are measurements, + not plans. The thread-leak hypothesis P6/P7 were built to test turned out to be **the weaker half of + the real result**: the leak is invisible in the thread count (5→5, 5→6), while the process being + unkillable by `SIGKILL` is unambiguous. +- **No load or duration testing of the recommended check.** P1/P2 were measured as single calls + (0.16–0.45 ms), not across a sustained reconcile loop on a box with many drives. Nothing suggests a + problem — they are two `/proc` reads the code already performs — but it was not measured. - **No fresh `/disks` payload and no fresh controller gate log** — deliberate, and the same call `DIAG-r116` §14 made: both were already captured verbatim on real hardware (twice and four times respectively), and running a controller here would have meant a hub customer, a pairing, a golden fetch diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index e0ad85b..86ca2dc 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -18,7 +18,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **WAITING-ON-OPERATOR (M)** | operator ruling on the release process | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design | CC | | **R-116** | ~~**The drive-absent alarm and its recovery were a MISMATCHED PAIR — absent fired the GENERIC `storage_disconnected`, return the SPECIFIC `backup_target_restored`; `backup_target_absent` never fired at all**~~ | **SHIPPED + PROVEN-LIVE** (agent v0.116.0, 2026-07-30) | — | **CLOSED. The full four-event sequence, on the wire, on a fresh box** (`audits/R116-v0116-2026-07-30.md`): `backup_target_absent (error)` on detach → `backup_target_restored (info)` on return for the TARGET, and `storage_disconnected (error)` → `storage_reconnected (info)` for a NON-target drive on the same box four minutes apart. **Two matched pairs, correctly discriminated — and discrimination is proven NON-trivially for the first time**, since both prior runs had the target itself emit the generic event. Gate fired in **3 s**; all four events reached the hub, so the specific alarm, its severity, its Hungarian copy and the hub routing are now exercised end-to-end. **Over-correction PASSES with a positive observable** (0 ABSENT lines / 0 drive events over 2m14s with both drives present, target `degraded:false`, while 2 `RETURNED` lines prove the gate was ticking). Ran on a nested PVE on **demo-hp** per `runbooks/target-selection.md` — through the **real day-0** from the v1.25.0 ISO, with the agent **installed unaided from the vouched Day-0 manifest** (published sha `b47c5c4dab641ee5…`, independent registry GET verified, manifest read back), drives enrolled through the real endpoints, device loss a real hot-detach. **THE FIX, and the ruling is the substantive part:** the mechanism was first isolated from the captured payload (`DIAG-r116-disks-payload-2026-07-30.md`) after two fixes aimed at shapes that do not occur. **Both smaller-looking options were REJECTED because they regress R-114** — `backup_target_offer.go:79` reads `BackupTarget && MountPath != ""` as *"a real drive with its own mountpoint — healthy"* and returns before its `TargetAbsent` branch, so back-filling `MountPath` on the Observe row **or** flagging the registry row (whose `MountPath` is the stale unit-file value) would have told the customer the backup target is fine while its drive was gone. **R-114's correctness was resting on R-116's bug** — a coupling invisible until the payload existed. Taken instead: the Observe row gets the **guest path only** (`mount_path` stays `""`, which is true) from a new `ConfigPath` (`json:"-"`, so the cross-repo golden + key-set contract is untouched), and the union row is deduped **on guest path** — the join being CONFIGURATION (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), the only identity that survives the device. Tests 845→849; 4 red-proofs each asserted to land, and red-proof 1 replays v0.115.0's code and fails, which is the empirical proof it was inert. Its green test had supplied a `MountPath` production never supplies AND left `DriveTargets` nil so the union loop never ran — both corrected. v0.115.0 left in place (inert, harmless). Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded. **Caveat: the drill's controller was 0.185.1 from the golden, which PREDATES R-114**, so its absent-state banner showed the old false copy — the golden being a release behind, not a regression → **R-120** | — | | **R-120** | ~~**The golden baked a controller that predated R-114 + R-112, so a FRESH box showed the customer the WRONG absent-target message**~~ | **CLOSED — golden rebaked + PROVEN-LIVE, and the class now has an ENFORCED gate** (golden 0.186.0 + hub v0.82.0, 2026-07-30) | — | **`audits/R120-golden-rebake-2026-07-30.md`.** **Half 1 — the artifact.** Golden **0.186.0** baked from `main`'s controller in the DooPlex bake fixture (overlay2 OK, **3 mounts**, FATAL 0, exclusions 0, 618 MB, upload **201**, `GOLDEN_SHA256=b760ac6a33e70700…`, token-leak grep 0, GL-1 teardown, `drill.qcow2` back to `virgin`). Three observables: **published** — anonymous GET (what the installer does) 200 / 648930639 bytes / sha identical to the bake; **vouched** — manifest read BACK; **resolved** — `Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)`. Floor **untouched** per publish-train rule 2 (`min_controller_version` still 0.156.0; it is a separate form); MinAgent left 0.113.0 as 0.186.0 declares. **Proven on a REAL day-0, not the fixture** (per the Part-1 rule now in `runbooks/target-selection.md`): VM 9402 on demo-hp from the v1.25.0 ISO → `Controller elindult (0.186.0)`. With the target detached the endpoint returned the **`TargetAbsent`** copy — *„A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod…"* — **and `offer_path` absent entirely**; the day-old read on the 0.185.1 golden had returned the false system-disk message **plus** an offer of the other drive. **Half 2 — the mechanism, operator ruling REFUSE.** hub **v0.82.0**: the gate sits in `hub/internal/web/configs.go` `handleSetArtifacts` immediately before the only write — the sole UI path to `SetArtifactManifest` — so it runs on every vouch without anyone choosing to, and it **refuses** rather than warning. Signal: `store.NewestReportedControllerVersion()` over `reports.controller_version`, **semver-compared in Go** (`MAX()` in SQL ranks 0.99.0 above 0.186.0 — a pair this fleet has shipped). Fail-open in exactly two deliberate cases: empty golden field, unknown fleet version. **NEAR-MISS RECORDED: the first draft read `guests.controller_version`, a column that exists and that NOTHING writes** — it would always have seen `""` and failed open, i.e. inert, this gate's own failure shape, one grep from shipping. 4 tests through the **production handler** over httptest (never a seam), the refusal asserting **both** the flash **and** that the manifest was not written; red-proof: deleting the block makes the stale golden vouchable again. **PROVEN LIVE on the deployed hub by re-attempting the original mistake:** vouching 0.185.1 → `HTTP 303 …flash=golden_behind_fleet` + `[WARN] artifact vouch REFUSED: golden 0.185.1 is older than the newest controller the fleet reports (0.186.0)`, and the manifest read back **unchanged at 0.186.0**. Recorded on **R-29's audit list** (`ROADMAP.md`) as the **first enforced gate** beside its three orphans, so the contrast is kept — the orphans are unchanged. Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded, exactly as `sess-e` was (and `sess-e` was deleted this run) | — | -| **R-117** | **A drive's guest bind becomes a DEAD MOUNT while every signal reads healthy — and it happens in TWO ways, only one of which the original framing covered.** (a) *after a detach/return*: the host raw mount heals onto the NEW device via its fs-UUID-keyed unit while the bind still names the OLD one, so the gate takes its `Return` branch and restarts the customer's apps onto a namespace that `EIO`s on every call; (b) *in STEADY STATE, no cycle at all* — a device that errors without disappearing leaves the raw mount `active`, `BoundUnderParent` `true` and the drive never `Disconnected`, so **the gate produces no action and NOTHING is emitted on any channel** | **MECHANISM + RECIPE ESTABLISHED (spike done 2026-07-30) — READY (S/M), not fixed** | — | **SPIKE: `audits/SPIKE-r117-bind-liveness-2026-07-30.md`** — reproduced deliberately **3/3**, two device classes, on a purpose-built scratch LXC (9301) on demo-hp; predicates evaluated by a throwaway probe calling the **real** `localapi` code from agent `d4eb259` (v0.116.0). **MECHANISM, confirmed at source and live:** both halves of the R-113 conjunction are *path-presence tests*. `GuestSeesMount` (`intermediary.go:276`) and `isHostMountpoint` (`intermediary.go:394`) each compare **field 5** (the mount point) of a mountinfo line and **never read field 3, the `major:minor`** — so neither can see that the bind and the raw mount name **different devices**. Measured: `raw 8:32 /dev/sdc` vs `bind 8:16 /dev/sdb …,shutdown`, `BoundUnderParent = TRUE`, `EIO`(5) on read **and** write, host and guest. **The node change is a CONSEQUENCE, not a precondition** — control test: with the stale bind held the drive returns as `sdc` (8:32); once it is `umount`ed the letter is **reused** (`sdb`, 8:16). The stale bind pins the dead superblock, which forces the new number — so a devno comparison is *sound* for case (a), not a heuristic. **RECIPE** (spike §10): `scsi_debug` is load-bearing — it is the only mechanism that removes a gendisk **while mounted** (dm and loop are held open by the stale bind: `dmsetup remove --force` → `Device or resource busy`). `modprobe scsi_debug` → mkfs → UUID-keyed `systemd-mount` → bind under a **scratch** shared parent → `echo 1 > /sys/block/sdb/device/delete` (systemd stops the raw unit itself, `BindsTo=`) → `echo '- - -' > /sys/class/scsi_host/hostN/scan` → re-mount by UUID. Case (b): swap a dm table to `error`. **TWO DEATH STATES, different signals** — device removed ⇒ superopts gain `shutdown`, `EIO` both ways; device errors in place ⇒ `emergency_ro`, write `EROFS`(30), reads still served from cache. **PROBE TRAP (R-117b): `statfs` and `getdents` SUCCEED on a namespace that `EIO`s every byte** — any liveness probe built on either is hollow by construction. **A pure-`/proc` check costs 0.17–0.45 ms**, cannot hang, spins up no disk and writes nothing, against the 3.85–5.03 ms `GuestSeesMount` already spends forking `lxc-info`. **CONSTRAINT: the fix needs BOTH a devno comparison (closes (a); blind to (b)) and the fs abort flags `shutdown`\|`emergency_ro` (closes (b); ext4-specific)** — the tempting devno-only fix leaves the silent half open. Keep the *cannot-tell → never absent* rule (`disks.go:983-986`). **R-117d — THE FIX IS ONE PREDICATE, AND IT SELF-HEALS:** `AttachDrive`'s normalize leg (`intermediary.go:238-247`) already performs exactly the repair — proven live, **guest never restarted** (init PID identical) — and **three** call sites already invoke it: the 20 s reconcile ticker (`main.go:973-985`), agent startup (`disks.go:951`), and **the controller's `Return` branch itself, at `intermediary.go:301-305`, BEFORE it restarts the apps at `:311`**. All three are defeated by `if n == 1 && b.GuestSeesMount(...)` returning early at `intermediary.go:235` and logging *"fully live"*. **Unblock the existing path; do not add a new one.** **R-117c:** three comments (`disks.go:83`, `:154-155`, `:155-165`) promise "live + usable in the guest"; nothing tests it and it is false in the measured state — the fix owes a test asserting the **consequence** (dead namespace ⇒ `BoundUnderParent=false` ⇒ alarm fires), plus a second scenario for case (b). **OPEN from the spike: the HANG case was NOT measured** (§6.3) — venue lost internet mid-transfer; the `dmsetup suspend --noflush` venue and the two blocking probes (write+`fsync`, cache-dropped read) are built and specified. It does not gate the `/proc` recommendation, but it **does** gate adding any I/O confirmation probe. **Teardown of the spike venue is OWED** (spike §14 lists the exact commands; nothing fenced was touched, no hub-side record was created). Note demo-hp's live agent is **0.113.0** — it does not run the R-113 conjunction, so live validation of the fix needs that box current first | CC | +| **R-117** | **A drive's guest bind becomes a DEAD MOUNT while every signal reads healthy — and it happens in TWO ways, only one of which the original framing covered.** (a) *after a detach/return*: the host raw mount heals onto the NEW device via its fs-UUID-keyed unit while the bind still names the OLD one, so the gate takes its `Return` branch and restarts the customer's apps onto a namespace that `EIO`s on every call; (b) *in STEADY STATE, no cycle at all* — a device that errors without disappearing leaves the raw mount `active`, `BoundUnderParent` `true` and the drive never `Disconnected`, so **the gate produces no action and NOTHING is emitted on any channel** | **MECHANISM + RECIPE + FIX CONSTRAINTS ESTABLISHED (spike complete 2026-07-30) — READY (S/M), not fixed** | — | **SPIKE: `audits/SPIKE-r117-bind-liveness-2026-07-30.md`** — reproduced deliberately **3/3**, two device classes, on a purpose-built scratch LXC (9301) on demo-hp; predicates evaluated by a throwaway probe calling the **real** `localapi` code from agent `d4eb259` (v0.116.0). **MECHANISM, confirmed at source and live:** both halves of the R-113 conjunction are *path-presence tests*. `GuestSeesMount` (`intermediary.go:276`) and `isHostMountpoint` (`intermediary.go:394`) each compare **field 5** (the mount point) of a mountinfo line and **never read field 3, the `major:minor`** — so neither can see that the bind and the raw mount name **different devices**. Measured: `raw 8:32 /dev/sdc` vs `bind 8:16 /dev/sdb …,shutdown`, `BoundUnderParent = TRUE`, `EIO`(5) on read **and** write, host and guest. **The node change is a CONSEQUENCE, not a precondition** — control test: with the stale bind held the drive returns as `sdc` (8:32); once it is `umount`ed the letter is **reused** (`sdb`, 8:16). The stale bind pins the dead superblock, which forces the new number — so a devno comparison is *sound* for case (a), not a heuristic. **RECIPE** (spike §10): `scsi_debug` is load-bearing — it is the only mechanism that removes a gendisk **while mounted** (dm and loop are held open by the stale bind: `dmsetup remove --force` → `Device or resource busy`). `modprobe scsi_debug` → mkfs → UUID-keyed `systemd-mount` → bind under a **scratch** shared parent → `echo 1 > /sys/block/sdb/device/delete` (systemd stops the raw unit itself, `BindsTo=`) → `echo '- - -' > /sys/class/scsi_host/hostN/scan` → re-mount by UUID. Case (b): swap a dm table to `error`. **TWO DEATH STATES, different signals** — device removed ⇒ superopts gain `shutdown`, `EIO` both ways; device errors in place ⇒ `emergency_ro`, write `EROFS`(30), reads still served from cache. **PROBE TRAP (R-117b): `statfs` and `getdents` SUCCEED on a namespace that `EIO`s every byte** — any liveness probe built on either is hollow by construction. **A pure-`/proc` check costs 0.17–0.45 ms**, cannot hang, spins up no disk and writes nothing, against the 3.85–5.03 ms `GuestSeesMount` already spends forking `lxc-info`. **CONSTRAINT: the fix needs BOTH a devno comparison (closes (a); blind to (b)) and the fs abort flags `shutdown`\|`emergency_ro` (closes (b); ext4-specific)** — the tempting devno-only fix leaves the silent half open. Keep the *cannot-tell → never absent* rule (`disks.go:983-986`). **R-117d — THE FIX IS ONE PREDICATE, AND IT SELF-HEALS:** `AttachDrive`'s normalize leg (`intermediary.go:238-247`) already performs exactly the repair — proven live, **guest never restarted** (init PID identical) — and **three** call sites already invoke it: the 20 s reconcile ticker (`main.go:973-985`), agent startup (`disks.go:951`), and **the controller's `Return` branch itself, at `intermediary.go:301-305`, BEFORE it restarts the apps at `:311`**. All three are defeated by `if n == 1 && b.GuestSeesMount(...)` returning early at `intermediary.go:235` and logging *"fully live"*. **Unblock the existing path; do not add a new one.** **R-117c:** three comments (`disks.go:83`, `:154-155`, `:155-165`) promise "live + usable in the guest"; nothing tests it and it is false in the measured state — the fix owes a test asserting the **consequence** (dead namespace ⇒ `BoundUnderParent=false` ⇒ alarm fires), plus a second scenario for case (b). **R-117f — THE HANG CASE, MEASURED, AND IT DISQUALIFIES I/O PROBING OUTRIGHT** (§6.3): against a `dmsetup suspend`ed device **every probe that touches it blocked — including a buffered write with no `fsync`** (the `O_CREAT` metadata path needs journal access) — while **P1 and P2 completed in 364 µs / 206 µs**. The blocked process then survived `SIGTERM` **and `SIGKILL`** (`stat=D`, `wchan=folio_wait_bit_common`, still alive 3m50s after `kill -9`) and died only when the device was resumed. **So `systemctl restart felhom-agent` would hang, leaving the agent unrecoverable until the device returns or the host reboots** — and the thread count does not reveal the leak (5→5, 5→6). A timeout protects the caller's control flow and nothing else. **The fix must issue no block I/O; that is now a fence, not a preference.** `statfs`/`getdents` reported HEALTHY on the wedged device too, confirming R-117b in a second failure mode. **Teardown DONE, all three layers** (spike §14): guest 9301 destroyed, `r117scratch` removed, both dm devices and both loop devices gone, `scsi_debug` unloaded, `local` back to 37.02 % (session start 37.00 %); fences re-verified after — 9201 `running`, `drill-r50` `stopped`, `local-lvm` **38.84 % byte-identical**, `felhom-backup` `content backup` unchanged, live `/mnt/felhom-drives` intact with both submounts, agent `active`. **Layer 3 genuinely empty** — 9301 had no NIC and ran no controller, so no hub-side record was ever created. Note demo-hp's live agent is **0.113.0** — it does not run the R-113 conjunction, so live validation of the fix needs that box current first | CC | | **R-118** | **An absent drive's union row advertises the ROOT filesystem's capacity as its own.** In the absent-state payload the registry-union row reports `total_bytes: 49675956224 / used_bytes: 4584579072` — **byte-identical to the `local` row** (`durable_id: path:/var/lib/vz`, i.e. `pve-root`) in the same response. The real drive is **4 GB** | **READY (XS) — NEW 2026-07-30** | — | Cause: `statfsCapacity(d.MountPath)` (`disks.go:335-338`) statfs's `/mnt/cel`, which with the device gone is a **bare directory on the root filesystem**. `observe.go:176-183`'s comment warns about exactly this trap and guards the Observe path ("*an unmounted removable dir-storage's mountpoint reverts to a bare directory on root … catastrophic DR mis-id*"); **the union path has no equivalent guard.** **Not a DR mis-id** — `durable_id` on that row is still the correct `uuid:…`, so re-attach identity is safe. It is a **false capacity** reaching every consumer of `total_bytes`/`used_fraction` (fill monitors, storage cards): a detached 4 GB drive advertises 46 GiB at 9.2 % used. Same class as `role.go:180-181` — an absent drive's fields decaying to the root filesystem's. Evidence: `audits/DIAG-r116-disks-payload-2026-07-30.md` §12 | CC | | **R-113** | ~~**The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.**~~ `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc//mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **SHIPPED + PROVEN-LIVE** (agent v0.114.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** Target drive hot-detached on a fresh box running the SHIPPED agent 0.114.0 (from the Day-0 manifest, not a hand build). **The gate fired in 4 seconds** — E-2d measured ZERO over 4½ minutes — and `SetDisconnected` was reached. It fired on exactly the shape that defeated it: raw `/mnt/mentes` NOT mounted while the bind `/mnt/felhom-drives/mentes` still read `/dev/sdb[/felhom-data]`. **Over-correction guard PASSED:** with both drives present, 0 ABSENT lines and the target stayed healthy. **Note: C5 still FAILED — but on a different defect (→ R-116), not on this one.** **SESSION C IS NOW UNBLOCKED.** Agent **0.114.0 published and vouched** 2026-07-29 (sha `5e4c15ebee2d7583…`, round-trip GET verified; hub manifest read back showing it) — a fresh drill box now installs the fix under test instead of the bug. The golden was NOT re-baked and `min_agent` was NOT raised: the golden bakes the controller, not the agent, and controller v0.185.0 declares MinAgent 0.113.0, which 0.114.0 already satisfies. **FIX: `BoundUnderParent` is now a CONJUNCTION — bound under the parent AND the drive's raw host mount still mounted** (`devicePresent`, new `deviceCheck` seam), at BOTH `/disks` construction sites. The raw mount is the device-bound systemd unit that dies with the device; the agent's bind is not — so the raw mount IS the device signal, grounded in E-2d's measurement rather than inference. **Conjunction, deliberately:** the device half alone would regress boot ordering (raw mounts early, bind lands ~18 s later — that window must keep reading absent), so existing behaviour is byte-identical and only the unreachable case is closed. **Unknown is never absent** (`devicePresent("")` = true) — a false absent stops a working customer's apps. **Controller UNCHANGED, no MinAgent bump:** `BoundUnderParent` has exactly one functional consumer (`planDriveGates:226`); a new `DevicePresent` bool was rejected because absent-from-JSON decodes to `false`, which would have made every drive on an older agent read ABSENT. +6 tests (208→214), 4 red-proofs run and reverted. **Deployed to demo-felhom; over-correction guard verified in production** — raw mount present, drive still reads present, 10/10 apps untouched, no gate action, no false alarm. **demo-hp deliberately left on 0.113.0** (the spec scoped deploy to felhom-pve) — it still carries the bug. **⚠️ SESSION C BLOCKER: the hub Day-0 manifest vouches agent 0.113.0**, so a fresh drill box would install WITHOUT this fix and validate nothing — publish + vouch 0.114.0 first (R-111's trap, same shape). **Awaiting live proof of exactly one leg:** device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. Original finding: **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 **Session C scope UNCHANGED by Session B** — R-114/R-112 shipped as controller v0.186.0 and neither touches the agent; the leg awaiting proof is still device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. One rebuild now validates all three. | CC | | **R-112** | **E-2's degraded banner and offer have NO UI CONSUMER — the endpoint is correct and the customer never sees it.** `GET /api/storage/backup-target` returns byte-exact copy (proven live), and **nothing fetches it**: `grep 'backup-target'` across all `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` + `degradedMessageFor` are consumed **only** by the JSON handler — **no page handler injects the state** | **SHIPPED + PROVEN-LIVE** (controller v0.186.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md` — the banner reached a customer's page for the first time.** Never-configured box: banner element 1, never-configured copy 1. After the wizard: offer block 1 with `data-path="/mnt/felhom-drives/mentes"`. Healthy after assign: all four markers 0, **proven POSITIVELY** — idle delta 0 `/backup/tiers` calls, page-load delta +1, single caller ⇒ the seam ran and chose silence. **FIXED: the state now has a consumer.** Server-rendered on `/backups` via `backupsHandler` → `backupTargetView` → `backups.html`, following the existing `SingleCopyWarning` banner pattern — NOT a 19th JS fetch, because a banner that needs JavaScript to appear is one more thing that can silently not happen. `backupTargetView` returns **nil** for healthy AND unknown, so those render nothing at all. **Scenario-E seam test drives `backupsHandler` over httptest and asserts the RENDERED HTML** — deleting the one line that sets `data["BackupTarget"]` reproduces the old state and fails every render assertion. **SEAM PROVEN LIVE on demo-felhom by a DIFFERENTIAL positive observable, not by an absent banner:** idle 8 s → 0 new `/backup/tiers` agent calls; each `/backups` load → exactly +1, and that call has only one caller (`resolveBackupTargetState`). The box is healthy (`degraded:false, target:felhom-backup`) and the page correctly rendered **nothing** — which matches its real state but, being a negative, is NOT by itself proof of wiring. **Still unproven live:** that a customer sees actual copy — impossible on a healthy box. **The decisive contrast: templates fetch 18 distinct `/api/storage/*` endpoints; `backup-target` and `backup-target/assign` are the only two with zero references.** The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"* — an invariant comment asserting a consumer that does not exist (7th instance of that class). v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* — it fixed the **router mount** and stopped one layer short of the **render**; its test `TestBackupTargetRoutesLiveUnderTheStorageAPIMount` pins dispatch, not reachability, which is exactly what `CLAUDE.md`'s seam rule warns about. **Fifth instance of seam-built-but-never-wired.** **Fix R-114 FIRST** — wiring this alone would start showing customers the wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | CC |