diff --git a/documentation/audits/R117-v0117-2026-07-30.md b/documentation/audits/R117-v0117-2026-07-30.md new file mode 100644 index 0000000..7563c59 --- /dev/null +++ b/documentation/audits/R117-v0117-2026-07-30.md @@ -0,0 +1,437 @@ +# R-117 v0.117.0 — the liveness signal now tests liveness, proven on hardware + +Implementation + live validation of `audits/SPIKE-r117-bind-liveness-2026-07-30.md`. Agent **v0.117.0**. +`felhom-controller` untouched: the spike proved the repair is already wired from the controller's +`Return` branch and dies inside the agent. + +--- + +## 1. Baselines — each established separately + +| Thing | Value | How | +|---|---|---| +| agent `main` before | **v0.116.0** @ `d4eb259` | `git log`, `CHANGELOG.md:1`; tree clean, `HEAD == origin/main` | +| agent shipped this run | **v0.117.0** @ `966d8f4` | commit below | +| **agent on demo-hp BEFORE** | **0.113.0** | `felhom-agent --version` on the box — **predates R-113 (0.114.0) entirely** | +| **agent on demo-hp AFTER** | **0.117.0** | same command, post-deploy | +| agent published + vouched before | published 0.113–0.116; **vouched 0.116.0** (`b47c5c4dab641ee5…`) | hub `/configuration`, read not assumed | +| controller `main` | **v0.186.0** @ `b331f18` | `git log` | +| golden bakes controller | **0.186.0** | hub `/configuration` — **matches `main`, so R-120's gate is holding them together** ✅ | +| `min_agent` / `min_controller_version` | 0.113.0 / 0.156.0 | hub `/configuration` | +| hub live | **0.82.0** | `kubectl -n felhom-system get deploy` | +| `felhom.eu` HEAD before | `e70b5fe` | as the task expected | +| demo-hp | PVE 9.2.2, kernel 7.0.2-6-pve | `pveversion` | + +--- + +## 2. Part 1 — the two record items, committed before any Go file was opened + +**`felhom.eu` `37515cd`** — `docs(R-117 Part 1)`. **`felhom-agent` `6be168d`** — the same rule repeated +where it binds. + +### 2.1 The `CLAUDE.md` rule, as written + +Placed beside the seam-wiring rule in `felhom.eu/CLAUDE.md` "Code quality rules": + +> **A health check issues no block I/O.** A probe that touches a wedged device enters uninterruptible +> sleep, survives `SIGKILL`, and cannot be recovered until the device returns or the host reboots — so +> `systemctl restart` hangs too. A timeout protects the caller's control flow and nothing else: the +> blocked thread remains. Liveness is decided from `/proc` and the kernel's own state, never by reading +> or writing the filesystem. Measured, R-117 spike §6.3 (…): a probe stayed in `D` state 3m50s after +> `kill -9`; a buffered write with no `fsync` blocked too (`O_CREAT` needs journal access); and +> `statfs`/`getdents` returned **healthy** on a namespace that `EIO`s every byte — fast, and wrong. + +**Repeated in `felhom-agent/CLAUDE.md`**, as a one-line pointer, and the reason is recorded there: health +checks are written in the agent repo, and `felhom.eu/CLAUDE.md` **does not load in an agent-only session**. +A standing rule that does not load where it binds is the inert-seam shape applied to a rule. The task named +one location; this is the one deviation, stated rather than absorbed (standing rule 4). + +### 2.2 The R-116 narrowing, applied + +The R-116 row's over-correction clause now carries: the 2 `RETURNED` lines are a genuine positive +observable so rule 3 is satisfied, but `degraded:false` was read off a drive whose bind was dead, so the +window evidences **"the gate did not over-fire"** and **not** **"the drive was healthy."** Nothing else +about the row changed. + +--- + +## 3. THE RULING (task §2.2) — what happens when P2 trips + +**Ruling: P2 must NOT self-heal. It surfaces. Operator decision NOT required — and the reasoning for that +is the substantive part, so it is stated in full for overruling.** + +The two states have different repairs, and *"would a re-bind help?"* — not *"is something aborted?"* — is +the question a verdict must answer: + +| Verdict | What a re-bind would do | Ruling | +|---|---|---| +| **`BindStaleDevice`** | the raw mount is a **different, healthy** superblock (the drive returned and the fs-UUID-keyed unit healed onto it), so umount + re-bind lands the namespace on working storage | **REPAIR.** Falls through to the existing normalize leg | +| **`BindAborted`** | the raw mount is the **same aborted** superblock — a fresh bind to a still-dead filesystem | **QUIET NO-OP, and surface via `BoundUnderParent=false`** | + +**Why `BindAborted` must not repair,** three reasons and the third is the decisive one: + +1. It cannot work. Re-binding does not clear an ext4 abort; that needs a remount or a fsck. +2. `AttachDrive` runs **every 20 s** (`cmd/felhom-agent/main.go` ticker), so re-binding would be an + **infinite silent retry** — R-117a's silence with more CPU. +3. It would **mask** the state. The whole point of the fix is that this state becomes visible. + +**Why no operator decision was needed.** The ruling changes customer-visible behaviour: apps that today +keep running against a read-only/`EIO` namespace will be **stopped**, and the customer gets an alarm. That +is a real change — but it is the **identical treatment an unplugged drive already receives**, through the +existing gate, the existing event types and the existing Hungarian copy. No new message is invented, no +new concept reaches the customer. The alternative is letting applications keep writing a customer's +documents into a filesystem that rejects every write. **Fail-closed is also the project's standing +direction for drive state.** So this is routing an already-broken state into an already-tested alarm, not a +new customer-facing decision. **Flagged prominently rather than decided quietly; overrule freely.** + +**Deliberately NOT done: automatic remount of an aborted filesystem.** That is a repair of the customer's +data-bearing filesystem, risks loss on a dirty abort, and is an operator call. Recorded as R-117g. + +--- + +## 4. The predicate as implemented + +`felhom-agent/internal/localapi/intermediary.go`: + +``` +bindLiveness(stable, raw) → + stable=="" or raw=="" → BindUnknown + no bind entry at `stable` → BindUnknown (isHostMountpoint's question, not this one) + no raw entry at `raw` → BindUnknown (devicePresent already reports device absence) + bind.devno != raw.devno → raw aborted ? BindAborted : BindStaleDevice ← P1 + (devnos agree) bind aborted → BindAborted ← P2 + (devnos agree) fstype unknown → BindUnknown + otherwise → BindLive +``` + +Wired as the **third term** of the conjunction at **both** `/disks` construction sites (`disks.go`): +`boundUnderParent(...) && devicePresent(...) && bindUsable(gp, )`. + +**Order is load-bearing, and a test caught it.** The first draft read the abort flag before comparing +devices. In the real return state the stale bind carries `shutdown` **as well as** a different device, so +abort-first classifies it `BindAborted` — **reporting stays correct, every payload test still passes, and +the repair silently never runs.** The abort flag is therefore read off the **raw** mount in the stale case +(the re-bind's target) and off the bind only once the devices already agree. + +### 4.1 How *unknown* is expressed + +A **three-state type**, never a bool: `BindLiveness ∈ {BindUnknown, BindLive, BindStaleDevice, +BindAborted}`, with `BindUnknown` as the **zero value**. Every caller reads it through **one** method: + +```go +func (l BindLiveness) Usable() bool { return l == BindLive || l == BindUnknown } +``` + +So the "cannot tell → never absent" rule lives in exactly one place and no caller can restate it wrongly. +The task named the `newestArchiveOn` trap — a `(value, bool)` shape whose comment promised a degradation +the signature could not express. Four routes to unknown are pinned by test: unreadable `/proc`, no raw +entry, empty paths, and a filesystem whose abort vocabulary is unmeasured. + +**P2's tokens: `shutdown` AND `emergency_ro`**, both measured. `ext4`/`ext3`/`ext2` (one driver); anything +else → **unknown, never live**. + +### 4.2 Not a new recovery path + +`AttachDrive`'s early return now switches on the verdict instead of declaring a dead namespace +*"fully live, no-op"*. The three call sites that already invoked the repair — the 20 s ticker, agent +startup, and the controller's `Return` branch **before** `restartStacks` — are unchanged. Also: +`isHostMountpoint` and `countHostMounts` are now one-liners over a single new parser `hostMountEntries`, +which yields devno/root/fstype/super-options (`REUSE.md`'s three-independent-readers note narrowed). + +--- + +## 5. Tests + +**849 → 863** (+14 top-level), **29/29 packages green**, `go build` / `go vet` / `go test` each with +`rc=0` read separately (never combined with a commit). The 849 baseline was **verified against a temporary +worktree at HEAD**, not inferred — the first draft of the CHANGELOG line said 858 → 873 from a count taken +mid-edit, and that was corrected before commit. + +| Test | Asserts (the consequence) | Result | +|---|---|---| +| `TestDisks_BindLiveness_StaleBindReadsAbsent` | R-117 case (a) → `bound_under_parent` false **through the real `/disks` handler** | PASS | +| `TestDisks_BindLiveness_AbortedFilesystemReadsAbsent` (×2 tokens) | R-117a case (b) → false, with a **row-shape guard** | PASS | +| `TestDisks_BindLiveness_UnionPath_StaleBindReadsAbsent` | union path, case (a) | PASS | +| `TestDisks_BindLiveness_UnionPath_AbortedReadsAbsent` | union path, case (b) — Role+State hardcoded there | PASS | +| `TestDisks_BindLiveness_HealthyReadsPresent` (observe+union) | no false negative | PASS | +| `TestDisks_BindLiveness_UnknownIsTreatedAsPresent` (5 subtests) | all four unknown routes → **present**; `Usable()` truth table | PASS | +| `TestBindLiveness_Verdicts` (4 states) | each verdict | PASS | +| `TestBindLiveness_AbortedWins_WhenDevnosAgree` | the P1-only fix cannot pass | PASS | +| `TestHostMountEntries_ParsesDevnoAndSuperOpts` | `" - "`-separator field extraction | PASS | +| `TestHostMountEntries_CountsStackedBinds` | double-bind convergence survives the refactor | PASS | +| `TestAttachDrive_StaleBind_Rebinds` | **the repair RUNS** (umount+rebind recorded) | PASS | +| `TestAttachDrive_AbortedFilesystem_DoesNotRebind` | **§3's ruling**: zero mount ops, no error | PASS | +| `TestAttachDrive_Healthy_IsStillANoOp` | 20 s idempotency | PASS | +| `TestAttachDrive_UnknownLiveness_IsANoOp` | cannot-tell must not churn | PASS | + +**Fixtures are the spike's captured mountinfo**, with two substitutions both recorded in the file: the +scratch shared parent mapped to the production `StableParentDir` (the code derives it), and the aborted +state transposed from dm onto the USB shape (§5.2 below). Tests redirect `procSelfMountinfo` / +`procGuestMountinfo` at fixture files, so the real parser, the real predicate, the real `AttachDrive` and +the real `/disks` handler all run — **the data is injected, the logic is not.** + +### 5.1 Red-proofs — 6, each verified to have landed + +| # | Mutation | Landing verified by | Fails | +|---|---|---|---| +| RP1 | term 3 removed at the **Observe** site | `bindUsable(gp, t.MountPath)` occurrences → **0** | StaleBind + Aborted(×2) | +| RP2 | term 3 removed at the **union** site | `bindUsable(gp, d.MountPath)` → **0** | both UnionPath tests | +| RP3 | **the P1-only fix** — abort check neutered | `RED-PROOF RP3` marker present | Aborted(×2), UnionPath_Aborted, Verdicts(×2), AbortedWins | +| RP4 | `emergency_ro` dropped from the token list | `RED-PROOF RP4` present | only the `emergency_ro` subtests — the "only `shutdown`" fix | +| RP5 | unknown reported as not-live | `RED-PROOF RP5` present | UnknownIsTreatedAsPresent/unreadable | +| RP6 | aborted arm falls through to the re-bind | `RED-PROOF RP6` present | AttachDrive_AbortedFilesystem_DoesNotRebind (`go test` **rc=1**, read from `go test` itself) | + +### 5.2 A HOLLOW TEST WAS CAUGHT — by RP1 failing to fail + +`TestDisks_BindLiveness_AbortedFilesystemReadsAbsent` first used the spike's dm capture verbatim +(`/dev/mapper/r117cel`). **`RoleForStorage` derives `role="system"` for that backing device** — a +system-role row never enters the block that computes `BoundUnderParent`, so the field stayed `false` **by +default**, the assertion passed, and **no mutation could ever fail it.** Pure decoration, and it is exactly +R-116's failure mode (a fixture supplying a shape production never emits). + +Found because **RP1 removed term 3 and the test still passed.** Fixed by transposing the aborted fixture +onto the USB drive shape (an in-place abort on a USB drive is the realistic case anyway; only the super +options and the matching devnos carry the claim) **and** by adding a guard that fails loudly if the row is +not the production shape: + +```go +if di.Role != "user-data" || di.GuestPath == "" { + t.Fatalf("fixture does not reproduce the production row shape: role=%q guest_path=%q — "+ + "the conjunction never runs on such a row, so any assertion below is vacuous", di.Role, di.GuestPath) +} +``` + +**The red-proof did not merely confirm the test; it is the only thing that found the test was empty.** + +--- + +## 6. Build, publish, vouch — three observables, quoted + +**1. Published** — the **anonymous** GET the installer performs, no credentials: + +``` +GET .../api/packages/admin/generic/felhom-agent/0.117.0/felhom-agent +http=200 bytes=14032908 +downloaded sha256: e6768a12eba1c9e0743367f04589845330151ac63d10caad794e8a4c6f36f735 +built sha256: e6768a12eba1c9e0743367f04589845330151ac63d10caad794e8a4c6f36f735 +MATCH — published bytes are the built bytes +$ /tmp/agent-dl-0.117.0 --version → felhom-agent 0.117.0 +``` + +**2. Vouched** — POST through the real operator endpoint, then the manifest **read back** (not the flash): + +``` +POST /configuration/artifacts → HTTP/1.1 303, Location: /configuration?flash=artifacts_set +hub log: [INFO] Artifact manifest set: agent=0.117.0 golden=0.186.0 min_agent="0.113.0" wrapper_sha=true + +READ BACK: agent_version=0.117.0 agent_sha256=e6768a12eba1c9e0… + golden_version=0.186.0 golden_sha256=b760ac6a33e70700… (preserved) + min_agent=0.113.0 wrapper_sha256=104db0a4401f65bb… (preserved) +``` + +R-120's gate did **not** refuse: golden 0.186.0 is not behind the fleet's newest controller (0.186.0). + +**3. Running on the box under test:** + +``` +BEFORE: felhom-agent 0.113.0 active +AFTER: felhom-agent 0.117.0 active +capabilities self-check ok=68 total=68 degraded=0 inactive=0 +reconcile: enrolled drive bound under shared parent (live, no reboot) vmid=9201 … ← loop ticking, new pid +``` + +Zero `ERROR` lines. `configs/` was untouched by this commit, so a binary-only deploy is correct (verified +with `git show --name-only`). + +--- + +## 7. §5.1 — demo-hp brought current, and the drift the R-120 gate does not cover + +demo-hp: **0.113.0 → 0.117.0**, prior binary backed up to `felhom-agent.bak-0.113.0`. + +**Does the R-120 vouch gate cover agent-on-a-box drift? NO — confirmed at source.** +`hub/internal/web/configs.go:1165-1169` compares `goldenVer` against +`store.NewestReportedControllerVersion()`: a **golden-artifact vs fleet-CONTROLLER** check. It says nothing +about the agent installed on a box. **`MinAgent` does not cover it either** — it *holds* the controller +floor for a box whose agent is too old (`hub/internal/api/handler.go:530-538`, `store.go:1857`), which is +protective, not an alarm — and demo-hp's 0.113.0 **equalled** `min_agent` 0.113.0, so even a floor +comparison was satisfied. + +**Filed as R-121 — the fourth instance of the drift family** (R-111, R-115, R-120, and now +installed-vs-vouched). Its measured cost: R-117's whole subject is the R-113 conjunction, which landed in +0.114.0, so **the designated drill host could not exercise the code under investigation at all**, and the +spike had to route every predicate result through an out-of-repo probe built from `main`. + +--- + +## 8. §5.2 — the live run + +**Venue:** demo-hp, the spike's §10 recipe. Scratch LXC **9301** (no NIC) on a scratch dir storage +`r117scratch` — never `local-lvm`, never `felhom-backup`. **Scratch** shared parent `/mnt/r117-drives`, so +the live `/mnt/felhom-drives` peer group (guest 9201's) was not used. `scsi_debug` for the drive, because +it is the only mechanism that removes a gendisk **while mounted**. Guest parent bind verified +`master:209` — a real slave of the host peer group. + +**Method:** a throwaway probe built from the **shipped tree at `966d8f4`** (exported in a copy outside both +repos), so `bindLiveness` and `AttachDrive` are the real functions, judging real kernel state. + +### 8.1 The predicate, every state, on hardware + +| State | host raw | host bind | verdict | `Usable()` | term 3 | cost | +|---|---|---|---|---|---|---| +| HEALTHY (scsi) | `8:16 rw,stripe=512` | `8:16 rw,stripe=512` | `live` | true | true | 442 µs | +| after real gendisk removal | **(NOT MOUNTED)** | `8:16 …,shutdown` | `unknown` | true | true | 340 µs | +| **R-117 RETURN** | `8:32` (`/dev/sdc`) | `8:16 …,shutdown` | **`stale-device`** | **false** | **false** | 365 µs | +| HEALTHY (dm) | `252:11 rw` | `252:11 rw` | `live` | true | true | 359 µs | +| **Q7 IN-PLACE ABORT** | `252:11 …,emergency_ro` | `252:11 …,emergency_ro` | **`filesystem-aborted`** | **false** | **false** | 497 µs | + +**Both dead states are detected on real hardware.** Note the two rows that matter most: + +- the **RETURN** row is the exact defect — `sdb`→`sdc`, the bind left on the dead superblock; +- the **IN-PLACE** row has **identical devnos on both sides** (the device never left) and the raw mount + still `active` — so the device comparison alone reads healthy there, and only the abort token catches + it. This is R-117a, the half that emits nothing today. + +The **absent** row is correct by design and worth stating: term 3 answers `unknown` because `devicePresent` +is the term that reports device absence, so term 3 abstains rather than double-counting. The overall +conjunction is false via term 2. It also demonstrates the cannot-tell rule working in a real state. + +**Cost: 340–497 µs**, against 3.85–5.03 ms that `GuestSeesMount` already spends forking `lxc-info`. + +### 8.2 No block I/O — the Part 1 fence applied to its own first consumer + +`strace -f -e trace=openat,statfs,fstatfs,read,pread64` while the predicate judged the **aborted** drive: + +``` +distinct openat targets: "/etc/localtime" "/proc/self/cgroup" "/proc/self/mountinfo" + "/sys/.../cpu.max" "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size" +openat/statfs against /mnt/r117*: NONE — the predicate never touched the mount +statfs syscalls: 0 +``` + +### 8.3 No regression on the live customer drive — through the REAL server pipeline + +`GET /disks` over the island bridge with the **controller's own credential** (from the guest's +`/etc/felhom-bootstrap/bootstrap.json`), i.e. the exact call the controller makes: + +``` +GET /disks http=200 bytes=2545 vmid=9201 +name=felhom-backup state=attached role=user-data +guest_path=/mnt/felhom-drives/nvme-1tb bound_under_parent=True backup_target=True +``` + +The live enrolled drive — which is also the whole-guest backup target — **reads present under 0.117.0.** +Host-side corroboration: exactly **1** bind entry at the stable path, `259:0` on both raw and bind, no +abort token ⇒ `BindLive`. And a positive observable that the consumer was actually evaluating (rule 3): +**32 gate-related controller log lines in 3 minutes**, with **zero** `ABSENT`/`RETURNED`/`backup_target` +transitions, and zero `bind is STALE` / `normalized to one bind` lines from the agent — so the new +predicate disturbed nothing. + +### 8.4 `AttachDrive` on hardware — and the one thing that was NOT repeated + +The healthy case ran through the real `AttachDrive` on the box: it returned the stable path, and the +**guest init PID was identical before and after (2555744)** — the guest was never restarted. + +**`AttachDrive` derives its target from `StablePathForRaw`, which hardcodes the production +`/mnt/felhom-drives`.** So invoking it for a scratch drive **necessarily creates a submount under the live +customer parent, which propagates into guest 9201** — which this task fences. That happened **once, +inadvertently, during the healthy case**: a bind appeared at `/mnt/felhom-drives/r117sd`. It was removed +within a minute and the revert was verified, not assumed: + +``` +live parent restored to exactly its two real submounts (nvme-1tb, Felhom-Share) +9201: running, init pid unchanged (never restarted) +stale scratch entries in 9201's namespace: 0 +9201 guest view: Felhom-Share nvme-1tb 9201 live drive read: OK +``` + +**It was deliberately NOT repeated for the stale-bind case**, so the *repair* leg is proven by +`TestAttachDrive_StaleBind_Rebinds` + RP6 through the real code, and on hardware only for the healthy +no-op. Doing it live would need ~20 s of a scratch submount inside guest 9201's namespace — harmless on +the evidence above, but the fence names 9201, so it is the operator's call, not one to take quietly. +**Named as the one gap in this run's hardware coverage.** + +--- + +## 9. Teardown — all three layers + +**Layer 1 — the machine.** `pct stop 9301` + `pct destroy 9301 --purge`; `pct list` back to `9201` alone. + +**Layer 2 — the host.** + +| | Before | After | +|---|---|---| +| `pvesm status` | `r117scratch` active, `local` **39.03 %** | `r117scratch` **removed**, `local` **37.07 %** (session start 37.02 %) | +| `df -h /var/lib/vz` | 16G, **42 %** | 15G, **40 %** (session start 40 %) | +| `dmsetup ls` | `r117cel (252:11)` + 11 pre-existing | **gone**; the 11 `pve-*` unchanged | +| `losetup -a` | 2 loops | **none** | +| `scsi_debug` | loaded | **unloaded**; `/dev/sd*` back to `sda1..3` | +| `/mnt` | `+ r117-drives, r117cel, r117sd` | `felhom-drives`, `nvme-1tb`, `vzsnap0` — as found | +| `/root`, `/var/lib/vz/r117*` | 5 scripts + `r117/` + probe | `ls -d /mnt/r117* /root/r117* /var/lib/vz/r117*` → **NONE** | + +The 124 MB template was removed. **One teardown defect, recorded because it is a reusable trap:** the +script hardcoded `losetup -d /dev/loop0`, but the loop numbers **swapped** this run (the guest rootfs took +loop0), so it detached the wrong device and left `/dev/loop1` attached to a deleted file. Fixed by +selecting loops by backing file (`losetup -a | grep r117`), not by number. **`local` is 0.05 % above the +session start** (~20 MB) — journal growth from the run, not a leftover artifact. + +**Layer 3 — the hub: one record was written, deliberately, and it is retained.** No customer, appliance, +pairing or claim was created (9301 had no NIC and ran no controller). The one hub-side change is the +**artifact manifest, now vouching agent 0.117.0** — the intended product of §6, not scratch, so it is +**retained** and is the fleet's vouched agent going forward. + +**Fences — re-verified after teardown, not assumed:** + +| Fence | State | +|---|---| +| guest **9201** | `running` throughout; guest view `Felhom-Share nvme-1tb`; live drive read **OK** | +| **`drill-r50`** (VM 300) | `stopped` — never started | +| **`local-lvm`** | **38.84 %** — byte-identical before, after, and to the session start | +| **`felhom-backup`** | `active`, 0.46 %, `content backup` unchanged — not re-targeted, not written | +| live `/mnt/felhom-drives` | **both** real submounts present; no `r117*` leftovers | +| `felhom-agent` on demo-hp | `active`, **0.117.0** | +| demo-felhom, Peti's cluster | never contacted | +| `felhom-controller` | **untouched**, as scoped | + +--- + +## 10. Findings — filed as `R-n`, none fixed + +| # | Finding | Severity | +|---|---|---| +| **R-121** | **A box's installed agent can sit releases behind the vouched one and nothing notices.** demo-hp ran 0.113.0 against a vouched 0.116.0 through the whole R-116/R-117 arc. The R-120 gate is golden-vs-fleet-controller and cannot see it; `MinAgent` is protective, not an alarm, and 0.113.0 *equalled* the floor. **Fourth instance of the drift family** | **MEDIUM** — filed with a fix shape and the honest tension (a box legitimately lags between publish and deploy) | +| **R-117g** | **An aborted filesystem is never cleared automatically, by design (§3), so a drive stuck in `emergency_ro` alarms forever until a human acts.** The alarm is correct and new; the absence of any guided recovery is the gap. A remount risks loss on a dirty abort, so it wants an operator-confirmed action, not a self-heal | **LOW-MEDIUM (follow-up to this fix)** | +| **R-117h** | **`AttachDrive` cannot be exercised against a non-production shared parent**, because `StablePathForRaw` hardcodes `StableParentDir`. Any hardware test of the repair path writes a submount into the live customer guest's namespace (§8.4). A test seam for the parent dir would make the repair leg testable on hardware without touching a customer | **LOW (testability)** | + +**Observations — noticed, documented, NOT acted on:** + +- **Guest 9201 carries 2 stacked entries** at `/mnt/felhom-drives/nvme-1tb` while the host has exactly 1. + Pre-existing, unrelated to this change, and harmless to the predicate (it reads the host table, and + `bindLiveness` iterates all entries). Worth a look — `AttachDrive` normalizes the *host* count only. +- **14 stale `felhom-agent.bak-*` binaries** on demo-hp (`0.92.1` … `0.113.0`), ~200 MB on `pve-root`. + Nothing prunes them. +- **11 files in the agent repo fail `gofmt -l`**, all pre-existing (`internal/capability/probe.go`, + `internal/storage/intent.go`, …). None are in `internal/localapi/`; none were touched here. + +--- + +## 11. What did not happen, and why + +- **The stale-bind repair was not exercised on hardware** (§8.4) — it would put a scratch submount inside + guest 9201's namespace, which this task fences. Covered by `TestAttachDrive_StaleBind_Rebinds` + RP6 + through the real code, and by the healthy-case `AttachDrive` on hardware (init PID identical). +- **No load or duration measurement of the predicate under a sustained reconcile loop.** The spike flagged + this as unmeasured and **it is still unmeasured** — said plainly rather than let to vanish. What exists: + single-call costs of 340–497 µs on hardware, and ~15 minutes of the real 20 s ticker on demo-hp with no + spurious re-bind and no CPU complaint. A many-drive box under sustained load was not tested. +- **`felhom-controller` was not touched**, per scope — the repair is already wired from its `Return` branch. +- **The R-116 row was not edited beyond §2.2's clause.** +- **felhom-pve was not upgraded** — it runs 0.115.0 and the task scoped the venue to demo-hp. The fleet is + now non-uniform: demo-hp 0.117.0, felhom-pve 0.115.0. Flagged, not acted on. +- **R-118 was not addressed** (the absent drive's union row advertising the root filesystem's capacity) — + a separate row, untouched by this fix. + +--- + +Every claim above is quoted from pushed Gitea source at `file:line` (agent `966d8f4`, hub `0.82.0`) or is +captured run output. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index c62877b..c421081 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -18,7 +18,8 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **WAITING-ON-OPERATOR (M)** | operator ruling on the release process | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design | CC | | **R-116** | ~~**The drive-absent alarm and its recovery were a MISMATCHED PAIR — absent fired the GENERIC `storage_disconnected`, return the SPECIFIC `backup_target_restored`; `backup_target_absent` never fired at all**~~ | **SHIPPED + PROVEN-LIVE** (agent v0.116.0, 2026-07-30) | — | **CLOSED. The full four-event sequence, on the wire, on a fresh box** (`audits/R116-v0116-2026-07-30.md`): `backup_target_absent (error)` on detach → `backup_target_restored (info)` on return for the TARGET, and `storage_disconnected (error)` → `storage_reconnected (info)` for a NON-target drive on the same box four minutes apart. **Two matched pairs, correctly discriminated — and discrimination is proven NON-trivially for the first time**, since both prior runs had the target itself emit the generic event. Gate fired in **3 s**; all four events reached the hub, so the specific alarm, its severity, its Hungarian copy and the hub routing are now exercised end-to-end. **Over-correction PASSES with a positive observable** (0 ABSENT lines / 0 drive events over 2m14s with both drives present, target `degraded:false`, while 2 `RETURNED` lines prove the gate was ticking). **NARROWED by the R-117 spike (`audits/SPIKE-r117-bind-liveness-2026-07-30.md` §12), and it stands as written:** the 2 `RETURNED` lines are a genuine positive observable, so rule 3 is satisfied — but `degraded:false` over that window was read off a drive whose bind was **dead** (R-117), so the window evidences **"the gate did not over-fire"** and **NOT** **"the drive was healthy."** No other part of this row changes: every input to the pairing fix is configuration-derived (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), which R-117 does not touch. Ran on a nested PVE on **demo-hp** per `runbooks/target-selection.md` — through the **real day-0** from the v1.25.0 ISO, with the agent **installed unaided from the vouched Day-0 manifest** (published sha `b47c5c4dab641ee5…`, independent registry GET verified, manifest read back), drives enrolled through the real endpoints, device loss a real hot-detach. **THE FIX, and the ruling is the substantive part:** the mechanism was first isolated from the captured payload (`DIAG-r116-disks-payload-2026-07-30.md`) after two fixes aimed at shapes that do not occur. **Both smaller-looking options were REJECTED because they regress R-114** — `backup_target_offer.go:79` reads `BackupTarget && MountPath != ""` as *"a real drive with its own mountpoint — healthy"* and returns before its `TargetAbsent` branch, so back-filling `MountPath` on the Observe row **or** flagging the registry row (whose `MountPath` is the stale unit-file value) would have told the customer the backup target is fine while its drive was gone. **R-114's correctness was resting on R-116's bug** — a coupling invisible until the payload existed. Taken instead: the Observe row gets the **guest path only** (`mount_path` stays `""`, which is true) from a new `ConfigPath` (`json:"-"`, so the cross-repo golden + key-set contract is untouched), and the union row is deduped **on guest path** — the join being CONFIGURATION (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), the only identity that survives the device. Tests 845→849; 4 red-proofs each asserted to land, and red-proof 1 replays v0.115.0's code and fails, which is the empirical proof it was inert. Its green test had supplied a `MountPath` production never supplies AND left `DriveTargets` nil so the union loop never ran — both corrected. v0.115.0 left in place (inert, harmless). Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded. **Caveat: the drill's controller was 0.185.1 from the golden, which PREDATES R-114**, so its absent-state banner showed the old false copy — the golden being a release behind, not a regression → **R-120** | — | | **R-120** | ~~**The golden baked a controller that predated R-114 + R-112, so a FRESH box showed the customer the WRONG absent-target message**~~ | **CLOSED — golden rebaked + PROVEN-LIVE, and the class now has an ENFORCED gate** (golden 0.186.0 + hub v0.82.0, 2026-07-30) | — | **`audits/R120-golden-rebake-2026-07-30.md`.** **Half 1 — the artifact.** Golden **0.186.0** baked from `main`'s controller in the DooPlex bake fixture (overlay2 OK, **3 mounts**, FATAL 0, exclusions 0, 618 MB, upload **201**, `GOLDEN_SHA256=b760ac6a33e70700…`, token-leak grep 0, GL-1 teardown, `drill.qcow2` back to `virgin`). Three observables: **published** — anonymous GET (what the installer does) 200 / 648930639 bytes / sha identical to the bake; **vouched** — manifest read BACK; **resolved** — `Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)`. Floor **untouched** per publish-train rule 2 (`min_controller_version` still 0.156.0; it is a separate form); MinAgent left 0.113.0 as 0.186.0 declares. **Proven on a REAL day-0, not the fixture** (per the Part-1 rule now in `runbooks/target-selection.md`): VM 9402 on demo-hp from the v1.25.0 ISO → `Controller elindult (0.186.0)`. With the target detached the endpoint returned the **`TargetAbsent`** copy — *„A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod…"* — **and `offer_path` absent entirely**; the day-old read on the 0.185.1 golden had returned the false system-disk message **plus** an offer of the other drive. **Half 2 — the mechanism, operator ruling REFUSE.** hub **v0.82.0**: the gate sits in `hub/internal/web/configs.go` `handleSetArtifacts` immediately before the only write — the sole UI path to `SetArtifactManifest` — so it runs on every vouch without anyone choosing to, and it **refuses** rather than warning. Signal: `store.NewestReportedControllerVersion()` over `reports.controller_version`, **semver-compared in Go** (`MAX()` in SQL ranks 0.99.0 above 0.186.0 — a pair this fleet has shipped). Fail-open in exactly two deliberate cases: empty golden field, unknown fleet version. **NEAR-MISS RECORDED: the first draft read `guests.controller_version`, a column that exists and that NOTHING writes** — it would always have seen `""` and failed open, i.e. inert, this gate's own failure shape, one grep from shipping. 4 tests through the **production handler** over httptest (never a seam), the refusal asserting **both** the flash **and** that the manifest was not written; red-proof: deleting the block makes the stale golden vouchable again. **PROVEN LIVE on the deployed hub by re-attempting the original mistake:** vouching 0.185.1 → `HTTP 303 …flash=golden_behind_fleet` + `[WARN] artifact vouch REFUSED: golden 0.185.1 is older than the newest controller the fleet reports (0.186.0)`, and the manifest read back **unchanged at 0.186.0**. Recorded on **R-29's audit list** (`ROADMAP.md`) as the **first enforced gate** beside its three orphans, so the contrast is kept — the orphans are unchanged. Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded, exactly as `sess-e` was (and `sess-e` was deleted this run) | — | -| **R-117** | **A drive's guest bind becomes a DEAD MOUNT while every signal reads healthy — and it happens in TWO ways, only one of which the original framing covered.** (a) *after a detach/return*: the host raw mount heals onto the NEW device via its fs-UUID-keyed unit while the bind still names the OLD one, so the gate takes its `Return` branch and restarts the customer's apps onto a namespace that `EIO`s on every call; (b) *in STEADY STATE, no cycle at all* — a device that errors without disappearing leaves the raw mount `active`, `BoundUnderParent` `true` and the drive never `Disconnected`, so **the gate produces no action and NOTHING is emitted on any channel** | **MECHANISM + RECIPE + FIX CONSTRAINTS ESTABLISHED (spike complete 2026-07-30) — READY (S/M), not fixed** | — | **SPIKE: `audits/SPIKE-r117-bind-liveness-2026-07-30.md`** — reproduced deliberately **3/3**, two device classes, on a purpose-built scratch LXC (9301) on demo-hp; predicates evaluated by a throwaway probe calling the **real** `localapi` code from agent `d4eb259` (v0.116.0). **MECHANISM, confirmed at source and live:** both halves of the R-113 conjunction are *path-presence tests*. `GuestSeesMount` (`intermediary.go:276`) and `isHostMountpoint` (`intermediary.go:394`) each compare **field 5** (the mount point) of a mountinfo line and **never read field 3, the `major:minor`** — so neither can see that the bind and the raw mount name **different devices**. Measured: `raw 8:32 /dev/sdc` vs `bind 8:16 /dev/sdb …,shutdown`, `BoundUnderParent = TRUE`, `EIO`(5) on read **and** write, host and guest. **The node change is a CONSEQUENCE, not a precondition** — control test: with the stale bind held the drive returns as `sdc` (8:32); once it is `umount`ed the letter is **reused** (`sdb`, 8:16). The stale bind pins the dead superblock, which forces the new number — so a devno comparison is *sound* for case (a), not a heuristic. **RECIPE** (spike §10): `scsi_debug` is load-bearing — it is the only mechanism that removes a gendisk **while mounted** (dm and loop are held open by the stale bind: `dmsetup remove --force` → `Device or resource busy`). `modprobe scsi_debug` → mkfs → UUID-keyed `systemd-mount` → bind under a **scratch** shared parent → `echo 1 > /sys/block/sdb/device/delete` (systemd stops the raw unit itself, `BindsTo=`) → `echo '- - -' > /sys/class/scsi_host/hostN/scan` → re-mount by UUID. Case (b): swap a dm table to `error`. **TWO DEATH STATES, different signals** — device removed ⇒ superopts gain `shutdown`, `EIO` both ways; device errors in place ⇒ `emergency_ro`, write `EROFS`(30), reads still served from cache. **PROBE TRAP (R-117b): `statfs` and `getdents` SUCCEED on a namespace that `EIO`s every byte** — any liveness probe built on either is hollow by construction. **A pure-`/proc` check costs 0.17–0.45 ms**, cannot hang, spins up no disk and writes nothing, against the 3.85–5.03 ms `GuestSeesMount` already spends forking `lxc-info`. **CONSTRAINT: the fix needs BOTH a devno comparison (closes (a); blind to (b)) and the fs abort flags `shutdown`\|`emergency_ro` (closes (b); ext4-specific)** — the tempting devno-only fix leaves the silent half open. Keep the *cannot-tell → never absent* rule (`disks.go:983-986`). **R-117d — THE FIX IS ONE PREDICATE, AND IT SELF-HEALS:** `AttachDrive`'s normalize leg (`intermediary.go:238-247`) already performs exactly the repair — proven live, **guest never restarted** (init PID identical) — and **three** call sites already invoke it: the 20 s reconcile ticker (`main.go:973-985`), agent startup (`disks.go:951`), and **the controller's `Return` branch itself, at `intermediary.go:301-305`, BEFORE it restarts the apps at `:311`**. All three are defeated by `if n == 1 && b.GuestSeesMount(...)` returning early at `intermediary.go:235` and logging *"fully live"*. **Unblock the existing path; do not add a new one.** **R-117c:** three comments (`disks.go:83`, `:154-155`, `:155-165`) promise "live + usable in the guest"; nothing tests it and it is false in the measured state — the fix owes a test asserting the **consequence** (dead namespace ⇒ `BoundUnderParent=false` ⇒ alarm fires), plus a second scenario for case (b). **R-117f — THE HANG CASE, MEASURED, AND IT DISQUALIFIES I/O PROBING OUTRIGHT** (§6.3): against a `dmsetup suspend`ed device **every probe that touches it blocked — including a buffered write with no `fsync`** (the `O_CREAT` metadata path needs journal access) — while **P1 and P2 completed in 364 µs / 206 µs**. The blocked process then survived `SIGTERM` **and `SIGKILL`** (`stat=D`, `wchan=folio_wait_bit_common`, still alive 3m50s after `kill -9`) and died only when the device was resumed. **So `systemctl restart felhom-agent` would hang, leaving the agent unrecoverable until the device returns or the host reboots** — and the thread count does not reveal the leak (5→5, 5→6). A timeout protects the caller's control flow and nothing else. **The fix must issue no block I/O; that is now a fence, not a preference.** `statfs`/`getdents` reported HEALTHY on the wedged device too, confirming R-117b in a second failure mode. **Teardown DONE, all three layers** (spike §14): guest 9301 destroyed, `r117scratch` removed, both dm devices and both loop devices gone, `scsi_debug` unloaded, `local` back to 37.02 % (session start 37.00 %); fences re-verified after — 9201 `running`, `drill-r50` `stopped`, `local-lvm` **38.84 % byte-identical**, `felhom-backup` `content backup` unchanged, live `/mnt/felhom-drives` intact with both submounts, agent `active`. **Layer 3 genuinely empty** — 9301 had no NIC and ran no controller, so no hub-side record was ever created. Note demo-hp's live agent is **0.113.0** — it does not run the R-113 conjunction, so live validation of the fix needs that box current first | CC | +| **R-117** | ~~**A drive's guest bind becomes a DEAD MOUNT while every signal reads healthy — and it happens in TWO ways, only one of which the original framing covered.** (a) *after a detach/return*: the host raw mount heals onto the NEW device via its fs-UUID-keyed unit while the bind still names the OLD one, so the gate takes its `Return` branch and restarts the customer's apps onto a namespace that `EIO`s on every call; (b) *in STEADY STATE, no cycle at all* — a device that errors without disappearing leaves the raw mount `active`, `BoundUnderParent` `true` and the drive never `Disconnected`, so **the gate produces no action and NOTHING is emitted on any channel**~~ | **SHIPPED + PROVEN-LIVE** (agent **v0.117.0**, 2026-07-30) | — | **CLOSED. `audits/R117-v0117-2026-07-30.md`.** `BoundUnderParent` gains a THIRD term at both /disks sites: `bindLiveness` reads `/proc` only and requires (a) **the bind names the same device as the raw mount** and (b) **the filesystem has not aborted** (`shutdown` **or** `emergency_ro`, both measured). **BOTH CHECKS ARE LOAD-BEARING and this is the substantive part:** R-117 was filed as a detach/return defect, but a device that fails WITHOUT disappearing gives the identical all-signals-healthy state with the **devnos EQUAL** and the drive never `Disconnected`, so the gate emits nothing at all, indefinitely (R-117a) — the device comparison alone cannot see it, and a P1-only fix passes every payload test (red-proof RP3 exists for exactly that). **THREE states, never a bool:** `{Unknown, Live, StaleDevice, Aborted}`, `Unknown` is the zero value, and every caller reads `Usable()` where unknown counts **PRESENT** (absent stops a customer's apps — the `newestArchiveOn` trap). **NO NEW RECOVERY PATH:** `AttachDrive`'s normalize leg already did the repair and three call sites already invoked it (20 s ticker, agent startup, and **the controller's `Return` branch BEFORE `restartStacks`**); all three were defeated by `if n == 1 && GuestSeesMount(...)` logging *"fully live, no-op"* about an EIO namespace. **RULING (asked for, given, flagged for overrule):** `StaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, guest never restarts — init PID identical); `Aborted` ⇒ **quiet no-op and SURFACE**, because a re-bind lands on the SAME dead superblock and this runs every 20 s = an infinite silent retry that masks the state. No operator decision required: it routes an already-broken state into the **existing** gate, event types and Hungarian copy — no new customer-facing concept — and the alternative is apps writing documents into a filesystem that rejects every write. **ORDERING TRAP caught by a test:** abort-first classifies the real return state as aborted (its stale bind carries `shutdown` too) and refuses the repair **while still reporting correctly**, so the abort flag is read off the RAW mount in the stale case. **LIVE on demo-hp** (brought 0.113.0 → 0.117.0 first — see R-121): RETURN `raw 8:32 / bind 8:16 shutdown` ⇒ `stale-device`, usable **false**; IN-PLACE `both 252:11 emergency_ro`, raw unit still `active` ⇒ `filesystem-aborted`, usable **false**; healthy ⇒ `live`; **340–497 µs**. **No block I/O proven by strace** (only `/proc/self/mountinfo`, **0** statfs) — the Part 1 `CLAUDE.md` fence applied to its own first consumer. **No regression through the REAL pipeline:** `GET /disks` with the controller's own credential shows the live backup-target drive `bound_under_parent=True`, with 32 gate lines in 3 min as the positive observable and zero spurious transitions. Tests **849→863**, 29/29 green, **6 red-proofs each verified to land** — and **RP1 failing to fail exposed a HOLLOW test**: the aborted fixture used a `/dev/mapper` device, for which `RoleForStorage` derives `role=system`, and a system row never runs the conjunction, so it reported false by DEFAULT and no mutation could fail it. Fixtures now assert the production row shape first. Teardown all 3 layers; hub layer = the vouched manifest, **retained** (it is the product, not scratch). **NOT covered:** the stale-bind repair on hardware — `StablePathForRaw` hardcodes the live parent, so it would write into guest 9201's namespace (R-117h); and sustained-load behaviour, still unmeasured. Follow-ups **R-117g** (no guided recovery for an aborted fs), **R-117h** (parent dir not test-seamable), **R-121** | CC | +| **R-121** | **A BOX's installed agent can sit releases behind the vouched one and nothing notices — the R-120 gate does not cover it.** demo-hp ran agent **0.113.0** while the hub vouched **0.116.0**, through the whole R-116/R-117 arc, and no signal existed on any channel | **READY (S) — NEW 2026-07-30** | — | **Fourth instance of the drift family** (R-111 golden's agent 17 releases behind, R-115 built+deployed but never published, R-120 golden a controller behind — and now installed-vs-vouched on a live box). **Confirmed at source that R-120's gate cannot catch it:** `hub/internal/web/configs.go:1165-1169` compares `goldenVer` against `store.NewestReportedControllerVersion()` — it is a **golden-artifact vs fleet-CONTROLLER** check and says nothing about the agent installed on a box. **`MinAgent` does not cover it either:** it is used to HOLD the controller floor for a box whose agent is too old (`hub/internal/api/handler.go:530-538`, `store.go:1857`) — protective, not an alarm — and demo-hp's 0.113.0 **equalled** `min_agent` 0.113.0, so even a floor comparison was satisfied. **The cost, measured:** R-117's whole subject is the R-113 conjunction, which landed in **0.114.0** — so the designated drill host could not exercise the code under investigation at all, and the R-117 spike had to route every predicate result through an out-of-repo probe built from `main` instead of the installed agent (`audits/SPIKE-r117-bind-liveness-2026-07-30.md` §1, §2.3). Discovered because the R-117 task made bringing the box current an explicit prerequisite. **Fix shape (not implemented):** the hub already receives `AgentVersion` on every host report, and already has semver comparison in Go — the missing piece is a checker comparing reported agent vs the **vouched** agent and surfacing it, operator-tier. Note the honest tension: a box legitimately lags between publish and deploy, so this wants a staleness window rather than an instant alarm | CC | | **R-118** | **An absent drive's union row advertises the ROOT filesystem's capacity as its own.** In the absent-state payload the registry-union row reports `total_bytes: 49675956224 / used_bytes: 4584579072` — **byte-identical to the `local` row** (`durable_id: path:/var/lib/vz`, i.e. `pve-root`) in the same response. The real drive is **4 GB** | **READY (XS) — NEW 2026-07-30** | — | Cause: `statfsCapacity(d.MountPath)` (`disks.go:335-338`) statfs's `/mnt/cel`, which with the device gone is a **bare directory on the root filesystem**. `observe.go:176-183`'s comment warns about exactly this trap and guards the Observe path ("*an unmounted removable dir-storage's mountpoint reverts to a bare directory on root … catastrophic DR mis-id*"); **the union path has no equivalent guard.** **Not a DR mis-id** — `durable_id` on that row is still the correct `uuid:…`, so re-attach identity is safe. It is a **false capacity** reaching every consumer of `total_bytes`/`used_fraction` (fill monitors, storage cards): a detached 4 GB drive advertises 46 GiB at 9.2 % used. Same class as `role.go:180-181` — an absent drive's fields decaying to the root filesystem's. Evidence: `audits/DIAG-r116-disks-payload-2026-07-30.md` §12 | CC | | **R-113** | ~~**The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.**~~ `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc//mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **SHIPPED + PROVEN-LIVE** (agent v0.114.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** Target drive hot-detached on a fresh box running the SHIPPED agent 0.114.0 (from the Day-0 manifest, not a hand build). **The gate fired in 4 seconds** — E-2d measured ZERO over 4½ minutes — and `SetDisconnected` was reached. It fired on exactly the shape that defeated it: raw `/mnt/mentes` NOT mounted while the bind `/mnt/felhom-drives/mentes` still read `/dev/sdb[/felhom-data]`. **Over-correction guard PASSED:** with both drives present, 0 ABSENT lines and the target stayed healthy. **Note: C5 still FAILED — but on a different defect (→ R-116), not on this one.** **SESSION C IS NOW UNBLOCKED.** Agent **0.114.0 published and vouched** 2026-07-29 (sha `5e4c15ebee2d7583…`, round-trip GET verified; hub manifest read back showing it) — a fresh drill box now installs the fix under test instead of the bug. The golden was NOT re-baked and `min_agent` was NOT raised: the golden bakes the controller, not the agent, and controller v0.185.0 declares MinAgent 0.113.0, which 0.114.0 already satisfies. **FIX: `BoundUnderParent` is now a CONJUNCTION — bound under the parent AND the drive's raw host mount still mounted** (`devicePresent`, new `deviceCheck` seam), at BOTH `/disks` construction sites. The raw mount is the device-bound systemd unit that dies with the device; the agent's bind is not — so the raw mount IS the device signal, grounded in E-2d's measurement rather than inference. **Conjunction, deliberately:** the device half alone would regress boot ordering (raw mounts early, bind lands ~18 s later — that window must keep reading absent), so existing behaviour is byte-identical and only the unreachable case is closed. **Unknown is never absent** (`devicePresent("")` = true) — a false absent stops a working customer's apps. **Controller UNCHANGED, no MinAgent bump:** `BoundUnderParent` has exactly one functional consumer (`planDriveGates:226`); a new `DevicePresent` bool was rejected because absent-from-JSON decodes to `false`, which would have made every drive on an older agent read ABSENT. +6 tests (208→214), 4 red-proofs run and reverted. **Deployed to demo-felhom; over-correction guard verified in production** — raw mount present, drive still reads present, 10/10 apps untouched, no gate action, no false alarm. **demo-hp deliberately left on 0.113.0** (the spec scoped deploy to felhom-pve) — it still carries the bug. **⚠️ SESSION C BLOCKER: the hub Day-0 manifest vouches agent 0.113.0**, so a fresh drill box would install WITHOUT this fix and validate nothing — publish + vouch 0.114.0 first (R-111's trap, same shape). **Awaiting live proof of exactly one leg:** device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. Original finding: **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 **Session C scope UNCHANGED by Session B** — R-114/R-112 shipped as controller v0.186.0 and neither touches the agent; the leg awaiting proof is still device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. One rebuild now validates all three. | CC | | **R-112** | **E-2's degraded banner and offer have NO UI CONSUMER — the endpoint is correct and the customer never sees it.** `GET /api/storage/backup-target` returns byte-exact copy (proven live), and **nothing fetches it**: `grep 'backup-target'` across all `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` + `degradedMessageFor` are consumed **only** by the JSON handler — **no page handler injects the state** | **SHIPPED + PROVEN-LIVE** (controller v0.186.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md` — the banner reached a customer's page for the first time.** Never-configured box: banner element 1, never-configured copy 1. After the wizard: offer block 1 with `data-path="/mnt/felhom-drives/mentes"`. Healthy after assign: all four markers 0, **proven POSITIVELY** — idle delta 0 `/backup/tiers` calls, page-load delta +1, single caller ⇒ the seam ran and chose silence. **FIXED: the state now has a consumer.** Server-rendered on `/backups` via `backupsHandler` → `backupTargetView` → `backups.html`, following the existing `SingleCopyWarning` banner pattern — NOT a 19th JS fetch, because a banner that needs JavaScript to appear is one more thing that can silently not happen. `backupTargetView` returns **nil** for healthy AND unknown, so those render nothing at all. **Scenario-E seam test drives `backupsHandler` over httptest and asserts the RENDERED HTML** — deleting the one line that sets `data["BackupTarget"]` reproduces the old state and fails every render assertion. **SEAM PROVEN LIVE on demo-felhom by a DIFFERENTIAL positive observable, not by an absent banner:** idle 8 s → 0 new `/backup/tiers` agent calls; each `/backups` load → exactly +1, and that call has only one caller (`resolveBackupTargetState`). The box is healthy (`degraded:false, target:felhom-backup`) and the page correctly rendered **nothing** — which matches its real state but, being a negative, is NOT by itself proof of wiring. **Still unproven live:** that a customer sees actual copy — impossible on a healthy box. **The decisive contrast: templates fetch 18 distinct `/api/storage/*` endpoints; `backup-target` and `backup-target/assign` are the only two with zero references.** The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"* — an invariant comment asserting a consumer that does not exist (7th instance of that class). v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* — it fixed the **router mount** and stopped one layer short of the **render**; its test `TestBackupTargetRoutesLiveUnderTheStorageAPIMount` pins dispatch, not reachability, which is exactly what `CLAUDE.md`'s seam rule warns about. **Fifth instance of seam-built-but-never-wired.** **Fix R-114 FIRST** — wiring this alone would start showing customers the wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | CC |