From 3d504d58c84db9b712290d1c87f226f64c666541 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 30 Jul 2026 12:42:26 +0200 Subject: [PATCH] =?UTF-8?q?docs(R-117):=20CLOSED=20=E2=80=94=20proven=20li?= =?UTF-8?q?ve=20on=20demo-hp;=20R-121=20filed=20for=20agent-on-box=20drift?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit R-117 row → SHIPPED + PROVEN-LIVE (agent v0.117.0), with the full validation in audits/R117-v0117-2026-07-30.md. Both dead states detected on real hardware through the shipped predicate: RETURN raw 8:32 /dev/sdc | bind 8:16 shutdown → stale-device, usable false IN-PLACE both 252:11 emergency_ro, raw unit active → filesystem-aborted, usable false healthy → live 340-497us per call. No block I/O proven by strace (only /proc/self/mountinfo, 0 statfs) — the Part 1 CLAUDE.md fence applied to its own first consumer. No regression through the real pipeline: the live backup-target drive reads bound_under_parent=True via GET /disks with the controller's own credential, with 32 gate lines in 3 min as the positive observable and zero spurious transitions. The ruling asked for in §2.2 is recorded in full and flagged for overrule: Aborted must NOT self-heal. A re-bind lands on the same dead superblock and the call site runs every 20s, so repairing would be an infinite silent retry that masks the state. It surfaces instead. No operator decision was taken quietly — the reasoning is that it routes an already-broken state into the existing gate, event types and Hungarian copy, so no new concept reaches the customer. R-121 filed: a box's installed agent can sit releases behind the vouched one and nothing notices. demo-hp ran 0.113.0 against a vouched 0.116.0 through the whole R-116/R-117 arc. Confirmed at source that R-120's gate cannot catch it — it compares goldenVer against NewestReportedControllerVersion(), i.e. golden-artifact vs fleet-CONTROLLER. MinAgent is protective, not an alarm, and 0.113.0 equalled the floor. Fourth instance of the drift family. Also filed: R-117g (an aborted filesystem is never cleared automatically by design, so it alarms until a human acts, with no guided recovery) and R-117h (StablePathForRaw hardcodes the parent, so the repair path cannot be exercised on hardware without writing into a live customer guest's namespace). --- documentation/audits/R117-v0117-2026-07-30.md | 437 ++++++++++++++++++ documentation/backlog/OPEN-ITEMS.md | 3 +- 2 files changed, 439 insertions(+), 1 deletion(-) create mode 100644 documentation/audits/R117-v0117-2026-07-30.md diff --git a/documentation/audits/R117-v0117-2026-07-30.md b/documentation/audits/R117-v0117-2026-07-30.md new file mode 100644 index 0000000..7563c59 --- /dev/null +++ b/documentation/audits/R117-v0117-2026-07-30.md @@ -0,0 +1,437 @@ +# R-117 v0.117.0 — the liveness signal now tests liveness, proven on hardware + +Implementation + live validation of `audits/SPIKE-r117-bind-liveness-2026-07-30.md`. Agent **v0.117.0**. +`felhom-controller` untouched: the spike proved the repair is already wired from the controller's +`Return` branch and dies inside the agent. + +--- + +## 1. Baselines — each established separately + +| Thing | Value | How | +|---|---|---| +| agent `main` before | **v0.116.0** @ `d4eb259` | `git log`, `CHANGELOG.md:1`; tree clean, `HEAD == origin/main` | +| agent shipped this run | **v0.117.0** @ `966d8f4` | commit below | +| **agent on demo-hp BEFORE** | **0.113.0** | `felhom-agent --version` on the box — **predates R-113 (0.114.0) entirely** | +| **agent on demo-hp AFTER** | **0.117.0** | same command, post-deploy | +| agent published + vouched before | published 0.113–0.116; **vouched 0.116.0** (`b47c5c4dab641ee5…`) | hub `/configuration`, read not assumed | +| controller `main` | **v0.186.0** @ `b331f18` | `git log` | +| golden bakes controller | **0.186.0** | hub `/configuration` — **matches `main`, so R-120's gate is holding them together** ✅ | +| `min_agent` / `min_controller_version` | 0.113.0 / 0.156.0 | hub `/configuration` | +| hub live | **0.82.0** | `kubectl -n felhom-system get deploy` | +| `felhom.eu` HEAD before | `e70b5fe` | as the task expected | +| demo-hp | PVE 9.2.2, kernel 7.0.2-6-pve | `pveversion` | + +--- + +## 2. Part 1 — the two record items, committed before any Go file was opened + +**`felhom.eu` `37515cd`** — `docs(R-117 Part 1)`. **`felhom-agent` `6be168d`** — the same rule repeated +where it binds. + +### 2.1 The `CLAUDE.md` rule, as written + +Placed beside the seam-wiring rule in `felhom.eu/CLAUDE.md` "Code quality rules": + +> **A health check issues no block I/O.** A probe that touches a wedged device enters uninterruptible +> sleep, survives `SIGKILL`, and cannot be recovered until the device returns or the host reboots — so +> `systemctl restart` hangs too. A timeout protects the caller's control flow and nothing else: the +> blocked thread remains. Liveness is decided from `/proc` and the kernel's own state, never by reading +> or writing the filesystem. Measured, R-117 spike §6.3 (…): a probe stayed in `D` state 3m50s after +> `kill -9`; a buffered write with no `fsync` blocked too (`O_CREAT` needs journal access); and +> `statfs`/`getdents` returned **healthy** on a namespace that `EIO`s every byte — fast, and wrong. + +**Repeated in `felhom-agent/CLAUDE.md`**, as a one-line pointer, and the reason is recorded there: health +checks are written in the agent repo, and `felhom.eu/CLAUDE.md` **does not load in an agent-only session**. +A standing rule that does not load where it binds is the inert-seam shape applied to a rule. The task named +one location; this is the one deviation, stated rather than absorbed (standing rule 4). + +### 2.2 The R-116 narrowing, applied + +The R-116 row's over-correction clause now carries: the 2 `RETURNED` lines are a genuine positive +observable so rule 3 is satisfied, but `degraded:false` was read off a drive whose bind was dead, so the +window evidences **"the gate did not over-fire"** and **not** **"the drive was healthy."** Nothing else +about the row changed. + +--- + +## 3. THE RULING (task §2.2) — what happens when P2 trips + +**Ruling: P2 must NOT self-heal. It surfaces. Operator decision NOT required — and the reasoning for that +is the substantive part, so it is stated in full for overruling.** + +The two states have different repairs, and *"would a re-bind help?"* — not *"is something aborted?"* — is +the question a verdict must answer: + +| Verdict | What a re-bind would do | Ruling | +|---|---|---| +| **`BindStaleDevice`** | the raw mount is a **different, healthy** superblock (the drive returned and the fs-UUID-keyed unit healed onto it), so umount + re-bind lands the namespace on working storage | **REPAIR.** Falls through to the existing normalize leg | +| **`BindAborted`** | the raw mount is the **same aborted** superblock — a fresh bind to a still-dead filesystem | **QUIET NO-OP, and surface via `BoundUnderParent=false`** | + +**Why `BindAborted` must not repair,** three reasons and the third is the decisive one: + +1. It cannot work. Re-binding does not clear an ext4 abort; that needs a remount or a fsck. +2. `AttachDrive` runs **every 20 s** (`cmd/felhom-agent/main.go` ticker), so re-binding would be an + **infinite silent retry** — R-117a's silence with more CPU. +3. It would **mask** the state. The whole point of the fix is that this state becomes visible. + +**Why no operator decision was needed.** The ruling changes customer-visible behaviour: apps that today +keep running against a read-only/`EIO` namespace will be **stopped**, and the customer gets an alarm. That +is a real change — but it is the **identical treatment an unplugged drive already receives**, through the +existing gate, the existing event types and the existing Hungarian copy. No new message is invented, no +new concept reaches the customer. The alternative is letting applications keep writing a customer's +documents into a filesystem that rejects every write. **Fail-closed is also the project's standing +direction for drive state.** So this is routing an already-broken state into an already-tested alarm, not a +new customer-facing decision. **Flagged prominently rather than decided quietly; overrule freely.** + +**Deliberately NOT done: automatic remount of an aborted filesystem.** That is a repair of the customer's +data-bearing filesystem, risks loss on a dirty abort, and is an operator call. Recorded as R-117g. + +--- + +## 4. The predicate as implemented + +`felhom-agent/internal/localapi/intermediary.go`: + +``` +bindLiveness(stable, raw) → + stable=="" or raw=="" → BindUnknown + no bind entry at `stable` → BindUnknown (isHostMountpoint's question, not this one) + no raw entry at `raw` → BindUnknown (devicePresent already reports device absence) + bind.devno != raw.devno → raw aborted ? BindAborted : BindStaleDevice ← P1 + (devnos agree) bind aborted → BindAborted ← P2 + (devnos agree) fstype unknown → BindUnknown + otherwise → BindLive +``` + +Wired as the **third term** of the conjunction at **both** `/disks` construction sites (`disks.go`): +`boundUnderParent(...) && devicePresent(...) && bindUsable(gp, )`. + +**Order is load-bearing, and a test caught it.** The first draft read the abort flag before comparing +devices. In the real return state the stale bind carries `shutdown` **as well as** a different device, so +abort-first classifies it `BindAborted` — **reporting stays correct, every payload test still passes, and +the repair silently never runs.** The abort flag is therefore read off the **raw** mount in the stale case +(the re-bind's target) and off the bind only once the devices already agree. + +### 4.1 How *unknown* is expressed + +A **three-state type**, never a bool: `BindLiveness ∈ {BindUnknown, BindLive, BindStaleDevice, +BindAborted}`, with `BindUnknown` as the **zero value**. Every caller reads it through **one** method: + +```go +func (l BindLiveness) Usable() bool { return l == BindLive || l == BindUnknown } +``` + +So the "cannot tell → never absent" rule lives in exactly one place and no caller can restate it wrongly. +The task named the `newestArchiveOn` trap — a `(value, bool)` shape whose comment promised a degradation +the signature could not express. Four routes to unknown are pinned by test: unreadable `/proc`, no raw +entry, empty paths, and a filesystem whose abort vocabulary is unmeasured. + +**P2's tokens: `shutdown` AND `emergency_ro`**, both measured. `ext4`/`ext3`/`ext2` (one driver); anything +else → **unknown, never live**. + +### 4.2 Not a new recovery path + +`AttachDrive`'s early return now switches on the verdict instead of declaring a dead namespace +*"fully live, no-op"*. The three call sites that already invoked the repair — the 20 s ticker, agent +startup, and the controller's `Return` branch **before** `restartStacks` — are unchanged. Also: +`isHostMountpoint` and `countHostMounts` are now one-liners over a single new parser `hostMountEntries`, +which yields devno/root/fstype/super-options (`REUSE.md`'s three-independent-readers note narrowed). + +--- + +## 5. Tests + +**849 → 863** (+14 top-level), **29/29 packages green**, `go build` / `go vet` / `go test` each with +`rc=0` read separately (never combined with a commit). The 849 baseline was **verified against a temporary +worktree at HEAD**, not inferred — the first draft of the CHANGELOG line said 858 → 873 from a count taken +mid-edit, and that was corrected before commit. + +| Test | Asserts (the consequence) | Result | +|---|---|---| +| `TestDisks_BindLiveness_StaleBindReadsAbsent` | R-117 case (a) → `bound_under_parent` false **through the real `/disks` handler** | PASS | +| `TestDisks_BindLiveness_AbortedFilesystemReadsAbsent` (×2 tokens) | R-117a case (b) → false, with a **row-shape guard** | PASS | +| `TestDisks_BindLiveness_UnionPath_StaleBindReadsAbsent` | union path, case (a) | PASS | +| `TestDisks_BindLiveness_UnionPath_AbortedReadsAbsent` | union path, case (b) — Role+State hardcoded there | PASS | +| `TestDisks_BindLiveness_HealthyReadsPresent` (observe+union) | no false negative | PASS | +| `TestDisks_BindLiveness_UnknownIsTreatedAsPresent` (5 subtests) | all four unknown routes → **present**; `Usable()` truth table | PASS | +| `TestBindLiveness_Verdicts` (4 states) | each verdict | PASS | +| `TestBindLiveness_AbortedWins_WhenDevnosAgree` | the P1-only fix cannot pass | PASS | +| `TestHostMountEntries_ParsesDevnoAndSuperOpts` | `" - "`-separator field extraction | PASS | +| `TestHostMountEntries_CountsStackedBinds` | double-bind convergence survives the refactor | PASS | +| `TestAttachDrive_StaleBind_Rebinds` | **the repair RUNS** (umount+rebind recorded) | PASS | +| `TestAttachDrive_AbortedFilesystem_DoesNotRebind` | **§3's ruling**: zero mount ops, no error | PASS | +| `TestAttachDrive_Healthy_IsStillANoOp` | 20 s idempotency | PASS | +| `TestAttachDrive_UnknownLiveness_IsANoOp` | cannot-tell must not churn | PASS | + +**Fixtures are the spike's captured mountinfo**, with two substitutions both recorded in the file: the +scratch shared parent mapped to the production `StableParentDir` (the code derives it), and the aborted +state transposed from dm onto the USB shape (§5.2 below). Tests redirect `procSelfMountinfo` / +`procGuestMountinfo` at fixture files, so the real parser, the real predicate, the real `AttachDrive` and +the real `/disks` handler all run — **the data is injected, the logic is not.** + +### 5.1 Red-proofs — 6, each verified to have landed + +| # | Mutation | Landing verified by | Fails | +|---|---|---|---| +| RP1 | term 3 removed at the **Observe** site | `bindUsable(gp, t.MountPath)` occurrences → **0** | StaleBind + Aborted(×2) | +| RP2 | term 3 removed at the **union** site | `bindUsable(gp, d.MountPath)` → **0** | both UnionPath tests | +| RP3 | **the P1-only fix** — abort check neutered | `RED-PROOF RP3` marker present | Aborted(×2), UnionPath_Aborted, Verdicts(×2), AbortedWins | +| RP4 | `emergency_ro` dropped from the token list | `RED-PROOF RP4` present | only the `emergency_ro` subtests — the "only `shutdown`" fix | +| RP5 | unknown reported as not-live | `RED-PROOF RP5` present | UnknownIsTreatedAsPresent/unreadable | +| RP6 | aborted arm falls through to the re-bind | `RED-PROOF RP6` present | AttachDrive_AbortedFilesystem_DoesNotRebind (`go test` **rc=1**, read from `go test` itself) | + +### 5.2 A HOLLOW TEST WAS CAUGHT — by RP1 failing to fail + +`TestDisks_BindLiveness_AbortedFilesystemReadsAbsent` first used the spike's dm capture verbatim +(`/dev/mapper/r117cel`). **`RoleForStorage` derives `role="system"` for that backing device** — a +system-role row never enters the block that computes `BoundUnderParent`, so the field stayed `false` **by +default**, the assertion passed, and **no mutation could ever fail it.** Pure decoration, and it is exactly +R-116's failure mode (a fixture supplying a shape production never emits). + +Found because **RP1 removed term 3 and the test still passed.** Fixed by transposing the aborted fixture +onto the USB drive shape (an in-place abort on a USB drive is the realistic case anyway; only the super +options and the matching devnos carry the claim) **and** by adding a guard that fails loudly if the row is +not the production shape: + +```go +if di.Role != "user-data" || di.GuestPath == "" { + t.Fatalf("fixture does not reproduce the production row shape: role=%q guest_path=%q — "+ + "the conjunction never runs on such a row, so any assertion below is vacuous", di.Role, di.GuestPath) +} +``` + +**The red-proof did not merely confirm the test; it is the only thing that found the test was empty.** + +--- + +## 6. Build, publish, vouch — three observables, quoted + +**1. Published** — the **anonymous** GET the installer performs, no credentials: + +``` +GET .../api/packages/admin/generic/felhom-agent/0.117.0/felhom-agent +http=200 bytes=14032908 +downloaded sha256: e6768a12eba1c9e0743367f04589845330151ac63d10caad794e8a4c6f36f735 +built sha256: e6768a12eba1c9e0743367f04589845330151ac63d10caad794e8a4c6f36f735 +MATCH — published bytes are the built bytes +$ /tmp/agent-dl-0.117.0 --version → felhom-agent 0.117.0 +``` + +**2. Vouched** — POST through the real operator endpoint, then the manifest **read back** (not the flash): + +``` +POST /configuration/artifacts → HTTP/1.1 303, Location: /configuration?flash=artifacts_set +hub log: [INFO] Artifact manifest set: agent=0.117.0 golden=0.186.0 min_agent="0.113.0" wrapper_sha=true + +READ BACK: agent_version=0.117.0 agent_sha256=e6768a12eba1c9e0… + golden_version=0.186.0 golden_sha256=b760ac6a33e70700… (preserved) + min_agent=0.113.0 wrapper_sha256=104db0a4401f65bb… (preserved) +``` + +R-120's gate did **not** refuse: golden 0.186.0 is not behind the fleet's newest controller (0.186.0). + +**3. Running on the box under test:** + +``` +BEFORE: felhom-agent 0.113.0 active +AFTER: felhom-agent 0.117.0 active +capabilities self-check ok=68 total=68 degraded=0 inactive=0 +reconcile: enrolled drive bound under shared parent (live, no reboot) vmid=9201 … ← loop ticking, new pid +``` + +Zero `ERROR` lines. `configs/` was untouched by this commit, so a binary-only deploy is correct (verified +with `git show --name-only`). + +--- + +## 7. §5.1 — demo-hp brought current, and the drift the R-120 gate does not cover + +demo-hp: **0.113.0 → 0.117.0**, prior binary backed up to `felhom-agent.bak-0.113.0`. + +**Does the R-120 vouch gate cover agent-on-a-box drift? NO — confirmed at source.** +`hub/internal/web/configs.go:1165-1169` compares `goldenVer` against +`store.NewestReportedControllerVersion()`: a **golden-artifact vs fleet-CONTROLLER** check. It says nothing +about the agent installed on a box. **`MinAgent` does not cover it either** — it *holds* the controller +floor for a box whose agent is too old (`hub/internal/api/handler.go:530-538`, `store.go:1857`), which is +protective, not an alarm — and demo-hp's 0.113.0 **equalled** `min_agent` 0.113.0, so even a floor +comparison was satisfied. + +**Filed as R-121 — the fourth instance of the drift family** (R-111, R-115, R-120, and now +installed-vs-vouched). Its measured cost: R-117's whole subject is the R-113 conjunction, which landed in +0.114.0, so **the designated drill host could not exercise the code under investigation at all**, and the +spike had to route every predicate result through an out-of-repo probe built from `main`. + +--- + +## 8. §5.2 — the live run + +**Venue:** demo-hp, the spike's §10 recipe. Scratch LXC **9301** (no NIC) on a scratch dir storage +`r117scratch` — never `local-lvm`, never `felhom-backup`. **Scratch** shared parent `/mnt/r117-drives`, so +the live `/mnt/felhom-drives` peer group (guest 9201's) was not used. `scsi_debug` for the drive, because +it is the only mechanism that removes a gendisk **while mounted**. Guest parent bind verified +`master:209` — a real slave of the host peer group. + +**Method:** a throwaway probe built from the **shipped tree at `966d8f4`** (exported in a copy outside both +repos), so `bindLiveness` and `AttachDrive` are the real functions, judging real kernel state. + +### 8.1 The predicate, every state, on hardware + +| State | host raw | host bind | verdict | `Usable()` | term 3 | cost | +|---|---|---|---|---|---|---| +| HEALTHY (scsi) | `8:16 rw,stripe=512` | `8:16 rw,stripe=512` | `live` | true | true | 442 µs | +| after real gendisk removal | **(NOT MOUNTED)** | `8:16 …,shutdown` | `unknown` | true | true | 340 µs | +| **R-117 RETURN** | `8:32` (`/dev/sdc`) | `8:16 …,shutdown` | **`stale-device`** | **false** | **false** | 365 µs | +| HEALTHY (dm) | `252:11 rw` | `252:11 rw` | `live` | true | true | 359 µs | +| **Q7 IN-PLACE ABORT** | `252:11 …,emergency_ro` | `252:11 …,emergency_ro` | **`filesystem-aborted`** | **false** | **false** | 497 µs | + +**Both dead states are detected on real hardware.** Note the two rows that matter most: + +- the **RETURN** row is the exact defect — `sdb`→`sdc`, the bind left on the dead superblock; +- the **IN-PLACE** row has **identical devnos on both sides** (the device never left) and the raw mount + still `active` — so the device comparison alone reads healthy there, and only the abort token catches + it. This is R-117a, the half that emits nothing today. + +The **absent** row is correct by design and worth stating: term 3 answers `unknown` because `devicePresent` +is the term that reports device absence, so term 3 abstains rather than double-counting. The overall +conjunction is false via term 2. It also demonstrates the cannot-tell rule working in a real state. + +**Cost: 340–497 µs**, against 3.85–5.03 ms that `GuestSeesMount` already spends forking `lxc-info`. + +### 8.2 No block I/O — the Part 1 fence applied to its own first consumer + +`strace -f -e trace=openat,statfs,fstatfs,read,pread64` while the predicate judged the **aborted** drive: + +``` +distinct openat targets: "/etc/localtime" "/proc/self/cgroup" "/proc/self/mountinfo" + "/sys/.../cpu.max" "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size" +openat/statfs against /mnt/r117*: NONE — the predicate never touched the mount +statfs syscalls: 0 +``` + +### 8.3 No regression on the live customer drive — through the REAL server pipeline + +`GET /disks` over the island bridge with the **controller's own credential** (from the guest's +`/etc/felhom-bootstrap/bootstrap.json`), i.e. the exact call the controller makes: + +``` +GET /disks http=200 bytes=2545 vmid=9201 +name=felhom-backup state=attached role=user-data +guest_path=/mnt/felhom-drives/nvme-1tb bound_under_parent=True backup_target=True +``` + +The live enrolled drive — which is also the whole-guest backup target — **reads present under 0.117.0.** +Host-side corroboration: exactly **1** bind entry at the stable path, `259:0` on both raw and bind, no +abort token ⇒ `BindLive`. And a positive observable that the consumer was actually evaluating (rule 3): +**32 gate-related controller log lines in 3 minutes**, with **zero** `ABSENT`/`RETURNED`/`backup_target` +transitions, and zero `bind is STALE` / `normalized to one bind` lines from the agent — so the new +predicate disturbed nothing. + +### 8.4 `AttachDrive` on hardware — and the one thing that was NOT repeated + +The healthy case ran through the real `AttachDrive` on the box: it returned the stable path, and the +**guest init PID was identical before and after (2555744)** — the guest was never restarted. + +**`AttachDrive` derives its target from `StablePathForRaw`, which hardcodes the production +`/mnt/felhom-drives`.** So invoking it for a scratch drive **necessarily creates a submount under the live +customer parent, which propagates into guest 9201** — which this task fences. That happened **once, +inadvertently, during the healthy case**: a bind appeared at `/mnt/felhom-drives/r117sd`. It was removed +within a minute and the revert was verified, not assumed: + +``` +live parent restored to exactly its two real submounts (nvme-1tb, Felhom-Share) +9201: running, init pid unchanged (never restarted) +stale scratch entries in 9201's namespace: 0 +9201 guest view: Felhom-Share nvme-1tb 9201 live drive read: OK +``` + +**It was deliberately NOT repeated for the stale-bind case**, so the *repair* leg is proven by +`TestAttachDrive_StaleBind_Rebinds` + RP6 through the real code, and on hardware only for the healthy +no-op. Doing it live would need ~20 s of a scratch submount inside guest 9201's namespace — harmless on +the evidence above, but the fence names 9201, so it is the operator's call, not one to take quietly. +**Named as the one gap in this run's hardware coverage.** + +--- + +## 9. Teardown — all three layers + +**Layer 1 — the machine.** `pct stop 9301` + `pct destroy 9301 --purge`; `pct list` back to `9201` alone. + +**Layer 2 — the host.** + +| | Before | After | +|---|---|---| +| `pvesm status` | `r117scratch` active, `local` **39.03 %** | `r117scratch` **removed**, `local` **37.07 %** (session start 37.02 %) | +| `df -h /var/lib/vz` | 16G, **42 %** | 15G, **40 %** (session start 40 %) | +| `dmsetup ls` | `r117cel (252:11)` + 11 pre-existing | **gone**; the 11 `pve-*` unchanged | +| `losetup -a` | 2 loops | **none** | +| `scsi_debug` | loaded | **unloaded**; `/dev/sd*` back to `sda1..3` | +| `/mnt` | `+ r117-drives, r117cel, r117sd` | `felhom-drives`, `nvme-1tb`, `vzsnap0` — as found | +| `/root`, `/var/lib/vz/r117*` | 5 scripts + `r117/` + probe | `ls -d /mnt/r117* /root/r117* /var/lib/vz/r117*` → **NONE** | + +The 124 MB template was removed. **One teardown defect, recorded because it is a reusable trap:** the +script hardcoded `losetup -d /dev/loop0`, but the loop numbers **swapped** this run (the guest rootfs took +loop0), so it detached the wrong device and left `/dev/loop1` attached to a deleted file. Fixed by +selecting loops by backing file (`losetup -a | grep r117`), not by number. **`local` is 0.05 % above the +session start** (~20 MB) — journal growth from the run, not a leftover artifact. + +**Layer 3 — the hub: one record was written, deliberately, and it is retained.** No customer, appliance, +pairing or claim was created (9301 had no NIC and ran no controller). The one hub-side change is the +**artifact manifest, now vouching agent 0.117.0** — the intended product of §6, not scratch, so it is +**retained** and is the fleet's vouched agent going forward. + +**Fences — re-verified after teardown, not assumed:** + +| Fence | State | +|---|---| +| guest **9201** | `running` throughout; guest view `Felhom-Share nvme-1tb`; live drive read **OK** | +| **`drill-r50`** (VM 300) | `stopped` — never started | +| **`local-lvm`** | **38.84 %** — byte-identical before, after, and to the session start | +| **`felhom-backup`** | `active`, 0.46 %, `content backup` unchanged — not re-targeted, not written | +| live `/mnt/felhom-drives` | **both** real submounts present; no `r117*` leftovers | +| `felhom-agent` on demo-hp | `active`, **0.117.0** | +| demo-felhom, Peti's cluster | never contacted | +| `felhom-controller` | **untouched**, as scoped | + +--- + +## 10. Findings — filed as `R-n`, none fixed + +| # | Finding | Severity | +|---|---|---| +| **R-121** | **A box's installed agent can sit releases behind the vouched one and nothing notices.** demo-hp ran 0.113.0 against a vouched 0.116.0 through the whole R-116/R-117 arc. The R-120 gate is golden-vs-fleet-controller and cannot see it; `MinAgent` is protective, not an alarm, and 0.113.0 *equalled* the floor. **Fourth instance of the drift family** | **MEDIUM** — filed with a fix shape and the honest tension (a box legitimately lags between publish and deploy) | +| **R-117g** | **An aborted filesystem is never cleared automatically, by design (§3), so a drive stuck in `emergency_ro` alarms forever until a human acts.** The alarm is correct and new; the absence of any guided recovery is the gap. A remount risks loss on a dirty abort, so it wants an operator-confirmed action, not a self-heal | **LOW-MEDIUM (follow-up to this fix)** | +| **R-117h** | **`AttachDrive` cannot be exercised against a non-production shared parent**, because `StablePathForRaw` hardcodes `StableParentDir`. Any hardware test of the repair path writes a submount into the live customer guest's namespace (§8.4). A test seam for the parent dir would make the repair leg testable on hardware without touching a customer | **LOW (testability)** | + +**Observations — noticed, documented, NOT acted on:** + +- **Guest 9201 carries 2 stacked entries** at `/mnt/felhom-drives/nvme-1tb` while the host has exactly 1. + Pre-existing, unrelated to this change, and harmless to the predicate (it reads the host table, and + `bindLiveness` iterates all entries). Worth a look — `AttachDrive` normalizes the *host* count only. +- **14 stale `felhom-agent.bak-*` binaries** on demo-hp (`0.92.1` … `0.113.0`), ~200 MB on `pve-root`. + Nothing prunes them. +- **11 files in the agent repo fail `gofmt -l`**, all pre-existing (`internal/capability/probe.go`, + `internal/storage/intent.go`, …). None are in `internal/localapi/`; none were touched here. + +--- + +## 11. What did not happen, and why + +- **The stale-bind repair was not exercised on hardware** (§8.4) — it would put a scratch submount inside + guest 9201's namespace, which this task fences. Covered by `TestAttachDrive_StaleBind_Rebinds` + RP6 + through the real code, and by the healthy-case `AttachDrive` on hardware (init PID identical). +- **No load or duration measurement of the predicate under a sustained reconcile loop.** The spike flagged + this as unmeasured and **it is still unmeasured** — said plainly rather than let to vanish. What exists: + single-call costs of 340–497 µs on hardware, and ~15 minutes of the real 20 s ticker on demo-hp with no + spurious re-bind and no CPU complaint. A many-drive box under sustained load was not tested. +- **`felhom-controller` was not touched**, per scope — the repair is already wired from its `Return` branch. +- **The R-116 row was not edited beyond §2.2's clause.** +- **felhom-pve was not upgraded** — it runs 0.115.0 and the task scoped the venue to demo-hp. The fleet is + now non-uniform: demo-hp 0.117.0, felhom-pve 0.115.0. Flagged, not acted on. +- **R-118 was not addressed** (the absent drive's union row advertising the root filesystem's capacity) — + a separate row, untouched by this fix. + +--- + +Every claim above is quoted from pushed Gitea source at `file:line` (agent `966d8f4`, hub `0.82.0`) or is +captured run output. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index c62877b..c421081 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -18,7 +18,8 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **WAITING-ON-OPERATOR (M)** | operator ruling on the release process | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design | CC | | **R-116** | ~~**The drive-absent alarm and its recovery were a MISMATCHED PAIR — absent fired the GENERIC `storage_disconnected`, return the SPECIFIC `backup_target_restored`; `backup_target_absent` never fired at all**~~ | **SHIPPED + PROVEN-LIVE** (agent v0.116.0, 2026-07-30) | — | **CLOSED. The full four-event sequence, on the wire, on a fresh box** (`audits/R116-v0116-2026-07-30.md`): `backup_target_absent (error)` on detach → `backup_target_restored (info)` on return for the TARGET, and `storage_disconnected (error)` → `storage_reconnected (info)` for a NON-target drive on the same box four minutes apart. **Two matched pairs, correctly discriminated — and discrimination is proven NON-trivially for the first time**, since both prior runs had the target itself emit the generic event. Gate fired in **3 s**; all four events reached the hub, so the specific alarm, its severity, its Hungarian copy and the hub routing are now exercised end-to-end. **Over-correction PASSES with a positive observable** (0 ABSENT lines / 0 drive events over 2m14s with both drives present, target `degraded:false`, while 2 `RETURNED` lines prove the gate was ticking). **NARROWED by the R-117 spike (`audits/SPIKE-r117-bind-liveness-2026-07-30.md` §12), and it stands as written:** the 2 `RETURNED` lines are a genuine positive observable, so rule 3 is satisfied — but `degraded:false` over that window was read off a drive whose bind was **dead** (R-117), so the window evidences **"the gate did not over-fire"** and **NOT** **"the drive was healthy."** No other part of this row changes: every input to the pairing fix is configuration-derived (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), which R-117 does not touch. Ran on a nested PVE on **demo-hp** per `runbooks/target-selection.md` — through the **real day-0** from the v1.25.0 ISO, with the agent **installed unaided from the vouched Day-0 manifest** (published sha `b47c5c4dab641ee5…`, independent registry GET verified, manifest read back), drives enrolled through the real endpoints, device loss a real hot-detach. **THE FIX, and the ruling is the substantive part:** the mechanism was first isolated from the captured payload (`DIAG-r116-disks-payload-2026-07-30.md`) after two fixes aimed at shapes that do not occur. **Both smaller-looking options were REJECTED because they regress R-114** — `backup_target_offer.go:79` reads `BackupTarget && MountPath != ""` as *"a real drive with its own mountpoint — healthy"* and returns before its `TargetAbsent` branch, so back-filling `MountPath` on the Observe row **or** flagging the registry row (whose `MountPath` is the stale unit-file value) would have told the customer the backup target is fine while its drive was gone. **R-114's correctness was resting on R-116's bug** — a coupling invisible until the payload existed. Taken instead: the Observe row gets the **guest path only** (`mount_path` stays `""`, which is true) from a new `ConfigPath` (`json:"-"`, so the cross-repo golden + key-set contract is untouched), and the union row is deduped **on guest path** — the join being CONFIGURATION (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), the only identity that survives the device. Tests 845→849; 4 red-proofs each asserted to land, and red-proof 1 replays v0.115.0's code and fails, which is the empirical proof it was inert. Its green test had supplied a `MountPath` production never supplies AND left `DriveTargets` nil so the union loop never ran — both corrected. v0.115.0 left in place (inert, harmless). Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded. **Caveat: the drill's controller was 0.185.1 from the golden, which PREDATES R-114**, so its absent-state banner showed the old false copy — the golden being a release behind, not a regression → **R-120** | — | | **R-120** | ~~**The golden baked a controller that predated R-114 + R-112, so a FRESH box showed the customer the WRONG absent-target message**~~ | **CLOSED — golden rebaked + PROVEN-LIVE, and the class now has an ENFORCED gate** (golden 0.186.0 + hub v0.82.0, 2026-07-30) | — | **`audits/R120-golden-rebake-2026-07-30.md`.** **Half 1 — the artifact.** Golden **0.186.0** baked from `main`'s controller in the DooPlex bake fixture (overlay2 OK, **3 mounts**, FATAL 0, exclusions 0, 618 MB, upload **201**, `GOLDEN_SHA256=b760ac6a33e70700…`, token-leak grep 0, GL-1 teardown, `drill.qcow2` back to `virgin`). Three observables: **published** — anonymous GET (what the installer does) 200 / 648930639 bytes / sha identical to the bake; **vouched** — manifest read BACK; **resolved** — `Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)`. Floor **untouched** per publish-train rule 2 (`min_controller_version` still 0.156.0; it is a separate form); MinAgent left 0.113.0 as 0.186.0 declares. **Proven on a REAL day-0, not the fixture** (per the Part-1 rule now in `runbooks/target-selection.md`): VM 9402 on demo-hp from the v1.25.0 ISO → `Controller elindult (0.186.0)`. With the target detached the endpoint returned the **`TargetAbsent`** copy — *„A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod…"* — **and `offer_path` absent entirely**; the day-old read on the 0.185.1 golden had returned the false system-disk message **plus** an offer of the other drive. **Half 2 — the mechanism, operator ruling REFUSE.** hub **v0.82.0**: the gate sits in `hub/internal/web/configs.go` `handleSetArtifacts` immediately before the only write — the sole UI path to `SetArtifactManifest` — so it runs on every vouch without anyone choosing to, and it **refuses** rather than warning. Signal: `store.NewestReportedControllerVersion()` over `reports.controller_version`, **semver-compared in Go** (`MAX()` in SQL ranks 0.99.0 above 0.186.0 — a pair this fleet has shipped). Fail-open in exactly two deliberate cases: empty golden field, unknown fleet version. **NEAR-MISS RECORDED: the first draft read `guests.controller_version`, a column that exists and that NOTHING writes** — it would always have seen `""` and failed open, i.e. inert, this gate's own failure shape, one grep from shipping. 4 tests through the **production handler** over httptest (never a seam), the refusal asserting **both** the flash **and** that the manifest was not written; red-proof: deleting the block makes the stale golden vouchable again. **PROVEN LIVE on the deployed hub by re-attempting the original mistake:** vouching 0.185.1 → `HTTP 303 …flash=golden_behind_fleet` + `[WARN] artifact vouch REFUSED: golden 0.185.1 is older than the newest controller the fleet reports (0.186.0)`, and the manifest read back **unchanged at 0.186.0**. Recorded on **R-29's audit list** (`ROADMAP.md`) as the **first enforced gate** beside its three orphans, so the contrast is kept — the orphans are unchanged. Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded, exactly as `sess-e` was (and `sess-e` was deleted this run) | — | -| **R-117** | **A drive's guest bind becomes a DEAD MOUNT while every signal reads healthy — and it happens in TWO ways, only one of which the original framing covered.** (a) *after a detach/return*: the host raw mount heals onto the NEW device via its fs-UUID-keyed unit while the bind still names the OLD one, so the gate takes its `Return` branch and restarts the customer's apps onto a namespace that `EIO`s on every call; (b) *in STEADY STATE, no cycle at all* — a device that errors without disappearing leaves the raw mount `active`, `BoundUnderParent` `true` and the drive never `Disconnected`, so **the gate produces no action and NOTHING is emitted on any channel** | **MECHANISM + RECIPE + FIX CONSTRAINTS ESTABLISHED (spike complete 2026-07-30) — READY (S/M), not fixed** | — | **SPIKE: `audits/SPIKE-r117-bind-liveness-2026-07-30.md`** — reproduced deliberately **3/3**, two device classes, on a purpose-built scratch LXC (9301) on demo-hp; predicates evaluated by a throwaway probe calling the **real** `localapi` code from agent `d4eb259` (v0.116.0). **MECHANISM, confirmed at source and live:** both halves of the R-113 conjunction are *path-presence tests*. `GuestSeesMount` (`intermediary.go:276`) and `isHostMountpoint` (`intermediary.go:394`) each compare **field 5** (the mount point) of a mountinfo line and **never read field 3, the `major:minor`** — so neither can see that the bind and the raw mount name **different devices**. Measured: `raw 8:32 /dev/sdc` vs `bind 8:16 /dev/sdb …,shutdown`, `BoundUnderParent = TRUE`, `EIO`(5) on read **and** write, host and guest. **The node change is a CONSEQUENCE, not a precondition** — control test: with the stale bind held the drive returns as `sdc` (8:32); once it is `umount`ed the letter is **reused** (`sdb`, 8:16). The stale bind pins the dead superblock, which forces the new number — so a devno comparison is *sound* for case (a), not a heuristic. **RECIPE** (spike §10): `scsi_debug` is load-bearing — it is the only mechanism that removes a gendisk **while mounted** (dm and loop are held open by the stale bind: `dmsetup remove --force` → `Device or resource busy`). `modprobe scsi_debug` → mkfs → UUID-keyed `systemd-mount` → bind under a **scratch** shared parent → `echo 1 > /sys/block/sdb/device/delete` (systemd stops the raw unit itself, `BindsTo=`) → `echo '- - -' > /sys/class/scsi_host/hostN/scan` → re-mount by UUID. Case (b): swap a dm table to `error`. **TWO DEATH STATES, different signals** — device removed ⇒ superopts gain `shutdown`, `EIO` both ways; device errors in place ⇒ `emergency_ro`, write `EROFS`(30), reads still served from cache. **PROBE TRAP (R-117b): `statfs` and `getdents` SUCCEED on a namespace that `EIO`s every byte** — any liveness probe built on either is hollow by construction. **A pure-`/proc` check costs 0.17–0.45 ms**, cannot hang, spins up no disk and writes nothing, against the 3.85–5.03 ms `GuestSeesMount` already spends forking `lxc-info`. **CONSTRAINT: the fix needs BOTH a devno comparison (closes (a); blind to (b)) and the fs abort flags `shutdown`\|`emergency_ro` (closes (b); ext4-specific)** — the tempting devno-only fix leaves the silent half open. Keep the *cannot-tell → never absent* rule (`disks.go:983-986`). **R-117d — THE FIX IS ONE PREDICATE, AND IT SELF-HEALS:** `AttachDrive`'s normalize leg (`intermediary.go:238-247`) already performs exactly the repair — proven live, **guest never restarted** (init PID identical) — and **three** call sites already invoke it: the 20 s reconcile ticker (`main.go:973-985`), agent startup (`disks.go:951`), and **the controller's `Return` branch itself, at `intermediary.go:301-305`, BEFORE it restarts the apps at `:311`**. All three are defeated by `if n == 1 && b.GuestSeesMount(...)` returning early at `intermediary.go:235` and logging *"fully live"*. **Unblock the existing path; do not add a new one.** **R-117c:** three comments (`disks.go:83`, `:154-155`, `:155-165`) promise "live + usable in the guest"; nothing tests it and it is false in the measured state — the fix owes a test asserting the **consequence** (dead namespace ⇒ `BoundUnderParent=false` ⇒ alarm fires), plus a second scenario for case (b). **R-117f — THE HANG CASE, MEASURED, AND IT DISQUALIFIES I/O PROBING OUTRIGHT** (§6.3): against a `dmsetup suspend`ed device **every probe that touches it blocked — including a buffered write with no `fsync`** (the `O_CREAT` metadata path needs journal access) — while **P1 and P2 completed in 364 µs / 206 µs**. The blocked process then survived `SIGTERM` **and `SIGKILL`** (`stat=D`, `wchan=folio_wait_bit_common`, still alive 3m50s after `kill -9`) and died only when the device was resumed. **So `systemctl restart felhom-agent` would hang, leaving the agent unrecoverable until the device returns or the host reboots** — and the thread count does not reveal the leak (5→5, 5→6). A timeout protects the caller's control flow and nothing else. **The fix must issue no block I/O; that is now a fence, not a preference.** `statfs`/`getdents` reported HEALTHY on the wedged device too, confirming R-117b in a second failure mode. **Teardown DONE, all three layers** (spike §14): guest 9301 destroyed, `r117scratch` removed, both dm devices and both loop devices gone, `scsi_debug` unloaded, `local` back to 37.02 % (session start 37.00 %); fences re-verified after — 9201 `running`, `drill-r50` `stopped`, `local-lvm` **38.84 % byte-identical**, `felhom-backup` `content backup` unchanged, live `/mnt/felhom-drives` intact with both submounts, agent `active`. **Layer 3 genuinely empty** — 9301 had no NIC and ran no controller, so no hub-side record was ever created. Note demo-hp's live agent is **0.113.0** — it does not run the R-113 conjunction, so live validation of the fix needs that box current first | CC | +| **R-117** | ~~**A drive's guest bind becomes a DEAD MOUNT while every signal reads healthy — and it happens in TWO ways, only one of which the original framing covered.** (a) *after a detach/return*: the host raw mount heals onto the NEW device via its fs-UUID-keyed unit while the bind still names the OLD one, so the gate takes its `Return` branch and restarts the customer's apps onto a namespace that `EIO`s on every call; (b) *in STEADY STATE, no cycle at all* — a device that errors without disappearing leaves the raw mount `active`, `BoundUnderParent` `true` and the drive never `Disconnected`, so **the gate produces no action and NOTHING is emitted on any channel**~~ | **SHIPPED + PROVEN-LIVE** (agent **v0.117.0**, 2026-07-30) | — | **CLOSED. `audits/R117-v0117-2026-07-30.md`.** `BoundUnderParent` gains a THIRD term at both /disks sites: `bindLiveness` reads `/proc` only and requires (a) **the bind names the same device as the raw mount** and (b) **the filesystem has not aborted** (`shutdown` **or** `emergency_ro`, both measured). **BOTH CHECKS ARE LOAD-BEARING and this is the substantive part:** R-117 was filed as a detach/return defect, but a device that fails WITHOUT disappearing gives the identical all-signals-healthy state with the **devnos EQUAL** and the drive never `Disconnected`, so the gate emits nothing at all, indefinitely (R-117a) — the device comparison alone cannot see it, and a P1-only fix passes every payload test (red-proof RP3 exists for exactly that). **THREE states, never a bool:** `{Unknown, Live, StaleDevice, Aborted}`, `Unknown` is the zero value, and every caller reads `Usable()` where unknown counts **PRESENT** (absent stops a customer's apps — the `newestArchiveOn` trap). **NO NEW RECOVERY PATH:** `AttachDrive`'s normalize leg already did the repair and three call sites already invoked it (20 s ticker, agent startup, and **the controller's `Return` branch BEFORE `restartStacks`**); all three were defeated by `if n == 1 && GuestSeesMount(...)` logging *"fully live, no-op"* about an EIO namespace. **RULING (asked for, given, flagged for overrule):** `StaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, guest never restarts — init PID identical); `Aborted` ⇒ **quiet no-op and SURFACE**, because a re-bind lands on the SAME dead superblock and this runs every 20 s = an infinite silent retry that masks the state. No operator decision required: it routes an already-broken state into the **existing** gate, event types and Hungarian copy — no new customer-facing concept — and the alternative is apps writing documents into a filesystem that rejects every write. **ORDERING TRAP caught by a test:** abort-first classifies the real return state as aborted (its stale bind carries `shutdown` too) and refuses the repair **while still reporting correctly**, so the abort flag is read off the RAW mount in the stale case. **LIVE on demo-hp** (brought 0.113.0 → 0.117.0 first — see R-121): RETURN `raw 8:32 / bind 8:16 shutdown` ⇒ `stale-device`, usable **false**; IN-PLACE `both 252:11 emergency_ro`, raw unit still `active` ⇒ `filesystem-aborted`, usable **false**; healthy ⇒ `live`; **340–497 µs**. **No block I/O proven by strace** (only `/proc/self/mountinfo`, **0** statfs) — the Part 1 `CLAUDE.md` fence applied to its own first consumer. **No regression through the REAL pipeline:** `GET /disks` with the controller's own credential shows the live backup-target drive `bound_under_parent=True`, with 32 gate lines in 3 min as the positive observable and zero spurious transitions. Tests **849→863**, 29/29 green, **6 red-proofs each verified to land** — and **RP1 failing to fail exposed a HOLLOW test**: the aborted fixture used a `/dev/mapper` device, for which `RoleForStorage` derives `role=system`, and a system row never runs the conjunction, so it reported false by DEFAULT and no mutation could fail it. Fixtures now assert the production row shape first. Teardown all 3 layers; hub layer = the vouched manifest, **retained** (it is the product, not scratch). **NOT covered:** the stale-bind repair on hardware — `StablePathForRaw` hardcodes the live parent, so it would write into guest 9201's namespace (R-117h); and sustained-load behaviour, still unmeasured. Follow-ups **R-117g** (no guided recovery for an aborted fs), **R-117h** (parent dir not test-seamable), **R-121** | CC | +| **R-121** | **A BOX's installed agent can sit releases behind the vouched one and nothing notices — the R-120 gate does not cover it.** demo-hp ran agent **0.113.0** while the hub vouched **0.116.0**, through the whole R-116/R-117 arc, and no signal existed on any channel | **READY (S) — NEW 2026-07-30** | — | **Fourth instance of the drift family** (R-111 golden's agent 17 releases behind, R-115 built+deployed but never published, R-120 golden a controller behind — and now installed-vs-vouched on a live box). **Confirmed at source that R-120's gate cannot catch it:** `hub/internal/web/configs.go:1165-1169` compares `goldenVer` against `store.NewestReportedControllerVersion()` — it is a **golden-artifact vs fleet-CONTROLLER** check and says nothing about the agent installed on a box. **`MinAgent` does not cover it either:** it is used to HOLD the controller floor for a box whose agent is too old (`hub/internal/api/handler.go:530-538`, `store.go:1857`) — protective, not an alarm — and demo-hp's 0.113.0 **equalled** `min_agent` 0.113.0, so even a floor comparison was satisfied. **The cost, measured:** R-117's whole subject is the R-113 conjunction, which landed in **0.114.0** — so the designated drill host could not exercise the code under investigation at all, and the R-117 spike had to route every predicate result through an out-of-repo probe built from `main` instead of the installed agent (`audits/SPIKE-r117-bind-liveness-2026-07-30.md` §1, §2.3). Discovered because the R-117 task made bringing the box current an explicit prerequisite. **Fix shape (not implemented):** the hub already receives `AgentVersion` on every host report, and already has semver comparison in Go — the missing piece is a checker comparing reported agent vs the **vouched** agent and surfacing it, operator-tier. Note the honest tension: a box legitimately lags between publish and deploy, so this wants a staleness window rather than an instant alarm | CC | | **R-118** | **An absent drive's union row advertises the ROOT filesystem's capacity as its own.** In the absent-state payload the registry-union row reports `total_bytes: 49675956224 / used_bytes: 4584579072` — **byte-identical to the `local` row** (`durable_id: path:/var/lib/vz`, i.e. `pve-root`) in the same response. The real drive is **4 GB** | **READY (XS) — NEW 2026-07-30** | — | Cause: `statfsCapacity(d.MountPath)` (`disks.go:335-338`) statfs's `/mnt/cel`, which with the device gone is a **bare directory on the root filesystem**. `observe.go:176-183`'s comment warns about exactly this trap and guards the Observe path ("*an unmounted removable dir-storage's mountpoint reverts to a bare directory on root … catastrophic DR mis-id*"); **the union path has no equivalent guard.** **Not a DR mis-id** — `durable_id` on that row is still the correct `uuid:…`, so re-attach identity is safe. It is a **false capacity** reaching every consumer of `total_bytes`/`used_fraction` (fill monitors, storage cards): a detached 4 GB drive advertises 46 GiB at 9.2 % used. Same class as `role.go:180-181` — an absent drive's fields decaying to the root filesystem's. Evidence: `audits/DIAG-r116-disks-payload-2026-07-30.md` §12 | CC | | **R-113** | ~~**The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.**~~ `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc//mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **SHIPPED + PROVEN-LIVE** (agent v0.114.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** Target drive hot-detached on a fresh box running the SHIPPED agent 0.114.0 (from the Day-0 manifest, not a hand build). **The gate fired in 4 seconds** — E-2d measured ZERO over 4½ minutes — and `SetDisconnected` was reached. It fired on exactly the shape that defeated it: raw `/mnt/mentes` NOT mounted while the bind `/mnt/felhom-drives/mentes` still read `/dev/sdb[/felhom-data]`. **Over-correction guard PASSED:** with both drives present, 0 ABSENT lines and the target stayed healthy. **Note: C5 still FAILED — but on a different defect (→ R-116), not on this one.** **SESSION C IS NOW UNBLOCKED.** Agent **0.114.0 published and vouched** 2026-07-29 (sha `5e4c15ebee2d7583…`, round-trip GET verified; hub manifest read back showing it) — a fresh drill box now installs the fix under test instead of the bug. The golden was NOT re-baked and `min_agent` was NOT raised: the golden bakes the controller, not the agent, and controller v0.185.0 declares MinAgent 0.113.0, which 0.114.0 already satisfies. **FIX: `BoundUnderParent` is now a CONJUNCTION — bound under the parent AND the drive's raw host mount still mounted** (`devicePresent`, new `deviceCheck` seam), at BOTH `/disks` construction sites. The raw mount is the device-bound systemd unit that dies with the device; the agent's bind is not — so the raw mount IS the device signal, grounded in E-2d's measurement rather than inference. **Conjunction, deliberately:** the device half alone would regress boot ordering (raw mounts early, bind lands ~18 s later — that window must keep reading absent), so existing behaviour is byte-identical and only the unreachable case is closed. **Unknown is never absent** (`devicePresent("")` = true) — a false absent stops a working customer's apps. **Controller UNCHANGED, no MinAgent bump:** `BoundUnderParent` has exactly one functional consumer (`planDriveGates:226`); a new `DevicePresent` bool was rejected because absent-from-JSON decodes to `false`, which would have made every drive on an older agent read ABSENT. +6 tests (208→214), 4 red-proofs run and reverted. **Deployed to demo-felhom; over-correction guard verified in production** — raw mount present, drive still reads present, 10/10 apps untouched, no gate action, no false alarm. **demo-hp deliberately left on 0.113.0** (the spec scoped deploy to felhom-pve) — it still carries the bug. **⚠️ SESSION C BLOCKER: the hub Day-0 manifest vouches agent 0.113.0**, so a fresh drill box would install WITHOUT this fix and validate nothing — publish + vouch 0.114.0 first (R-111's trap, same shape). **Awaiting live proof of exactly one leg:** device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. Original finding: **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 **Session C scope UNCHANGED by Session B** — R-114/R-112 shipped as controller v0.186.0 and neither touches the agent; the leg awaiting proof is still device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. One rebuild now validates all three. | CC | | **R-112** | **E-2's degraded banner and offer have NO UI CONSUMER — the endpoint is correct and the customer never sees it.** `GET /api/storage/backup-target` returns byte-exact copy (proven live), and **nothing fetches it**: `grep 'backup-target'` across all `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` + `degradedMessageFor` are consumed **only** by the JSON handler — **no page handler injects the state** | **SHIPPED + PROVEN-LIVE** (controller v0.186.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md` — the banner reached a customer's page for the first time.** Never-configured box: banner element 1, never-configured copy 1. After the wizard: offer block 1 with `data-path="/mnt/felhom-drives/mentes"`. Healthy after assign: all four markers 0, **proven POSITIVELY** — idle delta 0 `/backup/tiers` calls, page-load delta +1, single caller ⇒ the seam ran and chose silence. **FIXED: the state now has a consumer.** Server-rendered on `/backups` via `backupsHandler` → `backupTargetView` → `backups.html`, following the existing `SingleCopyWarning` banner pattern — NOT a 19th JS fetch, because a banner that needs JavaScript to appear is one more thing that can silently not happen. `backupTargetView` returns **nil** for healthy AND unknown, so those render nothing at all. **Scenario-E seam test drives `backupsHandler` over httptest and asserts the RENDERED HTML** — deleting the one line that sets `data["BackupTarget"]` reproduces the old state and fails every render assertion. **SEAM PROVEN LIVE on demo-felhom by a DIFFERENTIAL positive observable, not by an absent banner:** idle 8 s → 0 new `/backup/tiers` agent calls; each `/backups` load → exactly +1, and that call has only one caller (`resolveBackupTargetState`). The box is healthy (`degraded:false, target:felhom-backup`) and the page correctly rendered **nothing** — which matches its real state but, being a negative, is NOT by itself proof of wiring. **Still unproven live:** that a customer sees actual copy — impossible on a healthy box. **The decisive contrast: templates fetch 18 distinct `/api/storage/*` endpoints; `backup-target` and `backup-target/assign` are the only two with zero references.** The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"* — an invariant comment asserting a consumer that does not exist (7th instance of that class). v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* — it fixed the **router mount** and stopped one layer short of the **render**; its test `TestBackupTargetRoutesLiveUnderTheStorageAPIMount` pins dispatch, not reachability, which is exactly what `CLAUDE.md`'s seam rule warns about. **Fifth instance of seam-built-but-never-wired.** **Fix R-114 FIRST** — wiring this alone would start showing customers the wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | CC |