diff --git a/REPORT-r117-spike.md b/REPORT-r117-spike.md new file mode 100644 index 0000000..ad2f9dd --- /dev/null +++ b/REPORT-r117-spike.md @@ -0,0 +1,74 @@ +# REPORT — SPIKE R-117: a dead bind that reports healthy (2026-07-30) + +Written as `REPORT-.md`, not `REPORT.md`, per this repo's parallel-session rule and the +established local pattern (`REPORT-r116-diag.md`, `REPORT-session-c.md`, …). The shared `REPORT.md` +was not touched. + +**Class: Spike.** Deliverable is a findings document. **No production code was written; no `.go` file +in either repo was modified; nothing was built for deployment, published, or version-bumped.** + +## Deliverables + +| File | Action | +|---|---| +| `documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md` | **new** — the findings doc (Q1–Q7, evidence, recipe, probe comparison, recommendation) | +| `documentation/backlog/OPEN-ITEMS.md` | **R-117 row rewritten** with the mechanism, the reproduction recipe and the fix constraint — the one register edit, per the brief | + +**No CHANGELOG entry.** This repo's changelogs are per-area (`hub/`, `scripts/`, `website/`); a +documentation-only change belongs to none of them. Stated rather than silently skipped, per standing +rule 4. + +## Baselines + +agent `main` **v0.116.0** @ `d4eb259` · controller `main` **v0.186.0** @ `b331f18` · +`felhom.eu` HEAD `29bcfeb` · hub **live 0.82.0** · golden **0.186.0** · +demo-hp PVE 9.2.2 / kernel 7.0.2-6-pve, **live agent 0.113.0** (= manifest `MinAgent`; never used as +the source of a behavioural claim — every predicate result came from a probe built from `main`). + +## Results + +All seven questions answered empirically except the hang case (below). + +- **Q1 — reproduced 3/3**, two device classes, on a purpose-built scratch LXC (9301) on demo-hp. + **The device-node change is a consequence of the defect, not a precondition** — control test: with + the stale bind held the drive returns as `sdc` (8:32); released, the letter is reused (`sdb`, 8:16). +- **Q2 — two death states**: device removed ⇒ superopts gain `shutdown`, `EIO`(5) on read and write, + host and guest; device errors in place ⇒ `emergency_ro`, write `EROFS`(30), reads served from cache. + The raw host mount is genuinely healthy in both. **No cross-device mis-identification is possible** + on this path — the unit is fs-UUID-keyed. +- **Q3 — confirmed at source and live.** Both halves of the R-113 conjunction compare **field 5** of a + mountinfo line and **never read field 3 (`major:minor`)**, so neither can see that the bind and the + raw mount name different devices. Measured `BoundUnderParent = TRUE` over an `EIO` namespace. +- **Q4 — a pure-`/proc` check costs 0.17–0.45 ms**, cannot hang, spins up no disk, writes nothing. + **`statfs` and `getdents` both SUCCEED on a dead namespace** — probes built on either are hollow. +- **Q5 — the agent**, and not on balance: the controller runs inside the guest and cannot see the host + mount tables the check needs. +- **Q6 — recovery works in place, guest never restarted** (init PID identical). **The repair code + already exists and three call sites already invoke it**, including the controller's `Return` branch + *before* it restarts apps — all defeated by one early return. +- **Q7 — a bind can die in steady state, no cycle at all.** The gate produces no action and **nothing + is emitted on any channel.** A `Return`-branch fix cannot reach this half. + +## Not established + +**The hang case (findings §6.3).** The venue (`dmsetup suspend --noflush`) and the two blocking probes +(write+`fsync`, cache-dropped read, both with timeout-fires and thread-leak reporting) were built; the +run did not happen — the venue lost internet mid-transfer of the rebuilt binary. **The associated +thread-leak hypothesis is explicitly not claimed as a result.** It does not gate the `/proc` +recommendation but does gate adding any I/O confirmation probe. + +## Teardown — OWED + +**Not performed** — the second site outage began first. Findings §14 lists the exact commands. Nothing +fenced was touched at any point (9201 `running`, `drill-r50` `stopped`, `local-lvm` untouched, +`felhom-backup` not re-targeted, `/mnt/felhom-drives` untouched, ep0 read-only). **Layer 3 is +genuinely empty** — guest 9301 had no network interface and ran no controller, so no hub-side record +was created. + +## Findings filed, none fixed + +R-117 (mechanism + recipe), **R-117a** steady-state death with no event (HIGH, larger than R-117 as +filed), **R-117b** `statfs`/`getdents` are hollow liveness probes, **R-117c** three untested comments +promising "live + usable in the guest", **R-117d** the self-heal that already exists is short-circuited +(HIGH), **R-117e** both demo boxes share one failure domain — no route survives the site losing +internet, including the WireGuard OOB path. diff --git a/documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md b/documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md new file mode 100644 index 0000000..35f07c4 --- /dev/null +++ b/documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md @@ -0,0 +1,754 @@ +# SPIKE R-117 — a dead bind that reports healthy + +**Class: Spike. No production code was written; no `.go` file in either repo was modified.** +Output is this document. Findings are filed as `R-n`, none fixed. + +--- + +## 0. Verdict, up front + +| # | Question | Answer | +|---|---|---| +| Q1 | Reproduce deliberately | **YES — 3/3 attempts, two independent device classes.** A device-node change is **NOT a precondition — it is a CONSEQUENCE** of the stale bind, proven by a control test | +| Q2 | What is dead, at which layer | The **guest bind and the host bind** (same mount, one peer group); the **raw host mount is healthy on the new device**. **Two distinct death states** with different errnos — `EIO`(5) both directions, and `EROFS`(30) on write only | +| Q3 | Why `boundUnderParent` returns true | It is a **string comparison of field 5 of a mountinfo line**. It never touches the filesystem, and **neither half reads field 3 (the device number)** — so neither can see that the two entries name different devices | +| Q4 | Cost + failure modes of a real check | A **pure-`/proc` check costs 0.19–0.45 ms, cannot hang, cannot spin up a disk and writes nothing.** The two cheapest *I/O* probes — `statfs` and `getdents` — **succeed over a fully dead filesystem** and are worthless. The hang case was **NOT measured** (§6.3) | +| Q5 | Where the check belongs | **The agent**, and not on balance — the controller **cannot** perform it: the inputs are host mount tables it has no access to | +| Q6 | In-place recovery possible | **YES — proven, guest never restarted** (init PID identical across the repair). And **the repair code already exists, is already invoked at exactly the right moment, and is short-circuited by the same false predicate** | +| Q7 | Scope — steady state too? | **YES, and this is the worse half.** A drive can die **with no detach/return cycle at all**; the gate then never runs, the `Return` branch never fires, and **no event of any kind is emitted** | + +**The single most consequential finding is Q7.** R-117 was filed as a defect in the reattach path. It is +not confined to it: a device that fails **in place** produces the identical all-signals-healthy state +with no gate transition to hang a fix on. A fix that lives in the `Return` branch closes the half that +at least emits events, and leaves the silent half open. + +**The second is Q6.** `AttachDrive`'s normalize leg (`intermediary.go:238-247`) already performs exactly +the repair this defect needs — `umount` every layer, lay down one fresh bind — the controller's `Return` +branch already asks the agent to do it (`controller/internal/web/intermediary.go:301-305`) **before** it +restarts the apps, and the agent's 20-second reconcile ticker (`cmd/felhom-agent/main.go:973-985`) calls +it unconditionally. All three are defeated by one line: `if n == 1 && b.GuestSeesMount(...)` returns +early (`intermediary.go:235`). **Making the predicate honest turns three already-wired call sites into a +self-heal.** That is the cheapest fix available and it is not the one a fresh design would have reached. + +--- + +## 1. Confirmed baselines + +Established fresh this run, not carried. + +| Component | Version | Source | +|---|---|---| +| `felhom-agent` `main` | **v0.116.0** @ `d4eb259da241c4ae2af81d5dff6a51b86c411b42` | `git -C felhom-agent log`, `CHANGELOG.md:1` | +| `felhom-controller` `main` | **v0.186.0** @ `b331f18` | `git log`, CHANGELOG head | +| `felhom.eu` `main` HEAD | `29bcfeb` *(docs(R-120): CLOSED on both halves)* | `git log` | +| hub, **live** | **0.82.0** (`gitea.dooplex.hu/admin/felhom-hub:0.82.0`) | `kubectl -n felhom-system get deploy` | +| golden | **0.186.0** | `OPEN-ITEMS.md` R-120 row (rebaked + vouched 2026-07-30) | +| demo-hp PVE | `pve-manager/9.2.2/b9984c6d90a4bd80`, kernel `7.0.2-6-pve` | `pveversion` | +| **demo-hp agent, live** | **0.113.0** | `felhom-agent --version` on the box | + +> **The live agent on the venue is 0.113.0 — three releases behind `main` (0.116.0) and behind R-113 +> (0.114.0), the very conjunction under examination.** It equals the manifest's `MinAgent`, so it is the +> fleet's declared minimum, not drift. **It was therefore never used as the source of a behavioural +> claim.** Every predicate result in this document comes from a probe built from `main` +> @ `d4eb259` (§2.3), not from the installed agent — precisely the "do not silently reason about a +> different codebase" rule in the brief. + +--- + +## 2. Venue + +### 2.1 The named venue does not exist — recorded, not worked around + +The brief specifies *"the standing snapshot-reset fixture on demo-hp, reset to `virgin` first"*. +**There is no such fixture on demo-hp.** Enumerated on the box: + +``` +pct list -> 9201 running demo-hp (FENCED by the brief) +qm list -> 300 stopped drill-r50 (FENCED by the brief; the only drift fixture, R-93) +``` + +The only standing snapshot-reset fixture in the estate is the **nested-PVE golden-bake VM on DooPlex** +(`drill/drill.qcow2`, snapshot `virgin`) — a **Tier 2** machine, and +`runbooks/target-selection.md:56-61` states that using it as a drill victim "is not covered by that +exception and contradicts the 2026-07-25 ruling". So the two candidate venues were, respectively, +absent and prohibited. + +**Resolution: the venue named by the brief (demo-hp) was kept and the *fixture* premise corrected.** A +purpose-built **scratch LXC guest (VMID 9301)** was created on demo-hp — Tier 1 on a Tier 0 host, which +`target-selection.md:83-87` permits as "create and destroy freely". This is a **mechanism** claim, so +per `target-selection.md:89-104` a fixture is the correct instrument and a day-0 would have been waste. + +### 2.2 Fixture provenance — recorded per `target-selection.md:101` + +Built **2026-07-30 11:18–11:24 CEST**, from scratch, this session. Nothing about it is of unknown +vintage: + +| | | +|---|---| +| Guest | LXC **9301**, unprivileged, 256 MB / 1 core, **no network interface at all** | +| Template | `debian-13-standard_13.6-1_amd64.tar.zst`, downloaded this run, **checksum verified by `pveam`** | +| rootfs | **2 GB raw on a scratch dir storage `r117scratch`** at `/var/lib/vz/r117-scratch` — created for this run. **Never `local-lvm`** (fenced: over-subscribed thin pool backing live 9201) and **never `felhom-backup`** (the box's backup target) | +| Host | demo-hp, PVE 9.2.2, kernel 7.0.2-6-pve | +| Agent / controller **inside** it | **none** — see §2.4 | + +### 2.3 What actually evaluated the predicates + +A throwaway probe, `r117probe`, **outside both repos**, which calls the **real exported agent code**: + +- The agent tree was exported at `HEAD` (`git archive d4eb259`) into the session scratchpad and the + probe added there as `cmd/r117probe`. **`git status` against that worktree, excluding the probe dir, + is empty** — the copy is byte-identical to `main`. Neither repo was touched. (Direct cross-module + import is impossible: `internal/...` is unimportable from another module.) +- `HALF1` is a **real call** to `localapi.GuestBinder.GuestSeesMount` (`intermediary.go:276`) through a + real `proxmox.ExecRunner`, so the real `lxc-info` lookup and the real mountinfo parse both run. +- `HALF2` is a **verbatim copy** of `isHostMountpoint` (`intermediary.go:394`) — the implementation + behind `devicePresent` (`disks.go:972`), which is unexported. The copy is quoted against the original + in §5.2; it is 12 lines with no branching beyond the field-count guard. + +### 2.4 Fidelity gaps — stated, not glossed + +| Gap | Consequence | +|---|---| +| **No agent daemon and no controller ran in the venue.** | No fresh `/disks` payload and no fresh controller gate log were captured this run. **Not a hole in the evidence:** `/disks` in all three states was captured verbatim **twice on real hardware** (`DIAG-r116-disks-payload-2026-07-30.md` §11; `R116-v0116-2026-07-30.md` §5), and the gate log four times. What was missing from that record was the *layer beneath* it — the mount tables and the errnos — which is what this run supplies | +| The dm-based drive (§3.2 drive A) could not have its device node **removed** while mounted | Only affects drive A. Drive B used `scsi_debug`, whose gendisk **can** be deleted while mounted — real USB-pull semantics — and it is drive B that carries the headline reproduction | +| One step on drive A was **modelled**: `systemctl stop` of the raw mount unit stood in for systemd reacting to device loss | **Superseded by measurement.** On drive B the real device deletion made systemd stop the raw mount unit **on its own** (`mnt-r117sd.mount` → `inactive`, §3.3), confirming the `BindsTo=` mechanism empirically. The modelled step is not load-bearing for any conclusion | + +### 2.5 Two internet outages at the venue site + +The venue is at the operator's vacation location until ~08-02; both demo boxes share one uplink. + +- **Outage 1, ~11:07–11:13 CEST.** Both boxes went dark together. Four routes tried before concluding + (standing rule 2): DooPlex→tailnet (`100.76.96.79`, `100.70.170.35`) timeout, with + `tailscale status` reporting *both* peers `offline, last seen 7m ago`; DooPlex→LAN + (`192.168.0.87`, `192.168.0.162`) down; `demo-hp-lan` via `ProxyJump felhom-pve` timeout; + **ep0→WireGuard** (`ssh root@167.233.158.164` → `10.77.0.3`, `10.77.0.2`) ping down and ports 22 and + **8822** both closed, with ep0's own `wg show` putting the last handshake at ~5 minutes prior. + Cause confirmed by the operator: house internet. **demo-hp's `uptime` afterwards read `up 6 days`** — + the box never rebooted, so the 124 MB template download running at the time did not destabilise it. +- **Outage 2, from ~11:27 CEST.** Interrupted the run **after** all of Q1/Q2/Q3/Q6/Q7 and most of Q4 + were captured, and **before** the Q4 hang measurement (§6.3) and teardown (§10). + +--- + +## 3. Q1 — reproducing it deliberately + +**Reproduced 3/3 attempts.** Two device classes were used deliberately, because they fail differently +and the difference turned out to matter. + +### 3.1 Topology, faithful to the intermediary model + +Built by hand to match `intermediary.go:11-19` and `:26-42`: + +- scratch **shared parent** `/mnt/r117-drives` — self-bind, `--make-private`, `--make-shared`, the + verbatim logic of `sharedParentScript`. **Deliberately NOT the live `/mnt/felhom-drives`**, whose peer + group belongs to guest 9201. +- the guest's **permanent parent bind**, set before first start: + `pct set 9301 -mp0 /mnt/r117-drives,mp=/mnt/r117-drives`. +- per drive: raw host mount at `/mnt/`, namespace `/felhom-data` chowned `100000:100000`, + bound to `/mnt/r117-drives/`. +- raw mounts keyed by **fs-UUID**, matching `internal/storage/mountunit.go:51` + (`What=/dev/disk/by-uuid/`) — the detail that makes the host side heal onto a new device node. + +Propagation verified correct before any test — the guest's parent bind is a **slave** of the host's +shared peer group: + +``` +guest: 1176 1175 252:1 /mnt/r117-drives /mnt/r117-drives rw,relatime shared:428 master:196 - ext4 ... + ^^^^^^^^^^ +``` + +### 3.2 The two drives + +| | Drive A | Drive B | +|---|---|---| +| Stack | file → `loop0` → **dm-linear** | **`scsi_debug`** (`dev_size_mb=64`) | +| Appears as | `/dev/mapper/r117cel`, **252:11** | **`/dev/sdb`, 8:16** | +| Death mechanism | swap the dm table to an **`error`** target | `echo 1 > /sys/block/sdb/device/delete` — **real gendisk removal while mounted** | +| Models | a device that **errors in place** | a device that is **pulled** | + +### 3.3 The cycle, on drive B — the headline reproduction + +``` +HEALTHY host raw /mnt/r117sd -> /dev/sdb 8:16 rw,stripe=512 + host bind /mnt/r117-drives/sd -> /dev/sdb 8:16 rw,stripe=512 + guest bind /mnt/r117-drives/sd -> /dev/sdb 8:16 rw,stripe=512 master:450 + +PULL echo 1 > /sys/block/sdb/device/delete + /dev/sdb -> No such file or directory + mnt-r117sd.mount -> inactive <-- systemd stopped it ITSELF (BindsTo=) + +ABSENT host raw /mnt/r117sd -> NOT A MOUNTPOINT + host bind /mnt/r117-drives/sd -> /dev/sdb 8:16 rw,relatime,stripe=512,shutdown + ==> BoundUnderParent = FALSE (HALF1 true, HALF2 false) <-- R-113 works here + +RETURN echo '- - -' > /sys/class/scsi_host/host1/scan + returned as /dev/sdc devno=8:32 + by-uuid/71e155fc-... -> /dev/sdc <-- the UUID-keyed unit heals onto the new node + +R-117 host raw /mnt/r117sd -> /dev/sdc 8:32 rw,stripe=512 [HEALTHY] + host bind /mnt/r117-drives/sd -> /dev/sdb 8:16 ...,shutdown [DEAD] + guest bind /mnt/r117-drives/sd -> /dev/sdb 8:16 ...,shutdown [DEAD] + ==> BoundUnderParent = TRUE +``` + +This is **byte-for-byte the state reported three times as cosmetic** — a drive back on `sdc` while +`findmnt` still reads `sdb[/felhom-data]`, with `shutdown` in the options. + +### 3.4 The node change is a CONSEQUENCE, not a precondition — control test + +The brief asks whether the defect *requires* `sdb`→`sdc`. It does not require it; **it causes it.** + +``` +with the stale bind still held: + 8:16 in /proc/partitions -> 0 (the gendisk is gone) + 8:16 in /proc/self/mountinfo -> 1 (the stale bind still references its superblock) + device returns as -> /dev/sdc 8:32 <-- NEW devno + +after releasing the stale bind (umount), same pull + rescan: + device returns as -> /dev/sdb 8:16 <-- letter REUSED +``` + +The stale bind pins the dead superblock, which keeps the old device index allocated, which **forces the +returning device onto a different number.** The same effect appeared independently on drive A: +`dmsetup remove --force r117cel` failed with **`Device or resource busy`** — held open by the stale bind +alone. + +> **Consequence for any fix: a device-number comparison is not a heuristic for the return case, it is +> sound.** Whenever a stale bind exists, the numbers differ *because* it exists. Stated as measured +> here, twice, on two device classes — not as a kernel guarantee quoted from documentation. + +**Reliability: 3/3.** Two full detach/return cycles (drives A and B) and one in-place failure (§9), each +producing `BoundUnderParent = true` over a non-functioning namespace on the first attempt. No flakiness +observed; nothing was retried. + +--- + +## 4. Q2 — what exactly is dead, and at which layer + +**Dead: the bind.** The host bind and the guest bind are the same mount in one peer group, so they die +together and identically. **Alive and correct: the raw host mount**, re-mounted by fs-UUID on the new +device. The two disagree, and nothing compares them. + +**There are two death states, and they present differently.** This matters more than it looks: a probe +tuned to one misses the other. + +| | **(a) device removed** (drive B / real pull) | **(b) device errors in place** (drive A) | +|---|---|---| +| super options gain | **`shutdown`** | **`emergency_ro`** | +| guest `statfs` | **OK** | **OK** | +| guest `listdir` | **OK** | **OK** | +| guest read file content | **`errno=5 EIO`** | OK *(page cache)* | +| guest write | **`errno=5 EIO`** | **`errno=30 EROFS`** | +| host read via bind | **`errno=5 EIO`** | OK *(page cache)* | +| host write via bind | **`errno=5 EIO`** | **`errno=30 EROFS`** | +| host raw mount | not a mountpoint → then healthy on the **new** node | **stays mounted throughout** | +| `BoundUnderParent` | `false` while absent → **`true`** after return | **`true` the entire time** | + +Errnos are numeric, from `os.statvfs` / `os.listdir` / `open` inside the guest via `pct exec python3` +and on the host, in the same capture. + +Two consequences worth stating plainly: + +1. **`statfs` and `listdir` succeed on a filesystem that returns `EIO` for every byte of content.** They + are answered from the in-memory superblock and the dentry cache. Any liveness probe built on either + is **hollow by construction** — it reports healthy in exactly the state it exists to detect. +2. **The host-side raw mount is genuinely fine.** The device is present, readable and writable at + `/mnt/`. Nothing is wrong with the drive. The damage is confined to the namespace the customer's + apps actually use. + +### 4.1 The brief's "materially worse" case — answered: no + +The brief asks whether the host mount points at a device node *now belonging to a different device*. **It +does not.** In every state captured, the raw mount resolved through `/dev/disk/by-uuid/` to the +correct returning device (`by-uuid/71e155fc-... -> /dev/sdc`), because the unit is keyed by filesystem +UUID (`mountunit.go:51`) and not by node. **The stale reference is held by the *bind*, which names the +old device — and it points at a dead superblock, not at a live foreign one.** No cross-device +mis-identification was observed or is structurally possible on this path. + +--- + +## 5. Q3 — why `boundUnderParent` returns true + +**Because neither half of the conjunction looks at which device the mount is on.** Both are presence +tests on a *path string*. + +### 5.1 The conjunction + +`disks.go:222` (Observe path) and `disks.go:358` (registry path), identically: + +```go +di.BoundUnderParent = s.boundUnderParent(r.Context(), vmid, gp) && + s.devicePresent(t.MountPath) +``` + +### 5.2 Half 1 — `GuestSeesMount` + +`disks.go:949` → `boundUnderParent` → `s.guestAttach.GuestSeesMount(ctx, vmid, stablePath)` → +`intermediary.go:276`: + +```go +data, err := os.ReadFile("/proc/" + pid + "/mountinfo") +... +for _, line := range strings.Split(string(data), "\n") { + f := strings.Fields(line) + if len(f) >= 5 && f[4] == path { + return true + } +} +``` + +Half 2 — `devicePresent` (`disks.go:972`) → `isHostMountpoint` (`intermediary.go:394`) — is the same +test against `/proc/self/mountinfo` for the **raw** path: + +```go +fields := strings.Fields(sc.Text()) +if len(fields) >= 5 && fields[4] == path { + return true +} +``` + +**`f[4]` is the mount point. `f[2]` — the `major:minor` of the backing device — is never read by either +function.** The information that settles the question is present in the very lines both functions +already parse, one field away, and is discarded. + +Measured in the R-117 state, drive B: + +``` +HALF1 GuestSeesMount(9301, /mnt/r117-drives/sd) = true [4.643178ms] <- REAL agent code +HALF2 isHostMountpoint(/mnt/r117sd) = true [261.419µs] +=====> BoundUnderParent (disks.go:222 conjunction) = true + +host raw : 814 33 8:32 / /mnt/r117sd ... - ext4 /dev/sdc rw,stripe=512 +host bind : 755 118 8:16 /felhom-data /mnt/r117-drives/sd ... - ext4 /dev/sdb rw,stripe=512,shutdown +guest bind : 759 1176 8:16 /felhom-data /mnt/r117-drives/sd ... - ext4 /dev/sdb rw,stripe=512,shutdown + ^^^^ +``` + +Half 1 is satisfied by the **stale** entry; half 2 by the **new** device. Exactly as `OPEN-ITEMS.md` +predicted: neither half compares the two. **Confirmed, not inferred.** + +### 5.3 This is a gap, not a false invariant — and the doc comment is the reason it survived + +`disks.go:155-165` claims only the two cases the R-113 conjunction does fix (the boot-ordering window, +and a bind outliving its device *while still absent*). It does **not** claim staleness after a return, so +per the workspace rule this is a genuine gap rather than a seventh shipped false guarantee. + +But the *summary line* is a wish: + +> `// BoundUnderParent reports whether the drive is live + usable in the guest in the intermediary model.` + +**Measured: it reports neither.** In the R-117 state the drive is not live in the guest and not usable — +every content read and every write returns `EIO` — and the field reads `true`. The interface comment at +`disks.go:154-155` makes the same promise (*"the guest-usable signal — distinct from the host having the +bind"*), and `disks.go:83` a third time. Three statements of a property nothing tests. **Per the standing +rule, whichever fix lands owes a test that pins the consequence** — that a dead namespace reports +`BoundUnderParent = false` — and not merely the mechanism. + +--- + +## 6. Q4 — what a reliable check costs, and how it fails + +All timings from the R-117 state on drive B, on the venue, one run each, `-timeout 3s`. + +### 6.1 Measured + +| Probe | What it does | R-117 state (B) | in-place state (A) | healthy | Cost | Verdict | +|---|---|---|---|---|---|---| +| **P1 devno compare** | field 3 of the bind's mountinfo line vs field 3 of the raw mount's — **pure `/proc`, host-side only** | **STALE=true** ✅ | STALE=**false** ❌ | false ✅ | **0.38–0.45 ms** | **Sound for the return case (§3.4). Blind to in-place death** | +| **P2 ext4 abort options** | `shutdown` **or** `emergency_ro` in the super options — **pure `/proc`** | **DEAD=true** ✅ | **DEAD=true** ✅ | false ✅ | **0.17–0.24 ms** | **Catches both.** ext4-specific | +| P3 `statfs` | one syscall | **OK — MISSES IT** ❌ | OK — misses ❌ | OK | 0.042 ms | **Worthless** | +| P4 `getdents` | read one dir entry | **OK — MISSES IT** ❌ | OK — misses ❌ | OK | 0.045–0.061 ms | **Worthless** | +| P5 write + unlink | create, write 1 byte, remove | `EIO`(5) ✅ | `EROFS`(30) ✅ | OK | 0.042–0.152 ms | Catches both, **but writes**, and cannot distinguish a legitimately-`ro` mount from a dead one | +| P6 write + **fsync** | forces the byte to the device | **not measured** (§6.3) | — | — | — | **The probe that can hang** | +| P7 read with page cache dropped (`POSIX_FADV_DONTNEED`) | forces a read to reach the device | **not measured** (§6.3) | — | — | — | **The probe that can hang** | + +`GuestSeesMount` itself costs **3.85–5.03 ms**, an order of magnitude more than any probe above, +because it forks `lxc-info`. **Any of P1/P2 is free relative to the check it would strengthen.** + +### 6.2 The failure modes the brief named, answered from measurement + +- **"A write probe fails wrongly on a legitimately read-only mount."** Confirmed as a real hazard, and + worse than stated: in the in-place death state the mount *is* read-only (`emergency_ro`) and P5's + `EROFS` is indistinguishable from an intentionally `ro` mount by errno alone. **P5 cannot be the sole + signal.** +- **"A read probe on an empty drive has nothing to read."** Real, and it has a second and larger + problem: **a read probe passes from page cache over a dead filesystem** (P4 succeeded in every dead + state). Honest read probing requires dropping the cache first (P7), which converts it into a probe + that must reach the device — i.e. one that can block. +- **"Spinning up a sleeping disk on every poll has a real hardware cost."** **P1 and P2 eliminate this + concern entirely**: they read `/proc` and issue no block I/O, so a spun-down drive stays spun down. + This is the strongest argument for the pure-`/proc` pair over anything else, independent of the hang + question. + +### 6.3 The hang case — NOT ESTABLISHED + +**This is the one question in the brief that this run does not answer, and the gap is real.** + +Designed and built, not run: a `dm-linear` device suspended with `dmsetup suspend --noflush`, which +queues all I/O indefinitely instead of returning `EIO` — the controlled "half-dead or spun-down device" +the brief asks about. `r117probe` was extended with **P6 (write + `fsync`)** and **P7 (read with the page +cache dropped)** — the two probes that must reach the platter and therefore can block — plus reporting of +whether the blocked goroutine **ever returns** and the process's OS-thread count before and after. + +**It was not run: the second house-internet outage (§2.5) began while the rebuilt 3.6 MB binary was being +copied to the venue.** The build succeeded locally; the transfer did not complete. + +Not measured, therefore not claimed. What is *known* from what did run: + +- **P1 and P2 cannot hang** — they read `/proc/self/mountinfo`, which involves no block device. This is + a property of the files they read, and it is why the recommendation in §8 rests on them. +- The reasoning that a timeout around a blocked uninterruptible syscall leaves a **leaked thread per + poll** — and that at the reconcile ticker's 20-second cadence (`main.go:973`) that is 180 per hour — + is the hypothesis P6/P7 were built to test. **It is untested. It must not be quoted as a result.** + +--- + +## 7. Q5 — where the check belongs + +**The agent.** Not a balance of arguments — the controller is structurally incapable of it: + +1. **The inputs are host mount tables.** P1 needs the host's raw-mount entry for `/mnt/`; P2 needs + the super options of the host bind. The controller runs **inside** the guest and can see neither. The + guest's own mountinfo carries the stale entry (with `shutdown`, so P2 is available in-guest) but has + **no entry at all for the raw host mount**, so P1 is unavailable to it in principle. +2. **Both inputs are already in the agent's hand, in the same function.** `disks.go:222` already calls + `boundUnderParent(...)` and `devicePresent(t.MountPath)` with both paths in scope. The devno + comparison adds no new data source and no new dependency. +3. **The agent owns drive identity** (`durable_id`, enrolment, the fs-UUID-keyed mount units), so a + drive-liveness verdict belongs to the same owner under *one state, one owner*. +4. **The field already exists and is already documented to mean exactly this** (§5.3). The controller + consumes it and keys the gate on it (`controller/.../intermediary.go:226`). Fixing the producer fixes + every consumer at once; adding a second, in-guest liveness notion would create the competing + definition the hub's liveness-threshold rule exists to forbid. + +**Ruled on evidence, and the evidence points the same way as the principle** — but note it is (1) that is +decisive, not (3). + +--- + +## 8. Q6 — in-place recovery, and the cheapest fix available + +### 8.1 Recovery works, live, without stopping the guest — proven + +The repair is `AttachDrive`'s normalize leg (`intermediary.go:238-247`) verbatim: `umount` every layer at +the stable path, then one fresh `mount --bind`, which re-fires propagation into the guest's namespace. + +``` +guest init pid BEFORE: 2465515 + umount /mnt/r117-drives/sd ... ; mount --bind /mnt/r117sd/felhom-data /mnt/r117-drives/sd +guest init pid AFTER: 2465515 <-- unchanged: the guest was never restarted +``` + +Verified after the repair: + +``` +host raw /mnt/r117cel -> /dev/mapper/r117cel2 252:12 rw,relatime +host bind /mnt/r117-drives/cel -> /dev/mapper/r117cel2 252:12 rw,relatime +guest bind (mount id 744, NEW) -> 252:12 ... master:302 +GUEST statfs/listdir/read-canary/write -> all OK +P1 devno-compare raw="252:12" bind="252:12" -> STALE=false +``` + +**Two `mount` syscalls, no guest downtime, no app restart, full recovery.** So a fix can self-heal; it +does not have to settle for alarming. + +### 8.2 The repair is already wired — three times — and defeated by one line + +| Call site | What it does | Why it does nothing today | +|---|---|---| +| `cmd/felhom-agent/main.go:973-985` | 20-second ticker → `ReassertEnrolledMounts` then `ReassertGuestBinds` | → `disks.go:1108` → `AttachDrive` → short-circuits | +| `disks.go:951` | the same, at agent startup | same | +| `controller/internal/web/intermediary.go:301-305` | the `Return` branch calls `agent.GuestAttach(ctx, a.Raw)` — **before `restartStacks` at `:311`** | → `disks.go:624` → `AttachDrive` → short-circuits | + +All three land on `intermediary.go:234-236`: + +```go +n := countHostMounts(stable) +if n == 1 && b.GuestSeesMount(ctx, vmid, stable) { + return stable, nil // exactly one bind + guest-visible → fully live, no-op +} +``` + +In the R-117 state `n == 1` and `GuestSeesMount` is `true`, so **`AttachDrive` declares the namespace +"fully live" and returns without touching it.** The comment on that line is the sixth entry in the +workspace's false-invariant table in all but name: *"fully live"* is precisely what it is not. + +> **The ordering is already correct.** The controller asks the agent to re-attach at `:301-305` and only +> then restarts the apps at `:311`. If `AttachDrive` repaired the bind instead of short-circuiting, the +> apps would be restarted onto a **repaired** namespace by the code that exists today. **The defect is +> one predicate, not a missing recovery path.** + +--- + +## 9. Q7 — scope, and the half a `Return`-branch fix cannot reach + +**A bind can die in steady state, with no detach/return cycle at all — measured, not inferred.** + +Drive A, device swapped to an `error` target with everything running and nothing detached: + +``` +write into the bind from the guest -> Input/output error (immediately) +mnt-r117cel.mount -> active <-- still mounted + BindsTo="dev-dm\x2d11.device" the DEVICE never disappeared +host raw /mnt/r117cel -> /dev/mapper/r117cel 252:11 rw,relatime +host bind /mnt/r117-drives/cel -> /dev/mapper/r117cel 252:11 rw,relatime,emergency_ro +HALF1 GuestSeesMount = true HALF2 isHostMountpoint = true +=====> BoundUnderParent = TRUE +GUEST write -> errno=30 Read-only file system +``` + +**Nothing transitions.** `State` stays `attached`, `BoundUnderParent` stays `true`, the drive is never +`Disconnected`, so `planDriveGates` produces **neither** a `Stop` action nor a `Return` action +(`controller/.../intermediary.go:254-259` — both arms require `sp.Disconnected` to differ from +`present[sp.Path]`). **No gate log line, no event, no email, no banner. The customer's apps write into a +read-only namespace and every signal reads healthy indefinitely.** + +Reached in this run by a device that errors without disappearing. Real-world equivalents: a SATA/USB link +reset that recovers the *link* after ext4 has already aborted, a drive that develops write errors, an +enclosure that browns out briefly. **Not exotic.** + +Two consequences for the eventual fix: + +1. **A fix that lives in the `Return` branch cannot close this.** The branch never executes. +2. **P1 does not detect it either** — the device number is unchanged (252:11 on both sides), because the + device never left. **Only P2 caught it** (`emergency_ro`). This is the reason §8's recommendation is + **P1 ∨ P2** and not P1 alone; a fix built on the device-number comparison that the return case makes + so clean would ship with the silent half still open. + +**Probed deliberately, per the brief's instruction not to infer it from an absence of reports.** + +--- + +## 10. The reproduction recipe + +Exact enough to trigger on demand. `scsi_debug` is the load-bearing choice — it is the only mechanism +tried that can remove a gendisk **while it is mounted**, which is what a USB pull does and what dm and +loop cannot do (both are held open by the stale bind: `dmsetup remove --force` → `Device or resource +busy`). + +```bash +# --- venue: any Proxmox host + a scratch LXC. Shared parent MUST exist before the guest starts. --- +PARENT=/mnt/r117-drives; RAW=/mnt/r117sd; VMID=9301 +mkdir -p $PARENT +mountpoint -q $PARENT || { mount --bind $PARENT $PARENT + mount --make-private $PARENT; mount --make-shared $PARENT; } +pct set $VMID -mp0 $PARENT,mp=$PARENT # then start the guest; verify `master:` in its mountinfo +pct start $VMID + +# --- a drive whose device node can be removed while mounted --- +modprobe scsi_debug dev_size_mb=64 max_luns=1 num_tgts=1 +DEV=/dev/sdb # whichever /sys/block/sd*/device/model is scsi_debug +mkfs.ext4 -q -L R117SD $DEV +UUID=$(blkid -s UUID -o value $DEV) +mkdir -p $RAW +systemd-mount --no-block --collect -t ext4 /dev/disk/by-uuid/$UUID $RAW # UUID-keyed, per mountunit.go:51 +mkdir -p $RAW/felhom-data && chown 100000:100000 $RAW/felhom-data +echo canary > $RAW/felhom-data/.r117-canary && sync +mkdir -p $PARENT/sd && mount --bind $RAW/felhom-data $PARENT/sd +# the guest sees it immediately via propagation — no pct, no reboot + +# --- 1. PULL: real device removal. systemd stops the raw mount unit by itself (BindsTo=). --- +echo 1 > /sys/block/sdb/device/delete +# -> bind survives with `shutdown`; raw path is NOT a mountpoint; BoundUnderParent = false + +# --- 2. RETURN: rescan. It comes back on a DIFFERENT node because the stale bind pins the old one. --- +echo '- - -' > /sys/class/scsi_host/host1/scan # -> /dev/sdc, 8:32 +systemd-mount --no-block --collect -t ext4 /dev/disk/by-uuid/$UUID $RAW +# -> R-117: raw on 8:32 healthy, bind on 8:16 dead, BoundUnderParent = TRUE, EIO both directions + +# --- Q7 variant: steady-state death, no cycle. Needs a device that ERRORS without disappearing. --- +# dm-linear over a loop file, then swap the table to `error` and resume: +dmsetup load --table "0 error" && dmsetup resume +# -> raw mount stays ACTIVE, BoundUnderParent stays TRUE, writes EROFS, superopts gain emergency_ro + +# --- repair (Q6), no guest restart: AttachDrive's normalize leg, verbatim --- +umount $PARENT/sd && mount --bind $RAW/felhom-data $PARENT/sd + +# --- control test for §3.4: with NO stale bind holding it, the letter is REUSED --- +umount $PARENT/sd; echo 1 > /sys/block/sdc/device/delete +echo '- - -' > /sys/class/scsi_host/host1/scan # -> /dev/sdb, 8:16 again +``` + +**Traps, each paid for in this run:** + +- Establish the **shared parent before the guest starts**, or the guest's parent bind is `private` and no + drive swap propagates. Verify by finding `master:` on the guest's parent entry. +- The parent bind is **non-recursive**, so a drive already bound under the parent when the guest starts + is **invisible** to it. Re-fire with `umount` + `mount --bind`. Observed directly at guest first start. +- Do **not** reuse the live `/mnt/felhom-drives`: a bind under it propagates into the real customer guest, + and the agent's reconcile will act on it. +- `statfs` and `ls` **succeed** on the dead bind. Verify death by reading **file content** or writing. + +--- + +## 11. Recommendation, and its cost + +**Framed as a constraint on the fix, not as an implementation.** No code was written. + +**Make the liveness signal compare the two mountinfo entries it already reads, and consult the +filesystem's own abort state. Do not add block I/O.** + +``` +live(stable, raw) := bindEntry(stable) exists + AND rawEntry(raw) exists <-- devicePresent today + AND bindEntry.devno == rawEntry.devno <-- P1: closes the return case (§3.4) + AND bindEntry not aborted <-- P2: closes the steady-state case (§9) +``` + +**Cost:** two `/proc/self/mountinfo` reads that `isHostMountpoint` and `countHostMounts` **already +perform** on every pass, plus two field comparisons. **Measured 0.17–0.45 ms**, against the 3.85–5.03 ms +`GuestSeesMount` already spends forking `lxc-info` on the same call. **No block I/O, so no disk spin-up, +no write, and no possibility of blocking the report loop.** Effectively free. + +**Constraints the fix must respect, each earned above:** + +1. **Both halves are needed.** P1 alone leaves Q7 silent; P2 alone leaves any non-ext4 filesystem + uncovered. **Neither alone is sufficient, and the P1-only fix is the tempting one** because the return + case is what R-117 was filed as. +2. **Keep the "cannot tell → never absent" rule** (`disks.go:983-986`). An unreadable `/proc`, a missing + raw entry with an empty path, or an unrecognised filesystem must degrade to *unknown* and be treated + as present — reporting absent stops a customer's apps. The `(bool)` return shape that made this + impossible for `newestArchiveOn` (workspace table, row 2) is the trap to avoid: **`live` needs a third + state, or an explicit unknown-is-present convention pinned by a test.** +3. **P2's token list is filesystem-specific.** `shutdown` and `emergency_ro` are ext4. Both were measured + here; both must be in the list, because a probe checking only `shutdown` passes the entire Q7 state. + For any other filesystem the honest answer is *unknown* (see 2), not *live*. +4. **Fixing the predicate is the whole fix, and it self-heals.** §8.2: three call sites already invoke + the repair, including the controller's `Return` branch **before** it restarts apps, and the 20-second + ticker. An honest predicate makes `AttachDrive` stop short-circuiting, and the bind is repaired within + ~20 s with no guest restart. **Do not add a new recovery path; unblock the existing one.** +5. **The test must assert the consequence, not the mechanism.** Per the standing rule and R-97b's + Scenario F: assert that a **dead namespace yields `BoundUnderParent = false`** (and that the drive-gate + alarm fires), not merely that a devno comparison returns what it was fed. §5.3 lists **three** comments + promising a property nothing currently tests; the fix owes one that pins it. +6. **Q7 needs its own scenario.** A test that only exercises detach→return will pass while the silent + half stays broken. The in-place recipe in §10 is the second scenario. +7. **The hang question is open** (§6.3). It does not gate this recommendation — P1 and P2 read `/proc` + and cannot block — but it **does** gate any proposal to add an I/O confirmation probe on top. Settle + §6.3 before accepting one. + +--- + +## 12. What this changes about R-116's closure + +**R-116's verdict stands. Its evidence was narrower than it reads, and one caveat needs re-labelling.** + +- **The four-event sequence is unaffected.** R-116 was about *which event fires for which drive* — the + pairing of `backup_target_absent`/`backup_target_restored` and their discrimination from the generic + pair. That was proven on the wire, and every input to it (`isTarget`, the guest-path dedupe, the row + join) is **configuration-derived** — `storage.cfg`'s `path` versus the `.mount` unit's `Where` — which + R-117 does not touch. Nothing in this spike disturbs it. +- **`R116-v0116-2026-07-30.md` §5 already recorded the interference and drew the right line** + (*"a `backup_target_restored` there proves pairing, not recovery"*). This run **confirms that caveat + from the layer beneath**: on the reattach leg the namespace really was dead, `EIO` both directions, so + the run's `bound_under_parent = true` was a correct answer to the pairing question and a **false** + answer to the recovery question. +- **One thing does need re-labelling.** R-116's over-correction check is described in `OPEN-ITEMS.md` as + *"PASSES with a positive observable — 0 ABSENT lines / 0 drive events over 2m14s with both drives + present, target `degraded:false`, while 2 `RETURNED` lines prove the gate was ticking."* The two + `RETURNED` lines do prove the gate was ticking, so the check is **not** an absent-log verdict and rule + 3 is satisfied. But `degraded:false` over that window was read off a drive whose bind was dead — so it + evidences *"the gate did not over-fire"* and **not** *"the drive was healthy"*. **Narrower than it + reads, and still true as written.** Recommend the R-116 row gain one clause: the over-correction window + proves no spurious ABSENT, not drive health. +- **R-117 outranks R-116 in customer impact**, and §9 raises it further: the silent steady-state half has + no event on any channel at all, where R-116's worst case sent the *wrong* message. + +--- + +## 13. Findings filed — none fixed + +| # | Finding | Severity | Evidence | +|---|---|---|---| +| **R-117** | *(existing row, now with a mechanism and a recipe)* The bind-liveness signal is a path-presence test; the R-117 state is `BoundUnderParent = true` over an `EIO` namespace | **HIGH** | §3, §5 | +| **R-117a** | **A drive's bind can die in STEADY STATE, with no detach/return cycle, and no event is emitted on any channel.** The gate produces neither action, so a `Return`-branch fix cannot reach it. **P1 does not detect it; only the filesystem's abort flag does** | **HIGH — larger than R-117 as filed** | §9 | +| **R-117b** | **`statfs` and `getdents` both SUCCEED on a namespace returning `EIO` for all content.** Any liveness probe built on either is hollow by construction — it reports healthy in exactly the state it exists to detect | **MEDIUM (design trap)** | §4, §6.1 | +| **R-117c** | **Three comments** (`disks.go:83`, `:154-155`, `:155-165`) state that `BoundUnderParent` means "live + usable in the guest". **Nothing tests it, and it is false in the measured state.** Not counted as a shipped false invariant because `:158-165` scopes its own claim narrowly — but the summary lines do not | **MEDIUM** | §5.3 | +| **R-117d** | **`AttachDrive` logs and returns `"fully live"` for a namespace that is `EIO` on every call** (`intermediary.go:235`), disabling all three wired repair paths including the controller's pre-restart re-attach | **HIGH — this is the self-heal that already exists** | §8.2 | +| **R-117e** | **The two demo boxes share one uplink and one failure domain.** Both went unreachable together, twice in one session, on all four known routes including the WireGuard OOB path via ep0. There is no route to either box that survives the site losing internet — the "out-of-band" path is not out-of-band with respect to this failure | **LOW (operational, not a code defect)** | §2.5 | + +**Observed, out of scope, not acted on:** demo-hp's live agent is **0.113.0** against `main`'s 0.116.0 +(§1). It equals the manifest's declared `MinAgent`, so this is not R-115/R-111 drift — but it does mean +**the fleet's designated drill host does not run the R-113 conjunction whose successor this spike is +about.** Whoever fixes R-117 will need that box on a current agent to validate live. + +--- + +## 14. Teardown — **OWED, NOT YET DONE** + +**The second internet outage (§2.5) began before teardown. This section is a liability, not a record.** +Everything below is inert scratch on a Tier 0 box; nothing fenced was touched at any point (§14.2). + +### 14.1 Outstanding on demo-hp — the exact commands + +```bash +# layer 1 — the machine +pct stop 9301 && pct destroy 9301 --purge +# layer 2 — the host +umount /mnt/r117-drives/sd /mnt/r117-drives/cel 2>/dev/null +systemctl stop mnt-r117sd.mount mnt-r117cel.mount +umount /mnt/r117-drives # the scratch shared parent self-bind +dmsetup remove r117cel r117cel2 # now unheld once the binds are gone +losetup -d /dev/loop0 +modprobe -r scsi_debug # removes /dev/sdb (the fake disk) entirely +pvesm remove r117scratch && rm -rf /var/lib/vz/r117-scratch +rmdir /mnt/r117-drives/sd /mnt/r117-drives/cel /mnt/r117-drives /mnt/r117cel /mnt/r117sd +rm -rf /root/r117 /root/setup1.sh /root/setup2.sh /root/setup3.sh \ + /root/capture.sh /root/capture2.sh /root/r117probe /root/r117-pveam.log +# optional: rm /var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst (124 MB, downloaded +# this run; harmless to keep and useful for the next scratch guest) +# then verify: pct list | pvesm status | dmsetup ls | losetup -a | lsmod | grep scsi_debug | findmnt /mnt/r117-drives +``` + +### 14.2 Layer 3 — the hub: **nothing to dispose of** + +**This run created no hub-side record.** No customer, no appliance, no pairing, no claim: guest 9301 ran +no controller and had **no network interface at all**, and no agent was enrolled against it. Stated +explicitly per §13's rule that "teardown clean" without layer 3 is the `sess-c` failure — here layer 3 is +genuinely empty, and that is a different claim from silence. + +### 14.3 Fences — verified untouched throughout + +| Fence | State | +|---|---| +| demo-hp guest **9201** | `running` throughout; never a probe target | +| demo-hp **`drill-r50`** (VM 300) | `stopped`; never started, never touched | +| demo-hp **`local-lvm`** | untouched — scratch rootfs deliberately placed on a new dir storage on `pve-root` | +| demo-hp **`felhom-backup`** / `/mnt/nvme-1tb` | not re-targeted, not written to, `content backup` unchanged | +| demo-felhom, Peti's cluster | never contacted | +| **ep0** | **read-only** — one `ssh`, `wg show`, and TCP reachability probes toward `10.77.0.2/.3` during outage 1. Nothing written | +| `/mnt/felhom-drives` (live shared parent) | untouched; a **separate** scratch parent was used | +| **Code** | **no `.go` file in either repo modified**; nothing built for deployment, nothing published, no version bumped | +| Repos | the probe lives outside both, in the session scratchpad; the agent tree copy verified byte-identical to `d4eb259` | + +--- + +## 15. What did not happen, and why + +- **Q4's hang measurement was not taken** (§6.3) — the venue went offline mid-transfer. The venue + (`dmsetup suspend --noflush`) and the two probes (P6 write+`fsync`, P7 cache-dropped read, both with + timeout-fires reporting and thread-leak counting) are built and specified; only the run is missing. + **The associated hypothesis — that a timeout around a blocked syscall leaks a thread per poll — is + explicitly NOT claimed as a result.** +- **Teardown was not performed** (§14). Commands recorded; nothing fenced was touched. +- **No fresh `/disks` payload and no fresh controller gate log** — deliberate, and the same call + `DIAG-r116` §14 made: both were already captured verbatim on real hardware (twice and four times + respectively), and running a controller here would have meant a hub customer, a pairing, a golden fetch + and a claim — the work that consumed three prior sessions — to re-measure a known quantity. **The + missing layer was the mount tables and the errnos beneath that payload, which is what this run + supplies.** §5.2's capture is the join between the two. +- **The same-device-node return was not reproduced as a *defect* case** — because §3.4 showed it + **cannot** occur while a stale bind exists: the stale bind is what forces the new number. The control + test (letter reused once the bind is released) is the positive observable for that claim. +- **R-117 was not fixed**, per the fence — not even the one-line predicate change §8.2 identifies. The + design constraints are in §11 for a spec to accept or reject. +- **`felhotest` was not tried**; `Connection refused` was already recorded 2026-07-30 and it is + irrelevant to this mechanism. +- **Nothing was committed to `felhom-agent` or `felhom-controller`** — this run's only writes are this + document and the `R-117` row in `backlog/OPEN-ITEMS.md`. + +--- + +Every claim above is either quoted from pushed Gitea source at `file:line` (agent `d4eb259` / controller +`b331f18`) or is captured probe output from the venue. Where a result was inferred rather than measured, +it says so. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 28c5bdb..e0ad85b 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -18,7 +18,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **WAITING-ON-OPERATOR (M)** | operator ruling on the release process | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design | CC | | **R-116** | ~~**The drive-absent alarm and its recovery were a MISMATCHED PAIR — absent fired the GENERIC `storage_disconnected`, return the SPECIFIC `backup_target_restored`; `backup_target_absent` never fired at all**~~ | **SHIPPED + PROVEN-LIVE** (agent v0.116.0, 2026-07-30) | — | **CLOSED. The full four-event sequence, on the wire, on a fresh box** (`audits/R116-v0116-2026-07-30.md`): `backup_target_absent (error)` on detach → `backup_target_restored (info)` on return for the TARGET, and `storage_disconnected (error)` → `storage_reconnected (info)` for a NON-target drive on the same box four minutes apart. **Two matched pairs, correctly discriminated — and discrimination is proven NON-trivially for the first time**, since both prior runs had the target itself emit the generic event. Gate fired in **3 s**; all four events reached the hub, so the specific alarm, its severity, its Hungarian copy and the hub routing are now exercised end-to-end. **Over-correction PASSES with a positive observable** (0 ABSENT lines / 0 drive events over 2m14s with both drives present, target `degraded:false`, while 2 `RETURNED` lines prove the gate was ticking). Ran on a nested PVE on **demo-hp** per `runbooks/target-selection.md` — through the **real day-0** from the v1.25.0 ISO, with the agent **installed unaided from the vouched Day-0 manifest** (published sha `b47c5c4dab641ee5…`, independent registry GET verified, manifest read back), drives enrolled through the real endpoints, device loss a real hot-detach. **THE FIX, and the ruling is the substantive part:** the mechanism was first isolated from the captured payload (`DIAG-r116-disks-payload-2026-07-30.md`) after two fixes aimed at shapes that do not occur. **Both smaller-looking options were REJECTED because they regress R-114** — `backup_target_offer.go:79` reads `BackupTarget && MountPath != ""` as *"a real drive with its own mountpoint — healthy"* and returns before its `TargetAbsent` branch, so back-filling `MountPath` on the Observe row **or** flagging the registry row (whose `MountPath` is the stale unit-file value) would have told the customer the backup target is fine while its drive was gone. **R-114's correctness was resting on R-116's bug** — a coupling invisible until the payload existed. Taken instead: the Observe row gets the **guest path only** (`mount_path` stays `""`, which is true) from a new `ConfigPath` (`json:"-"`, so the cross-repo golden + key-set contract is untouched), and the union row is deduped **on guest path** — the join being CONFIGURATION (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), the only identity that survives the device. Tests 845→849; 4 red-proofs each asserted to land, and red-proof 1 replays v0.115.0's code and fails, which is the empirical proof it was inert. Its green test had supplied a `MountPath` production never supplies AND left `DriveTargets` nil so the union loop never ran — both corrected. v0.115.0 left in place (inert, harmless). Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded. **Caveat: the drill's controller was 0.185.1 from the golden, which PREDATES R-114**, so its absent-state banner showed the old false copy — the golden being a release behind, not a regression → **R-120** | — | | **R-120** | ~~**The golden baked a controller that predated R-114 + R-112, so a FRESH box showed the customer the WRONG absent-target message**~~ | **CLOSED — golden rebaked + PROVEN-LIVE, and the class now has an ENFORCED gate** (golden 0.186.0 + hub v0.82.0, 2026-07-30) | — | **`audits/R120-golden-rebake-2026-07-30.md`.** **Half 1 — the artifact.** Golden **0.186.0** baked from `main`'s controller in the DooPlex bake fixture (overlay2 OK, **3 mounts**, FATAL 0, exclusions 0, 618 MB, upload **201**, `GOLDEN_SHA256=b760ac6a33e70700…`, token-leak grep 0, GL-1 teardown, `drill.qcow2` back to `virgin`). Three observables: **published** — anonymous GET (what the installer does) 200 / 648930639 bytes / sha identical to the bake; **vouched** — manifest read BACK; **resolved** — `Artifact manifest served for customer sess-f (agent=0.116.0 golden=0.186.0)`. Floor **untouched** per publish-train rule 2 (`min_controller_version` still 0.156.0; it is a separate form); MinAgent left 0.113.0 as 0.186.0 declares. **Proven on a REAL day-0, not the fixture** (per the Part-1 rule now in `runbooks/target-selection.md`): VM 9402 on demo-hp from the v1.25.0 ISO → `Controller elindult (0.186.0)`. With the target detached the endpoint returned the **`TargetAbsent`** copy — *„A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod…"* — **and `offer_path` absent entirely**; the day-old read on the 0.185.1 golden had returned the false system-disk message **plus** an offer of the other drive. **Half 2 — the mechanism, operator ruling REFUSE.** hub **v0.82.0**: the gate sits in `hub/internal/web/configs.go` `handleSetArtifacts` immediately before the only write — the sole UI path to `SetArtifactManifest` — so it runs on every vouch without anyone choosing to, and it **refuses** rather than warning. Signal: `store.NewestReportedControllerVersion()` over `reports.controller_version`, **semver-compared in Go** (`MAX()` in SQL ranks 0.99.0 above 0.186.0 — a pair this fleet has shipped). Fail-open in exactly two deliberate cases: empty golden field, unknown fleet version. **NEAR-MISS RECORDED: the first draft read `guests.controller_version`, a column that exists and that NOTHING writes** — it would always have seen `""` and failed open, i.e. inert, this gate's own failure shape, one grep from shipping. 4 tests through the **production handler** over httptest (never a seam), the refusal asserting **both** the flash **and** that the manifest was not written; red-proof: deleting the block makes the stale golden vouchable again. **PROVEN LIVE on the deployed hub by re-attempting the original mistake:** vouching 0.185.1 → `HTTP 303 …flash=golden_behind_fleet` + `[WARN] artifact vouch REFUSED: golden 0.185.1 is older than the newest controller the fleet reports (0.186.0)`, and the manifest read back **unchanged at 0.186.0**. Recorded on **R-29's audit list** (`ROADMAP.md`) as the **first enforced gate** beside its three orphans, so the contrast is kept — the orphans are unchanged. Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded, exactly as `sess-e` was (and `sess-e` was deleted this run) | — | -| **R-117** | **A returned drive's guest bind is a DEAD MOUNT, and the gate restarts the customer's apps onto it while every signal reads healthy.** After a detach/reattach the host raw mount is fine on the NEW device node, but the guest's bind still names the OLD one and every operation through it returns `EIO`: host `findmnt /mnt/cel` → `/dev/vdc`; guest `findmnt /mnt/felhom-drives/cel` → `/dev/vdb[/felhom-data] …,shutdown`; guest `ls` and write → **`Input/output error`**; and `/disks` reports `state:"attached"`, `role:"user-data"`, `bound_under_parent:TRUE`, `backup_target:true` | **READY (M) — NEW 2026-07-30, deliberately not chased** | — | **Worse than R-116 and it outranks it.** `planDriveGates` computes `present[gp] = … \|\| d.BoundUnderParent` (`intermediary.go:226`) = `true`, and the drive is `Disconnected`, so the gate takes the **`Return`** branch (`intermediary.go:281`, `:301-320`): it clears the flag, **restarts the gate-stopped apps onto a namespace that `EIO`s on every call**, reports the drive healthy and emails `backup_target_restored`. **No alarm on any channel.** R-113's conjunction cannot catch it — half 1 (guest mountinfo has an entry at the guest path) is satisfied by the STALE entry and half 2 (`devicePresent`) by the NEW device; **neither half compares the two**. Not a false invariant: `disks.go:158-165` claims only the boot-ordering and outlived-bind cases, so this is a genuine gap. **Seen but uninvestigated in E-2d, Session C and Part 5 (three consecutive runs) as "the drive returned as /dev/sdc while findmnt still read /dev/sdb"** — that framing read as cosmetic; it is not. Reproduced deliberately and probed to EIO in `audits/DIAG-r116-disks-payload-2026-07-30.md` §11. Deserves its own run | CC | +| **R-117** | **A drive's guest bind becomes a DEAD MOUNT while every signal reads healthy — and it happens in TWO ways, only one of which the original framing covered.** (a) *after a detach/return*: the host raw mount heals onto the NEW device via its fs-UUID-keyed unit while the bind still names the OLD one, so the gate takes its `Return` branch and restarts the customer's apps onto a namespace that `EIO`s on every call; (b) *in STEADY STATE, no cycle at all* — a device that errors without disappearing leaves the raw mount `active`, `BoundUnderParent` `true` and the drive never `Disconnected`, so **the gate produces no action and NOTHING is emitted on any channel** | **MECHANISM + RECIPE ESTABLISHED (spike done 2026-07-30) — READY (S/M), not fixed** | — | **SPIKE: `audits/SPIKE-r117-bind-liveness-2026-07-30.md`** — reproduced deliberately **3/3**, two device classes, on a purpose-built scratch LXC (9301) on demo-hp; predicates evaluated by a throwaway probe calling the **real** `localapi` code from agent `d4eb259` (v0.116.0). **MECHANISM, confirmed at source and live:** both halves of the R-113 conjunction are *path-presence tests*. `GuestSeesMount` (`intermediary.go:276`) and `isHostMountpoint` (`intermediary.go:394`) each compare **field 5** (the mount point) of a mountinfo line and **never read field 3, the `major:minor`** — so neither can see that the bind and the raw mount name **different devices**. Measured: `raw 8:32 /dev/sdc` vs `bind 8:16 /dev/sdb …,shutdown`, `BoundUnderParent = TRUE`, `EIO`(5) on read **and** write, host and guest. **The node change is a CONSEQUENCE, not a precondition** — control test: with the stale bind held the drive returns as `sdc` (8:32); once it is `umount`ed the letter is **reused** (`sdb`, 8:16). The stale bind pins the dead superblock, which forces the new number — so a devno comparison is *sound* for case (a), not a heuristic. **RECIPE** (spike §10): `scsi_debug` is load-bearing — it is the only mechanism that removes a gendisk **while mounted** (dm and loop are held open by the stale bind: `dmsetup remove --force` → `Device or resource busy`). `modprobe scsi_debug` → mkfs → UUID-keyed `systemd-mount` → bind under a **scratch** shared parent → `echo 1 > /sys/block/sdb/device/delete` (systemd stops the raw unit itself, `BindsTo=`) → `echo '- - -' > /sys/class/scsi_host/hostN/scan` → re-mount by UUID. Case (b): swap a dm table to `error`. **TWO DEATH STATES, different signals** — device removed ⇒ superopts gain `shutdown`, `EIO` both ways; device errors in place ⇒ `emergency_ro`, write `EROFS`(30), reads still served from cache. **PROBE TRAP (R-117b): `statfs` and `getdents` SUCCEED on a namespace that `EIO`s every byte** — any liveness probe built on either is hollow by construction. **A pure-`/proc` check costs 0.17–0.45 ms**, cannot hang, spins up no disk and writes nothing, against the 3.85–5.03 ms `GuestSeesMount` already spends forking `lxc-info`. **CONSTRAINT: the fix needs BOTH a devno comparison (closes (a); blind to (b)) and the fs abort flags `shutdown`\|`emergency_ro` (closes (b); ext4-specific)** — the tempting devno-only fix leaves the silent half open. Keep the *cannot-tell → never absent* rule (`disks.go:983-986`). **R-117d — THE FIX IS ONE PREDICATE, AND IT SELF-HEALS:** `AttachDrive`'s normalize leg (`intermediary.go:238-247`) already performs exactly the repair — proven live, **guest never restarted** (init PID identical) — and **three** call sites already invoke it: the 20 s reconcile ticker (`main.go:973-985`), agent startup (`disks.go:951`), and **the controller's `Return` branch itself, at `intermediary.go:301-305`, BEFORE it restarts the apps at `:311`**. All three are defeated by `if n == 1 && b.GuestSeesMount(...)` returning early at `intermediary.go:235` and logging *"fully live"*. **Unblock the existing path; do not add a new one.** **R-117c:** three comments (`disks.go:83`, `:154-155`, `:155-165`) promise "live + usable in the guest"; nothing tests it and it is false in the measured state — the fix owes a test asserting the **consequence** (dead namespace ⇒ `BoundUnderParent=false` ⇒ alarm fires), plus a second scenario for case (b). **OPEN from the spike: the HANG case was NOT measured** (§6.3) — venue lost internet mid-transfer; the `dmsetup suspend --noflush` venue and the two blocking probes (write+`fsync`, cache-dropped read) are built and specified. It does not gate the `/proc` recommendation, but it **does** gate adding any I/O confirmation probe. **Teardown of the spike venue is OWED** (spike §14 lists the exact commands; nothing fenced was touched, no hub-side record was created). Note demo-hp's live agent is **0.113.0** — it does not run the R-113 conjunction, so live validation of the fix needs that box current first | CC | | **R-118** | **An absent drive's union row advertises the ROOT filesystem's capacity as its own.** In the absent-state payload the registry-union row reports `total_bytes: 49675956224 / used_bytes: 4584579072` — **byte-identical to the `local` row** (`durable_id: path:/var/lib/vz`, i.e. `pve-root`) in the same response. The real drive is **4 GB** | **READY (XS) — NEW 2026-07-30** | — | Cause: `statfsCapacity(d.MountPath)` (`disks.go:335-338`) statfs's `/mnt/cel`, which with the device gone is a **bare directory on the root filesystem**. `observe.go:176-183`'s comment warns about exactly this trap and guards the Observe path ("*an unmounted removable dir-storage's mountpoint reverts to a bare directory on root … catastrophic DR mis-id*"); **the union path has no equivalent guard.** **Not a DR mis-id** — `durable_id` on that row is still the correct `uuid:…`, so re-attach identity is safe. It is a **false capacity** reaching every consumer of `total_bytes`/`used_fraction` (fill monitors, storage cards): a detached 4 GB drive advertises 46 GiB at 9.2 % used. Same class as `role.go:180-181` — an absent drive's fields decaying to the root filesystem's. Evidence: `audits/DIAG-r116-disks-payload-2026-07-30.md` §12 | CC | | **R-113** | ~~**The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.**~~ `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc//mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **SHIPPED + PROVEN-LIVE** (agent v0.114.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** Target drive hot-detached on a fresh box running the SHIPPED agent 0.114.0 (from the Day-0 manifest, not a hand build). **The gate fired in 4 seconds** — E-2d measured ZERO over 4½ minutes — and `SetDisconnected` was reached. It fired on exactly the shape that defeated it: raw `/mnt/mentes` NOT mounted while the bind `/mnt/felhom-drives/mentes` still read `/dev/sdb[/felhom-data]`. **Over-correction guard PASSED:** with both drives present, 0 ABSENT lines and the target stayed healthy. **Note: C5 still FAILED — but on a different defect (→ R-116), not on this one.** **SESSION C IS NOW UNBLOCKED.** Agent **0.114.0 published and vouched** 2026-07-29 (sha `5e4c15ebee2d7583…`, round-trip GET verified; hub manifest read back showing it) — a fresh drill box now installs the fix under test instead of the bug. The golden was NOT re-baked and `min_agent` was NOT raised: the golden bakes the controller, not the agent, and controller v0.185.0 declares MinAgent 0.113.0, which 0.114.0 already satisfies. **FIX: `BoundUnderParent` is now a CONJUNCTION — bound under the parent AND the drive's raw host mount still mounted** (`devicePresent`, new `deviceCheck` seam), at BOTH `/disks` construction sites. The raw mount is the device-bound systemd unit that dies with the device; the agent's bind is not — so the raw mount IS the device signal, grounded in E-2d's measurement rather than inference. **Conjunction, deliberately:** the device half alone would regress boot ordering (raw mounts early, bind lands ~18 s later — that window must keep reading absent), so existing behaviour is byte-identical and only the unreachable case is closed. **Unknown is never absent** (`devicePresent("")` = true) — a false absent stops a working customer's apps. **Controller UNCHANGED, no MinAgent bump:** `BoundUnderParent` has exactly one functional consumer (`planDriveGates:226`); a new `DevicePresent` bool was rejected because absent-from-JSON decodes to `false`, which would have made every drive on an older agent read ABSENT. +6 tests (208→214), 4 red-proofs run and reverted. **Deployed to demo-felhom; over-correction guard verified in production** — raw mount present, drive still reads present, 10/10 apps untouched, no gate action, no false alarm. **demo-hp deliberately left on 0.113.0** (the spec scoped deploy to felhom-pve) — it still carries the bug. **⚠️ SESSION C BLOCKER: the hub Day-0 manifest vouches agent 0.113.0**, so a fresh drill box would install WITHOUT this fix and validate nothing — publish + vouch 0.114.0 first (R-111's trap, same shape). **Awaiting live proof of exactly one leg:** device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. Original finding: **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 **Session C scope UNCHANGED by Session B** — R-114/R-112 shipped as controller v0.186.0 and neither touches the agent; the leg awaiting proof is still device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. One rebuild now validates all three. | CC | | **R-112** | **E-2's degraded banner and offer have NO UI CONSUMER — the endpoint is correct and the customer never sees it.** `GET /api/storage/backup-target` returns byte-exact copy (proven live), and **nothing fetches it**: `grep 'backup-target'` across all `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` + `degradedMessageFor` are consumed **only** by the JSON handler — **no page handler injects the state** | **SHIPPED + PROVEN-LIVE** (controller v0.186.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md` — the banner reached a customer's page for the first time.** Never-configured box: banner element 1, never-configured copy 1. After the wizard: offer block 1 with `data-path="/mnt/felhom-drives/mentes"`. Healthy after assign: all four markers 0, **proven POSITIVELY** — idle delta 0 `/backup/tiers` calls, page-load delta +1, single caller ⇒ the seam ran and chose silence. **FIXED: the state now has a consumer.** Server-rendered on `/backups` via `backupsHandler` → `backupTargetView` → `backups.html`, following the existing `SingleCopyWarning` banner pattern — NOT a 19th JS fetch, because a banner that needs JavaScript to appear is one more thing that can silently not happen. `backupTargetView` returns **nil** for healthy AND unknown, so those render nothing at all. **Scenario-E seam test drives `backupsHandler` over httptest and asserts the RENDERED HTML** — deleting the one line that sets `data["BackupTarget"]` reproduces the old state and fails every render assertion. **SEAM PROVEN LIVE on demo-felhom by a DIFFERENTIAL positive observable, not by an absent banner:** idle 8 s → 0 new `/backup/tiers` agent calls; each `/backups` load → exactly +1, and that call has only one caller (`resolveBackupTargetState`). The box is healthy (`degraded:false, target:felhom-backup`) and the page correctly rendered **nothing** — which matches its real state but, being a negative, is NOT by itself proof of wiring. **Still unproven live:** that a customer sees actual copy — impossible on a healthy box. **The decisive contrast: templates fetch 18 distinct `/api/storage/*` endpoints; `backup-target` and `backup-target/assign` are the only two with zero references.** The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"* — an invariant comment asserting a consumer that does not exist (7th instance of that class). v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* — it fixed the **router mount** and stopped one layer short of the **render**; its test `TestBackupTargetRoutesLiveUnderTheStorageAPIMount` pins dispatch, not reachability, which is exactly what `CLAUDE.md`'s seam rule warns about. **Fifth instance of seam-built-but-never-wired.** **Fix R-114 FIRST** — wiring this alone would start showing customers the wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | CC |