felhom.eu/documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md

SPIKE R-117 — a dead bind that reports healthy

Class: Spike. No production code was written; no .go file in either repo was modified. Output is this document. Findings are filed as R-n, none fixed.

---

0. Verdict, up front

#	Question	Answer
Q1	Reproduce deliberately	YES — 3/3 attempts, two independent device classes. A device-node change is NOT a precondition — it is a CONSEQUENCE of the stale bind, proven by a control test
Q2	What is dead, at which layer	The guest bind and the host bind (same mount, one peer group); the raw host mount is healthy on the new device. Two distinct death states with different errnos — EIO(5) both directions, and EROFS(30) on write only
Q3	Why boundUnderParent returns true	It is a string comparison of field 5 of a mountinfo line. It never touches the filesystem, and neither half reads field 3 (the device number) — so neither can see that the two entries name different devices
Q4	Cost + failure modes of a real check	A pure-/proc check costs 0.16–0.45 ms, cannot hang, cannot spin up a disk and writes nothing. The two cheapest I/O probes — statfs and getdents — succeed over a fully dead filesystem and are worthless. Every probe that reaches the device blocks on a wedged one, and the process then cannot be killed even with SIGKILL (§6.3)
Q5	Where the check belongs	The agent, and not on balance — the controller cannot perform it: the inputs are host mount tables it has no access to
Q6	In-place recovery possible	YES — proven, guest never restarted (init PID identical across the repair). And the repair code already exists, is already invoked at exactly the right moment, and is short-circuited by the same false predicate
Q7	Scope — steady state too?	YES, and this is the worse half. A drive can die with no detach/return cycle at all; the gate then never runs, the Return branch never fires, and no event of any kind is emitted

The single most consequential finding is Q7. R-117 was filed as a defect in the reattach path. It is not confined to it: a device that fails in place produces the identical all-signals-healthy state with no gate transition to hang a fix on. A fix that lives in the Return branch closes the half that at least emits events, and leaves the silent half open.

The second is Q6. AttachDrive's normalize leg (intermediary.go:238-247) already performs exactly the repair this defect needs — umount every layer, lay down one fresh bind — the controller's Return branch already asks the agent to do it (controller/internal/web/intermediary.go:301-305) before it restarts the apps, and the agent's 20-second reconcile ticker (cmd/felhom-agent/main.go:973-985) calls it unconditionally. All three are defeated by one line: if n == 1 && b.GuestSeesMount(...) returns early (intermediary.go:235). Making the predicate honest turns three already-wired call sites into a self-heal. That is the cheapest fix available and it is not the one a fresh design would have reached.

---

1. Confirmed baselines

Established fresh this run, not carried.

Component	Version	Source
felhom-agent main	v0.116.0 @ d4eb259da241c4ae2af81d5dff6a51b86c411b42	git -C felhom-agent log, CHANGELOG.md:1
felhom-controller main	v0.186.0 @ b331f18	git log, CHANGELOG head
felhom.eu main HEAD	29bcfeb (docs(R-120): CLOSED on both halves)	git log
hub, live	0.82.0 (gitea.dooplex.hu/admin/felhom-hub:0.82.0)	kubectl -n felhom-system get deploy
golden	0.186.0	OPEN-ITEMS.md R-120 row (rebaked + vouched 2026-07-30)
demo-hp PVE	pve-manager/9.2.2/b9984c6d90a4bd80, kernel 7.0.2-6-pve	pveversion
demo-hp agent, live	0.113.0	felhom-agent --version on the box

The live agent on the venue is 0.113.0 — three releases behind main (0.116.0) and behind R-113 (0.114.0), the very conjunction under examination. It equals the manifest's MinAgent, so it is the fleet's declared minimum, not drift. It was therefore never used as the source of a behavioural claim. Every predicate result in this document comes from a probe built from main @ d4eb259 (§2.3), not from the installed agent — precisely the "do not silently reason about a different codebase" rule in the brief.

---

2. Venue

2.1 The named venue does not exist — recorded, not worked around

The brief specifies "the standing snapshot-reset fixture on demo-hp, reset to virgin first". There is no such fixture on demo-hp. Enumerated on the box:

pct list  ->  9201 running demo-hp          (FENCED by the brief)
qm list   ->  300  stopped drill-r50        (FENCED by the brief; the only drift fixture, R-93)

The only standing snapshot-reset fixture in the estate is the nested-PVE golden-bake VM on DooPlex (drill/drill.qcow2, snapshot virgin) — a Tier 2 machine, and runbooks/target-selection.md:56-61 states that using it as a drill victim "is not covered by that exception and contradicts the 2026-07-25 ruling". So the two candidate venues were, respectively, absent and prohibited.

Resolution: the venue named by the brief (demo-hp) was kept and the fixture premise corrected. A purpose-built scratch LXC guest (VMID 9301) was created on demo-hp — Tier 1 on a Tier 0 host, which target-selection.md:83-87 permits as "create and destroy freely". This is a mechanism claim, so per target-selection.md:89-104 a fixture is the correct instrument and a day-0 would have been waste.

2.2 Fixture provenance — recorded per target-selection.md:101

Built 2026-07-30 11:18–11:24 CEST, from scratch, this session. Nothing about it is of unknown vintage:

Guest	LXC 9301, unprivileged, 256 MB / 1 core, no network interface at all
Template	debian-13-standard_13.6-1_amd64.tar.zst, downloaded this run, checksum verified by pveam
rootfs	2 GB raw on a scratch dir storage r117scratch at /var/lib/vz/r117-scratch — created for this run. Never local-lvm (fenced: over-subscribed thin pool backing live 9201) and never felhom-backup (the box's backup target)
Host	demo-hp, PVE 9.2.2, kernel 7.0.2-6-pve
Agent / controller inside it	none — see §2.4

2.3 What actually evaluated the predicates

A throwaway probe, r117probe, outside both repos, which calls the real exported agent code:

• The agent tree was exported at HEAD (git archive d4eb259) into the session scratchpad and the probe added there as cmd/r117probe. git status against that worktree, excluding the probe dir, is empty — the copy is byte-identical to main. Neither repo was touched. (Direct cross-module import is impossible: internal/... is unimportable from another module.)
• HALF1 is a real call to localapi.GuestBinder.GuestSeesMount (intermediary.go:276) through a real proxmox.ExecRunner, so the real lxc-info lookup and the real mountinfo parse both run.
• HALF2 is a verbatim copy of isHostMountpoint (intermediary.go:394) — the implementation behind devicePresent (disks.go:972), which is unexported. The copy is quoted against the original in §5.2; it is 12 lines with no branching beyond the field-count guard.

2.4 Fidelity gaps — stated, not glossed

Gap	Consequence
No agent daemon and no controller ran in the venue.	No fresh /disks payload and no fresh controller gate log were captured this run. Not a hole in the evidence: /disks in all three states was captured verbatim twice on real hardware (DIAG-r116-disks-payload-2026-07-30.md §11; R116-v0116-2026-07-30.md §5), and the gate log four times. What was missing from that record was the layer beneath it — the mount tables and the errnos — which is what this run supplies
The dm-based drive (§3.2 drive A) could not have its device node removed while mounted	Only affects drive A. Drive B used scsi_debug, whose gendisk can be deleted while mounted — real USB-pull semantics — and it is drive B that carries the headline reproduction
One step on drive A was modelled: systemctl stop of the raw mount unit stood in for systemd reacting to device loss	Superseded by measurement. On drive B the real device deletion made systemd stop the raw mount unit on its own (mnt-r117sd.mount → inactive, §3.3), confirming the BindsTo= mechanism empirically. The modelled step is not load-bearing for any conclusion

2.5 Two internet outages at the venue site

The venue is at the operator's vacation location until ~08-02; both demo boxes share one uplink.

• Outage 1, ~11:07–11:13 CEST. Both boxes went dark together. Four routes tried before concluding (standing rule 2): DooPlex→tailnet (100.76.96.79, 100.70.170.35) timeout, with tailscale status reporting both peers offline, last seen 7m ago; DooPlex→LAN (192.168.0.87, 192.168.0.162) down; demo-hp-lan via ProxyJump felhom-pve timeout; ep0→WireGuard (ssh root@167.233.158.164 → 10.77.0.3, 10.77.0.2) ping down and ports 22 and 8822 both closed, with ep0's own wg show putting the last handshake at ~5 minutes prior. Cause confirmed by the operator: house internet. demo-hp's uptime afterwards read up 6 days — the box never rebooted, so the 124 MB template download running at the time did not destabilise it.
• Outage 2, from ~11:27 CEST. Interrupted the run after all of Q1/Q2/Q3/Q6/Q7 and most of Q4 were captured, and before the Q4 hang measurement (§6.3) and teardown (§10).

---

3. Q1 — reproducing it deliberately

Reproduced 3/3 attempts. Two device classes were used deliberately, because they fail differently and the difference turned out to matter.

3.1 Topology, faithful to the intermediary model

Built by hand to match intermediary.go:11-19 and :26-42:

• scratch shared parent /mnt/r117-drives — self-bind, --make-private, --make-shared, the verbatim logic of sharedParentScript. Deliberately NOT the live /mnt/felhom-drives, whose peer group belongs to guest 9201.
• the guest's permanent parent bind, set before first start: pct set 9301 -mp0 /mnt/r117-drives,mp=/mnt/r117-drives.
• per drive: raw host mount at /mnt/<name>, namespace <raw>/felhom-data chowned 100000:100000, bound to /mnt/r117-drives/<name>.
• raw mounts keyed by fs-UUID, matching internal/storage/mountunit.go:51 (What=/dev/disk/by-uuid/<UUID>) — the detail that makes the host side heal onto a new device node.

Propagation verified correct before any test — the guest's parent bind is a slave of the host's shared peer group:

guest: 1176 1175 252:1 /mnt/r117-drives /mnt/r117-drives rw,relatime shared:428 master:196 - ext4 ...
                                                                                 ^^^^^^^^^^

3.2 The two drives

	Drive A	Drive B
Stack	file → loop0 → dm-linear	scsi_debug (dev_size_mb=64)
Appears as	/dev/mapper/r117cel, 252:11	/dev/sdb, 8:16
Death mechanism	swap the dm table to an error target	echo 1 > /sys/block/sdb/device/delete — real gendisk removal while mounted
Models	a device that errors in place	a device that is pulled

3.3 The cycle, on drive B — the headline reproduction

HEALTHY        host raw   /mnt/r117sd        -> /dev/sdb  8:16  rw,stripe=512
               host bind  /mnt/r117-drives/sd -> /dev/sdb  8:16  rw,stripe=512
               guest bind /mnt/r117-drives/sd -> /dev/sdb  8:16  rw,stripe=512   master:450

PULL           echo 1 > /sys/block/sdb/device/delete
               /dev/sdb            -> No such file or directory
               mnt-r117sd.mount    -> inactive          <-- systemd stopped it ITSELF (BindsTo=)

ABSENT         host raw   /mnt/r117sd         -> NOT A MOUNTPOINT
               host bind  /mnt/r117-drives/sd -> /dev/sdb 8:16 rw,relatime,stripe=512,shutdown
               ==> BoundUnderParent = FALSE   (HALF1 true, HALF2 false)   <-- R-113 works here

RETURN         echo '- - -' > /sys/class/scsi_host/host1/scan
               returned as /dev/sdc  devno=8:32
               by-uuid/71e155fc-... -> /dev/sdc          <-- the UUID-keyed unit heals onto the new node

R-117          host raw   /mnt/r117sd         -> /dev/sdc  8:32  rw,stripe=512          [HEALTHY]
               host bind  /mnt/r117-drives/sd -> /dev/sdb  8:16  ...,shutdown           [DEAD]
               guest bind /mnt/r117-drives/sd -> /dev/sdb  8:16  ...,shutdown           [DEAD]
               ==> BoundUnderParent = TRUE

This is byte-for-byte the state reported three times as cosmetic — a drive back on sdc while findmnt still reads sdb[/felhom-data], with shutdown in the options.

3.4 The node change is a CONSEQUENCE, not a precondition — control test

The brief asks whether the defect requires sdb→sdc. It does not require it; it causes it.

with the stale bind still held:
  8:16 in /proc/partitions        -> 0        (the gendisk is gone)
  8:16 in /proc/self/mountinfo    -> 1        (the stale bind still references its superblock)
  device returns as               -> /dev/sdc  8:32      <-- NEW devno

after releasing the stale bind (umount), same pull + rescan:
  device returns as               -> /dev/sdb  8:16      <-- letter REUSED

The stale bind pins the dead superblock, which keeps the old device index allocated, which forces the returning device onto a different number. The same effect appeared independently on drive A: dmsetup remove --force r117cel failed with Device or resource busy — held open by the stale bind alone.

Consequence for any fix: a device-number comparison is not a heuristic for the return case, it is sound. Whenever a stale bind exists, the numbers differ because it exists. Stated as measured here, twice, on two device classes — not as a kernel guarantee quoted from documentation.

Reliability: 3/3. Two full detach/return cycles (drives A and B) and one in-place failure (§9), each producing BoundUnderParent = true over a non-functioning namespace on the first attempt. No flakiness observed; nothing was retried.

---

4. Q2 — what exactly is dead, and at which layer

Dead: the bind. The host bind and the guest bind are the same mount in one peer group, so they die together and identically. Alive and correct: the raw host mount, re-mounted by fs-UUID on the new device. The two disagree, and nothing compares them.

There are two death states, and they present differently. This matters more than it looks: a probe tuned to one misses the other.

	(a) device removed (drive B / real pull)	(b) device errors in place (drive A)
super options gain	shutdown	emergency_ro
guest statfs	OK	OK
guest listdir	OK	OK
guest read file content	errno=5 EIO	OK (page cache)
guest write	errno=5 EIO	errno=30 EROFS
host read via bind	errno=5 EIO	OK (page cache)
host write via bind	errno=5 EIO	errno=30 EROFS
host raw mount	not a mountpoint → then healthy on the new node	stays mounted throughout
BoundUnderParent	false while absent → true after return	true the entire time

Errnos are numeric, from os.statvfs / os.listdir / open inside the guest via pct exec python3 and on the host, in the same capture.

Two consequences worth stating plainly:

1. statfs and listdir succeed on a filesystem that returns EIO for every byte of content. They are answered from the in-memory superblock and the dentry cache. Any liveness probe built on either is hollow by construction — it reports healthy in exactly the state it exists to detect.
2. The host-side raw mount is genuinely fine. The device is present, readable and writable at /mnt/<name>. Nothing is wrong with the drive. The damage is confined to the namespace the customer's apps actually use.

4.1 The brief's "materially worse" case — answered: no

The brief asks whether the host mount points at a device node now belonging to a different device. It does not. In every state captured, the raw mount resolved through /dev/disk/by-uuid/<UUID> to the correct returning device (by-uuid/71e155fc-... -> /dev/sdc), because the unit is keyed by filesystem UUID (mountunit.go:51) and not by node. The stale reference is held by the bind, which names the old device — and it points at a dead superblock, not at a live foreign one. No cross-device mis-identification was observed or is structurally possible on this path.

---

5. Q3 — why boundUnderParent returns true

Because neither half of the conjunction looks at which device the mount is on. Both are presence tests on a path string.

5.1 The conjunction

disks.go:222 (Observe path) and disks.go:358 (registry path), identically:

di.BoundUnderParent = s.boundUnderParent(r.Context(), vmid, gp) &&
    s.devicePresent(t.MountPath)

5.2 Half 1 — GuestSeesMount

disks.go:949 → boundUnderParent → s.guestAttach.GuestSeesMount(ctx, vmid, stablePath) → intermediary.go:276:

data, err := os.ReadFile("/proc/" + pid + "/mountinfo")
...
for _, line := range strings.Split(string(data), "\n") {
    f := strings.Fields(line)
    if len(f) >= 5 && f[4] == path {
        return true
    }
}

Half 2 — devicePresent (disks.go:972) → isHostMountpoint (intermediary.go:394) — is the same test against /proc/self/mountinfo for the raw path:

fields := strings.Fields(sc.Text())
if len(fields) >= 5 && fields[4] == path {
    return true
}

f[4] is the mount point. f[2] — the major:minor of the backing device — is never read by either function. The information that settles the question is present in the very lines both functions already parse, one field away, and is discarded.

Measured in the R-117 state, drive B:

HALF1  GuestSeesMount(9301, /mnt/r117-drives/sd) = true    [4.643178ms]   <- REAL agent code
HALF2  isHostMountpoint(/mnt/r117sd)             = true    [261.419µs]
=====> BoundUnderParent (disks.go:222 conjunction) = true

host  raw   : 814 33 8:32 /            /mnt/r117sd         ... - ext4 /dev/sdc rw,stripe=512
host  bind  : 755 118 8:16 /felhom-data /mnt/r117-drives/sd ... - ext4 /dev/sdb rw,stripe=512,shutdown
guest bind  : 759 1176 8:16 /felhom-data /mnt/r117-drives/sd ... - ext4 /dev/sdb rw,stripe=512,shutdown
                    ^^^^

Half 1 is satisfied by the stale entry; half 2 by the new device. Exactly as OPEN-ITEMS.md predicted: neither half compares the two. Confirmed, not inferred.

5.3 This is a gap, not a false invariant — and the doc comment is the reason it survived

disks.go:155-165 claims only the two cases the R-113 conjunction does fix (the boot-ordering window, and a bind outliving its device while still absent). It does not claim staleness after a return, so per the workspace rule this is a genuine gap rather than a seventh shipped false guarantee.

But the summary line is a wish:

// BoundUnderParent reports whether the drive is live + usable in the guest in the intermediary model.

Measured: it reports neither. In the R-117 state the drive is not live in the guest and not usable — every content read and every write returns EIO — and the field reads true. The interface comment at disks.go:154-155 makes the same promise ("the guest-usable signal — distinct from the host having the bind"), and disks.go:83 a third time. Three statements of a property nothing tests. Per the standing rule, whichever fix lands owes a test that pins the consequence — that a dead namespace reports BoundUnderParent = false — and not merely the mechanism.

---

6. Q4 — what a reliable check costs, and how it fails

All timings from the R-117 state on drive B, on the venue, one run each, -timeout 3s.

6.1 Measured

Probe	What it does	R-117 state (B)	in-place state (A)	healthy	Cost	Verdict
P1 devno compare	field 3 of the bind's mountinfo line vs field 3 of the raw mount's — pure /proc, host-side only	STALE=true ✅	STALE=false ❌	false ✅	0.38–0.45 ms	Sound for the return case (§3.4). Blind to in-place death
P2 ext4 abort options	shutdown or emergency_ro in the super options — pure /proc	DEAD=true ✅	DEAD=true ✅	false ✅	0.17–0.24 ms	Catches both. ext4-specific
P3 statfs	one syscall	OK — MISSES IT ❌	OK — misses ❌	OK	0.042 ms	Worthless
P4 getdents	read one dir entry	OK — MISSES IT ❌	OK — misses ❌	OK	0.045–0.061 ms	Worthless
P5 write + unlink	create, write 1 byte, remove	EIO(5) ✅	EROFS(30) ✅	OK	0.042–0.152 ms	Catches both, but writes, cannot distinguish a legitimately-ro mount from a dead one, and BLOCKS on a wedged device (§6.3)
P6 write + fsync	forces the byte to the device	—	—	OK, 11.4 ms	11.4 ms healthy	BLOCKS on a wedged device. 25–70× the /proc probes even when healthy, on a loop device with no seek cost
P7 read with page cache dropped (POSIX_FADV_DONTNEED)	forces a read to reach the device	—	—	OK, 0.18 ms	0.18 ms	BLOCKS on a wedged device

GuestSeesMount itself costs 3.85–5.03 ms, an order of magnitude more than any probe above, because it forks lxc-info. Any of P1/P2 is free relative to the check it would strengthen.

6.2 The failure modes the brief named, answered from measurement

• "A write probe fails wrongly on a legitimately read-only mount." Confirmed as a real hazard, and worse than stated: in the in-place death state the mount is read-only (emergency_ro) and P5's EROFS is indistinguishable from an intentionally ro mount by errno alone. P5 cannot be the sole signal.
• "A read probe on an empty drive has nothing to read." Real, and it has a second and larger problem: a read probe passes from page cache over a dead filesystem (P4 succeeded in every dead state). Honest read probing requires dropping the cache first (P7), which converts it into a probe that must reach the device — i.e. one that can block.
• "Spinning up a sleeping disk on every poll has a real hardware cost." P1 and P2 eliminate this concern entirely: they read /proc and issue no block I/O, so a spun-down drive stays spun down. This is the strongest argument for the pure-/proc pair over anything else, independent of the hang question.

6.3 The hang case — MEASURED, and it is worse than "a probe that hangs"

Venue: the dm-linear drive suspended with dmsetup suspend --noflush --nolockfs, which queues all I/O indefinitely instead of returning EIO — the controlled "half-dead or spun-down device" the brief asks about. Verified suspended (dmsetup info -o attr → L-sw, the s), then the full probe set with -timeout 3s, then dmsetup resume.

=== PROBES against a WEDGED (not erroring) device, timeout 3s ===
HALF1  GuestSeesMount = true    [3.983519ms]
HALF2  isHostMountpoint = true  [234.181µs]
=====> BoundUnderParent = true
P1 devno-compare   raw="252:12" bind="252:12" -> STALE=false  [363.878µs]   <-- COMPLETED
P2 ext4-abort-opts superopts="rw" -> DEAD=false               [206.104µs]   <-- COMPLETED
P3 statfs        COMPLETED in 40µs   err=<nil>                              <-- reports HEALTHY
P4 getdents      COMPLETED in 64µs   err=<nil>                              <-- reports HEALTHY
P6 write+fsync   TIMED OUT after 3.002s — caller proceeds; goroutine still blocked
P6                 ...still blocked 8s in; threads 5 -> 5 (LEAKED — the syscall is uninterruptible)
P7 read-uncached TIMED OUT after 3.002s — caller proceeds; goroutine still blocked
P7                 ...still blocked 8s in; threads 5 -> 5 (LEAKED)
P5 write+unlink  TIMED OUT after 3.002s — caller proceeds; goroutine still blocked
P5                 ...still blocked 8s in; threads 5 -> 6 (LEAKED)

Four results, in ascending order of severity.

1. P1 and P2 completed normally, in microseconds. They read /proc/self/mountinfo, which involves no block device. This is the whole argument for the §11 recommendation: on a wedged device the pure-/proc check is not merely faster, it is the only one that returns at all.
2. P3 and P4 completed and reported healthy. On a wedged device they do not even have the decency to hang — they lie, exactly as they do on a dead one (§6.1). Confirms R-117b in a second failure mode.
3. Every probe that touches the device blocked — including P5. P5 is a buffered write with no fsync, which was expected to complete out of page cache. It did not: the O_CREAT metadata path needs journal access (wchan=do_get_write_access). There is no such thing as a cheap write probe that is safe on a wedged device.
4. The process could not be killed — not by SIGTERM, not by SIGKILL. This is the finding that changes the recommendation from a preference into a fence:

# after `timeout 200` had already sent SIGTERM:
    PID STAT WCHAN                        ELAPSED COMMAND
2493045 Dl   folio_wait_bit_common          03:20 r117probe
  tid 2493045 stat=D wchan=folio_wait_bit_common
  tid 2493049 stat=D wchan=do_get_write_access

# explicit SIGKILL:
kill -9 2493045
after SIGKILL: 2493045 Dl  folio_wait_bit_common  03:50      <-- STILL ALIVE

# only the device recovering released it:
dmsetup resume r117cel2  -> probe still alive? no — it died once I/O completed

A timeout protects the caller's control flow and nothing else. The goroutine and its OS thread are consumed permanently, and the process enters uninterruptible sleep. Applied to the agent, whose reconcile ticker runs every 20 s (main.go:973):

• the report loop's caller proceeds, so the timeout "works" — and the evidence of the problem is a thread you cannot see and cannot reclaim;
• systemctl restart felhom-agent would hang, because SIGKILL does not reach a D-state thread. The agent becomes unrecoverable until the device recovers or the host reboots — on a box whose whole purpose is to be recovered remotely;
• the thread count is not a reliable tell (5→5, 5→5, 5→6): the Go runtime reuses threads, so the leak does not announce itself in the one metric an operator might watch.

This disqualifies I/O probing on this path outright. Not "use a timeout", not "use it carefully" — a liveness probe that issues block I/O converts a dead drive into an unkillable agent. §11's recommendation issues none.

---

7. Q5 — where the check belongs

The agent. Not a balance of arguments — the controller is structurally incapable of it:

1. The inputs are host mount tables. P1 needs the host's raw-mount entry for /mnt/<name>; P2 needs the super options of the host bind. The controller runs inside the guest and can see neither. The guest's own mountinfo carries the stale entry (with shutdown, so P2 is available in-guest) but has no entry at all for the raw host mount, so P1 is unavailable to it in principle.
2. Both inputs are already in the agent's hand, in the same function. disks.go:222 already calls boundUnderParent(...) and devicePresent(t.MountPath) with both paths in scope. The devno comparison adds no new data source and no new dependency.
3. The agent owns drive identity (durable_id, enrolment, the fs-UUID-keyed mount units), so a drive-liveness verdict belongs to the same owner under one state, one owner.
4. The field already exists and is already documented to mean exactly this (§5.3). The controller consumes it and keys the gate on it (controller/.../intermediary.go:226). Fixing the producer fixes every consumer at once; adding a second, in-guest liveness notion would create the competing definition the hub's liveness-threshold rule exists to forbid.

Ruled on evidence, and the evidence points the same way as the principle — but note it is (1) that is decisive, not (3).

---

8. Q6 — in-place recovery, and the cheapest fix available

8.1 Recovery works, live, without stopping the guest — proven

The repair is AttachDrive's normalize leg (intermediary.go:238-247) verbatim: umount every layer at the stable path, then one fresh mount --bind, which re-fires propagation into the guest's namespace.

guest init pid BEFORE: 2465515
  umount /mnt/r117-drives/sd ... ; mount --bind /mnt/r117sd/felhom-data /mnt/r117-drives/sd
guest init pid AFTER:  2465515      <-- unchanged: the guest was never restarted

Verified after the repair:

host raw   /mnt/r117cel        -> /dev/mapper/r117cel2  252:12  rw,relatime
host bind  /mnt/r117-drives/cel -> /dev/mapper/r117cel2 252:12  rw,relatime
guest bind (mount id 744, NEW) -> 252:12 ... master:302
GUEST statfs/listdir/read-canary/write  ->  all OK
P1 devno-compare  raw="252:12" bind="252:12"  ->  STALE=false

Two mount syscalls, no guest downtime, no app restart, full recovery. So a fix can self-heal; it does not have to settle for alarming.

8.2 The repair is already wired — three times — and defeated by one line

Call site	What it does	Why it does nothing today
cmd/felhom-agent/main.go:973-985	20-second ticker → ReassertEnrolledMounts then ReassertGuestBinds	→ disks.go:1108 → AttachDrive → short-circuits
disks.go:951	the same, at agent startup	same
controller/internal/web/intermediary.go:301-305	the Return branch calls agent.GuestAttach(ctx, a.Raw) — before restartStacks at :311	→ disks.go:624 → AttachDrive → short-circuits

All three land on intermediary.go:234-236:

n := countHostMounts(stable)
if n == 1 && b.GuestSeesMount(ctx, vmid, stable) {
    return stable, nil // exactly one bind + guest-visible → fully live, no-op
}

In the R-117 state n == 1 and GuestSeesMount is true, so AttachDrive declares the namespace "fully live" and returns without touching it. The comment on that line is the sixth entry in the workspace's false-invariant table in all but name: "fully live" is precisely what it is not.

The ordering is already correct. The controller asks the agent to re-attach at :301-305 and only then restarts the apps at :311. If AttachDrive repaired the bind instead of short-circuiting, the apps would be restarted onto a repaired namespace by the code that exists today. The defect is one predicate, not a missing recovery path.

---

9. Q7 — scope, and the half a Return-branch fix cannot reach

A bind can die in steady state, with no detach/return cycle at all — measured, not inferred.

Drive A, device swapped to an error target with everything running and nothing detached:

write into the bind from the guest      ->  Input/output error   (immediately)
mnt-r117cel.mount                      ->  active               <-- still mounted
  BindsTo="dev-dm\x2d11.device"                                     the DEVICE never disappeared
host raw   /mnt/r117cel                ->  /dev/mapper/r117cel  252:11  rw,relatime
host bind  /mnt/r117-drives/cel        ->  /dev/mapper/r117cel  252:11  rw,relatime,emergency_ro
HALF1 GuestSeesMount = true    HALF2 isHostMountpoint = true
=====> BoundUnderParent = TRUE
GUEST write -> errno=30 Read-only file system

Nothing transitions. State stays attached, BoundUnderParent stays true, the drive is never Disconnected, so planDriveGates produces neither a Stop action nor a Return action (controller/.../intermediary.go:254-259 — both arms require sp.Disconnected to differ from present[sp.Path]). No gate log line, no event, no email, no banner. The customer's apps write into a read-only namespace and every signal reads healthy indefinitely.

Reached in this run by a device that errors without disappearing. Real-world equivalents: a SATA/USB link reset that recovers the link after ext4 has already aborted, a drive that develops write errors, an enclosure that browns out briefly. Not exotic.

Two consequences for the eventual fix:

1. A fix that lives in the Return branch cannot close this. The branch never executes.
2. P1 does not detect it either — the device number is unchanged (252:11 on both sides), because the device never left. Only P2 caught it (emergency_ro). This is the reason §8's recommendation is P1 ∨ P2 and not P1 alone; a fix built on the device-number comparison that the return case makes so clean would ship with the silent half still open.

Probed deliberately, per the brief's instruction not to infer it from an absence of reports.

---

10. The reproduction recipe

Exact enough to trigger on demand. scsi_debug is the load-bearing choice — it is the only mechanism tried that can remove a gendisk while it is mounted, which is what a USB pull does and what dm and loop cannot do (both are held open by the stale bind: dmsetup remove --force → Device or resource busy).

# --- venue: any Proxmox host + a scratch LXC. Shared parent MUST exist before the guest starts. ---
PARENT=/mnt/r117-drives; RAW=/mnt/r117sd; VMID=9301
mkdir -p $PARENT
mountpoint -q $PARENT || { mount --bind $PARENT $PARENT
                           mount --make-private $PARENT; mount --make-shared $PARENT; }
pct set $VMID -mp0 $PARENT,mp=$PARENT     # then start the guest; verify `master:` in its mountinfo
pct start $VMID

# --- a drive whose device node can be removed while mounted ---
modprobe scsi_debug dev_size_mb=64 max_luns=1 num_tgts=1
DEV=/dev/sdb                                     # whichever /sys/block/sd*/device/model is scsi_debug
mkfs.ext4 -q -L R117SD $DEV
UUID=$(blkid -s UUID -o value $DEV)
mkdir -p $RAW
systemd-mount --no-block --collect -t ext4 /dev/disk/by-uuid/$UUID $RAW   # UUID-keyed, per mountunit.go:51
mkdir -p $RAW/felhom-data && chown 100000:100000 $RAW/felhom-data
echo canary > $RAW/felhom-data/.r117-canary && sync
mkdir -p $PARENT/sd && mount --bind $RAW/felhom-data $PARENT/sd
# the guest sees it immediately via propagation — no pct, no reboot

# --- 1. PULL: real device removal. systemd stops the raw mount unit by itself (BindsTo=). ---
echo 1 > /sys/block/sdb/device/delete
#     -> bind survives with `shutdown`; raw path is NOT a mountpoint; BoundUnderParent = false

# --- 2. RETURN: rescan. It comes back on a DIFFERENT node because the stale bind pins the old one. ---
echo '- - -' > /sys/class/scsi_host/host1/scan          # -> /dev/sdc, 8:32
systemd-mount --no-block --collect -t ext4 /dev/disk/by-uuid/$UUID $RAW
#     -> R-117: raw on 8:32 healthy, bind on 8:16 dead, BoundUnderParent = TRUE, EIO both directions

# --- Q7 variant: steady-state death, no cycle. Needs a device that ERRORS without disappearing. ---
#     dm-linear over a loop file, then swap the table to `error` and resume:
dmsetup load <name> --table "0 <sectors> error" && dmsetup resume <name>
#     -> raw mount stays ACTIVE, BoundUnderParent stays TRUE, writes EROFS, superopts gain emergency_ro

# --- repair (Q6), no guest restart: AttachDrive's normalize leg, verbatim ---
umount $PARENT/sd && mount --bind $RAW/felhom-data $PARENT/sd

# --- control test for §3.4: with NO stale bind holding it, the letter is REUSED ---
umount $PARENT/sd; echo 1 > /sys/block/sdc/device/delete
echo '- - -' > /sys/class/scsi_host/host1/scan           # -> /dev/sdb, 8:16 again

Traps, each paid for in this run:

• Establish the shared parent before the guest starts, or the guest's parent bind is private and no drive swap propagates. Verify by finding master:<n> on the guest's parent entry.
• The parent bind is non-recursive, so a drive already bound under the parent when the guest starts is invisible to it. Re-fire with umount + mount --bind. Observed directly at guest first start.
• Do not reuse the live /mnt/felhom-drives: a bind under it propagates into the real customer guest, and the agent's reconcile will act on it.
• statfs and ls succeed on the dead bind. Verify death by reading file content or writing.

---

11. Recommendation, and its cost

Framed as a constraint on the fix, not as an implementation. No code was written.

Make the liveness signal compare the two mountinfo entries it already reads, and consult the filesystem's own abort state. Do not add block I/O.

live(stable, raw) :=  bindEntry(stable) exists
                  AND rawEntry(raw)     exists            <-- devicePresent today
                  AND bindEntry.devno == rawEntry.devno    <-- P1: closes the return case (§3.4)
                  AND bindEntry not aborted                <-- P2: closes the steady-state case (§9)

Cost: two /proc/self/mountinfo reads that isHostMountpoint and countHostMounts already perform on every pass, plus two field comparisons. Measured 0.17–0.45 ms, against the 3.85–5.03 ms GuestSeesMount already spends forking lxc-info on the same call. No block I/O, so no disk spin-up, no write, and no possibility of blocking the report loop. Effectively free.

Constraints the fix must respect, each earned above:

1. Both halves are needed. P1 alone leaves Q7 silent; P2 alone leaves any non-ext4 filesystem uncovered. Neither alone is sufficient, and the P1-only fix is the tempting one because the return case is what R-117 was filed as.
2. Keep the "cannot tell → never absent" rule (disks.go:983-986). An unreadable /proc, a missing raw entry with an empty path, or an unrecognised filesystem must degrade to unknown and be treated as present — reporting absent stops a customer's apps. The (bool) return shape that made this impossible for newestArchiveOn (workspace table, row 2) is the trap to avoid: live needs a third state, or an explicit unknown-is-present convention pinned by a test.
3. P2's token list is filesystem-specific. shutdown and emergency_ro are ext4. Both were measured here; both must be in the list, because a probe checking only shutdown passes the entire Q7 state. For any other filesystem the honest answer is unknown (see 2), not live.
4. Fixing the predicate is the whole fix, and it self-heals. §8.2: three call sites already invoke the repair, including the controller's Return branch before it restarts apps, and the 20-second ticker. An honest predicate makes AttachDrive stop short-circuiting, and the bind is repaired within ~20 s with no guest restart. Do not add a new recovery path; unblock the existing one.
5. The test must assert the consequence, not the mechanism. Per the standing rule and R-97b's Scenario F: assert that a dead namespace yields BoundUnderParent = false (and that the drive-gate alarm fires), not merely that a devno comparison returns what it was fed. §5.3 lists three comments promising a property nothing currently tests; the fix owes one that pins it.
6. Q7 needs its own scenario. A test that only exercises detach→return will pass while the silent half stays broken. The in-place recipe in §10 is the second scenario.
7. Do not add an I/O confirmation probe. Measured, not preferred (§6.3). On a wedged device every probe that touches it blocks — including a buffered write with no fsync — and the process then cannot be killed by SIGKILL, so the agent could not be restarted until the device recovered or the host rebooted. A timeout protects the caller's control flow and nothing else. P1 and P2 both returned in microseconds on that same wedged device; the recommendation issues no block I/O, and that is now a fence rather than a preference.

---

12. What this changes about R-116's closure

R-116's verdict stands. Its evidence was narrower than it reads, and one caveat needs re-labelling.

• The four-event sequence is unaffected. R-116 was about which event fires for which drive — the pairing of backup_target_absent/backup_target_restored and their discrimination from the generic pair. That was proven on the wire, and every input to it (isTarget, the guest-path dedupe, the row join) is configuration-derived — storage.cfg's path versus the .mount unit's Where — which R-117 does not touch. Nothing in this spike disturbs it.
• R116-v0116-2026-07-30.md §5 already recorded the interference and drew the right line ("a backup_target_restored there proves pairing, not recovery"). This run confirms that caveat from the layer beneath: on the reattach leg the namespace really was dead, EIO both directions, so the run's bound_under_parent = true was a correct answer to the pairing question and a false answer to the recovery question.
• One thing does need re-labelling. R-116's over-correction check is described in OPEN-ITEMS.md as "PASSES with a positive observable — 0 ABSENT lines / 0 drive events over 2m14s with both drives present, target degraded:false, while 2 RETURNED lines prove the gate was ticking." The two RETURNED lines do prove the gate was ticking, so the check is not an absent-log verdict and rule 3 is satisfied. But degraded:false over that window was read off a drive whose bind was dead — so it evidences "the gate did not over-fire" and not "the drive was healthy". Narrower than it reads, and still true as written. Recommend the R-116 row gain one clause: the over-correction window proves no spurious ABSENT, not drive health.
• R-117 outranks R-116 in customer impact, and §9 raises it further: the silent steady-state half has no event on any channel at all, where R-116's worst case sent the wrong message.

---

13. Findings filed — none fixed

#	Finding	Severity	Evidence
R-117	(existing row, now with a mechanism and a recipe) The bind-liveness signal is a path-presence test; the R-117 state is BoundUnderParent = true over an EIO namespace	HIGH	§3, §5
R-117a	A drive's bind can die in STEADY STATE, with no detach/return cycle, and no event is emitted on any channel. The gate produces neither action, so a Return-branch fix cannot reach it. P1 does not detect it; only the filesystem's abort flag does	HIGH — larger than R-117 as filed	§9
R-117b	statfs and getdents both SUCCEED on a namespace returning EIO for all content — and also on a wedged one, where they do not even hang. Any liveness probe built on either is hollow by construction — it reports healthy in exactly the state it exists to detect	MEDIUM (design trap)	§4, §6.1, §6.3
R-117f	A liveness probe that issues block I/O turns a wedged drive into an UNKILLABLE AGENT. On a suspended device every I/O probe blocked — including a buffered write with no fsync — and the process survived SIGTERM and SIGKILL, dying only when the device recovered. systemctl restart felhom-agent would therefore hang, leaving the agent unrecoverable until the device returns or the host reboots. A timeout protects the caller's control flow and nothing else, and the thread count does not reveal the leak (5→5, 5→6)	HIGH (constraint on the fix — it disqualifies an entire probe class)	§6.3
R-117c	Three comments (disks.go:83, :154-155, :155-165) state that BoundUnderParent means "live + usable in the guest". Nothing tests it, and it is false in the measured state. Not counted as a shipped false invariant because :158-165 scopes its own claim narrowly — but the summary lines do not	MEDIUM	§5.3
R-117d	AttachDrive logs and returns "fully live" for a namespace that is EIO on every call (intermediary.go:235), disabling all three wired repair paths including the controller's pre-restart re-attach	HIGH — this is the self-heal that already exists	§8.2
R-117e	The two demo boxes share one uplink and one failure domain. Both went unreachable together, twice in one session, on all four known routes including the WireGuard OOB path via ep0. There is no route to either box that survives the site losing internet — the "out-of-band" path is not out-of-band with respect to this failure	LOW (operational, not a code defect)	§2.5

Observed, out of scope, not acted on: demo-hp's live agent is 0.113.0 against main's 0.116.0 (§1). It equals the manifest's declared MinAgent, so this is not R-115/R-111 drift — but it does mean the fleet's designated drill host does not run the R-113 conjunction whose successor this spike is about. Whoever fixes R-117 will need that box on a current agent to validate live.

---

14. Teardown — DONE, all three layers

Run after the venue came back; the site outage delayed it, it did not skip it. Ordering note, recorded because it is a real trap: any suspended dm device must be dmsetup resumed before any umount, or every unmount in the teardown blocks on the wedged device — the same uninterruptible-sleep mechanism as §6.3. Both devices were verified attr=L--w (not suspended) first.

14.1 Layer 1 — the machine

BEFORE   pct list -> 9201 running demo-hp / 9301 running r117
         pct stop 9301 ; pct destroy 9301 --purge   ("purging CT 9301 from related configurations..")
AFTER    pct list -> 9201 running demo-hp          <-- only the pre-existing guest

14.2 Layer 2 — the host, with the space actually returned

	Before	After
pvesm status	r117scratch active, local 38.97 %	r117scratch removed, local 37.02 % (session start: 37.00 %)
df -h /var/lib/vz	16G used, 42 %	15G used, 40 % (session start: 40 %)
dmsetup ls	r117cel (252:11), r117cel2 (252:12) + 11 pre-existing	both gone; the 11 pre-existing pve-* unchanged
losetup -a	/dev/loop0 (cel.img), /dev/loop1 (9301 rootfs)	empty — no loop devices
scsi_debug	loaded (1)	unloaded (0); /dev/sd* back to sda1..3 only
r117 mounts	raw + bind + scratch shared parent	findmnt | grep r117 → (none)
/mnt	+ r117-drives, r117cel, r117sd	felhom-drives, nvme-1tb, vzsnap0 — exactly as found
/root	7 scratch files + r117/	ls /root | grep -i r117 → (none)

The transient mnt-r117cel.mount / mnt-r117sd.mount units died with the unmounts (Unit mnt-r117sd.mount not loaded — created by systemd-mount, so nothing was left in /etc). The 124 MB debian-13-standard template was also removed.

14.3 Layer 3 — the hub: nothing to dispose of

This run created no hub-side record. No customer, no appliance, no pairing, no claim: guest 9301 ran no controller and had no network interface at all, and no agent was enrolled against it. Stated explicitly per §13 of the template — "teardown clean" without layer 3 is the sess-c failure. Here layer 3 is genuinely empty, which is a different claim from silence.

14.4 Fences — re-verified after teardown, not assumed

Fence	State
demo-hp guest 9201	running — before, during and after; never a probe target
demo-hp drill-r50 (VM 300)	stopped — never started, never touched
demo-hp local-lvm	38.84 % — byte-identical before and after, and to the session-start reading
demo-hp felhom-backup	active, 0.46 %, content backup unchanged — not re-targeted, not written to
live shared parent /mnt/felhom-drives	intact with both submounts (nvme-1tb on /dev/nvme0n1[/felhom-data], Felhom-Share autofs) — a separate scratch parent was used throughout
felhom-agent service on demo-hp	active
demo-felhom, Peti's cluster	never contacted
ep0	read-only — one ssh, wg show, and TCP reachability probes toward 10.77.0.2/.3 during outage 1. Nothing written
Code	no .go file in either repo modified; nothing built for deployment, nothing published, no version bumped
Repos	the probe lives outside both, in the session scratchpad; the agent tree copy verified byte-identical to d4eb259

---

15. What did not happen, and why

• Everything the brief asked for was measured. The two items the first internet outage deferred — Q4's hang case and teardown — were completed once the site came back: §6.3 and §14 are measurements, not plans. The thread-leak hypothesis P6/P7 were built to test turned out to be the weaker half of the real result: the leak is invisible in the thread count (5→5, 5→6), while the process being unkillable by SIGKILL is unambiguous.
• No load or duration testing of the recommended check. P1/P2 were measured as single calls (0.16–0.45 ms), not across a sustained reconcile loop on a box with many drives. Nothing suggests a problem — they are two /proc reads the code already performs — but it was not measured.
• No fresh /disks payload and no fresh controller gate log — deliberate, and the same call DIAG-r116 §14 made: both were already captured verbatim on real hardware (twice and four times respectively), and running a controller here would have meant a hub customer, a pairing, a golden fetch and a claim — the work that consumed three prior sessions — to re-measure a known quantity. The missing layer was the mount tables and the errnos beneath that payload, which is what this run supplies. §5.2's capture is the join between the two.
• The same-device-node return was not reproduced as a defect case — because §3.4 showed it cannot occur while a stale bind exists: the stale bind is what forces the new number. The control test (letter reused once the bind is released) is the positive observable for that claim.
• R-117 was not fixed, per the fence — not even the one-line predicate change §8.2 identifies. The design constraints are in §11 for a spec to accept or reject.
• felhotest was not tried; Connection refused was already recorded 2026-07-30 and it is irrelevant to this mechanism.
• Nothing was committed to felhom-agent or felhom-controller — this run's only writes are this document and the R-117 row in backlog/OPEN-ITEMS.md.

---

Every claim above is either quoted from pushed Gitea source at file:line (agent d4eb259 / controller b331f18) or is captured probe output from the venue. Where a result was inferred rather than measured, it says so.