admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs(R-117): the hang case measured — an I/O probe turns a wedged drive into an unkillable agent

Browse Source 
Completes the spike once the venue came back. Q4's hang case and teardown are
now measurements, not plans.

Against a dmsetup-suspended device (I/O queues instead of returning EIO):

- P1 (devno compare) and P2 (ext4 abort flags) completed in 364us / 206us.
  They read /proc, so no block device is involved.
- statfs and getdents completed and reported HEALTHY — on a wedged device they
  do not even hang. R-117b confirmed in a second failure mode.
- EVERY probe that touches the device blocked, including a buffered write with
  no fsync: the O_CREAT metadata path needs journal access
  (wchan=do_get_write_access). There is no cheap-and-safe write probe.
- The blocked process survived SIGTERM AND SIGKILL (stat=D,
  wchan=folio_wait_bit_common, still alive 3m50s after kill -9) and died only
  when the device was resumed. So `systemctl restart felhom-agent` would hang,
  leaving the agent unrecoverable until the device returns or the host reboots.
  The thread count does not reveal the leak (5->5, 5->6).

Filed as R-117f. A timeout protects the caller's control flow and nothing else,
so "the fix must issue no block I/O" is now a fence rather than a preference —
the thread-leak hypothesis the probes were built to test turned out to be the
weaker half of the result.

Teardown done, all three layers: guest 9301 destroyed, r117scratch removed, both
dm and both loop devices gone, scsi_debug unloaded, local back to 37.02% against
a 37.00% session start. Fences re-verified AFTER teardown: 9201 running,
drill-r50 stopped, local-lvm 38.84% byte-identical, felhom-backup content
unchanged, live /mnt/felhom-drives intact with both submounts, agent active.
Layer 3 genuinely empty — 9301 had no NIC and ran no controller.

Trap recorded: a suspended dm device must be resumed BEFORE any umount, or the
teardown blocks on the same uninterruptible sleep.
This commit is contained in:
authentik Default Admin
2026-07-30 11:49:59 +02:00
parent c949389c95
commit e70b5feebe
3 changed files with 161 additions and 77 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md
+121 -59
View File
@@ -12,7 +12,7 @@ Output is this document. Findings are filed as `R-n`, none fixed.
| Q1 | Reproduce deliberately | **YES — 3/3 attempts, two independent device classes.** A device-node change is **NOT a precondition — it is a CONSEQUENCE** of the stale bind, proven by a control test |
| Q2 | What is dead, at which layer | The **guest bind and the host bind** (same mount, one peer group); the **raw host mount is healthy on the new device**. **Two distinct death states** with different errnos — `EIO`(5) both directions, and `EROFS`(30) on write only |
| Q3 | Why `boundUnderParent` returns true | It is a **string comparison of field 5 of a mountinfo line**. It never touches the filesystem, and **neither half reads field 3 (the device number)** — so neither can see that the two entries name different devices |
| Q4 | Cost + failure modes of a real check | A **pure-`/proc` check costs 0.19–0.45 ms, cannot hang, cannot spin up a disk and writes nothing.** The two cheapest *I/O* probes — `statfs` and `getdents` — **succeed over a fully dead filesystem** and are worthless. The hang case was **NOT measured** (§6.3) |
| Q4 | Cost + failure modes of a real check | A **pure-`/proc` check costs 0.16–0.45 ms, cannot hang, cannot spin up a disk and writes nothing.** The two cheapest *I/O* probes — `statfs` and `getdents` — **succeed over a fully dead filesystem** and are worthless. **Every probe that reaches the device blocks on a wedged one, and the process then cannot be killed even with `SIGKILL`** (§6.3) |
| Q5 | Where the check belongs | **The agent**, and not on balance — the controller **cannot** perform it: the inputs are host mount tables it has no access to |
| Q6 | In-place recovery possible | **YES — proven, guest never restarted** (init PID identical across the repair). And **the repair code already exists, is already invoked at exactly the right moment, and is short-circuited by the same false predicate** |
| Q7 | Scope — steady state too? | **YES, and this is the worse half.** A drive can die **with no detach/return cycle at all**; the gate then never runs, the `Return` branch never fires, and **no event of any kind is emitted** |
@@ -358,9 +358,9 @@ All timings from the R-117 state on drive B, on the venue, one run each, `-timeo
| **P2 ext4 abort options** | `shutdown` **or** `emergency_ro` in the super options — **pure `/proc`** | **DEAD=true** ✅ | **DEAD=true** ✅ | false ✅ | **0.17–0.24 ms** | **Catches both.** ext4-specific |
| P3 `statfs` | one syscall | **OK — MISSES IT** ❌ | OK — misses ❌ | OK | 0.042 ms | **Worthless** |
| P4 `getdents` | read one dir entry | **OK — MISSES IT** ❌ | OK — misses ❌ | OK | 0.045–0.061 ms | **Worthless** |
| P5 write + unlink | create, write 1 byte, remove | `EIO`(5) ✅ | `EROFS`(30) ✅ | OK | 0.042–0.152 ms | Catches both, **but writes**, and cannot distinguish a legitimately-`ro` mount from a dead one |
| P6 write + **fsync** | forces the byte to the device | **not measured** (§6.3) | — | — | — | **The probe that can hang** |
| P7 read with page cache dropped (`POSIX_FADV_DONTNEED`) | forces a read to reach the device | **not measured** (§6.3) | — | — | — | **The probe that can hang** |
| P5 write + unlink | create, write 1 byte, remove | `EIO`(5) ✅ | `EROFS`(30) ✅ | OK | 0.042–0.152 ms | Catches both, **but writes**, cannot distinguish a legitimately-`ro` mount from a dead one, **and BLOCKS on a wedged device** (§6.3) |
| P6 write + **fsync** | forces the byte to the device | — | — | OK, **11.4 ms** | **11.4 ms healthy** | **BLOCKS on a wedged device.** 25–70× the `/proc` probes even when healthy, on a loop device with no seek cost |
| P7 read with page cache dropped (`POSIX_FADV_DONTNEED`) | forces a read to reach the device | — | — | OK, 0.18 ms | 0.18 ms | **BLOCKS on a wedged device** |
`GuestSeesMount` itself costs **3.85–5.03 ms**, an order of magnitude more than any probe above,
because it forks `lxc-info`. **Any of P1/P2 is free relative to the check it would strengthen.**
@@ -380,26 +380,75 @@ because it forks `lxc-info`. **Any of P1/P2 is free relative to the check it wou
This is the strongest argument for the pure-`/proc` pair over anything else, independent of the hang
question.
### 6.3 The hang case — NOT ESTABLISHED
### 6.3 The hang case — MEASURED, and it is worse than "a probe that hangs"
**This is the one question in the brief that this run does not answer, and the gap is real.**
Venue: the `dm-linear` drive suspended with `dmsetup suspend --noflush --nolockfs`, which **queues all
I/O indefinitely instead of returning `EIO`** — the controlled "half-dead or spun-down device" the brief
asks about. Verified suspended (`dmsetup info -o attr` → `L-sw`, the `s`), then the full probe set with
`-timeout 3s`, then `dmsetup resume`.
Designed and built, not run: a `dm-linear` device suspended with `dmsetup suspend --noflush`, which
queues all I/O indefinitely instead of returning `EIO` — the controlled "half-dead or spun-down device"
the brief asks about. `r117probe` was extended with **P6 (write + `fsync`)** and **P7 (read with the page
cache dropped)** — the two probes that must reach the platter and therefore can block — plus reporting of
whether the blocked goroutine **ever returns** and the process's OS-thread count before and after.
```
=== PROBES against a WEDGED (not erroring) device, timeout 3s ===
HALF1 GuestSeesMount = true [3.983519ms]
HALF2 isHostMountpoint = true [234.181µs]
=====> BoundUnderParent = true
P1 devno-compare raw="252:12" bind="252:12" -> STALE=false [363.878µs] <-- COMPLETED
P2 ext4-abort-opts superopts="rw" -> DEAD=false [206.104µs] <-- COMPLETED
P3 statfs COMPLETED in 40µs err=<nil> <-- reports HEALTHY
P4 getdents COMPLETED in 64µs err=<nil> <-- reports HEALTHY
P6 write+fsync TIMED OUT after 3.002s — caller proceeds; goroutine still blocked
P6 ...still blocked 8s in; threads 5 -> 5 (LEAKED — the syscall is uninterruptible)
P7 read-uncached TIMED OUT after 3.002s — caller proceeds; goroutine still blocked
P7 ...still blocked 8s in; threads 5 -> 5 (LEAKED)
P5 write+unlink TIMED OUT after 3.002s — caller proceeds; goroutine still blocked
P5 ...still blocked 8s in; threads 5 -> 6 (LEAKED)
```
**It was not run: the second house-internet outage (§2.5) began while the rebuilt 3.6 MB binary was being
copied to the venue.** The build succeeded locally; the transfer did not complete.
**Four results, in ascending order of severity.**
Not measured, therefore not claimed. What is *known* from what did run:
1. **P1 and P2 completed normally, in microseconds.** They read `/proc/self/mountinfo`, which involves
no block device. **This is the whole argument for the §11 recommendation**: on a wedged device the
pure-`/proc` check is not merely faster, it is the only one that returns at all.
2. **P3 and P4 completed *and reported healthy*.** On a wedged device they do not even have the decency
to hang — they lie, exactly as they do on a dead one (§6.1). Confirms **R-117b** in a second failure
mode.
3. **Every probe that touches the device blocked — including P5.** P5 is a buffered `write` with no
`fsync`, which was *expected* to complete out of page cache. It did not: the `O_CREAT` metadata path
needs journal access (`wchan=do_get_write_access`). **There is no such thing as a cheap write probe
that is safe on a wedged device.**
4. **The process could not be killed — not by `SIGTERM`, not by `SIGKILL`.** This is the finding that
changes the recommendation from a preference into a fence:
- **P1 and P2 cannot hang** — they read `/proc/self/mountinfo`, which involves no block device. This is
a property of the files they read, and it is why the recommendation in §8 rests on them.
- The reasoning that a timeout around a blocked uninterruptible syscall leaves a **leaked thread per
poll** — and that at the reconcile ticker's 20-second cadence (`main.go:973`) that is 180 per hour —
is the hypothesis P6/P7 were built to test. **It is untested. It must not be quoted as a result.**
```
# after `timeout 200` had already sent SIGTERM:
PID STAT WCHAN ELAPSED COMMAND
2493045 Dl folio_wait_bit_common 03:20 r117probe
tid 2493045 stat=D wchan=folio_wait_bit_common
tid 2493049 stat=D wchan=do_get_write_access
# explicit SIGKILL:
kill -9 2493045
after SIGKILL: 2493045 Dl folio_wait_bit_common 03:50 <-- STILL ALIVE
# only the device recovering released it:
dmsetup resume r117cel2 -> probe still alive? no — it died once I/O completed
```
**A timeout protects the caller's control flow and nothing else.** The goroutine and its OS thread are
consumed permanently, and the *process* enters uninterruptible sleep. Applied to the agent, whose
reconcile ticker runs every 20 s (`main.go:973`):
- the report loop's caller proceeds, so the timeout "works" — and the evidence of the problem is a
thread you cannot see and cannot reclaim;
- **`systemctl restart felhom-agent` would hang**, because `SIGKILL` does not reach a `D`-state thread.
The agent becomes unrecoverable **until the device recovers or the host reboots** — on a box whose
whole purpose is to be recovered remotely;
- the thread count is **not** a reliable tell (5→5, 5→5, 5→6): the Go runtime reuses threads, so the
leak does not announce itself in the one metric an operator might watch.
> **This disqualifies I/O probing on this path outright.** Not "use a timeout", not "use it carefully" —
> a liveness probe that issues block I/O converts a dead drive into an unkillable agent. §11's
> recommendation issues none.
---
@@ -623,9 +672,12 @@ no write, and no possibility of blocking the report loop.** Effectively free.
promising a property nothing currently tests; the fix owes one that pins it.
6. **Q7 needs its own scenario.** A test that only exercises detach→return will pass while the silent
half stays broken. The in-place recipe in §10 is the second scenario.
7. **The hang question is open** (§6.3). It does not gate this recommendation — P1 and P2 read `/proc`
and cannot block — but it **does** gate any proposal to add an I/O confirmation probe on top. Settle
§6.3 before accepting one.
7. **Do not add an I/O confirmation probe. Measured, not preferred** (§6.3). On a wedged device every
probe that touches it blocks — including a buffered write with no `fsync` — and the process then
**cannot be killed by `SIGKILL`**, so the agent could not be restarted until the device recovered or
the host rebooted. A timeout protects the caller's control flow and nothing else. **P1 and P2 both
returned in microseconds on that same wedged device**; the recommendation issues no block I/O, and
that is now a fence rather than a preference.
---
@@ -662,7 +714,8 @@ no write, and no possibility of blocking the report loop.** Effectively free.
|---|---|---|---|
| **R-117** | *(existing row, now with a mechanism and a recipe)* The bind-liveness signal is a path-presence test; the R-117 state is `BoundUnderParent = true` over an `EIO` namespace | **HIGH** | §3, §5 |
| **R-117a** | **A drive's bind can die in STEADY STATE, with no detach/return cycle, and no event is emitted on any channel.** The gate produces neither action, so a `Return`-branch fix cannot reach it. **P1 does not detect it; only the filesystem's abort flag does** | **HIGH — larger than R-117 as filed** | §9 |
| **R-117b** | **`statfs` and `getdents` both SUCCEED on a namespace returning `EIO` for all content.** Any liveness probe built on either is hollow by construction — it reports healthy in exactly the state it exists to detect | **MEDIUM (design trap)** | §4, §6.1 |
| **R-117b** | **`statfs` and `getdents` both SUCCEED on a namespace returning `EIO` for all content** — and also on a **wedged** one, where they do not even hang. Any liveness probe built on either is hollow by construction — it reports healthy in exactly the state it exists to detect | **MEDIUM (design trap)** | §4, §6.1, §6.3 |
| **R-117f** | **A liveness probe that issues block I/O turns a wedged drive into an UNKILLABLE AGENT.** On a suspended device every I/O probe blocked — including a buffered write with no `fsync` — and the process survived `SIGTERM` **and `SIGKILL`**, dying only when the device recovered. `systemctl restart felhom-agent` would therefore hang, leaving the agent unrecoverable until the device returns or the host reboots. **A timeout protects the caller's control flow and nothing else**, and the thread count does not reveal the leak (5→5, 5→6) | **HIGH (constraint on the fix — it disqualifies an entire probe class)** | §6.3 |
| **R-117c** | **Three comments** (`disks.go:83`, `:154-155`, `:155-165`) state that `BoundUnderParent` means "live + usable in the guest". **Nothing tests it, and it is false in the measured state.** Not counted as a shipped false invariant because `:158-165` scopes its own claim narrowly — but the summary lines do not | **MEDIUM** | §5.3 |
| **R-117d** | **`AttachDrive` logs and returns `"fully live"` for a namespace that is `EIO` on every call** (`intermediary.go:235`), disabling all three wired repair paths including the controller's pre-restart re-attach | **HIGH — this is the self-heal that already exists** | §8.2 |
| **R-117e** | **The two demo boxes share one uplink and one failure domain.** Both went unreachable together, twice in one session, on all four known routes including the WireGuard OOB path via ep0. There is no route to either box that survives the site losing internet — the "out-of-band" path is not out-of-band with respect to this failure | **LOW (operational, not a code defect)** | §2.5 |
@@ -674,50 +727,57 @@ about.** Whoever fixes R-117 will need that box on a current agent to validate l
---
## 14. Teardown — **OWED, NOT YET DONE**
## 14. Teardown — DONE, all three layers
**The second internet outage (§2.5) began before teardown. This section is a liability, not a record.**
Everything below is inert scratch on a Tier 0 box; nothing fenced was touched at any point (§14.2).
Run after the venue came back; the site outage delayed it, it did not skip it.
**Ordering note, recorded because it is a real trap:** any suspended dm device must be `dmsetup resume`d
**before** any `umount`, or every unmount in the teardown blocks on the wedged device — the same
uninterruptible-sleep mechanism as §6.3. Both devices were verified `attr=L--w` (not suspended) first.
### 14.1 Outstanding on demo-hp — the exact commands
### 14.1 Layer 1 — the machine
```bash
# layer 1 — the machine
pct stop 9301 && pct destroy 9301 --purge
# layer 2 — the host
umount /mnt/r117-drives/sd /mnt/r117-drives/cel 2>/dev/null
systemctl stop mnt-r117sd.mount mnt-r117cel.mount
umount /mnt/r117-drives # the scratch shared parent self-bind
dmsetup remove r117cel r117cel2 # now unheld once the binds are gone
losetup -d /dev/loop0
modprobe -r scsi_debug # removes /dev/sdb (the fake disk) entirely
pvesm remove r117scratch && rm -rf /var/lib/vz/r117-scratch
rmdir /mnt/r117-drives/sd /mnt/r117-drives/cel /mnt/r117-drives /mnt/r117cel /mnt/r117sd
rm -rf /root/r117 /root/setup1.sh /root/setup2.sh /root/setup3.sh \
/root/capture.sh /root/capture2.sh /root/r117probe /root/r117-pveam.log
# optional: rm /var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst (124 MB, downloaded
# this run; harmless to keep and useful for the next scratch guest)
# then verify: pct list | pvesm status | dmsetup ls | losetup -a | lsmod | grep scsi_debug | findmnt /mnt/r117-drives
```
BEFORE pct list -> 9201 running demo-hp / 9301 running r117
pct stop 9301 ; pct destroy 9301 --purge ("purging CT 9301 from related configurations..")
AFTER pct list -> 9201 running demo-hp <-- only the pre-existing guest
```
### 14.2 Layer 3 — the hub: **nothing to dispose of**
### 14.2 Layer 2 — the host, with the space actually returned
| | Before | After |
|---|---|---|
| `pvesm status` | `r117scratch` **active**, `local` **38.97 %** | `r117scratch` **removed**, `local` **37.02 %** *(session start: 37.00 %)* |
| `df -h /var/lib/vz` | 16G used, **42 %** | 15G used, **40 %** *(session start: 40 %)* |
| `dmsetup ls` | `r117cel (252:11)`, `r117cel2 (252:12)` + 11 pre-existing | **both gone**; the 11 pre-existing `pve-*` unchanged |
| `losetup -a` | `/dev/loop0` (cel.img), `/dev/loop1` (9301 rootfs) | **empty — no loop devices** |
| `scsi_debug` | loaded (1) | **unloaded (0)**; `/dev/sd*` back to `sda1..3` only |
| r117 mounts | raw + bind + scratch shared parent | `findmnt \| grep r117` → **(none)** |
| `/mnt` | `+ r117-drives`, `r117cel`, `r117sd` | **`felhom-drives`, `nvme-1tb`, `vzsnap0`** — exactly as found |
| `/root` | 7 scratch files + `r117/` | `ls /root \| grep -i r117` → **(none)** |
The transient `mnt-r117cel.mount` / `mnt-r117sd.mount` units died with the unmounts
(`Unit mnt-r117sd.mount not loaded` — created by `systemd-mount`, so nothing was left in `/etc`). The
124 MB `debian-13-standard` template was also removed.
### 14.3 Layer 3 — the hub: nothing to dispose of
**This run created no hub-side record.** No customer, no appliance, no pairing, no claim: guest 9301 ran
no controller and had **no network interface at all**, and no agent was enrolled against it. Stated
explicitly per §13's rule that "teardown clean" without layer 3 is the `sess-c` failure — here layer 3 is
genuinely empty, and that is a different claim from silence.
explicitly per §13 of the template — "teardown clean" without layer 3 is the `sess-c` failure. Here layer
3 is genuinely empty, which is a different claim from silence.
### 14.3 Fences — verified untouched throughout
### 14.4 Fences — re-verified after teardown, not assumed
| Fence | State |
|---|---|
| demo-hp guest **9201** | `running` throughout; never a probe target |
| demo-hp **`drill-r50`** (VM 300) | `stopped`; never started, never touched |
| demo-hp **`local-lvm`** | untouched — scratch rootfs deliberately placed on a new dir storage on `pve-root` |
| demo-hp **`felhom-backup`** / `/mnt/nvme-1tb` | not re-targeted, not written to, `content backup` unchanged |
| demo-hp guest **9201** | **`running`** — before, during and after; never a probe target |
| demo-hp **`drill-r50`** (VM 300) | **`stopped`** — never started, never touched |
| demo-hp **`local-lvm`** | **38.84 %** — byte-identical before and after, and to the session-start reading |
| demo-hp **`felhom-backup`** | **`active`, 0.46 %**, `content backup` **unchanged** — not re-targeted, not written to |
| live shared parent `/mnt/felhom-drives` | intact with **both** submounts (`nvme-1tb` on `/dev/nvme0n1[/felhom-data]`, `Felhom-Share` autofs) — a **separate** scratch parent was used throughout |
| `felhom-agent` service on demo-hp | **`active`** |
| demo-felhom, Peti's cluster | never contacted |
| **ep0** | **read-only** — one `ssh`, `wg show`, and TCP reachability probes toward `10.77.0.2/.3` during outage 1. Nothing written |
| `/mnt/felhom-drives` (live shared parent) | untouched; a **separate** scratch parent was used |
| **Code** | **no `.go` file in either repo modified**; nothing built for deployment, nothing published, no version bumped |
| Repos | the probe lives outside both, in the session scratchpad; the agent tree copy verified byte-identical to `d4eb259` |
@@ -725,12 +785,14 @@ genuinely empty, and that is a different claim from silence.
## 15. What did not happen, and why
- **Q4's hang measurement was not taken** (§6.3) — the venue went offline mid-transfer. The venue
(`dmsetup suspend --noflush`) and the two probes (P6 write+`fsync`, P7 cache-dropped read, both with
timeout-fires reporting and thread-leak counting) are built and specified; only the run is missing.
**The associated hypothesis — that a timeout around a blocked syscall leaks a thread per poll — is
explicitly NOT claimed as a result.**
- **Teardown was not performed** (§14). Commands recorded; nothing fenced was touched.
- **Everything the brief asked for was measured.** The two items the first internet outage deferred —
Q4's hang case and teardown — were completed once the site came back: §6.3 and §14 are measurements,
not plans. The thread-leak hypothesis P6/P7 were built to test turned out to be **the weaker half of
the real result**: the leak is invisible in the thread count (5→5, 5→6), while the process being
unkillable by `SIGKILL` is unambiguous.
- **No load or duration testing of the recommended check.** P1/P2 were measured as single calls
(0.16–0.45 ms), not across a sustained reconcile loop on a box with many drives. Nothing suggests a
problem — they are two `/proc` reads the code already performs — but it was not measured.
- **No fresh `/disks` payload and no fresh controller gate log** — deliberate, and the same call
`DIAG-r116` §14 made: both were already captured verbatim on real hardware (twice and four times
respectively), and running a controller here would have meant a hub customer, a pairing, a golden fetch