docs(R-117): spike — the mechanism, a recipe, and a steady-state half nobody had looked for

Both halves of the R-113 conjunction are path-presence tests: GuestSeesMount
(intermediary.go:276) and isHostMountpoint (:394) compare field 5 of a mountinfo
line and never read field 3, so neither can see that the bind and the raw mount
name different devices. Measured BoundUnderParent=TRUE over a namespace that
EIOs on every read and write.

Reproduced 3/3 on a purpose-built scratch LXC on demo-hp; predicates evaluated
by a throwaway probe calling the real localapi code from d4eb259.

Three results that change the shape of the fix:

- Q7: a bind can die in STEADY STATE with no detach/return cycle. The gate
  produces no action and nothing is emitted on any channel. A Return-branch fix
  cannot reach this half, and a devno comparison does not detect it.
- Q6/R-117d: AttachDrive's normalize leg already performs the repair, and three
  call sites already invoke it - including the controller's Return branch before
  it restarts apps. All defeated by one early return at :235. Unblock the
  existing path; do not add a new one.
- Q1: the device-node change is a CONSEQUENCE, not a precondition. The stale
  bind pins the dead superblock, forcing the returning device onto a new number.
  Control test: released, the letter is reused.

Not established: the hang case. Venue and probes built, run lost to a site
internet outage; the thread-leak hypothesis is not claimed as a result.
Teardown of the spike venue is owed - commands in the findings doc; nothing
fenced was touched and no hub-side record was created.
This commit is contained in:
2026-07-30 11:38:22 +02:00
parent 29bcfeb214
commit c949389c95
3 changed files with 829 additions and 1 deletions
+74
View File
@@ -0,0 +1,74 @@
# REPORT — SPIKE R-117: a dead bind that reports healthy (2026-07-30)
Written as `REPORT-<topic>.md`, not `REPORT.md`, per this repo's parallel-session rule and the
established local pattern (`REPORT-r116-diag.md`, `REPORT-session-c.md`, …). The shared `REPORT.md`
was not touched.
**Class: Spike.** Deliverable is a findings document. **No production code was written; no `.go` file
in either repo was modified; nothing was built for deployment, published, or version-bumped.**
## Deliverables
| File | Action |
|---|---|
| `documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md` | **new** — the findings doc (Q1Q7, evidence, recipe, probe comparison, recommendation) |
| `documentation/backlog/OPEN-ITEMS.md` | **R-117 row rewritten** with the mechanism, the reproduction recipe and the fix constraint — the one register edit, per the brief |
**No CHANGELOG entry.** This repo's changelogs are per-area (`hub/`, `scripts/`, `website/`); a
documentation-only change belongs to none of them. Stated rather than silently skipped, per standing
rule 4.
## Baselines
agent `main` **v0.116.0** @ `d4eb259` · controller `main` **v0.186.0** @ `b331f18` ·
`felhom.eu` HEAD `29bcfeb` · hub **live 0.82.0** · golden **0.186.0** ·
demo-hp PVE 9.2.2 / kernel 7.0.2-6-pve, **live agent 0.113.0** (= manifest `MinAgent`; never used as
the source of a behavioural claim — every predicate result came from a probe built from `main`).
## Results
All seven questions answered empirically except the hang case (below).
- **Q1 — reproduced 3/3**, two device classes, on a purpose-built scratch LXC (9301) on demo-hp.
**The device-node change is a consequence of the defect, not a precondition** — control test: with
the stale bind held the drive returns as `sdc` (8:32); released, the letter is reused (`sdb`, 8:16).
- **Q2 — two death states**: device removed ⇒ superopts gain `shutdown`, `EIO`(5) on read and write,
host and guest; device errors in place ⇒ `emergency_ro`, write `EROFS`(30), reads served from cache.
The raw host mount is genuinely healthy in both. **No cross-device mis-identification is possible**
on this path — the unit is fs-UUID-keyed.
- **Q3 — confirmed at source and live.** Both halves of the R-113 conjunction compare **field 5** of a
mountinfo line and **never read field 3 (`major:minor`)**, so neither can see that the bind and the
raw mount name different devices. Measured `BoundUnderParent = TRUE` over an `EIO` namespace.
- **Q4 — a pure-`/proc` check costs 0.170.45 ms**, cannot hang, spins up no disk, writes nothing.
**`statfs` and `getdents` both SUCCEED on a dead namespace** — probes built on either are hollow.
- **Q5 — the agent**, and not on balance: the controller runs inside the guest and cannot see the host
mount tables the check needs.
- **Q6 — recovery works in place, guest never restarted** (init PID identical). **The repair code
already exists and three call sites already invoke it**, including the controller's `Return` branch
*before* it restarts apps — all defeated by one early return.
- **Q7 — a bind can die in steady state, no cycle at all.** The gate produces no action and **nothing
is emitted on any channel.** A `Return`-branch fix cannot reach this half.
## Not established
**The hang case (findings §6.3).** The venue (`dmsetup suspend --noflush`) and the two blocking probes
(write+`fsync`, cache-dropped read, both with timeout-fires and thread-leak reporting) were built; the
run did not happen — the venue lost internet mid-transfer of the rebuilt binary. **The associated
thread-leak hypothesis is explicitly not claimed as a result.** It does not gate the `/proc`
recommendation but does gate adding any I/O confirmation probe.
## Teardown — OWED
**Not performed** — the second site outage began first. Findings §14 lists the exact commands. Nothing
fenced was touched at any point (9201 `running`, `drill-r50` `stopped`, `local-lvm` untouched,
`felhom-backup` not re-targeted, `/mnt/felhom-drives` untouched, ep0 read-only). **Layer 3 is
genuinely empty** — guest 9301 had no network interface and ran no controller, so no hub-side record
was created.
## Findings filed, none fixed
R-117 (mechanism + recipe), **R-117a** steady-state death with no event (HIGH, larger than R-117 as
filed), **R-117b** `statfs`/`getdents` are hollow liveness probes, **R-117c** three untested comments
promising "live + usable in the guest", **R-117d** the self-heal that already exists is short-circuited
(HIGH), **R-117e** both demo boxes share one failure domain — no route survives the site losing
internet, including the WireGuard OOB path.