v0.117.0 — R-117: the liveness signal now tests liveness
BoundUnderParent reported a namespace that returned EIO on every read and write
as healthy, and the gate restarted the customer's apps onto it. Both existing
terms parse a mountinfo line and then test only fields[4], the mount POINT.
Field 3 — major:minor — sat in the same parsed slice and was discarded.
Measured on hardware: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb with `shutdown`,
bound_under_parent true, EIO both directions, and the controller taking its
Return branch and emailing backup_target_restored with no alarm on any channel.
BoundUnderParent gains a third term at both /disks construction sites. The new
bindLiveness reads /proc only and asks two questions: the bind must name the
same device as the raw mount, and the filesystem must not have aborted (ext4
`shutdown` or `emergency_ro`).
The second check is not optional. A device that fails WITHOUT disappearing gives
the identical all-signals-healthy state with the devnos EQUAL and the drive never
Disconnected, so the gate produces neither a Stop nor a Return and nothing is
emitted on any channel, indefinitely (R-117a). A devno-only fix would have passed
every payload test.
Three states, never a bool: {Unknown, Live, StaleDevice, Aborted}, read through
Usable(), where Unknown counts as PRESENT — reporting absent stops a working
customer's apps.
No new recovery path; the existing one was unblocked. AttachDrive's normalize leg
already did the repair and three call sites already invoked it, including the
controller's Return branch before it restarts apps. All three died on
`if n == 1 && GuestSeesMount(...)` returning early. Now: StaleDevice ⇒ re-bind
(repairs live, guest never restarts); Aborted ⇒ quiet no-op, because a re-bind
lands on the same dead superblock and this runs every 20s — an infinite silent
retry that masks the state; it surfaces via BoundUnderParent=false instead.
Ordering trap caught by a test: reading the abort flag before comparing devices
classifies the real return state as aborted (its stale bind carries `shutdown`
too) and refuses the repair while still reporting correctly. The abort flag is
read off the RAW mount in the stale case.
Tests 849 → 863, 29/29 packages green. 6 red-proofs, each verified to have
landed. A hollow test was caught during them: the aborted fixture first used a
/dev/mapper device, for which RoleForStorage derives role=system — a system row
has no GuestPath, never runs the conjunction, and reports false by default, so
the assertion passed vacuously and no mutation could fail it. Found because RP1
failed to fail.
This commit is contained in:
@@ -1,3 +1,66 @@
|
||||
## v0.117.0 — R-117: the liveness signal now tests liveness (2026-07-30)
|
||||
|
||||
**`BoundUnderParent` reported a namespace that returned `EIO` on every read and write as healthy, and the
|
||||
gate restarted the customer's apps onto it.** Both existing terms — `GuestSeesMount` and
|
||||
`isHostMountpoint` — parse a mountinfo line and then test only `fields[4]`, the mount POINT. Field 3, the
|
||||
`major:minor`, sat in the same parsed slice and was discarded. So after a drive was detached and returned,
|
||||
the raw host mount healed onto the NEW device via its fs-UUID-keyed unit while the bind still named the
|
||||
OLD one, and both terms stayed true. Measured on hardware: raw `8:32 /dev/sdc`, bind `8:16 /dev/sdb` with
|
||||
`shutdown`, `bound_under_parent: true`, `EIO` both directions — and the controller taking its `Return`
|
||||
branch, restarting the gate-stopped apps and emailing `backup_target_restored`, with no alarm on any
|
||||
channel (`felhom.eu/documentation/audits/SPIKE-r117-bind-liveness-2026-07-30.md`).
|
||||
|
||||
**THE FIX is one predicate.** `BoundUnderParent` gains a third term, `bindUsable`, at both /disks
|
||||
construction sites (`disks.go`). The new `bindLiveness` (`intermediary.go`) reads `/proc` only and asks
|
||||
two questions:
|
||||
|
||||
- **the bind must name the same device as the raw mount** — sound rather than heuristic, because a stale
|
||||
bind pins the dead superblock, which keeps the old device index allocated, which FORCES the returning
|
||||
device onto a different number (measured both ways: releasing the bind let the letter be reused);
|
||||
- **the filesystem must not have aborted** — ext4's `shutdown` (device removed) or `emergency_ro`
|
||||
(`errors=remount-ro` fired in place). Both tokens were measured and both are load-bearing.
|
||||
|
||||
**The second check is not optional, and this is the substantive part.** R-117 was filed as a
|
||||
detach/return defect, but a device that fails WITHOUT disappearing produces the identical
|
||||
all-signals-healthy state with the devnos EQUAL and the drive never `Disconnected` — so the gate produces
|
||||
neither a `Stop` nor a `Return` action and **nothing is emitted on any channel, indefinitely** (R-117a).
|
||||
The device comparison alone cannot see it. A devno-only fix would have passed every payload test.
|
||||
|
||||
**THREE states, never a bool.** `BindLiveness` is `{Unknown, Live, StaleDevice, Aborted}` and every
|
||||
caller reads it through `Usable()`, where `Unknown` counts as PRESENT — unreadable `/proc`, no raw mount
|
||||
to compare against, or a filesystem whose abort vocabulary we have not measured all degrade to unknown,
|
||||
because reporting absent stops a working customer's apps. The workspace's false-invariant table records
|
||||
`newestArchiveOn` promising exactly this over a `(value, bool)` shape that could not express it.
|
||||
|
||||
**NO NEW RECOVERY PATH — the existing one was unblocked.** `AttachDrive`'s normalize leg already performed
|
||||
the needed repair, and three call sites already invoked it: the 20 s reconcile ticker
|
||||
(`cmd/felhom-agent/main.go`), agent startup, and **the controller's `Return` branch, before it restarts
|
||||
the apps**. All three were defeated by `if n == 1 && b.GuestSeesMount(...)` returning early and logging
|
||||
*"fully live, no-op"* about a dead namespace. The arm now switches on the verdict:
|
||||
**`BindStaleDevice` ⇒ re-bind** (repairs live, guest never restarts — proven, init PID identical);
|
||||
**`BindAborted` ⇒ quiet no-op**, because a re-bind lands on the SAME aborted superblock and this runs
|
||||
every 20 s, so re-binding would be an infinite silent retry that also masks the state — it surfaces via
|
||||
`BoundUnderParent=false` instead, and clearing an aborted filesystem needs a remount or a fsck, which is
|
||||
an operator decision. **Ordering trap, caught by a test:** reading the abort flag before comparing devices
|
||||
classifies the real return state as aborted (its stale bind carries `shutdown` too) and refuses the repair
|
||||
while still reporting correctly — so the abort flag is read off the RAW mount in the stale case.
|
||||
|
||||
**Also:** `isHostMountpoint` and `countHostMounts` are now one-liners over the single new parser
|
||||
`hostMountEntries`, which yields devno, root, fstype and super options (REUSE.md's
|
||||
three-independent-readers note narrowed). The three comments that promised `BoundUnderParent` meant
|
||||
"live + usable in the guest" — true for three releases only as a wish — now name the tests that pin them.
|
||||
`CLAUDE.md` gained a standing rule: **a health check issues no block I/O** (a probe on a wedged device
|
||||
survives `SIGKILL`; measured).
|
||||
|
||||
Tests **849 → 863** (+14 top-level, verified against a temp worktree at HEAD — not inferred from a count taken mid-edit, which is how the first draft of this line read 858 → 873), all green: 29/29 packages, `go build`/`go vet`/`go test` each with `rc=0` read separately. **6 red-proofs, each verified to have landed**: term 3 removed at
|
||||
each construction site; the abort check neutered (the P1-only fix — fails the aborted tests ALONE);
|
||||
`emergency_ro` dropped from the token list; unknown reported as not-live; and the aborted arm falling
|
||||
through to a re-bind. **A hollow test was caught and fixed during the red-proofs**: the aborted fixture
|
||||
first used a `/dev/mapper` device, for which `RoleForStorage` derives `role="system"` — a system row has
|
||||
no `GuestPath`, never runs the conjunction, and reports `false` by default, so the assertion passed
|
||||
vacuously and NO mutation could fail it. Found because RP1 failed to fail. The fixtures now assert the
|
||||
production row shape before asserting the field.
|
||||
|
||||
## v0.116.0 — R-116: the flag and the key finally share a row (2026-07-30)
|
||||
|
||||
**Closes the mechanism `v0.115.0` guessed at.** The absent-drive alarm was the generic
|
||||
|
||||
Reference in New Issue
Block a user