v0.117.0 — R-117: the liveness signal now tests liveness
BoundUnderParent reported a namespace that returned EIO on every read and write
as healthy, and the gate restarted the customer's apps onto it. Both existing
terms parse a mountinfo line and then test only fields[4], the mount POINT.
Field 3 — major:minor — sat in the same parsed slice and was discarded.
Measured on hardware: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb with `shutdown`,
bound_under_parent true, EIO both directions, and the controller taking its
Return branch and emailing backup_target_restored with no alarm on any channel.
BoundUnderParent gains a third term at both /disks construction sites. The new
bindLiveness reads /proc only and asks two questions: the bind must name the
same device as the raw mount, and the filesystem must not have aborted (ext4
`shutdown` or `emergency_ro`).
The second check is not optional. A device that fails WITHOUT disappearing gives
the identical all-signals-healthy state with the devnos EQUAL and the drive never
Disconnected, so the gate produces neither a Stop nor a Return and nothing is
emitted on any channel, indefinitely (R-117a). A devno-only fix would have passed
every payload test.
Three states, never a bool: {Unknown, Live, StaleDevice, Aborted}, read through
Usable(), where Unknown counts as PRESENT — reporting absent stops a working
customer's apps.
No new recovery path; the existing one was unblocked. AttachDrive's normalize leg
already did the repair and three call sites already invoked it, including the
controller's Return branch before it restarts apps. All three died on
`if n == 1 && GuestSeesMount(...)` returning early. Now: StaleDevice ⇒ re-bind
(repairs live, guest never restarts); Aborted ⇒ quiet no-op, because a re-bind
lands on the same dead superblock and this runs every 20s — an infinite silent
retry that masks the state; it surfaces via BoundUnderParent=false instead.
Ordering trap caught by a test: reading the abort flag before comparing devices
classifies the real return state as aborted (its stale bind carries `shutdown`
too) and refuses the repair while still reporting correctly. The abort flag is
read off the RAW mount in the stale case.
Tests 849 → 863, 29/29 packages green. 6 red-proofs, each verified to have
landed. A hollow test was caught during them: the aborted fixture first used a
/dev/mapper device, for which RoleForStorage derives role=system — a system row
has no GuestPath, never runs the conjunction, and reports false by default, so
the assertion passed vacuously and no mutation could fail it. Found because RP1
failed to fail.
This commit is contained in:
@@ -324,6 +324,15 @@ type Server struct {
|
||||
// Optional — nil defaults to isHostMountpoint; tests inject a fake.
|
||||
deviceCheck func(string) bool
|
||||
|
||||
// livenessCheck answers whether the bind at a stable guest path is USABLE, not merely present — the
|
||||
// third term of the BoundUnderParent conjunction (R-117). Deliberately separate from boundCheck and
|
||||
// deviceCheck because it is the only one of the three that compares them: boundCheck asks "does the
|
||||
// guest see a mount by that name", deviceCheck asks "is the raw mount still there", and BOTH are
|
||||
// satisfied by a bind that names the drive that went away while the raw mount healed onto the
|
||||
// returning one. Optional — nil defaults to bindLiveness. Prefer redirecting procSelfMountinfo at a
|
||||
// captured fixture over injecting here: that exercises the real parser and predicate.
|
||||
livenessCheck func(stable, raw string) BindLiveness
|
||||
|
||||
jobsMu sync.Mutex
|
||||
// jobs is per-guest-PER-TARGET backup job state (slice 8B; keyed by target too since R-82).
|
||||
// Keying by vmid alone would let a PBS backup started inside the same quiesce window collide
|
||||
|
||||
Reference in New Issue
Block a user