admin/felhom-agent
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v0.117.0 — R-117: the liveness signal now tests liveness

Browse Source 
BoundUnderParent reported a namespace that returned EIO on every read and write
as healthy, and the gate restarted the customer's apps onto it. Both existing
terms parse a mountinfo line and then test only fields[4], the mount POINT.
Field 3 — major:minor — sat in the same parsed slice and was discarded.

Measured on hardware: raw 8:32 /dev/sdc, bind 8:16 /dev/sdb with `shutdown`,
bound_under_parent true, EIO both directions, and the controller taking its
Return branch and emailing backup_target_restored with no alarm on any channel.

BoundUnderParent gains a third term at both /disks construction sites. The new
bindLiveness reads /proc only and asks two questions: the bind must name the
same device as the raw mount, and the filesystem must not have aborted (ext4
`shutdown` or `emergency_ro`).

The second check is not optional. A device that fails WITHOUT disappearing gives
the identical all-signals-healthy state with the devnos EQUAL and the drive never
Disconnected, so the gate produces neither a Stop nor a Return and nothing is
emitted on any channel, indefinitely (R-117a). A devno-only fix would have passed
every payload test.

Three states, never a bool: {Unknown, Live, StaleDevice, Aborted}, read through
Usable(), where Unknown counts as PRESENT — reporting absent stops a working
customer's apps.

No new recovery path; the existing one was unblocked. AttachDrive's normalize leg
already did the repair and three call sites already invoked it, including the
controller's Return branch before it restarts apps. All three died on
`if n == 1 && GuestSeesMount(...)` returning early. Now: StaleDevice ⇒ re-bind
(repairs live, guest never restarts); Aborted ⇒ quiet no-op, because a re-bind
lands on the same dead superblock and this runs every 20s — an infinite silent
retry that masks the state; it surfaces via BoundUnderParent=false instead.

Ordering trap caught by a test: reading the abort flag before comparing devices
classifies the real return state as aborted (its stale bind carries `shutdown`
too) and refuses the repair while still reporting correctly. The abort flag is
read off the RAW mount in the stale case.

Tests 849 → 863, 29/29 packages green. 6 red-proofs, each verified to have
landed. A hollow test was caught during them: the aborted fixture first used a
/dev/mapper device, for which RoleForStorage derives role=system — a system row
has no GuestPath, never runs the conjunction, and reports false by default, so
the assertion passed vacuously and no mutation could fail it. Found because RP1
failed to fail.
This commit is contained in:
authentik Default Admin
2026-07-30 12:26:42 +02:00
parent 6be168d1a0
commit 966d8f41ff
7 changed files with 947 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files
internal/localapi/disks.go
+59 -18
View File
@@ -79,8 +79,10 @@ type GuestAttacher interface {
DetachDrive(ctx context.Context, where string) error
// EnsureSharedParent makes the host stable parent shared + installs the boot-persistence unit.
EnsureSharedParent(ctx context.Context) error
// GuestSeesMount reports whether vmid's guest sees `path` as a mount in its own namespace (the
// guest-usable signal — distinct from the host having the bind). Backs BoundUnderParent.
// GuestSeesMount reports whether vmid's guest sees `path` as a mount in its own namespace — the
// guest-VISIBILITY signal, distinct from the host having the bind. One of the three terms behind
// BoundUnderParent. It is a path-presence test and NOT a liveness signal (R-117): a stale bind over a
// dead device is still "seen". Liveness is bindLiveness's job.
GuestSeesMount(ctx context.Context, vmid int, path string) bool
// GuestBootID returns a token that changes on every guest boot (host or guest) but is stable across a
// controller-only restart — the deterministic guest-reboot signal the controller recreates apps on.
@@ -155,14 +157,30 @@ type DiskInfo struct {
// BoundUnderParent reports whether the drive is live + usable in the guest in the intermediary model.
// The controller's drive-absent gate + auto-restart key on this (and State).
//
// It is a CONJUNCTION of two facts, and both halves are load-bearing (R-113, v0.114.0):
// It is a CONJUNCTION of THREE facts, and all three are load-bearing:
// 1. felhom-data is bound under the shared parent at GuestPath (the guest-visible mount check), and
// 2. the drive's RAW host mount is still mounted — i.e. the DEVICE is still there.
// Half 1 alone was the bug: the raw mount is device-bound and dies with its device, but the agent's
// own bind is not, so half 1 stays true over a stale shell after the device is pulled. The controller
// read that survivor as "present" and the drive-absent alarm could never fire — measured live in E-2d
// (felhom.eu audits/E2D-fresh-vm-2026-07-29.md §5.2). Half 2 alone would regress boot ordering, where
// the raw mounts early and the bind lands ~18s later; the conjunction keeps that window reading absent.
// 2. the drive's RAW host mount is still mounted — i.e. the DEVICE is still there (R-113), and
// 3. the bind actually WORKS: it names the same device as the raw mount, and that filesystem has
// not aborted (R-117, v0.117.0 — see bindLiveness).
// Half 1 alone was the bug R-113 fixed: the raw mount is device-bound and dies with its device, but
// the agent's own bind is not, so half 1 stays true over a stale shell after the device is pulled. The
// controller read that survivor as "present" and the drive-absent alarm could never fire — measured
// live in E-2d (felhom.eu audits/E2D-fresh-vm-2026-07-29.md §5.2). Half 2 alone would regress boot
// ordering, where the raw mounts early and the bind lands ~18s later; the conjunction keeps that
// window reading absent.
//
// Terms 1 and 2 TOGETHER were still not liveness, which is R-117: both are path-presence tests
// comparing only field 5 of a mountinfo line, so both stay true over a bind that names the drive that
// went away while the raw mount healed onto the returning one via its fs-UUID-keyed unit. Measured
// live: raw on 8:32 /dev/sdc, bind on 8:16 /dev/sdb with `shutdown`, this field TRUE, EIO on every
// read and write, and the gate restarting the customer's apps onto it with no alarm on any channel
// (felhom.eu audits/SPIKE-r117-bind-liveness-2026-07-30.md §5.2).
//
// THE TESTS THAT PIN THIS COMMENT, because for three releases it promised a property nothing tested
// (spike §5.3): disks_bind_liveness_test.go — TestDisks_BindLiveness_StaleBindReadsAbsent (term 3,
// case a), _AbortedFilesystemReadsAbsent (term 3, case b, the steady-state case that emits nothing
// today), _UnknownIsTreatedAsPresent (the cannot-tell rule) and _HealthyReadsPresent (no false
// negative). Each asserts the CONSEQUENCE — what this field reads — not the mechanism.
BoundUnderParent bool `json:"bound_under_parent"`
// Smart is the already-computed per-disk SMART health summary (v0.94.0), serialized here so the
// controller can render a disk-health card + degradation alert WITHOUT any new smartctl load — the
@@ -219,8 +237,12 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
// boot-ordering behaviour the controller's gate depends on exactly as it was
// (raw mounted early, bind not yet ⇒ still absent) while closing the case the
// gate could never see (bind outlived the device ⇒ now absent).
// R-117: AND in bind LIVENESS. The two terms above are both path-presence tests, so
// both stay true over a bind that names the drive that went away while the raw mount
// healed onto the returning one — EIO on every call, payload healthy.
di.BoundUnderParent = s.boundUnderParent(r.Context(), vmid, gp) &&
s.devicePresent(t.MountPath)
s.devicePresent(t.MountPath) &&
s.bindUsable(gp, t.MountPath)
}
}
// R-116: carry the GUEST PATH on the backup-target row even when its role has flipped to
@@ -351,12 +373,13 @@ func (s *Server) handleDisks(w http.ResponseWriter, r *http.Request, vmid int) {
// exactly like the Observe path — else the controller reads a registry drive as "Leválasztva".
if gp := StablePathForRaw(d.MountPath); gp != "" {
di.GuestPath = gp
// R-113: same conjunction as the Observe path. This path matters MORE, not less —
// a registry drive with no PVE dir-storage is exactly the shape E-2d detached, and
// its State is hardcoded "attached" below, so the raw-mount check is the only
// device truth this row carries.
// R-113 + R-117: same three-term conjunction as the Observe path. This path matters
// MORE, not less — a registry drive with no PVE dir-storage is exactly the shape
// E-2d detached, and its State is hardcoded "attached" below, so these checks are
// the only device truth this row carries.
di.BoundUnderParent = s.boundUnderParent(r.Context(), vmid, gp) &&
s.devicePresent(d.MountPath)
s.devicePresent(d.MountPath) &&
s.bindUsable(gp, d.MountPath)
}
// A registry drive has no PVE `pvesm status` snapshot, so fill backing device + capacity
// from the host directly: resolve the device by fs-UUID, and statfs the mount for size —
@@ -943,9 +966,12 @@ func (s *Server) handleDiskFormat(w http.ResponseWriter, r *http.Request, vmid i
}
// boundUnderParent reports whether a drive's felhom-data is bound at its stable guest path AND visible
// inside the guest (the usable-in-guest signal the controller's gate keys on — a guest reboot leaves the
// host bind in place but invisible to the guest until re-propagated). Injectable via s.boundCheck for
// tests; defaults to the guest-namespace mount check.
// inside the guest — a guest reboot leaves the host bind in place but invisible to the guest until
// re-propagated. Injectable via s.boundCheck for tests; defaults to the guest-namespace mount check.
//
// This is VISIBILITY, not liveness (R-117). It compares only the mount point, so it stays true over a
// bind whose device has gone; do not read it as "usable in the guest" — that is the whole three-term
// conjunction at the two /disks construction sites, whose third term is bindUsable.
func (s *Server) boundUnderParent(ctx context.Context, vmid int, stablePath string) bool {
if s.boundCheck != nil {
return s.boundCheck(stablePath)
@@ -979,6 +1005,21 @@ func (s *Server) devicePresent(rawMountPath string) bool {
return isHostMountpoint(rawMountPath)
}
// bindUsable is the THIRD term of the BoundUnderParent conjunction (R-117): the bind must not only exist
// and be guest-visible, it must actually WORK. The first two terms are path-presence tests and are both
// satisfied by a bind that names the drive that went away — measured live, with EIO on every read and
// write while the payload read healthy and the gate restarted the customer's apps onto it.
//
// UNKNOWN counts as usable, via BindLiveness.Usable — the same "cannot tell → never absent" rule
// devicePresent applies above, and for the same reason: a false absent stops a working customer's apps.
// Injectable via s.livenessCheck; the default reads /proc only and issues NO block I/O (CLAUDE.md).
func (s *Server) bindUsable(stable, rawMountPath string) bool {
if s.livenessCheck != nil {
return s.livenessCheck(stable, rawMountPath).Usable()
}
return bindLiveness(stable, rawMountPath).Usable()
}
// guestBoundPaths returns the set of guest mountpoint paths (the `mp=` of each entry in the guest's
// config) — i.e. the host drives actually BOUND into the guest. F9: this is the guest-attached signal
// (`GuestAttached`) that distinguishes a guest-usable drive from one merely present on the host. A bind