felhom-agent/REPORT.md

REPORT — v0.117.0: R-117, the liveness signal now tests liveness (2026-07-30)

Overwritten per the standing rule. Full evidence: felhom.eu/documentation/audits/R117-v0117-2026-07-30.md. Spike this builds on: audits/SPIKE-r117-bind-liveness-2026-07-30.md.

agent	0.116.0 → 0.117.0 @ 966d8f4
published + vouched	0.117.0, sha e6768a12eba1c9e0… (anonymous GET verified, manifest read back)
deployed	demo-hp 0.117.0 (from 0.113.0 — see R-121) · felhom-pve still 0.115.0, out of scope
controller / golden / hub	0.186.0 / 0.186.0 / 0.82.0 — untouched
tests	849 → 863, 29/29 packages, build/vet/test each rc=0 read separately

The defect

BoundUnderParent reported a namespace returning EIO on every read and write as healthy, and the gate restarted the customer's apps onto it. Both existing terms — GuestSeesMount and isHostMountpoint — parse a mountinfo line and then test only fields[4], the mount POINT. Field 3, major:minor, sat in the same parsed slice and was discarded.

The fix — one predicate, both halves mandatory

bindLiveness (internal/localapi/intermediary.go) becomes the third term of the conjunction at both /disks construction sites. /proc only, no block I/O:

• the bind must name the same device as the raw mount — sound, not heuristic: a stale bind pins the dead superblock, which keeps the old device index allocated, which forces the returning device onto a different number;
• the filesystem must not have aborted — shutdown or emergency_ro, both measured.

The second is not optional. A device that fails without disappearing gives the identical all-signals-healthy state with the devnos equal and the drive never Disconnected, so the gate emits nothing at all, indefinitely (R-117a). A devno-only fix passes every payload test; red-proof RP3 exists for exactly that.

Three states, never a bool: {Unknown, Live, StaleDevice, Aborted}, Unknown the zero value, and Usable() is the single place where unknown counts as present — absent stops a working customer's apps.

The ruling (task §2.2) — and it is flagged for overrule

StaleDevice ⇒ re-bind (the raw mount is a healthy new superblock; repairs live, guest never restarts). Aborted ⇒ quiet no-op, and surface via BoundUnderParent=false — a re-bind lands on the same dead superblock and this runs every 20 s, so repairing would be an infinite silent retry that also masks the state. No operator decision taken quietly: it does change customer-visible behaviour (apps stop and an alarm fires where today nothing happens), but it routes an already-broken state into the existing gate, event types and Hungarian copy — no new concept reaches the customer — and the alternative is applications writing documents into a filesystem that rejects every write.

No new recovery path. AttachDrive's normalize leg already did the repair and three call sites already invoked it (20 s ticker, agent startup, and the controller's Return branch before restartStacks). All three died on if n == 1 && GuestSeesMount(...) logging "fully live, no-op" about a dead namespace.

Ordering trap, caught by a test: reading the abort flag before comparing devices classifies the real return state as aborted (its stale bind carries shutdown too) and refuses the repair while still reporting correctly — so the abort flag is read off the RAW mount in the stale case.

Live validation, demo-hp

State	raw	bind	verdict	usable
healthy	8:16	8:16	live	true
R-117 return	8:32 /dev/sdc	8:16 …,shutdown	stale-device	false
Q7 in-place abort	252:11 …,emergency_ro	252:11 …,emergency_ro	filesystem-aborted	false

340–497 µs per call. No block I/O proven by strace: only /proc/self/mountinfo, 0 statfs. No regression through the real pipeline: GET /disks with the controller's own credential shows the live backup-target drive bound_under_parent=True, with 32 gate lines in 3 min as the positive observable and zero spurious transitions.

Tests + red-proofs

14 new tests; the payload ones drive the real /disks handler and the AttachDrive ones record the actual privileged calls. Fixtures are the spike's captured mountinfo, injected as data via procSelfMountinfo/procGuestMountinfo, so the real parser, predicate, AttachDrive and handler all run.

6 red-proofs, each verified to have landed. RP1/RP2 (term 3 removed per site), RP3 (the P1-only fix), RP4 (emergency_ro dropped), RP5 (unknown reported dead), RP6 (aborted arm falls through to a re-bind).

A hollow test was caught, by RP1 failing to fail. The aborted fixture first used a /dev/mapper device, for which RoleForStorage derives role="system" — a system row never enters the block that computes BoundUnderParent, so the field stayed false by default, the assertion passed, and no mutation could ever fail it. Fixtures now assert the production row shape before asserting the field.

Not yet done / not covered

• The stale-bind repair on hardware. StablePathForRaw hardcodes the production /mnt/felhom-drives, so exercising AttachDrive for a scratch drive writes a submount into guest 9201's namespace, which the task fences. It happened once inadvertently, was reverted within a minute and 9201 verified unaffected, and was deliberately not repeated. Filed R-117h.
• Sustained-load behaviour of the predicate — flagged unmeasured by the spike and still unmeasured.
• felhom-pve not upgraded (0.115.0). The fleet is non-uniform.
• R-117g — an aborted filesystem is never cleared automatically, by design, so it alarms until a human acts; there is no guided recovery.
• R-121 — a box's installed agent can sit releases behind the vouched one and nothing notices; the R-120 gate is golden-vs-fleet-controller and cannot see it.

Observations (not acted on)

Guest 9201 has 2 stacked entries at /mnt/felhom-drives/nvme-1tb while the host has 1 (pre-existing; AttachDrive normalizes the host count only). 14 stale felhom-agent.bak-* binaries on demo-hp (~200 MB), nothing prunes them. 11 files in this repo fail gofmt -l, all pre-existing and none in internal/localapi/.