felhom.eu/documentation/audits/R117-v0117-2026-07-30.md

R-117 v0.117.0 — the liveness signal now tests liveness, proven on hardware

Implementation + live validation of audits/SPIKE-r117-bind-liveness-2026-07-30.md. Agent v0.117.0. felhom-controller untouched: the spike proved the repair is already wired from the controller's Return branch and dies inside the agent.

---

1. Baselines — each established separately

Thing	Value	How
agent main before	v0.116.0 @ d4eb259	git log, CHANGELOG.md:1; tree clean, HEAD == origin/main
agent shipped this run	v0.117.0 @ 966d8f4	commit below
agent on demo-hp BEFORE	0.113.0	felhom-agent --version on the box — predates R-113 (0.114.0) entirely
agent on demo-hp AFTER	0.117.0	same command, post-deploy
agent published + vouched before	published 0.113–0.116; vouched 0.116.0 (b47c5c4dab641ee5…)	hub /configuration, read not assumed
controller main	v0.186.0 @ b331f18	git log
golden bakes controller	0.186.0	hub /configuration — matches main, so R-120's gate is holding them together ✅
min_agent / min_controller_version	0.113.0 / 0.156.0	hub /configuration
hub live	0.82.0	kubectl -n felhom-system get deploy
felhom.eu HEAD before	e70b5fe	as the task expected
demo-hp	PVE 9.2.2, kernel 7.0.2-6-pve	pveversion

---

2. Part 1 — the two record items, committed before any Go file was opened

felhom.eu 37515cd — docs(R-117 Part 1). felhom-agent 6be168d — the same rule repeated where it binds.

2.1 The CLAUDE.md rule, as written

Placed beside the seam-wiring rule in felhom.eu/CLAUDE.md "Code quality rules":

A health check issues no block I/O. A probe that touches a wedged device enters uninterruptible sleep, survives SIGKILL, and cannot be recovered until the device returns or the host reboots — so systemctl restart hangs too. A timeout protects the caller's control flow and nothing else: the blocked thread remains. Liveness is decided from /proc and the kernel's own state, never by reading or writing the filesystem. Measured, R-117 spike §6.3 (…): a probe stayed in D state 3m50s after kill -9; a buffered write with no fsync blocked too (O_CREAT needs journal access); and statfs/getdents returned healthy on a namespace that EIOs every byte — fast, and wrong.

Repeated in felhom-agent/CLAUDE.md, as a one-line pointer, and the reason is recorded there: health checks are written in the agent repo, and felhom.eu/CLAUDE.md does not load in an agent-only session. A standing rule that does not load where it binds is the inert-seam shape applied to a rule. The task named one location; this is the one deviation, stated rather than absorbed (standing rule 4).

2.2 The R-116 narrowing, applied

The R-116 row's over-correction clause now carries: the 2 RETURNED lines are a genuine positive observable so rule 3 is satisfied, but degraded:false was read off a drive whose bind was dead, so the window evidences "the gate did not over-fire" and not "the drive was healthy." Nothing else about the row changed.

---

3. THE RULING (task §2.2) — what happens when P2 trips

Ruling: P2 must NOT self-heal. It surfaces. Operator decision NOT required — and the reasoning for that is the substantive part, so it is stated in full for overruling.

The two states have different repairs, and "would a re-bind help?" — not "is something aborted?" — is the question a verdict must answer:

Verdict	What a re-bind would do	Ruling
BindStaleDevice	the raw mount is a different, healthy superblock (the drive returned and the fs-UUID-keyed unit healed onto it), so umount + re-bind lands the namespace on working storage	REPAIR. Falls through to the existing normalize leg
BindAborted	the raw mount is the same aborted superblock — a fresh bind to a still-dead filesystem	QUIET NO-OP, and surface via BoundUnderParent=false

Why BindAborted must not repair, three reasons and the third is the decisive one:

1. It cannot work. Re-binding does not clear an ext4 abort; that needs a remount or a fsck.
2. AttachDrive runs every 20 s (cmd/felhom-agent/main.go ticker), so re-binding would be an infinite silent retry — R-117a's silence with more CPU.
3. It would mask the state. The whole point of the fix is that this state becomes visible.

Why no operator decision was needed. The ruling changes customer-visible behaviour: apps that today keep running against a read-only/EIO namespace will be stopped, and the customer gets an alarm. That is a real change — but it is the identical treatment an unplugged drive already receives, through the existing gate, the existing event types and the existing Hungarian copy. No new message is invented, no new concept reaches the customer. The alternative is letting applications keep writing a customer's documents into a filesystem that rejects every write. Fail-closed is also the project's standing direction for drive state. So this is routing an already-broken state into an already-tested alarm, not a new customer-facing decision. Flagged prominently rather than decided quietly; overrule freely.

Deliberately NOT done: automatic remount of an aborted filesystem. That is a repair of the customer's data-bearing filesystem, risks loss on a dirty abort, and is an operator call. Recorded as R-117g.

---

4. The predicate as implemented

felhom-agent/internal/localapi/intermediary.go:

bindLiveness(stable, raw) →
  stable=="" or raw==""            → BindUnknown
  no bind entry at `stable`        → BindUnknown   (isHostMountpoint's question, not this one)
  no raw entry at `raw`            → BindUnknown   (devicePresent already reports device absence)
  bind.devno != raw.devno          → raw aborted ? BindAborted : BindStaleDevice     ← P1
  (devnos agree) bind aborted      → BindAborted                                      ← P2
  (devnos agree) fstype unknown    → BindUnknown
  otherwise                        → BindLive

Wired as the third term of the conjunction at both /disks construction sites (disks.go): boundUnderParent(...) && devicePresent(...) && bindUsable(gp, <raw>).

Order is load-bearing, and a test caught it. The first draft read the abort flag before comparing devices. In the real return state the stale bind carries shutdown as well as a different device, so abort-first classifies it BindAborted — reporting stays correct, every payload test still passes, and the repair silently never runs. The abort flag is therefore read off the raw mount in the stale case (the re-bind's target) and off the bind only once the devices already agree.

4.1 How unknown is expressed

A three-state type, never a bool: BindLiveness ∈ {BindUnknown, BindLive, BindStaleDevice, BindAborted}, with BindUnknown as the zero value. Every caller reads it through one method:

func (l BindLiveness) Usable() bool { return l == BindLive || l == BindUnknown }

So the "cannot tell → never absent" rule lives in exactly one place and no caller can restate it wrongly. The task named the newestArchiveOn trap — a (value, bool) shape whose comment promised a degradation the signature could not express. Four routes to unknown are pinned by test: unreadable /proc, no raw entry, empty paths, and a filesystem whose abort vocabulary is unmeasured.

P2's tokens: shutdown AND emergency_ro, both measured. ext4/ext3/ext2 (one driver); anything else → unknown, never live.

4.2 Not a new recovery path

AttachDrive's early return now switches on the verdict instead of declaring a dead namespace "fully live, no-op". The three call sites that already invoked the repair — the 20 s ticker, agent startup, and the controller's Return branch before restartStacks — are unchanged. Also: isHostMountpoint and countHostMounts are now one-liners over a single new parser hostMountEntries, which yields devno/root/fstype/super-options (REUSE.md's three-independent-readers note narrowed).

---

5. Tests

849 → 863 (+14 top-level), 29/29 packages green, go build / go vet / go test each with rc=0 read separately (never combined with a commit). The 849 baseline was verified against a temporary worktree at HEAD, not inferred — the first draft of the CHANGELOG line said 858 → 873 from a count taken mid-edit, and that was corrected before commit.

Test	Asserts (the consequence)	Result
TestDisks_BindLiveness_StaleBindReadsAbsent	R-117 case (a) → bound_under_parent false through the real /disks handler	PASS
TestDisks_BindLiveness_AbortedFilesystemReadsAbsent (×2 tokens)	R-117a case (b) → false, with a row-shape guard	PASS
TestDisks_BindLiveness_UnionPath_StaleBindReadsAbsent	union path, case (a)	PASS
TestDisks_BindLiveness_UnionPath_AbortedReadsAbsent	union path, case (b) — Role+State hardcoded there	PASS
TestDisks_BindLiveness_HealthyReadsPresent (observe+union)	no false negative	PASS
TestDisks_BindLiveness_UnknownIsTreatedAsPresent (5 subtests)	all four unknown routes → present; Usable() truth table	PASS
TestBindLiveness_Verdicts (4 states)	each verdict	PASS
TestBindLiveness_AbortedWins_WhenDevnosAgree	the P1-only fix cannot pass	PASS
TestHostMountEntries_ParsesDevnoAndSuperOpts	" - "-separator field extraction	PASS
TestHostMountEntries_CountsStackedBinds	double-bind convergence survives the refactor	PASS
TestAttachDrive_StaleBind_Rebinds	the repair RUNS (umount+rebind recorded)	PASS
TestAttachDrive_AbortedFilesystem_DoesNotRebind	§3's ruling: zero mount ops, no error	PASS
TestAttachDrive_Healthy_IsStillANoOp	20 s idempotency	PASS
TestAttachDrive_UnknownLiveness_IsANoOp	cannot-tell must not churn	PASS

Fixtures are the spike's captured mountinfo, with two substitutions both recorded in the file: the scratch shared parent mapped to the production StableParentDir (the code derives it), and the aborted state transposed from dm onto the USB shape (§5.2 below). Tests redirect procSelfMountinfo / procGuestMountinfo at fixture files, so the real parser, the real predicate, the real AttachDrive and the real /disks handler all run — the data is injected, the logic is not.

5.1 Red-proofs — 6, each verified to have landed

#	Mutation	Landing verified by	Fails
RP1	term 3 removed at the Observe site	bindUsable(gp, t.MountPath) occurrences → 0	StaleBind + Aborted(×2)
RP2	term 3 removed at the union site	bindUsable(gp, d.MountPath) → 0	both UnionPath tests
RP3	the P1-only fix — abort check neutered	RED-PROOF RP3 marker present	Aborted(×2), UnionPath_Aborted, Verdicts(×2), AbortedWins
RP4	emergency_ro dropped from the token list	RED-PROOF RP4 present	only the emergency_ro subtests — the "only shutdown" fix
RP5	unknown reported as not-live	RED-PROOF RP5 present	UnknownIsTreatedAsPresent/unreadable
RP6	aborted arm falls through to the re-bind	RED-PROOF RP6 present	AttachDrive_AbortedFilesystem_DoesNotRebind (go test rc=1, read from go test itself)

5.2 A HOLLOW TEST WAS CAUGHT — by RP1 failing to fail

TestDisks_BindLiveness_AbortedFilesystemReadsAbsent first used the spike's dm capture verbatim (/dev/mapper/r117cel). RoleForStorage derives role="system" for that backing device — a system-role row never enters the block that computes BoundUnderParent, so the field stayed false by default, the assertion passed, and no mutation could ever fail it. Pure decoration, and it is exactly R-116's failure mode (a fixture supplying a shape production never emits).

Found because RP1 removed term 3 and the test still passed. Fixed by transposing the aborted fixture onto the USB drive shape (an in-place abort on a USB drive is the realistic case anyway; only the super options and the matching devnos carry the claim) and by adding a guard that fails loudly if the row is not the production shape:

if di.Role != "user-data" || di.GuestPath == "" {
    t.Fatalf("fixture does not reproduce the production row shape: role=%q guest_path=%q — "+
        "the conjunction never runs on such a row, so any assertion below is vacuous", di.Role, di.GuestPath)
}

The red-proof did not merely confirm the test; it is the only thing that found the test was empty.

---

6. Build, publish, vouch — three observables, quoted

1. Published — the anonymous GET the installer performs, no credentials:

GET .../api/packages/admin/generic/felhom-agent/0.117.0/felhom-agent
http=200 bytes=14032908
downloaded sha256: e6768a12eba1c9e0743367f04589845330151ac63d10caad794e8a4c6f36f735
built      sha256: e6768a12eba1c9e0743367f04589845330151ac63d10caad794e8a4c6f36f735
MATCH — published bytes are the built bytes
$ /tmp/agent-dl-0.117.0 --version  →  felhom-agent 0.117.0

2. Vouched — POST through the real operator endpoint, then the manifest read back (not the flash):

POST /configuration/artifacts → HTTP/1.1 303, Location: /configuration?flash=artifacts_set
hub log: [INFO] Artifact manifest set: agent=0.117.0 golden=0.186.0 min_agent="0.113.0" wrapper_sha=true

READ BACK:  agent_version=0.117.0   agent_sha256=e6768a12eba1c9e0…
            golden_version=0.186.0  golden_sha256=b760ac6a33e70700…   (preserved)
            min_agent=0.113.0       wrapper_sha256=104db0a4401f65bb…  (preserved)

R-120's gate did not refuse: golden 0.186.0 is not behind the fleet's newest controller (0.186.0).

3. Running on the box under test:

BEFORE: felhom-agent 0.113.0   active
AFTER:  felhom-agent 0.117.0   active
capabilities self-check ok=68 total=68 degraded=0 inactive=0
reconcile: enrolled drive bound under shared parent (live, no reboot) vmid=9201 …  ← loop ticking, new pid

Zero ERROR lines. configs/ was untouched by this commit, so a binary-only deploy is correct (verified with git show --name-only).

---

7. §5.1 — demo-hp brought current, and the drift the R-120 gate does not cover

demo-hp: 0.113.0 → 0.117.0, prior binary backed up to felhom-agent.bak-0.113.0.

Does the R-120 vouch gate cover agent-on-a-box drift? NO — confirmed at source. hub/internal/web/configs.go:1165-1169 compares goldenVer against store.NewestReportedControllerVersion(): a golden-artifact vs fleet-CONTROLLER check. It says nothing about the agent installed on a box. MinAgent does not cover it either — it holds the controller floor for a box whose agent is too old (hub/internal/api/handler.go:530-538, store.go:1857), which is protective, not an alarm — and demo-hp's 0.113.0 equalled min_agent 0.113.0, so even a floor comparison was satisfied.

Filed as R-121 — the fourth instance of the drift family (R-111, R-115, R-120, and now installed-vs-vouched). Its measured cost: R-117's whole subject is the R-113 conjunction, which landed in 0.114.0, so the designated drill host could not exercise the code under investigation at all, and the spike had to route every predicate result through an out-of-repo probe built from main.

---

8. §5.2 — the live run

Venue: demo-hp, the spike's §10 recipe. Scratch LXC 9301 (no NIC) on a scratch dir storage r117scratch — never local-lvm, never felhom-backup. Scratch shared parent /mnt/r117-drives, so the live /mnt/felhom-drives peer group (guest 9201's) was not used. scsi_debug for the drive, because it is the only mechanism that removes a gendisk while mounted. Guest parent bind verified master:209 — a real slave of the host peer group.

Method: a throwaway probe built from the shipped tree at 966d8f4 (exported in a copy outside both repos), so bindLiveness and AttachDrive are the real functions, judging real kernel state.

8.1 The predicate, every state, on hardware

State	host raw	host bind	verdict	Usable()	term 3	cost
HEALTHY (scsi)	8:16 rw,stripe=512	8:16 rw,stripe=512	live	true	true	442 µs
after real gendisk removal	(NOT MOUNTED)	8:16 …,shutdown	unknown	true	true	340 µs
R-117 RETURN	8:32 (/dev/sdc)	8:16 …,shutdown	stale-device	false	false	365 µs
HEALTHY (dm)	252:11 rw	252:11 rw	live	true	true	359 µs
Q7 IN-PLACE ABORT	252:11 …,emergency_ro	252:11 …,emergency_ro	filesystem-aborted	false	false	497 µs

Both dead states are detected on real hardware. Note the two rows that matter most:

• the RETURN row is the exact defect — sdb→sdc, the bind left on the dead superblock;
• the IN-PLACE row has identical devnos on both sides (the device never left) and the raw mount still active — so the device comparison alone reads healthy there, and only the abort token catches it. This is R-117a, the half that emits nothing today.

The absent row is correct by design and worth stating: term 3 answers unknown because devicePresent is the term that reports device absence, so term 3 abstains rather than double-counting. The overall conjunction is false via term 2. It also demonstrates the cannot-tell rule working in a real state.

Cost: 340–497 µs, against 3.85–5.03 ms that GuestSeesMount already spends forking lxc-info.

8.2 No block I/O — the Part 1 fence applied to its own first consumer

strace -f -e trace=openat,statfs,fstatfs,read,pread64 while the predicate judged the aborted drive:

distinct openat targets:  "/etc/localtime"  "/proc/self/cgroup"  "/proc/self/mountinfo"
                          "/sys/.../cpu.max"  "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size"
openat/statfs against /mnt/r117*:  NONE — the predicate never touched the mount
statfs syscalls: 0

8.3 No regression on the live customer drive — through the REAL server pipeline

GET /disks over the island bridge with the controller's own credential (from the guest's /etc/felhom-bootstrap/bootstrap.json), i.e. the exact call the controller makes:

GET /disks http=200 bytes=2545     vmid=9201
name=felhom-backup  state=attached  role=user-data
guest_path=/mnt/felhom-drives/nvme-1tb   bound_under_parent=True   backup_target=True

The live enrolled drive — which is also the whole-guest backup target — reads present under 0.117.0. Host-side corroboration: exactly 1 bind entry at the stable path, 259:0 on both raw and bind, no abort token ⇒ BindLive. And a positive observable that the consumer was actually evaluating (rule 3): 32 gate-related controller log lines in 3 minutes, with zero ABSENT/RETURNED/backup_target transitions, and zero bind is STALE / normalized to one bind lines from the agent — so the new predicate disturbed nothing.

8.4 AttachDrive on hardware — and the one thing that was NOT repeated

The healthy case ran through the real AttachDrive on the box: it returned the stable path, and the guest init PID was identical before and after (2555744) — the guest was never restarted.

AttachDrive derives its target from StablePathForRaw, which hardcodes the production /mnt/felhom-drives. So invoking it for a scratch drive necessarily creates a submount under the live customer parent, which propagates into guest 9201 — which this task fences. That happened once, inadvertently, during the healthy case: a bind appeared at /mnt/felhom-drives/r117sd. It was removed within a minute and the revert was verified, not assumed:

live parent restored to exactly its two real submounts (nvme-1tb, Felhom-Share)
9201: running, init pid unchanged (never restarted)
stale scratch entries in 9201's namespace: 0
9201 guest view: Felhom-Share  nvme-1tb          9201 live drive read: OK

It was deliberately NOT repeated for the stale-bind case, so the repair leg is proven by TestAttachDrive_StaleBind_Rebinds + RP6 through the real code, and on hardware only for the healthy no-op. Doing it live would need ~20 s of a scratch submount inside guest 9201's namespace — harmless on the evidence above, but the fence names 9201, so it is the operator's call, not one to take quietly. Named as the one gap in this run's hardware coverage.

---

9. Teardown — all three layers

Layer 1 — the machine. pct stop 9301 + pct destroy 9301 --purge; pct list back to 9201 alone.

Layer 2 — the host.

	Before	After
pvesm status	r117scratch active, local 39.03 %	r117scratch removed, local 37.07 % (session start 37.02 %)
df -h /var/lib/vz	16G, 42 %	15G, 40 % (session start 40 %)
dmsetup ls	r117cel (252:11) + 11 pre-existing	gone; the 11 pve-* unchanged
losetup -a	2 loops	none
scsi_debug	loaded	unloaded; /dev/sd* back to sda1..3
/mnt	+ r117-drives, r117cel, r117sd	felhom-drives, nvme-1tb, vzsnap0 — as found
/root, /var/lib/vz/r117*	5 scripts + r117/ + probe	ls -d /mnt/r117* /root/r117* /var/lib/vz/r117* → NONE

The 124 MB template was removed. One teardown defect, recorded because it is a reusable trap: the script hardcoded losetup -d /dev/loop0, but the loop numbers swapped this run (the guest rootfs took loop0), so it detached the wrong device and left /dev/loop1 attached to a deleted file. Fixed by selecting loops by backing file (losetup -a | grep r117), not by number. local is 0.05 % above the session start (~20 MB) — journal growth from the run, not a leftover artifact.

Layer 3 — the hub: one record was written, deliberately, and it is retained. No customer, appliance, pairing or claim was created (9301 had no NIC and ran no controller). The one hub-side change is the artifact manifest, now vouching agent 0.117.0 — the intended product of §6, not scratch, so it is retained and is the fleet's vouched agent going forward.

Fences — re-verified after teardown, not assumed:

Fence	State
guest 9201	running throughout; guest view Felhom-Share nvme-1tb; live drive read OK
drill-r50 (VM 300)	stopped — never started
local-lvm	38.84 % — byte-identical before, after, and to the session start
felhom-backup	active, 0.46 %, content backup unchanged — not re-targeted, not written
live /mnt/felhom-drives	both real submounts present; no r117* leftovers
felhom-agent on demo-hp	active, 0.117.0
demo-felhom, Peti's cluster	never contacted
felhom-controller	untouched, as scoped

---

10. Findings — filed as R-n, none fixed

#	Finding	Severity
R-121	A box's installed agent can sit releases behind the vouched one and nothing notices. demo-hp ran 0.113.0 against a vouched 0.116.0 through the whole R-116/R-117 arc. The R-120 gate is golden-vs-fleet-controller and cannot see it; MinAgent is protective, not an alarm, and 0.113.0 equalled the floor. Fourth instance of the drift family	MEDIUM — filed with a fix shape and the honest tension (a box legitimately lags between publish and deploy)
R-117g	An aborted filesystem is never cleared automatically, by design (§3), so a drive stuck in emergency_ro alarms forever until a human acts. The alarm is correct and new; the absence of any guided recovery is the gap. A remount risks loss on a dirty abort, so it wants an operator-confirmed action, not a self-heal	LOW-MEDIUM (follow-up to this fix)
R-117h	AttachDrive cannot be exercised against a non-production shared parent, because StablePathForRaw hardcodes StableParentDir. Any hardware test of the repair path writes a submount into the live customer guest's namespace (§8.4). A test seam for the parent dir would make the repair leg testable on hardware without touching a customer	LOW (testability)

Observations — noticed, documented, NOT acted on:

• Guest 9201 carries 2 stacked entries at /mnt/felhom-drives/nvme-1tb while the host has exactly 1. Pre-existing, unrelated to this change, and harmless to the predicate (it reads the host table, and bindLiveness iterates all entries). Worth a look — AttachDrive normalizes the host count only.
• 14 stale felhom-agent.bak-* binaries on demo-hp (0.92.1 … 0.113.0), ~200 MB on pve-root. Nothing prunes them.
• 11 files in the agent repo fail gofmt -l, all pre-existing (internal/capability/probe.go, internal/storage/intent.go, …). None are in internal/localapi/; none were touched here.

---

11. What did not happen, and why

• The stale-bind repair was not exercised on hardware (§8.4) — it would put a scratch submount inside guest 9201's namespace, which this task fences. Covered by TestAttachDrive_StaleBind_Rebinds + RP6 through the real code, and by the healthy-case AttachDrive on hardware (init PID identical).
• No load or duration measurement of the predicate under a sustained reconcile loop. The spike flagged this as unmeasured and it is still unmeasured — said plainly rather than let to vanish. What exists: single-call costs of 340–497 µs on hardware, and ~15 minutes of the real 20 s ticker on demo-hp with no spurious re-bind and no CPU complaint. A many-drive box under sustained load was not tested.
• felhom-controller was not touched, per scope — the repair is already wired from its Return branch.
• The R-116 row was not edited beyond §2.2's clause.
• felhom-pve was not upgraded — it runs 0.115.0 and the task scoped the venue to demo-hp. The fleet is now non-uniform: demo-hp 0.117.0, felhom-pve 0.115.0. Flagged, not acted on.
• R-118 was not addressed (the absent drive's union row advertising the root filesystem's capacity) — a separate row, untouched by this fix.

---

Every claim above is quoted from pushed Gitea source at file:line (agent 966d8f4, hub 0.82.0) or is captured run output.