admin
1913e12031
Predicate validated on hardware for both dead states (stale-device and filesystem-aborted), 340-497us per call, no block I/O proven by strace (only /proc/self/mountinfo, 0 statfs). No regression through the real pipeline: the live backup-target drive reads bound_under_parent=True via GET /disks with the controller's own credential. Records what was NOT covered: the stale-bind repair on hardware (StablePathForRaw hardcodes the live parent, so it would write into guest 9201's namespace - R-117h), and sustained-load behaviour, still unmeasured.
104 lines
6.3 KiB
Markdown
104 lines
6.3 KiB
Markdown
# REPORT — v0.117.0: R-117, the liveness signal now tests liveness (2026-07-30)
|
||
|
||
**Overwritten** per the standing rule. Full evidence:
|
||
`felhom.eu/documentation/audits/R117-v0117-2026-07-30.md`. Spike this builds on:
|
||
`audits/SPIKE-r117-bind-liveness-2026-07-30.md`.
|
||
|
||
| | |
|
||
|---|---|
|
||
| agent | 0.116.0 → **0.117.0** @ `966d8f4` |
|
||
| published + vouched | **0.117.0**, sha `e6768a12eba1c9e0…` (anonymous GET verified, manifest read back) |
|
||
| deployed | **demo-hp 0.117.0** (from **0.113.0** — see R-121) · felhom-pve still **0.115.0**, out of scope |
|
||
| controller / golden / hub | 0.186.0 / 0.186.0 / 0.82.0 — **untouched** |
|
||
| tests | **849 → 863**, 29/29 packages, `build`/`vet`/`test` each `rc=0` read separately |
|
||
|
||
## The defect
|
||
|
||
`BoundUnderParent` reported a namespace returning `EIO` on every read and write as healthy, and the gate
|
||
restarted the customer's apps onto it. Both existing terms — `GuestSeesMount` and `isHostMountpoint` —
|
||
parse a mountinfo line and then test only `fields[4]`, the mount POINT. **Field 3, `major:minor`, sat in
|
||
the same parsed slice and was discarded.**
|
||
|
||
## The fix — one predicate, both halves mandatory
|
||
|
||
`bindLiveness` (`internal/localapi/intermediary.go`) becomes the third term of the conjunction at both
|
||
`/disks` construction sites. `/proc` only, **no block I/O**:
|
||
|
||
- **the bind must name the same device as the raw mount** — sound, not heuristic: a stale bind pins the
|
||
dead superblock, which keeps the old device index allocated, which forces the returning device onto a
|
||
different number;
|
||
- **the filesystem must not have aborted** — `shutdown` or `emergency_ro`, both measured.
|
||
|
||
**The second is not optional.** A device that fails *without disappearing* gives the identical
|
||
all-signals-healthy state with the **devnos equal** and the drive never `Disconnected`, so the gate emits
|
||
nothing at all, indefinitely (R-117a). A devno-only fix passes every payload test; red-proof RP3 exists for
|
||
exactly that.
|
||
|
||
**Three states, never a bool:** `{Unknown, Live, StaleDevice, Aborted}`, `Unknown` the zero value, and
|
||
`Usable()` is the single place where unknown counts as **present** — absent stops a working customer's apps.
|
||
|
||
## The ruling (task §2.2) — and it is flagged for overrule
|
||
|
||
`StaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, guest never restarts).
|
||
`Aborted` ⇒ **quiet no-op, and surface via `BoundUnderParent=false`** — a re-bind lands on the *same* dead
|
||
superblock and this runs every 20 s, so repairing would be an infinite silent retry that also masks the
|
||
state. **No operator decision taken quietly:** it does change customer-visible behaviour (apps stop and an
|
||
alarm fires where today nothing happens), but it routes an already-broken state into the **existing** gate,
|
||
event types and Hungarian copy — no new concept reaches the customer — and the alternative is applications
|
||
writing documents into a filesystem that rejects every write.
|
||
|
||
**No new recovery path.** `AttachDrive`'s normalize leg already did the repair and three call sites already
|
||
invoked it (20 s ticker, agent startup, and the controller's `Return` branch **before** `restartStacks`).
|
||
All three died on `if n == 1 && GuestSeesMount(...)` logging *"fully live, no-op"* about a dead namespace.
|
||
|
||
**Ordering trap, caught by a test:** reading the abort flag before comparing devices classifies the real
|
||
return state as aborted (its stale bind carries `shutdown` too) and refuses the repair **while still
|
||
reporting correctly** — so the abort flag is read off the RAW mount in the stale case.
|
||
|
||
## Live validation, demo-hp
|
||
|
||
| State | raw | bind | verdict | usable |
|
||
|---|---|---|---|---|
|
||
| healthy | `8:16` | `8:16` | `live` | true |
|
||
| **R-117 return** | `8:32` `/dev/sdc` | `8:16 …,shutdown` | **`stale-device`** | **false** |
|
||
| **Q7 in-place abort** | `252:11 …,emergency_ro` | `252:11 …,emergency_ro` | **`filesystem-aborted`** | **false** |
|
||
|
||
340–497 µs per call. **No block I/O proven by `strace`**: only `/proc/self/mountinfo`, **0** `statfs`.
|
||
**No regression through the real pipeline:** `GET /disks` with the controller's own credential shows the
|
||
live backup-target drive `bound_under_parent=True`, with 32 gate lines in 3 min as the positive observable
|
||
and zero spurious transitions.
|
||
|
||
## Tests + red-proofs
|
||
|
||
14 new tests; the payload ones drive the real `/disks` handler and the `AttachDrive` ones record the actual
|
||
privileged calls. Fixtures are the spike's captured mountinfo, injected as **data** via
|
||
`procSelfMountinfo`/`procGuestMountinfo`, so the real parser, predicate, `AttachDrive` and handler all run.
|
||
|
||
**6 red-proofs, each verified to have landed.** RP1/RP2 (term 3 removed per site), RP3 (**the P1-only
|
||
fix**), RP4 (`emergency_ro` dropped), RP5 (unknown reported dead), RP6 (aborted arm falls through to a
|
||
re-bind).
|
||
|
||
**A hollow test was caught, by RP1 failing to fail.** The aborted fixture first used a `/dev/mapper`
|
||
device, for which `RoleForStorage` derives `role="system"` — a system row never enters the block that
|
||
computes `BoundUnderParent`, so the field stayed false **by default**, the assertion passed, and no mutation
|
||
could ever fail it. Fixtures now assert the production row shape before asserting the field.
|
||
|
||
## Not yet done / not covered
|
||
|
||
- **The stale-bind repair on hardware.** `StablePathForRaw` hardcodes the production
|
||
`/mnt/felhom-drives`, so exercising `AttachDrive` for a scratch drive writes a submount into guest
|
||
9201's namespace, which the task fences. It happened once inadvertently, was reverted within a minute
|
||
and 9201 verified unaffected, and was deliberately not repeated. Filed **R-117h**.
|
||
- **Sustained-load behaviour of the predicate** — flagged unmeasured by the spike and **still unmeasured**.
|
||
- **felhom-pve not upgraded** (0.115.0). The fleet is non-uniform.
|
||
- **R-117g** — an aborted filesystem is never cleared automatically, by design, so it alarms until a human
|
||
acts; there is no guided recovery.
|
||
- **R-121** — a box's installed agent can sit releases behind the vouched one and nothing notices; the
|
||
R-120 gate is golden-vs-fleet-controller and cannot see it.
|
||
|
||
## Observations (not acted on)
|
||
|
||
Guest 9201 has 2 stacked entries at `/mnt/felhom-drives/nvme-1tb` while the host has 1 (pre-existing;
|
||
`AttachDrive` normalizes the host count only). 14 stale `felhom-agent.bak-*` binaries on demo-hp (~200 MB),
|
||
nothing prunes them. 11 files in this repo fail `gofmt -l`, all pre-existing and none in `internal/localapi/`.
|