diff --git a/REPORT.md b/REPORT.md index 680ce4f..0bd20ec 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,143 +1,103 @@ -# REPORT — v0.116.0: R-116, the flag and the key share a row (2026-07-30) +# REPORT — v0.117.0: R-117, the liveness signal now tests liveness (2026-07-30) -**Overwritten** per the standing rule. Agent-only fix: the controller was **not** touched (one fix, one -place) and `v0.115.0` was **not** reverted. Mechanism evidence: `felhom.eu` -`audits/DIAG-r116-disks-payload-2026-07-30.md`. +**Overwritten** per the standing rule. Full evidence: +`felhom.eu/documentation/audits/R117-v0117-2026-07-30.md`. Spike this builds on: +`audits/SPIKE-r117-bind-liveness-2026-07-30.md`. -## Baselines as actually running (not as on `main`) +| | | +|---|---| +| agent | 0.116.0 → **0.117.0** @ `966d8f4` | +| published + vouched | **0.117.0**, sha `e6768a12eba1c9e0…` (anonymous GET verified, manifest read back) | +| deployed | **demo-hp 0.117.0** (from **0.113.0** — see R-121) · felhom-pve still **0.115.0**, out of scope | +| controller / golden / hub | 0.186.0 / 0.186.0 / 0.82.0 — **untouched** | +| tests | **849 → 863**, 29/29 packages, `build`/`vet`/`test` each `rc=0` read separately | -| | `main` | running | a FRESH box gets (hub Day-0 manifest) | -|---|---|---|---| -| agent | 0.115.0 → **0.116.0** | felhom-pve **0.115.0**, demo-hp **0.113.0** | **0.115.0** | -| controller | 0.186.0 | felhom-pve **0.186.0**, demo-hp **0.185.1** | golden bakes **0.185.1** | -| hub | 0.81.0 | 0.81.0 | `min_agent` 0.113.0, `min_controller` 0.156.0 | -| `felhom.eu` | `1aa1bd1` | — | — | +## The defect -**The fleet is not uniform on either component.** The golden baking controller **0.185.1** while -demo-felhom runs **0.186.0** is load-bearing for the ruling below: a controller-side fix would not reach -a fresh box without a re-bake, whereas the agent channel already serves the newest published version. +`BoundUnderParent` reported a namespace returning `EIO` on every read and write as healthy, and the gate +restarted the customer's apps onto it. Both existing terms — `GuestSeesMount` and `isHostMountpoint` — +parse a mountinfo line and then test only `fields[4]`, the mount POINT. **Field 3, `major:minor`, sat in +the same parsed slice and was discarded.** -## The ruling: neither (a) nor (b) — a hybrid, because both regress R-114 +## The fix — one predicate, both halves mandatory -`primaryTargetID` is a PVE storage name (`felhom-backup`); the registry row's `Name` is the drive UUID. -With the device gone the two records of one drive share **no runtime field** — no mount, no backing -device, and the Observe row's `DurableID` has degraded off the fs-UUID to `path:/mnt/cel`. +`bindLiveness` (`internal/localapi/intermediary.go`) becomes the third term of the conjunction at both +`/disks` construction sites. `/proc` only, **no block I/O**: -**Rejected (a) — back-fill `MountPath` on the Observe row.** The spec's lean, and it does re-activate -v0.115.0. Rejected on source evidence: `backup_target_offer.go:79` reads -`d.BackupTarget && d.MountPath != ""` as *"a real drive with its own mountpoint — healthy"* and returns -**before** its `TargetAbsent` branch. Option (a) manufactures exactly that row while the drive is -missing, so it would have silently flipped **R-114** — shipped 2026-07-29, proven live — back to a false -*healthy* banner. Also `MountPath`'s own contract is "the RAW host PVE mount" (`disks.go:152-153`); a path -that is not mounted is not that, so the downstream reading is correct and the change would be the bug. +- **the bind must name the same device as the raw mount** — sound, not heuristic: a stale bind pins the + dead superblock, which keeps the old device index allocated, which forces the returning device onto a + different number; +- **the filesystem must not have aborted** — `shutdown` or `emergency_ro`, both measured. -**Rejected (b) — carry `BackupTarget` on the registry row.** A join **does** exist (below), so this was -available — but the registry row's `MountPath` is non-empty, read from the by-then-stale `.mount` unit -file. It trips `:79` identically. Same regression, same reason. +**The second is not optional.** A device that fails *without disappearing* gives the identical +all-signals-healthy state with the **devnos equal** and the drive never `Disconnected`, so the gate emits +nothing at all, indefinitely (R-117a). A devno-only fix passes every payload test; red-proof RP3 exists for +exactly that. -**Rejected (c) — resolve controller-side.** The mapping is not there to find, and the *one state, one -owner* ruling stands: the agent owns drive identity. +**Three states, never a bool:** `{Unknown, Live, StaleDevice, Aborted}`, `Unknown` the zero value, and +`Usable()` is the single place where unknown counts as **present** — absent stops a working customer's apps. -**Taken (d) — give the Observe row the guest path (and NO MountPath), then dedup the registry row on -guest path.** The gate keys on **GuestPath**; R-114 keys on **MountPath**. Separating them satisfies both: -the surviving row carries the flag and the guest path with `mount_path: ""`, so `isTarget[guestPath]` is -`true` **and** `:79` does not match, leaving R-114's `TargetAbsent` branch reachable. +## The ruling (task §2.2) — and it is flagged for overrule -**The join:** with the device gone the shared identity is **configuration**, not state — the storage's -`path` from `storage.cfg` (Observe side) and the `.mount` unit's `Where` (registry side) are the same -path, so both derive the same stable guest path. That is the key both sides can still compute. +`StaleDevice` ⇒ **re-bind** (the raw mount is a healthy new superblock; repairs live, guest never restarts). +`Aborted` ⇒ **quiet no-op, and surface via `BoundUnderParent=false`** — a re-bind lands on the *same* dead +superblock and this runs every 20 s, so repairing would be an infinite silent retry that also masks the +state. **No operator decision taken quietly:** it does change customer-visible behaviour (apps stop and an +alarm fires where today nothing happens), but it routes an already-broken state into the **existing** gate, +event types and Hungarian copy — no new concept reaches the customer — and the alternative is applications +writing documents into a filesystem that rejects every write. -**What (d)'s dedup removes in the absent state**, enumerated: the registry row's `uuid:` `durable_id`, its -hardcoded and **false** `state: "attached"`, its `role: user-data`, and its `total_bytes`/`used_bytes` -copied from the **root** filesystem. No consumer found — wipe/decommission require the drive present; -`firstOfferableDrive` (`:138`) already requires `MountPath != ""` **and** `Role == "user-data"`; `diskKey` -(`disk_health.go:130`) changes key transiently but the health check excludes UNKNOWN verdicts, so no false -degradation. **R-118's symptom disappears incidentally — R-118 is NOT fixed and stays open.** +**No new recovery path.** `AttachDrive`'s normalize leg already did the repair and three call sites already +invoked it (20 s ticker, agent startup, and the controller's `Return` branch **before** `restartStacks`). +All three died on `if n == 1 && GuestSeesMount(...)` logging *"fully live, no-op"* about a dead namespace. -**Deliberately unchanged:** `Role` (the absent target still reads `system`), the `BoundUnderParent` -conjunction, and every wire field. +**Ordering trap, caught by a test:** reading the abort flag before comparing devices classifies the real +return state as aborted (its stale bind carries `shutdown` too) and refuses the repair **while still +reporting correctly** — so the abort flag is read off the RAW mount in the stale case. -## Files +## Live validation, demo-hp -`internal/hub/report.go` (`ConfigPath`, **`json:"-"`** — that struct is a cross-repo contract pinned by -`testdata/host-report.golden.json` + `contract_test.go`'s key-set comparison, so a wire-visible field -would have needed a matching `felhom.eu/hub` change; nothing off-box needs the value) · -`internal/storage/observe.go` (set from `s.Path`) · `internal/localapi/disks.go` (ConfigPath fallback + -guest-path dedup) · `internal/localapi/disks_backup_target_row_test.go` (fixture corrected + 4 tests) · -`CHANGELOG.md`. +| State | raw | bind | verdict | usable | +|---|---|---|---|---| +| healthy | `8:16` | `8:16` | `live` | true | +| **R-117 return** | `8:32` `/dev/sdc` | `8:16 …,shutdown` | **`stale-device`** | **false** | +| **Q7 in-place abort** | `252:11 …,emergency_ro` | `252:11 …,emergency_ro` | **`filesystem-aborted`** | **false** | -`observe.go`'s change is **not** the fallthrough its `:176-183` comment forbids: that prohibition is about -resolving a *device or UUID* from the **containing** filesystem, which would hand back root's identity and -mis-target a DR re-attach. `s.Path` is the storage's own declaration, identifies nothing but itself, and is -used for no resolution. `MountPath` stays empty, which is the truth. +340–497 µs per call. **No block I/O proven by `strace`**: only `/proc/self/mountinfo`, **0** `statfs`. +**No regression through the real pipeline:** `GET /disks` with the controller's own credential shows the +live backup-target drive `bound_under_parent=True`, with 32 gate lines in 3 min as the positive observable +and zero spurious transitions. -## §2 reconfirmation +## Tests + red-proofs -| Cited | Line now | Holds | -|---|---|---| -| `mountPath` assigned only on exactMount success | `observe.go:184-190` | **HOLDS** verbatim | -| empty `backingDevice` ⇒ `RoleSystem` | `role.go:180-181` | **HOLDS** verbatim | -| guest-path guard skipped for non-user-data | `disks.go:215-217` | **HOLDS** | -| `Known()` reads unit FILES, never the mount table | `registry_known.go:40-75`, `MountPath: spec.Where` at `:70` | **HOLDS** | -| `driveTargetByPath` keys only from non-empty fields | `intermediary.go:602-618` (`:611`, `:614`) | **HOLDS** | -| the dedup keys on `d.MountPath` at `:263` | `:263` is `if t.BackingDevice != ""`. Actual: `seen` built `:290-295`, dedup `:298` | **HOLDS; the line number in the spec is wrong** | -| "`seen[""]` is set, so the union row is not skipped" | **DOES NOT HOLD as stated** — `seen[d.MountPath] = true` is guarded by `if d.MountPath != ""`, so `seen` never gains `""`. The union row survives because `/mnt/` is *absent* from `seen`, not because `""` is present. Same outcome, different reason | corrected | +14 new tests; the payload ones drive the real `/disks` handler and the `AttachDrive` ones record the actual +privileged calls. Fixtures are the spike's captured mountinfo, injected as **data** via +`procSelfMountinfo`/`procGuestMountinfo`, so the real parser, predicate, `AttachDrive` and handler all run. -## Tests — 845 → 849, suite green (`go build ./... && go vet ./... && go test ./...` rc=0) +**6 red-proofs, each verified to have landed.** RP1/RP2 (term 3 removed per site), RP3 (**the P1-only +fix**), RP4 (`emergency_ro` dropped), RP5 (unknown reported dead), RP6 (aborted arm falls through to a +re-bind). -All PASS: `TestAbsentBackupTargetIsResolvableByGuestPath` (rewritten — real shape **and** the union loop), -`TestAbsentTargetAppearsExactlyOnce` (new, the join), `TestAbsentTargetKeepsR114DegradedSignal` (new, the -regression guard), `TestPresentTargetPayloadUnchanged` (new, fleet-wide parity), -`TestAbsentNonTargetKeepsItsRegistryRowAndNoFlag` (new, negative + over-suppression), -`TestAbsentTargetRowDoesNotRegisterPresence`, `TestCaseBLocalTargetGetsNoGuestPath`, -`TestSystemBackedTargetUnderMntGetsNoGuestPath`, `TestNonTargetDriveNeverCarriesTheFlag`. +**A hollow test was caught, by RP1 failing to fail.** The aborted fixture first used a `/dev/mapper` +device, for which `RoleForStorage` derives `role="system"` — a system row never enters the block that +computes `BoundUnderParent`, so the field stayed false **by default**, the assertion passed, and no mutation +could ever fail it. Fixtures now assert the production row shape before asserting the field. -**Red-proofs — each mutation asserted to have landed before the run:** +## Not yet done / not covered -1. v0.115.0's MountPath-only fallback → `isTarget[…] = FALSE … rows=2`. **The empirical proof v0.115.0 - was inert**, rather than an argument for it. -2. Drop the guest-path dedup → `the absent drive is carried by 2 rows, want exactly 1`. -3. Give the absent row a `MountPath` (what option (a) produces) → the R-114 guard fires. -4. Over-broad dedup (`gp != ""` without `seenGuest`) → the non-target drive loses its own row, and two - pre-existing R-113 union tests fail as well. +- **The stale-bind repair on hardware.** `StablePathForRaw` hardcodes the production + `/mnt/felhom-drives`, so exercising `AttachDrive` for a scratch drive writes a submount into guest + 9201's namespace, which the task fences. It happened once inadvertently, was reverted within a minute + and 9201 verified unaffected, and was deliberately not repeated. Filed **R-117h**. +- **Sustained-load behaviour of the predicate** — flagged unmeasured by the spike and **still unmeasured**. +- **felhom-pve not upgraded** (0.115.0). The fleet is non-uniform. +- **R-117g** — an aborted filesystem is never cleared automatically, by design, so it alarms until a human + acts; there is no guided recovery. +- **R-121** — a box's installed agent can sit releases behind the vouched one and nothing notices; the + R-120 gate is golden-vs-fleet-controller and cannot see it. -**The corrected fixture is the headline test change.** `theAbsentTarget` supplied -`MountPath: "/mnt/mentes"`, which production never supplies, and `targetRowServer` left `DriveTargets` -**nil**, so the union loop never ran and the two-row shape was invisible to the entire suite. Both fixed — -that combination is why an inert fix shipped green. +## Observations (not acted on) -## Process note - -One red-proof was restored with `git checkout `, which discarded the whole implementation rather -than the one-line mutation. Re-applied and re-verified; later red-proofs restored from pristine copies -kept outside the repo. `git checkout` is not an undo for a mutation on a work-in-progress file. - -## LIVE VALIDATION — all claims PASS (`felhom.eu` `audits/R116-v0116-2026-07-30.md`) - -Fresh box, real day-0 from the v1.25.0 ISO on a nested PVE on **demo-hp** (per `runbooks/target-selection.md`), -agent **installed unaided from the vouched Day-0 manifest**, both drives enrolled through the real endpoints, -device loss a real hot-detach. - -``` -07:20:04 backup_target_absent (error) Cel meghajto ← TARGET, specific (was generic through v0.115.0) -07:22:34 backup_target_restored (info) Cel meghajto ← its matching pair -07:24:04 storage_disconnected (error) Adat meghajto ← NON-target, generic, same box 4 min later -07:25:34 storage_reconnected (info) Adat meghajto -``` - -All four at the hub; gate fired in 3 s. **Discrimination proven non-trivially for the first time.** -Over-correction passes on a positive observable (0 ABSENT lines / 0 drive events over 2m14s with both drives -present, target `degraded:false`, while two `RETURNED` lines prove the gate ticked). Publish observables: -registry GET of the published bytes matched `b47c5c4d…`, the manifest was read back showing 0.116.0, and the -box reported it running. - -**One caveat, not a regression:** the drill's controller was **0.185.1** from the golden, which predates -R-114, so its absent-state banner showed the old false "the backup is on the system disk" copy. The R-114 -guard is pinned by unit test and by the payload (no row combines the flag with a mount path) but could not be -confirmed on that box. Filed as **R-120** — the golden is a release behind, which is R-115's class one layer up. - -## Teardown — all three §13 layers - -VM 9401 destroyed `--purge`; `r116-images` removed and the space measured back (`felhom-backup` 928787076 KiB -available vs 928787080 before); hub records **gate-blocked** — the cascade was attempted and correctly refused -`HTTP 409 host … is ONLINE`, with the command recorded for once it ages to DOWN. Secrets shredded. +Guest 9201 has 2 stacked entries at `/mnt/felhom-drives/nvme-1tb` while the host has 1 (pre-existing; +`AttachDrive` normalizes the host count only). 14 stale `felhom-agent.bak-*` binaries on demo-hp (~200 MB), +nothing prunes them. 11 files in this repo fail `gofmt -l`, all pre-existing and none in `internal/localapi/`.