docs(R-117): CLOSED — proven live on demo-hp; R-121 filed for agent-on-box drift
R-117 row → SHIPPED + PROVEN-LIVE (agent v0.117.0), with the full validation in audits/R117-v0117-2026-07-30.md. Both dead states detected on real hardware through the shipped predicate: RETURN raw 8:32 /dev/sdc | bind 8:16 shutdown → stale-device, usable false IN-PLACE both 252:11 emergency_ro, raw unit active → filesystem-aborted, usable false healthy → live 340-497us per call. No block I/O proven by strace (only /proc/self/mountinfo, 0 statfs) — the Part 1 CLAUDE.md fence applied to its own first consumer. No regression through the real pipeline: the live backup-target drive reads bound_under_parent=True via GET /disks with the controller's own credential, with 32 gate lines in 3 min as the positive observable and zero spurious transitions. The ruling asked for in §2.2 is recorded in full and flagged for overrule: Aborted must NOT self-heal. A re-bind lands on the same dead superblock and the call site runs every 20s, so repairing would be an infinite silent retry that masks the state. It surfaces instead. No operator decision was taken quietly — the reasoning is that it routes an already-broken state into the existing gate, event types and Hungarian copy, so no new concept reaches the customer. R-121 filed: a box's installed agent can sit releases behind the vouched one and nothing notices. demo-hp ran 0.113.0 against a vouched 0.116.0 through the whole R-116/R-117 arc. Confirmed at source that R-120's gate cannot catch it — it compares goldenVer against NewestReportedControllerVersion(), i.e. golden-artifact vs fleet-CONTROLLER. MinAgent is protective, not an alarm, and 0.113.0 equalled the floor. Fourth instance of the drift family. Also filed: R-117g (an aborted filesystem is never cleared automatically by design, so it alarms until a human acts, with no guided recovery) and R-117h (StablePathForRaw hardcodes the parent, so the repair path cannot be exercised on hardware without writing into a live customer guest's namespace).
This commit is contained in:
@@ -0,0 +1,437 @@
|
||||
# R-117 v0.117.0 — the liveness signal now tests liveness, proven on hardware
|
||||
|
||||
Implementation + live validation of `audits/SPIKE-r117-bind-liveness-2026-07-30.md`. Agent **v0.117.0**.
|
||||
`felhom-controller` untouched: the spike proved the repair is already wired from the controller's
|
||||
`Return` branch and dies inside the agent.
|
||||
|
||||
---
|
||||
|
||||
## 1. Baselines — each established separately
|
||||
|
||||
| Thing | Value | How |
|
||||
|---|---|---|
|
||||
| agent `main` before | **v0.116.0** @ `d4eb259` | `git log`, `CHANGELOG.md:1`; tree clean, `HEAD == origin/main` |
|
||||
| agent shipped this run | **v0.117.0** @ `966d8f4` | commit below |
|
||||
| **agent on demo-hp BEFORE** | **0.113.0** | `felhom-agent --version` on the box — **predates R-113 (0.114.0) entirely** |
|
||||
| **agent on demo-hp AFTER** | **0.117.0** | same command, post-deploy |
|
||||
| agent published + vouched before | published 0.113–0.116; **vouched 0.116.0** (`b47c5c4dab641ee5…`) | hub `/configuration`, read not assumed |
|
||||
| controller `main` | **v0.186.0** @ `b331f18` | `git log` |
|
||||
| golden bakes controller | **0.186.0** | hub `/configuration` — **matches `main`, so R-120's gate is holding them together** ✅ |
|
||||
| `min_agent` / `min_controller_version` | 0.113.0 / 0.156.0 | hub `/configuration` |
|
||||
| hub live | **0.82.0** | `kubectl -n felhom-system get deploy` |
|
||||
| `felhom.eu` HEAD before | `e70b5fe` | as the task expected |
|
||||
| demo-hp | PVE 9.2.2, kernel 7.0.2-6-pve | `pveversion` |
|
||||
|
||||
---
|
||||
|
||||
## 2. Part 1 — the two record items, committed before any Go file was opened
|
||||
|
||||
**`felhom.eu` `37515cd`** — `docs(R-117 Part 1)`. **`felhom-agent` `6be168d`** — the same rule repeated
|
||||
where it binds.
|
||||
|
||||
### 2.1 The `CLAUDE.md` rule, as written
|
||||
|
||||
Placed beside the seam-wiring rule in `felhom.eu/CLAUDE.md` "Code quality rules":
|
||||
|
||||
> **A health check issues no block I/O.** A probe that touches a wedged device enters uninterruptible
|
||||
> sleep, survives `SIGKILL`, and cannot be recovered until the device returns or the host reboots — so
|
||||
> `systemctl restart` hangs too. A timeout protects the caller's control flow and nothing else: the
|
||||
> blocked thread remains. Liveness is decided from `/proc` and the kernel's own state, never by reading
|
||||
> or writing the filesystem. Measured, R-117 spike §6.3 (…): a probe stayed in `D` state 3m50s after
|
||||
> `kill -9`; a buffered write with no `fsync` blocked too (`O_CREAT` needs journal access); and
|
||||
> `statfs`/`getdents` returned **healthy** on a namespace that `EIO`s every byte — fast, and wrong.
|
||||
|
||||
**Repeated in `felhom-agent/CLAUDE.md`**, as a one-line pointer, and the reason is recorded there: health
|
||||
checks are written in the agent repo, and `felhom.eu/CLAUDE.md` **does not load in an agent-only session**.
|
||||
A standing rule that does not load where it binds is the inert-seam shape applied to a rule. The task named
|
||||
one location; this is the one deviation, stated rather than absorbed (standing rule 4).
|
||||
|
||||
### 2.2 The R-116 narrowing, applied
|
||||
|
||||
The R-116 row's over-correction clause now carries: the 2 `RETURNED` lines are a genuine positive
|
||||
observable so rule 3 is satisfied, but `degraded:false` was read off a drive whose bind was dead, so the
|
||||
window evidences **"the gate did not over-fire"** and **not** **"the drive was healthy."** Nothing else
|
||||
about the row changed.
|
||||
|
||||
---
|
||||
|
||||
## 3. THE RULING (task §2.2) — what happens when P2 trips
|
||||
|
||||
**Ruling: P2 must NOT self-heal. It surfaces. Operator decision NOT required — and the reasoning for that
|
||||
is the substantive part, so it is stated in full for overruling.**
|
||||
|
||||
The two states have different repairs, and *"would a re-bind help?"* — not *"is something aborted?"* — is
|
||||
the question a verdict must answer:
|
||||
|
||||
| Verdict | What a re-bind would do | Ruling |
|
||||
|---|---|---|
|
||||
| **`BindStaleDevice`** | the raw mount is a **different, healthy** superblock (the drive returned and the fs-UUID-keyed unit healed onto it), so umount + re-bind lands the namespace on working storage | **REPAIR.** Falls through to the existing normalize leg |
|
||||
| **`BindAborted`** | the raw mount is the **same aborted** superblock — a fresh bind to a still-dead filesystem | **QUIET NO-OP, and surface via `BoundUnderParent=false`** |
|
||||
|
||||
**Why `BindAborted` must not repair,** three reasons and the third is the decisive one:
|
||||
|
||||
1. It cannot work. Re-binding does not clear an ext4 abort; that needs a remount or a fsck.
|
||||
2. `AttachDrive` runs **every 20 s** (`cmd/felhom-agent/main.go` ticker), so re-binding would be an
|
||||
**infinite silent retry** — R-117a's silence with more CPU.
|
||||
3. It would **mask** the state. The whole point of the fix is that this state becomes visible.
|
||||
|
||||
**Why no operator decision was needed.** The ruling changes customer-visible behaviour: apps that today
|
||||
keep running against a read-only/`EIO` namespace will be **stopped**, and the customer gets an alarm. That
|
||||
is a real change — but it is the **identical treatment an unplugged drive already receives**, through the
|
||||
existing gate, the existing event types and the existing Hungarian copy. No new message is invented, no
|
||||
new concept reaches the customer. The alternative is letting applications keep writing a customer's
|
||||
documents into a filesystem that rejects every write. **Fail-closed is also the project's standing
|
||||
direction for drive state.** So this is routing an already-broken state into an already-tested alarm, not a
|
||||
new customer-facing decision. **Flagged prominently rather than decided quietly; overrule freely.**
|
||||
|
||||
**Deliberately NOT done: automatic remount of an aborted filesystem.** That is a repair of the customer's
|
||||
data-bearing filesystem, risks loss on a dirty abort, and is an operator call. Recorded as R-117g.
|
||||
|
||||
---
|
||||
|
||||
## 4. The predicate as implemented
|
||||
|
||||
`felhom-agent/internal/localapi/intermediary.go`:
|
||||
|
||||
```
|
||||
bindLiveness(stable, raw) →
|
||||
stable=="" or raw=="" → BindUnknown
|
||||
no bind entry at `stable` → BindUnknown (isHostMountpoint's question, not this one)
|
||||
no raw entry at `raw` → BindUnknown (devicePresent already reports device absence)
|
||||
bind.devno != raw.devno → raw aborted ? BindAborted : BindStaleDevice ← P1
|
||||
(devnos agree) bind aborted → BindAborted ← P2
|
||||
(devnos agree) fstype unknown → BindUnknown
|
||||
otherwise → BindLive
|
||||
```
|
||||
|
||||
Wired as the **third term** of the conjunction at **both** `/disks` construction sites (`disks.go`):
|
||||
`boundUnderParent(...) && devicePresent(...) && bindUsable(gp, <raw>)`.
|
||||
|
||||
**Order is load-bearing, and a test caught it.** The first draft read the abort flag before comparing
|
||||
devices. In the real return state the stale bind carries `shutdown` **as well as** a different device, so
|
||||
abort-first classifies it `BindAborted` — **reporting stays correct, every payload test still passes, and
|
||||
the repair silently never runs.** The abort flag is therefore read off the **raw** mount in the stale case
|
||||
(the re-bind's target) and off the bind only once the devices already agree.
|
||||
|
||||
### 4.1 How *unknown* is expressed
|
||||
|
||||
A **three-state type**, never a bool: `BindLiveness ∈ {BindUnknown, BindLive, BindStaleDevice,
|
||||
BindAborted}`, with `BindUnknown` as the **zero value**. Every caller reads it through **one** method:
|
||||
|
||||
```go
|
||||
func (l BindLiveness) Usable() bool { return l == BindLive || l == BindUnknown }
|
||||
```
|
||||
|
||||
So the "cannot tell → never absent" rule lives in exactly one place and no caller can restate it wrongly.
|
||||
The task named the `newestArchiveOn` trap — a `(value, bool)` shape whose comment promised a degradation
|
||||
the signature could not express. Four routes to unknown are pinned by test: unreadable `/proc`, no raw
|
||||
entry, empty paths, and a filesystem whose abort vocabulary is unmeasured.
|
||||
|
||||
**P2's tokens: `shutdown` AND `emergency_ro`**, both measured. `ext4`/`ext3`/`ext2` (one driver); anything
|
||||
else → **unknown, never live**.
|
||||
|
||||
### 4.2 Not a new recovery path
|
||||
|
||||
`AttachDrive`'s early return now switches on the verdict instead of declaring a dead namespace
|
||||
*"fully live, no-op"*. The three call sites that already invoked the repair — the 20 s ticker, agent
|
||||
startup, and the controller's `Return` branch **before** `restartStacks` — are unchanged. Also:
|
||||
`isHostMountpoint` and `countHostMounts` are now one-liners over a single new parser `hostMountEntries`,
|
||||
which yields devno/root/fstype/super-options (`REUSE.md`'s three-independent-readers note narrowed).
|
||||
|
||||
---
|
||||
|
||||
## 5. Tests
|
||||
|
||||
**849 → 863** (+14 top-level), **29/29 packages green**, `go build` / `go vet` / `go test` each with
|
||||
`rc=0` read separately (never combined with a commit). The 849 baseline was **verified against a temporary
|
||||
worktree at HEAD**, not inferred — the first draft of the CHANGELOG line said 858 → 873 from a count taken
|
||||
mid-edit, and that was corrected before commit.
|
||||
|
||||
| Test | Asserts (the consequence) | Result |
|
||||
|---|---|---|
|
||||
| `TestDisks_BindLiveness_StaleBindReadsAbsent` | R-117 case (a) → `bound_under_parent` false **through the real `/disks` handler** | PASS |
|
||||
| `TestDisks_BindLiveness_AbortedFilesystemReadsAbsent` (×2 tokens) | R-117a case (b) → false, with a **row-shape guard** | PASS |
|
||||
| `TestDisks_BindLiveness_UnionPath_StaleBindReadsAbsent` | union path, case (a) | PASS |
|
||||
| `TestDisks_BindLiveness_UnionPath_AbortedReadsAbsent` | union path, case (b) — Role+State hardcoded there | PASS |
|
||||
| `TestDisks_BindLiveness_HealthyReadsPresent` (observe+union) | no false negative | PASS |
|
||||
| `TestDisks_BindLiveness_UnknownIsTreatedAsPresent` (5 subtests) | all four unknown routes → **present**; `Usable()` truth table | PASS |
|
||||
| `TestBindLiveness_Verdicts` (4 states) | each verdict | PASS |
|
||||
| `TestBindLiveness_AbortedWins_WhenDevnosAgree` | the P1-only fix cannot pass | PASS |
|
||||
| `TestHostMountEntries_ParsesDevnoAndSuperOpts` | `" - "`-separator field extraction | PASS |
|
||||
| `TestHostMountEntries_CountsStackedBinds` | double-bind convergence survives the refactor | PASS |
|
||||
| `TestAttachDrive_StaleBind_Rebinds` | **the repair RUNS** (umount+rebind recorded) | PASS |
|
||||
| `TestAttachDrive_AbortedFilesystem_DoesNotRebind` | **§3's ruling**: zero mount ops, no error | PASS |
|
||||
| `TestAttachDrive_Healthy_IsStillANoOp` | 20 s idempotency | PASS |
|
||||
| `TestAttachDrive_UnknownLiveness_IsANoOp` | cannot-tell must not churn | PASS |
|
||||
|
||||
**Fixtures are the spike's captured mountinfo**, with two substitutions both recorded in the file: the
|
||||
scratch shared parent mapped to the production `StableParentDir` (the code derives it), and the aborted
|
||||
state transposed from dm onto the USB shape (§5.2 below). Tests redirect `procSelfMountinfo` /
|
||||
`procGuestMountinfo` at fixture files, so the real parser, the real predicate, the real `AttachDrive` and
|
||||
the real `/disks` handler all run — **the data is injected, the logic is not.**
|
||||
|
||||
### 5.1 Red-proofs — 6, each verified to have landed
|
||||
|
||||
| # | Mutation | Landing verified by | Fails |
|
||||
|---|---|---|---|
|
||||
| RP1 | term 3 removed at the **Observe** site | `bindUsable(gp, t.MountPath)` occurrences → **0** | StaleBind + Aborted(×2) |
|
||||
| RP2 | term 3 removed at the **union** site | `bindUsable(gp, d.MountPath)` → **0** | both UnionPath tests |
|
||||
| RP3 | **the P1-only fix** — abort check neutered | `RED-PROOF RP3` marker present | Aborted(×2), UnionPath_Aborted, Verdicts(×2), AbortedWins |
|
||||
| RP4 | `emergency_ro` dropped from the token list | `RED-PROOF RP4` present | only the `emergency_ro` subtests — the "only `shutdown`" fix |
|
||||
| RP5 | unknown reported as not-live | `RED-PROOF RP5` present | UnknownIsTreatedAsPresent/unreadable |
|
||||
| RP6 | aborted arm falls through to the re-bind | `RED-PROOF RP6` present | AttachDrive_AbortedFilesystem_DoesNotRebind (`go test` **rc=1**, read from `go test` itself) |
|
||||
|
||||
### 5.2 A HOLLOW TEST WAS CAUGHT — by RP1 failing to fail
|
||||
|
||||
`TestDisks_BindLiveness_AbortedFilesystemReadsAbsent` first used the spike's dm capture verbatim
|
||||
(`/dev/mapper/r117cel`). **`RoleForStorage` derives `role="system"` for that backing device** — a
|
||||
system-role row never enters the block that computes `BoundUnderParent`, so the field stayed `false` **by
|
||||
default**, the assertion passed, and **no mutation could ever fail it.** Pure decoration, and it is exactly
|
||||
R-116's failure mode (a fixture supplying a shape production never emits).
|
||||
|
||||
Found because **RP1 removed term 3 and the test still passed.** Fixed by transposing the aborted fixture
|
||||
onto the USB drive shape (an in-place abort on a USB drive is the realistic case anyway; only the super
|
||||
options and the matching devnos carry the claim) **and** by adding a guard that fails loudly if the row is
|
||||
not the production shape:
|
||||
|
||||
```go
|
||||
if di.Role != "user-data" || di.GuestPath == "" {
|
||||
t.Fatalf("fixture does not reproduce the production row shape: role=%q guest_path=%q — "+
|
||||
"the conjunction never runs on such a row, so any assertion below is vacuous", di.Role, di.GuestPath)
|
||||
}
|
||||
```
|
||||
|
||||
**The red-proof did not merely confirm the test; it is the only thing that found the test was empty.**
|
||||
|
||||
---
|
||||
|
||||
## 6. Build, publish, vouch — three observables, quoted
|
||||
|
||||
**1. Published** — the **anonymous** GET the installer performs, no credentials:
|
||||
|
||||
```
|
||||
GET .../api/packages/admin/generic/felhom-agent/0.117.0/felhom-agent
|
||||
http=200 bytes=14032908
|
||||
downloaded sha256: e6768a12eba1c9e0743367f04589845330151ac63d10caad794e8a4c6f36f735
|
||||
built sha256: e6768a12eba1c9e0743367f04589845330151ac63d10caad794e8a4c6f36f735
|
||||
MATCH — published bytes are the built bytes
|
||||
$ /tmp/agent-dl-0.117.0 --version → felhom-agent 0.117.0
|
||||
```
|
||||
|
||||
**2. Vouched** — POST through the real operator endpoint, then the manifest **read back** (not the flash):
|
||||
|
||||
```
|
||||
POST /configuration/artifacts → HTTP/1.1 303, Location: /configuration?flash=artifacts_set
|
||||
hub log: [INFO] Artifact manifest set: agent=0.117.0 golden=0.186.0 min_agent="0.113.0" wrapper_sha=true
|
||||
|
||||
READ BACK: agent_version=0.117.0 agent_sha256=e6768a12eba1c9e0…
|
||||
golden_version=0.186.0 golden_sha256=b760ac6a33e70700… (preserved)
|
||||
min_agent=0.113.0 wrapper_sha256=104db0a4401f65bb… (preserved)
|
||||
```
|
||||
|
||||
R-120's gate did **not** refuse: golden 0.186.0 is not behind the fleet's newest controller (0.186.0).
|
||||
|
||||
**3. Running on the box under test:**
|
||||
|
||||
```
|
||||
BEFORE: felhom-agent 0.113.0 active
|
||||
AFTER: felhom-agent 0.117.0 active
|
||||
capabilities self-check ok=68 total=68 degraded=0 inactive=0
|
||||
reconcile: enrolled drive bound under shared parent (live, no reboot) vmid=9201 … ← loop ticking, new pid
|
||||
```
|
||||
|
||||
Zero `ERROR` lines. `configs/` was untouched by this commit, so a binary-only deploy is correct (verified
|
||||
with `git show --name-only`).
|
||||
|
||||
---
|
||||
|
||||
## 7. §5.1 — demo-hp brought current, and the drift the R-120 gate does not cover
|
||||
|
||||
demo-hp: **0.113.0 → 0.117.0**, prior binary backed up to `felhom-agent.bak-0.113.0`.
|
||||
|
||||
**Does the R-120 vouch gate cover agent-on-a-box drift? NO — confirmed at source.**
|
||||
`hub/internal/web/configs.go:1165-1169` compares `goldenVer` against
|
||||
`store.NewestReportedControllerVersion()`: a **golden-artifact vs fleet-CONTROLLER** check. It says nothing
|
||||
about the agent installed on a box. **`MinAgent` does not cover it either** — it *holds* the controller
|
||||
floor for a box whose agent is too old (`hub/internal/api/handler.go:530-538`, `store.go:1857`), which is
|
||||
protective, not an alarm — and demo-hp's 0.113.0 **equalled** `min_agent` 0.113.0, so even a floor
|
||||
comparison was satisfied.
|
||||
|
||||
**Filed as R-121 — the fourth instance of the drift family** (R-111, R-115, R-120, and now
|
||||
installed-vs-vouched). Its measured cost: R-117's whole subject is the R-113 conjunction, which landed in
|
||||
0.114.0, so **the designated drill host could not exercise the code under investigation at all**, and the
|
||||
spike had to route every predicate result through an out-of-repo probe built from `main`.
|
||||
|
||||
---
|
||||
|
||||
## 8. §5.2 — the live run
|
||||
|
||||
**Venue:** demo-hp, the spike's §10 recipe. Scratch LXC **9301** (no NIC) on a scratch dir storage
|
||||
`r117scratch` — never `local-lvm`, never `felhom-backup`. **Scratch** shared parent `/mnt/r117-drives`, so
|
||||
the live `/mnt/felhom-drives` peer group (guest 9201's) was not used. `scsi_debug` for the drive, because
|
||||
it is the only mechanism that removes a gendisk **while mounted**. Guest parent bind verified
|
||||
`master:209` — a real slave of the host peer group.
|
||||
|
||||
**Method:** a throwaway probe built from the **shipped tree at `966d8f4`** (exported in a copy outside both
|
||||
repos), so `bindLiveness` and `AttachDrive` are the real functions, judging real kernel state.
|
||||
|
||||
### 8.1 The predicate, every state, on hardware
|
||||
|
||||
| State | host raw | host bind | verdict | `Usable()` | term 3 | cost |
|
||||
|---|---|---|---|---|---|---|
|
||||
| HEALTHY (scsi) | `8:16 rw,stripe=512` | `8:16 rw,stripe=512` | `live` | true | true | 442 µs |
|
||||
| after real gendisk removal | **(NOT MOUNTED)** | `8:16 …,shutdown` | `unknown` | true | true | 340 µs |
|
||||
| **R-117 RETURN** | `8:32` (`/dev/sdc`) | `8:16 …,shutdown` | **`stale-device`** | **false** | **false** | 365 µs |
|
||||
| HEALTHY (dm) | `252:11 rw` | `252:11 rw` | `live` | true | true | 359 µs |
|
||||
| **Q7 IN-PLACE ABORT** | `252:11 …,emergency_ro` | `252:11 …,emergency_ro` | **`filesystem-aborted`** | **false** | **false** | 497 µs |
|
||||
|
||||
**Both dead states are detected on real hardware.** Note the two rows that matter most:
|
||||
|
||||
- the **RETURN** row is the exact defect — `sdb`→`sdc`, the bind left on the dead superblock;
|
||||
- the **IN-PLACE** row has **identical devnos on both sides** (the device never left) and the raw mount
|
||||
still `active` — so the device comparison alone reads healthy there, and only the abort token catches
|
||||
it. This is R-117a, the half that emits nothing today.
|
||||
|
||||
The **absent** row is correct by design and worth stating: term 3 answers `unknown` because `devicePresent`
|
||||
is the term that reports device absence, so term 3 abstains rather than double-counting. The overall
|
||||
conjunction is false via term 2. It also demonstrates the cannot-tell rule working in a real state.
|
||||
|
||||
**Cost: 340–497 µs**, against 3.85–5.03 ms that `GuestSeesMount` already spends forking `lxc-info`.
|
||||
|
||||
### 8.2 No block I/O — the Part 1 fence applied to its own first consumer
|
||||
|
||||
`strace -f -e trace=openat,statfs,fstatfs,read,pread64` while the predicate judged the **aborted** drive:
|
||||
|
||||
```
|
||||
distinct openat targets: "/etc/localtime" "/proc/self/cgroup" "/proc/self/mountinfo"
|
||||
"/sys/.../cpu.max" "/sys/kernel/mm/transparent_hugepage/hpage_pmd_size"
|
||||
openat/statfs against /mnt/r117*: NONE — the predicate never touched the mount
|
||||
statfs syscalls: 0
|
||||
```
|
||||
|
||||
### 8.3 No regression on the live customer drive — through the REAL server pipeline
|
||||
|
||||
`GET /disks` over the island bridge with the **controller's own credential** (from the guest's
|
||||
`/etc/felhom-bootstrap/bootstrap.json`), i.e. the exact call the controller makes:
|
||||
|
||||
```
|
||||
GET /disks http=200 bytes=2545 vmid=9201
|
||||
name=felhom-backup state=attached role=user-data
|
||||
guest_path=/mnt/felhom-drives/nvme-1tb bound_under_parent=True backup_target=True
|
||||
```
|
||||
|
||||
The live enrolled drive — which is also the whole-guest backup target — **reads present under 0.117.0.**
|
||||
Host-side corroboration: exactly **1** bind entry at the stable path, `259:0` on both raw and bind, no
|
||||
abort token ⇒ `BindLive`. And a positive observable that the consumer was actually evaluating (rule 3):
|
||||
**32 gate-related controller log lines in 3 minutes**, with **zero** `ABSENT`/`RETURNED`/`backup_target`
|
||||
transitions, and zero `bind is STALE` / `normalized to one bind` lines from the agent — so the new
|
||||
predicate disturbed nothing.
|
||||
|
||||
### 8.4 `AttachDrive` on hardware — and the one thing that was NOT repeated
|
||||
|
||||
The healthy case ran through the real `AttachDrive` on the box: it returned the stable path, and the
|
||||
**guest init PID was identical before and after (2555744)** — the guest was never restarted.
|
||||
|
||||
**`AttachDrive` derives its target from `StablePathForRaw`, which hardcodes the production
|
||||
`/mnt/felhom-drives`.** So invoking it for a scratch drive **necessarily creates a submount under the live
|
||||
customer parent, which propagates into guest 9201** — which this task fences. That happened **once,
|
||||
inadvertently, during the healthy case**: a bind appeared at `/mnt/felhom-drives/r117sd`. It was removed
|
||||
within a minute and the revert was verified, not assumed:
|
||||
|
||||
```
|
||||
live parent restored to exactly its two real submounts (nvme-1tb, Felhom-Share)
|
||||
9201: running, init pid unchanged (never restarted)
|
||||
stale scratch entries in 9201's namespace: 0
|
||||
9201 guest view: Felhom-Share nvme-1tb 9201 live drive read: OK
|
||||
```
|
||||
|
||||
**It was deliberately NOT repeated for the stale-bind case**, so the *repair* leg is proven by
|
||||
`TestAttachDrive_StaleBind_Rebinds` + RP6 through the real code, and on hardware only for the healthy
|
||||
no-op. Doing it live would need ~20 s of a scratch submount inside guest 9201's namespace — harmless on
|
||||
the evidence above, but the fence names 9201, so it is the operator's call, not one to take quietly.
|
||||
**Named as the one gap in this run's hardware coverage.**
|
||||
|
||||
---
|
||||
|
||||
## 9. Teardown — all three layers
|
||||
|
||||
**Layer 1 — the machine.** `pct stop 9301` + `pct destroy 9301 --purge`; `pct list` back to `9201` alone.
|
||||
|
||||
**Layer 2 — the host.**
|
||||
|
||||
| | Before | After |
|
||||
|---|---|---|
|
||||
| `pvesm status` | `r117scratch` active, `local` **39.03 %** | `r117scratch` **removed**, `local` **37.07 %** (session start 37.02 %) |
|
||||
| `df -h /var/lib/vz` | 16G, **42 %** | 15G, **40 %** (session start 40 %) |
|
||||
| `dmsetup ls` | `r117cel (252:11)` + 11 pre-existing | **gone**; the 11 `pve-*` unchanged |
|
||||
| `losetup -a` | 2 loops | **none** |
|
||||
| `scsi_debug` | loaded | **unloaded**; `/dev/sd*` back to `sda1..3` |
|
||||
| `/mnt` | `+ r117-drives, r117cel, r117sd` | `felhom-drives`, `nvme-1tb`, `vzsnap0` — as found |
|
||||
| `/root`, `/var/lib/vz/r117*` | 5 scripts + `r117/` + probe | `ls -d /mnt/r117* /root/r117* /var/lib/vz/r117*` → **NONE** |
|
||||
|
||||
The 124 MB template was removed. **One teardown defect, recorded because it is a reusable trap:** the
|
||||
script hardcoded `losetup -d /dev/loop0`, but the loop numbers **swapped** this run (the guest rootfs took
|
||||
loop0), so it detached the wrong device and left `/dev/loop1` attached to a deleted file. Fixed by
|
||||
selecting loops by backing file (`losetup -a | grep r117`), not by number. **`local` is 0.05 % above the
|
||||
session start** (~20 MB) — journal growth from the run, not a leftover artifact.
|
||||
|
||||
**Layer 3 — the hub: one record was written, deliberately, and it is retained.** No customer, appliance,
|
||||
pairing or claim was created (9301 had no NIC and ran no controller). The one hub-side change is the
|
||||
**artifact manifest, now vouching agent 0.117.0** — the intended product of §6, not scratch, so it is
|
||||
**retained** and is the fleet's vouched agent going forward.
|
||||
|
||||
**Fences — re-verified after teardown, not assumed:**
|
||||
|
||||
| Fence | State |
|
||||
|---|---|
|
||||
| guest **9201** | `running` throughout; guest view `Felhom-Share nvme-1tb`; live drive read **OK** |
|
||||
| **`drill-r50`** (VM 300) | `stopped` — never started |
|
||||
| **`local-lvm`** | **38.84 %** — byte-identical before, after, and to the session start |
|
||||
| **`felhom-backup`** | `active`, 0.46 %, `content backup` unchanged — not re-targeted, not written |
|
||||
| live `/mnt/felhom-drives` | **both** real submounts present; no `r117*` leftovers |
|
||||
| `felhom-agent` on demo-hp | `active`, **0.117.0** |
|
||||
| demo-felhom, Peti's cluster | never contacted |
|
||||
| `felhom-controller` | **untouched**, as scoped |
|
||||
|
||||
---
|
||||
|
||||
## 10. Findings — filed as `R-n`, none fixed
|
||||
|
||||
| # | Finding | Severity |
|
||||
|---|---|---|
|
||||
| **R-121** | **A box's installed agent can sit releases behind the vouched one and nothing notices.** demo-hp ran 0.113.0 against a vouched 0.116.0 through the whole R-116/R-117 arc. The R-120 gate is golden-vs-fleet-controller and cannot see it; `MinAgent` is protective, not an alarm, and 0.113.0 *equalled* the floor. **Fourth instance of the drift family** | **MEDIUM** — filed with a fix shape and the honest tension (a box legitimately lags between publish and deploy) |
|
||||
| **R-117g** | **An aborted filesystem is never cleared automatically, by design (§3), so a drive stuck in `emergency_ro` alarms forever until a human acts.** The alarm is correct and new; the absence of any guided recovery is the gap. A remount risks loss on a dirty abort, so it wants an operator-confirmed action, not a self-heal | **LOW-MEDIUM (follow-up to this fix)** |
|
||||
| **R-117h** | **`AttachDrive` cannot be exercised against a non-production shared parent**, because `StablePathForRaw` hardcodes `StableParentDir`. Any hardware test of the repair path writes a submount into the live customer guest's namespace (§8.4). A test seam for the parent dir would make the repair leg testable on hardware without touching a customer | **LOW (testability)** |
|
||||
|
||||
**Observations — noticed, documented, NOT acted on:**
|
||||
|
||||
- **Guest 9201 carries 2 stacked entries** at `/mnt/felhom-drives/nvme-1tb` while the host has exactly 1.
|
||||
Pre-existing, unrelated to this change, and harmless to the predicate (it reads the host table, and
|
||||
`bindLiveness` iterates all entries). Worth a look — `AttachDrive` normalizes the *host* count only.
|
||||
- **14 stale `felhom-agent.bak-*` binaries** on demo-hp (`0.92.1` … `0.113.0`), ~200 MB on `pve-root`.
|
||||
Nothing prunes them.
|
||||
- **11 files in the agent repo fail `gofmt -l`**, all pre-existing (`internal/capability/probe.go`,
|
||||
`internal/storage/intent.go`, …). None are in `internal/localapi/`; none were touched here.
|
||||
|
||||
---
|
||||
|
||||
## 11. What did not happen, and why
|
||||
|
||||
- **The stale-bind repair was not exercised on hardware** (§8.4) — it would put a scratch submount inside
|
||||
guest 9201's namespace, which this task fences. Covered by `TestAttachDrive_StaleBind_Rebinds` + RP6
|
||||
through the real code, and by the healthy-case `AttachDrive` on hardware (init PID identical).
|
||||
- **No load or duration measurement of the predicate under a sustained reconcile loop.** The spike flagged
|
||||
this as unmeasured and **it is still unmeasured** — said plainly rather than let to vanish. What exists:
|
||||
single-call costs of 340–497 µs on hardware, and ~15 minutes of the real 20 s ticker on demo-hp with no
|
||||
spurious re-bind and no CPU complaint. A many-drive box under sustained load was not tested.
|
||||
- **`felhom-controller` was not touched**, per scope — the repair is already wired from its `Return` branch.
|
||||
- **The R-116 row was not edited beyond §2.2's clause.**
|
||||
- **felhom-pve was not upgraded** — it runs 0.115.0 and the task scoped the venue to demo-hp. The fleet is
|
||||
now non-uniform: demo-hp 0.117.0, felhom-pve 0.115.0. Flagged, not acted on.
|
||||
- **R-118 was not addressed** (the absent drive's union row advertising the root filesystem's capacity) —
|
||||
a separate row, untouched by this fix.
|
||||
|
||||
---
|
||||
|
||||
Every claim above is quoted from pushed Gitea source at `file:line` (agent `966d8f4`, hub `0.82.0`) or is
|
||||
captured run output.
|
||||
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user