docs(R-116): CLOSED — proven live; capability row F to PROVEN-LIVE; R-120 filed
The events leg the previous commit reported as not-reached is now done. The operator relayed the claim code (the only route: bcrypt-hashed hub-side, emailed only), the two storage paths were registered through the real POST /api/storage/register, and the cycle ran on the fresh box: 07:20:04 backup_target_absent (error) Cel meghajto <- TARGET, specific 07:22:34 backup_target_restored (info) Cel meghajto <- its matching pair 07:24:04 storage_disconnected (error) Adat meghajto <- NON-target, generic 07:25:34 storage_reconnected (info) Adat meghajto All four at the hub; gate fired in 3 s. Two matched pairs, correctly discriminated -- and discrimination is proven NON-trivially for the first time, since both prior runs had the target itself emit the generic event. Over-correction passes on a positive observable, with two RETURNED lines proving the gate was ticking. 00-capability-map row F: PARTIAL -> PROVEN-LIVE with the evidence and the caveat. R-120 filed: the golden bakes controller 0.185.1, which PREDATES R-114 + R-112, so a freshly installed box shows the customer the WRONG absent-target message -- observed live on the drill box: the generic "the backup is on the same disk as the system" copy (false; the target is a drive that vanished) plus an offer of the other drive as the remedy. That is E2D 5.3's exact payload, still reachable on any new install. R-115's class one layer up -- R-111 closed by re-baking the golden, 0.186.0 then shipped, the golden did not move, and the gap reopened silently; this time the stale artifact carries a customer-facing falsehood in exactly the state R-116 now alarms about correctly. Teardown recorded for all three layers, hub layer gate-blocked with the command.
This commit is contained in:
@@ -121,7 +121,7 @@
|
||||
|
||||
| Scenario | Components | Status | Evidence | Gap / roadmap |
|
||||
|---|---|---|---|---|
|
||||
| An ABSENT backup-target drive raises its OWN alarm, paired with a matching recovery | agent v0.114.0, controller v0.184.1+, hub v0.81.0 | **PARTIAL** | `SESSION-C-2026-07-29` C5: the drive-absent gate fires (4 s) and an alarm reaches the hub — but it is the **generic** `storage_disconnected`, while the return fires the **specific** `backup_target_restored`, so the pair cannot be matched. `backup_target_absent` never fired (count 0) | RE-TESTED 2026-07-30 on the fully shipped stack (`R116-PART5-2026-07-30`) — **still generic**; the v0.115.0 fix was ineffective and the mechanism is not isolated. The specific alarm, its severity, Hungarian copy and hub routing remain unexercised end-to-end → **R-116** |
|
||||
| An ABSENT backup-target drive raises its OWN alarm, paired with a matching recovery | agent **v0.116.0**, controller v0.184.1+, hub v0.81.0 | **PROVEN-LIVE** (2026-07-30) | `audits/R116-v0116-2026-07-30.md`. On a fresh box built through the **real day-0** on demo-hp, running the agent it installed **unaided from the vouched Day-0 manifest** (0.116.0), both drives enrolled through the real endpoints and device loss a real hot-detach — **the full four-event sequence, two matched pairs, correctly discriminated:** `07:20:04 backup_target_absent (error)` / `07:22:34 backup_target_restored (info)` for the TARGET, and `07:24:04 storage_disconnected (error)` / `07:25:34 storage_reconnected (info)` for a NON-target drive on the same box minutes apart. Gate fired in **3 s**. All four reached the hub — specific alarm, severity, Hungarian copy and hub routing now exercised end-to-end. **Discrimination is proven NON-trivially for the first time**: both prior runs had the target itself emit the generic event, so the mirror proved nothing. **Over-correction guard PASSES** with a positive observable — 0 ABSENT lines and 0 drive events over a 2m14s window with both drives present, target `degraded:false`, while 2 `RETURNED` lines prove the gate was ticking. | **The mechanism was isolated from the captured payload first** (`DIAG-r116-disks-payload-2026-07-30.md`), after two fixes aimed at shapes that do not occur. v0.116.0 joins the two records of one drive on the only identity that survives the device — the CONFIGURED path — so one row carries both the flag and the guest path the gate keys on. **Both smaller-looking fixes were rejected because they regress R-114** (`backup_target_offer.go:79` reads flag+mount_path as *healthy*). **Caveat worth reading: the drill box ran controller 0.185.1 from the golden, which PREDATES R-114** — so its absent-state banner showed the old false "backup is on the system disk" copy. That is the golden being a release behind, not a regression → **R-120** |
|
||||
| Health-degradation email (edge-triggered, cooldowns, Hungarian) via hub → Resend | controller, hub | **IMPLEMENTED** | delivery pipeline live-proven for the **enlarge-block** trigger (`CAMPAIGN-6D` P3-DELIVERY, op+customer "Kedves Ügyfél!"); `NotifyHealthChange` ok→warn/fail edge-trigger implemented | The **health-degradation** trigger specifically has never fired an email live in any doc. Demoted (pipeline proven for a different event). Deliverability to HU freemail → R-4 |
|
||||
| Event catalog: app_start_failed, dead-app, offbox_enlarge_blocked, claim/reset codes, critical severity | controller, hub v0.31/48/50/55 | **PROVEN-LIVE** | live-delivered: `CAMPAIGN-6D` P3-DELIVERY (enlarge-block, op+customer); `DRILL-day0-vm` F-4 (claim code); `DRILL-day0-take2` F-15 (reset code) | `app_start_failed`/`dead-app` delivery is unit-only (6C inconclusive) — the pipeline + 3 event families are live, those two are not |
|
||||
| Prefs safety: empty-email wipe guard | controller v0.137 + hub v0.71.0 | **IMPLEMENTED** | controller leg red-proofed 07-15; hub-side no-clobber belt (`handleSavePreferences` preserves a stored non-empty address on an empty-email push) red-proofed 07-22 | Born from a live incident; controller 0.160.0 guards both its push legs, so the hub belt covers older/rogue boxes |
|
||||
|
||||
@@ -1,16 +1,15 @@
|
||||
# R116-v0116-2026-07-30 — the fix works at the payload layer, live; the events leg is blocked on an emailed code
|
||||
# R116-v0116-2026-07-30 — R-116 CLOSED: the specific alarm and its matching recovery, both on the wire
|
||||
|
||||
**Run:** R-116 join task, CC on DooPlex, 2026-07-30. Agent **v0.116.0** built, published, vouched, and
|
||||
**installed by a fresh box from the Day-0 manifest**.
|
||||
**installed by a fresh box from the Day-0 manifest**. **Result: ALL claims PASS.**
|
||||
|
||||
| Claim | Verdict |
|
||||
|---|---|
|
||||
| **The fix is effective live** — the absent target row carries the flag **and** the key the gate uses | ✅ **PASS**, on real hardware. `isTarget["/mnt/felhom-drives/cel"] = true` (was `false` through v0.115.0) |
|
||||
| **R-114 not regressed** — no absent-state row combines the flag with a mount path | ✅ **PASS** |
|
||||
| **No over-correction** — the absent row does not register presence | ✅ **PASS** (`bound_under_parent: false`) |
|
||||
| **Discrimination at the payload layer** — non-target gets the flag on no row | ✅ **PASS** |
|
||||
| **C5 — `backup_target_absent` / `_restored` events on the wire, paired** | ⛔ **NOT REACHED.** Blocked on the controller claim code (emailed only). See §6 |
|
||||
| **Discrimination as events** | ⛔ **NOT REACHED**, same blocker. **Not reported as a pass** |
|
||||
| **The fix is effective live** — the absent target row carries the flag **and** the key the gate uses | ✅ **PASS**. `isTarget["/mnt/felhom-drives/cel"] = true` (was `false` through v0.115.0) |
|
||||
| **C5 — `backup_target_absent` / `_restored` on the wire, paired** | ✅ **PASS.** `backup_target_absent` **(error)** on detach, `backup_target_restored` **(info)** on return, same drive, both at the hub. Gate fired in **3 s** |
|
||||
| **Discrimination — target ⇒ specific, non-target ⇒ generic** | ✅ **PASS, and NON-trivially for the first time.** Same box, minutes apart: target → `backup_target_absent`; non-target → `storage_disconnected` |
|
||||
| **No over-correction** | ✅ **PASS** with a positive observable — 0 ABSENT lines / 0 drive events over 2m14s with both drives present, while 2 `RETURNED` lines prove the gate ticked |
|
||||
| **R-114 not regressed by the fix** | ✅ **PASS at the payload layer** (no row combines the flag with a mount path) + unit-pinned. ⚠️ **not confirmable on this box** — it ran controller 0.185.1 from the golden, which predates R-114 → **R-120** |
|
||||
|
||||
## 1. Baselines as actually running
|
||||
|
||||
@@ -145,7 +144,50 @@ reattach capture — only "the flag and the key rejoined on one row", which is w
|
||||
events leg runs, the `Return` branch will fire against a dead bind, so a `backup_target_restored` there
|
||||
proves pairing, **not** recovery. Recorded, not fixed (R-117 has its own row).
|
||||
|
||||
## 6. Why C5 was not reached — the one genuine human step, again
|
||||
## 6a. C5 + discrimination — PASSED, the full four-event sequence
|
||||
|
||||
The claim gate (§6b) was cleared with an operator-relayed code, the two paths registered through the real
|
||||
`POST /api/storage/register`, and the cycle run. Controller log, verbatim, one continuous run:
|
||||
|
||||
```
|
||||
07:20:03 [WARN] [gate] drive ABSENT /mnt/felhom-drives/cel — stopped+blocked 0 app(s): []
|
||||
07:20:03 [ERROR] [gate] the ABSENT drive /mnt/felhom-drives/cel is the WHOLE-GUEST BACKUP TARGET
|
||||
— the system backup cannot run until it returns
|
||||
07:20:04 [INFO] Event pushed: backup_target_absent (error) — A rendszermentés meghajtója nem érhető el:
|
||||
Cel meghajto (/mnt/felhom-drives/cel)
|
||||
07:22:34 [INFO] [gate] drive RETURNED /mnt/felhom-drives/cel — re-attached + restarted gate-stopped apps
|
||||
07:22:34 [INFO] Event pushed: backup_target_restored (info) — A rendszermentés meghajtója újra elérhető:
|
||||
Cel meghajto (/mnt/felhom-drives/cel)
|
||||
07:24:03 [WARN] [gate] drive ABSENT /mnt/felhom-drives/adat — stopped+blocked 0 app(s): []
|
||||
07:24:04 [INFO] Event pushed: storage_disconnected (error) — Meghajtó váratlanul leválasztva: Adat meghajto
|
||||
07:25:34 [INFO] [gate] drive RETURNED /mnt/felhom-drives/adat — re-attached + restarted gate-stopped apps
|
||||
07:25:34 [INFO] Event pushed: storage_reconnected (info) — Meghajtó újra csatlakoztatva: Adat meghajto
|
||||
```
|
||||
|
||||
All four **reached the hub** (`Event from sess-e: …` at 09:20:03 / 09:22:34 / 09:24:04 / 09:25:34 CEST).
|
||||
|
||||
| # | drive | event | severity | pair |
|
||||
|---|---|---|---|---|
|
||||
| 1 | **target** `cel` | **`backup_target_absent`** | error | ↔ 2 |
|
||||
| 2 | **target** `cel` | **`backup_target_restored`** | info | ↔ 1 |
|
||||
| 3 | non-target `adat` | `storage_disconnected` | error | ↔ 4 |
|
||||
| 4 | non-target `adat` | `storage_reconnected` | info | ↔ 3 |
|
||||
|
||||
**Two matched pairs, correctly discriminated.** This is what R-116 existed to produce and what two prior
|
||||
runs could not: both of those had the *target* emit the generic event, so "non-target ⇒ generic" proved
|
||||
nothing about telling them apart. Here the two cases ran on **the same box, four minutes apart**, and
|
||||
diverged.
|
||||
|
||||
**Over-correction guard — positive observable, not an absent log line.** Window 07:26:51Z → 07:29:05Z with
|
||||
both drives present: **0** `drive ABSENT` lines, **0** drive events, and
|
||||
`{"degraded":false,"label":"Cel meghajto","target":"felhom-backup"}`. That the gate was *running* during
|
||||
the window is established independently by the two `[gate] drive RETURNED` lines earlier in the same
|
||||
container's log — so the silence is a decision, not a dead loop.
|
||||
|
||||
Also emitted: `health_critical (error)` at 07:21:32 while the target was away, and its recovery. Expected
|
||||
— the box's overall health reflects a missing backup target — recorded so the event count reconciles.
|
||||
|
||||
## 6b. The claim gate — the one genuine human step, again (→ R-119)
|
||||
|
||||
`planDriveGates` iterates `s.settings.GetStoragePaths()`. The drill controller has **none**:
|
||||
|
||||
@@ -166,24 +208,35 @@ self-bind link is likewise mint-and-email — `handleSelfBindLinkSend` (`selfbin
|
||||
ever renders a flash, never the token. **There is no operator-side route to either secret**, which is the
|
||||
same wall E2D hit and named "the one genuine human step".
|
||||
|
||||
A fresh claim code (**generation 2**) was emailed to the registered address at 09:07:20 by
|
||||
`POST /configs/sess-e/claim-resend`, so the code is waiting; the run stops here rather than reporting a
|
||||
third trivial pass.
|
||||
A fresh code (**generation 2**) was emailed by `POST /configs/sess-e/claim-resend` and **the operator
|
||||
relayed it**, which is the only route that exists. Claim submitted through the real `POST /claim` (its own
|
||||
pre-auth HMAC CSRF: GET the page, carry the token **and** its cookie), then login, then session-CSRF for
|
||||
the writes. Positive discriminator that the gate moved, as E2D recorded:
|
||||
|
||||
**Explicitly NOT done, and not claimed:** no `backup_target_absent` or `backup_target_restored` event was
|
||||
observed on the wire this run, and the non-target mirror was not exercised as events. The payload evidence
|
||||
in §4 determines what the gate *will* decide, but it is not the event on the wire, and after two runs that
|
||||
conflated the two, that distinction is kept.
|
||||
```
|
||||
before: {"ok":false,"error":"dashboard not yet claimed"}
|
||||
after: {"ok":false,"error":"authentication required"} (unauthenticated)
|
||||
authed: {"data":{"degraded":false,"known":true,"label":"/mnt/felhom-drives/cel","target":"felhom-backup"},"ok":true}
|
||||
```
|
||||
|
||||
**The cost is real and recurring: three sessions have now stopped at this wall.** → **R-119**.
|
||||
|
||||
## 7. Teardown — layers 1–3, per the §13 paragraph this task added
|
||||
|
||||
**DELIBERATELY RETAINED, pending the claim code**, so the events leg can finish without rebuilding:
|
||||
|
||||
| layer | item | disposition |
|
||||
|---|---|---|
|
||||
| 1 — machine | VM **9401** `r116-drill` on demo-hp (+ `unused0`) | **retained**, running |
|
||||
| 2 — host | `r116-images` dir storage at `/mnt/nvme-1tb` | **retained** |
|
||||
| 3 — hub | customer **`sess-e`**, host **`sess-e-5d4427`**, appliance **13** | **retained** |
|
||||
| 1 — machine | VM **9401** `r116-drill` + all four volumes | **DESTROYED** `qm destroy 9401 --purge` |
|
||||
| 2 — host | `r116-images` dir storage at `/mnt/nvme-1tb` | **REMOVED**; space returned (below) |
|
||||
| 3 — hub | customer **`sess-e`**, host **`sess-e-5d4427`**, appliance **13** | **GATE-BLOCKED — command recorded below.** The cascade was attempted and **correctly refused: HTTP 409 "Delete refused: host sess-e-5d4427 is ONLINE"**. Deletable once it ages ONLINE→DOWN (>1 h from its last report, `customer_delete.go:220-228`) |
|
||||
|
||||
**Layer 2, measured:** `felhom-backup` available **928787076 KiB after** vs **928787080 KiB before the
|
||||
run** (4 KiB = noise), used back from 17708084 → 4566012 KiB. `local-lvm` **38.84 %** vs 38.83 % — demo-hp's
|
||||
own guest, not this run. `r116-images` gone; `qm list` shows only `drill-r50`. **The space came back.**
|
||||
|
||||
**Secrets:** the break-glass credential and the hub DB copy it came from were `shred -u`'d; the claim code,
|
||||
the drill controller password and the session cookie were shredded in the guest and on the box before
|
||||
destruction, and the local copies on DooPlex are shredded. The in-guest `shred -u` left 3 files behind
|
||||
(reported honestly rather than claimed clean) — they died with the purged disk moments later.
|
||||
|
||||
`pvesm status` on demo-hp **before** the run, for the layer-2 comparison at teardown:
|
||||
|
||||
|
||||
@@ -16,7 +16,8 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
|
||||
| **R-110** | **`main` is the installer's publish channel — there is no staging.** `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a 30 s period and nginx serves that working tree directly (`location /scripts/`, `root …/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as **root on a virgin box**, the single most privileged thing Felhom ships | **WAITING-ON-OPERATOR (S)** | operator ruling | **Two consequences worth stating:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" **was never available to take**. **Open question for the operator, not a defect to fix blind:** whether `/scripts/` should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether `main`-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now. **SECOND INSTANCE, found 2026-07-29 by the E-2d run and filed here rather than as a new ID:** `felhom-host-install.sh` fetches **nine** files from `raw/branch/main` (`:2072`–`:2206`) and the hub manifest vouches a sha for exactly **one** (`wrapper_sha256` → `felhom-pbs-apply`; re-checked this run, no drift). E-2a's `felhom-backup-target-apply` (`:2116`) is installed **0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n` — a root-executed artifact taken from `main` with no pinned integrity, which is this row's class exactly | CC |
|
||||
| **R-111** | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`.**~~ `felhom-host-install.sh` does not use `main`: it reads the hub-vouched manifest (`:423-436`) and fetches Gitea generic packages (agent `:1945`, golden `:2573`). Gitea's newest are **agent 0.96.0** and **golden 0.161.0**, and the hub's manifest selects exactly those — so a fresh box lands on **agent 0.96.0 + controller 0.161.0** (global floor `v0.156.0` < the golden's 0.161.0, so no self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and was never published** | **SHIPPED 2026-07-29 — the channel now serves agent 0.113.0 + golden 0.185.1** | — | **FIXED the same day it was found.** Agent **0.113.0** built from the clean tree @ `58b598b` and published (`scripts/publish-agent.sh`), sha `5f3247f756cb658e…`, round-trip GET verified. Golden **0.185.1** baked on the nested drill VM embedding controller `0.185.1`, published, sha `dba00f3e845c415e…` — bake clean: `Result=success`, overlay2, **all 3 mounts included** (rootfs+mp0+mp1), 0 FATAL/exclusions, upload HTTP 201, token-leak grep 0; log `drill/bake-0.185.1.log`; GL-1 teardown done (guest 9100 purged, secrets shredded, disk restored to `virgin`). Hub Day-0 manifest moved **both together in one POST** so it never vouched a new agent against an old golden; `min_agent` **0.93.0 → 0.113.0**, which is what controller v0.185.0 declares (`felhom-controller/CHANGELOG.md:15`) — **zero fleet impact, verified: all three enrolled hosts already run agent 0.113.0, so no box is held.** `wrapper_sha256` preserved verbatim (re-checked against `configs/felhom-pbs-apply` — no drift). **The global controller floor was deliberately NOT raised**: the golden now bakes 0.185.1, so a fresh box needs no self-update, and raising it would have been an unnecessary fleet-wide write. Original finding follows. **Found 2026-07-29 by the E-2d Phase 0 gate, which stopped the run before a VM was created.** 17 unpublished releases (v0.97.0–v0.113.0) strand the **entire R-82 tiered-backup arc** plus **F-CRIT-2** (a failed backup looking fresh — 7 days silent) and **F-REBOOT** (a guest rebooted mid-backup never returns): a new customer's box would install without them. **Blocks E-2d's C3/C4/C5** — those test endpoints and events that do not exist in 0.96.0/0.161.0. The **controller is fine** (registry has 0.185.1, floor-driven self-update), so the gap is specific to the two Gitea-generic artifacts. **Mirror of R-110, not a duplicate:** R-110 = the installer publishes instantly with no staging; R-111 = the agent/golden publish gate exists and was never walked. Fix should decide whether publishing joins the release train rather than staying a remembered step (R-29's shape, one layer up). Evidence: `audits/E2D-fresh-vm-2026-07-29.md` **DEFERRED LEG, AND IT RECURRED → R-115.** This row's shipped half stands and is not reopened: the bump happened, was verified, and was proven end-to-end by the E-2d install. But its own closing line — *decide whether publishing joins the release train rather than staying a remembered step* — was never acted on, and agent 0.114.0 reproduced the exact condition the same afternoon. The recurrence is filed as **R-115**, not as a reopen, because the stale-channel finding is closed while the process defect that caused it is a distinct problem with a distinct fix. | CC |
|
||||
| **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **WAITING-ON-OPERATOR (M)** | operator ruling on the release process | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design | CC |
|
||||
| **R-116** | **The drive-absent alarm and its recovery are a MISMATCHED PAIR — absent fired the GENERIC `storage_disconnected`, return the SPECIFIC `backup_target_restored`** | **FIX SHIPPED + PROVEN-LIVE AT THE PAYLOAD LAYER (agent v0.116.0, 2026-07-30); the EVENTS leg is the one open item** | an emailed controller claim code (→ R-119) | **v0.116.0 works live where v0.115.0 was inert** (`audits/R116-v0116-2026-07-30.md`). Built, published (sha `b47c5c4dab641ee5…`, independent registry GET verified), vouched in the Day-0 manifest, and **installed unaided by a fresh box** — `Artifact manifest served for customer sess-e (agent=0.116.0 golden=0.185.1)`, host `sess-e-5d4427 … 0.116.0 ONLINE`. Real day-0 on a nested PVE on **demo-hp** (per `runbooks/target-selection.md`, which sent this run to demo-hp rather than the DooPlex fixture the prior run used), drives enrolled through the real endpoints, device loss a real hot-detach. **CAPTURED LIVE: the absent target is now ONE row carrying `backup_target:true` AND `guest_path:/mnt/felhom-drives/cel` with `mount_path:""`, so `isTarget["/mnt/felhom-drives/cel"] = TRUE`** (it was `false` through v0.115.0) — the gate will take the specific branch, and RETURNED gives `true` too so the pair matches. All three guards pass from the same payload: **R-114 preserved** (no row combines the flag with a non-empty `mount_path`, so `backup_target_offer.go:79` does not match and the `TargetAbsent` branch stays reachable), **no over-correction** (`bound_under_parent:false`), **discrimination at the payload layer** (the non-target carries the flag on no row — the thing both prior runs could not show). **THE RULING, and it is the substantive output:** the spec's preferred option (back-fill `MountPath` on the Observe row) and its alternative (flag the registry row, whose `MountPath` comes from the stale unit file) **both regress R-114**, which shipped 2026-07-29 — either manufactures a row that `:79` reads as *healthy* while the drive is missing. R-114's correctness rested on the absent rows NOT combining those two fields, a coupling invisible until the payload existed. Taken instead: give the Observe row the **guest path only** (from a new `ConfigPath`, `json:"-"` so the cross-repo golden/key-set contract is untouched) and dedup the registry row **on guest path** — the join being CONFIGURATION (storage.cfg `path` vs the `.mount` unit's `Where`), the only identity that survives the device. Tests 845→849, 4 red-proofs each verified to land; red-proof 1 replays v0.115.0's code and fails, which is the empirical proof it was inert. Its green test had supplied a `MountPath` production never supplies AND left `DriveTargets` nil so the union loop never ran — both corrected. **STILL OPEN — the events on the wire.** `planDriveGates` iterates registered StoragePaths; the drill controller has none (`[WARN] Storage paths: no storage paths registered`) and every storage route answers `401 dashboard not yet claimed`. The claim code is bcrypt-hashed and emailed-only, and `handleSelfBindLinkSend` (`selfbind_mint.go:139-161`) renders a flash, never the token — **no operator-side route exists** (→ **R-119**). A gen-2 code was re-sent 09:07:20Z; the drill VM 9401, `r116-images` and customer `sess-e` are **deliberately RETAINED** with teardown commands recorded, so the leg finishes without a rebuild. **No `backup_target_absent` was seen on the wire this run and none is claimed.** v0.115.0 left in place (inert, harmless). R-118's symptom vanishes incidentally on this one row; R-118 NOT fixed | CC |
|
||||
| **R-116** | ~~**The drive-absent alarm and its recovery were a MISMATCHED PAIR — absent fired the GENERIC `storage_disconnected`, return the SPECIFIC `backup_target_restored`; `backup_target_absent` never fired at all**~~ | **SHIPPED + PROVEN-LIVE** (agent v0.116.0, 2026-07-30) | — | **CLOSED. The full four-event sequence, on the wire, on a fresh box** (`audits/R116-v0116-2026-07-30.md`): `backup_target_absent (error)` on detach → `backup_target_restored (info)` on return for the TARGET, and `storage_disconnected (error)` → `storage_reconnected (info)` for a NON-target drive on the same box four minutes apart. **Two matched pairs, correctly discriminated — and discrimination is proven NON-trivially for the first time**, since both prior runs had the target itself emit the generic event. Gate fired in **3 s**; all four events reached the hub, so the specific alarm, its severity, its Hungarian copy and the hub routing are now exercised end-to-end. **Over-correction PASSES with a positive observable** (0 ABSENT lines / 0 drive events over 2m14s with both drives present, target `degraded:false`, while 2 `RETURNED` lines prove the gate was ticking). Ran on a nested PVE on **demo-hp** per `runbooks/target-selection.md` — through the **real day-0** from the v1.25.0 ISO, with the agent **installed unaided from the vouched Day-0 manifest** (published sha `b47c5c4dab641ee5…`, independent registry GET verified, manifest read back), drives enrolled through the real endpoints, device loss a real hot-detach. **THE FIX, and the ruling is the substantive part:** the mechanism was first isolated from the captured payload (`DIAG-r116-disks-payload-2026-07-30.md`) after two fixes aimed at shapes that do not occur. **Both smaller-looking options were REJECTED because they regress R-114** — `backup_target_offer.go:79` reads `BackupTarget && MountPath != ""` as *"a real drive with its own mountpoint — healthy"* and returns before its `TargetAbsent` branch, so back-filling `MountPath` on the Observe row **or** flagging the registry row (whose `MountPath` is the stale unit-file value) would have told the customer the backup target is fine while its drive was gone. **R-114's correctness was resting on R-116's bug** — a coupling invisible until the payload existed. Taken instead: the Observe row gets the **guest path only** (`mount_path` stays `""`, which is true) from a new `ConfigPath` (`json:"-"`, so the cross-repo golden + key-set contract is untouched), and the union row is deduped **on guest path** — the join being CONFIGURATION (`storage.cfg`'s `path` vs the `.mount` unit's `Where`), the only identity that survives the device. Tests 845→849; 4 red-proofs each asserted to land, and red-proof 1 replays v0.115.0's code and fails, which is the empirical proof it was inert. Its green test had supplied a `MountPath` production never supplies AND left `DriveTargets` nil so the union loop never ran — both corrected. v0.115.0 left in place (inert, harmless). Teardown all 3 layers; hub layer gate-blocked on ONLINE with the command recorded. **Caveat: the drill's controller was 0.185.1 from the golden, which PREDATES R-114**, so its absent-state banner showed the old false copy — the golden being a release behind, not a regression → **R-120** | — |
|
||||
| **R-120** | **The golden bakes a controller that predates R-114 + R-112, so a FRESH customer box shows the WRONG absent-target message.** The Day-0 golden is **0.185.1**; both R-114 (the third state — *configured and its drive is gone* vs *never configured*) and R-112 (the banner having any UI consumer at all) shipped in **0.186.0**. Observed live this run on the drill box with its target detached: the endpoint returned the generic degraded branch — *"A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer"* (**false** — the target is a drive that vanished) **plus** `offer_path: /mnt/felhom-drives/adat`, offering another drive as the remedy. That is E2D §5.3's exact payload, still reachable on any newly installed box | **READY (S)** | — | **This is R-115's class one layer over: baking is a remembered step.** R-111 was closed by re-baking the golden to 0.185.1; 0.186.0 then shipped and the golden was not moved, so the gap reopened silently — and this time the stale artifact carries a **customer-facing falsehood** in exactly the state R-116 now alarms about correctly. Fix is mechanical (bake + publish + vouch a 0.186.0 golden), but the decision that matters is whether the golden joins the release train or the manifest gains a gate that refuses to vouch a golden older than the newest controller — **reminders have already failed twice here** (R-111 → R-115). Evidence: `audits/R116-v0116-2026-07-30.md` §6a caveat + §1 | CC |
|
||||
| **R-117** | **A returned drive's guest bind is a DEAD MOUNT, and the gate restarts the customer's apps onto it while every signal reads healthy.** After a detach/reattach the host raw mount is fine on the NEW device node, but the guest's bind still names the OLD one and every operation through it returns `EIO`: host `findmnt /mnt/cel` → `/dev/vdc`; guest `findmnt /mnt/felhom-drives/cel` → `/dev/vdb[/felhom-data] …,shutdown`; guest `ls` and write → **`Input/output error`**; and `/disks` reports `state:"attached"`, `role:"user-data"`, `bound_under_parent:TRUE`, `backup_target:true` | **READY (M) — NEW 2026-07-30, deliberately not chased** | — | **Worse than R-116 and it outranks it.** `planDriveGates` computes `present[gp] = … \|\| d.BoundUnderParent` (`intermediary.go:226`) = `true`, and the drive is `Disconnected`, so the gate takes the **`Return`** branch (`intermediary.go:281`, `:301-320`): it clears the flag, **restarts the gate-stopped apps onto a namespace that `EIO`s on every call**, reports the drive healthy and emails `backup_target_restored`. **No alarm on any channel.** R-113's conjunction cannot catch it — half 1 (guest mountinfo has an entry at the guest path) is satisfied by the STALE entry and half 2 (`devicePresent`) by the NEW device; **neither half compares the two**. Not a false invariant: `disks.go:158-165` claims only the boot-ordering and outlived-bind cases, so this is a genuine gap. **Seen but uninvestigated in E-2d, Session C and Part 5 (three consecutive runs) as "the drive returned as /dev/sdc while findmnt still read /dev/sdb"** — that framing read as cosmetic; it is not. Reproduced deliberately and probed to EIO in `audits/DIAG-r116-disks-payload-2026-07-30.md` §11. Deserves its own run | CC |
|
||||
| **R-118** | **An absent drive's union row advertises the ROOT filesystem's capacity as its own.** In the absent-state payload the registry-union row reports `total_bytes: 49675956224 / used_bytes: 4584579072` — **byte-identical to the `local` row** (`durable_id: path:/var/lib/vz`, i.e. `pve-root`) in the same response. The real drive is **4 GB** | **READY (XS) — NEW 2026-07-30** | — | Cause: `statfsCapacity(d.MountPath)` (`disks.go:335-338`) statfs's `/mnt/cel`, which with the device gone is a **bare directory on the root filesystem**. `observe.go:176-183`'s comment warns about exactly this trap and guards the Observe path ("*an unmounted removable dir-storage's mountpoint reverts to a bare directory on root … catastrophic DR mis-id*"); **the union path has no equivalent guard.** **Not a DR mis-id** — `durable_id` on that row is still the correct `uuid:…`, so re-attach identity is safe. It is a **false capacity** reaching every consumer of `total_bytes`/`used_fraction` (fill monitors, storage cards): a detached 4 GB drive advertises 46 GiB at 9.2 % used. Same class as `role.go:180-181` — an absent drive's fields decaying to the root filesystem's. Evidence: `audits/DIAG-r116-disks-payload-2026-07-30.md` §12 | CC |
|
||||
| **R-113** | ~~**The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.**~~ `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc/<pid>/mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **SHIPPED + PROVEN-LIVE** (agent v0.114.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** Target drive hot-detached on a fresh box running the SHIPPED agent 0.114.0 (from the Day-0 manifest, not a hand build). **The gate fired in 4 seconds** — E-2d measured ZERO over 4½ minutes — and `SetDisconnected` was reached. It fired on exactly the shape that defeated it: raw `/mnt/mentes` NOT mounted while the bind `/mnt/felhom-drives/mentes` still read `/dev/sdb[/felhom-data]`. **Over-correction guard PASSED:** with both drives present, 0 ABSENT lines and the target stayed healthy. **Note: C5 still FAILED — but on a different defect (→ R-116), not on this one.** **SESSION C IS NOW UNBLOCKED.** Agent **0.114.0 published and vouched** 2026-07-29 (sha `5e4c15ebee2d7583…`, round-trip GET verified; hub manifest read back showing it) — a fresh drill box now installs the fix under test instead of the bug. The golden was NOT re-baked and `min_agent` was NOT raised: the golden bakes the controller, not the agent, and controller v0.185.0 declares MinAgent 0.113.0, which 0.114.0 already satisfies. **FIX: `BoundUnderParent` is now a CONJUNCTION — bound under the parent AND the drive's raw host mount still mounted** (`devicePresent`, new `deviceCheck` seam), at BOTH `/disks` construction sites. The raw mount is the device-bound systemd unit that dies with the device; the agent's bind is not — so the raw mount IS the device signal, grounded in E-2d's measurement rather than inference. **Conjunction, deliberately:** the device half alone would regress boot ordering (raw mounts early, bind lands ~18 s later — that window must keep reading absent), so existing behaviour is byte-identical and only the unreachable case is closed. **Unknown is never absent** (`devicePresent("")` = true) — a false absent stops a working customer's apps. **Controller UNCHANGED, no MinAgent bump:** `BoundUnderParent` has exactly one functional consumer (`planDriveGates:226`); a new `DevicePresent` bool was rejected because absent-from-JSON decodes to `false`, which would have made every drive on an older agent read ABSENT. +6 tests (208→214), 4 red-proofs run and reverted. **Deployed to demo-felhom; over-correction guard verified in production** — raw mount present, drive still reads present, 10/10 apps untouched, no gate action, no false alarm. **demo-hp deliberately left on 0.113.0** (the spec scoped deploy to felhom-pve) — it still carries the bug. **⚠️ SESSION C BLOCKER: the hub Day-0 manifest vouches agent 0.113.0**, so a fresh drill box would install WITHOUT this fix and validate nothing — publish + vouch 0.114.0 first (R-111's trap, same shape). **Awaiting live proof of exactly one leg:** device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. Original finding: **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 **Session C scope UNCHANGED by Session B** — R-114/R-112 shipped as controller v0.186.0 and neither touches the agent; the leg awaiting proof is still device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. One rebuild now validates all three. | CC |
|
||||
|
||||
Reference in New Issue
Block a user