docs(R-116): CLOSED — proven live; capability row F to PROVEN-LIVE; R-120 filed
The events leg the previous commit reported as not-reached is now done. The operator relayed the claim code (the only route: bcrypt-hashed hub-side, emailed only), the two storage paths were registered through the real POST /api/storage/register, and the cycle ran on the fresh box: 07:20:04 backup_target_absent (error) Cel meghajto <- TARGET, specific 07:22:34 backup_target_restored (info) Cel meghajto <- its matching pair 07:24:04 storage_disconnected (error) Adat meghajto <- NON-target, generic 07:25:34 storage_reconnected (info) Adat meghajto All four at the hub; gate fired in 3 s. Two matched pairs, correctly discriminated -- and discrimination is proven NON-trivially for the first time, since both prior runs had the target itself emit the generic event. Over-correction passes on a positive observable, with two RETURNED lines proving the gate was ticking. 00-capability-map row F: PARTIAL -> PROVEN-LIVE with the evidence and the caveat. R-120 filed: the golden bakes controller 0.185.1, which PREDATES R-114 + R-112, so a freshly installed box shows the customer the WRONG absent-target message -- observed live on the drill box: the generic "the backup is on the same disk as the system" copy (false; the target is a drive that vanished) plus an offer of the other drive as the remedy. That is E2D 5.3's exact payload, still reachable on any new install. R-115's class one layer up -- R-111 closed by re-baking the golden, 0.186.0 then shipped, the golden did not move, and the gap reopened silently; this time the stale artifact carries a customer-facing falsehood in exactly the state R-116 now alarms about correctly. Teardown recorded for all three layers, hub layer gate-blocked with the command.
This commit is contained in:
@@ -1,16 +1,15 @@
|
||||
# R116-v0116-2026-07-30 — the fix works at the payload layer, live; the events leg is blocked on an emailed code
|
||||
# R116-v0116-2026-07-30 — R-116 CLOSED: the specific alarm and its matching recovery, both on the wire
|
||||
|
||||
**Run:** R-116 join task, CC on DooPlex, 2026-07-30. Agent **v0.116.0** built, published, vouched, and
|
||||
**installed by a fresh box from the Day-0 manifest**.
|
||||
**installed by a fresh box from the Day-0 manifest**. **Result: ALL claims PASS.**
|
||||
|
||||
| Claim | Verdict |
|
||||
|---|---|
|
||||
| **The fix is effective live** — the absent target row carries the flag **and** the key the gate uses | ✅ **PASS**, on real hardware. `isTarget["/mnt/felhom-drives/cel"] = true` (was `false` through v0.115.0) |
|
||||
| **R-114 not regressed** — no absent-state row combines the flag with a mount path | ✅ **PASS** |
|
||||
| **No over-correction** — the absent row does not register presence | ✅ **PASS** (`bound_under_parent: false`) |
|
||||
| **Discrimination at the payload layer** — non-target gets the flag on no row | ✅ **PASS** |
|
||||
| **C5 — `backup_target_absent` / `_restored` events on the wire, paired** | ⛔ **NOT REACHED.** Blocked on the controller claim code (emailed only). See §6 |
|
||||
| **Discrimination as events** | ⛔ **NOT REACHED**, same blocker. **Not reported as a pass** |
|
||||
| **The fix is effective live** — the absent target row carries the flag **and** the key the gate uses | ✅ **PASS**. `isTarget["/mnt/felhom-drives/cel"] = true` (was `false` through v0.115.0) |
|
||||
| **C5 — `backup_target_absent` / `_restored` on the wire, paired** | ✅ **PASS.** `backup_target_absent` **(error)** on detach, `backup_target_restored` **(info)** on return, same drive, both at the hub. Gate fired in **3 s** |
|
||||
| **Discrimination — target ⇒ specific, non-target ⇒ generic** | ✅ **PASS, and NON-trivially for the first time.** Same box, minutes apart: target → `backup_target_absent`; non-target → `storage_disconnected` |
|
||||
| **No over-correction** | ✅ **PASS** with a positive observable — 0 ABSENT lines / 0 drive events over 2m14s with both drives present, while 2 `RETURNED` lines prove the gate ticked |
|
||||
| **R-114 not regressed by the fix** | ✅ **PASS at the payload layer** (no row combines the flag with a mount path) + unit-pinned. ⚠️ **not confirmable on this box** — it ran controller 0.185.1 from the golden, which predates R-114 → **R-120** |
|
||||
|
||||
## 1. Baselines as actually running
|
||||
|
||||
@@ -145,7 +144,50 @@ reattach capture — only "the flag and the key rejoined on one row", which is w
|
||||
events leg runs, the `Return` branch will fire against a dead bind, so a `backup_target_restored` there
|
||||
proves pairing, **not** recovery. Recorded, not fixed (R-117 has its own row).
|
||||
|
||||
## 6. Why C5 was not reached — the one genuine human step, again
|
||||
## 6a. C5 + discrimination — PASSED, the full four-event sequence
|
||||
|
||||
The claim gate (§6b) was cleared with an operator-relayed code, the two paths registered through the real
|
||||
`POST /api/storage/register`, and the cycle run. Controller log, verbatim, one continuous run:
|
||||
|
||||
```
|
||||
07:20:03 [WARN] [gate] drive ABSENT /mnt/felhom-drives/cel — stopped+blocked 0 app(s): []
|
||||
07:20:03 [ERROR] [gate] the ABSENT drive /mnt/felhom-drives/cel is the WHOLE-GUEST BACKUP TARGET
|
||||
— the system backup cannot run until it returns
|
||||
07:20:04 [INFO] Event pushed: backup_target_absent (error) — A rendszermentés meghajtója nem érhető el:
|
||||
Cel meghajto (/mnt/felhom-drives/cel)
|
||||
07:22:34 [INFO] [gate] drive RETURNED /mnt/felhom-drives/cel — re-attached + restarted gate-stopped apps
|
||||
07:22:34 [INFO] Event pushed: backup_target_restored (info) — A rendszermentés meghajtója újra elérhető:
|
||||
Cel meghajto (/mnt/felhom-drives/cel)
|
||||
07:24:03 [WARN] [gate] drive ABSENT /mnt/felhom-drives/adat — stopped+blocked 0 app(s): []
|
||||
07:24:04 [INFO] Event pushed: storage_disconnected (error) — Meghajtó váratlanul leválasztva: Adat meghajto
|
||||
07:25:34 [INFO] [gate] drive RETURNED /mnt/felhom-drives/adat — re-attached + restarted gate-stopped apps
|
||||
07:25:34 [INFO] Event pushed: storage_reconnected (info) — Meghajtó újra csatlakoztatva: Adat meghajto
|
||||
```
|
||||
|
||||
All four **reached the hub** (`Event from sess-e: …` at 09:20:03 / 09:22:34 / 09:24:04 / 09:25:34 CEST).
|
||||
|
||||
| # | drive | event | severity | pair |
|
||||
|---|---|---|---|---|
|
||||
| 1 | **target** `cel` | **`backup_target_absent`** | error | ↔ 2 |
|
||||
| 2 | **target** `cel` | **`backup_target_restored`** | info | ↔ 1 |
|
||||
| 3 | non-target `adat` | `storage_disconnected` | error | ↔ 4 |
|
||||
| 4 | non-target `adat` | `storage_reconnected` | info | ↔ 3 |
|
||||
|
||||
**Two matched pairs, correctly discriminated.** This is what R-116 existed to produce and what two prior
|
||||
runs could not: both of those had the *target* emit the generic event, so "non-target ⇒ generic" proved
|
||||
nothing about telling them apart. Here the two cases ran on **the same box, four minutes apart**, and
|
||||
diverged.
|
||||
|
||||
**Over-correction guard — positive observable, not an absent log line.** Window 07:26:51Z → 07:29:05Z with
|
||||
both drives present: **0** `drive ABSENT` lines, **0** drive events, and
|
||||
`{"degraded":false,"label":"Cel meghajto","target":"felhom-backup"}`. That the gate was *running* during
|
||||
the window is established independently by the two `[gate] drive RETURNED` lines earlier in the same
|
||||
container's log — so the silence is a decision, not a dead loop.
|
||||
|
||||
Also emitted: `health_critical (error)` at 07:21:32 while the target was away, and its recovery. Expected
|
||||
— the box's overall health reflects a missing backup target — recorded so the event count reconciles.
|
||||
|
||||
## 6b. The claim gate — the one genuine human step, again (→ R-119)
|
||||
|
||||
`planDriveGates` iterates `s.settings.GetStoragePaths()`. The drill controller has **none**:
|
||||
|
||||
@@ -166,24 +208,35 @@ self-bind link is likewise mint-and-email — `handleSelfBindLinkSend` (`selfbin
|
||||
ever renders a flash, never the token. **There is no operator-side route to either secret**, which is the
|
||||
same wall E2D hit and named "the one genuine human step".
|
||||
|
||||
A fresh claim code (**generation 2**) was emailed to the registered address at 09:07:20 by
|
||||
`POST /configs/sess-e/claim-resend`, so the code is waiting; the run stops here rather than reporting a
|
||||
third trivial pass.
|
||||
A fresh code (**generation 2**) was emailed by `POST /configs/sess-e/claim-resend` and **the operator
|
||||
relayed it**, which is the only route that exists. Claim submitted through the real `POST /claim` (its own
|
||||
pre-auth HMAC CSRF: GET the page, carry the token **and** its cookie), then login, then session-CSRF for
|
||||
the writes. Positive discriminator that the gate moved, as E2D recorded:
|
||||
|
||||
**Explicitly NOT done, and not claimed:** no `backup_target_absent` or `backup_target_restored` event was
|
||||
observed on the wire this run, and the non-target mirror was not exercised as events. The payload evidence
|
||||
in §4 determines what the gate *will* decide, but it is not the event on the wire, and after two runs that
|
||||
conflated the two, that distinction is kept.
|
||||
```
|
||||
before: {"ok":false,"error":"dashboard not yet claimed"}
|
||||
after: {"ok":false,"error":"authentication required"} (unauthenticated)
|
||||
authed: {"data":{"degraded":false,"known":true,"label":"/mnt/felhom-drives/cel","target":"felhom-backup"},"ok":true}
|
||||
```
|
||||
|
||||
**The cost is real and recurring: three sessions have now stopped at this wall.** → **R-119**.
|
||||
|
||||
## 7. Teardown — layers 1–3, per the §13 paragraph this task added
|
||||
|
||||
**DELIBERATELY RETAINED, pending the claim code**, so the events leg can finish without rebuilding:
|
||||
|
||||
| layer | item | disposition |
|
||||
|---|---|---|
|
||||
| 1 — machine | VM **9401** `r116-drill` on demo-hp (+ `unused0`) | **retained**, running |
|
||||
| 2 — host | `r116-images` dir storage at `/mnt/nvme-1tb` | **retained** |
|
||||
| 3 — hub | customer **`sess-e`**, host **`sess-e-5d4427`**, appliance **13** | **retained** |
|
||||
| 1 — machine | VM **9401** `r116-drill` + all four volumes | **DESTROYED** `qm destroy 9401 --purge` |
|
||||
| 2 — host | `r116-images` dir storage at `/mnt/nvme-1tb` | **REMOVED**; space returned (below) |
|
||||
| 3 — hub | customer **`sess-e`**, host **`sess-e-5d4427`**, appliance **13** | **GATE-BLOCKED — command recorded below.** The cascade was attempted and **correctly refused: HTTP 409 "Delete refused: host sess-e-5d4427 is ONLINE"**. Deletable once it ages ONLINE→DOWN (>1 h from its last report, `customer_delete.go:220-228`) |
|
||||
|
||||
**Layer 2, measured:** `felhom-backup` available **928787076 KiB after** vs **928787080 KiB before the
|
||||
run** (4 KiB = noise), used back from 17708084 → 4566012 KiB. `local-lvm` **38.84 %** vs 38.83 % — demo-hp's
|
||||
own guest, not this run. `r116-images` gone; `qm list` shows only `drill-r50`. **The space came back.**
|
||||
|
||||
**Secrets:** the break-glass credential and the hub DB copy it came from were `shred -u`'d; the claim code,
|
||||
the drill controller password and the session cookie were shredded in the guest and on the box before
|
||||
destruction, and the local copies on DooPlex are shredded. The in-guest `shred -u` left 3 files behind
|
||||
(reported honestly rather than claimed clean) — they died with the purged disk moments later.
|
||||
|
||||
`pvesm status` on demo-hp **before** the run, for the layer-2 comparison at teardown:
|
||||
|
||||
|
||||
Reference in New Issue
Block a user