# SESSION-C-2026-07-29 — R-113, R-114, R-112 proven on a real box; C5 fails on a new defect **Run:** `RUNBOOK-session-c-2026-07-29.md`, executed by CC on DooPlex, 2026-07-29. **Session C of three.** A → R-113 (agent v0.114.0) · B → R-114 + R-112 (controller v0.186.0) · **C → this.** | Claim | Fix | Verdict | |---|---|---| | **C4** — the offer appears and moves the target when accepted | R-112 | ✅ **PASS** | | **C3** — with the target absent the customer is told the drive is missing, and offered nothing | R-114 | ✅ **PASS** | | **C5** — drive loss produces `backup_target_absent` (error) and `backup_target_restored` (info) | R-113 | ❌ **FAIL** — the alarm fires, but as the **generic** event; the recovery is the **specific** one. Unmatched pair → **R-116** | **All three fixes are proven to work.** R-113's gate fires, R-114's message is correct, R-112's banner reaches the customer. C5 fails on a **fourth, separate defect** that only became reachable *because* R-113 made the gate fire at all — it could not have been seen before. --- ## 1. Baselines, and what the drill box actually ran | Artifact | Vouched / used | How it got there | |---|---|---| | agent | **0.114.0** | **from the Day-0 manifest** — `Artifact manifest served for sess-c (agent=0.114.0 golden=0.185.1)`. C5 therefore tested the **shipped** binary, closing R-115 observation 1 for R-113 | | golden | 0.185.1 | vouched; bakes controller 0.185.1 | | controller | **0.186.0** | **hand-deployed after install** (§3.1 ruling (a)) via the bootstrap path — `0.185.1 → 0.186.0`, `Up 21 seconds (healthy)` | | hub | 0.81.0 | unchanged | | host-install | 1.22.0 | unchanged | Repos at run time: controller `b331f18`, agent `b58d7bc`, felhom.eu `af518ba`, all clean. **§3.1 ruling — (a), deploy 0.186.0 by hand.** A bake is its own session and would have inserted one before any verdict; the install path is already proven by E-2d. **Caveat, as required: C3/C4 prove the CODE, not the SHIPPED GOLDEN.** The vouched golden lagging a controller release is R-115's shape and is filed there, not as a new ID. **§3.2 cadence** — `driveGateLoop` registered `server.go:224`, defined `intermediary.go:328`, ticker **30 s** (`:339`) ⇒ C5 budget 60 s. **§3.3** — SCSI hot-detach, volume survives as `unused0`. Fence re-measured before and after: `local-lvm` **38.78 %** both times. ## 2. Timeline (VM 9301 `sessionc`, nested PVE on demo-hp) | UTC | Event | |---|---| | 18:29 | VM created — one disk, q35/OVMF SB-off, `hotplug disk`, outside the `felhom` pool | | 18:33:47 | registered unclaimed, pairing code `2XT-3PV` | | 18:34:08 | **bound to `sess-c` by CC** — no operator STOP (E-2d retired it) | | 18:34:57 | manifest served: **agent 0.114.0**, golden 0.185.1 | | 18:38:03 | `controller_started (0.185.1)` — install complete, bind→controller in **3 m 55 s** | | ~18:40 | controller **0.186.0** deployed; dashboard claimed with the emailed code | | 18:43:50 | **target drive hot-detached** | | **18:43:54** | **gate fired — 4 seconds** | | 18:47:01 | reattached | | 18:47:24 | gate RETURNED (23 s) | | 18:50:32 | non-target drive detached (mirror) | | 18:50:54 | gate fired (22 s) | ## 3. C4 — PASS (R-112 + the assign flow) **The banner reached a customer's page for the first time.** E-2d could only prove the JSON. Never-configured box, before any second drive: ``` GET /backups http=200 banner element : 1 never-configured copy : 1 <- "ugyanazon a lemezen van" IS in the HTML absent copy : 0 offer block : 0 <- correct: no eligible drive registered yet ``` After attaching a 100 GB disk and bringing it in **through the dashboard wizard**: ``` banner element : 1 offer block : 1 data-path="/mnt/felhom-drives/mentes" ``` **Decline path — PROVEN.** After registering and *not* accepting: `target` still `local`, **no** `felhom-backup` storage, `agent.json` `local_backup_target = local`. Registration confers no role. **Accept:** `{"assigned":"/mnt/felhom-drives/mentes","restart_required":true}`. **The agent did NOT self-restart** — `ActiveEnterTimestamp` 20:35:15 CEST, unchanged at 20:42:20. **In-flight check performed and recorded before restarting:** 0 `vzdump` processes, 0 backup lines in the agent journal (5 min), 0 running PVE tasks. Wrapper created `dir: felhom-backup, path /mnt/mentes, is_mountpoint 1` — the drive's **own** mountpoint. **After restart, healthy renders nothing — and it is proven POSITIVELY, not by an empty page:** ``` banner element 0 · offer block 0 · degraded copy 0 · absent copy 0 /backup/tiers calls before 6 · after idle 8s 6 (delta 0) · after page load 7 (delta +1) ``` That call has exactly one caller (`resolveBackupTargetState`), so **the seam ran and chose silence**. ## 4. C3 — PASS (R-114) With the target drive detached, the customer's page: ``` banner element : 1 ABSENT copy (correct) : 1 <- "vissza nem csatlakoztatod" system-disk copy (BAD): 0 <- E-2d saw THIS offer block (BAD) : 0 <- E-2d offered the VANISHED drive ``` API: `{"degraded":true,"target":"felhom-backup","message":"A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod, a teljes rendszermentés nem készül el."}` Both of E-2d's falsehoods are gone. ## 5. C5 — FAIL, and R-113 is nonetheless PROVEN **R-113 works. This is the headline.** E-2d measured **zero** gate activity over 4½ minutes. Here: ``` 18:43:54 [INFO] [settings] Storage path /mnt/felhom-drives/mentes disconnected=true 18:43:54 [WARN] [gate] drive ABSENT /mnt/felhom-drives/mentes — stopped+blocked 0 app(s): [] ``` **4 seconds**, and `SetDisconnected` reached — the flag R-114 and the offer suppression depend on. And it fired on **exactly the shape that defeated it before**: ``` raw /mnt/mentes : NOT mounted <- device-bound, died with the device bind /mnt/felhom-drives/mentes : /dev/sdb[/felhom-data] <- the stale bind SURVIVED ``` The bind outlived the device, as always; the gate now sees past it. **What fails is which event is sent.** ``` absent : Event pushed: storage_disconnected (error) <- GENERIC return : Event pushed: backup_target_restored (info) <- SPECIFIC ``` C5 required `backup_target_absent`. It never fired — `backup_target_absent` count **0** across the run. Worse than a missing label: the alarm and its recovery are a **mismatched pair**, which is precisely what `notifyDriveReturned`'s own comment forbids — *"a target that alarmed as backup_target_absent has to recover as backup_target_restored, not as a generic reconnect the operator cannot match"*. Here it is inverted: generic alarm, specific recovery. ### Root cause, established at source `notifyDriveAbsent` (`intermediary.go:635-646`) branches on `isTarget[path]`, built by `driveTargetByPath` (`:602-616`) as `out[GuestPath] = d.BackupTarget` / `out[MountPath] = d.BackupTarget`. The gate's `a.Path` is the **guest** path. The drive is represented by **two `/disks` rows**, and the flag and the guest path sit on different ones: | Row | Source | `BackupTarget` | `GuestPath` | |---|---|---|---| | the `felhom-backup` storage | Observe path, `felhom-agent/internal/localapi/disks.go:211` (`t.Name == primaryTargetID`) | **true** | set only when role is user-data | | the registry drive | union path, `disks.go:265-267` — **never assigns `BackupTarget` at all** | **false** | `/mnt/felhom-drives/mentes` | > **CORRECTION 2026-07-29 (R-116 Phase 0).** This section originally said the union row *"writes > `false`"*. **That is wrong**, and the real mechanism is worse — the union row is not written at all. > > The step left un-isolated above has now been isolated: `RoleForStorage` returns `RoleSystem` whenever > `backingDevice == ""` (`felhom-agent/internal/storage/role.go:180-181`). So when the device vanishes, > Observe's `exactMountDevice` fails, `t.BackingDevice` becomes `""`, the row's role flips to system and > it loses its guest path — **but it keeps its `MountPath`**. The union loop skips any drive whose > `MountPath` is already `seen` (`disks.go:263`), so the registry row is **DEDUPED AWAY ENTIRELY**. > > `/disks` therefore carries **no row with that guest path at all**: `isTarget[guestPath]` is a > **missing key**, not a `false` value. The practical difference is decisive — the obvious fix (set > `BackupTarget` on the union row) **could not have worked**, because that row is not emitted in the > state where the alarm is needed. On **return** the rows rejoin and the specific event fires — which is exactly the asymmetry observed. v0.184.1 fixed the *keying* (both keys are mapped); it did not fix the flag being absent from the row that carries the guest path. **Filed as R-116.** Not fixed: a defect found in a validation run is a finding. ## 6. Mirror scenario + over-correction guard — PASS, with a caveat A **non-target** drive (`adat`, 70 GB, registered but never assigned) detached at 18:50:32: ``` 18:50:54 [WARN] [gate] drive ABSENT /mnt/felhom-drives/adat — stopped+blocked 0 app(s) 18:50:54 [INFO] Event pushed: storage_disconnected (error) — Meghajtó váratlanul leválasztva: Adat meghajto backup_target_absent count : 0 storage_disconnected count : 1 ``` **Over-correction guard — the real risk of R-113 — PASSES.** With both drives present the gate produced **0** ABSENT lines, and the target stayed `degraded:false` with banner element 0 throughout the whole mirror sequence. R-113's stricter presence did not make a healthy drive read absent. **Caveat, stated rather than glossed:** because the *target* drive also produced the generic event, the mirror passes **trivially**. It confirms no over-correction; it cannot confirm specific-vs-generic discrimination, because nothing produced the specific event on absence at all. ## 7. Findings 1. **R-116 (new)** — the absent alarm is generic while the recovery is specific; the pair cannot be matched by an operator. Root cause §5. **This defect was unreachable before R-113**: the gate never ran, so the branch was never taken. Fixing one bug exposed the next, which is what a validation run is for. 2. **The stale bind survives a return, still naming a dead device node.** After the drive came back as `/dev/sdc`, `findmnt /mnt/felhom-drives/mentes` still read `/dev/sdb[/felhom-data]` while the state reported `degraded:false`. E-2d noticed this and did not chase it; it is now confirmed. Whether apps writing through that path reach the live device was **not** established — worth its own look. 3. **The capability map still has no E-2 / backup-target rows at all**, so no row could be moved to PROVEN-LIVE. Creating them is a design act, not a validation act; filed rather than invented (third session in a row this has been noted). ## 8. Teardown VM 9301 destroyed `--purge`, `sc-images` storage removed, scratch dir removed. **`pvesm status` after == before** (`local-lvm` **38.78 %**, `felhom-backup` 931059224 KiB available). **Guest 9201 and `drill-r50` untouched.** **Customer ruling: DELETE** (R-93's lesson — no half-real fixtures). Attempted and **correctly refused**: `delete sess-c REFUSED: host sess-c-8158e4 is ONLINE`. The VM is destroyed so no further reports come; the host ages OK → WARN (30 m) → DOWN (>1 h) and is then deletable. Command, unchanged from E-2d: ``` POST /configs/sess-c/delete ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-c expect_hosts=1 ``` ## 9. What did not happen - **`backup_target_absent` never fired**, so its severity, its Hungarian customer copy and its hub routing are all still unexercised end-to-end. R-116 blocks them. - **The offer control was accepted via the endpoint it POSTs**, not by a browser click — no browser automation exists on DooPlex. The rendered control, its `data-path` and its non-auto-submission were verified in the HTML. - The stale-bind-after-return question (finding 2) was observed, not investigated.