felhom.eu/documentation/audits/SESSION-C-2026-07-29.md

SESSION-C-2026-07-29 — R-113, R-114, R-112 proven on a real box; C5 fails on a new defect

Run: RUNBOOK-session-c-2026-07-29.md, executed by CC on DooPlex, 2026-07-29. Session C of three. A → R-113 (agent v0.114.0) · B → R-114 + R-112 (controller v0.186.0) · C → this.

Claim	Fix	Verdict
C4 — the offer appears and moves the target when accepted	R-112	✅ PASS
C3 — with the target absent the customer is told the drive is missing, and offered nothing	R-114	✅ PASS
C5 — drive loss produces backup_target_absent (error) and backup_target_restored (info)	R-113	❌ FAIL — the alarm fires, but as the generic event; the recovery is the specific one. Unmatched pair → R-116

All three fixes are proven to work. R-113's gate fires, R-114's message is correct, R-112's banner reaches the customer. C5 fails on a fourth, separate defect that only became reachable because R-113 made the gate fire at all — it could not have been seen before.

---

1. Baselines, and what the drill box actually ran

Artifact	Vouched / used	How it got there
agent	0.114.0	from the Day-0 manifest — Artifact manifest served for sess-c (agent=0.114.0 golden=0.185.1). C5 therefore tested the shipped binary, closing R-115 observation 1 for R-113
golden	0.185.1	vouched; bakes controller 0.185.1
controller	0.186.0	hand-deployed after install (§3.1 ruling (a)) via the bootstrap path — 0.185.1 → 0.186.0, Up 21 seconds (healthy)
hub	0.81.0	unchanged
host-install	1.22.0	unchanged

Repos at run time: controller b331f18, agent b58d7bc, felhom.eu af518ba, all clean.

§3.1 ruling — (a), deploy 0.186.0 by hand. A bake is its own session and would have inserted one before any verdict; the install path is already proven by E-2d. Caveat, as required: C3/C4 prove the CODE, not the SHIPPED GOLDEN. The vouched golden lagging a controller release is R-115's shape and is filed there, not as a new ID.

§3.2 cadence — driveGateLoop registered server.go:224, defined intermediary.go:328, ticker 30 s (:339) ⇒ C5 budget 60 s. §3.3 — SCSI hot-detach, volume survives as unused0.

Fence re-measured before and after: local-lvm 38.78 % both times.

2. Timeline (VM 9301 sessionc, nested PVE on demo-hp)

UTC	Event
18:29	VM created — one disk, q35/OVMF SB-off, hotplug disk, outside the felhom pool
18:33:47	registered unclaimed, pairing code 2XT-3PV
18:34:08	bound to sess-c by CC — no operator STOP (E-2d retired it)
18:34:57	manifest served: agent 0.114.0, golden 0.185.1
18:38:03	controller_started (0.185.1) — install complete, bind→controller in 3 m 55 s
~18:40	controller 0.186.0 deployed; dashboard claimed with the emailed code
18:43:50	target drive hot-detached
18:43:54	gate fired — 4 seconds
18:47:01	reattached
18:47:24	gate RETURNED (23 s)
18:50:32	non-target drive detached (mirror)
18:50:54	gate fired (22 s)

3. C4 — PASS (R-112 + the assign flow)

The banner reached a customer's page for the first time. E-2d could only prove the JSON.

Never-configured box, before any second drive:

GET /backups http=200
  banner element        : 1
  never-configured copy : 1      <- "ugyanazon a lemezen van" IS in the HTML
  absent copy           : 0
  offer block           : 0      <- correct: no eligible drive registered yet

After attaching a 100 GB disk and bringing it in through the dashboard wizard:

  banner element : 1     offer block : 1     data-path="/mnt/felhom-drives/mentes"

Decline path — PROVEN. After registering and not accepting: target still local, no felhom-backup storage, agent.json local_backup_target = local. Registration confers no role.

Accept: {"assigned":"/mnt/felhom-drives/mentes","restart_required":true}. The agent did NOT self-restart — ActiveEnterTimestamp 20:35:15 CEST, unchanged at 20:42:20. In-flight check performed and recorded before restarting: 0 vzdump processes, 0 backup lines in the agent journal (5 min), 0 running PVE tasks. Wrapper created dir: felhom-backup, path /mnt/mentes, is_mountpoint 1 — the drive's own mountpoint.

After restart, healthy renders nothing — and it is proven POSITIVELY, not by an empty page:

  banner element 0 · offer block 0 · degraded copy 0 · absent copy 0
  /backup/tiers calls  before 6 · after idle 8s 6 (delta 0) · after page load 7 (delta +1)

That call has exactly one caller (resolveBackupTargetState), so the seam ran and chose silence.

4. C3 — PASS (R-114)

With the target drive detached, the customer's page:

  banner element        : 1
  ABSENT copy (correct) : 1     <- "vissza nem csatlakoztatod"
  system-disk copy (BAD): 0     <- E-2d saw THIS
  offer block (BAD)     : 0     <- E-2d offered the VANISHED drive

API: {"degraded":true,"target":"felhom-backup","message":"A rendszermentés meghajtója nem érhető el — amíg vissza nem csatlakoztatod, a teljes rendszermentés nem készül el."}

Both of E-2d's falsehoods are gone.

5. C5 — FAIL, and R-113 is nonetheless PROVEN

R-113 works. This is the headline. E-2d measured zero gate activity over 4½ minutes. Here:

18:43:54 [INFO] [settings] Storage path /mnt/felhom-drives/mentes disconnected=true
18:43:54 [WARN] [gate] drive ABSENT /mnt/felhom-drives/mentes — stopped+blocked 0 app(s): []

4 seconds, and SetDisconnected reached — the flag R-114 and the offer suppression depend on. And it fired on exactly the shape that defeated it before:

raw   /mnt/mentes               : NOT mounted        <- device-bound, died with the device
bind  /mnt/felhom-drives/mentes : /dev/sdb[/felhom-data]   <- the stale bind SURVIVED

The bind outlived the device, as always; the gate now sees past it.

What fails is which event is sent.

absent : Event pushed: storage_disconnected (error)      <- GENERIC
return : Event pushed: backup_target_restored (info)     <- SPECIFIC

C5 required backup_target_absent. It never fired — backup_target_absent count 0 across the run. Worse than a missing label: the alarm and its recovery are a mismatched pair, which is precisely what notifyDriveReturned's own comment forbids — "a target that alarmed as backup_target_absent has to recover as backup_target_restored, not as a generic reconnect the operator cannot match". Here it is inverted: generic alarm, specific recovery.

Root cause, established at source

notifyDriveAbsent (intermediary.go:635-646) branches on isTarget[path], built by driveTargetByPath (:602-616) as out[GuestPath] = d.BackupTarget / out[MountPath] = d.BackupTarget. The gate's a.Path is the guest path.

The drive is represented by two /disks rows, and the flag and the guest path sit on different ones:

Row	Source	BackupTarget	GuestPath
the felhom-backup storage	Observe path, felhom-agent/internal/localapi/disks.go:211 (t.Name == primaryTargetID)	true	set only when role is user-data
the registry drive	union path, disks.go:265-267 — never assigns BackupTarget at all	false	/mnt/felhom-drives/mentes

CORRECTION 2026-07-29 (R-116 Phase 0). This section originally said the union row "writes false". That is wrong, and the real mechanism is worse — the union row is not written at all.

The step left un-isolated above has now been isolated: RoleForStorage returns RoleSystem whenever backingDevice == "" (felhom-agent/internal/storage/role.go:180-181). So when the device vanishes, Observe's exactMountDevice fails, t.BackingDevice becomes "", the row's role flips to system and it loses its guest path — but it keeps its MountPath. The union loop skips any drive whose MountPath is already seen (disks.go:263), so the registry row is DEDUPED AWAY ENTIRELY.

/disks therefore carries no row with that guest path at all: isTarget[guestPath] is a missing key, not a false value. The practical difference is decisive — the obvious fix (set BackupTarget on the union row) could not have worked, because that row is not emitted in the state where the alarm is needed.

On return the rows rejoin and the specific event fires — which is exactly the asymmetry observed. v0.184.1 fixed the keying (both keys are mapped); it did not fix the flag being absent from the row that carries the guest path.

Filed as R-116. Not fixed: a defect found in a validation run is a finding.

6. Mirror scenario + over-correction guard — PASS, with a caveat

A non-target drive (adat, 70 GB, registered but never assigned) detached at 18:50:32:

18:50:54 [WARN] [gate] drive ABSENT /mnt/felhom-drives/adat — stopped+blocked 0 app(s)
18:50:54 [INFO] Event pushed: storage_disconnected (error) — Meghajtó váratlanul leválasztva: Adat meghajto
  backup_target_absent count : 0     storage_disconnected count : 1

Over-correction guard — the real risk of R-113 — PASSES. With both drives present the gate produced 0 ABSENT lines, and the target stayed degraded:false with banner element 0 throughout the whole mirror sequence. R-113's stricter presence did not make a healthy drive read absent.

Caveat, stated rather than glossed: because the target drive also produced the generic event, the mirror passes trivially. It confirms no over-correction; it cannot confirm specific-vs-generic discrimination, because nothing produced the specific event on absence at all.

7. Findings

1. R-116 (new) — the absent alarm is generic while the recovery is specific; the pair cannot be matched by an operator. Root cause §5. This defect was unreachable before R-113: the gate never ran, so the branch was never taken. Fixing one bug exposed the next, which is what a validation run is for.
2. The stale bind survives a return, still naming a dead device node. After the drive came back as /dev/sdc, findmnt /mnt/felhom-drives/mentes still read /dev/sdb[/felhom-data] while the state reported degraded:false. E-2d noticed this and did not chase it; it is now confirmed. Whether apps writing through that path reach the live device was not established — worth its own look.
3. The capability map still has no E-2 / backup-target rows at all, so no row could be moved to PROVEN-LIVE. Creating them is a design act, not a validation act; filed rather than invented (third session in a row this has been noted).

8. Teardown

VM 9301 destroyed --purge, sc-images storage removed, scratch dir removed. pvesm status after == before (local-lvm 38.78 %, felhom-backup 931059224 KiB available). Guest 9201 and drill-r50 untouched.

Customer ruling: DELETE (R-93's lesson — no half-real fixtures). Attempted and correctly refused: delete sess-c REFUSED: host sess-c-8158e4 is ONLINE. The VM is destroyed so no further reports come; the host ages OK → WARN (30 m) → DOWN (>1 h) and is then deletable. Command, unchanged from E-2d:

POST /configs/sess-c/delete  ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-c expect_hosts=1

9. What did not happen

• backup_target_absent never fired, so its severity, its Hungarian customer copy and its hub routing are all still unexercised end-to-end. R-116 blocks them.
• The offer control was accepted via the endpoint it POSTs, not by a browser click — no browser automation exists on DooPlex. The rendered control, its data-path and its non-auto-submission were verified in the HTML.
• The stale-bind-after-return question (finding 2) was observed, not investigated.