felhom.eu/documentation/audits/R116-v0116-2026-07-30.md

# R116-v0116-2026-07-30 — the fix works at the payload layer, live; the events leg is blocked on an emailed code

**Run:** R-116 join task, CC on DooPlex, 2026-07-30. Agent **v0.116.0** built, published, vouched, and
**installed by a fresh box from the Day-0 manifest**.

| Claim | Verdict |
|---|---|
| **The fix is effective live** — the absent target row carries the flag **and** the key the gate uses | ✅ **PASS**, on real hardware. `isTarget["/mnt/felhom-drives/cel"] = true` (was `false` through v0.115.0) |
| **R-114 not regressed** — no absent-state row combines the flag with a mount path | ✅ **PASS** |
| **No over-correction** — the absent row does not register presence | ✅ **PASS** (`bound_under_parent: false`) |
| **Discrimination at the payload layer** — non-target gets the flag on no row | ✅ **PASS** |
| **C5 — `backup_target_absent` / `_restored` events on the wire, paired** | ⛔ **NOT REACHED.** Blocked on the controller claim code (emailed only). See §6 |
| **Discrimination as events** | ⛔ **NOT REACHED**, same blocker. **Not reported as a pass** |

## 1. Baselines as actually running

| | `main` | running | a FRESH box gets |
|---|---|---|---|
| agent | **0.116.0** (`21b0164`) | felhom-pve 0.115.0 · demo-hp 0.113.0 · **drill `sess-e-5d4427` 0.116.0** | **0.116.0** (vouched this run) |
| controller | 0.186.0 (`b331f18`) | felhom-pve **0.186.0** · demo-hp **0.185.1** · drill **0.185.1** | golden bakes **0.185.1** |
| hub | 0.81.0 | 0.81.0 | `min_agent` 0.113.0 · `min_controller` 0.156.0 |
| `felhom.eu` | `1aa1bd1` | — | — |

**The golden bakes controller 0.185.1** — which is why the fix had to land in the agent: a controller-side
fix would not reach a fresh box without a re-bake, whereas the agent channel serves the newest vouched
version immediately. Confirmed by this run: the drill box installed **0.116.0** unaided.

## 2. §5's three publish observables, as returned

```
(1) registry, independent GET of the published bytes:
    HTTP 200  bytes=14022347
    b47c5c4dab641ee57e2cf5893c2aebf1806a41c376ab06322342b02c86940a98  /tmp/rt-0.116.0
    /tmp/rt-0.116.0 --version → felhom-agent 0.116.0
(2) hub Day-0 manifest, read BACK after the POST (not the 303):
    agent_version: SELECTED=['0.116.0']      agent_sha256 = b47c5c4dab641ee5…
    golden_version: SELECTED=['0.185.1']     min_agent = 0.113.0
    wrapper_sha256 = 104db0a4401f65bb…  (re-checked against configs/felhom-pbs-apply — NO drift)
(3) the box under test reports it running:
    hub log  → Artifact manifest served for customer sess-e (agent=0.116.0 golden=0.185.1)
    host row → sess-e-5d4427 … 0.116.0 ONLINE
    on the box → felhom-agent --version → felhom-agent 0.116.0
```

`min_agent` deliberately **not** raised: controller v0.185.0 declares MinAgent 0.113.0, and raising it
would hold demo-hp (0.113.0) for no reason. The golden was not moved — the controller is unchanged.

## 3. The rig — real pipeline, no bypass

**Machine: `demo-hp`**, per `runbooks/target-selection.md` (Tier 0, the designated drill+build VM host).
**This differs from the previous R-116 run, which used the DooPlex fixture** — the document says DooPlex is
Tier 2 and its `drill.qcow2` is a bake fixture, not a drill target, so this run followed the document.

VM **9401** `r116-drill`: nested PVE, 8 GiB/4 vCPU cpu=host, OVMF, disks on a **dir storage
`r116-images` at `/mnt/nvme-1tb` root** — **not** `local-lvm` (over-subscribed thin pool hosting a live
guest). Installed from the **real v1.25.0 felhom ISO** already on demo-hp, through the **real day-0**:
self-register (appliance 13, pairing `D54-DG5`) → operator bind to `sess-e` → credentials delivered once →
guest 9201 provisioned from the vouched golden → `controller_started (0.185.1)` → reports flowing.

Drives, both enrolled through the **real endpoints** (`/disks/format` → `/disks/assign` →
`/disks/guest-attach` → `/backup/target`), no hand-set state:

| | device | host mount | guest bind | role |
|---|---|---|---|---|
| target | `/dev/sdb` (scsi1) | `/mnt/cel` | `/mnt/felhom-drives/cel` | primary backup target (`felhom-backup`, `is_mountpoint 1`) |
| non-target | `/dev/sdc` (scsi2) | `/mnt/adat` | `/mnt/felhom-drives/adat` | user-data only |

`backup tier armed target=felhom-backup … primary=true` after the required restart.

Device loss is a **real hot-detach** (`qm set 9401 --delete scsi1`); the volume survived as `unused0` and
was reattached, so the present-state control is re-runnable.

## 4. The payloads — PASS, and this is what v0.115.0 could not do

Host state at the absent capture, identical to every prior run:

```
ls /dev/sdb        → No such file or directory
findmnt /mnt/cel   → rc=1 (not mounted)
pvesm status       → unable to activate storage 'felhom-backup' - directory is expected to be a
                     mount point but is not mounted: '/mnt/cel'
                     felhom-backup   dir   inactive   0 0 0
```

### The target drive's row, all three states

| state | rows | rows for the target | `mount_path` | `guest_path` | `backup_target` | `bound_under_parent` | `state` | `role` |
|---|---|---|---|---|---|---|---|---|
| PRESENT | 4 | **1** | `/mnt/cel` | `/mnt/felhom-drives/cel` | **`true`** | `true` | attached | user-data |
| **ABSENT** | 4 | **1** | **`""`** | **`/mnt/felhom-drives/cel`** | **`true`** | `false` | disconnected | system |
| RETURNED | 4 | **1** | `/mnt/cel` | `/mnt/felhom-drives/cel` | **`true`** | `true` | attached | user-data |

**Exactly one row in every state** — the registry row for the target is deduped by guest path in the
absent state (`registry-row-for-target = 0` in all three captures). Before this fix the absent state
carried it **twice**.

### `driveTargetByPath` computed from the captured absent payload

```
driveTargetByPath = {'/mnt/felhom-drives/cel': True,
                     '/mnt/felhom-drives/adat': False, '/mnt/adat': False}
isTarget[/mnt/felhom-drives/cel] = True      ← was FALSE through v0.115.0
```

`notifyDriveAbsent` therefore takes the **specific** branch. RETURNED gives `True` as well, so the pair
matches.

### The three guards, from the same payload

- **R-114 preserved.** No row has `backup_target: true` **and** `mount_path != ""`. The absent target row
  reports `mount_path: ""`, so `backup_target_offer.go:79` does **not** match and R-114's `TargetAbsent`
  branch stays reachable. This is why option (a) was rejected — see `felhom-agent/REPORT.md`.
- **No over-correction.** `bound_under_parent: false` on the absent row, so
  `present[gp] = … || d.BoundUnderParent` stays false and the Stop branch is still reachable.
- **Discrimination, payload layer.** The non-target `adat` carries `backup_target` **absent ⇒ false** on
  both its keys, in both states. So the two are now genuinely distinguishable — which is the thing both
  prior runs could not show.

### Raw payloads

Absent (all 4 rows, verbatim):

```json
{"ok":true,"data":{"disks":[{"name":"felhom-backup","type":"local-dir","state":"disconnected","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":0,"used_bytes":0,"used_fraction":0,"durable_id":"path:/mnt/cel","guest_attached":false,"backup_target":true,"guest_path":"/mnt/felhom-drives/cel","bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}},{"name":"local","type":"local","state":"attached","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":14484905984,"used_bytes":5096075264,"used_fraction":0.351819699045967,"durable_id":"path:/var/lib/vz","guest_attached":false,"bound_under_parent":false,"smart":{"health":"UNKNOWN","model_name":"QEMU QEMU HARDDISK","temperature_c":0,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}},{"name":"local-lvm","type":"lvmthin","state":"attached","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":12675186688,"used_bytes":4224639723,"used_fraction":0.33329999999129,"durable_id":"pve/data","guest_attached":false,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}},{"name":"d248508a-0c4f-466f-9c43-0dae988efc62","type":"usb","state":"attached","backing_device":"/dev/sdc","mount_path":"/mnt/adat","class":"","role":"user-data","data_bearing":false,"total_bytes":8350298112,"used_bytes":2125824,"used_fraction":0.00025458061155266215,"durable_id":"uuid:d248508a-0c4f-466f-9c43-0dae988efc62","guest_attached":false,"guest_path":"/mnt/felhom-drives/adat","bound_under_parent":true,"smart":{"health":"UNKNOWN","model_name":"QEMU QEMU HARDDISK","temperature_c":0,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}}],"guest_boot_id":"1785394603-28275","vmid":9201}}
```

Present and returned payloads: `d-PRESENT.json` / `d-RETURNED.json`, on the box at `/root/` and in the
session scratchpad. The target row differs from the absent one only in the fields tabulated above.

## 5. Where R-117 interfered — and it did

**Reproduced again, fourth consecutive run, and on real hardware this time:**

```
after reattach:  host findmnt /mnt/cel                     → /dev/sdd
                 guest findmnt /mnt/felhom-drives/cel       → /dev/sdb[/felhom-data] rw,relatime,shutdown
                 guest read through the bind                → GUEST-READ-EIO
                 /disks says                                → state attached, bound_under_parent TRUE
```

**How it interferes with reading this run:** the reattach leg's `isTarget = true` and
`bound_under_parent = true` are *correct as answers to the questions this fix asks*, but they are read off
a drive whose guest namespace is **dead**. So "the drive came back healthy" cannot be concluded from the
reattach capture — only "the flag and the key rejoined on one row", which is what was under test. When the
events leg runs, the `Return` branch will fire against a dead bind, so a `backup_target_restored` there
proves pairing, **not** recovery. Recorded, not fixed (R-117 has its own row).

## 6. Why C5 was not reached — the one genuine human step, again

`planDriveGates` iterates `s.settings.GetStoragePaths()`. The drill controller has **none**:

```
controller log → [WARN] Storage paths: no storage paths registered
```

Registering one requires the controller's storage API, and **every** route is behind the claim gate:

```
GET /api/storage/backup-target  (Host: felhom.sess-e.test)  → 401 {"ok":false,"error":"dashboard not yet claimed"}
GET /api/storage/paths                                       → 401
GET /api/storage/list                                        → 401
```

The claim code is **bcrypt-hashed in the hub and only ever emailed** (`claim/engine.go`), and the
self-bind link is likewise mint-and-email — `handleSelfBindLinkSend` (`selfbind_mint.go:139-161`) only
ever renders a flash, never the token. **There is no operator-side route to either secret**, which is the
same wall E2D hit and named "the one genuine human step".

A fresh claim code (**generation 2**) was emailed to the registered address at 09:07:20 by
`POST /configs/sess-e/claim-resend`, so the code is waiting; the run stops here rather than reporting a
third trivial pass.

**Explicitly NOT done, and not claimed:** no `backup_target_absent` or `backup_target_restored` event was
observed on the wire this run, and the non-target mirror was not exercised as events. The payload evidence
in §4 determines what the gate *will* decide, but it is not the event on the wire, and after two runs that
conflated the two, that distinction is kept.

## 7. Teardown — layers 1–3, per the §13 paragraph this task added

**DELIBERATELY RETAINED, pending the claim code**, so the events leg can finish without rebuilding:

| layer | item | disposition |
|---|---|---|
| 1 — machine | VM **9401** `r116-drill` on demo-hp (+ `unused0`) | **retained**, running |
| 2 — host | `r116-images` dir storage at `/mnt/nvme-1tb` | **retained** |
| 3 — hub | customer **`sess-e`**, host **`sess-e-5d4427`**, appliance **13** | **retained** |

`pvesm status` on demo-hp **before** the run, for the layer-2 comparison at teardown:

```
felhom-backup  dir     active  983379700  4566008  928787080  0.46%
felhom-pbs     pbs     active          0        0          0  0.00%
local          dir     active   40516856 14961808   23464656 36.93%
local-lvm  lvmthin     active   56545280 21956532   34588747 38.83%    ← the fence figure, unchanged
```

Teardown commands, recorded now so they cannot be forgotten:

```bash
ssh demo-hp 'qm stop 9401; qm destroy 9401 --purge; pvesm remove r116-images; pvesm status'
# hub, once the host ages ONLINE→DOWN (customer_delete.go:220-228 refuses only on ONLINE):
POST /configs/sess-e/delete  ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-e expect_hosts=1
```

**Already clear, and not by this session:** `sess-c` and `sess-d` — both `/customers/<id>` → **HTTP 404**,
absent from `/configs` and `/hosts`. The operator removed them using the commands the previous report
recorded. Verified directly rather than inferred.

**Fences held:** guest 9201 on both demo boxes untouched · `drill-r50` (VM 300) untouched, `stopped` ·
neither demo box re-targeted · nothing on `local-lvm` · Peti untouched · v0.115.0 not reverted ·
the hub DB copy taken for the break-glass credential was `shred -u`'d immediately.

## 8. Findings — filed, none fixed

- **R-117** — reproduced on real hardware with the read/write probe (`EIO` both directions) while `/disks`
  reports `attached` + `bound_under_parent: true`. Already filed; this run adds the metal-adjacent
  reproduction and the note in §5 about how it colours the reattach leg.
- **R-118** — the union row's root-filesystem capacity. Its **symptom** is now absent in the target's
  absent state because that row is suppressed; **R-118 is not fixed** and remains open for every other
  registry-only drive (the non-target `adat` row still carries statfs values from its own live mount, so
  the class is unchanged).
- **R-119 (new) — the claim gate makes the drive-gate legs unreachable to CC, by design, every time.**
  Three sessions have now stopped at the same wall: the controller's storage API is claim-gated, the claim
  code is emailed-only, and `planDriveGates` cannot act until a storage path is registered. This is not a
  defect in the gate — it is correct customer-ownership — but it means **every** future validation of a
  drive-gate behaviour costs an operator email round-trip. Worth a ruling: either a documented
  operator-side test affordance (an operator-scoped claim, explicitly audit-logged), or accept the
  round-trip and put it in the runbook as a **prerequisite step** rather than a mid-run surprise. Filed,
  not designed.