docs(R-116): isolate the mechanism from the real /disks payload; file R-117 + R-118

The absent-state /disks payload was captured on a genuine device loss, after a
present-drive control run proved the query works (Part 5's three attempts failed
on token extraction, and its control returned 0 rows).

The answer is theory #1 -- "the registry-union row writes false" -- which was
raised, declared wrong and retracted. The retraction was the error.

Absent state returns 4 rows, not 3. The drive appears twice and the two facts the
controller needs sit on different rows: the Observe row has backup_target:true but
mount_path:"" and guest_path:"", so it contributes no key to driveTargetByPath;
the registry-union row owns /mnt/felhom-drives/<name> and omits BackupTarget from
its struct literal (disks.go:301-306) => false. The union row is not deduped
because seen is keyed on MountPath (:290-295), the one field the absent state
empties, and its own MountPath comes from the systemd .mount unit FILE
(registry_known.go:40-75), which never reads the mount table.

Theory #2 (the basis of the shipped v0.115.0) is false on both halves; #3 is false
too. v0.115.0 is provably inert: StablePathForRaw("") returns "".

Also files the read path verbatim -- the token plaintext lives only in
bootstrap.json on the Proxmox host; the agent's store keeps hashes only.

New: R-117 (READY M, outranks R-116) -- a returned drive's guest bind is a DEAD
mount (EIO both ways) while /disks reports attached + bound_under_parent:true, so
the gate restarts the customer's apps onto it and reports healthy with no alarm.
R-118 (READY XS) -- an absent drive's union row advertises the root filesystem's
capacity as its own.

Docs only. No code written, nothing built or published; v0.115.0 untouched.
Both demo boxes read-only; drill fixture restored to virgin.
This commit is contained in:
admin committed 2026-07-30 08:06:21 +02:00
1 parent c3ce4c7b20
commit d56e395a2a
3 files changed
+604 -1

No files matched your search

+113
View File
@@ -0,0 +1,113 @@
# REPORT-r116-diag — the `/disks` payload captured, R-116's mechanism isolated (2026-07-30)
Read-only diagnosis run by CC on DooPlex. **No code written, nothing built, nothing published.**
Full evidence: `documentation/audits/DIAG-r116-disks-payload-2026-07-30.md`.
A `REPORT-*.md` sibling, not the shared `REPORT.md` (`CLAUDE.md` parallel-session rule).
## Outcome
**Both goals met.** The `/disks` read path is solved and written down verbatim, proven by a
present-drive control run *first*; and the absent-state payload was captured, which isolates the
mechanism.
**R-116 is theory #1 — "the registry-union row writes `false`" — the theory that was raised, declared
wrong, and retracted. The retraction was the error.**
In the absent state `/disks` returns **4 rows, not 3**. The drive appears twice and the two facts the
controller needs sit on different rows:
| row | source | `mount_path` | `guest_path` | `backup_target` |
|---|---|---|---|---|
| `felhom-backup` | Observe (`disks.go:196-284`) | `""` | `""` | **`true`** |
| `694034cc-…` (the UUID) | registry union (`disks.go:297-339`) | `/mnt/cel` | `/mnt/felhom-drives/cel` | **field ABSENT ⇒ `false`** |
So the row holding the flag contributes **no key** to `driveTargetByPath`, and the row that owns the key
says `false` → `isTarget[a.Path]` is `false` → generic `storage_disconnected`. On return the rows
re-merge into one carrying both facts → specific `backup_target_restored`. Applying
`intermediary.go:602-618` to the captured payloads gives PRESENT `True` / ABSENT `False` /
RETURNED `True` — **the live asymmetry reproduced from payload alone.**
The union row's `MountPath` survives the device because the union source is the systemd **`.mount` unit
file** (`registry_known.go:40-75` via `main.go:605`→`:764`), which never reads the mount table. The
dedup at `:298` therefore does not fire, because `seen` is keyed on the one field the absent state
empties (`:290-295`).
**Theory #2 (the basis of the shipped v0.115.0) is false on both halves**; **#3 is false too**
(`isTarget["/mnt/cel"]` is `false` as well). **v0.115.0 is provably inert** — its fallback calls
`StablePathForRaw("")`, which returns `""` (`intermediary.go:69-75`), so it assigns nothing.
## The read path (this cost two prior sessions — it should never cost again)
The token plaintext exists in exactly one place: `bootstrap.json` **on the Proxmox host**. The agent's
own store keeps SHA-256 hashes only (`tokenstore.go:26-32`), which is what defeated the earlier attempts.
```bash
ssh felhom-pve
B=/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json
TOK=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['token'])")
EP=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['endpoint'])")
curl -sS -k -H "Authorization: Bearer $TOK" "https://$EP/disks" | python3 -m json.tool
```
Control run, live felhom-pve, drive present: **HTTP 200, 2483 bytes, 4 plausible rows** — so Part 5's
0-rows-on-a-present-drive failure mode is excluded.
## Where the absent state was staged
No new box. The existing DooPlex **nested-PVE drill fixture** (`drill/drill.qcow2`, snapshot `virgin`) —
my own host, zero production exposure, and it can hot-unplug a disk for a genuine device loss. Run with
the **byte-identical live agent binary** (`sha256 f48544ad…`, `--version` 0.115.0) and every
state-producing step through the real endpoints (`format` → `assign` → `guest-attach` →
`backup/target`). Its present-state row matched felhom-pve's control run field-for-field before it was
trusted. Non-production aspects (root/direct privileged mode, stubbed hub, a hand-written bearer-token
record, no controller) are enumerated in the audit §4.
## Two new findings, filed not chased
- **R-117 (READY M) — outranks R-116.** After a detach/reattach the guest's bind is a **dead mount**:
host is healthy on the new device node, guest still names the old one, and `ls`/write through it
return **`EIO`** — while `/disks` reports `attached` + `bound_under_parent:true` + `backup_target:true`.
`planDriveGates` therefore takes the `Return` branch and **restarts the customer's apps onto a dead
namespace, reporting healthy, with no alarm on any channel.** R-113's conjunction cannot catch it:
one half is satisfied by the stale entry, the other by the new device, and neither compares them.
This is the "stale bind" seen and dismissed as cosmetic in three consecutive runs.
- **R-118 (READY XS).** An absent drive's union row reports the **root filesystem's** capacity as its own
(46 GiB / 9.2 % for a 4 GB drive) — `statfsCapacity` at `disks.go:335-338` statfs's a bare directory on
root. `observe.go:176-183` guards the Observe path against exactly this; the union path does not.
`durable_id` is still correct, so re-attach identity is safe — it is a false capacity, not a DR mis-id.
## Register
`documentation/backlog/OPEN-ITEMS.md` — R-116 updated with the mechanism and the fix constraints;
R-117 and R-118 added. The single register edit this session makes.
## Record correction
The brief's baseline `controller 0.185.1` is the version the **golden bakes**; both demo boxes actually
run **0.186.0** (R-114 + R-112, 2026-07-29). Both are true of different things, and the distinction
matters because R-114's `TargetAbsent` branch exists only in 0.186.0. Confirmed: agent **0.115.0**,
controller **0.186.0** live, hub **0.81.0** (manifest pin and live pod image agree), host-install
**1.22.0**, `felhom.eu` HEAD `c3ce4c7`.
## Teardown and fences
Drill guest destroyed, scratch storage removed, mount unit deleted, secrets `shred -u`'d, VM powered
off, **`drill.qcow2` restored to `virgin`** (the golden-bake fixture is exactly as found), scratch qcow2
and console dumps deleted. DooPlex `/mnt/5_hdd` at 24 %, unchanged.
Both demo boxes **read-only throughout** and re-verified after teardown: demo-hp `local-lvm` **38.83 %**
(identical before/after and to Part 5), `drill-r50` still stopped, felhom-pve `felhom-backup` still
active on `/dev/sdb`, both guests running, **v0.115.0 untouched**.
`sess-d-0452c4` now reads **STALE**, and the delete gate refuses only on ONLINE
(`hub/internal/web/customer_delete.go:220-228`) — so it **is** now deletable; the command is recorded in
the audit rather than executed (customer delete runs external teardown plus a DB purge). **`sess-c` is
also still present and was not recorded by the Session-C audit** — same terms.
## Not done, deliberately
No fresh controller gate-log / hub-event correlation: that observable was already captured live and
identically twice, and only the payload was missing. Staging a controller meant a hub customer, a
pairing, a golden fetch and a claim — the work that consumed the three prior sessions. The audit §8
correlates the payload to those existing measurements and labels that step as inference.
@@ -0,0 +1,488 @@
# DIAG-r116-disks-payload-2026-07-30 — the `/disks` payload, and the mechanism isolated from it
**Run:** read-only diagnosis by CC on DooPlex, 2026-07-30. **No code written, nothing shipped.**
**Result: the payload was captured in both states and the mechanism is isolated.** It is theory **#1**
— *the registry-union row writes `false`* — the theory that was raised, declared wrong, and retracted.
**The retraction was the error.**
---
## 1. Baselines
All confirmed this run, not carried from the brief.
| Artifact | Version | How confirmed |
|---|---|---|
| agent | **0.115.0** | `felhom-agent --version` on felhom-pve; `felhom-agent/CHANGELOG.md:1` = `## v0.115.0`; repo HEAD `a58239f6ded367fe27b4eb57cf2fcb0c54414b80` |
| controller | **0.186.0** ⚠ | `pct exec 9201 -- docker ps` on felhom-pve → `felhom-controller:0.186.0`; `CHANGELOG.md:3` = `### v0.186.0`; HEAD `b331f18424b7cf8d0883e7709f74595808264080` |
| hub | **0.81.0** | `manifests/hub.yaml:128` pin **and** the live pod image (`kubectl -n felhom-system get pod hub-… -o jsonpath=…`) both `felhom-hub:0.81.0`; `hub/CHANGELOG.md:1` = `## v0.81.0` |
| host-install | **1.22.0** | `scripts/felhom-host-install.sh:187` `SCRIPT_VERSION="1.22.0"` |
| `felhom.eu` HEAD | `c3ce4c7b20ce4c0d3673590d710d13e5d28b68bb` | `git rev-parse HEAD` |
⚠ **The brief's `controller 0.185.1` is not what is deployed.** 0.185.1 is what the **golden bakes**
(so it is what a fresh drill box installs, which is why Part 5 correctly recorded it); **0.186.0** is
what both demo boxes actually run, shipped 2026-07-29 as R-114 + R-112. Both numbers are true of
different things; the distinction matters because R-114's `TargetAbsent` branch exists only in 0.186.0.
Hub host table, read this run: `demo-felhom-8363b5` agent 0.115.0 ONLINE · `demo-hp-bb76ea` agent
0.113.0 ONLINE · `drill-r50-0a4f9a` DOWN · `sess-c-8158e4` DOWN · `sess-d-0452c4` **STALE**.
---
## 2. §3 reconfirmation — every cited fact, at `file:line`
| Cited fact | Line now | Holds? |
|---|---|---|
| `observe.go` — `mountPath` comes from the mount table, assigned only on exact-mount success | `internal/storage/observe.go:184-190` | **HOLDS**, verbatim as quoted |
| `role.go` — `if backingDevice == "" { return RoleSystem }` for USB/LocalDir | `internal/storage/role.go:180-181` | **HOLDS**, verbatim |
| `primaryTargetID := s.primaryTier().TargetID`, in scope for both loops | `internal/localapi/disks.go:194` | **HOLDS** |
| `BackupTarget: t.Name == primaryTargetID` | `disks.go:211` | **HOLDS** |
| `GuestPath` set only inside `if di.Role == RoleUserData` | `disks.go:215-217` | **HOLDS** |
| first loop `out = append(out, di)` | `disks.go:284` (brief said `:249`) | **HOLDS — line moved**, v0.115.0 inserted `:226-260` |
| `seen[d.MountPath] = true` built by iterating `out` | `disks.go:290-295` (brief said `:258`) | **HOLDS — line moved.** Note the guard: `seen` is written **only when `d.MountPath != ""`** (`:292`) |
| union dedup `if d.MountPath == "" \|\| seen[d.MountPath] { continue }` | `disks.go:298` (brief said `:263`) | **HOLDS — line moved** |
| union row hardcodes `Role: RoleUserData` | `disks.go:304` | **HOLDS** |
| union row's `GuestPath` set from `StablePathForRaw(d.MountPath)` | `disks.go:309-310` | **HOLDS** |
| **`BackupTarget` is absent from the union struct literal** | `disks.go:301-306` | **HOLDS** — the literal sets `Name/Type/State/MountPath/DurableID/Role/GuestAttached` and nothing else |
| v0.115.0's fallback `di.GuestPath = StablePathForRaw(t.MountPath)` | `disks.go:258-260` | **HOLDS** |
| `StablePathForRaw("") == ""` (so the fallback sets nothing when `MountPath` is empty) | `internal/localapi/intermediary.go:69-75` → `DriveNameFromRaw:79-88` (no `/mnt/` prefix → `""`) | **HOLDS** |
| PVE reports the detached target `dir inactive 0 0 0` | reproduced live, §5 | **HOLDS** |
**One fact the brief did not state, and it is load-bearing.** `/disks`' union source is the
**systemd-unit-file** registry, not PVE and not the mount table:
`storage.NewRegistryKnownTargets(driveUnitDir, …)` (`cmd/felhom-agent/main.go:605`) is the value passed
as `DriveTargets` (`main.go:764` → `buildLocalAPIServer` → `disks.go:296` `s.driveTargets.Known()`).
`Known()` (`internal/storage/registry_known.go:40-75`) `os.ReadDir`s `/etc/systemd/system`, parses each
Felhom `.mount` unit, and returns `MountPath: spec.Where` (`:70`) — **it never consults the mount table
or the device.** So the union row's `MountPath` survives the device's disappearance by construction.
Verified on the live box (felhom-pve), unit present and device-bound:
```
/etc/systemd/system/mnt-hdd_1.mount
What=/dev/disk/by-uuid/47a3361a-91e0-4831-a69d-27f540ed3f48
Where=/mnt/hdd_1
/var/lib/felhom-agent/drive-intents.json → intent recorded (not `new`, so Known() includes it)
```
---
## 3. The working `/disks` invocation — verbatim
Two sessions were lost to this. **The token's plaintext exists in exactly one place: the guest's
bootstrap mount, which the agent owns host-side.** The agent's own token store holds only SHA-256
hashes (`internal/localapi/tokenstore.go:26-32`, `:136`), so it is *not* a source — that is what the
earlier extraction attempts were fighting.
Authoritative source, on the **Proxmox host**, as root:
```
/var/lib/felhom-agent/guests/<VMID>/bootstrap/bootstrap.json # 0600, owner 100000:100000
```
(written by `internal/provision/backhalf.go:124-130`; the in-guest view is
`/etc/felhom-bootstrap/bootstrap.json`, mp9, read-only — the host path avoids the guest hop entirely).
**The invocation. Run it ON the Proxmox host so the token never crosses a hop or reaches a log:**
```bash
ssh felhom-pve
export LC_ALL=C
B=/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json
TOK=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['token'])")
EP=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['endpoint'])") # 169.254.253.1:8443
curl -sS -k -H "Authorization: Bearer $TOK" "https://$EP/disks" | python3 -m json.tool
```
`-k` is correct and not a shortcut: the leaf is self-signed and the controller pins it by SHA-256
(`local_api.fingerprint` in the same file, `2d325e89…c191` on felhom-pve) rather than by a CA. The
endpoint is the island bridge `vmbr9` (`169.254.253.1/30`), reachable from the host itself.
### 3.1 The control run — it proves the query works
**Present-drive control, live felhom-pve, before anything else:**
```
endpoint=169.254.253.1:8443
HTTP 200 bytes=2483
rows: 4 (local-lvm, felhom-pbs, felhom-backup, local)
```
The backup-target row in that control run:
```json
{ "name": "felhom-backup", "type": "local-dir", "state": "attached",
"backing_device": "/dev/sdb", "mount_path": "/mnt/hdd_1", "role": "user-data",
"backup_target": true, "guest_path": "/mnt/felhom-drives/hdd_1",
"bound_under_parent": true, "durable_id": "uuid:47a3361a-91e0-4831-a69d-27f540ed3f48" }
```
Non-empty, plausible, four rows. **Part 5's failure mode (0 rows on a present drive) is excluded.**
---
## 4. Where the absent state was staged, and why there
The demo boxes' targets are fenced, and `drill-r50` is the drift fixture. **No new box was built.** The
absent state was staged on the **existing nested-Proxmox drill VM on DooPlex**
(`/mnt/5_hdd/felhom.eu/drill/drill.qcow2`, PVE 9.2, snapshot `virgin`, the golden-bake fixture) — my own
host, zero production exposure, and it can hot-unplug a disk for a **genuine device loss** rather than a
simulated one.
**Fidelity — what was real:**
- the **byte-identical live agent binary**: `scp` from felhom-pve `/usr/local/bin/felhom-agent`,
`sha256 f48544ad5c4242a719c080ff83d7a6bc759e14272987416eb26c9683386803b7`, `--version` 0.115.0.
- the real `configs/felhom-backup-target-apply` (E-2a) and `felhom-mkfs-guarded` wrappers.
- a real LXC guest (9201), in the `felhom` pool, with the mp8 parent bind created **exactly** as
`provision.BackHalf` does it (`backhalf.go:200-206`: `pct set 9201 -mp8 /mnt/felhom-drives,mp=/mnt/felhom-drives`).
- **every state-producing step through the real endpoints**, not by hand:
`POST /disks/format` → `POST /disks/assign` → `POST /disks/guest-attach` → `POST /backup/target`.
That produced the `.mount` unit, `drive-intents.json: enrolled`, the guest bind, the
`dir: felhom-backup / path /mnt/cel / is_mountpoint 1` storage and the `local_backup_target` rewrite.
- device loss by QMP `device_del` on the virtio-blk frontend — the device node vanishes.
**What was NOT production-shaped, stated plainly:** (a) the agent ran as root with
`privileged.mode: "direct"` and a `root@pam` PVE token instead of non-root + sudoers + `felhom-agent@pve`
— this changes *who* runs the privileged ops, not what `/disks` computes; (b) `hub.url` pointed at a
dead local port, so nothing was reported to the hub; (c) **the local-API bearer token record was
hand-written** into `local-tokens.log` (one `{"v":9201,"h":"<sha256>"}` line) because minting is only
reachable through `--selftest=provision`, which needs a golden and a real customer. That is the *key to
the door*, not the state under measurement — every drive/storage/target fact was produced by the real
endpoints. (d) No controller ran, so §7 correlates against the two prior live runs rather than a fresh one.
**The rig validated itself against production before being trusted:** its present-state target row is
field-for-field identical in shape to felhom-pve's control run (`role: user-data`,
`backup_target: true`, `guest_path` set, `bound_under_parent: true`, `durable_id: uuid:…`) — only the
device node and mountpoint names differ.
---
## 5. Host state at each capture
| | PRESENT (05:53:49Z) | ABSENT (05:54:32Z) | RETURNED (05:55:45Z) |
|---|---|---|---|
| device node | `/dev/vdb` | **gone** (`ls /dev/vdb` → No such file) | `/dev/vdc` |
| `findmnt /mnt/cel` | `/dev/vdb ext4` | **not mounted** (rc=1) | `/dev/vdc ext4` |
| guest `findmnt /mnt/felhom-drives/cel` | `/dev/vdb[/felhom-data]` | `/dev/vdb[/felhom-data] …,shutdown` | `/dev/vdb[/felhom-data] …,shutdown` |
| `pvesm status` | `felhom-backup dir active` | `felhom-backup dir inactive 0 0 0` | `felhom-backup dir active` |
The detached state reproduced PVE's exact Part-5 string:
```
unable to activate storage 'felhom-backup' - directory is expected to be a mount point
but is not mounted: '/mnt/cel'
```
and the agent's exact Session-C / E-2d lines:
```
07:54:08 WARN storage: watchdog detected target state change from=attached to=disconnected
07:54:09 INFO storage: enrolled drive absent by UUID — not re-asserting (will retry when it enumerates)
07:54:09 WARN reconcile: enrolled drive not present (durable-id absent) — skipping
```
The R-113 asymmetry is present as measured before: **raw `/mnt/cel` NOT mounted while the guest's bind
still names `/dev/vdb[/felhom-data]`.**
---
## 6. The raw payloads
### 6.1 PRESENT — `GET /disks`, HTTP 200, 1824 bytes, **3 rows**
```json
{"ok":true,"data":{"disks":[
{"name":"felhom-backup","type":"local-dir","state":"attached","backing_device":"/dev/vdb","mount_path":"/mnt/cel","class":"slow","role":"user-data","data_bearing":true,"data_reason":"device is mounted","total_bytes":4143677440,"used_bytes":1081344,"used_fraction":0.000260962397690878,"durable_id":"uuid:694034cc-eee6-4b17-901f-708f50df9b09","wipe_durable_id":"byid:virtio-R116CEL","guest_attached":false,"backup_target":true,"guest_path":"/mnt/felhom-drives/cel","bound_under_parent":true,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}},
{"name":"local-lvm","type":"lvmthin","state":"attached","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":81533075456,"used_bytes":807177447,"used_fraction":0.00989999999982339,"durable_id":"pve/data","guest_attached":false,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}},
{"name":"local","type":"local","state":"attached","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":49675956224,"used_bytes":4584443904,"used_fraction":0.092286978499774,"durable_id":"path:/var/lib/vz","guest_attached":false,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}}
],"guest_boot_id":"1785390256-46753","vmid":9201}}
```
### 6.2 ABSENT — `GET /disks`, HTTP 200, 2040 bytes, **4 rows** (one MORE than present)
```json
{"ok":true,"data":{"disks":[
{"name":"felhom-backup","type":"local-dir","state":"disconnected","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":0,"used_bytes":0,"used_fraction":0,"durable_id":"path:/mnt/cel","guest_attached":false,"backup_target":true,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}},
{"name":"local-lvm","type":"lvmthin","state":"attached","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":81533075456,"used_bytes":807177447,"used_fraction":0.00989999999982339,"durable_id":"pve/data","guest_attached":false,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}},
{"name":"local","type":"local","state":"attached","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":49675956224,"used_bytes":4584579072,"used_fraction":0.0922896994942001,"durable_id":"path:/var/lib/vz","guest_attached":false,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}},
{"name":"694034cc-eee6-4b17-901f-708f50df9b09","type":"usb","state":"attached","backing_device":"","mount_path":"/mnt/cel","class":"","role":"user-data","data_bearing":false,"total_bytes":49675956224,"used_bytes":4584579072,"used_fraction":0.09228969949420011,"durable_id":"uuid:694034cc-eee6-4b17-901f-708f50df9b09","guest_attached":false,"guest_path":"/mnt/felhom-drives/cel","bound_under_parent":false}
],"guest_boot_id":"1785390256-46753","vmid":9201}}
```
### 6.3 RETURNED — `GET /disks`, HTTP 200, 1825 bytes, **3 rows** (back to one row for the drive)
```json
{"ok":true,"data":{"disks":[
{"name":"felhom-backup","type":"local-dir","state":"attached","backing_device":"/dev/vdc","mount_path":"/mnt/cel","class":"slow","role":"user-data","data_bearing":true,"data_reason":"device is mounted","total_bytes":4143677440,"used_bytes":1081344,"used_fraction":0.000260962397690878,"durable_id":"uuid:694034cc-eee6-4b17-901f-708f50df9b09","wipe_durable_id":"byid:virtio-R116CEL","guest_attached":false,"backup_target":true,"guest_path":"/mnt/felhom-drives/cel","bound_under_parent":true,"smart":{"health":"UNKNOWN", …}},
{"name":"local-lvm", … as above … },
{"name":"local", … as above … }
],"guest_boot_id":"1785390256-46753","vmid":9201}}
```
*(the `local-lvm` / `local` rows in 6.3 are byte-identical to 6.1 apart from `used_bytes`; elided only
there, and nowhere else in this document.)*
---
## 7. The per-row table for the backup-target drive
**PAYLOAD.** Every cell read from the JSON above.
### PRESENT — 1 row mentions the drive
| Name | Type | State | Role | MountPath | GuestPath | BackingDevice | BackupTarget | BoundUnderParent |
|---|---|---|---|---|---|---|---|---|
| `felhom-backup` | `local-dir` | `attached` | `user-data` | `/mnt/cel` | `/mnt/felhom-drives/cel` | `/dev/vdb` | **`true`** | `true` |
### ABSENT — **2 rows** mention the drive, and the two facts are on DIFFERENT rows
| Name | Type | State | Role | MountPath | GuestPath | BackingDevice | BackupTarget | BoundUnderParent | Source |
|---|---|---|---|---|---|---|---|---|---|
| `felhom-backup` | `local-dir` | `disconnected` | **`system`** | **`""`** | **`""`** (field absent) | `""` | **`true`** | `false` | Observe loop, `disks.go:196-284` |
| `694034cc-…9b09` | `usb` | **`attached`** (hardcoded `:302`) | `user-data` (hardcoded `:304`) | **`/mnt/cel`** | **`/mnt/felhom-drives/cel`** | `""` | **field ABSENT ⇒ `false`** | `false` | union loop, `disks.go:297-339` |
`durable_id` also degrades on the Observe row: `uuid:694034cc…` → **`path:/mnt/cel`**.
### RETURNED — 1 row again
| Name | Type | State | Role | MountPath | GuestPath | BackingDevice | BackupTarget | BoundUnderParent |
|---|---|---|---|---|---|---|---|---|
| `felhom-backup` | `local-dir` | `attached` | `user-data` | `/mnt/cel` | `/mnt/felhom-drives/cel` | **`/dev/vdc`** | **`true`** | `true` |
---
## 8. The mechanism
**Which row the lookup lands on, which key it uses, what that row says.**
The controller's key is `a.Path` — the **registered `StoragePath`**, which for an external drive is the
GUEST path `/mnt/felhom-drives/cel` (`intermediary.go:297`, `:318`). `driveTargetByPath`
(`intermediary.go:602-618`) builds the map with **plain assignment**, one entry per non-empty
`GuestPath` (`:612`) and per non-empty `MountPath` (`:615`).
**PAYLOAD — the map, computed by applying `intermediary.go:602-618` verbatim to the captured JSON:**
```
PRESENT rows=3 isTarget["/mnt/felhom-drives/cel"] = True → SPECIFIC
map: {'/mnt/felhom-drives/cel': True, '/mnt/cel': True}
ABSENT rows=4 isTarget["/mnt/felhom-drives/cel"] = False → GENERIC
map: {'/mnt/felhom-drives/cel': False, '/mnt/cel': False}
RETURNED rows=3 isTarget["/mnt/felhom-drives/cel"] = True → SPECIFIC
map: {'/mnt/felhom-drives/cel': True, '/mnt/cel': True}
```
That is the live asymmetry — generic on detach, specific on return — **reproduced from the payload
alone**, matching Session C and Part 5 exactly.
**Why it fails, step by step. Steps 1–4 are payload; step 5 is the map above.**
1. The device goes. `exactMountDevice` fails, so `backingDevice` and `mountPath` are both `""`
(`observe.go:184-190`). **Payload: Observe row has `backing_device:""`, `mount_path:""`.**
2. `RoleForStorage` therefore returns `RoleSystem` (`role.go:180-181`), so the `GuestPath` block at
`disks.go:215-217` is skipped. **Payload: `role:"system"`, no `guest_path`.**
3. v0.115.0's fallback (`disks.go:258-260`) fires — its three gates all pass — but it calls
`StablePathForRaw(t.MountPath)` with `MountPath == ""`, which returns `""`
(`intermediary.go:69-75`). **It assigns an empty string. Payload confirms: no `guest_path` on that
row.** This is exactly why v0.115.0 is inert.
4. `seen` is built only from rows whose `MountPath != ""` (`disks.go:290-295`). The Observe row's is
empty, so **`/mnt/cel` is not in `seen`**, so the union row is **not** deduped at `:298`. It is
emitted with `MountPath` from the `.mount` unit file and `GuestPath` from `:309-310` — and
**`BackupTarget` is not in its struct literal (`:301-306`), so it serialises as `false`.**
**Payload: row 4 exists, carries both paths, and `backup_target` is absent.**
5. So the row holding `backup_target:true` contributes **no key at all** (both its keys are empty
strings and are skipped at `:611` and `:614`), while the row that owns the key
`/mnt/felhom-drives/cel` says `false`. `isTarget[a.Path]` → `false` → `notifyDriveAbsent` takes the
else branch (`intermediary.go:645`) → **generic `storage_disconnected`**.
6. On return the two rows re-merge into one (the Observe row regains `MountPath`, so the union row is
deduped again) — that single row carries `backup_target:true` **and** the guest path, so
`notifyDriveReturned` takes the specific branch (`:656`) → **`backup_target_restored`**. Hence the
unmatchable pair.
**INFERENCE (small, and labelled):** step 5's map-construction and step 6's branch selection were
computed by applying the controller's published source to the captured payload, not observed from a
running controller this session. The observable they predict — generic out, specific back — is exactly
what Session C and Part 5 *did* measure live on real boxes, twice. So the prediction is corroborated,
but the controller was not itself re-run here.
**There is no map-overwrite race.** An earlier worry was that two rows might both write the same key
with the last one winning. They do not: the Observe row contributes nothing, so the union row is the
sole writer of `/mnt/felhom-drives/cel`. Row order is irrelevant.
---
## 9. Which theory this is
| Theory | Verdict from payload |
|---|---|
| **#1 — "the union row writes `false`"** (raised, then declared wrong and retracted) | ✅ **CORRECT.** The union row is emitted (`rows` 3 → 4), owns the key, and its `backup_target` field is absent ⇒ `false`. **The retraction was the error, not the theory.** |
| **#2 — "no row carries the guest path; the union row is deduped away"** (the basis of the shipped v0.115.0 fix) | ❌ **FALSE on both halves.** A row *does* carry `guest_path:"/mnt/felhom-drives/cel"` (row 4), and the union row is *not* deduped — precisely *because* `MountPath` does not survive on the Observe row, which is the fact #2 was built on. #2 had the premise right and drew the opposite conclusion from it. |
| **#3 — "`isTarget["/mnt/<n>"]` is already `true` in the absent state, so the controller could fall back to it"** | ❌ **FALSE.** `isTarget["/mnt/cel"] = false` as well — the same union row writes both keys. A fallback to the raw path changes nothing. |
So: **#1**, and it was available all along. The one genuinely new fact is *why* the dedup does not fire
— `seen` is keyed on the very field the absent state empties (`disks.go:290-295` vs `:298`), which makes
the union row's emission a **consequence** of the vanished device rather than an independent condition.
---
## 10. What a fix would have to change — constraints for the next session's spec
**Not implemented here. Stated as constraints, not as a design.**
1. **The necessary and sufficient condition is that the row owning the key
`/mnt/felhom-drives/<name>` must report `backup_target: true` when that drive is the primary
target.** Everything else is a means. Whether that is achieved by carrying the flag onto the union
row, by merging the two rows, or by making `driveTargetByPath` OR rather than assign, is the spec's
choice — but it must be verified against a payload, not a fixture.
2. **`disks.go:301-306` omits `BackupTarget` — and the omission is invisible.** A struct literal that
silently defaults a bool is the whole defect. Whatever the fix, this literal is the site. Note the
union loop has no access to `t` (there is no Observe target for it) — it would have to compare
`d.Name`/`d.MountPath` against `primaryTargetID`/the target's path, and **`d.Name` for a union row is
the drive's UUID, not the storage id** (payload: `"694034cc-…"` vs `"felhom-backup"`), so a naive
`d.Name == primaryTargetID` comparison will silently never match. That trap is worth stating in the
spec explicitly.
3. **Fixing only the agent may be enough, but the controller half is worth a look in the same pass.**
`driveTargetByPath` uses assignment, so it is order-dependent the moment two rows ever carry the
same key with different values — which is exactly the state a partial fix would create. An OR (or a
documented single-writer invariant with a test) removes the hazard.
4. **v0.115.0's fallback should not simply be extended.** It is gated on `t.BackingDevice == ""` and
feeds `StablePathForRaw(t.MountPath)`; in the failing state `MountPath` is also `""`, so there is no
value on that row to derive a guest path *from*. Any fix on the Observe row needs a path source that
survives device loss — and the only one that does is the same `.mount` unit the union row already
reads. That argues for fixing the union row, not the Observe row.
5. **Do not let the fix make an absent drive read as present.** `planDriveGates` computes
`present[gp] = present[gp] || d.BoundUnderParent` (`intermediary.go:226`). The union row's
`BoundUnderParent` is `false` in the payload (R-113's conjunction holds), and it must stay false. The
existing `TestAbsentTargetRowDoesNotRegisterPresence` pins this for the Observe row; the union row
needs the equivalent.
6. **The red-proof must be the payload shape, not the Session-C fixture.** v0.115.0 was unit-tested and
red-proofed against a fixture that supplied a `MountPath` and a `disconnected` state — a shape that
does not occur. The fixture for the next fix is §6.2 of this document: **4 rows, the target's flag
and its guest path on different rows, `state:"attached"` on the union row.**
7. **Whether to keep v0.115.0 is now answerable.** It is provably inert (it assigns `""`). It is not
harmful. Reverting it is a separate change with its own risk; extending it is ruled out by (4).
Recommendation, for the spec to accept or reject: **leave it, and note in its comment that the case
it targets cannot arise** — or delete it in the same commit that fixes the union row, so the tree
does not carry two explanations of one defect.
---
## 11. R-117 (new) — the returned drive's guest bind is a DEAD MOUNT, and everything reports healthy
The brief's §9 "stale bind, unchased after three runs". **It is not cosmetic.** Captured this run, and
it is worse than a wrong device name.
**PAYLOAD + live probe, RETURNED state (drive back, host healthy on `/dev/vdc`):**
```
host findmnt -no SOURCE /mnt/cel → /dev/vdc
guest findmnt -no SOURCE,OPTIONS /mnt/felhom-drives/cel → /dev/vdb[/felhom-data] rw,relatime,shutdown
guest ls /mnt/felhom-drives/cel → ls: reading directory …: Input/output error
guest write into it → cannot create …: Input/output error (GUEST-WRITE-FAIL)
host ls /mnt/cel/felhom-data/ → fine, empty, live
/disks says: state "attached", role "user-data", bound_under_parent TRUE, backup_target true
```
**The consequence, and this is why it outranks R-116.** `planDriveGates` computes
`present["/mnt/felhom-drives/cel"] = … || d.BoundUnderParent` (`intermediary.go:226`), which is `true`.
The drive is currently `Disconnected`, so the gate takes the **`Return`** branch
(`intermediary.go:281`, `:301-320`): it clears the disconnected flag and **restarts the gate-stopped
apps onto a mount that returns `EIO` for every operation** — then reports the drive healthy and emails
`backup_target_restored`. A customer's apps come back up pointed at a dead namespace with no alarm on
any channel.
R-113's conjunction cannot catch it: half 1 (guest mountinfo has an entry at the guest path) is
satisfied by the *stale* entry, and half 2 (`devicePresent("/mnt/cel")`) is satisfied by the *new*
device. **Neither half compares the two.** `disks.go:158-165`'s comment claims only that the conjunction
fixes the boot-ordering and outlived-bind cases — it does not claim this, so this is a gap rather than a
false invariant.
Reproduced deliberately and cleanly here; consistent with the same asymmetry seen (uninvestigated) in
E-2d, Session C and Part 5. **Filed, not chased, per the brief.**
---
## 12. R-118 (new) — the absent drive's union row reports the ROOT filesystem's size as its own
**PAYLOAD, §6.2 row 4:** `total_bytes: 49675956224`, `used_bytes: 4584579072` — **byte-identical to the
`local` row** (`durable_id: path:/var/lib/vz`, i.e. `pve-root`) in the same response. The real drive is
**4 GB** (`4143677440` in §6.1).
Cause, from source: `statfsCapacity(d.MountPath)` at `disks.go:335-338` statfs's `/mnt/cel`, which with
the device gone is a **bare directory on the root filesystem**. `observe.go:176-183`'s comment warns
about exactly this trap and guards the Observe path against it ("*an unmounted removable dir-storage's
mountpoint reverts to a bare directory on root … would be a catastrophic DR mis-id*"); **the union path
at `:335` has no equivalent guard.**
Not a DR mis-id — `durable_id` on that row is still the correct `uuid:694034cc…`, so the re-attach
identity is safe. It is a **false capacity**: a detached 4 GB drive advertises 46 GiB with 9.2 % used, to
any consumer of `total_bytes`/`used_fraction` (fill monitors, the storage cards). Same class as the
`role.go:180-181` flip that drives R-116 — an absent drive's fields decaying to the root filesystem's.
---
## 13. Teardown
| | Before | After |
|---|---|---|
| drill VM `pvesm status` | `felhom-backup` active, `local-lvm` 0.99 %, `local` 9.23 % | `felhom-backup` **removed**, `local-lvm` **0.00 %**, `local` 9.23 % |
| drill VM `pct list` | `9201 running` | **empty** |
- guest 9201 `pct destroy --purge` (logical volume removed, config purged); `pvesm remove felhom-backup`;
`mnt-cel.mount` disabled + deleted + `daemon-reload`; `shred -u` on the local-API token file and the
PVE token file.
- `poweroff`, qemu exited (`0` procs), **`qemu-img snapshot -a virgin drill.qcow2`** → the golden-bake
fixture is exactly as found (`virgin`, ID 1, 2026-07-03 14:12:15).
- scratch artifacts deleted: `r116-cel.qcow2`, `qmp.sock`, `qemu.pid`, both console `.ppm` dumps.
- DooPlex `df -h /mnt/5_hdd`: **2.1T used / 6.6T avail, 24 %** — same as at session start.
- The disposable `root@pam!r116` PVE token and the drill local-API token died with the snapshot revert;
neither is recorded in any committed file.
**Fences — re-verified after teardown, not assumed:**
| Fence | State |
|---|---|
| demo-hp `local-lvm` | **38.83 %** — identical before, after, and to Part 5's reading |
| demo-hp `drill-r50` (VM 300) | `stopped` — untouched |
| demo-hp guest 9201 | `running` |
| felhom-pve `felhom-backup` | `active`, `/mnt/hdd_1` on `/dev/sdb` — **target not re-pointed** |
| felhom-pve guest 9201 | `running` |
| Both demo boxes | **read-only access only** — `GET /disks`, file reads, `pvesm status`, `pct list`, one `scp` of the agent binary |
| Code | **no `.go` file edited, nothing built, nothing published, v0.115.0 untouched** |
### `sess-d`, and a leftover the prior runs did not record
`sess-d-0452c4` now reads **STALE**, not ONLINE. The delete gate refuses only on
`s.hostStatus(...) == "ok"` (`hub/internal/web/customer_delete.go:220-228`) — ONLINE — so **`sess-d` is
now deletable.** Not executed here: deleting a customer runs external teardown plus a DB purge, and the
brief scoped this session to recording the command.
```
POST /configs/sess-d/delete ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-d expect_hosts=1
```
**Also still present, and not mentioned in the Session-C audit: `sess-c` / host `sess-c-8158e4`
(agent 0.114.0, DOWN).** Same shape, same gate, deletable on the same terms —
`confirm_id=sess-c expect_hosts=1`. Two drill customers are now accumulating in the hub; that is the
operator's cleanup call, not a register row.
---
## 14. What did not happen, and why
- **No controller was run, so no fresh gate-log / hub-event correlation was captured** (brief §6.4).
Deliberate: that observable was already captured live and identically **twice** (Session C, Part 5),
and the *payload* was the only missing input. Staging a controller meant a hub customer, a pairing, a
golden fetch and a claim — the work that consumed the three prior sessions. §8 correlates the payload
to those existing measurements instead, and labels that as the inference it is.
- **No new drill box was built** — the brief's stated preference. The existing DooPlex nested-PVE
fixture was reused and restored.
- **R-117 and R-118 were filed, not investigated.** R-117 in particular deserves its own run; on the
evidence here it is more customer-affecting than R-116.
- **v0.115.0 was neither reverted nor extended**, per the fence.
- **Two boot attempts of the drill VM failed before the run started** and are recorded because the cause
is a reusable trap: adding `-device` entries at launch shifts PCI slot assignment, which renames the
guest NIC (`ens3` → gone), so vmbr0 loses its port and SSH is unreachable while the console shows a
healthy PVE login. `bootindex` fixes the boot-order half but not the NIC half. **The fix is to boot
the memory's known-good command line unchanged and hot-plug the extra disk over QMP afterwards** —
which is also the more faithful device-loss mechanism.
+3 -1
View File
@@ -16,7 +16,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
| **R-110** | **`main` is the installer's publish channel — there is no staging.** `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a 30 s period and nginx serves that working tree directly (`location /scripts/`, `root …/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as **root on a virgin box**, the single most privileged thing Felhom ships | **WAITING-ON-OPERATOR (S)** | operator ruling | **Two consequences worth stating:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" **was never available to take**. **Open question for the operator, not a defect to fix blind:** whether `/scripts/` should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether `main`-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now. **SECOND INSTANCE, found 2026-07-29 by the E-2d run and filed here rather than as a new ID:** `felhom-host-install.sh` fetches **nine** files from `raw/branch/main` (`:2072`–`:2206`) and the hub manifest vouches a sha for exactly **one** (`wrapper_sha256` → `felhom-pbs-apply`; re-checked this run, no drift). E-2a's `felhom-backup-target-apply` (`:2116`) is installed **0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n` — a root-executed artifact taken from `main` with no pinned integrity, which is this row's class exactly | CC |
| **R-111** | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`.**~~ `felhom-host-install.sh` does not use `main`: it reads the hub-vouched manifest (`:423-436`) and fetches Gitea generic packages (agent `:1945`, golden `:2573`). Gitea's newest are **agent 0.96.0** and **golden 0.161.0**, and the hub's manifest selects exactly those — so a fresh box lands on **agent 0.96.0 + controller 0.161.0** (global floor `v0.156.0` < the golden's 0.161.0, so no self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and was never published** | **SHIPPED 2026-07-29 — the channel now serves agent 0.113.0 + golden 0.185.1** | — | **FIXED the same day it was found.** Agent **0.113.0** built from the clean tree @ `58b598b` and published (`scripts/publish-agent.sh`), sha `5f3247f756cb658e…`, round-trip GET verified. Golden **0.185.1** baked on the nested drill VM embedding controller `0.185.1`, published, sha `dba00f3e845c415e…` — bake clean: `Result=success`, overlay2, **all 3 mounts included** (rootfs+mp0+mp1), 0 FATAL/exclusions, upload HTTP 201, token-leak grep 0; log `drill/bake-0.185.1.log`; GL-1 teardown done (guest 9100 purged, secrets shredded, disk restored to `virgin`). Hub Day-0 manifest moved **both together in one POST** so it never vouched a new agent against an old golden; `min_agent` **0.93.0 → 0.113.0**, which is what controller v0.185.0 declares (`felhom-controller/CHANGELOG.md:15`) — **zero fleet impact, verified: all three enrolled hosts already run agent 0.113.0, so no box is held.** `wrapper_sha256` preserved verbatim (re-checked against `configs/felhom-pbs-apply` — no drift). **The global controller floor was deliberately NOT raised**: the golden now bakes 0.185.1, so a fresh box needs no self-update, and raising it would have been an unnecessary fleet-wide write. Original finding follows. **Found 2026-07-29 by the E-2d Phase 0 gate, which stopped the run before a VM was created.** 17 unpublished releases (v0.97.0–v0.113.0) strand the **entire R-82 tiered-backup arc** plus **F-CRIT-2** (a failed backup looking fresh — 7 days silent) and **F-REBOOT** (a guest rebooted mid-backup never returns): a new customer's box would install without them. **Blocks E-2d's C3/C4/C5** — those test endpoints and events that do not exist in 0.96.0/0.161.0. The **controller is fine** (registry has 0.185.1, floor-driven self-update), so the gap is specific to the two Gitea-generic artifacts. **Mirror of R-110, not a duplicate:** R-110 = the installer publishes instantly with no staging; R-111 = the agent/golden publish gate exists and was never walked. Fix should decide whether publishing joins the release train rather than staying a remembered step (R-29's shape, one layer up). Evidence: `audits/E2D-fresh-vm-2026-07-29.md` **DEFERRED LEG, AND IT RECURRED → R-115.** This row's shipped half stands and is not reopened: the bump happened, was verified, and was proven end-to-end by the E-2d install. But its own closing line — *decide whether publishing joins the release train rather than staying a remembered step* — was never acted on, and agent 0.114.0 reproduced the exact condition the same afternoon. The recurrence is filed as **R-115**, not as a reopen, because the stale-channel finding is closed while the process defect that caused it is a distinct problem with a distinct fix. | CC |
| **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **WAITING-ON-OPERATOR (M)** | operator ruling on the release process | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design | CC |
| **R-116** | **The drive-absent alarm and its recovery are a MISMATCHED PAIR — absent fires the GENERIC `storage_disconnected`, return fires the SPECIFIC `backup_target_restored`.** `backup_target_absent` never fires at all (count 0 across a full Session-C run). An operator gets an alarm they cannot match to its recovery, which is exactly what `notifyDriveReturned`'s own comment forbids | **STILL OPEN — fix shipped (agent v0.115.0) and INEFFECTIVE; mechanism NOT isolated** | a working `/disks` read | **PART 5 RE-VALIDATION FAILED 2026-07-30 (`audits/R116-PART5-2026-07-30.md`).** A fresh box running the FULLY SHIPPED stack — agent **0.115.0 from the Day-0 manifest** + controller 0.185.1 from the vouched golden — still fired the **generic** `storage_disconnected` on detach and the **specific** `backup_target_restored` on return. `backup_target_absent` count 0. Identical to Session C. **Discrimination FAILS** (the target itself produced the generic event, so the non-target leg was pointless to stage — reported as a fail, not as Session C's trivial pass). **Over-correction guard PASSES** (0 ABSENT lines with the drive present). **The v0.115.0 fix targets a shape that does not occur live, and WHICH shape does occur is NOT ISOLATED.** With the drive detached PVE reports the storage `inactive` with zeroed fields — a shape the unit fixture did not model. Three attempts to read the real `/disks` payload failed on token extraction, and **a present-drive control also returned 0 rows, proving the QUERY was broken, not the payload** — without that control this would have been a third false mechanism, after "the union row writes false" (wrong, corrected) and "no row carries the guest path" (unverified). **NEXT SESSION'S FIRST JOB: a working `/disks` read with a present-drive control run FIRST, before any code.** v0.115.0 is published, vouched and **inert**; not reverted (reverting is itself a change, and the runbook forbids fixing mid-run) | CC |
| **R-116** | **The drive-absent alarm and its recovery are a MISMATCHED PAIR — absent fires the GENERIC `storage_disconnected`, return fires the SPECIFIC `backup_target_restored`.** `backup_target_absent` never fires at all (count 0 across a full Session-C run). An operator gets an alarm they cannot match to its recovery, which is exactly what `notifyDriveReturned`'s own comment forbids | **OPEN — MECHANISM ISOLATED 2026-07-30, fix NOT yet written** | nothing — READY (S) | **MECHANISM ISOLATED FROM THE REAL PAYLOAD (`audits/DIAG-r116-disks-payload-2026-07-30.md`).** The `/disks` read path is solved and written down verbatim: the token's plaintext lives ONLY in `/var/lib/felhom-agent/guests/<vmid>/bootstrap/bootstrap.json` **on the Proxmox host** (the agent's token store keeps hashes only, `tokenstore.go:26-32` — which is what defeated three prior extraction attempts); curl the island endpoint from the host itself. **Present-drive control run PASSED first** (felhom-pve, HTTP 200, 4 rows) so a 0-row query could not be mistaken for an empty payload. Absent state then staged on the existing DooPlex nested-PVE drill fixture with the **byte-identical live 0.115.0 binary** (`sha256 f48544ad…`) and every state-producing step through the real endpoints (format→assign→guest-attach→backup/target); device loss by QMP hot-unplug. **THE ANSWER IS THEORY #1 — "the registry-union row writes `false`" — the theory that was raised, declared wrong and RETRACTED. The retraction was the error.** In the absent state `/disks` returns **4 rows, not 3**: the drive appears TWICE and the two facts the controller needs sit on DIFFERENT rows. The Observe row (`felhom-backup`) has `backup_target:true` but `mount_path:""` **and** `guest_path:""` (role flipped to `system`; v0.115.0's fallback calls `StablePathForRaw("")` which returns `""` — `intermediary.go:69-75` — so **it assigns nothing, which is exactly why it is inert**), so it contributes NO key to `driveTargetByPath` at all. The registry-union row (`disks.go:297-339`) is **NOT deduped** — `seen` is keyed on `MountPath` (`:290-295`) and the Observe row's is now empty — and it owns both keys while **`BackupTarget` is absent from its struct literal (`:301-306`) ⇒ `false`**. Its `MountPath` survives the device because the union source is the **systemd `.mount` unit FILE** (`registry_known.go:40-75` via `main.go:605`→`:764`), which never consults the mount table. So `isTarget["/mnt/felhom-drives/cel"] = false` → generic; on return the two rows re-merge into one carrying both facts → specific. Verified by applying `intermediary.go:602-618` to the captured payload: PRESENT `True` / ABSENT `False` / RETURNED `True` — **the live asymmetry reproduced from payload alone.** **Theory #2 (the basis of v0.115.0) is FALSE on both halves** — a row DOES carry the guest path, and the union row is not deduped *because of* the very premise #2 was built on; **project-Claude's #3 is FALSE too** (`isTarget["/mnt/cel"]` is `false` as well — same row writes both keys). **FIX CONSTRAINTS, not a design** (see §10 of the diag): the necessary+sufficient condition is that the row owning `/mnt/felhom-drives/<name>` reports `backup_target:true`; the site is the union struct literal; **trap — `d.Name` on a union row is the drive UUID, not the storage id, so a naive `d.Name == primaryTargetID` silently never matches**; do NOT extend v0.115.0 (there is no path value on that row to derive from — the unit file is the only source that survives, which argues for fixing the union row); keep `BoundUnderParent` false there so the gate cannot read absent as present; and **red-proof against §6.2's real shape, not Session C's fixture** (v0.115.0 was red-proofed against a shape that does not occur). v0.115.0 is provably INERT and harmless — keep-or-delete is the fix spec's call. **Original Part-5 record follows.** **PART 5 RE-VALIDATION FAILED 2026-07-30 (`audits/R116-PART5-2026-07-30.md`).** A fresh box running the FULLY SHIPPED stack — agent **0.115.0 from the Day-0 manifest** + controller 0.185.1 from the vouched golden — still fired the **generic** `storage_disconnected` on detach and the **specific** `backup_target_restored` on return. `backup_target_absent` count 0. Identical to Session C. **Discrimination FAILS** (the target itself produced the generic event, so the non-target leg was pointless to stage — reported as a fail, not as Session C's trivial pass). **Over-correction guard PASSES** (0 ABSENT lines with the drive present). **The v0.115.0 fix targets a shape that does not occur live, and WHICH shape does occur is NOT ISOLATED.** With the drive detached PVE reports the storage `inactive` with zeroed fields — a shape the unit fixture did not model. Three attempts to read the real `/disks` payload failed on token extraction, and **a present-drive control also returned 0 rows, proving the QUERY was broken, not the payload** — without that control this wouldLine truncated
| **R-117** | **A returned drive's guest bind is a DEAD MOUNT, and the gate restarts the customer's apps onto it while every signal reads healthy.** After a detach/reattach the host raw mount is fine on the NEW device node, but the guest's bind still names the OLD one and every operation through it returns `EIO`: host `findmnt /mnt/cel` → `/dev/vdc`; guest `findmnt /mnt/felhom-drives/cel` → `/dev/vdb[/felhom-data] …,shutdown`; guest `ls` and write → **`Input/output error`**; and `/disks` reports `state:"attached"`, `role:"user-data"`, `bound_under_parent:TRUE`, `backup_target:true` | **READY (M) — NEW 2026-07-30, deliberately not chased** | — | **Worse than R-116 and it outranks it.** `planDriveGates` computes `present[gp] = … \|\| d.BoundUnderParent` (`intermediary.go:226`) = `true`, and the drive is `Disconnected`, so the gate takes the **`Return`** branch (`intermediary.go:281`, `:301-320`): it clears the flag, **restarts the gate-stopped apps onto a namespace that `EIO`s on every call**, reports the drive healthy and emails `backup_target_restored`. **No alarm on any channel.** R-113's conjunction cannot catch it — half 1 (guest mountinfo has an entry at the guest path) is satisfied by the STALE entry and half 2 (`devicePresent`) by the NEW device; **neither half compares the two**. Not a false invariant: `disks.go:158-165` claims only the boot-ordering and outlived-bind cases, so this is a genuine gap. **Seen but uninvestigated in E-2d, Session C and Part 5 (three consecutive runs) as "the drive returned as /dev/sdc while findmnt still read /dev/sdb"** — that framing read as cosmetic; it is not. Reproduced deliberately and probed to EIO in `audits/DIAG-r116-disks-payload-2026-07-30.md` §11. Deserves its own run | CC |
| **R-118** | **An absent drive's union row advertises the ROOT filesystem's capacity as its own.** In the absent-state payload the registry-union row reports `total_bytes: 49675956224 / used_bytes: 4584579072` — **byte-identical to the `local` row** (`durable_id: path:/var/lib/vz`, i.e. `pve-root`) in the same response. The real drive is **4 GB** | **READY (XS) — NEW 2026-07-30** | — | Cause: `statfsCapacity(d.MountPath)` (`disks.go:335-338`) statfs's `/mnt/cel`, which with the device gone is a **bare directory on the root filesystem**. `observe.go:176-183`'s comment warns about exactly this trap and guards the Observe path ("*an unmounted removable dir-storage's mountpoint reverts to a bare directory on root … catastrophic DR mis-id*"); **the union path has no equivalent guard.** **Not a DR mis-id** — `durable_id` on that row is still the correct `uuid:…`, so re-attach identity is safe. It is a **false capacity** reaching every consumer of `total_bytes`/`used_fraction` (fill monitors, storage cards): a detached 4 GB drive advertises 46 GiB at 9.2 % used. Same class as `role.go:180-181` — an absent drive's fields decaying to the root filesystem's. Evidence: `audits/DIAG-r116-disks-payload-2026-07-30.md` §12 | CC |
| **R-113** | ~~**The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.**~~ `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc/<pid>/mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **SHIPPED + PROVEN-LIVE** (agent v0.114.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** Target drive hot-detached on a fresh box running the SHIPPED agent 0.114.0 (from the Day-0 manifest, not a hand build). **The gate fired in 4 seconds** — E-2d measured ZERO over 4½ minutes — and `SetDisconnected` was reached. It fired on exactly the shape that defeated it: raw `/mnt/mentes` NOT mounted while the bind `/mnt/felhom-drives/mentes` still read `/dev/sdb[/felhom-data]`. **Over-correction guard PASSED:** with both drives present, 0 ABSENT lines and the target stayed healthy. **Note: C5 still FAILED — but on a different defect (→ R-116), not on this one.** **SESSION C IS NOW UNBLOCKED.** Agent **0.114.0 published and vouched** 2026-07-29 (sha `5e4c15ebee2d7583…`, round-trip GET verified; hub manifest read back showing it) — a fresh drill box now installs the fix under test instead of the bug. The golden was NOT re-baked and `min_agent` was NOT raised: the golden bakes the controller, not the agent, and controller v0.185.0 declares MinAgent 0.113.0, which 0.114.0 already satisfies. **FIX: `BoundUnderParent` is now a CONJUNCTION — bound under the parent AND the drive's raw host mount still mounted** (`devicePresent`, new `deviceCheck` seam), at BOTH `/disks` construction sites. The raw mount is the device-bound systemd unit that dies with the device; the agent's bind is not — so the raw mount IS the device signal, grounded in E-2d's measurement rather than inference. **Conjunction, deliberately:** the device half alone would regress boot ordering (raw mounts early, bind lands ~18 s later — that window must keep reading absent), so existing behaviour is byte-identical and only the unreachable case is closed. **Unknown is never absent** (`devicePresent("")` = true) — a false absent stops a working customer's apps. **Controller UNCHANGED, no MinAgent bump:** `BoundUnderParent` has exactly one functional consumer (`planDriveGates:226`); a new `DevicePresent` bool was rejected because absent-from-JSON decodes to `false`, which would have made every drive on an older agent read ABSENT. +6 tests (208→214), 4 red-proofs run and reverted. **Deployed to demo-felhom; over-correction guard verified in production** — raw mount present, drive still reads present, 10/10 apps untouched, no gate action, no false alarm. **demo-hp deliberately left on 0.113.0** (the spec scoped deploy to felhom-pve) — it still carries the bug. **⚠️ SESSION C BLOCKER: the hub Day-0 manifest vouches agent 0.113.0**, so a fresh drill box would install WITHOUT this fix and validate nothing — publish + vouch 0.114.0 first (R-111's trap, same shape). **Awaiting live proof of exactly one leg:** device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. Original finding: **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 **Session C scope UNCHANGED by Session B** — R-114/R-112 shipped as controller v0.186.0 and neither touches the agent; the leg awaiting proof is still device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. One rebuild now validates all three. | CC |
| **R-112** | **E-2's degraded banner and offer have NO UI CONSUMER — the endpoint is correct and the customer never sees it.** `GET /api/storage/backup-target` returns byte-exact copy (proven live), and **nothing fetches it**: `grep 'backup-target'` across all `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` + `degradedMessageFor` are consumed **only** by the JSON handler — **no page handler injects the state** | **SHIPPED + PROVEN-LIVE** (controller v0.186.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md` — the banner reached a customer's page for the first time.** Never-configured box: banner element 1, never-configured copy 1. After the wizard: offer block 1 with `data-path="/mnt/felhom-drives/mentes"`. Healthy after assign: all four markers 0, **proven POSITIVELY** — idle delta 0 `/backup/tiers` calls, page-load delta +1, single caller ⇒ the seam ran and chose silence. **FIXED: the state now has a consumer.** Server-rendered on `/backups` via `backupsHandler` → `backupTargetView` → `backups.html`, following the existing `SingleCopyWarning` banner pattern — NOT a 19th JS fetch, because a banner that needs JavaScript to appear is one more thing that can silently not happen. `backupTargetView` returns **nil** for healthy AND unknown, so those render nothing at all. **Scenario-E seam test drives `backupsHandler` over httptest and asserts the RENDERED HTML** — deleting the one line that sets `data["BackupTarget"]` reproduces the old state and fails every render assertion. **SEAM PROVEN LIVE on demo-felhom by a DIFFERENTIAL positive observable, not by an absent banner:** idle 8 s → 0 new `/backup/tiers` agent calls; each `/backups` load → exactly +1, and that call has only one caller (`resolveBackupTargetState`). The box is healthy (`degraded:false, target:felhom-backup`) and the page correctly rendered **nothing** — which matches its real state but, being a negative, is NOT by itself proof of wiring. **Still unproven live:** that a customer sees actual copy — impossible on a healthy box. **The decisive contrast: templates fetch 18 distinct `/api/storage/*` endpoints; `backup-target` and `backup-target/assign` are the only two with zero references.** The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"* — an invariant comment asserting a consumer that does not exist (7th instance of that class). v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* — it fixed the **router mount** and stopped one layer short of the **render**; its test `TestBackupTargetRoutesLiveUnderTheStorageAPIMount` pins dispatch, not reachability, which is exactly what `CLAUDE.md`'s seam rule warns about. **Fifth instance of seam-built-but-never-wired.** **Fix R-114 FIRST** — wiring this alone would start showing customers the wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | CC |
| **R-114** | **On target-drive loss the customer is told the wrong story and offered the drive that just vanished.** With the assigned target absent, the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"a rendszermentés ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a drive that has disappeared, not the system disk — **and** `offer_path` pointing at the missing drive as the remedy | **SHIPPED + PROVEN-LIVE** (controller v0.186.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** With the target absent the page rendered the ABSENT copy (1), the system-disk copy 0, the offer block 0 — both of E-2d's falsehoods gone. API carried `message:"A rendszermentés meghajtója nem érhető el…"` with `target:felhom-backup`. **FIXED: the third state exists.** New `BackupTargetState.TargetAbsent` separates *configured-and-gone* from *never-configured*. `Degraded` keeps its meaning (is there a problem) so the wire contract is unchanged for every consumer; `TargetAbsent` answers which problem, because the remedies are OPPOSITE — attach any second drive vs reconnect *that* one. Copy routed through `degradedMessageFor` (still one decision point) and taken **verbatim** from the hub's `backup_target_absent` email so banner and mail tell one story. **Offer suppressed on the branch itself**, deliberately not left to `firstOfferableDrive`'s `Disconnected` skip — that flag is set by R-113 in another repo, and this state must be right without it. Red-proof: deleting the branch reproduces E-2d's exact payload, offering `/mnt/felhom-drives/mentes2`, the drive that had vanished. **MinAgent unchanged 0.113.0** — R-114 reads `BackupTarget`/`MountPath`/`GuestPath`/`Role`, none of which R-113 altered, so demo-hp is not held. **NOT live-validated: Scenario C cannot occur on a healthy box.** `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing **never configured** from **configured and now missing**. Shares R-113's root cause — two disagreeing presence signals — but is a different code path with a different fix. **Currently invisible ONLY because of R-112; fix this before wiring that.** Also seen: after reattach the drive returned as `/dev/sdc` while the stable bind still recorded `/dev/sdb`, and the state read healthy. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 | CC |