diff --git a/REPORT-r116-diag.md b/REPORT-r116-diag.md new file mode 100644 index 0000000..d5d527e --- /dev/null +++ b/REPORT-r116-diag.md @@ -0,0 +1,113 @@ +# REPORT-r116-diag — the `/disks` payload captured, R-116's mechanism isolated (2026-07-30) + +Read-only diagnosis run by CC on DooPlex. **No code written, nothing built, nothing published.** +Full evidence: `documentation/audits/DIAG-r116-disks-payload-2026-07-30.md`. + +A `REPORT-*.md` sibling, not the shared `REPORT.md` (`CLAUDE.md` parallel-session rule). + +## Outcome + +**Both goals met.** The `/disks` read path is solved and written down verbatim, proven by a +present-drive control run *first*; and the absent-state payload was captured, which isolates the +mechanism. + +**R-116 is theory #1 — "the registry-union row writes `false`" — the theory that was raised, declared +wrong, and retracted. The retraction was the error.** + +In the absent state `/disks` returns **4 rows, not 3**. The drive appears twice and the two facts the +controller needs sit on different rows: + +| row | source | `mount_path` | `guest_path` | `backup_target` | +|---|---|---|---|---| +| `felhom-backup` | Observe (`disks.go:196-284`) | `""` | `""` | **`true`** | +| `694034cc-…` (the UUID) | registry union (`disks.go:297-339`) | `/mnt/cel` | `/mnt/felhom-drives/cel` | **field ABSENT ⇒ `false`** | + +So the row holding the flag contributes **no key** to `driveTargetByPath`, and the row that owns the key +says `false` → `isTarget[a.Path]` is `false` → generic `storage_disconnected`. On return the rows +re-merge into one carrying both facts → specific `backup_target_restored`. Applying +`intermediary.go:602-618` to the captured payloads gives PRESENT `True` / ABSENT `False` / +RETURNED `True` — **the live asymmetry reproduced from payload alone.** + +The union row's `MountPath` survives the device because the union source is the systemd **`.mount` unit +file** (`registry_known.go:40-75` via `main.go:605`→`:764`), which never reads the mount table. The +dedup at `:298` therefore does not fire, because `seen` is keyed on the one field the absent state +empties (`:290-295`). + +**Theory #2 (the basis of the shipped v0.115.0) is false on both halves**; **#3 is false too** +(`isTarget["/mnt/cel"]` is `false` as well). **v0.115.0 is provably inert** — its fallback calls +`StablePathForRaw("")`, which returns `""` (`intermediary.go:69-75`), so it assigns nothing. + +## The read path (this cost two prior sessions — it should never cost again) + +The token plaintext exists in exactly one place: `bootstrap.json` **on the Proxmox host**. The agent's +own store keeps SHA-256 hashes only (`tokenstore.go:26-32`), which is what defeated the earlier attempts. + +```bash +ssh felhom-pve +B=/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json +TOK=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['token'])") +EP=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['endpoint'])") +curl -sS -k -H "Authorization: Bearer $TOK" "https://$EP/disks" | python3 -m json.tool +``` + +Control run, live felhom-pve, drive present: **HTTP 200, 2483 bytes, 4 plausible rows** — so Part 5's +0-rows-on-a-present-drive failure mode is excluded. + +## Where the absent state was staged + +No new box. The existing DooPlex **nested-PVE drill fixture** (`drill/drill.qcow2`, snapshot `virgin`) — +my own host, zero production exposure, and it can hot-unplug a disk for a genuine device loss. Run with +the **byte-identical live agent binary** (`sha256 f48544ad…`, `--version` 0.115.0) and every +state-producing step through the real endpoints (`format` → `assign` → `guest-attach` → +`backup/target`). Its present-state row matched felhom-pve's control run field-for-field before it was +trusted. Non-production aspects (root/direct privileged mode, stubbed hub, a hand-written bearer-token +record, no controller) are enumerated in the audit §4. + +## Two new findings, filed not chased + +- **R-117 (READY M) — outranks R-116.** After a detach/reattach the guest's bind is a **dead mount**: + host is healthy on the new device node, guest still names the old one, and `ls`/write through it + return **`EIO`** — while `/disks` reports `attached` + `bound_under_parent:true` + `backup_target:true`. + `planDriveGates` therefore takes the `Return` branch and **restarts the customer's apps onto a dead + namespace, reporting healthy, with no alarm on any channel.** R-113's conjunction cannot catch it: + one half is satisfied by the stale entry, the other by the new device, and neither compares them. + This is the "stale bind" seen and dismissed as cosmetic in three consecutive runs. +- **R-118 (READY XS).** An absent drive's union row reports the **root filesystem's** capacity as its own + (46 GiB / 9.2 % for a 4 GB drive) — `statfsCapacity` at `disks.go:335-338` statfs's a bare directory on + root. `observe.go:176-183` guards the Observe path against exactly this; the union path does not. + `durable_id` is still correct, so re-attach identity is safe — it is a false capacity, not a DR mis-id. + +## Register + +`documentation/backlog/OPEN-ITEMS.md` — R-116 updated with the mechanism and the fix constraints; +R-117 and R-118 added. The single register edit this session makes. + +## Record correction + +The brief's baseline `controller 0.185.1` is the version the **golden bakes**; both demo boxes actually +run **0.186.0** (R-114 + R-112, 2026-07-29). Both are true of different things, and the distinction +matters because R-114's `TargetAbsent` branch exists only in 0.186.0. Confirmed: agent **0.115.0**, +controller **0.186.0** live, hub **0.81.0** (manifest pin and live pod image agree), host-install +**1.22.0**, `felhom.eu` HEAD `c3ce4c7`. + +## Teardown and fences + +Drill guest destroyed, scratch storage removed, mount unit deleted, secrets `shred -u`'d, VM powered +off, **`drill.qcow2` restored to `virgin`** (the golden-bake fixture is exactly as found), scratch qcow2 +and console dumps deleted. DooPlex `/mnt/5_hdd` at 24 %, unchanged. + +Both demo boxes **read-only throughout** and re-verified after teardown: demo-hp `local-lvm` **38.83 %** +(identical before/after and to Part 5), `drill-r50` still stopped, felhom-pve `felhom-backup` still +active on `/dev/sdb`, both guests running, **v0.115.0 untouched**. + +`sess-d-0452c4` now reads **STALE**, and the delete gate refuses only on ONLINE +(`hub/internal/web/customer_delete.go:220-228`) — so it **is** now deletable; the command is recorded in +the audit rather than executed (customer delete runs external teardown plus a DB purge). **`sess-c` is +also still present and was not recorded by the Session-C audit** — same terms. + +## Not done, deliberately + +No fresh controller gate-log / hub-event correlation: that observable was already captured live and +identically twice, and only the payload was missing. Staging a controller meant a hub customer, a +pairing, a golden fetch and a claim — the work that consumed the three prior sessions. The audit §8 +correlates the payload to those existing measurements and labels that step as inference. diff --git a/documentation/audits/DIAG-r116-disks-payload-2026-07-30.md b/documentation/audits/DIAG-r116-disks-payload-2026-07-30.md new file mode 100644 index 0000000..7bae28c --- /dev/null +++ b/documentation/audits/DIAG-r116-disks-payload-2026-07-30.md @@ -0,0 +1,488 @@ +# DIAG-r116-disks-payload-2026-07-30 — the `/disks` payload, and the mechanism isolated from it + +**Run:** read-only diagnosis by CC on DooPlex, 2026-07-30. **No code written, nothing shipped.** +**Result: the payload was captured in both states and the mechanism is isolated.** It is theory **#1** +— *the registry-union row writes `false`* — the theory that was raised, declared wrong, and retracted. +**The retraction was the error.** + +--- + +## 1. Baselines + +All confirmed this run, not carried from the brief. + +| Artifact | Version | How confirmed | +|---|---|---| +| agent | **0.115.0** | `felhom-agent --version` on felhom-pve; `felhom-agent/CHANGELOG.md:1` = `## v0.115.0`; repo HEAD `a58239f6ded367fe27b4eb57cf2fcb0c54414b80` | +| controller | **0.186.0** ⚠ | `pct exec 9201 -- docker ps` on felhom-pve → `felhom-controller:0.186.0`; `CHANGELOG.md:3` = `### v0.186.0`; HEAD `b331f18424b7cf8d0883e7709f74595808264080` | +| hub | **0.81.0** | `manifests/hub.yaml:128` pin **and** the live pod image (`kubectl -n felhom-system get pod hub-… -o jsonpath=…`) both `felhom-hub:0.81.0`; `hub/CHANGELOG.md:1` = `## v0.81.0` | +| host-install | **1.22.0** | `scripts/felhom-host-install.sh:187` `SCRIPT_VERSION="1.22.0"` | +| `felhom.eu` HEAD | `c3ce4c7b20ce4c0d3673590d710d13e5d28b68bb` | `git rev-parse HEAD` | + +⚠ **The brief's `controller 0.185.1` is not what is deployed.** 0.185.1 is what the **golden bakes** +(so it is what a fresh drill box installs, which is why Part 5 correctly recorded it); **0.186.0** is +what both demo boxes actually run, shipped 2026-07-29 as R-114 + R-112. Both numbers are true of +different things; the distinction matters because R-114's `TargetAbsent` branch exists only in 0.186.0. + +Hub host table, read this run: `demo-felhom-8363b5` agent 0.115.0 ONLINE · `demo-hp-bb76ea` agent +0.113.0 ONLINE · `drill-r50-0a4f9a` DOWN · `sess-c-8158e4` DOWN · `sess-d-0452c4` **STALE**. + +--- + +## 2. §3 reconfirmation — every cited fact, at `file:line` + +| Cited fact | Line now | Holds? | +|---|---|---| +| `observe.go` — `mountPath` comes from the mount table, assigned only on exact-mount success | `internal/storage/observe.go:184-190` | **HOLDS**, verbatim as quoted | +| `role.go` — `if backingDevice == "" { return RoleSystem }` for USB/LocalDir | `internal/storage/role.go:180-181` | **HOLDS**, verbatim | +| `primaryTargetID := s.primaryTier().TargetID`, in scope for both loops | `internal/localapi/disks.go:194` | **HOLDS** | +| `BackupTarget: t.Name == primaryTargetID` | `disks.go:211` | **HOLDS** | +| `GuestPath` set only inside `if di.Role == RoleUserData` | `disks.go:215-217` | **HOLDS** | +| first loop `out = append(out, di)` | `disks.go:284` (brief said `:249`) | **HOLDS — line moved**, v0.115.0 inserted `:226-260` | +| `seen[d.MountPath] = true` built by iterating `out` | `disks.go:290-295` (brief said `:258`) | **HOLDS — line moved.** Note the guard: `seen` is written **only when `d.MountPath != ""`** (`:292`) | +| union dedup `if d.MountPath == "" \|\| seen[d.MountPath] { continue }` | `disks.go:298` (brief said `:263`) | **HOLDS — line moved** | +| union row hardcodes `Role: RoleUserData` | `disks.go:304` | **HOLDS** | +| union row's `GuestPath` set from `StablePathForRaw(d.MountPath)` | `disks.go:309-310` | **HOLDS** | +| **`BackupTarget` is absent from the union struct literal** | `disks.go:301-306` | **HOLDS** — the literal sets `Name/Type/State/MountPath/DurableID/Role/GuestAttached` and nothing else | +| v0.115.0's fallback `di.GuestPath = StablePathForRaw(t.MountPath)` | `disks.go:258-260` | **HOLDS** | +| `StablePathForRaw("") == ""` (so the fallback sets nothing when `MountPath` is empty) | `internal/localapi/intermediary.go:69-75` → `DriveNameFromRaw:79-88` (no `/mnt/` prefix → `""`) | **HOLDS** | +| PVE reports the detached target `dir inactive 0 0 0` | reproduced live, §5 | **HOLDS** | + +**One fact the brief did not state, and it is load-bearing.** `/disks`' union source is the +**systemd-unit-file** registry, not PVE and not the mount table: +`storage.NewRegistryKnownTargets(driveUnitDir, …)` (`cmd/felhom-agent/main.go:605`) is the value passed +as `DriveTargets` (`main.go:764` → `buildLocalAPIServer` → `disks.go:296` `s.driveTargets.Known()`). +`Known()` (`internal/storage/registry_known.go:40-75`) `os.ReadDir`s `/etc/systemd/system`, parses each +Felhom `.mount` unit, and returns `MountPath: spec.Where` (`:70`) — **it never consults the mount table +or the device.** So the union row's `MountPath` survives the device's disappearance by construction. + +Verified on the live box (felhom-pve), unit present and device-bound: + +``` +/etc/systemd/system/mnt-hdd_1.mount + What=/dev/disk/by-uuid/47a3361a-91e0-4831-a69d-27f540ed3f48 + Where=/mnt/hdd_1 +/var/lib/felhom-agent/drive-intents.json → intent recorded (not `new`, so Known() includes it) +``` + +--- + +## 3. The working `/disks` invocation — verbatim + +Two sessions were lost to this. **The token's plaintext exists in exactly one place: the guest's +bootstrap mount, which the agent owns host-side.** The agent's own token store holds only SHA-256 +hashes (`internal/localapi/tokenstore.go:26-32`, `:136`), so it is *not* a source — that is what the +earlier extraction attempts were fighting. + +Authoritative source, on the **Proxmox host**, as root: + +``` +/var/lib/felhom-agent/guests//bootstrap/bootstrap.json # 0600, owner 100000:100000 +``` + +(written by `internal/provision/backhalf.go:124-130`; the in-guest view is +`/etc/felhom-bootstrap/bootstrap.json`, mp9, read-only — the host path avoids the guest hop entirely). + +**The invocation. Run it ON the Proxmox host so the token never crosses a hop or reaches a log:** + +```bash +ssh felhom-pve +export LC_ALL=C +B=/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json +TOK=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['token'])") +EP=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['endpoint'])") # 169.254.253.1:8443 +curl -sS -k -H "Authorization: Bearer $TOK" "https://$EP/disks" | python3 -m json.tool +``` + +`-k` is correct and not a shortcut: the leaf is self-signed and the controller pins it by SHA-256 +(`local_api.fingerprint` in the same file, `2d325e89…c191` on felhom-pve) rather than by a CA. The +endpoint is the island bridge `vmbr9` (`169.254.253.1/30`), reachable from the host itself. + +### 3.1 The control run — it proves the query works + +**Present-drive control, live felhom-pve, before anything else:** + +``` +endpoint=169.254.253.1:8443 +HTTP 200 bytes=2483 +rows: 4 (local-lvm, felhom-pbs, felhom-backup, local) +``` + +The backup-target row in that control run: + +```json +{ "name": "felhom-backup", "type": "local-dir", "state": "attached", + "backing_device": "/dev/sdb", "mount_path": "/mnt/hdd_1", "role": "user-data", + "backup_target": true, "guest_path": "/mnt/felhom-drives/hdd_1", + "bound_under_parent": true, "durable_id": "uuid:47a3361a-91e0-4831-a69d-27f540ed3f48" } +``` + +Non-empty, plausible, four rows. **Part 5's failure mode (0 rows on a present drive) is excluded.** + +--- + +## 4. Where the absent state was staged, and why there + +The demo boxes' targets are fenced, and `drill-r50` is the drift fixture. **No new box was built.** The +absent state was staged on the **existing nested-Proxmox drill VM on DooPlex** +(`/mnt/5_hdd/felhom.eu/drill/drill.qcow2`, PVE 9.2, snapshot `virgin`, the golden-bake fixture) — my own +host, zero production exposure, and it can hot-unplug a disk for a **genuine device loss** rather than a +simulated one. + +**Fidelity — what was real:** + +- the **byte-identical live agent binary**: `scp` from felhom-pve `/usr/local/bin/felhom-agent`, + `sha256 f48544ad5c4242a719c080ff83d7a6bc759e14272987416eb26c9683386803b7`, `--version` 0.115.0. +- the real `configs/felhom-backup-target-apply` (E-2a) and `felhom-mkfs-guarded` wrappers. +- a real LXC guest (9201), in the `felhom` pool, with the mp8 parent bind created **exactly** as + `provision.BackHalf` does it (`backhalf.go:200-206`: `pct set 9201 -mp8 /mnt/felhom-drives,mp=/mnt/felhom-drives`). +- **every state-producing step through the real endpoints**, not by hand: + `POST /disks/format` → `POST /disks/assign` → `POST /disks/guest-attach` → `POST /backup/target`. + That produced the `.mount` unit, `drive-intents.json: enrolled`, the guest bind, the + `dir: felhom-backup / path /mnt/cel / is_mountpoint 1` storage and the `local_backup_target` rewrite. +- device loss by QMP `device_del` on the virtio-blk frontend — the device node vanishes. + +**What was NOT production-shaped, stated plainly:** (a) the agent ran as root with +`privileged.mode: "direct"` and a `root@pam` PVE token instead of non-root + sudoers + `felhom-agent@pve` +— this changes *who* runs the privileged ops, not what `/disks` computes; (b) `hub.url` pointed at a +dead local port, so nothing was reported to the hub; (c) **the local-API bearer token record was +hand-written** into `local-tokens.log` (one `{"v":9201,"h":""}` line) because minting is only +reachable through `--selftest=provision`, which needs a golden and a real customer. That is the *key to +the door*, not the state under measurement — every drive/storage/target fact was produced by the real +endpoints. (d) No controller ran, so §7 correlates against the two prior live runs rather than a fresh one. + +**The rig validated itself against production before being trusted:** its present-state target row is +field-for-field identical in shape to felhom-pve's control run (`role: user-data`, +`backup_target: true`, `guest_path` set, `bound_under_parent: true`, `durable_id: uuid:…`) — only the +device node and mountpoint names differ. + +--- + +## 5. Host state at each capture + +| | PRESENT (05:53:49Z) | ABSENT (05:54:32Z) | RETURNED (05:55:45Z) | +|---|---|---|---| +| device node | `/dev/vdb` | **gone** (`ls /dev/vdb` → No such file) | `/dev/vdc` | +| `findmnt /mnt/cel` | `/dev/vdb ext4` | **not mounted** (rc=1) | `/dev/vdc ext4` | +| guest `findmnt /mnt/felhom-drives/cel` | `/dev/vdb[/felhom-data]` | `/dev/vdb[/felhom-data] …,shutdown` | `/dev/vdb[/felhom-data] …,shutdown` | +| `pvesm status` | `felhom-backup dir active` | `felhom-backup dir inactive 0 0 0` | `felhom-backup dir active` | + +The detached state reproduced PVE's exact Part-5 string: + +``` +unable to activate storage 'felhom-backup' - directory is expected to be a mount point +but is not mounted: '/mnt/cel' +``` + +and the agent's exact Session-C / E-2d lines: + +``` +07:54:08 WARN storage: watchdog detected target state change from=attached to=disconnected +07:54:09 INFO storage: enrolled drive absent by UUID — not re-asserting (will retry when it enumerates) +07:54:09 WARN reconcile: enrolled drive not present (durable-id absent) — skipping +``` + +The R-113 asymmetry is present as measured before: **raw `/mnt/cel` NOT mounted while the guest's bind +still names `/dev/vdb[/felhom-data]`.** + +--- + +## 6. The raw payloads + +### 6.1 PRESENT — `GET /disks`, HTTP 200, 1824 bytes, **3 rows** + +```json +{"ok":true,"data":{"disks":[ +{"name":"felhom-backup","type":"local-dir","state":"attached","backing_device":"/dev/vdb","mount_path":"/mnt/cel","class":"slow","role":"user-data","data_bearing":true,"data_reason":"device is mounted","total_bytes":4143677440,"used_bytes":1081344,"used_fraction":0.000260962397690878,"durable_id":"uuid:694034cc-eee6-4b17-901f-708f50df9b09","wipe_durable_id":"byid:virtio-R116CEL","guest_attached":false,"backup_target":true,"guest_path":"/mnt/felhom-drives/cel","bound_under_parent":true,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}}, +{"name":"local-lvm","type":"lvmthin","state":"attached","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":81533075456,"used_bytes":807177447,"used_fraction":0.00989999999982339,"durable_id":"pve/data","guest_attached":false,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}}, +{"name":"local","type":"local","state":"attached","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":49675956224,"used_bytes":4584443904,"used_fraction":0.092286978499774,"durable_id":"path:/var/lib/vz","guest_attached":false,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}} +],"guest_boot_id":"1785390256-46753","vmid":9201}} +``` + +### 6.2 ABSENT — `GET /disks`, HTTP 200, 2040 bytes, **4 rows** (one MORE than present) + +```json +{"ok":true,"data":{"disks":[ +{"name":"felhom-backup","type":"local-dir","state":"disconnected","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":0,"used_bytes":0,"used_fraction":0,"durable_id":"path:/mnt/cel","guest_attached":false,"backup_target":true,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}}, +{"name":"local-lvm","type":"lvmthin","state":"attached","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":81533075456,"used_bytes":807177447,"used_fraction":0.00989999999982339,"durable_id":"pve/data","guest_attached":false,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}}, +{"name":"local","type":"local","state":"attached","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":49675956224,"used_bytes":4584579072,"used_fraction":0.0922896994942001,"durable_id":"path:/var/lib/vz","guest_attached":false,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}}, +{"name":"694034cc-eee6-4b17-901f-708f50df9b09","type":"usb","state":"attached","backing_device":"","mount_path":"/mnt/cel","class":"","role":"user-data","data_bearing":false,"total_bytes":49675956224,"used_bytes":4584579072,"used_fraction":0.09228969949420011,"durable_id":"uuid:694034cc-eee6-4b17-901f-708f50df9b09","guest_attached":false,"guest_path":"/mnt/felhom-drives/cel","bound_under_parent":false} +],"guest_boot_id":"1785390256-46753","vmid":9201}} +``` + +### 6.3 RETURNED — `GET /disks`, HTTP 200, 1825 bytes, **3 rows** (back to one row for the drive) + +```json +{"ok":true,"data":{"disks":[ +{"name":"felhom-backup","type":"local-dir","state":"attached","backing_device":"/dev/vdc","mount_path":"/mnt/cel","class":"slow","role":"user-data","data_bearing":true,"data_reason":"device is mounted","total_bytes":4143677440,"used_bytes":1081344,"used_fraction":0.000260962397690878,"durable_id":"uuid:694034cc-eee6-4b17-901f-708f50df9b09","wipe_durable_id":"byid:virtio-R116CEL","guest_attached":false,"backup_target":true,"guest_path":"/mnt/felhom-drives/cel","bound_under_parent":true,"smart":{"health":"UNKNOWN", …}}, +{"name":"local-lvm", … as above … }, +{"name":"local", … as above … } +],"guest_boot_id":"1785390256-46753","vmid":9201}} +``` + +*(the `local-lvm` / `local` rows in 6.3 are byte-identical to 6.1 apart from `used_bytes`; elided only +there, and nowhere else in this document.)* + +--- + +## 7. The per-row table for the backup-target drive + +**PAYLOAD.** Every cell read from the JSON above. + +### PRESENT — 1 row mentions the drive + +| Name | Type | State | Role | MountPath | GuestPath | BackingDevice | BackupTarget | BoundUnderParent | +|---|---|---|---|---|---|---|---|---| +| `felhom-backup` | `local-dir` | `attached` | `user-data` | `/mnt/cel` | `/mnt/felhom-drives/cel` | `/dev/vdb` | **`true`** | `true` | + +### ABSENT — **2 rows** mention the drive, and the two facts are on DIFFERENT rows + +| Name | Type | State | Role | MountPath | GuestPath | BackingDevice | BackupTarget | BoundUnderParent | Source | +|---|---|---|---|---|---|---|---|---|---| +| `felhom-backup` | `local-dir` | `disconnected` | **`system`** | **`""`** | **`""`** (field absent) | `""` | **`true`** | `false` | Observe loop, `disks.go:196-284` | +| `694034cc-…9b09` | `usb` | **`attached`** (hardcoded `:302`) | `user-data` (hardcoded `:304`) | **`/mnt/cel`** | **`/mnt/felhom-drives/cel`** | `""` | **field ABSENT ⇒ `false`** | `false` | union loop, `disks.go:297-339` | + +`durable_id` also degrades on the Observe row: `uuid:694034cc…` → **`path:/mnt/cel`**. + +### RETURNED — 1 row again + +| Name | Type | State | Role | MountPath | GuestPath | BackingDevice | BackupTarget | BoundUnderParent | +|---|---|---|---|---|---|---|---|---| +| `felhom-backup` | `local-dir` | `attached` | `user-data` | `/mnt/cel` | `/mnt/felhom-drives/cel` | **`/dev/vdc`** | **`true`** | `true` | + +--- + +## 8. The mechanism + +**Which row the lookup lands on, which key it uses, what that row says.** + +The controller's key is `a.Path` — the **registered `StoragePath`**, which for an external drive is the +GUEST path `/mnt/felhom-drives/cel` (`intermediary.go:297`, `:318`). `driveTargetByPath` +(`intermediary.go:602-618`) builds the map with **plain assignment**, one entry per non-empty +`GuestPath` (`:612`) and per non-empty `MountPath` (`:615`). + +**PAYLOAD — the map, computed by applying `intermediary.go:602-618` verbatim to the captured JSON:** + +``` +PRESENT rows=3 isTarget["/mnt/felhom-drives/cel"] = True → SPECIFIC + map: {'/mnt/felhom-drives/cel': True, '/mnt/cel': True} +ABSENT rows=4 isTarget["/mnt/felhom-drives/cel"] = False → GENERIC + map: {'/mnt/felhom-drives/cel': False, '/mnt/cel': False} +RETURNED rows=3 isTarget["/mnt/felhom-drives/cel"] = True → SPECIFIC + map: {'/mnt/felhom-drives/cel': True, '/mnt/cel': True} +``` + +That is the live asymmetry — generic on detach, specific on return — **reproduced from the payload +alone**, matching Session C and Part 5 exactly. + +**Why it fails, step by step. Steps 1–4 are payload; step 5 is the map above.** + +1. The device goes. `exactMountDevice` fails, so `backingDevice` and `mountPath` are both `""` + (`observe.go:184-190`). **Payload: Observe row has `backing_device:""`, `mount_path:""`.** +2. `RoleForStorage` therefore returns `RoleSystem` (`role.go:180-181`), so the `GuestPath` block at + `disks.go:215-217` is skipped. **Payload: `role:"system"`, no `guest_path`.** +3. v0.115.0's fallback (`disks.go:258-260`) fires — its three gates all pass — but it calls + `StablePathForRaw(t.MountPath)` with `MountPath == ""`, which returns `""` + (`intermediary.go:69-75`). **It assigns an empty string. Payload confirms: no `guest_path` on that + row.** This is exactly why v0.115.0 is inert. +4. `seen` is built only from rows whose `MountPath != ""` (`disks.go:290-295`). The Observe row's is + empty, so **`/mnt/cel` is not in `seen`**, so the union row is **not** deduped at `:298`. It is + emitted with `MountPath` from the `.mount` unit file and `GuestPath` from `:309-310` — and + **`BackupTarget` is not in its struct literal (`:301-306`), so it serialises as `false`.** + **Payload: row 4 exists, carries both paths, and `backup_target` is absent.** +5. So the row holding `backup_target:true` contributes **no key at all** (both its keys are empty + strings and are skipped at `:611` and `:614`), while the row that owns the key + `/mnt/felhom-drives/cel` says `false`. `isTarget[a.Path]` → `false` → `notifyDriveAbsent` takes the + else branch (`intermediary.go:645`) → **generic `storage_disconnected`**. +6. On return the two rows re-merge into one (the Observe row regains `MountPath`, so the union row is + deduped again) — that single row carries `backup_target:true` **and** the guest path, so + `notifyDriveReturned` takes the specific branch (`:656`) → **`backup_target_restored`**. Hence the + unmatchable pair. + +**INFERENCE (small, and labelled):** step 5's map-construction and step 6's branch selection were +computed by applying the controller's published source to the captured payload, not observed from a +running controller this session. The observable they predict — generic out, specific back — is exactly +what Session C and Part 5 *did* measure live on real boxes, twice. So the prediction is corroborated, +but the controller was not itself re-run here. + +**There is no map-overwrite race.** An earlier worry was that two rows might both write the same key +with the last one winning. They do not: the Observe row contributes nothing, so the union row is the +sole writer of `/mnt/felhom-drives/cel`. Row order is irrelevant. + +--- + +## 9. Which theory this is + +| Theory | Verdict from payload | +|---|---| +| **#1 — "the union row writes `false`"** (raised, then declared wrong and retracted) | ✅ **CORRECT.** The union row is emitted (`rows` 3 → 4), owns the key, and its `backup_target` field is absent ⇒ `false`. **The retraction was the error, not the theory.** | +| **#2 — "no row carries the guest path; the union row is deduped away"** (the basis of the shipped v0.115.0 fix) | ❌ **FALSE on both halves.** A row *does* carry `guest_path:"/mnt/felhom-drives/cel"` (row 4), and the union row is *not* deduped — precisely *because* `MountPath` does not survive on the Observe row, which is the fact #2 was built on. #2 had the premise right and drew the opposite conclusion from it. | +| **#3 — "`isTarget["/mnt/"]` is already `true` in the absent state, so the controller could fall back to it"** | ❌ **FALSE.** `isTarget["/mnt/cel"] = false` as well — the same union row writes both keys. A fallback to the raw path changes nothing. | + +So: **#1**, and it was available all along. The one genuinely new fact is *why* the dedup does not fire +— `seen` is keyed on the very field the absent state empties (`disks.go:290-295` vs `:298`), which makes +the union row's emission a **consequence** of the vanished device rather than an independent condition. + +--- + +## 10. What a fix would have to change — constraints for the next session's spec + +**Not implemented here. Stated as constraints, not as a design.** + +1. **The necessary and sufficient condition is that the row owning the key + `/mnt/felhom-drives/` must report `backup_target: true` when that drive is the primary + target.** Everything else is a means. Whether that is achieved by carrying the flag onto the union + row, by merging the two rows, or by making `driveTargetByPath` OR rather than assign, is the spec's + choice — but it must be verified against a payload, not a fixture. +2. **`disks.go:301-306` omits `BackupTarget` — and the omission is invisible.** A struct literal that + silently defaults a bool is the whole defect. Whatever the fix, this literal is the site. Note the + union loop has no access to `t` (there is no Observe target for it) — it would have to compare + `d.Name`/`d.MountPath` against `primaryTargetID`/the target's path, and **`d.Name` for a union row is + the drive's UUID, not the storage id** (payload: `"694034cc-…"` vs `"felhom-backup"`), so a naive + `d.Name == primaryTargetID` comparison will silently never match. That trap is worth stating in the + spec explicitly. +3. **Fixing only the agent may be enough, but the controller half is worth a look in the same pass.** + `driveTargetByPath` uses assignment, so it is order-dependent the moment two rows ever carry the + same key with different values — which is exactly the state a partial fix would create. An OR (or a + documented single-writer invariant with a test) removes the hazard. +4. **v0.115.0's fallback should not simply be extended.** It is gated on `t.BackingDevice == ""` and + feeds `StablePathForRaw(t.MountPath)`; in the failing state `MountPath` is also `""`, so there is no + value on that row to derive a guest path *from*. Any fix on the Observe row needs a path source that + survives device loss — and the only one that does is the same `.mount` unit the union row already + reads. That argues for fixing the union row, not the Observe row. +5. **Do not let the fix make an absent drive read as present.** `planDriveGates` computes + `present[gp] = present[gp] || d.BoundUnderParent` (`intermediary.go:226`). The union row's + `BoundUnderParent` is `false` in the payload (R-113's conjunction holds), and it must stay false. The + existing `TestAbsentTargetRowDoesNotRegisterPresence` pins this for the Observe row; the union row + needs the equivalent. +6. **The red-proof must be the payload shape, not the Session-C fixture.** v0.115.0 was unit-tested and + red-proofed against a fixture that supplied a `MountPath` and a `disconnected` state — a shape that + does not occur. The fixture for the next fix is §6.2 of this document: **4 rows, the target's flag + and its guest path on different rows, `state:"attached"` on the union row.** +7. **Whether to keep v0.115.0 is now answerable.** It is provably inert (it assigns `""`). It is not + harmful. Reverting it is a separate change with its own risk; extending it is ruled out by (4). + Recommendation, for the spec to accept or reject: **leave it, and note in its comment that the case + it targets cannot arise** — or delete it in the same commit that fixes the union row, so the tree + does not carry two explanations of one defect. + +--- + +## 11. R-117 (new) — the returned drive's guest bind is a DEAD MOUNT, and everything reports healthy + +The brief's §9 "stale bind, unchased after three runs". **It is not cosmetic.** Captured this run, and +it is worse than a wrong device name. + +**PAYLOAD + live probe, RETURNED state (drive back, host healthy on `/dev/vdc`):** + +``` +host findmnt -no SOURCE /mnt/cel → /dev/vdc +guest findmnt -no SOURCE,OPTIONS /mnt/felhom-drives/cel → /dev/vdb[/felhom-data] rw,relatime,shutdown +guest ls /mnt/felhom-drives/cel → ls: reading directory …: Input/output error +guest write into it → cannot create …: Input/output error (GUEST-WRITE-FAIL) +host ls /mnt/cel/felhom-data/ → fine, empty, live +/disks says: state "attached", role "user-data", bound_under_parent TRUE, backup_target true +``` + +**The consequence, and this is why it outranks R-116.** `planDriveGates` computes +`present["/mnt/felhom-drives/cel"] = … || d.BoundUnderParent` (`intermediary.go:226`), which is `true`. +The drive is currently `Disconnected`, so the gate takes the **`Return`** branch +(`intermediary.go:281`, `:301-320`): it clears the disconnected flag and **restarts the gate-stopped +apps onto a mount that returns `EIO` for every operation** — then reports the drive healthy and emails +`backup_target_restored`. A customer's apps come back up pointed at a dead namespace with no alarm on +any channel. + +R-113's conjunction cannot catch it: half 1 (guest mountinfo has an entry at the guest path) is +satisfied by the *stale* entry, and half 2 (`devicePresent("/mnt/cel")`) is satisfied by the *new* +device. **Neither half compares the two.** `disks.go:158-165`'s comment claims only that the conjunction +fixes the boot-ordering and outlived-bind cases — it does not claim this, so this is a gap rather than a +false invariant. + +Reproduced deliberately and cleanly here; consistent with the same asymmetry seen (uninvestigated) in +E-2d, Session C and Part 5. **Filed, not chased, per the brief.** + +--- + +## 12. R-118 (new) — the absent drive's union row reports the ROOT filesystem's size as its own + +**PAYLOAD, §6.2 row 4:** `total_bytes: 49675956224`, `used_bytes: 4584579072` — **byte-identical to the +`local` row** (`durable_id: path:/var/lib/vz`, i.e. `pve-root`) in the same response. The real drive is +**4 GB** (`4143677440` in §6.1). + +Cause, from source: `statfsCapacity(d.MountPath)` at `disks.go:335-338` statfs's `/mnt/cel`, which with +the device gone is a **bare directory on the root filesystem**. `observe.go:176-183`'s comment warns +about exactly this trap and guards the Observe path against it ("*an unmounted removable dir-storage's +mountpoint reverts to a bare directory on root … would be a catastrophic DR mis-id*"); **the union path +at `:335` has no equivalent guard.** + +Not a DR mis-id — `durable_id` on that row is still the correct `uuid:694034cc…`, so the re-attach +identity is safe. It is a **false capacity**: a detached 4 GB drive advertises 46 GiB with 9.2 % used, to +any consumer of `total_bytes`/`used_fraction` (fill monitors, the storage cards). Same class as the +`role.go:180-181` flip that drives R-116 — an absent drive's fields decaying to the root filesystem's. + +--- + +## 13. Teardown + +| | Before | After | +|---|---|---| +| drill VM `pvesm status` | `felhom-backup` active, `local-lvm` 0.99 %, `local` 9.23 % | `felhom-backup` **removed**, `local-lvm` **0.00 %**, `local` 9.23 % | +| drill VM `pct list` | `9201 running` | **empty** | + +- guest 9201 `pct destroy --purge` (logical volume removed, config purged); `pvesm remove felhom-backup`; + `mnt-cel.mount` disabled + deleted + `daemon-reload`; `shred -u` on the local-API token file and the + PVE token file. +- `poweroff`, qemu exited (`0` procs), **`qemu-img snapshot -a virgin drill.qcow2`** → the golden-bake + fixture is exactly as found (`virgin`, ID 1, 2026-07-03 14:12:15). +- scratch artifacts deleted: `r116-cel.qcow2`, `qmp.sock`, `qemu.pid`, both console `.ppm` dumps. +- DooPlex `df -h /mnt/5_hdd`: **2.1T used / 6.6T avail, 24 %** — same as at session start. +- The disposable `root@pam!r116` PVE token and the drill local-API token died with the snapshot revert; + neither is recorded in any committed file. + +**Fences — re-verified after teardown, not assumed:** + +| Fence | State | +|---|---| +| demo-hp `local-lvm` | **38.83 %** — identical before, after, and to Part 5's reading | +| demo-hp `drill-r50` (VM 300) | `stopped` — untouched | +| demo-hp guest 9201 | `running` | +| felhom-pve `felhom-backup` | `active`, `/mnt/hdd_1` on `/dev/sdb` — **target not re-pointed** | +| felhom-pve guest 9201 | `running` | +| Both demo boxes | **read-only access only** — `GET /disks`, file reads, `pvesm status`, `pct list`, one `scp` of the agent binary | +| Code | **no `.go` file edited, nothing built, nothing published, v0.115.0 untouched** | + +### `sess-d`, and a leftover the prior runs did not record + +`sess-d-0452c4` now reads **STALE**, not ONLINE. The delete gate refuses only on +`s.hostStatus(...) == "ok"` (`hub/internal/web/customer_delete.go:220-228`) — ONLINE — so **`sess-d` is +now deletable.** Not executed here: deleting a customer runs external teardown plus a DB purge, and the +brief scoped this session to recording the command. + +``` +POST /configs/sess-d/delete ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-d expect_hosts=1 +``` + +**Also still present, and not mentioned in the Session-C audit: `sess-c` / host `sess-c-8158e4` +(agent 0.114.0, DOWN).** Same shape, same gate, deletable on the same terms — +`confirm_id=sess-c expect_hosts=1`. Two drill customers are now accumulating in the hub; that is the +operator's cleanup call, not a register row. + +--- + +## 14. What did not happen, and why + +- **No controller was run, so no fresh gate-log / hub-event correlation was captured** (brief §6.4). + Deliberate: that observable was already captured live and identically **twice** (Session C, Part 5), + and the *payload* was the only missing input. Staging a controller meant a hub customer, a pairing, a + golden fetch and a claim — the work that consumed the three prior sessions. §8 correlates the payload + to those existing measurements instead, and labels that as the inference it is. +- **No new drill box was built** — the brief's stated preference. The existing DooPlex nested-PVE + fixture was reused and restored. +- **R-117 and R-118 were filed, not investigated.** R-117 in particular deserves its own run; on the + evidence here it is more customer-affecting than R-116. +- **v0.115.0 was neither reverted nor extended**, per the fence. +- **Two boot attempts of the drill VM failed before the run started** and are recorded because the cause + is a reusable trap: adding `-device` entries at launch shifts PCI slot assignment, which renames the + guest NIC (`ens3` → gone), so vmbr0 loses its port and SSH is unreachable while the console shows a + healthy PVE login. `bootindex` fixes the boot-order half but not the NIC half. **The fix is to boot + the memory's known-good command line unchanged and hot-plug the extra disk over QMP afterwards** — + which is also the more faithful device-loss mechanism. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 6fc3dd1..d57048b 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -16,7 +16,9 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **R-110** | **`main` is the installer's publish channel — there is no staging.** `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a 30 s period and nginx serves that working tree directly (`location /scripts/`, `root …/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as **root on a virgin box**, the single most privileged thing Felhom ships | **WAITING-ON-OPERATOR (S)** | operator ruling | **Two consequences worth stating:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" **was never available to take**. **Open question for the operator, not a defect to fix blind:** whether `/scripts/` should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether `main`-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now. **SECOND INSTANCE, found 2026-07-29 by the E-2d run and filed here rather than as a new ID:** `felhom-host-install.sh` fetches **nine** files from `raw/branch/main` (`:2072`–`:2206`) and the hub manifest vouches a sha for exactly **one** (`wrapper_sha256` → `felhom-pbs-apply`; re-checked this run, no drift). E-2a's `felhom-backup-target-apply` (`:2116`) is installed **0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n` — a root-executed artifact taken from `main` with no pinned integrity, which is this row's class exactly | CC | | **R-111** | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`.**~~ `felhom-host-install.sh` does not use `main`: it reads the hub-vouched manifest (`:423-436`) and fetches Gitea generic packages (agent `:1945`, golden `:2573`). Gitea's newest are **agent 0.96.0** and **golden 0.161.0**, and the hub's manifest selects exactly those — so a fresh box lands on **agent 0.96.0 + controller 0.161.0** (global floor `v0.156.0` < the golden's 0.161.0, so no self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and was never published** | **SHIPPED 2026-07-29 — the channel now serves agent 0.113.0 + golden 0.185.1** | — | **FIXED the same day it was found.** Agent **0.113.0** built from the clean tree @ `58b598b` and published (`scripts/publish-agent.sh`), sha `5f3247f756cb658e…`, round-trip GET verified. Golden **0.185.1** baked on the nested drill VM embedding controller `0.185.1`, published, sha `dba00f3e845c415e…` — bake clean: `Result=success`, overlay2, **all 3 mounts included** (rootfs+mp0+mp1), 0 FATAL/exclusions, upload HTTP 201, token-leak grep 0; log `drill/bake-0.185.1.log`; GL-1 teardown done (guest 9100 purged, secrets shredded, disk restored to `virgin`). Hub Day-0 manifest moved **both together in one POST** so it never vouched a new agent against an old golden; `min_agent` **0.93.0 → 0.113.0**, which is what controller v0.185.0 declares (`felhom-controller/CHANGELOG.md:15`) — **zero fleet impact, verified: all three enrolled hosts already run agent 0.113.0, so no box is held.** `wrapper_sha256` preserved verbatim (re-checked against `configs/felhom-pbs-apply` — no drift). **The global controller floor was deliberately NOT raised**: the golden now bakes 0.185.1, so a fresh box needs no self-update, and raising it would have been an unnecessary fleet-wide write. Original finding follows. **Found 2026-07-29 by the E-2d Phase 0 gate, which stopped the run before a VM was created.** 17 unpublished releases (v0.97.0–v0.113.0) strand the **entire R-82 tiered-backup arc** plus **F-CRIT-2** (a failed backup looking fresh — 7 days silent) and **F-REBOOT** (a guest rebooted mid-backup never returns): a new customer's box would install without them. **Blocks E-2d's C3/C4/C5** — those test endpoints and events that do not exist in 0.96.0/0.161.0. The **controller is fine** (registry has 0.185.1, floor-driven self-update), so the gap is specific to the two Gitea-generic artifacts. **Mirror of R-110, not a duplicate:** R-110 = the installer publishes instantly with no staging; R-111 = the agent/golden publish gate exists and was never walked. Fix should decide whether publishing joins the release train rather than staying a remembered step (R-29's shape, one layer up). Evidence: `audits/E2D-fresh-vm-2026-07-29.md` **DEFERRED LEG, AND IT RECURRED → R-115.** This row's shipped half stands and is not reopened: the bump happened, was verified, and was proven end-to-end by the E-2d install. But its own closing line — *decide whether publishing joins the release train rather than staying a remembered step* — was never acted on, and agent 0.114.0 reproduced the exact condition the same afternoon. The recurrence is filed as **R-115**, not as a reopen, because the stale-channel finding is closed while the process defect that caused it is a distinct problem with a distinct fix. | CC | | **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **WAITING-ON-OPERATOR (M)** | operator ruling on the release process | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design | CC | -| **R-116** | **The drive-absent alarm and its recovery are a MISMATCHED PAIR — absent fires the GENERIC `storage_disconnected`, return fires the SPECIFIC `backup_target_restored`.** `backup_target_absent` never fires at all (count 0 across a full Session-C run). An operator gets an alarm they cannot match to its recovery, which is exactly what `notifyDriveReturned`'s own comment forbids | **STILL OPEN — fix shipped (agent v0.115.0) and INEFFECTIVE; mechanism NOT isolated** | a working `/disks` read | **PART 5 RE-VALIDATION FAILED 2026-07-30 (`audits/R116-PART5-2026-07-30.md`).** A fresh box running the FULLY SHIPPED stack — agent **0.115.0 from the Day-0 manifest** + controller 0.185.1 from the vouched golden — still fired the **generic** `storage_disconnected` on detach and the **specific** `backup_target_restored` on return. `backup_target_absent` count 0. Identical to Session C. **Discrimination FAILS** (the target itself produced the generic event, so the non-target leg was pointless to stage — reported as a fail, not as Session C's trivial pass). **Over-correction guard PASSES** (0 ABSENT lines with the drive present). **The v0.115.0 fix targets a shape that does not occur live, and WHICH shape does occur is NOT ISOLATED.** With the drive detached PVE reports the storage `inactive` with zeroed fields — a shape the unit fixture did not model. Three attempts to read the real `/disks` payload failed on token extraction, and **a present-drive control also returned 0 rows, proving the QUERY was broken, not the payload** — without that control this would have been a third false mechanism, after "the union row writes false" (wrong, corrected) and "no row carries the guest path" (unverified). **NEXT SESSION'S FIRST JOB: a working `/disks` read with a present-drive control run FIRST, before any code.** v0.115.0 is published, vouched and **inert**; not reverted (reverting is itself a change, and the runbook forbids fixing mid-run) | CC | +| **R-116** | **The drive-absent alarm and its recovery are a MISMATCHED PAIR — absent fires the GENERIC `storage_disconnected`, return fires the SPECIFIC `backup_target_restored`.** `backup_target_absent` never fires at all (count 0 across a full Session-C run). An operator gets an alarm they cannot match to its recovery, which is exactly what `notifyDriveReturned`'s own comment forbids | **OPEN — MECHANISM ISOLATED 2026-07-30, fix NOT yet written** | nothing — READY (S) | **MECHANISM ISOLATED FROM THE REAL PAYLOAD (`audits/DIAG-r116-disks-payload-2026-07-30.md`).** The `/disks` read path is solved and written down verbatim: the token's plaintext lives ONLY in `/var/lib/felhom-agent/guests//bootstrap/bootstrap.json` **on the Proxmox host** (the agent's token store keeps hashes only, `tokenstore.go:26-32` — which is what defeated three prior extraction attempts); curl the island endpoint from the host itself. **Present-drive control run PASSED first** (felhom-pve, HTTP 200, 4 rows) so a 0-row query could not be mistaken for an empty payload. Absent state then staged on the existing DooPlex nested-PVE drill fixture with the **byte-identical live 0.115.0 binary** (`sha256 f48544ad…`) and every state-producing step through the real endpoints (format→assign→guest-attach→backup/target); device loss by QMP hot-unplug. **THE ANSWER IS THEORY #1 — "the registry-union row writes `false`" — the theory that was raised, declared wrong and RETRACTED. The retraction was the error.** In the absent state `/disks` returns **4 rows, not 3**: the drive appears TWICE and the two facts the controller needs sit on DIFFERENT rows. The Observe row (`felhom-backup`) has `backup_target:true` but `mount_path:""` **and** `guest_path:""` (role flipped to `system`; v0.115.0's fallback calls `StablePathForRaw("")` which returns `""` — `intermediary.go:69-75` — so **it assigns nothing, which is exactly why it is inert**), so it contributes NO key to `driveTargetByPath` at all. The registry-union row (`disks.go:297-339`) is **NOT deduped** — `seen` is keyed on `MountPath` (`:290-295`) and the Observe row's is now empty — and it owns both keys while **`BackupTarget` is absent from its struct literal (`:301-306`) ⇒ `false`**. Its `MountPath` survives the device because the union source is the **systemd `.mount` unit FILE** (`registry_known.go:40-75` via `main.go:605`→`:764`), which never consults the mount table. So `isTarget["/mnt/felhom-drives/cel"] = false` → generic; on return the two rows re-merge into one carrying both facts → specific. Verified by applying `intermediary.go:602-618` to the captured payload: PRESENT `True` / ABSENT `False` / RETURNED `True` — **the live asymmetry reproduced from payload alone.** **Theory #2 (the basis of v0.115.0) is FALSE on both halves** — a row DOES carry the guest path, and the union row is not deduped *because of* the very premise #2 was built on; **project-Claude's #3 is FALSE too** (`isTarget["/mnt/cel"]` is `false` as well — same row writes both keys). **FIX CONSTRAINTS, not a design** (see §10 of the diag): the necessary+sufficient condition is that the row owning `/mnt/felhom-drives/` reports `backup_target:true`; the site is the union struct literal; **trap — `d.Name` on a union row is the drive UUID, not the storage id, so a naive `d.Name == primaryTargetID` silently never matches**; do NOT extend v0.115.0 (there is no path value on that row to derive from — the unit file is the only source that survives, which argues for fixing the union row); keep `BoundUnderParent` false there so the gate cannot read absent as present; and **red-proof against §6.2's real shape, not Session C's fixture** (v0.115.0 was red-proofed against a shape that does not occur). v0.115.0 is provably INERT and harmless — keep-or-delete is the fix spec's call. **Original Part-5 record follows.** **PART 5 RE-VALIDATION FAILED 2026-07-30 (`audits/R116-PART5-2026-07-30.md`).** A fresh box running the FULLY SHIPPED stack — agent **0.115.0 from the Day-0 manifest** + controller 0.185.1 from the vouched golden — still fired the **generic** `storage_disconnected` on detach and the **specific** `backup_target_restored` on return. `backup_target_absent` count 0. Identical to Session C. **Discrimination FAILS** (the target itself produced the generic event, so the non-target leg was pointless to stage — reported as a fail, not as Session C's trivial pass). **Over-correction guard PASSES** (0 ABSENT lines with the drive present). **The v0.115.0 fix targets a shape that does not occur live, and WHICH shape does occur is NOT ISOLATED.** With the drive detached PVE reports the storage `inactive` with zeroed fields — a shape the unit fixture did not model. Three attempts to read the real `/disks` payload failed on token extraction, and **a present-drive control also returned 0 rows, proving the QUERY was broken, not the payload** — without that control this would have been a third false mechanism, after "the union row writes false" (wrong, corrected) and "no row carries the guest path" (unverified). **NEXT SESSION'S FIRST JOB: a working `/disks` read with a present-drive control run FIRST, before any code.** v0.115.0 is published, vouched and **inert**; not reverted (reverting is itself a change, and the runbook forbids fixing mid-run) | CC | +| **R-117** | **A returned drive's guest bind is a DEAD MOUNT, and the gate restarts the customer's apps onto it while every signal reads healthy.** After a detach/reattach the host raw mount is fine on the NEW device node, but the guest's bind still names the OLD one and every operation through it returns `EIO`: host `findmnt /mnt/cel` → `/dev/vdc`; guest `findmnt /mnt/felhom-drives/cel` → `/dev/vdb[/felhom-data] …,shutdown`; guest `ls` and write → **`Input/output error`**; and `/disks` reports `state:"attached"`, `role:"user-data"`, `bound_under_parent:TRUE`, `backup_target:true` | **READY (M) — NEW 2026-07-30, deliberately not chased** | — | **Worse than R-116 and it outranks it.** `planDriveGates` computes `present[gp] = … \|\| d.BoundUnderParent` (`intermediary.go:226`) = `true`, and the drive is `Disconnected`, so the gate takes the **`Return`** branch (`intermediary.go:281`, `:301-320`): it clears the flag, **restarts the gate-stopped apps onto a namespace that `EIO`s on every call**, reports the drive healthy and emails `backup_target_restored`. **No alarm on any channel.** R-113's conjunction cannot catch it — half 1 (guest mountinfo has an entry at the guest path) is satisfied by the STALE entry and half 2 (`devicePresent`) by the NEW device; **neither half compares the two**. Not a false invariant: `disks.go:158-165` claims only the boot-ordering and outlived-bind cases, so this is a genuine gap. **Seen but uninvestigated in E-2d, Session C and Part 5 (three consecutive runs) as "the drive returned as /dev/sdc while findmnt still read /dev/sdb"** — that framing read as cosmetic; it is not. Reproduced deliberately and probed to EIO in `audits/DIAG-r116-disks-payload-2026-07-30.md` §11. Deserves its own run | CC | +| **R-118** | **An absent drive's union row advertises the ROOT filesystem's capacity as its own.** In the absent-state payload the registry-union row reports `total_bytes: 49675956224 / used_bytes: 4584579072` — **byte-identical to the `local` row** (`durable_id: path:/var/lib/vz`, i.e. `pve-root`) in the same response. The real drive is **4 GB** | **READY (XS) — NEW 2026-07-30** | — | Cause: `statfsCapacity(d.MountPath)` (`disks.go:335-338`) statfs's `/mnt/cel`, which with the device gone is a **bare directory on the root filesystem**. `observe.go:176-183`'s comment warns about exactly this trap and guards the Observe path ("*an unmounted removable dir-storage's mountpoint reverts to a bare directory on root … catastrophic DR mis-id*"); **the union path has no equivalent guard.** **Not a DR mis-id** — `durable_id` on that row is still the correct `uuid:…`, so re-attach identity is safe. It is a **false capacity** reaching every consumer of `total_bytes`/`used_fraction` (fill monitors, storage cards): a detached 4 GB drive advertises 46 GiB at 9.2 % used. Same class as `role.go:180-181` — an absent drive's fields decaying to the root filesystem's. Evidence: `audits/DIAG-r116-disks-payload-2026-07-30.md` §12 | CC | | **R-113** | ~~**The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.**~~ `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc//mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **SHIPPED + PROVEN-LIVE** (agent v0.114.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** Target drive hot-detached on a fresh box running the SHIPPED agent 0.114.0 (from the Day-0 manifest, not a hand build). **The gate fired in 4 seconds** — E-2d measured ZERO over 4½ minutes — and `SetDisconnected` was reached. It fired on exactly the shape that defeated it: raw `/mnt/mentes` NOT mounted while the bind `/mnt/felhom-drives/mentes` still read `/dev/sdb[/felhom-data]`. **Over-correction guard PASSED:** with both drives present, 0 ABSENT lines and the target stayed healthy. **Note: C5 still FAILED — but on a different defect (→ R-116), not on this one.** **SESSION C IS NOW UNBLOCKED.** Agent **0.114.0 published and vouched** 2026-07-29 (sha `5e4c15ebee2d7583…`, round-trip GET verified; hub manifest read back showing it) — a fresh drill box now installs the fix under test instead of the bug. The golden was NOT re-baked and `min_agent` was NOT raised: the golden bakes the controller, not the agent, and controller v0.185.0 declares MinAgent 0.113.0, which 0.114.0 already satisfies. **FIX: `BoundUnderParent` is now a CONJUNCTION — bound under the parent AND the drive's raw host mount still mounted** (`devicePresent`, new `deviceCheck` seam), at BOTH `/disks` construction sites. The raw mount is the device-bound systemd unit that dies with the device; the agent's bind is not — so the raw mount IS the device signal, grounded in E-2d's measurement rather than inference. **Conjunction, deliberately:** the device half alone would regress boot ordering (raw mounts early, bind lands ~18 s later — that window must keep reading absent), so existing behaviour is byte-identical and only the unreachable case is closed. **Unknown is never absent** (`devicePresent("")` = true) — a false absent stops a working customer's apps. **Controller UNCHANGED, no MinAgent bump:** `BoundUnderParent` has exactly one functional consumer (`planDriveGates:226`); a new `DevicePresent` bool was rejected because absent-from-JSON decodes to `false`, which would have made every drive on an older agent read ABSENT. +6 tests (208→214), 4 red-proofs run and reverted. **Deployed to demo-felhom; over-correction guard verified in production** — raw mount present, drive still reads present, 10/10 apps untouched, no gate action, no false alarm. **demo-hp deliberately left on 0.113.0** (the spec scoped deploy to felhom-pve) — it still carries the bug. **⚠️ SESSION C BLOCKER: the hub Day-0 manifest vouches agent 0.113.0**, so a fresh drill box would install WITHOUT this fix and validate nothing — publish + vouch 0.114.0 first (R-111's trap, same shape). **Awaiting live proof of exactly one leg:** device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. Original finding: **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 **Session C scope UNCHANGED by Session B** — R-114/R-112 shipped as controller v0.186.0 and neither touches the agent; the leg awaiting proof is still device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. One rebuild now validates all three. | CC | | **R-112** | **E-2's degraded banner and offer have NO UI CONSUMER — the endpoint is correct and the customer never sees it.** `GET /api/storage/backup-target` returns byte-exact copy (proven live), and **nothing fetches it**: `grep 'backup-target'` across all `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` + `degradedMessageFor` are consumed **only** by the JSON handler — **no page handler injects the state** | **SHIPPED + PROVEN-LIVE** (controller v0.186.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md` — the banner reached a customer's page for the first time.** Never-configured box: banner element 1, never-configured copy 1. After the wizard: offer block 1 with `data-path="/mnt/felhom-drives/mentes"`. Healthy after assign: all four markers 0, **proven POSITIVELY** — idle delta 0 `/backup/tiers` calls, page-load delta +1, single caller ⇒ the seam ran and chose silence. **FIXED: the state now has a consumer.** Server-rendered on `/backups` via `backupsHandler` → `backupTargetView` → `backups.html`, following the existing `SingleCopyWarning` banner pattern — NOT a 19th JS fetch, because a banner that needs JavaScript to appear is one more thing that can silently not happen. `backupTargetView` returns **nil** for healthy AND unknown, so those render nothing at all. **Scenario-E seam test drives `backupsHandler` over httptest and asserts the RENDERED HTML** — deleting the one line that sets `data["BackupTarget"]` reproduces the old state and fails every render assertion. **SEAM PROVEN LIVE on demo-felhom by a DIFFERENTIAL positive observable, not by an absent banner:** idle 8 s → 0 new `/backup/tiers` agent calls; each `/backups` load → exactly +1, and that call has only one caller (`resolveBackupTargetState`). The box is healthy (`degraded:false, target:felhom-backup`) and the page correctly rendered **nothing** — which matches its real state but, being a negative, is NOT by itself proof of wiring. **Still unproven live:** that a customer sees actual copy — impossible on a healthy box. **The decisive contrast: templates fetch 18 distinct `/api/storage/*` endpoints; `backup-target` and `backup-target/assign` are the only two with zero references.** The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"* — an invariant comment asserting a consumer that does not exist (7th instance of that class). v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* — it fixed the **router mount** and stopped one layer short of the **render**; its test `TestBackupTargetRoutesLiveUnderTheStorageAPIMount` pins dispatch, not reachability, which is exactly what `CLAUDE.md`'s seam rule warns about. **Fifth instance of seam-built-but-never-wired.** **Fix R-114 FIRST** — wiring this alone would start showing customers the wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | CC | | **R-114** | **On target-drive loss the customer is told the wrong story and offered the drive that just vanished.** With the assigned target absent, the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"a rendszermentés ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a drive that has disappeared, not the system disk — **and** `offer_path` pointing at the missing drive as the remedy | **SHIPPED + PROVEN-LIVE** (controller v0.186.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** With the target absent the page rendered the ABSENT copy (1), the system-disk copy 0, the offer block 0 — both of E-2d's falsehoods gone. API carried `message:"A rendszermentés meghajtója nem érhető el…"` with `target:felhom-backup`. **FIXED: the third state exists.** New `BackupTargetState.TargetAbsent` separates *configured-and-gone* from *never-configured*. `Degraded` keeps its meaning (is there a problem) so the wire contract is unchanged for every consumer; `TargetAbsent` answers which problem, because the remedies are OPPOSITE — attach any second drive vs reconnect *that* one. Copy routed through `degradedMessageFor` (still one decision point) and taken **verbatim** from the hub's `backup_target_absent` email so banner and mail tell one story. **Offer suppressed on the branch itself**, deliberately not left to `firstOfferableDrive`'s `Disconnected` skip — that flag is set by R-113 in another repo, and this state must be right without it. Red-proof: deleting the branch reproduces E-2d's exact payload, offering `/mnt/felhom-drives/mentes2`, the drive that had vanished. **MinAgent unchanged 0.113.0** — R-114 reads `BackupTarget`/`MountPath`/`GuestPath`/`Role`, none of which R-113 altered, so demo-hp is not held. **NOT live-validated: Scenario C cannot occur on a healthy box.** `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing **never configured** from **configured and now missing**. Shares R-113's root cause — two disagreeing presence signals — but is a different code path with a different fix. **Currently invisible ONLY because of R-112; fix this before wiring that.** Also seen: after reattach the drive returned as `/dev/sdc` while the stable bind still recorded `/dev/sdb`, and the state read healthy. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 | CC |