docs(R-116): isolate the mechanism from the real /disks payload; file R-117 + R-118
The absent-state /disks payload was captured on a genuine device loss, after a present-drive control run proved the query works (Part 5's three attempts failed on token extraction, and its control returned 0 rows). The answer is theory #1 -- "the registry-union row writes false" -- which was raised, declared wrong and retracted. The retraction was the error. Absent state returns 4 rows, not 3. The drive appears twice and the two facts the controller needs sit on different rows: the Observe row has backup_target:true but mount_path:"" and guest_path:"", so it contributes no key to driveTargetByPath; the registry-union row owns /mnt/felhom-drives/<name> and omits BackupTarget from its struct literal (disks.go:301-306) => false. The union row is not deduped because seen is keyed on MountPath (:290-295), the one field the absent state empties, and its own MountPath comes from the systemd .mount unit FILE (registry_known.go:40-75), which never reads the mount table. Theory #2 (the basis of the shipped v0.115.0) is false on both halves; #3 is false too. v0.115.0 is provably inert: StablePathForRaw("") returns "". Also files the read path verbatim -- the token plaintext lives only in bootstrap.json on the Proxmox host; the agent's store keeps hashes only. New: R-117 (READY M, outranks R-116) -- a returned drive's guest bind is a DEAD mount (EIO both ways) while /disks reports attached + bound_under_parent:true, so the gate restarts the customer's apps onto it and reports healthy with no alarm. R-118 (READY XS) -- an absent drive's union row advertises the root filesystem's capacity as its own. Docs only. No code written, nothing built or published; v0.115.0 untouched. Both demo boxes read-only; drill fixture restored to virgin.
This commit is contained in:
@@ -0,0 +1,488 @@
|
||||
# DIAG-r116-disks-payload-2026-07-30 — the `/disks` payload, and the mechanism isolated from it
|
||||
|
||||
**Run:** read-only diagnosis by CC on DooPlex, 2026-07-30. **No code written, nothing shipped.**
|
||||
**Result: the payload was captured in both states and the mechanism is isolated.** It is theory **#1**
|
||||
— *the registry-union row writes `false`* — the theory that was raised, declared wrong, and retracted.
|
||||
**The retraction was the error.**
|
||||
|
||||
---
|
||||
|
||||
## 1. Baselines
|
||||
|
||||
All confirmed this run, not carried from the brief.
|
||||
|
||||
| Artifact | Version | How confirmed |
|
||||
|---|---|---|
|
||||
| agent | **0.115.0** | `felhom-agent --version` on felhom-pve; `felhom-agent/CHANGELOG.md:1` = `## v0.115.0`; repo HEAD `a58239f6ded367fe27b4eb57cf2fcb0c54414b80` |
|
||||
| controller | **0.186.0** ⚠ | `pct exec 9201 -- docker ps` on felhom-pve → `felhom-controller:0.186.0`; `CHANGELOG.md:3` = `### v0.186.0`; HEAD `b331f18424b7cf8d0883e7709f74595808264080` |
|
||||
| hub | **0.81.0** | `manifests/hub.yaml:128` pin **and** the live pod image (`kubectl -n felhom-system get pod hub-… -o jsonpath=…`) both `felhom-hub:0.81.0`; `hub/CHANGELOG.md:1` = `## v0.81.0` |
|
||||
| host-install | **1.22.0** | `scripts/felhom-host-install.sh:187` `SCRIPT_VERSION="1.22.0"` |
|
||||
| `felhom.eu` HEAD | `c3ce4c7b20ce4c0d3673590d710d13e5d28b68bb` | `git rev-parse HEAD` |
|
||||
|
||||
⚠ **The brief's `controller 0.185.1` is not what is deployed.** 0.185.1 is what the **golden bakes**
|
||||
(so it is what a fresh drill box installs, which is why Part 5 correctly recorded it); **0.186.0** is
|
||||
what both demo boxes actually run, shipped 2026-07-29 as R-114 + R-112. Both numbers are true of
|
||||
different things; the distinction matters because R-114's `TargetAbsent` branch exists only in 0.186.0.
|
||||
|
||||
Hub host table, read this run: `demo-felhom-8363b5` agent 0.115.0 ONLINE · `demo-hp-bb76ea` agent
|
||||
0.113.0 ONLINE · `drill-r50-0a4f9a` DOWN · `sess-c-8158e4` DOWN · `sess-d-0452c4` **STALE**.
|
||||
|
||||
---
|
||||
|
||||
## 2. §3 reconfirmation — every cited fact, at `file:line`
|
||||
|
||||
| Cited fact | Line now | Holds? |
|
||||
|---|---|---|
|
||||
| `observe.go` — `mountPath` comes from the mount table, assigned only on exact-mount success | `internal/storage/observe.go:184-190` | **HOLDS**, verbatim as quoted |
|
||||
| `role.go` — `if backingDevice == "" { return RoleSystem }` for USB/LocalDir | `internal/storage/role.go:180-181` | **HOLDS**, verbatim |
|
||||
| `primaryTargetID := s.primaryTier().TargetID`, in scope for both loops | `internal/localapi/disks.go:194` | **HOLDS** |
|
||||
| `BackupTarget: t.Name == primaryTargetID` | `disks.go:211` | **HOLDS** |
|
||||
| `GuestPath` set only inside `if di.Role == RoleUserData` | `disks.go:215-217` | **HOLDS** |
|
||||
| first loop `out = append(out, di)` | `disks.go:284` (brief said `:249`) | **HOLDS — line moved**, v0.115.0 inserted `:226-260` |
|
||||
| `seen[d.MountPath] = true` built by iterating `out` | `disks.go:290-295` (brief said `:258`) | **HOLDS — line moved.** Note the guard: `seen` is written **only when `d.MountPath != ""`** (`:292`) |
|
||||
| union dedup `if d.MountPath == "" \|\| seen[d.MountPath] { continue }` | `disks.go:298` (brief said `:263`) | **HOLDS — line moved** |
|
||||
| union row hardcodes `Role: RoleUserData` | `disks.go:304` | **HOLDS** |
|
||||
| union row's `GuestPath` set from `StablePathForRaw(d.MountPath)` | `disks.go:309-310` | **HOLDS** |
|
||||
| **`BackupTarget` is absent from the union struct literal** | `disks.go:301-306` | **HOLDS** — the literal sets `Name/Type/State/MountPath/DurableID/Role/GuestAttached` and nothing else |
|
||||
| v0.115.0's fallback `di.GuestPath = StablePathForRaw(t.MountPath)` | `disks.go:258-260` | **HOLDS** |
|
||||
| `StablePathForRaw("") == ""` (so the fallback sets nothing when `MountPath` is empty) | `internal/localapi/intermediary.go:69-75` → `DriveNameFromRaw:79-88` (no `/mnt/` prefix → `""`) | **HOLDS** |
|
||||
| PVE reports the detached target `dir inactive 0 0 0` | reproduced live, §5 | **HOLDS** |
|
||||
|
||||
**One fact the brief did not state, and it is load-bearing.** `/disks`' union source is the
|
||||
**systemd-unit-file** registry, not PVE and not the mount table:
|
||||
`storage.NewRegistryKnownTargets(driveUnitDir, …)` (`cmd/felhom-agent/main.go:605`) is the value passed
|
||||
as `DriveTargets` (`main.go:764` → `buildLocalAPIServer` → `disks.go:296` `s.driveTargets.Known()`).
|
||||
`Known()` (`internal/storage/registry_known.go:40-75`) `os.ReadDir`s `/etc/systemd/system`, parses each
|
||||
Felhom `.mount` unit, and returns `MountPath: spec.Where` (`:70`) — **it never consults the mount table
|
||||
or the device.** So the union row's `MountPath` survives the device's disappearance by construction.
|
||||
|
||||
Verified on the live box (felhom-pve), unit present and device-bound:
|
||||
|
||||
```
|
||||
/etc/systemd/system/mnt-hdd_1.mount
|
||||
What=/dev/disk/by-uuid/47a3361a-91e0-4831-a69d-27f540ed3f48
|
||||
Where=/mnt/hdd_1
|
||||
/var/lib/felhom-agent/drive-intents.json → intent recorded (not `new`, so Known() includes it)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 3. The working `/disks` invocation — verbatim
|
||||
|
||||
Two sessions were lost to this. **The token's plaintext exists in exactly one place: the guest's
|
||||
bootstrap mount, which the agent owns host-side.** The agent's own token store holds only SHA-256
|
||||
hashes (`internal/localapi/tokenstore.go:26-32`, `:136`), so it is *not* a source — that is what the
|
||||
earlier extraction attempts were fighting.
|
||||
|
||||
Authoritative source, on the **Proxmox host**, as root:
|
||||
|
||||
```
|
||||
/var/lib/felhom-agent/guests/<VMID>/bootstrap/bootstrap.json # 0600, owner 100000:100000
|
||||
```
|
||||
|
||||
(written by `internal/provision/backhalf.go:124-130`; the in-guest view is
|
||||
`/etc/felhom-bootstrap/bootstrap.json`, mp9, read-only — the host path avoids the guest hop entirely).
|
||||
|
||||
**The invocation. Run it ON the Proxmox host so the token never crosses a hop or reaches a log:**
|
||||
|
||||
```bash
|
||||
ssh felhom-pve
|
||||
export LC_ALL=C
|
||||
B=/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json
|
||||
TOK=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['token'])")
|
||||
EP=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['endpoint'])") # 169.254.253.1:8443
|
||||
curl -sS -k -H "Authorization: Bearer $TOK" "https://$EP/disks" | python3 -m json.tool
|
||||
```
|
||||
|
||||
`-k` is correct and not a shortcut: the leaf is self-signed and the controller pins it by SHA-256
|
||||
(`local_api.fingerprint` in the same file, `2d325e89…c191` on felhom-pve) rather than by a CA. The
|
||||
endpoint is the island bridge `vmbr9` (`169.254.253.1/30`), reachable from the host itself.
|
||||
|
||||
### 3.1 The control run — it proves the query works
|
||||
|
||||
**Present-drive control, live felhom-pve, before anything else:**
|
||||
|
||||
```
|
||||
endpoint=169.254.253.1:8443
|
||||
HTTP 200 bytes=2483
|
||||
rows: 4 (local-lvm, felhom-pbs, felhom-backup, local)
|
||||
```
|
||||
|
||||
The backup-target row in that control run:
|
||||
|
||||
```json
|
||||
{ "name": "felhom-backup", "type": "local-dir", "state": "attached",
|
||||
"backing_device": "/dev/sdb", "mount_path": "/mnt/hdd_1", "role": "user-data",
|
||||
"backup_target": true, "guest_path": "/mnt/felhom-drives/hdd_1",
|
||||
"bound_under_parent": true, "durable_id": "uuid:47a3361a-91e0-4831-a69d-27f540ed3f48" }
|
||||
```
|
||||
|
||||
Non-empty, plausible, four rows. **Part 5's failure mode (0 rows on a present drive) is excluded.**
|
||||
|
||||
---
|
||||
|
||||
## 4. Where the absent state was staged, and why there
|
||||
|
||||
The demo boxes' targets are fenced, and `drill-r50` is the drift fixture. **No new box was built.** The
|
||||
absent state was staged on the **existing nested-Proxmox drill VM on DooPlex**
|
||||
(`/mnt/5_hdd/felhom.eu/drill/drill.qcow2`, PVE 9.2, snapshot `virgin`, the golden-bake fixture) — my own
|
||||
host, zero production exposure, and it can hot-unplug a disk for a **genuine device loss** rather than a
|
||||
simulated one.
|
||||
|
||||
**Fidelity — what was real:**
|
||||
|
||||
- the **byte-identical live agent binary**: `scp` from felhom-pve `/usr/local/bin/felhom-agent`,
|
||||
`sha256 f48544ad5c4242a719c080ff83d7a6bc759e14272987416eb26c9683386803b7`, `--version` 0.115.0.
|
||||
- the real `configs/felhom-backup-target-apply` (E-2a) and `felhom-mkfs-guarded` wrappers.
|
||||
- a real LXC guest (9201), in the `felhom` pool, with the mp8 parent bind created **exactly** as
|
||||
`provision.BackHalf` does it (`backhalf.go:200-206`: `pct set 9201 -mp8 /mnt/felhom-drives,mp=/mnt/felhom-drives`).
|
||||
- **every state-producing step through the real endpoints**, not by hand:
|
||||
`POST /disks/format` → `POST /disks/assign` → `POST /disks/guest-attach` → `POST /backup/target`.
|
||||
That produced the `.mount` unit, `drive-intents.json: enrolled`, the guest bind, the
|
||||
`dir: felhom-backup / path /mnt/cel / is_mountpoint 1` storage and the `local_backup_target` rewrite.
|
||||
- device loss by QMP `device_del` on the virtio-blk frontend — the device node vanishes.
|
||||
|
||||
**What was NOT production-shaped, stated plainly:** (a) the agent ran as root with
|
||||
`privileged.mode: "direct"` and a `root@pam` PVE token instead of non-root + sudoers + `felhom-agent@pve`
|
||||
— this changes *who* runs the privileged ops, not what `/disks` computes; (b) `hub.url` pointed at a
|
||||
dead local port, so nothing was reported to the hub; (c) **the local-API bearer token record was
|
||||
hand-written** into `local-tokens.log` (one `{"v":9201,"h":"<sha256>"}` line) because minting is only
|
||||
reachable through `--selftest=provision`, which needs a golden and a real customer. That is the *key to
|
||||
the door*, not the state under measurement — every drive/storage/target fact was produced by the real
|
||||
endpoints. (d) No controller ran, so §7 correlates against the two prior live runs rather than a fresh one.
|
||||
|
||||
**The rig validated itself against production before being trusted:** its present-state target row is
|
||||
field-for-field identical in shape to felhom-pve's control run (`role: user-data`,
|
||||
`backup_target: true`, `guest_path` set, `bound_under_parent: true`, `durable_id: uuid:…`) — only the
|
||||
device node and mountpoint names differ.
|
||||
|
||||
---
|
||||
|
||||
## 5. Host state at each capture
|
||||
|
||||
| | PRESENT (05:53:49Z) | ABSENT (05:54:32Z) | RETURNED (05:55:45Z) |
|
||||
|---|---|---|---|
|
||||
| device node | `/dev/vdb` | **gone** (`ls /dev/vdb` → No such file) | `/dev/vdc` |
|
||||
| `findmnt /mnt/cel` | `/dev/vdb ext4` | **not mounted** (rc=1) | `/dev/vdc ext4` |
|
||||
| guest `findmnt /mnt/felhom-drives/cel` | `/dev/vdb[/felhom-data]` | `/dev/vdb[/felhom-data] …,shutdown` | `/dev/vdb[/felhom-data] …,shutdown` |
|
||||
| `pvesm status` | `felhom-backup dir active` | `felhom-backup dir inactive 0 0 0` | `felhom-backup dir active` |
|
||||
|
||||
The detached state reproduced PVE's exact Part-5 string:
|
||||
|
||||
```
|
||||
unable to activate storage 'felhom-backup' - directory is expected to be a mount point
|
||||
but is not mounted: '/mnt/cel'
|
||||
```
|
||||
|
||||
and the agent's exact Session-C / E-2d lines:
|
||||
|
||||
```
|
||||
07:54:08 WARN storage: watchdog detected target state change from=attached to=disconnected
|
||||
07:54:09 INFO storage: enrolled drive absent by UUID — not re-asserting (will retry when it enumerates)
|
||||
07:54:09 WARN reconcile: enrolled drive not present (durable-id absent) — skipping
|
||||
```
|
||||
|
||||
The R-113 asymmetry is present as measured before: **raw `/mnt/cel` NOT mounted while the guest's bind
|
||||
still names `/dev/vdb[/felhom-data]`.**
|
||||
|
||||
---
|
||||
|
||||
## 6. The raw payloads
|
||||
|
||||
### 6.1 PRESENT — `GET /disks`, HTTP 200, 1824 bytes, **3 rows**
|
||||
|
||||
```json
|
||||
{"ok":true,"data":{"disks":[
|
||||
{"name":"felhom-backup","type":"local-dir","state":"attached","backing_device":"/dev/vdb","mount_path":"/mnt/cel","class":"slow","role":"user-data","data_bearing":true,"data_reason":"device is mounted","total_bytes":4143677440,"used_bytes":1081344,"used_fraction":0.000260962397690878,"durable_id":"uuid:694034cc-eee6-4b17-901f-708f50df9b09","wipe_durable_id":"byid:virtio-R116CEL","guest_attached":false,"backup_target":true,"guest_path":"/mnt/felhom-drives/cel","bound_under_parent":true,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}},
|
||||
{"name":"local-lvm","type":"lvmthin","state":"attached","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":81533075456,"used_bytes":807177447,"used_fraction":0.00989999999982339,"durable_id":"pve/data","guest_attached":false,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}},
|
||||
{"name":"local","type":"local","state":"attached","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":49675956224,"used_bytes":4584443904,"used_fraction":0.092286978499774,"durable_id":"path:/var/lib/vz","guest_attached":false,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}}
|
||||
],"guest_boot_id":"1785390256-46753","vmid":9201}}
|
||||
```
|
||||
|
||||
### 6.2 ABSENT — `GET /disks`, HTTP 200, 2040 bytes, **4 rows** (one MORE than present)
|
||||
|
||||
```json
|
||||
{"ok":true,"data":{"disks":[
|
||||
{"name":"felhom-backup","type":"local-dir","state":"disconnected","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":0,"used_bytes":0,"used_fraction":0,"durable_id":"path:/mnt/cel","guest_attached":false,"backup_target":true,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}},
|
||||
{"name":"local-lvm","type":"lvmthin","state":"attached","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":81533075456,"used_bytes":807177447,"used_fraction":0.00989999999982339,"durable_id":"pve/data","guest_attached":false,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}},
|
||||
{"name":"local","type":"local","state":"attached","backing_device":"","mount_path":"","class":"","role":"system","data_bearing":false,"total_bytes":49675956224,"used_bytes":4584579072,"used_fraction":0.0922896994942001,"durable_id":"path:/var/lib/vz","guest_attached":false,"bound_under_parent":false,"smart":{"health":"UNKNOWN","temperature_c":null,"power_on_hours":null,"reallocated_sectors":null,"pending_sectors":null,"offline_uncorrectable":null,"critical_warning":null,"media_errors":null,"percentage_used":null}},
|
||||
{"name":"694034cc-eee6-4b17-901f-708f50df9b09","type":"usb","state":"attached","backing_device":"","mount_path":"/mnt/cel","class":"","role":"user-data","data_bearing":false,"total_bytes":49675956224,"used_bytes":4584579072,"used_fraction":0.09228969949420011,"durable_id":"uuid:694034cc-eee6-4b17-901f-708f50df9b09","guest_attached":false,"guest_path":"/mnt/felhom-drives/cel","bound_under_parent":false}
|
||||
],"guest_boot_id":"1785390256-46753","vmid":9201}}
|
||||
```
|
||||
|
||||
### 6.3 RETURNED — `GET /disks`, HTTP 200, 1825 bytes, **3 rows** (back to one row for the drive)
|
||||
|
||||
```json
|
||||
{"ok":true,"data":{"disks":[
|
||||
{"name":"felhom-backup","type":"local-dir","state":"attached","backing_device":"/dev/vdc","mount_path":"/mnt/cel","class":"slow","role":"user-data","data_bearing":true,"data_reason":"device is mounted","total_bytes":4143677440,"used_bytes":1081344,"used_fraction":0.000260962397690878,"durable_id":"uuid:694034cc-eee6-4b17-901f-708f50df9b09","wipe_durable_id":"byid:virtio-R116CEL","guest_attached":false,"backup_target":true,"guest_path":"/mnt/felhom-drives/cel","bound_under_parent":true,"smart":{"health":"UNKNOWN", …}},
|
||||
{"name":"local-lvm", … as above … },
|
||||
{"name":"local", … as above … }
|
||||
],"guest_boot_id":"1785390256-46753","vmid":9201}}
|
||||
```
|
||||
|
||||
*(the `local-lvm` / `local` rows in 6.3 are byte-identical to 6.1 apart from `used_bytes`; elided only
|
||||
there, and nowhere else in this document.)*
|
||||
|
||||
---
|
||||
|
||||
## 7. The per-row table for the backup-target drive
|
||||
|
||||
**PAYLOAD.** Every cell read from the JSON above.
|
||||
|
||||
### PRESENT — 1 row mentions the drive
|
||||
|
||||
| Name | Type | State | Role | MountPath | GuestPath | BackingDevice | BackupTarget | BoundUnderParent |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| `felhom-backup` | `local-dir` | `attached` | `user-data` | `/mnt/cel` | `/mnt/felhom-drives/cel` | `/dev/vdb` | **`true`** | `true` |
|
||||
|
||||
### ABSENT — **2 rows** mention the drive, and the two facts are on DIFFERENT rows
|
||||
|
||||
| Name | Type | State | Role | MountPath | GuestPath | BackingDevice | BackupTarget | BoundUnderParent | Source |
|
||||
|---|---|---|---|---|---|---|---|---|---|
|
||||
| `felhom-backup` | `local-dir` | `disconnected` | **`system`** | **`""`** | **`""`** (field absent) | `""` | **`true`** | `false` | Observe loop, `disks.go:196-284` |
|
||||
| `694034cc-…9b09` | `usb` | **`attached`** (hardcoded `:302`) | `user-data` (hardcoded `:304`) | **`/mnt/cel`** | **`/mnt/felhom-drives/cel`** | `""` | **field ABSENT ⇒ `false`** | `false` | union loop, `disks.go:297-339` |
|
||||
|
||||
`durable_id` also degrades on the Observe row: `uuid:694034cc…` → **`path:/mnt/cel`**.
|
||||
|
||||
### RETURNED — 1 row again
|
||||
|
||||
| Name | Type | State | Role | MountPath | GuestPath | BackingDevice | BackupTarget | BoundUnderParent |
|
||||
|---|---|---|---|---|---|---|---|---|
|
||||
| `felhom-backup` | `local-dir` | `attached` | `user-data` | `/mnt/cel` | `/mnt/felhom-drives/cel` | **`/dev/vdc`** | **`true`** | `true` |
|
||||
|
||||
---
|
||||
|
||||
## 8. The mechanism
|
||||
|
||||
**Which row the lookup lands on, which key it uses, what that row says.**
|
||||
|
||||
The controller's key is `a.Path` — the **registered `StoragePath`**, which for an external drive is the
|
||||
GUEST path `/mnt/felhom-drives/cel` (`intermediary.go:297`, `:318`). `driveTargetByPath`
|
||||
(`intermediary.go:602-618`) builds the map with **plain assignment**, one entry per non-empty
|
||||
`GuestPath` (`:612`) and per non-empty `MountPath` (`:615`).
|
||||
|
||||
**PAYLOAD — the map, computed by applying `intermediary.go:602-618` verbatim to the captured JSON:**
|
||||
|
||||
```
|
||||
PRESENT rows=3 isTarget["/mnt/felhom-drives/cel"] = True → SPECIFIC
|
||||
map: {'/mnt/felhom-drives/cel': True, '/mnt/cel': True}
|
||||
ABSENT rows=4 isTarget["/mnt/felhom-drives/cel"] = False → GENERIC
|
||||
map: {'/mnt/felhom-drives/cel': False, '/mnt/cel': False}
|
||||
RETURNED rows=3 isTarget["/mnt/felhom-drives/cel"] = True → SPECIFIC
|
||||
map: {'/mnt/felhom-drives/cel': True, '/mnt/cel': True}
|
||||
```
|
||||
|
||||
That is the live asymmetry — generic on detach, specific on return — **reproduced from the payload
|
||||
alone**, matching Session C and Part 5 exactly.
|
||||
|
||||
**Why it fails, step by step. Steps 1–4 are payload; step 5 is the map above.**
|
||||
|
||||
1. The device goes. `exactMountDevice` fails, so `backingDevice` and `mountPath` are both `""`
|
||||
(`observe.go:184-190`). **Payload: Observe row has `backing_device:""`, `mount_path:""`.**
|
||||
2. `RoleForStorage` therefore returns `RoleSystem` (`role.go:180-181`), so the `GuestPath` block at
|
||||
`disks.go:215-217` is skipped. **Payload: `role:"system"`, no `guest_path`.**
|
||||
3. v0.115.0's fallback (`disks.go:258-260`) fires — its three gates all pass — but it calls
|
||||
`StablePathForRaw(t.MountPath)` with `MountPath == ""`, which returns `""`
|
||||
(`intermediary.go:69-75`). **It assigns an empty string. Payload confirms: no `guest_path` on that
|
||||
row.** This is exactly why v0.115.0 is inert.
|
||||
4. `seen` is built only from rows whose `MountPath != ""` (`disks.go:290-295`). The Observe row's is
|
||||
empty, so **`/mnt/cel` is not in `seen`**, so the union row is **not** deduped at `:298`. It is
|
||||
emitted with `MountPath` from the `.mount` unit file and `GuestPath` from `:309-310` — and
|
||||
**`BackupTarget` is not in its struct literal (`:301-306`), so it serialises as `false`.**
|
||||
**Payload: row 4 exists, carries both paths, and `backup_target` is absent.**
|
||||
5. So the row holding `backup_target:true` contributes **no key at all** (both its keys are empty
|
||||
strings and are skipped at `:611` and `:614`), while the row that owns the key
|
||||
`/mnt/felhom-drives/cel` says `false`. `isTarget[a.Path]` → `false` → `notifyDriveAbsent` takes the
|
||||
else branch (`intermediary.go:645`) → **generic `storage_disconnected`**.
|
||||
6. On return the two rows re-merge into one (the Observe row regains `MountPath`, so the union row is
|
||||
deduped again) — that single row carries `backup_target:true` **and** the guest path, so
|
||||
`notifyDriveReturned` takes the specific branch (`:656`) → **`backup_target_restored`**. Hence the
|
||||
unmatchable pair.
|
||||
|
||||
**INFERENCE (small, and labelled):** step 5's map-construction and step 6's branch selection were
|
||||
computed by applying the controller's published source to the captured payload, not observed from a
|
||||
running controller this session. The observable they predict — generic out, specific back — is exactly
|
||||
what Session C and Part 5 *did* measure live on real boxes, twice. So the prediction is corroborated,
|
||||
but the controller was not itself re-run here.
|
||||
|
||||
**There is no map-overwrite race.** An earlier worry was that two rows might both write the same key
|
||||
with the last one winning. They do not: the Observe row contributes nothing, so the union row is the
|
||||
sole writer of `/mnt/felhom-drives/cel`. Row order is irrelevant.
|
||||
|
||||
---
|
||||
|
||||
## 9. Which theory this is
|
||||
|
||||
| Theory | Verdict from payload |
|
||||
|---|---|
|
||||
| **#1 — "the union row writes `false`"** (raised, then declared wrong and retracted) | ✅ **CORRECT.** The union row is emitted (`rows` 3 → 4), owns the key, and its `backup_target` field is absent ⇒ `false`. **The retraction was the error, not the theory.** |
|
||||
| **#2 — "no row carries the guest path; the union row is deduped away"** (the basis of the shipped v0.115.0 fix) | ❌ **FALSE on both halves.** A row *does* carry `guest_path:"/mnt/felhom-drives/cel"` (row 4), and the union row is *not* deduped — precisely *because* `MountPath` does not survive on the Observe row, which is the fact #2 was built on. #2 had the premise right and drew the opposite conclusion from it. |
|
||||
| **#3 — "`isTarget["/mnt/<n>"]` is already `true` in the absent state, so the controller could fall back to it"** | ❌ **FALSE.** `isTarget["/mnt/cel"] = false` as well — the same union row writes both keys. A fallback to the raw path changes nothing. |
|
||||
|
||||
So: **#1**, and it was available all along. The one genuinely new fact is *why* the dedup does not fire
|
||||
— `seen` is keyed on the very field the absent state empties (`disks.go:290-295` vs `:298`), which makes
|
||||
the union row's emission a **consequence** of the vanished device rather than an independent condition.
|
||||
|
||||
---
|
||||
|
||||
## 10. What a fix would have to change — constraints for the next session's spec
|
||||
|
||||
**Not implemented here. Stated as constraints, not as a design.**
|
||||
|
||||
1. **The necessary and sufficient condition is that the row owning the key
|
||||
`/mnt/felhom-drives/<name>` must report `backup_target: true` when that drive is the primary
|
||||
target.** Everything else is a means. Whether that is achieved by carrying the flag onto the union
|
||||
row, by merging the two rows, or by making `driveTargetByPath` OR rather than assign, is the spec's
|
||||
choice — but it must be verified against a payload, not a fixture.
|
||||
2. **`disks.go:301-306` omits `BackupTarget` — and the omission is invisible.** A struct literal that
|
||||
silently defaults a bool is the whole defect. Whatever the fix, this literal is the site. Note the
|
||||
union loop has no access to `t` (there is no Observe target for it) — it would have to compare
|
||||
`d.Name`/`d.MountPath` against `primaryTargetID`/the target's path, and **`d.Name` for a union row is
|
||||
the drive's UUID, not the storage id** (payload: `"694034cc-…"` vs `"felhom-backup"`), so a naive
|
||||
`d.Name == primaryTargetID` comparison will silently never match. That trap is worth stating in the
|
||||
spec explicitly.
|
||||
3. **Fixing only the agent may be enough, but the controller half is worth a look in the same pass.**
|
||||
`driveTargetByPath` uses assignment, so it is order-dependent the moment two rows ever carry the
|
||||
same key with different values — which is exactly the state a partial fix would create. An OR (or a
|
||||
documented single-writer invariant with a test) removes the hazard.
|
||||
4. **v0.115.0's fallback should not simply be extended.** It is gated on `t.BackingDevice == ""` and
|
||||
feeds `StablePathForRaw(t.MountPath)`; in the failing state `MountPath` is also `""`, so there is no
|
||||
value on that row to derive a guest path *from*. Any fix on the Observe row needs a path source that
|
||||
survives device loss — and the only one that does is the same `.mount` unit the union row already
|
||||
reads. That argues for fixing the union row, not the Observe row.
|
||||
5. **Do not let the fix make an absent drive read as present.** `planDriveGates` computes
|
||||
`present[gp] = present[gp] || d.BoundUnderParent` (`intermediary.go:226`). The union row's
|
||||
`BoundUnderParent` is `false` in the payload (R-113's conjunction holds), and it must stay false. The
|
||||
existing `TestAbsentTargetRowDoesNotRegisterPresence` pins this for the Observe row; the union row
|
||||
needs the equivalent.
|
||||
6. **The red-proof must be the payload shape, not the Session-C fixture.** v0.115.0 was unit-tested and
|
||||
red-proofed against a fixture that supplied a `MountPath` and a `disconnected` state — a shape that
|
||||
does not occur. The fixture for the next fix is §6.2 of this document: **4 rows, the target's flag
|
||||
and its guest path on different rows, `state:"attached"` on the union row.**
|
||||
7. **Whether to keep v0.115.0 is now answerable.** It is provably inert (it assigns `""`). It is not
|
||||
harmful. Reverting it is a separate change with its own risk; extending it is ruled out by (4).
|
||||
Recommendation, for the spec to accept or reject: **leave it, and note in its comment that the case
|
||||
it targets cannot arise** — or delete it in the same commit that fixes the union row, so the tree
|
||||
does not carry two explanations of one defect.
|
||||
|
||||
---
|
||||
|
||||
## 11. R-117 (new) — the returned drive's guest bind is a DEAD MOUNT, and everything reports healthy
|
||||
|
||||
The brief's §9 "stale bind, unchased after three runs". **It is not cosmetic.** Captured this run, and
|
||||
it is worse than a wrong device name.
|
||||
|
||||
**PAYLOAD + live probe, RETURNED state (drive back, host healthy on `/dev/vdc`):**
|
||||
|
||||
```
|
||||
host findmnt -no SOURCE /mnt/cel → /dev/vdc
|
||||
guest findmnt -no SOURCE,OPTIONS /mnt/felhom-drives/cel → /dev/vdb[/felhom-data] rw,relatime,shutdown
|
||||
guest ls /mnt/felhom-drives/cel → ls: reading directory …: Input/output error
|
||||
guest write into it → cannot create …: Input/output error (GUEST-WRITE-FAIL)
|
||||
host ls /mnt/cel/felhom-data/ → fine, empty, live
|
||||
/disks says: state "attached", role "user-data", bound_under_parent TRUE, backup_target true
|
||||
```
|
||||
|
||||
**The consequence, and this is why it outranks R-116.** `planDriveGates` computes
|
||||
`present["/mnt/felhom-drives/cel"] = … || d.BoundUnderParent` (`intermediary.go:226`), which is `true`.
|
||||
The drive is currently `Disconnected`, so the gate takes the **`Return`** branch
|
||||
(`intermediary.go:281`, `:301-320`): it clears the disconnected flag and **restarts the gate-stopped
|
||||
apps onto a mount that returns `EIO` for every operation** — then reports the drive healthy and emails
|
||||
`backup_target_restored`. A customer's apps come back up pointed at a dead namespace with no alarm on
|
||||
any channel.
|
||||
|
||||
R-113's conjunction cannot catch it: half 1 (guest mountinfo has an entry at the guest path) is
|
||||
satisfied by the *stale* entry, and half 2 (`devicePresent("/mnt/cel")`) is satisfied by the *new*
|
||||
device. **Neither half compares the two.** `disks.go:158-165`'s comment claims only that the conjunction
|
||||
fixes the boot-ordering and outlived-bind cases — it does not claim this, so this is a gap rather than a
|
||||
false invariant.
|
||||
|
||||
Reproduced deliberately and cleanly here; consistent with the same asymmetry seen (uninvestigated) in
|
||||
E-2d, Session C and Part 5. **Filed, not chased, per the brief.**
|
||||
|
||||
---
|
||||
|
||||
## 12. R-118 (new) — the absent drive's union row reports the ROOT filesystem's size as its own
|
||||
|
||||
**PAYLOAD, §6.2 row 4:** `total_bytes: 49675956224`, `used_bytes: 4584579072` — **byte-identical to the
|
||||
`local` row** (`durable_id: path:/var/lib/vz`, i.e. `pve-root`) in the same response. The real drive is
|
||||
**4 GB** (`4143677440` in §6.1).
|
||||
|
||||
Cause, from source: `statfsCapacity(d.MountPath)` at `disks.go:335-338` statfs's `/mnt/cel`, which with
|
||||
the device gone is a **bare directory on the root filesystem**. `observe.go:176-183`'s comment warns
|
||||
about exactly this trap and guards the Observe path against it ("*an unmounted removable dir-storage's
|
||||
mountpoint reverts to a bare directory on root … would be a catastrophic DR mis-id*"); **the union path
|
||||
at `:335` has no equivalent guard.**
|
||||
|
||||
Not a DR mis-id — `durable_id` on that row is still the correct `uuid:694034cc…`, so the re-attach
|
||||
identity is safe. It is a **false capacity**: a detached 4 GB drive advertises 46 GiB with 9.2 % used, to
|
||||
any consumer of `total_bytes`/`used_fraction` (fill monitors, the storage cards). Same class as the
|
||||
`role.go:180-181` flip that drives R-116 — an absent drive's fields decaying to the root filesystem's.
|
||||
|
||||
---
|
||||
|
||||
## 13. Teardown
|
||||
|
||||
| | Before | After |
|
||||
|---|---|---|
|
||||
| drill VM `pvesm status` | `felhom-backup` active, `local-lvm` 0.99 %, `local` 9.23 % | `felhom-backup` **removed**, `local-lvm` **0.00 %**, `local` 9.23 % |
|
||||
| drill VM `pct list` | `9201 running` | **empty** |
|
||||
|
||||
- guest 9201 `pct destroy --purge` (logical volume removed, config purged); `pvesm remove felhom-backup`;
|
||||
`mnt-cel.mount` disabled + deleted + `daemon-reload`; `shred -u` on the local-API token file and the
|
||||
PVE token file.
|
||||
- `poweroff`, qemu exited (`0` procs), **`qemu-img snapshot -a virgin drill.qcow2`** → the golden-bake
|
||||
fixture is exactly as found (`virgin`, ID 1, 2026-07-03 14:12:15).
|
||||
- scratch artifacts deleted: `r116-cel.qcow2`, `qmp.sock`, `qemu.pid`, both console `.ppm` dumps.
|
||||
- DooPlex `df -h /mnt/5_hdd`: **2.1T used / 6.6T avail, 24 %** — same as at session start.
|
||||
- The disposable `root@pam!r116` PVE token and the drill local-API token died with the snapshot revert;
|
||||
neither is recorded in any committed file.
|
||||
|
||||
**Fences — re-verified after teardown, not assumed:**
|
||||
|
||||
| Fence | State |
|
||||
|---|---|
|
||||
| demo-hp `local-lvm` | **38.83 %** — identical before, after, and to Part 5's reading |
|
||||
| demo-hp `drill-r50` (VM 300) | `stopped` — untouched |
|
||||
| demo-hp guest 9201 | `running` |
|
||||
| felhom-pve `felhom-backup` | `active`, `/mnt/hdd_1` on `/dev/sdb` — **target not re-pointed** |
|
||||
| felhom-pve guest 9201 | `running` |
|
||||
| Both demo boxes | **read-only access only** — `GET /disks`, file reads, `pvesm status`, `pct list`, one `scp` of the agent binary |
|
||||
| Code | **no `.go` file edited, nothing built, nothing published, v0.115.0 untouched** |
|
||||
|
||||
### `sess-d`, and a leftover the prior runs did not record
|
||||
|
||||
`sess-d-0452c4` now reads **STALE**, not ONLINE. The delete gate refuses only on
|
||||
`s.hostStatus(...) == "ok"` (`hub/internal/web/customer_delete.go:220-228`) — ONLINE — so **`sess-d` is
|
||||
now deletable.** Not executed here: deleting a customer runs external teardown plus a DB purge, and the
|
||||
brief scoped this session to recording the command.
|
||||
|
||||
```
|
||||
POST /configs/sess-d/delete ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-d expect_hosts=1
|
||||
```
|
||||
|
||||
**Also still present, and not mentioned in the Session-C audit: `sess-c` / host `sess-c-8158e4`
|
||||
(agent 0.114.0, DOWN).** Same shape, same gate, deletable on the same terms —
|
||||
`confirm_id=sess-c expect_hosts=1`. Two drill customers are now accumulating in the hub; that is the
|
||||
operator's cleanup call, not a register row.
|
||||
|
||||
---
|
||||
|
||||
## 14. What did not happen, and why
|
||||
|
||||
- **No controller was run, so no fresh gate-log / hub-event correlation was captured** (brief §6.4).
|
||||
Deliberate: that observable was already captured live and identically **twice** (Session C, Part 5),
|
||||
and the *payload* was the only missing input. Staging a controller meant a hub customer, a pairing, a
|
||||
golden fetch and a claim — the work that consumed the three prior sessions. §8 correlates the payload
|
||||
to those existing measurements instead, and labels that as the inference it is.
|
||||
- **No new drill box was built** — the brief's stated preference. The existing DooPlex nested-PVE
|
||||
fixture was reused and restored.
|
||||
- **R-117 and R-118 were filed, not investigated.** R-117 in particular deserves its own run; on the
|
||||
evidence here it is more customer-affecting than R-116.
|
||||
- **v0.115.0 was neither reverted nor extended**, per the fence.
|
||||
- **Two boot attempts of the drill VM failed before the run started** and are recorded because the cause
|
||||
is a reusable trap: adding `-device` entries at launch shifts PCI slot assignment, which renames the
|
||||
guest NIC (`ens3` → gone), so vmbr0 loses its port and SSH is unreachable while the console shows a
|
||||
healthy PVE login. `bootindex` fixes the boot-order half but not the NIC half. **The fix is to boot
|
||||
the memory's known-good command line unchanged and hot-plug the extra disk over QMP afterwards** —
|
||||
which is also the more faithful device-loss mechanism.
|
||||
File diff suppressed because one or more lines are too long
Reference in New Issue
Block a user