docs(R-116): isolate the mechanism from the real /disks payload; file R-117 + R-118
The absent-state /disks payload was captured on a genuine device loss, after a present-drive control run proved the query works (Part 5's three attempts failed on token extraction, and its control returned 0 rows). The answer is theory #1 -- "the registry-union row writes false" -- which was raised, declared wrong and retracted. The retraction was the error. Absent state returns 4 rows, not 3. The drive appears twice and the two facts the controller needs sit on different rows: the Observe row has backup_target:true but mount_path:"" and guest_path:"", so it contributes no key to driveTargetByPath; the registry-union row owns /mnt/felhom-drives/<name> and omits BackupTarget from its struct literal (disks.go:301-306) => false. The union row is not deduped because seen is keyed on MountPath (:290-295), the one field the absent state empties, and its own MountPath comes from the systemd .mount unit FILE (registry_known.go:40-75), which never reads the mount table. Theory #2 (the basis of the shipped v0.115.0) is false on both halves; #3 is false too. v0.115.0 is provably inert: StablePathForRaw("") returns "". Also files the read path verbatim -- the token plaintext lives only in bootstrap.json on the Proxmox host; the agent's store keeps hashes only. New: R-117 (READY M, outranks R-116) -- a returned drive's guest bind is a DEAD mount (EIO both ways) while /disks reports attached + bound_under_parent:true, so the gate restarts the customer's apps onto it and reports healthy with no alarm. R-118 (READY XS) -- an absent drive's union row advertises the root filesystem's capacity as its own. Docs only. No code written, nothing built or published; v0.115.0 untouched. Both demo boxes read-only; drill fixture restored to virgin.
This commit is contained in:
@@ -0,0 +1,113 @@
|
||||
# REPORT-r116-diag — the `/disks` payload captured, R-116's mechanism isolated (2026-07-30)
|
||||
|
||||
Read-only diagnosis run by CC on DooPlex. **No code written, nothing built, nothing published.**
|
||||
Full evidence: `documentation/audits/DIAG-r116-disks-payload-2026-07-30.md`.
|
||||
|
||||
A `REPORT-*.md` sibling, not the shared `REPORT.md` (`CLAUDE.md` parallel-session rule).
|
||||
|
||||
## Outcome
|
||||
|
||||
**Both goals met.** The `/disks` read path is solved and written down verbatim, proven by a
|
||||
present-drive control run *first*; and the absent-state payload was captured, which isolates the
|
||||
mechanism.
|
||||
|
||||
**R-116 is theory #1 — "the registry-union row writes `false`" — the theory that was raised, declared
|
||||
wrong, and retracted. The retraction was the error.**
|
||||
|
||||
In the absent state `/disks` returns **4 rows, not 3**. The drive appears twice and the two facts the
|
||||
controller needs sit on different rows:
|
||||
|
||||
| row | source | `mount_path` | `guest_path` | `backup_target` |
|
||||
|---|---|---|---|---|
|
||||
| `felhom-backup` | Observe (`disks.go:196-284`) | `""` | `""` | **`true`** |
|
||||
| `694034cc-…` (the UUID) | registry union (`disks.go:297-339`) | `/mnt/cel` | `/mnt/felhom-drives/cel` | **field ABSENT ⇒ `false`** |
|
||||
|
||||
So the row holding the flag contributes **no key** to `driveTargetByPath`, and the row that owns the key
|
||||
says `false` → `isTarget[a.Path]` is `false` → generic `storage_disconnected`. On return the rows
|
||||
re-merge into one carrying both facts → specific `backup_target_restored`. Applying
|
||||
`intermediary.go:602-618` to the captured payloads gives PRESENT `True` / ABSENT `False` /
|
||||
RETURNED `True` — **the live asymmetry reproduced from payload alone.**
|
||||
|
||||
The union row's `MountPath` survives the device because the union source is the systemd **`.mount` unit
|
||||
file** (`registry_known.go:40-75` via `main.go:605`→`:764`), which never reads the mount table. The
|
||||
dedup at `:298` therefore does not fire, because `seen` is keyed on the one field the absent state
|
||||
empties (`:290-295`).
|
||||
|
||||
**Theory #2 (the basis of the shipped v0.115.0) is false on both halves**; **#3 is false too**
|
||||
(`isTarget["/mnt/cel"]` is `false` as well). **v0.115.0 is provably inert** — its fallback calls
|
||||
`StablePathForRaw("")`, which returns `""` (`intermediary.go:69-75`), so it assigns nothing.
|
||||
|
||||
## The read path (this cost two prior sessions — it should never cost again)
|
||||
|
||||
The token plaintext exists in exactly one place: `bootstrap.json` **on the Proxmox host**. The agent's
|
||||
own store keeps SHA-256 hashes only (`tokenstore.go:26-32`), which is what defeated the earlier attempts.
|
||||
|
||||
```bash
|
||||
ssh felhom-pve
|
||||
B=/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json
|
||||
TOK=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['token'])")
|
||||
EP=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['endpoint'])")
|
||||
curl -sS -k -H "Authorization: Bearer $TOK" "https://$EP/disks" | python3 -m json.tool
|
||||
```
|
||||
|
||||
Control run, live felhom-pve, drive present: **HTTP 200, 2483 bytes, 4 plausible rows** — so Part 5's
|
||||
0-rows-on-a-present-drive failure mode is excluded.
|
||||
|
||||
## Where the absent state was staged
|
||||
|
||||
No new box. The existing DooPlex **nested-PVE drill fixture** (`drill/drill.qcow2`, snapshot `virgin`) —
|
||||
my own host, zero production exposure, and it can hot-unplug a disk for a genuine device loss. Run with
|
||||
the **byte-identical live agent binary** (`sha256 f48544ad…`, `--version` 0.115.0) and every
|
||||
state-producing step through the real endpoints (`format` → `assign` → `guest-attach` →
|
||||
`backup/target`). Its present-state row matched felhom-pve's control run field-for-field before it was
|
||||
trusted. Non-production aspects (root/direct privileged mode, stubbed hub, a hand-written bearer-token
|
||||
record, no controller) are enumerated in the audit §4.
|
||||
|
||||
## Two new findings, filed not chased
|
||||
|
||||
- **R-117 (READY M) — outranks R-116.** After a detach/reattach the guest's bind is a **dead mount**:
|
||||
host is healthy on the new device node, guest still names the old one, and `ls`/write through it
|
||||
return **`EIO`** — while `/disks` reports `attached` + `bound_under_parent:true` + `backup_target:true`.
|
||||
`planDriveGates` therefore takes the `Return` branch and **restarts the customer's apps onto a dead
|
||||
namespace, reporting healthy, with no alarm on any channel.** R-113's conjunction cannot catch it:
|
||||
one half is satisfied by the stale entry, the other by the new device, and neither compares them.
|
||||
This is the "stale bind" seen and dismissed as cosmetic in three consecutive runs.
|
||||
- **R-118 (READY XS).** An absent drive's union row reports the **root filesystem's** capacity as its own
|
||||
(46 GiB / 9.2 % for a 4 GB drive) — `statfsCapacity` at `disks.go:335-338` statfs's a bare directory on
|
||||
root. `observe.go:176-183` guards the Observe path against exactly this; the union path does not.
|
||||
`durable_id` is still correct, so re-attach identity is safe — it is a false capacity, not a DR mis-id.
|
||||
|
||||
## Register
|
||||
|
||||
`documentation/backlog/OPEN-ITEMS.md` — R-116 updated with the mechanism and the fix constraints;
|
||||
R-117 and R-118 added. The single register edit this session makes.
|
||||
|
||||
## Record correction
|
||||
|
||||
The brief's baseline `controller 0.185.1` is the version the **golden bakes**; both demo boxes actually
|
||||
run **0.186.0** (R-114 + R-112, 2026-07-29). Both are true of different things, and the distinction
|
||||
matters because R-114's `TargetAbsent` branch exists only in 0.186.0. Confirmed: agent **0.115.0**,
|
||||
controller **0.186.0** live, hub **0.81.0** (manifest pin and live pod image agree), host-install
|
||||
**1.22.0**, `felhom.eu` HEAD `c3ce4c7`.
|
||||
|
||||
## Teardown and fences
|
||||
|
||||
Drill guest destroyed, scratch storage removed, mount unit deleted, secrets `shred -u`'d, VM powered
|
||||
off, **`drill.qcow2` restored to `virgin`** (the golden-bake fixture is exactly as found), scratch qcow2
|
||||
and console dumps deleted. DooPlex `/mnt/5_hdd` at 24 %, unchanged.
|
||||
|
||||
Both demo boxes **read-only throughout** and re-verified after teardown: demo-hp `local-lvm` **38.83 %**
|
||||
(identical before/after and to Part 5), `drill-r50` still stopped, felhom-pve `felhom-backup` still
|
||||
active on `/dev/sdb`, both guests running, **v0.115.0 untouched**.
|
||||
|
||||
`sess-d-0452c4` now reads **STALE**, and the delete gate refuses only on ONLINE
|
||||
(`hub/internal/web/customer_delete.go:220-228`) — so it **is** now deletable; the command is recorded in
|
||||
the audit rather than executed (customer delete runs external teardown plus a DB purge). **`sess-c` is
|
||||
also still present and was not recorded by the Session-C audit** — same terms.
|
||||
|
||||
## Not done, deliberately
|
||||
|
||||
No fresh controller gate-log / hub-event correlation: that observable was already captured live and
|
||||
identically twice, and only the payload was missing. Staging a controller meant a hub customer, a
|
||||
pairing, a golden fetch and a claim — the work that consumed the three prior sessions. The audit §8
|
||||
correlates the payload to those existing measurements and labels that step as inference.
|
||||
Reference in New Issue
Block a user