admin
d56e395a2a
The absent-state /disks payload was captured on a genuine device loss, after a present-drive control run proved the query works (Part 5's three attempts failed on token extraction, and its control returned 0 rows). The answer is theory #1 -- "the registry-union row writes false" -- which was raised, declared wrong and retracted. The retraction was the error. Absent state returns 4 rows, not 3. The drive appears twice and the two facts the controller needs sit on different rows: the Observe row has backup_target:true but mount_path:"" and guest_path:"", so it contributes no key to driveTargetByPath; the registry-union row owns /mnt/felhom-drives/<name> and omits BackupTarget from its struct literal (disks.go:301-306) => false. The union row is not deduped because seen is keyed on MountPath (:290-295), the one field the absent state empties, and its own MountPath comes from the systemd .mount unit FILE (registry_known.go:40-75), which never reads the mount table. Theory #2 (the basis of the shipped v0.115.0) is false on both halves; #3 is false too. v0.115.0 is provably inert: StablePathForRaw("") returns "". Also files the read path verbatim -- the token plaintext lives only in bootstrap.json on the Proxmox host; the agent's store keeps hashes only. New: R-117 (READY M, outranks R-116) -- a returned drive's guest bind is a DEAD mount (EIO both ways) while /disks reports attached + bound_under_parent:true, so the gate restarts the customer's apps onto it and reports healthy with no alarm. R-118 (READY XS) -- an absent drive's union row advertises the root filesystem's capacity as its own. Docs only. No code written, nothing built or published; v0.115.0 untouched. Both demo boxes read-only; drill fixture restored to virgin.
6.8 KiB
REPORT-r116-diag — the /disks payload captured, R-116's mechanism isolated (2026-07-30)
Read-only diagnosis run by CC on DooPlex. No code written, nothing built, nothing published.
Full evidence: documentation/audits/DIAG-r116-disks-payload-2026-07-30.md.
A REPORT-*.md sibling, not the shared REPORT.md (CLAUDE.md parallel-session rule).
Outcome
Both goals met. The /disks read path is solved and written down verbatim, proven by a
present-drive control run first; and the absent-state payload was captured, which isolates the
mechanism.
R-116 is theory #1 — "the registry-union row writes false" — the theory that was raised, declared
wrong, and retracted. The retraction was the error.
In the absent state /disks returns 4 rows, not 3. The drive appears twice and the two facts the
controller needs sit on different rows:
| row | source | mount_path |
guest_path |
backup_target |
|---|---|---|---|---|
felhom-backup |
Observe (disks.go:196-284) |
"" |
"" |
true |
694034cc-… (the UUID) |
registry union (disks.go:297-339) |
/mnt/cel |
/mnt/felhom-drives/cel |
field ABSENT ⇒ false |
So the row holding the flag contributes no key to driveTargetByPath, and the row that owns the key
says false → isTarget[a.Path] is false → generic storage_disconnected. On return the rows
re-merge into one carrying both facts → specific backup_target_restored. Applying
intermediary.go:602-618 to the captured payloads gives PRESENT True / ABSENT False /
RETURNED True — the live asymmetry reproduced from payload alone.
The union row's MountPath survives the device because the union source is the systemd .mount unit
file (registry_known.go:40-75 via main.go:605→:764), which never reads the mount table. The
dedup at :298 therefore does not fire, because seen is keyed on the one field the absent state
empties (:290-295).
Theory #2 (the basis of the shipped v0.115.0) is false on both halves; #3 is false too
(isTarget["/mnt/cel"] is false as well). v0.115.0 is provably inert — its fallback calls
StablePathForRaw(""), which returns "" (intermediary.go:69-75), so it assigns nothing.
The read path (this cost two prior sessions — it should never cost again)
The token plaintext exists in exactly one place: bootstrap.json on the Proxmox host. The agent's
own store keeps SHA-256 hashes only (tokenstore.go:26-32), which is what defeated the earlier attempts.
ssh felhom-pve
B=/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json
TOK=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['token'])")
EP=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['endpoint'])")
curl -sS -k -H "Authorization: Bearer $TOK" "https://$EP/disks" | python3 -m json.tool
Control run, live felhom-pve, drive present: HTTP 200, 2483 bytes, 4 plausible rows — so Part 5's 0-rows-on-a-present-drive failure mode is excluded.
Where the absent state was staged
No new box. The existing DooPlex nested-PVE drill fixture (drill/drill.qcow2, snapshot virgin) —
my own host, zero production exposure, and it can hot-unplug a disk for a genuine device loss. Run with
the byte-identical live agent binary (sha256 f48544ad…, --version 0.115.0) and every
state-producing step through the real endpoints (format → assign → guest-attach →
backup/target). Its present-state row matched felhom-pve's control run field-for-field before it was
trusted. Non-production aspects (root/direct privileged mode, stubbed hub, a hand-written bearer-token
record, no controller) are enumerated in the audit §4.
Two new findings, filed not chased
- R-117 (READY M) — outranks R-116. After a detach/reattach the guest's bind is a dead mount:
host is healthy on the new device node, guest still names the old one, and
ls/write through it returnEIO— while/disksreportsattached+bound_under_parent:true+backup_target:true.planDriveGatestherefore takes theReturnbranch and restarts the customer's apps onto a dead namespace, reporting healthy, with no alarm on any channel. R-113's conjunction cannot catch it: one half is satisfied by the stale entry, the other by the new device, and neither compares them. This is the "stale bind" seen and dismissed as cosmetic in three consecutive runs. - R-118 (READY XS). An absent drive's union row reports the root filesystem's capacity as its own
(46 GiB / 9.2 % for a 4 GB drive) —
statfsCapacityatdisks.go:335-338statfs's a bare directory on root.observe.go:176-183guards the Observe path against exactly this; the union path does not.durable_idis still correct, so re-attach identity is safe — it is a false capacity, not a DR mis-id.
Register
documentation/backlog/OPEN-ITEMS.md — R-116 updated with the mechanism and the fix constraints;
R-117 and R-118 added. The single register edit this session makes.
Record correction
The brief's baseline controller 0.185.1 is the version the golden bakes; both demo boxes actually
run 0.186.0 (R-114 + R-112, 2026-07-29). Both are true of different things, and the distinction
matters because R-114's TargetAbsent branch exists only in 0.186.0. Confirmed: agent 0.115.0,
controller 0.186.0 live, hub 0.81.0 (manifest pin and live pod image agree), host-install
1.22.0, felhom.eu HEAD c3ce4c7.
Teardown and fences
Drill guest destroyed, scratch storage removed, mount unit deleted, secrets shred -u'd, VM powered
off, drill.qcow2 restored to virgin (the golden-bake fixture is exactly as found), scratch qcow2
and console dumps deleted. DooPlex /mnt/5_hdd at 24 %, unchanged.
Both demo boxes read-only throughout and re-verified after teardown: demo-hp local-lvm 38.83 %
(identical before/after and to Part 5), drill-r50 still stopped, felhom-pve felhom-backup still
active on /dev/sdb, both guests running, v0.115.0 untouched.
sess-d-0452c4 now reads STALE, and the delete gate refuses only on ONLINE
(hub/internal/web/customer_delete.go:220-228) — so it is now deletable; the command is recorded in
the audit rather than executed (customer delete runs external teardown plus a DB purge). sess-c is
also still present and was not recorded by the Session-C audit — same terms.
Not done, deliberately
No fresh controller gate-log / hub-event correlation: that observable was already captured live and identically twice, and only the payload was missing. Staging a controller meant a hub customer, a pairing, a golden fetch and a claim — the work that consumed the three prior sessions. The audit §8 correlates the payload to those existing measurements and labels that step as inference.