admin
699790b12d
Nothing in the repo said which machines are safe to break. The host table gave access and role and stopped there, so a session needing a victim had to guess -- and the guessing inverted: the two boxes that exist to be broken were treated as sacred, and DooPlex (the recovery chain) got used because it was the only box no spec had fenced. New documentation/runbooks/target-selection.md -- one page, three tiers, and per machine what is freely permitted / needs care / forbidden, each carrying its REASON so a rule can be correctly narrowed later instead of ossifying. States the selection rule positively (start at Tier 0; a Tier 2 box only when a task says so explicitly; an absent fence is not permission) and that fences name ACTS, not machines -- demo-hp's over-subscribed local-lvm is one dangerous storage, not a dangerous box. CLAUDE.md: host table gains a Blast radius column, gains the missing demo-hp row (it was where the drill VMs ran and it was not in the table at all), and a pointer line to the new runbook. CORRECTION to the spec's problem statement: the designation was not missing. The 2026-07-25 operator ruling naming the t740 as drill+build VM host -- explicitly "moved off DooPlex" -- already existed in operations/nodes.md. It sat where no session reads at start, while the prohibitions were repeated in every task spec. The defect is reachability of the ruling, not its absence, and the R-116 drill on DooPlex contradicted a written ruling rather than filling a vacuum. CORRECTION to the R-116 record, same commit: the baseline claimed controller 0.186.0 on both demo boxes. Only felhom-pve was sampled and generalised; demo-hp re-checked directly runs 0.185.1, so the fleet is split and R-114's TargetAbsent branch is absent from demo-hp. Fixed in the audit table and REPORT-r116-diag. Docs only -- no code, no build, no deploy, no host reconfigured, no host renamed.
7.0 KiB
REPORT-r116-diag — the /disks payload captured, R-116's mechanism isolated (2026-07-30)
Read-only diagnosis run by CC on DooPlex. No code written, nothing built, nothing published.
Full evidence: documentation/audits/DIAG-r116-disks-payload-2026-07-30.md.
A REPORT-*.md sibling, not the shared REPORT.md (CLAUDE.md parallel-session rule).
Outcome
Both goals met. The /disks read path is solved and written down verbatim, proven by a
present-drive control run first; and the absent-state payload was captured, which isolates the
mechanism.
R-116 is theory #1 — "the registry-union row writes false" — the theory that was raised, declared
wrong, and retracted. The retraction was the error.
In the absent state /disks returns 4 rows, not 3. The drive appears twice and the two facts the
controller needs sit on different rows:
| row | source | mount_path |
guest_path |
backup_target |
|---|---|---|---|---|
felhom-backup |
Observe (disks.go:196-284) |
"" |
"" |
true |
694034cc-… (the UUID) |
registry union (disks.go:297-339) |
/mnt/cel |
/mnt/felhom-drives/cel |
field ABSENT ⇒ false |
So the row holding the flag contributes no key to driveTargetByPath, and the row that owns the key
says false → isTarget[a.Path] is false → generic storage_disconnected. On return the rows
re-merge into one carrying both facts → specific backup_target_restored. Applying
intermediary.go:602-618 to the captured payloads gives PRESENT True / ABSENT False /
RETURNED True — the live asymmetry reproduced from payload alone.
The union row's MountPath survives the device because the union source is the systemd .mount unit
file (registry_known.go:40-75 via main.go:605→:764), which never reads the mount table. The
dedup at :298 therefore does not fire, because seen is keyed on the one field the absent state
empties (:290-295).
Theory #2 (the basis of the shipped v0.115.0) is false on both halves; #3 is false too
(isTarget["/mnt/cel"] is false as well). v0.115.0 is provably inert — its fallback calls
StablePathForRaw(""), which returns "" (intermediary.go:69-75), so it assigns nothing.
The read path (this cost two prior sessions — it should never cost again)
The token plaintext exists in exactly one place: bootstrap.json on the Proxmox host. The agent's
own store keeps SHA-256 hashes only (tokenstore.go:26-32), which is what defeated the earlier attempts.
ssh felhom-pve
B=/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json
TOK=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['token'])")
EP=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['endpoint'])")
curl -sS -k -H "Authorization: Bearer $TOK" "https://$EP/disks" | python3 -m json.tool
Control run, live felhom-pve, drive present: HTTP 200, 2483 bytes, 4 plausible rows — so Part 5's 0-rows-on-a-present-drive failure mode is excluded.
Where the absent state was staged
No new box. The existing DooPlex nested-PVE drill fixture (drill/drill.qcow2, snapshot virgin) —
my own host, zero production exposure, and it can hot-unplug a disk for a genuine device loss. Run with
the byte-identical live agent binary (sha256 f48544ad…, --version 0.115.0) and every
state-producing step through the real endpoints (format → assign → guest-attach →
backup/target). Its present-state row matched felhom-pve's control run field-for-field before it was
trusted. Non-production aspects (root/direct privileged mode, stubbed hub, a hand-written bearer-token
record, no controller) are enumerated in the audit §4.
Two new findings, filed not chased
- R-117 (READY M) — outranks R-116. After a detach/reattach the guest's bind is a dead mount:
host is healthy on the new device node, guest still names the old one, and
ls/write through it returnEIO— while/disksreportsattached+bound_under_parent:true+backup_target:true.planDriveGatestherefore takes theReturnbranch and restarts the customer's apps onto a dead namespace, reporting healthy, with no alarm on any channel. R-113's conjunction cannot catch it: one half is satisfied by the stale entry, the other by the new device, and neither compares them. This is the "stale bind" seen and dismissed as cosmetic in three consecutive runs. - R-118 (READY XS). An absent drive's union row reports the root filesystem's capacity as its own
(46 GiB / 9.2 % for a 4 GB drive) —
statfsCapacityatdisks.go:335-338statfs's a bare directory on root.observe.go:176-183guards the Observe path against exactly this; the union path does not.durable_idis still correct, so re-attach identity is safe — it is a false capacity, not a DR mis-id.
Register
documentation/backlog/OPEN-ITEMS.md — R-116 updated with the mechanism and the fix constraints;
R-117 and R-118 added. The single register edit this session makes.
Record correction
The brief's baseline controller 0.185.1 is the version the golden bakes. 0.186.0 (R-114 +
R-112, 2026-07-29) is what demo-felhom runs — demo-hp is still on 0.185.1, so the fleet is
split, and R-114's TargetAbsent branch exists only on demo-felhom. Confirmed: agent 0.115.0
(felhom-pve) / 0.113.0 (demo-hp), hub 0.81.0 (manifest pin and live pod image agree),
host-install 1.22.0, felhom.eu HEAD c3ce4c7.
Correction, 2026-07-30. As first written this section said 0.186.0 was what both demo boxes run. That was wrong — only felhom-pve's guest was sampled and the result generalised to the fleet. demo-hp re-checked directly →
0.185.1. Fixed here and in the audit's baseline table.
Teardown and fences
Drill guest destroyed, scratch storage removed, mount unit deleted, secrets shred -u'd, VM powered
off, drill.qcow2 restored to virgin (the golden-bake fixture is exactly as found), scratch qcow2
and console dumps deleted. DooPlex /mnt/5_hdd at 24 %, unchanged.
Both demo boxes read-only throughout and re-verified after teardown: demo-hp local-lvm 38.83 %
(identical before/after and to Part 5), drill-r50 still stopped, felhom-pve felhom-backup still
active on /dev/sdb, both guests running, v0.115.0 untouched.
sess-d-0452c4 now reads STALE, and the delete gate refuses only on ONLINE
(hub/internal/web/customer_delete.go:220-228) — so it is now deletable; the command is recorded in
the audit rather than executed (customer delete runs external teardown plus a DB purge). sess-c is
also still present and was not recorded by the Session-C audit — same terms.
Not done, deliberately
No fresh controller gate-log / hub-event correlation: that observable was already captured live and identically twice, and only the payload was missing. Staging a controller meant a hub customer, a pairing, a golden fetch and a claim — the work that consumed the three prior sessions. The audit §8 correlates the payload to those existing measurements and labels that step as inference.