admin
699790b12d
Nothing in the repo said which machines are safe to break. The host table gave access and role and stopped there, so a session needing a victim had to guess -- and the guessing inverted: the two boxes that exist to be broken were treated as sacred, and DooPlex (the recovery chain) got used because it was the only box no spec had fenced. New documentation/runbooks/target-selection.md -- one page, three tiers, and per machine what is freely permitted / needs care / forbidden, each carrying its REASON so a rule can be correctly narrowed later instead of ossifying. States the selection rule positively (start at Tier 0; a Tier 2 box only when a task says so explicitly; an absent fence is not permission) and that fences name ACTS, not machines -- demo-hp's over-subscribed local-lvm is one dangerous storage, not a dangerous box. CLAUDE.md: host table gains a Blast radius column, gains the missing demo-hp row (it was where the drill VMs ran and it was not in the table at all), and a pointer line to the new runbook. CORRECTION to the spec's problem statement: the designation was not missing. The 2026-07-25 operator ruling naming the t740 as drill+build VM host -- explicitly "moved off DooPlex" -- already existed in operations/nodes.md. It sat where no session reads at start, while the prohibitions were repeated in every task spec. The defect is reachability of the ruling, not its absence, and the R-116 drill on DooPlex contradicted a written ruling rather than filling a vacuum. CORRECTION to the R-116 record, same commit: the baseline claimed controller 0.186.0 on both demo boxes. Only felhom-pve was sampled and generalised; demo-hp re-checked directly runs 0.185.1, so the fleet is split and R-114's TargetAbsent branch is absent from demo-hp. Fixed in the audit table and REPORT-r116-diag. Docs only -- no code, no build, no deploy, no host reconfigured, no host renamed.
118 lines
7.0 KiB
Markdown
118 lines
7.0 KiB
Markdown
# REPORT-r116-diag — the `/disks` payload captured, R-116's mechanism isolated (2026-07-30)
|
|
|
|
Read-only diagnosis run by CC on DooPlex. **No code written, nothing built, nothing published.**
|
|
Full evidence: `documentation/audits/DIAG-r116-disks-payload-2026-07-30.md`.
|
|
|
|
A `REPORT-*.md` sibling, not the shared `REPORT.md` (`CLAUDE.md` parallel-session rule).
|
|
|
|
## Outcome
|
|
|
|
**Both goals met.** The `/disks` read path is solved and written down verbatim, proven by a
|
|
present-drive control run *first*; and the absent-state payload was captured, which isolates the
|
|
mechanism.
|
|
|
|
**R-116 is theory #1 — "the registry-union row writes `false`" — the theory that was raised, declared
|
|
wrong, and retracted. The retraction was the error.**
|
|
|
|
In the absent state `/disks` returns **4 rows, not 3**. The drive appears twice and the two facts the
|
|
controller needs sit on different rows:
|
|
|
|
| row | source | `mount_path` | `guest_path` | `backup_target` |
|
|
|---|---|---|---|---|
|
|
| `felhom-backup` | Observe (`disks.go:196-284`) | `""` | `""` | **`true`** |
|
|
| `694034cc-…` (the UUID) | registry union (`disks.go:297-339`) | `/mnt/cel` | `/mnt/felhom-drives/cel` | **field ABSENT ⇒ `false`** |
|
|
|
|
So the row holding the flag contributes **no key** to `driveTargetByPath`, and the row that owns the key
|
|
says `false` → `isTarget[a.Path]` is `false` → generic `storage_disconnected`. On return the rows
|
|
re-merge into one carrying both facts → specific `backup_target_restored`. Applying
|
|
`intermediary.go:602-618` to the captured payloads gives PRESENT `True` / ABSENT `False` /
|
|
RETURNED `True` — **the live asymmetry reproduced from payload alone.**
|
|
|
|
The union row's `MountPath` survives the device because the union source is the systemd **`.mount` unit
|
|
file** (`registry_known.go:40-75` via `main.go:605`→`:764`), which never reads the mount table. The
|
|
dedup at `:298` therefore does not fire, because `seen` is keyed on the one field the absent state
|
|
empties (`:290-295`).
|
|
|
|
**Theory #2 (the basis of the shipped v0.115.0) is false on both halves**; **#3 is false too**
|
|
(`isTarget["/mnt/cel"]` is `false` as well). **v0.115.0 is provably inert** — its fallback calls
|
|
`StablePathForRaw("")`, which returns `""` (`intermediary.go:69-75`), so it assigns nothing.
|
|
|
|
## The read path (this cost two prior sessions — it should never cost again)
|
|
|
|
The token plaintext exists in exactly one place: `bootstrap.json` **on the Proxmox host**. The agent's
|
|
own store keeps SHA-256 hashes only (`tokenstore.go:26-32`), which is what defeated the earlier attempts.
|
|
|
|
```bash
|
|
ssh felhom-pve
|
|
B=/var/lib/felhom-agent/guests/9201/bootstrap/bootstrap.json
|
|
TOK=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['token'])")
|
|
EP=$(python3 -c "import json;print(json.load(open('$B'))['local_api']['endpoint'])")
|
|
curl -sS -k -H "Authorization: Bearer $TOK" "https://$EP/disks" | python3 -m json.tool
|
|
```
|
|
|
|
Control run, live felhom-pve, drive present: **HTTP 200, 2483 bytes, 4 plausible rows** — so Part 5's
|
|
0-rows-on-a-present-drive failure mode is excluded.
|
|
|
|
## Where the absent state was staged
|
|
|
|
No new box. The existing DooPlex **nested-PVE drill fixture** (`drill/drill.qcow2`, snapshot `virgin`) —
|
|
my own host, zero production exposure, and it can hot-unplug a disk for a genuine device loss. Run with
|
|
the **byte-identical live agent binary** (`sha256 f48544ad…`, `--version` 0.115.0) and every
|
|
state-producing step through the real endpoints (`format` → `assign` → `guest-attach` →
|
|
`backup/target`). Its present-state row matched felhom-pve's control run field-for-field before it was
|
|
trusted. Non-production aspects (root/direct privileged mode, stubbed hub, a hand-written bearer-token
|
|
record, no controller) are enumerated in the audit §4.
|
|
|
|
## Two new findings, filed not chased
|
|
|
|
- **R-117 (READY M) — outranks R-116.** After a detach/reattach the guest's bind is a **dead mount**:
|
|
host is healthy on the new device node, guest still names the old one, and `ls`/write through it
|
|
return **`EIO`** — while `/disks` reports `attached` + `bound_under_parent:true` + `backup_target:true`.
|
|
`planDriveGates` therefore takes the `Return` branch and **restarts the customer's apps onto a dead
|
|
namespace, reporting healthy, with no alarm on any channel.** R-113's conjunction cannot catch it:
|
|
one half is satisfied by the stale entry, the other by the new device, and neither compares them.
|
|
This is the "stale bind" seen and dismissed as cosmetic in three consecutive runs.
|
|
- **R-118 (READY XS).** An absent drive's union row reports the **root filesystem's** capacity as its own
|
|
(46 GiB / 9.2 % for a 4 GB drive) — `statfsCapacity` at `disks.go:335-338` statfs's a bare directory on
|
|
root. `observe.go:176-183` guards the Observe path against exactly this; the union path does not.
|
|
`durable_id` is still correct, so re-attach identity is safe — it is a false capacity, not a DR mis-id.
|
|
|
|
## Register
|
|
|
|
`documentation/backlog/OPEN-ITEMS.md` — R-116 updated with the mechanism and the fix constraints;
|
|
R-117 and R-118 added. The single register edit this session makes.
|
|
|
|
## Record correction
|
|
|
|
The brief's baseline `controller 0.185.1` is the version the **golden bakes**. **0.186.0** (R-114 +
|
|
R-112, 2026-07-29) is what **demo-felhom** runs — **demo-hp is still on 0.185.1**, so the fleet is
|
|
split, and R-114's `TargetAbsent` branch exists only on demo-felhom. Confirmed: agent **0.115.0**
|
|
(felhom-pve) / **0.113.0** (demo-hp), hub **0.81.0** (manifest pin and live pod image agree),
|
|
host-install **1.22.0**, `felhom.eu` HEAD `c3ce4c7`.
|
|
|
|
> **Correction, 2026-07-30.** As first written this section said 0.186.0 was what *both* demo boxes run.
|
|
> That was wrong — only felhom-pve's guest was sampled and the result generalised to the fleet. demo-hp
|
|
> re-checked directly → `0.185.1`. Fixed here and in the audit's baseline table.
|
|
|
|
## Teardown and fences
|
|
|
|
Drill guest destroyed, scratch storage removed, mount unit deleted, secrets `shred -u`'d, VM powered
|
|
off, **`drill.qcow2` restored to `virgin`** (the golden-bake fixture is exactly as found), scratch qcow2
|
|
and console dumps deleted. DooPlex `/mnt/5_hdd` at 24 %, unchanged.
|
|
|
|
Both demo boxes **read-only throughout** and re-verified after teardown: demo-hp `local-lvm` **38.83 %**
|
|
(identical before/after and to Part 5), `drill-r50` still stopped, felhom-pve `felhom-backup` still
|
|
active on `/dev/sdb`, both guests running, **v0.115.0 untouched**.
|
|
|
|
`sess-d-0452c4` now reads **STALE**, and the delete gate refuses only on ONLINE
|
|
(`hub/internal/web/customer_delete.go:220-228`) — so it **is** now deletable; the command is recorded in
|
|
the audit rather than executed (customer delete runs external teardown plus a DB purge). **`sess-c` is
|
|
also still present and was not recorded by the Session-C audit** — same terms.
|
|
|
|
## Not done, deliberately
|
|
|
|
No fresh controller gate-log / hub-event correlation: that observable was already captured live and
|
|
identically twice, and only the payload was missing. Staging a controller meant a hub customer, a
|
|
pairing, a golden fetch and a claim — the work that consumed the three prior sessions. The audit §8
|
|
correlates the payload to those existing measurements and labels that step as inference.
|