admin
c3ce4c7b20
A fresh box running the fully shipped stack -- agent 0.115.0 from the Day-0 manifest plus controller 0.185.1 from the vouched golden, no hand-deploy -- still fired the GENERIC storage_disconnected on detach and the SPECIFIC backup_target_restored on return. backup_target_absent count 0. Identical to Session C. The v0.115.0 fix changed nothing observable. Part 4's three positive observables were all obtained before the run (registry newest 0.115.0, hub vouches 0.115.0, felhom-pve running 0.115.0 clean), so the publish step forgotten twice was not forgotten a third time, and the box demonstrably installed the fix under test. Discrimination FAILS: the target itself produced the generic event, so the two cannot be told apart regardless of the non-target leg -- which was therefore not staged. Reported as a fail, not as Session C's trivial pass. Over-correction guard PASSES: 0 ABSENT lines with the drive present, target degraded:false. THE HONEST PART. The fix targets a shape that does not occur live, and which shape does occur is NOT ISOLATED. With the drive detached PVE reports the storage inactive with zeroed fields -- a shape the unit fixture did not model. Three attempts to read the real /disks payload failed on token extraction across the ssh -> guest -> container layers, and a present-drive CONTROL query also returned 0 rows, proving the query was broken rather than the payload. Without that control this run would have recorded a third false mechanism, after "the union row writes false" (wrong, corrected yesterday) and "no row carries the guest path" (unverified). The leading hypothesis -- an inactive storage reaching Observe with an empty MountPath, so StablePathForRaw returns "" -- is consistent with the pvesm output but is NOT evidence and is recorded as such. Next session's first job is a working /disks read, with a present-drive control run FIRST, before any further code. agent v0.115.0 is published, vouched and INERT. Not reverted: reverting is itself a change, the runbook forbids fixing mid-run, and the code is tested and harmless. Capability-map row F stays PARTIAL, now citing the re-test. Teardown clean: pvesm status after == before (local-lvm 38.83%), guest 9201 and drill-r50 untouched. Customer sess-d pending the usual ONLINE-ages-to-DOWN gate.
5.4 KiB
R116-PART5-2026-07-30 — the fix shipped, C5 still fails, and the mechanism is NOT isolated
Run: RUNBOOK R-116 Part 5, executed by CC on DooPlex, 2026-07-30.
Result: FAILED. Agent 0.115.0 — the R-116 fix, installed from the Day-0 manifest — did not
change the outcome. The absent-drive alarm is still the generic one.
| Claim | Verdict |
|---|---|
C5 — backup_target_absent on detach, backup_target_restored on return, paired |
❌ FAIL — detach still fires storage_disconnected; return fires the specific event. Same asymmetry as Session C |
| Discrimination — target ⇒ specific, non-target ⇒ generic | ❌ FAIL — the target produced the generic event, so the two are still indistinguishable. Not achieved |
| No over-correction | ✅ PASS — 0 ABSENT lines with the drive present; target degraded:false |
1. What ran — the fully shipped stack
| Artifact | Version | How |
|---|---|---|
| agent | 0.115.0 | from the Day-0 manifest — Artifact manifest served for customer sess-d (agent=0.115.0 golden=0.185.1); host row sess-d-0452c4 … 0.115.0 ONLINE |
| controller | 0.185.1 | from the vouched golden — no hand-deploy this run, deliberately, so the whole stack is the shipped article |
Part 4's three positive observables were all obtained before the run: registry newest 0.115.0,
hub vouches 0.115.0, felhom-pve reports felhom-agent 0.115.0 active with 0 errors. The publish
step that was forgotten twice was not forgotten.
Fence reconfirmed: local-lvm 38.83 % before and after. Guest 9201 and drill-r50 untouched.
2. The observation
05:07:12 [gate] drive ABSENT /mnt/felhom-drives/cel — stopped+blocked 0 app(s)
05:07:15 Event pushed: storage_disconnected (error) — Meghajtó váratlanul leválasztva: Cel meghajto
backup_target_absent : 0 storage_disconnected : 1
05:11:43 [gate] drive RETURNED /mnt/felhom-drives/cel
05:11:43 Event pushed: backup_target_restored (info)
Identical to Session C. The gate fires correctly (R-113 holds, 0 s–3 s), SetDisconnected is reached,
and the wrong event goes out.
3. Why the fix did not fire — NOT ISOLATED, and that is the honest answer
The v0.115.0 change sets di.GuestPath on the Observe row when
di.GuestPath == "" && di.BackupTarget && t.BackingDevice == "". It is unit-tested and red-proofed
against a fixture built from the Session-C shape. On the live box it changed nothing.
What was established:
- With the drive detached,
pvesm statusreports the storage asfelhom-backup dir inactive 0 0 0, and PVE emitsunable to activate storage 'felhom-backup' - directory is expected to be a mount point but is not mounted: '/mnt/cel'. So the storage is listed but inactive with zeroed fields — a shape the unit fixture did not model (it supplied aMountPathand adisconnectedstate).
What was NOT established, and must not be guessed at again: the actual /disks payload. Three
attempts to read it failed on token extraction across the ssh → guest → container layers. A
control query with the drive PRESENT also returned 0 rows, which proves the query itself was
broken, not the payload. Without that control this run would have recorded a third false mechanism —
after "the union row writes false" (wrong, corrected) and "no row carries the guest path" (unverified).
So the honest state is: the fix addresses a shape that does not occur on the live box, and which
shape does occur is unknown. The leading hypothesis — that an inactive storage reaches Observe with
an empty MountPath, so StablePathForRaw("") returns "" and the fallback sets nothing — is
consistent with the pvesm output but is not evidence.
The next session's first job is a working /disks read, with a present-drive control run first,
before any further code. Everything after that is guesswork until it exists.
4. Also observed
- The stale bind survives the return and still names a dead device node. After the drive came back
as
/dev/sdc,findmnt /mnt/felhom-drives/celstill read/dev/sdb[/felhom-data]while the state reporteddegraded:false. Third consecutive run showing this; still not investigated. - Discrimination could not be reached. Because the target itself produced the generic event, there was no point staging the non-target leg: the two cannot be told apart regardless of what it emits. Reported as a FAIL, not as Session C's trivial pass.
5. Teardown
VM 9302 destroyed --purge, p5-images removed, scratch dir removed. pvesm status after == before
(local-lvm 38.83 %). Guest 9201 and drill-r50 untouched.
Customer sess-d + host sess-d-0452c4 remain — the delete gate refuses while the host still reads
ONLINE, as in both prior runs. Deletable once it ages to DOWN (>1 h):
POST /configs/sess-d/delete ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-d expect_hosts=1
6. Standing of agent v0.115.0
It is published and vouched, and it is inert — it adds a guest path in a case that apparently
does not arise, so it neither fixes nor breaks anything observable. It was NOT reverted: reverting is
itself a change, the runbook forbids fixing mid-run, and the code is well-tested and harmless. Whether
to keep, extend or revert it is the next session's decision, taken with a working /disks read in hand.