From 06d778839251ea1ff59989e5b24056cec451f5ac Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Wed, 29 Jul 2026 20:55:48 +0200 Subject: [PATCH] Session C: R-113/R-114/R-112 PROVEN LIVE; C5 fails on a new defect (R-116) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full ISO/PAIRING run on a fresh nested box. Agent 0.114.0 came from the Day-0 manifest -- the SHIPPED binary -- so C5 tested the real artifact. Controller 0.186.0 hand-deployed after install per the §3.1 ruling; the vouched golden bakes 0.185.1, so C3/C4 prove the code not the shipped golden, and that lag is filed against R-115 rather than a new ID. R-113 PROVEN: detach 18:43:50, gate fired 18:43:54 -- four seconds, where E-2d measured zero over 4.5 minutes -- and SetDisconnected was reached. It fired on exactly the shape that defeated it: raw /mnt/mentes NOT mounted while the bind /mnt/felhom-drives/mentes still read /dev/sdb[/felhom-data]. R-114 PROVEN: with the target absent the page rendered the absent copy, the system-disk copy 0 and the offer block 0. Both of E-2d's falsehoods are gone. R-112 PROVEN: the banner reached a customer's page for the first time. Healthy renders nothing, proven POSITIVELY -- idle delta 0 /backup/tiers calls, page load delta +1, single caller, so the seam ran and chose silence. C5 FAILED on a fourth, separate defect. The alarm fires but as the GENERIC storage_disconnected, while the recovery is the SPECIFIC backup_target_restored -- a pair an operator cannot match, which is what notifyDriveReturned's own comment forbids. backup_target_absent count 0 across the run. Root cause: the drive is TWO /disks rows and BackupTarget and GuestPath sit on different ones; absent they separate, on return they rejoin. v0.184.1 fixed the keying, not this. Only reachable because R-113 made the gate fire at all. Filed as R-116. Mirror + over-correction guard PASS: non-target drive -> storage_disconnected, backup_target_absent 0; both drives present -> 0 ABSENT lines and the target stayed healthy. Caveat recorded: the mirror passes trivially because the target also produced the generic event. E-2 and E-2d CLOSED as partially proven with R-116 the one named open leg, per the runbook's §9 rule decided in advance rather than mid-run. Capability map NOT touched: it has no E-2 rows at all, so nothing could move to PROVEN-LIVE. Creating them is a design act, not a validation act. Teardown clean: pvesm status after == before (local-lvm 38.78%), guest 9201 and drill-r50 untouched. Customer delete attempted and correctly refused while the host still reads ONLINE; command recorded for once it ages to DOWN. --- REPORT-session-c.md | 94 ++++++++ documentation/audits/SESSION-C-2026-07-29.md | 220 +++++++++++++++++++ documentation/backlog/OPEN-ITEMS.md | 11 +- documentation/backlog/ROADMAP.md | 1 + 4 files changed, 321 insertions(+), 5 deletions(-) create mode 100644 REPORT-session-c.md create mode 100644 documentation/audits/SESSION-C-2026-07-29.md diff --git a/REPORT-session-c.md b/REPORT-session-c.md new file mode 100644 index 0000000..39284ad --- /dev/null +++ b/REPORT-session-c.md @@ -0,0 +1,94 @@ +# REPORT — Session C: R-113, R-114, R-112 proven on a real box; C5 fails on a new defect (2026-07-29) + +`RUNBOOK-session-c-2026-07-29.md`. Full evidence: `documentation/audits/SESSION-C-2026-07-29.md`. +Root `REPORT.md` untouched. + +## Verdicts + +| Claim | Fix | Verdict | +|---|---|---| +| **C4** — offer appears and moves the target | R-112 | ✅ **PASS** | +| **C3** — absent target tells the truth, offers nothing | R-114 | ✅ **PASS** | +| **C5** — `backup_target_absent` + `backup_target_restored` | R-113 | ❌ **FAIL** — generic alarm, specific recovery → **R-116** | + +**All three shipped fixes work.** R-113's gate fires in **4 seconds** (E-2d measured zero over 4½ +minutes). R-114's message is correct. R-112's banner reaches the customer. C5 fails on a **fourth, +separate defect that was unreachable until R-113 made the gate fire at all.** + +## What the drill box ran + +Agent **0.114.0 from the Day-0 manifest** — the shipped binary, so C5 tested the real artifact and +closed R-115's observation 1 for R-113. Controller **0.186.0 hand-deployed** after install (§3.1 +ruling (a)); the vouched golden bakes 0.185.1, so **C3/C4 prove the code, not the shipped golden** — +that lag is filed against R-115, not a new ID. + +## The three headline observables + +**R-113** — detach at 18:43:50, gate at **18:43:54**, on exactly the shape that defeated it before: +``` +raw /mnt/mentes : NOT mounted +bind /mnt/felhom-drives/mentes : /dev/sdb[/felhom-data] <- the stale bind SURVIVED +``` + +**R-114** — with the target absent: absent copy 1, **system-disk copy 0**, **offer block 0**. Both of +E-2d's falsehoods gone. + +**R-112** — banner element 1 and the never-configured copy in the HTML; after the wizard, the offer +with `data-path="/mnt/felhom-drives/mentes"`. Healthy renders nothing, **proven positively**: idle +delta 0 `/backup/tiers` calls, page-load delta **+1**, single caller ⇒ the seam ran and chose silence. + +**Decline path proven** (registration confers no role), `restart_required:true`, agent did **not** +self-restart, in-flight check recorded before I restarted it, wrapper created the storage at the +drive's own mountpoint. + +## C5's failure + +``` +absent : Event pushed: storage_disconnected (error) <- GENERIC +return : Event pushed: backup_target_restored (info) <- SPECIFIC +``` + +`backup_target_absent` count **0** across the whole run. The alarm and its recovery cannot be matched +— precisely what `notifyDriveReturned`'s own comment forbids. + +**Root cause (R-116):** `driveTargetByPath` builds `out[GuestPath] = d.BackupTarget`, but the drive is +**two `/disks` rows** and the flag and the guest path sit on different ones — the `felhom-backup` +storage row carries `BackupTarget: true` and gets a guest path only while classified user-data; the +registry union row carries the guest path and **never assigns `BackupTarget`**. Absent ⇒ they separate +⇒ generic. Return ⇒ they rejoin ⇒ specific. v0.184.1 fixed the *keying*, not this. + +## Mirror + over-correction guard — PASS, with a caveat + +Non-target drive detached ⇒ `storage_disconnected`, `backup_target_absent` count 0. **Over-correction +guard passes**: both drives present ⇒ 0 ABSENT lines, target stayed healthy — R-113's stricter presence +did not make a healthy drive read absent. **Caveat: the mirror passes trivially**, because the target +also produced the generic event; it confirms no over-correction but cannot confirm discrimination. + +## Record + +- `OPEN-ITEMS.md` — **R-113, R-114, R-112 → SHIPPED + PROVEN-LIVE**; **R-116 opened** (READY (S), P1); + **E-2 and E-2d CLOSED as partially proven** with R-116 as the one named open leg, per the runbook's + §9 decided-in-advance rule. +- `ROADMAP.md` — R-116 under P1. +- **Capability map NOT touched** — it still has **no E-2 / backup-target rows at all**, so no row could + be moved to PROVEN-LIVE. Creating them is a design act, not a validation act. Third session running + that this has been noted. + +## Teardown + +VM destroyed, storage removed, **`pvesm status` after == before** (`local-lvm` 38.78 %), guest 9201 and +`drill-r50` untouched. **Customer ruling: DELETE**; attempted and correctly refused (`host … is +ONLINE`) — deletable once the destroyed host ages to DOWN (>1 h), command recorded in `OPEN-ITEMS.md`. + +## What did not happen + +`backup_target_absent` never fired, so its severity, Hungarian copy and hub routing remain unexercised +— R-116 blocks them. The offer was accepted via the endpoint the button POSTs, not a browser click (no +browser automation on DooPlex); the rendered control and its non-auto-submission were verified in HTML. +The stale bind still naming a dead device node after return was observed, not investigated. + +## The arc + +E-2 ends here. Its stated definition of done is **R-106 + R-109, R-108 and D5** — none of which this +detour touched. The detour was worth taking: it found six real defects (R-111 through R-116), four of +them customer-affecting, none of which any unit suite had caught. diff --git a/documentation/audits/SESSION-C-2026-07-29.md b/documentation/audits/SESSION-C-2026-07-29.md new file mode 100644 index 0000000..39509b2 --- /dev/null +++ b/documentation/audits/SESSION-C-2026-07-29.md @@ -0,0 +1,220 @@ +# SESSION-C-2026-07-29 — R-113, R-114, R-112 proven on a real box; C5 fails on a new defect + +**Run:** `RUNBOOK-session-c-2026-07-29.md`, executed by CC on DooPlex, 2026-07-29. +**Session C of three.** A → R-113 (agent v0.114.0) · B → R-114 + R-112 (controller v0.186.0) · **C → this.** + +| Claim | Fix | Verdict | +|---|---|---| +| **C4** — the offer appears and moves the target when accepted | R-112 | ✅ **PASS** | +| **C3** — with the target absent the customer is told the drive is missing, and offered nothing | R-114 | ✅ **PASS** | +| **C5** — drive loss produces `backup_target_absent` (error) and `backup_target_restored` (info) | R-113 | ❌ **FAIL** — the alarm fires, but as the **generic** event; the recovery is the **specific** one. Unmatched pair → **R-116** | + +**All three fixes are proven to work.** R-113's gate fires, R-114's message is correct, R-112's banner +reaches the customer. C5 fails on a **fourth, separate defect** that only became reachable *because* +R-113 made the gate fire at all — it could not have been seen before. + +--- + +## 1. Baselines, and what the drill box actually ran + +| Artifact | Vouched / used | How it got there | +|---|---|---| +| agent | **0.114.0** | **from the Day-0 manifest** — `Artifact manifest served for sess-c (agent=0.114.0 golden=0.185.1)`. C5 therefore tested the **shipped** binary, closing R-115 observation 1 for R-113 | +| golden | 0.185.1 | vouched; bakes controller 0.185.1 | +| controller | **0.186.0** | **hand-deployed after install** (§3.1 ruling (a)) via the bootstrap path — `0.185.1 → 0.186.0`, `Up 21 seconds (healthy)` | +| hub | 0.81.0 | unchanged | +| host-install | 1.22.0 | unchanged | + +Repos at run time: controller `b331f18`, agent `b58d7bc`, felhom.eu `af518ba`, all clean. + +**§3.1 ruling — (a), deploy 0.186.0 by hand.** A bake is its own session and would have inserted one +before any verdict; the install path is already proven by E-2d. **Caveat, as required: C3/C4 prove the +CODE, not the SHIPPED GOLDEN.** The vouched golden lagging a controller release is R-115's shape and is +filed there, not as a new ID. + +**§3.2 cadence** — `driveGateLoop` registered `server.go:224`, defined `intermediary.go:328`, ticker +**30 s** (`:339`) ⇒ C5 budget 60 s. **§3.3** — SCSI hot-detach, volume survives as `unused0`. + +Fence re-measured before and after: `local-lvm` **38.78 %** both times. + +## 2. Timeline (VM 9301 `sessionc`, nested PVE on demo-hp) + +| UTC | Event | +|---|---| +| 18:29 | VM created — one disk, q35/OVMF SB-off, `hotplug disk`, outside the `felhom` pool | +| 18:33:47 | registered unclaimed, pairing code `2XT-3PV` | +| 18:34:08 | **bound to `sess-c` by CC** — no operator STOP (E-2d retired it) | +| 18:34:57 | manifest served: **agent 0.114.0**, golden 0.185.1 | +| 18:38:03 | `controller_started (0.185.1)` — install complete, bind→controller in **3 m 55 s** | +| ~18:40 | controller **0.186.0** deployed; dashboard claimed with the emailed code | +| 18:43:50 | **target drive hot-detached** | +| **18:43:54** | **gate fired — 4 seconds** | +| 18:47:01 | reattached | +| 18:47:24 | gate RETURNED (23 s) | +| 18:50:32 | non-target drive detached (mirror) | +| 18:50:54 | gate fired (22 s) | + +## 3. C4 — PASS (R-112 + the assign flow) + +**The banner reached a customer's page for the first time.** E-2d could only prove the JSON. + +Never-configured box, before any second drive: + +``` +GET /backups http=200 + banner element : 1 + never-configured copy : 1 <- "ugyanazon a lemezen van" IS in the HTML + absent copy : 0 + offer block : 0 <- correct: no eligible drive registered yet +``` + +After attaching a 100 GB disk and bringing it in **through the dashboard wizard**: + +``` + banner element : 1 offer block : 1 data-path="/mnt/felhom-drives/mentes" +``` + +**Decline path — PROVEN.** After registering and *not* accepting: `target` still `local`, **no** +`felhom-backup` storage, `agent.json` `local_backup_target = local`. Registration confers no role. + +**Accept:** `{"assigned":"/mnt/felhom-drives/mentes","restart_required":true}`. +**The agent did NOT self-restart** — `ActiveEnterTimestamp` 20:35:15 CEST, unchanged at 20:42:20. +**In-flight check performed and recorded before restarting:** 0 `vzdump` processes, 0 backup lines in +the agent journal (5 min), 0 running PVE tasks. Wrapper created `dir: felhom-backup, path /mnt/mentes, +is_mountpoint 1` — the drive's **own** mountpoint. + +**After restart, healthy renders nothing — and it is proven POSITIVELY, not by an empty page:** + +``` + banner element 0 · offer block 0 · degraded copy 0 · absent copy 0 + /backup/tiers calls before 6 · after idle 8s 6 (delta 0) · after page load 7 (delta +1) +``` + +That call has exactly one caller (`resolveBackupTargetState`), so **the seam ran and chose silence**. + +## 4. C3 — PASS (R-114) + +With the target drive detached, the customer's page: + +``` + banner element : 1 + ABSENT copy (correct) : 1 <- "vissza nem csatlakoztatod" + system-disk copy (BAD): 0 <- E-2d saw THIS + offer block (BAD) : 0 <- E-2d offered the VANISHED drive +``` + +API: `{"degraded":true,"target":"felhom-backup","message":"A rendszermentés meghajtója nem érhető el — +amíg vissza nem csatlakoztatod, a teljes rendszermentés nem készül el."}` + +Both of E-2d's falsehoods are gone. + +## 5. C5 — FAIL, and R-113 is nonetheless PROVEN + +**R-113 works. This is the headline.** E-2d measured **zero** gate activity over 4½ minutes. Here: + +``` +18:43:54 [INFO] [settings] Storage path /mnt/felhom-drives/mentes disconnected=true +18:43:54 [WARN] [gate] drive ABSENT /mnt/felhom-drives/mentes — stopped+blocked 0 app(s): [] +``` + +**4 seconds**, and `SetDisconnected` reached — the flag R-114 and the offer suppression depend on. +And it fired on **exactly the shape that defeated it before**: + +``` +raw /mnt/mentes : NOT mounted <- device-bound, died with the device +bind /mnt/felhom-drives/mentes : /dev/sdb[/felhom-data] <- the stale bind SURVIVED +``` + +The bind outlived the device, as always; the gate now sees past it. + +**What fails is which event is sent.** + +``` +absent : Event pushed: storage_disconnected (error) <- GENERIC +return : Event pushed: backup_target_restored (info) <- SPECIFIC +``` + +C5 required `backup_target_absent`. It never fired — `backup_target_absent` count **0** across the run. +Worse than a missing label: the alarm and its recovery are a **mismatched pair**, which is precisely +what `notifyDriveReturned`'s own comment forbids — *"a target that alarmed as backup_target_absent has +to recover as backup_target_restored, not as a generic reconnect the operator cannot match"*. Here it +is inverted: generic alarm, specific recovery. + +### Root cause, established at source + +`notifyDriveAbsent` (`intermediary.go:635-646`) branches on `isTarget[path]`, built by +`driveTargetByPath` (`:602-616`) as `out[GuestPath] = d.BackupTarget` / `out[MountPath] = d.BackupTarget`. +The gate's `a.Path` is the **guest** path. + +The drive is represented by **two `/disks` rows**, and the flag and the guest path sit on different ones: + +| Row | Source | `BackupTarget` | `GuestPath` | +|---|---|---|---| +| the `felhom-backup` storage | Observe path, `felhom-agent/internal/localapi/disks.go:211` (`t.Name == primaryTargetID`) | **true** | set only when role is user-data | +| the registry drive | union path, `disks.go:265-267` — **never assigns `BackupTarget` at all** | **false** | `/mnt/felhom-drives/mentes` | + +While the device is absent the target-flagged row loses the classification that gives it a guest path, +so `out["/mnt/felhom-drives/mentes"]` is written **false** by the union row and the specific branch is +skipped. On **return** the rows rejoin and the specific event fires — which is exactly the asymmetry +observed. v0.184.1 fixed the *keying* (both keys are mapped); it did not fix the flag being absent from +the row that carries the guest path. + +*(The precise role-classification step that drops the guest path while absent was not isolated — the +two-row split and the observed absent/return asymmetry are the evidence.)* + +**Filed as R-116.** Not fixed: a defect found in a validation run is a finding. + +## 6. Mirror scenario + over-correction guard — PASS, with a caveat + +A **non-target** drive (`adat`, 70 GB, registered but never assigned) detached at 18:50:32: + +``` +18:50:54 [WARN] [gate] drive ABSENT /mnt/felhom-drives/adat — stopped+blocked 0 app(s) +18:50:54 [INFO] Event pushed: storage_disconnected (error) — Meghajtó váratlanul leválasztva: Adat meghajto + backup_target_absent count : 0 storage_disconnected count : 1 +``` + +**Over-correction guard — the real risk of R-113 — PASSES.** With both drives present the gate produced +**0** ABSENT lines, and the target stayed `degraded:false` with banner element 0 throughout the whole +mirror sequence. R-113's stricter presence did not make a healthy drive read absent. + +**Caveat, stated rather than glossed:** because the *target* drive also produced the generic event, the +mirror passes **trivially**. It confirms no over-correction; it cannot confirm specific-vs-generic +discrimination, because nothing produced the specific event on absence at all. + +## 7. Findings + +1. **R-116 (new)** — the absent alarm is generic while the recovery is specific; the pair cannot be + matched by an operator. Root cause §5. **This defect was unreachable before R-113**: the gate never + ran, so the branch was never taken. Fixing one bug exposed the next, which is what a validation run + is for. +2. **The stale bind survives a return, still naming a dead device node.** After the drive came back as + `/dev/sdc`, `findmnt /mnt/felhom-drives/mentes` still read `/dev/sdb[/felhom-data]` while the state + reported `degraded:false`. E-2d noticed this and did not chase it; it is now confirmed. Whether apps + writing through that path reach the live device was **not** established — worth its own look. +3. **The capability map still has no E-2 / backup-target rows at all**, so no row could be moved to + PROVEN-LIVE. Creating them is a design act, not a validation act; filed rather than invented (third + session in a row this has been noted). + +## 8. Teardown + +VM 9301 destroyed `--purge`, `sc-images` storage removed, scratch dir removed. +**`pvesm status` after == before** (`local-lvm` **38.78 %**, `felhom-backup` 931059224 KiB available). +**Guest 9201 and `drill-r50` untouched.** + +**Customer ruling: DELETE** (R-93's lesson — no half-real fixtures). Attempted and **correctly refused**: +`delete sess-c REFUSED: host sess-c-8158e4 is ONLINE`. The VM is destroyed so no further reports come; +the host ages OK → WARN (30 m) → DOWN (>1 h) and is then deletable. Command, unchanged from E-2d: + +``` +POST /configs/sess-c/delete ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-c expect_hosts=1 +``` + +## 9. What did not happen + +- **`backup_target_absent` never fired**, so its severity, its Hungarian customer copy and its hub + routing are all still unexercised end-to-end. R-116 blocks them. +- **The offer control was accepted via the endpoint it POSTs**, not by a browser click — no browser + automation exists on DooPlex. The rendered control, its `data-path` and its non-auto-submission were + verified in the HTML. +- The stale-bind-after-return question (finding 2) was observed, not investigated. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index ffd6145..53d2634 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -11,14 +11,15 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha |---|---|---|---|---|---| | **R-88a** | ~~Failing backup re-quiesces every 5 min, no backoff~~ | **SHIPPED** (controller v0.176.0, 2026-07-27) | — | Live on both boxes; breaker 15m→4h, per-tier, never permanent | — | | **R-88b** | ~~`/backup/due` cannot say *unknown*~~ | **SHIPPED + PROVEN-LIVE** (agent v0.105.0 + controller v0.178.0, 2026-07-27) | — | `age_state=unknown` captured on real hardware during a deliberate ep0 outage; controller deferred, **zero app stacks stopped** | — | -| **E-2d** | **Prove E-2 on a fresh VM on the t740** — the only remaining route to four unproven items: a real `felhom-host-install.sh` **1.22.0** run (never done), Case B naturally (single-drive install renders the degraded banner without degrading a live box), a **claimable** customer so the three claim-gated items stop being gated, then add a drive (the offer) and unplug it (`backup_target_absent` end-to-end) | **READY (M)** | — | **Space checked 2026-07-29 — NOT a blocker, with one constraint: the VM disk must NOT go on `local-lvm`.** That thin pool is over-subscribed (144 GB allocated against 54 GB, 38.8% used) on a box running a live customer guest, and a full thin pool corrupts every guest on it. `local` has only 23.7 GB and sits on `pve-root`. **Use `/mnt/nvme-1tb` (888 GB free).** Caveat found: a dir storage at a SUBDIRECTORY there will fail the agent's `exactMount` check and report `disconnected` in the host report — cosmetic, but decide the placement deliberately. **Do NOT unblock drill-r50** (deliberately blocked; unblocking it means the fixture stops representing anything real). **CORRECTED 2026-07-29 — the ISO is not an obstacle to the 1.22.0 proof, it is the best route to it.** `felhom-bootstrap.sh:96` fetches from `https://felhom.eu/scripts/felhom-host-install.sh`, not the hub, and that URL serves **1.22.0** (git-sync from `main`, ≤30 s — live fetch confirmed). A fresh ISO install therefore exercises 1.22.0 **automatically** — which makes the ISO leg the *stronger* proof, because it is the real customer path rather than a proxy for it, and it retires the "run 1.22.0 manually" step. **The Phase 0 question that decides the shape is ANSWERED — it was a source read, and the answer is yes.** A fresh VM with no baked customer-id lands in **PAIRING** mode (`felhom-bootstrap.sh:537-541`), not **DIRECT** (`:312`), and only DIRECT passes `--customer-id / --mode / --passphrase-file`. But on a 200 from `/api/v1/appliance/poll` the pairing loop writes the hub-delivered `FELHOM_CUSTOMER_ID` + `FELHOM_RETRIEVAL_PASSPHRASE` into the 0600 env, re-sources it and calls `run_direct` **in the same invocation** (`:495-499`) — so pairing reaches the identical installer invocation (`:322-343` — the single `$INSTALL_URL` fetch at `:322-330`, the `--customer-id/--mode/--hub-url/--passphrase-file` args array at `:334`, and the `bash "$SCRIPT_TMP" "${args[@]}"` call itself at `:343`) and the customer it yields is the one the operator bound, i.e. claimable. **So the ISO leg is the spine**; a manual 1.22.0 run is not needed as a separate scenario. **RUN ATTEMPTED 2026-07-29 — STOPPED AT PHASE 0, no VM created, nothing touched (`audits/E2D-fresh-vm-2026-07-29.md`).** The blocker is **R-111**: a fresh box installs **agent 0.96.0 + controller 0.161.0**, not `main`'s 0.113.0/0.185.1, so **C3/C4/C5 test surfaces that do not exist on it** — the degraded banner + `GET /api/storage/backup-target` landed in controller **v0.185.1** (`cdaeb36`) with the copy in **v0.185.0** (`3f7cf2a`); `backup_target_absent` in **v0.184.0** (`c1a63de`); the offer's apply needs agent **v0.113.0** (`58b598b`). **C1 (a real rc=0 1.22.0 install) and C2 (Case B natural, `configure_backup_target()` `felhom-host-install.sh:627`, warnings `:653-655`) remain ACHIEVABLE TODAY** — both are installer-side and host-install is served at 1.22.0. **C3 unblocks cheaply** by raising the hub global floor 0.156.0 → ≥0.185.0 (controller 0.185.1 IS in the registry): measured fleet impact is **nil** — both demo boxes already run 0.185.1, and Peti is DOWN 14 d and already below the current floor. **C4/C5 need R-111 first.** Phase 0 answers are all recorded in the audit, so a resumed run does not re-derive them: drive-gate cadence **30 s** (`intermediary.go:337`, registered `server.go:217`) ⇒ a 60 s two-cycle budget; hot-detach available (`virtio-scsi-single` + default hotplug, VM 300 is the working reference); ISO present (`…v1.25.0-nested-vm-generic-mkimage.iso`); `/mnt/nvme-1tb` 888 G free and the `local-lvm` fence re-measured (38.77 %, unchanged). **The §5.1a operator STOP is retired** — `HUB_PW` is in `~/.config/credentials` and hub auth was verified, so CC can bind. **One open decision carried forward:** where the VM disk's dir storage goes, since `local-lvm` is forbidden and `felhom-backup` is the live backup target — see the audit §4.1. **RUN EXECUTED 2026-07-29 after R-111 was fixed — `audits/E2D-fresh-vm-2026-07-29.md`.** Full ISO/PAIRING route on a nested VM on demo-hp; bind → running controller in **3 m 35 s**; teardown clean (`pvesm status` after == before, `local-lvm` 38.77 %, guest 9201 untouched). **C1 PROVEN** (`felhom-host-install v1.22.0`, `Day-0 provision SUCCESS`, guest 9201 running, golden = the one baked 20 min earlier). **C2 PROVEN** (both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort). **C3/C4 PARTIAL — API exact and complete, but customer-invisible → R-112.** **C5 FAILED → R-113.** C4's decline path PROVEN (registration confers no role), `restart_required:true` PROVEN, agent did **not** self-restart, E-2a wrapper created the storage at the drive's own mountpoint, healthy renders nothing. **E-2d's own premise needed amending:** a fresh install does NOT yield a CC-drivable claimable customer — the claim code is bcrypt-hashed and email-only, so one operator relay was required (and the claim flow is now proven end to end). **This row stays OPEN only for the residue:** C5 re-test after R-113, and the C3/C4 UI legs after R-112 | CC | +| **E-2d** | **Prove E-2 on a fresh VM on the t740** — the only remaining route to four unproven items: a real `felhom-host-install.sh` **1.22.0** run (never done), Case B naturally (single-drive install renders the degraded banner without degrading a live box), a **claimable** customer so the three claim-gated items stop being gated, then add a drive (the offer) and unplug it (`backup_target_absent` end-to-end) | **CLOSED — PARTIALLY PROVEN** (Session C, 2026-07-29) | — | **CLOSED by `audits/SESSION-C-2026-07-29.md`.** C1/C2 proven in E-2d; **C3 and C4 PROVEN LIVE** this session (R-114, R-112); **C5 FAILED** — the gate fires and an alarm reaches the hub, but it is the generic event, not `backup_target_absent` (→ **R-116**, the one named open leg). Per the runbook's §9, decided in advance: a failed claim closes E-2 as partially proven with a named leg rather than re-running. The arc's stated definition of done is R-106+R-109, R-108 and D5 — none of which this detour touched. **Space checked 2026-07-29 — NOT a blocker, with one constraint: the VM disk must NOT go on `local-lvm`.** That thin pool is over-subscribed (144 GB allocated against 54 GB, 38.8% used) on a box running a live customer guest, and a full thin pool corrupts every guest on it. `local` has only 23.7 GB and sits on `pve-root`. **Use `/mnt/nvme-1tb` (888 GB free).** Caveat found: a dir storage at a SUBDIRECTORY there will fail the agent's `exactMount` check and report `disconnected` in the host report — cosmetic, but decide the placement deliberately. **Do NOT unblock drill-r50** (deliberately blocked; unblocking it means the fixture stops representing anything real). **CORRECTED 2026-07-29 — the ISO is not an obstacle to the 1.22.0 proof, it is the best route to it.** `felhom-bootstrap.sh:96` fetches from `https://felhom.eu/scripts/felhom-host-install.sh`, not the hub, and that URL serves **1.22.0** (git-sync from `main`, ≤30 s — live fetch confirmed). A fresh ISO install therefore exercises 1.22.0 **automatically** — which makes the ISO leg the *stronger* proof, because it is the real customer path rather than a proxy for it, and it retires the "run 1.22.0 manually" step. **The Phase 0 question that decides the shape is ANSWERED — it was a source read, and the answer is yes.** A fresh VM with no baked customer-id lands in **PAIRING** mode (`felhom-bootstrap.sh:537-541`), not **DIRECT** (`:312`), and only DIRECT passes `--customer-id / --mode / --passphrase-file`. But on a 200 from `/api/v1/appliance/poll` the pairing loop writes the hub-delivered `FELHOM_CUSTOMER_ID` + `FELHOM_RETRIEVAL_PASSPHRASE` into the 0600 env, re-sources it and calls `run_direct` **in the same invocation** (`:495-499`) — so pairing reaches the identical installer invocation (`:322-343` — the single `$INSTALL_URL` fetch at `:322-330`, the `--customer-id/--mode/--hub-url/--passphrase-file` args array at `:334`, and the `bash "$SCRIPT_TMP" "${args[@]}"` call itself at `:343`) and the customer it yields is the one the operator bound, i.e. claimable. **So the ISO leg is the spine**; a manual 1.22.0 run is not needed as a separate scenario. **RUN ATTEMPTED 2026-07-29 — STOPPED AT PHASE 0, no VM created, nothing touched (`audits/E2D-fresh-vm-2026-07-29.md`).** The blocker is **R-111**: a fresh box installs **agent 0.96.0 + controller 0.161.0**, not `main`'s 0.113.0/0.185.1, so **C3/C4/C5 test surfaces that do not exist on it** — the degraded banner + `GET /api/storage/backup-target` landed in controller **v0.185.1** (`cdaeb36`) with the copy in **v0.185.0** (`3f7cf2a`); `backup_target_absent` in **v0.184.0** (`c1a63de`); the offer's apply needs agent **v0.113.0** (`58b598b`). **C1 (a real rc=0 1.22.0 install) and C2 (Case B natural, `configure_backup_target()` `felhom-host-install.sh:627`, warnings `:653-655`) remain ACHIEVABLE TODAY** — both are installer-side and host-install is served at 1.22.0. **C3 unblocks cheaply** by raising the hub global floor 0.156.0 → ≥0.185.0 (controller 0.185.1 IS in the registry): measured fleet impact is **nil** — both demo boxes already run 0.185.1, and Peti is DOWN 14 d and already below the current floor. **C4/C5 need R-111 first.** Phase 0 answers are all recorded in the audit, so a resumed run does not re-derive them: drive-gate cadence **30 s** (`intermediary.go:337`, registered `server.go:217`) ⇒ a 60 s two-cycle budget; hot-detach available (`virtio-scsi-single` + default hotplug, VM 300 is the working reference); ISO present (`…v1.25.0-nested-vm-generic-mkimage.iso`); `/mnt/nvme-1tb` 888 G free and the `local-lvm` fence re-measured (38.77 %, unchanged). **The §5.1a operator STOP is retired** — `HUB_PW` is in `~/.config/credentials` and hub auth was verified, so CC can bind. **One open decision carried forward:** where the VM disk's dir storage goes, since `local-lvm` is forbidden and `felhom-backup` is the live backup target — see the audit §4.1. **RUN EXECUTED 2026-07-29 after R-111 was fixed — `audits/E2D-fresh-vm-2026-07-29.md`.** Full ISO/PAIRING route on a nested VM on demo-hp; bind → running controller in **3 m 35 s**; teardown clean (`pvesm status` after == before, `local-lvm` 38.77 %, guest 9201 untouched). **C1 PROVEN** (`felhom-host-install v1.22.0`, `Day-0 provision SUCCESS`, guest 9201 running, golden = the one baked 20 min earlier). **C2 PROVEN** (both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort). **C3/C4 PARTIAL — API exact and complete, but customer-invisible → R-112.** **C5 FAILED → R-113.** C4's decline path PROVEN (registration confers no role), `restart_required:true` PROVEN, agent did **not** self-restart, E-2a wrapper created the storage at the drive's own mountpoint, healthy renders nothing. **E-2d's own premise needed amending:** a fresh install does NOT yield a CC-drivable claimable customer — the claim code is bcrypt-hashed and email-only, so one operator relay was required (and the claim flow is now proven end to end). **This row stays OPEN only for the residue:** C5 re-test after R-113, and the C3/C4 UI legs after R-112 | CC | | **R-94** | **A hand-synced version constant drifts, and the gate that would catch it is never run** — `hub/internal/web/configs.go:28` pins `hostInstallVersion = "1.19.0"` while `scripts/felhom-host-install.sh:187` is `SCRIPT_VERSION="1.22.0"` | **READY (XS)** | — | **CORRECTED 2026-07-29 — the earlier framing of this row was false and is retracted.** The constant selects no script: its only consumers are `configs.go:487` (`ScriptVersion`) and `render_test.go:219`, and it renders as a label at `customer_unified.html:494`. The install command beneath that label fetches `https://felhom.eu/scripts/felhom-host-install.sh` (`customer_unified.html:563`, `:1262`), which the website git-syncs from `main` on a 30 s period (`manifests/webpage.yaml`) — so **1.22.0 is what every install already gets** (live fetch, 2026-07-29). Every flag the generator emits is parsed by 1.22.0 (`customer_unified.html`~`:1210`–`:1238` vs `felhom-host-install.sh:1177`–`:1210`): **no functional gap, only a wrong number on the operator's screen.** Three legs, all XS: **(a)** derive the label from `SCRIPT_VERSION` rather than hand-syncing it, or delete it; **(b)** `scripts/hostinstall_gates.py` **fails today** and is invoked by no Makefile, hook or `CLAUDE.md` — wire it next to `site_gates.py` or delete it, because a gate nobody runs reads as coverage it is not providing (**this leg is one instance of → R-29**, which is the class: gates are enforced nowhere, and the enforcement decision belongs there, not here); **(c)** `render_test.go:219` compares the constant to itself and passes at any value — replace it with the cross-file assertion. **No longer blocked on E-2d** — it never gated anything. **2026-07-29: a real 1.22.0 install has now happened** (`audits/E2D-fresh-vm-2026-07-29.md`), so even the original (retracted) precaution is discharged — nothing stands in front of this row | CC | | **R-110** | **`main` is the installer's publish channel — there is no staging.** `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a 30 s period and nginx serves that working tree directly (`location /scripts/`, `root …/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as **root on a virgin box**, the single most privileged thing Felhom ships | **WAITING-ON-OPERATOR (S)** | operator ruling | **Two consequences worth stating:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" **was never available to take**. **Open question for the operator, not a defect to fix blind:** whether `/scripts/` should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether `main`-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now. **SECOND INSTANCE, found 2026-07-29 by the E-2d run and filed here rather than as a new ID:** `felhom-host-install.sh` fetches **nine** files from `raw/branch/main` (`:2072`–`:2206`) and the hub manifest vouches a sha for exactly **one** (`wrapper_sha256` → `felhom-pbs-apply`; re-checked this run, no drift). E-2a's `felhom-backup-target-apply` (`:2116`) is installed **0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n` — a root-executed artifact taken from `main` with no pinned integrity, which is this row's class exactly | CC | | **R-111** | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`.**~~ `felhom-host-install.sh` does not use `main`: it reads the hub-vouched manifest (`:423-436`) and fetches Gitea generic packages (agent `:1945`, golden `:2573`). Gitea's newest are **agent 0.96.0** and **golden 0.161.0**, and the hub's manifest selects exactly those — so a fresh box lands on **agent 0.96.0 + controller 0.161.0** (global floor `v0.156.0` < the golden's 0.161.0, so no self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and was never published** | **SHIPPED 2026-07-29 — the channel now serves agent 0.113.0 + golden 0.185.1** | — | **FIXED the same day it was found.** Agent **0.113.0** built from the clean tree @ `58b598b` and published (`scripts/publish-agent.sh`), sha `5f3247f756cb658e…`, round-trip GET verified. Golden **0.185.1** baked on the nested drill VM embedding controller `0.185.1`, published, sha `dba00f3e845c415e…` — bake clean: `Result=success`, overlay2, **all 3 mounts included** (rootfs+mp0+mp1), 0 FATAL/exclusions, upload HTTP 201, token-leak grep 0; log `drill/bake-0.185.1.log`; GL-1 teardown done (guest 9100 purged, secrets shredded, disk restored to `virgin`). Hub Day-0 manifest moved **both together in one POST** so it never vouched a new agent against an old golden; `min_agent` **0.93.0 → 0.113.0**, which is what controller v0.185.0 declares (`felhom-controller/CHANGELOG.md:15`) — **zero fleet impact, verified: all three enrolled hosts already run agent 0.113.0, so no box is held.** `wrapper_sha256` preserved verbatim (re-checked against `configs/felhom-pbs-apply` — no drift). **The global controller floor was deliberately NOT raised**: the golden now bakes 0.185.1, so a fresh box needs no self-update, and raising it would have been an unnecessary fleet-wide write. Original finding follows. **Found 2026-07-29 by the E-2d Phase 0 gate, which stopped the run before a VM was created.** 17 unpublished releases (v0.97.0–v0.113.0) strand the **entire R-82 tiered-backup arc** plus **F-CRIT-2** (a failed backup looking fresh — 7 days silent) and **F-REBOOT** (a guest rebooted mid-backup never returns): a new customer's box would install without them. **Blocks E-2d's C3/C4/C5** — those test endpoints and events that do not exist in 0.96.0/0.161.0. The **controller is fine** (registry has 0.185.1, floor-driven self-update), so the gap is specific to the two Gitea-generic artifacts. **Mirror of R-110, not a duplicate:** R-110 = the installer publishes instantly with no staging; R-111 = the agent/golden publish gate exists and was never walked. Fix should decide whether publishing joins the release train rather than staying a remembered step (R-29's shape, one layer up). Evidence: `audits/E2D-fresh-vm-2026-07-29.md` **DEFERRED LEG, AND IT RECURRED → R-115.** This row's shipped half stands and is not reopened: the bump happened, was verified, and was proven end-to-end by the E-2d install. But its own closing line — *decide whether publishing joins the release train rather than staying a remembered step* — was never acted on, and agent 0.114.0 reproduced the exact condition the same afternoon. The recurrence is filed as **R-115**, not as a reopen, because the stale-channel finding is closed while the process defect that caused it is a distinct problem with a distinct fix. | CC | | **R-115** | **Publishing is a remembered step, and it was forgotten within eight hours of being documented as forgettable.** A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published — so "deployed" and "installable" are independent states that drift silently. **Two instances, both real:** **R-111** (2026-07-29 morning) — 17 agent releases v0.97.0–v0.113.0 stranded, so a new customer would have installed without the entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT; found only because the E-2d Phase 0 gate happened to look. **Agent 0.114.0** (same afternoon) — the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, and **unpublished until this task**, which blocked Session C: a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix | **WAITING-ON-OPERATOR (M)** | operator ruling on the release process | **The finding is the RECURRENCE, not either instance** — both instances are fixed. R-111's own text already named this leg (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; the leg then recurred the same day, which is the evidence that a note is not a mechanism. **Class: → R-29, one layer up** — a control that exists and is never walked; deliberately NOT given its own ID. **The decision is the operator's; the options, mechanisms first:** (a) **publish as a step in the build/release path**, so deployed and installable cannot diverge; (b) **a gate that refuses to deploy a version that is not published+vouched** — the strongest, and it fails closed; (c) a session-end checklist entry; (d) accept it as manual and add a pre-Session-C verification. **(a) and (b) are mechanisms; (c) and (d) are reminders — and R-29's whole finding is that reminders do not hold.** No code this session by design | CC | -| **R-113** | ~~**The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.**~~ `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc//mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **CODE SHIPPED (agent v0.114.0, `b58d7bc`) — NOT LIVE-VALIDATED** | — (Session C unblocked 2026-07-29) | **SESSION C IS NOW UNBLOCKED.** Agent **0.114.0 published and vouched** 2026-07-29 (sha `5e4c15ebee2d7583…`, round-trip GET verified; hub manifest read back showing it) — a fresh drill box now installs the fix under test instead of the bug. The golden was NOT re-baked and `min_agent` was NOT raised: the golden bakes the controller, not the agent, and controller v0.185.0 declares MinAgent 0.113.0, which 0.114.0 already satisfies. **FIX: `BoundUnderParent` is now a CONJUNCTION — bound under the parent AND the drive's raw host mount still mounted** (`devicePresent`, new `deviceCheck` seam), at BOTH `/disks` construction sites. The raw mount is the device-bound systemd unit that dies with the device; the agent's bind is not — so the raw mount IS the device signal, grounded in E-2d's measurement rather than inference. **Conjunction, deliberately:** the device half alone would regress boot ordering (raw mounts early, bind lands ~18 s later — that window must keep reading absent), so existing behaviour is byte-identical and only the unreachable case is closed. **Unknown is never absent** (`devicePresent("")` = true) — a false absent stops a working customer's apps. **Controller UNCHANGED, no MinAgent bump:** `BoundUnderParent` has exactly one functional consumer (`planDriveGates:226`); a new `DevicePresent` bool was rejected because absent-from-JSON decodes to `false`, which would have made every drive on an older agent read ABSENT. +6 tests (208→214), 4 red-proofs run and reverted. **Deployed to demo-felhom; over-correction guard verified in production** — raw mount present, drive still reads present, 10/10 apps untouched, no gate action, no false alarm. **demo-hp deliberately left on 0.113.0** (the spec scoped deploy to felhom-pve) — it still carries the bug. **⚠️ SESSION C BLOCKER: the hub Day-0 manifest vouches agent 0.113.0**, so a fresh drill box would install WITHOUT this fix and validate nothing — publish + vouch 0.114.0 first (R-111's trap, same shape). **Awaiting live proof of exactly one leg:** device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. Original finding: **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 **Session C scope UNCHANGED by Session B** — R-114/R-112 shipped as controller v0.186.0 and neither touches the agent; the leg awaiting proof is still device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. One rebuild now validates all three. | CC | -| **R-112** | **E-2's degraded banner and offer have NO UI CONSUMER — the endpoint is correct and the customer never sees it.** `GET /api/storage/backup-target` returns byte-exact copy (proven live), and **nothing fetches it**: `grep 'backup-target'` across all `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` + `degradedMessageFor` are consumed **only** by the JSON handler — **no page handler injects the state** | **CODE SHIPPED (controller v0.186.0, `b331f18`) — SEAM PROVEN LIVE, copy NOT** | — | **FIXED: the state now has a consumer.** Server-rendered on `/backups` via `backupsHandler` → `backupTargetView` → `backups.html`, following the existing `SingleCopyWarning` banner pattern — NOT a 19th JS fetch, because a banner that needs JavaScript to appear is one more thing that can silently not happen. `backupTargetView` returns **nil** for healthy AND unknown, so those render nothing at all. **Scenario-E seam test drives `backupsHandler` over httptest and asserts the RENDERED HTML** — deleting the one line that sets `data["BackupTarget"]` reproduces the old state and fails every render assertion. **SEAM PROVEN LIVE on demo-felhom by a DIFFERENTIAL positive observable, not by an absent banner:** idle 8 s → 0 new `/backup/tiers` agent calls; each `/backups` load → exactly +1, and that call has only one caller (`resolveBackupTargetState`). The box is healthy (`degraded:false, target:felhom-backup`) and the page correctly rendered **nothing** — which matches its real state but, being a negative, is NOT by itself proof of wiring. **Still unproven live:** that a customer sees actual copy — impossible on a healthy box. **The decisive contrast: templates fetch 18 distinct `/api/storage/*` endpoints; `backup-target` and `backup-target/assign` are the only two with zero references.** The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"* — an invariant comment asserting a consumer that does not exist (7th instance of that class). v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* — it fixed the **router mount** and stopped one layer short of the **render**; its test `TestBackupTargetRoutesLiveUnderTheStorageAPIMount` pins dispatch, not reachability, which is exactly what `CLAUDE.md`'s seam rule warns about. **Fifth instance of seam-built-but-never-wired.** **Fix R-114 FIRST** — wiring this alone would start showing customers the wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | CC | -| **R-114** | **On target-drive loss the customer is told the wrong story and offered the drive that just vanished.** With the assigned target absent, the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"a rendszermentés ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a drive that has disappeared, not the system disk — **and** `offer_path` pointing at the missing drive as the remedy | **CODE SHIPPED (controller v0.186.0, `b331f18`) — NOT LIVE-VALIDATED** | Session C (Scenario C cannot occur on a healthy box) | **FIXED: the third state exists.** New `BackupTargetState.TargetAbsent` separates *configured-and-gone* from *never-configured*. `Degraded` keeps its meaning (is there a problem) so the wire contract is unchanged for every consumer; `TargetAbsent` answers which problem, because the remedies are OPPOSITE — attach any second drive vs reconnect *that* one. Copy routed through `degradedMessageFor` (still one decision point) and taken **verbatim** from the hub's `backup_target_absent` email so banner and mail tell one story. **Offer suppressed on the branch itself**, deliberately not left to `firstOfferableDrive`'s `Disconnected` skip — that flag is set by R-113 in another repo, and this state must be right without it. Red-proof: deleting the branch reproduces E-2d's exact payload, offering `/mnt/felhom-drives/mentes2`, the drive that had vanished. **MinAgent unchanged 0.113.0** — R-114 reads `BackupTarget`/`MountPath`/`GuestPath`/`Role`, none of which R-113 altered, so demo-hp is not held. **NOT live-validated: Scenario C cannot occur on a healthy box.** `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing **never configured** from **configured and now missing**. Shares R-113's root cause — two disagreeing presence signals — but is a different code path with a different fix. **Currently invisible ONLY because of R-112; fix this before wiring that.** Also seen: after reattach the drive returned as `/dev/sdc` while the stable bind still recorded `/dev/sdb`, and the state read healthy. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 | CC | +| **R-116** | **The drive-absent alarm and its recovery are a MISMATCHED PAIR — absent fires the GENERIC `storage_disconnected`, return fires the SPECIFIC `backup_target_restored`.** `backup_target_absent` never fires at all (count 0 across a full Session-C run). An operator gets an alarm they cannot match to its recovery, which is exactly what `notifyDriveReturned`'s own comment forbids | **READY (S) — P1** | — | **PROVEN LIVE 2026-07-29 (`audits/SESSION-C-2026-07-29.md` §5).** Root cause at source: `notifyDriveAbsent` (`intermediary.go:635-646`) branches on `isTarget[a.Path]` where `a.Path` is the GUEST path, and `driveTargetByPath` (`:602-616`) builds that map as `out[GuestPath] = d.BackupTarget`. **The drive is TWO `/disks` rows and the flag and the guest path sit on different ones:** the `felhom-backup` storage row carries `BackupTarget: true` (`felhom-agent/internal/localapi/disks.go:211`) but gets a `GuestPath` only while its role classifies as user-data; the registry union row carries the `GuestPath` but **never assigns `BackupTarget` at all** (`disks.go:265-267`). While the device is absent the flagged row loses its guest path, so the union row writes `false` and the specific branch is skipped — on RETURN the rows rejoin and the specific event fires, which is the observed asymmetry. **v0.184.1 fixed the KEYING** (both keys are mapped) **and not this** — the flag is missing from the row that has the guest path. **Only reachable because R-113 made the gate fire at all**; before that the branch was never taken. Fix likely belongs agent-side (the union row should carry the flag), so decide the repo before implementing. Blocks E-2's C5 | CC | +| **R-113** | ~~**The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.**~~ `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc//mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **SHIPPED + PROVEN-LIVE** (agent v0.114.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** Target drive hot-detached on a fresh box running the SHIPPED agent 0.114.0 (from the Day-0 manifest, not a hand build). **The gate fired in 4 seconds** — E-2d measured ZERO over 4½ minutes — and `SetDisconnected` was reached. It fired on exactly the shape that defeated it: raw `/mnt/mentes` NOT mounted while the bind `/mnt/felhom-drives/mentes` still read `/dev/sdb[/felhom-data]`. **Over-correction guard PASSED:** with both drives present, 0 ABSENT lines and the target stayed healthy. **Note: C5 still FAILED — but on a different defect (→ R-116), not on this one.** **SESSION C IS NOW UNBLOCKED.** Agent **0.114.0 published and vouched** 2026-07-29 (sha `5e4c15ebee2d7583…`, round-trip GET verified; hub manifest read back showing it) — a fresh drill box now installs the fix under test instead of the bug. The golden was NOT re-baked and `min_agent` was NOT raised: the golden bakes the controller, not the agent, and controller v0.185.0 declares MinAgent 0.113.0, which 0.114.0 already satisfies. **FIX: `BoundUnderParent` is now a CONJUNCTION — bound under the parent AND the drive's raw host mount still mounted** (`devicePresent`, new `deviceCheck` seam), at BOTH `/disks` construction sites. The raw mount is the device-bound systemd unit that dies with the device; the agent's bind is not — so the raw mount IS the device signal, grounded in E-2d's measurement rather than inference. **Conjunction, deliberately:** the device half alone would regress boot ordering (raw mounts early, bind lands ~18 s later — that window must keep reading absent), so existing behaviour is byte-identical and only the unreachable case is closed. **Unknown is never absent** (`devicePresent("")` = true) — a false absent stops a working customer's apps. **Controller UNCHANGED, no MinAgent bump:** `BoundUnderParent` has exactly one functional consumer (`planDriveGates:226`); a new `DevicePresent` bool was rejected because absent-from-JSON decodes to `false`, which would have made every drive on an older agent read ABSENT. +6 tests (208→214), 4 red-proofs run and reverted. **Deployed to demo-felhom; over-correction guard verified in production** — raw mount present, drive still reads present, 10/10 apps untouched, no gate action, no false alarm. **demo-hp deliberately left on 0.113.0** (the spec scoped deploy to felhom-pve) — it still carries the bug. **⚠️ SESSION C BLOCKER: the hub Day-0 manifest vouches agent 0.113.0**, so a fresh drill box would install WITHOUT this fix and validate nothing — publish + vouch 0.114.0 first (R-111's trap, same shape). **Awaiting live proof of exactly one leg:** device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. Original finding: **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 **Session C scope UNCHANGED by Session B** — R-114/R-112 shipped as controller v0.186.0 and neither touches the agent; the leg awaiting proof is still device loss → gate `Stop` → `SetDisconnected` → `backup_target_absent` on the wire. One rebuild now validates all three. | CC | +| **R-112** | **E-2's degraded banner and offer have NO UI CONSUMER — the endpoint is correct and the customer never sees it.** `GET /api/storage/backup-target` returns byte-exact copy (proven live), and **nothing fetches it**: `grep 'backup-target'` across all `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` + `degradedMessageFor` are consumed **only** by the JSON handler — **no page handler injects the state** | **SHIPPED + PROVEN-LIVE** (controller v0.186.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md` — the banner reached a customer's page for the first time.** Never-configured box: banner element 1, never-configured copy 1. After the wizard: offer block 1 with `data-path="/mnt/felhom-drives/mentes"`. Healthy after assign: all four markers 0, **proven POSITIVELY** — idle delta 0 `/backup/tiers` calls, page-load delta +1, single caller ⇒ the seam ran and chose silence. **FIXED: the state now has a consumer.** Server-rendered on `/backups` via `backupsHandler` → `backupTargetView` → `backups.html`, following the existing `SingleCopyWarning` banner pattern — NOT a 19th JS fetch, because a banner that needs JavaScript to appear is one more thing that can silently not happen. `backupTargetView` returns **nil** for healthy AND unknown, so those render nothing at all. **Scenario-E seam test drives `backupsHandler` over httptest and asserts the RENDERED HTML** — deleting the one line that sets `data["BackupTarget"]` reproduces the old state and fails every render assertion. **SEAM PROVEN LIVE on demo-felhom by a DIFFERENTIAL positive observable, not by an absent banner:** idle 8 s → 0 new `/backup/tiers` agent calls; each `/backups` load → exactly +1, and that call has only one caller (`resolveBackupTargetState`). The box is healthy (`degraded:false, target:felhom-backup`) and the page correctly rendered **nothing** — which matches its real state but, being a negative, is NOT by itself proof of wiring. **Still unproven live:** that a customer sees actual copy — impossible on a healthy box. **The decisive contrast: templates fetch 18 distinct `/api/storage/*` endpoints; `backup-target` and `backup-target/assign` are the only two with zero references.** The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"* — an invariant comment asserting a consumer that does not exist (7th instance of that class). v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* — it fixed the **router mount** and stopped one layer short of the **render**; its test `TestBackupTargetRoutesLiveUnderTheStorageAPIMount` pins dispatch, not reachability, which is exactly what `CLAUDE.md`'s seam rule warns about. **Fifth instance of seam-built-but-never-wired.** **Fix R-114 FIRST** — wiring this alone would start showing customers the wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | CC | +| **R-114** | **On target-drive loss the customer is told the wrong story and offered the drive that just vanished.** With the assigned target absent, the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"a rendszermentés ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a drive that has disappeared, not the system disk — **and** `offer_path` pointing at the missing drive as the remedy | **SHIPPED + PROVEN-LIVE** (controller v0.186.0, 2026-07-29) | — | **PROVEN LIVE `audits/SESSION-C-2026-07-29.md`.** With the target absent the page rendered the ABSENT copy (1), the system-disk copy 0, the offer block 0 — both of E-2d's falsehoods gone. API carried `message:"A rendszermentés meghajtója nem érhető el…"` with `target:felhom-backup`. **FIXED: the third state exists.** New `BackupTargetState.TargetAbsent` separates *configured-and-gone* from *never-configured*. `Degraded` keeps its meaning (is there a problem) so the wire contract is unchanged for every consumer; `TargetAbsent` answers which problem, because the remedies are OPPOSITE — attach any second drive vs reconnect *that* one. Copy routed through `degradedMessageFor` (still one decision point) and taken **verbatim** from the hub's `backup_target_absent` email so banner and mail tell one story. **Offer suppressed on the branch itself**, deliberately not left to `firstOfferableDrive`'s `Disconnected` skip — that flag is set by R-113 in another repo, and this state must be right without it. Red-proof: deleting the branch reproduces E-2d's exact payload, offering `/mnt/felhom-drives/mentes2`, the drive that had vanished. **MinAgent unchanged 0.113.0** — R-114 reads `BackupTarget`/`MountPath`/`GuestPath`/`Role`, none of which R-113 altered, so demo-hp is not held. **NOT live-validated: Scenario C cannot occur on a healthy box.** `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing **never configured** from **configured and now missing**. Shares R-113's root cause — two disagreeing presence signals — but is a different code path with a different fix. **Currently invisible ONLY because of R-112; fix this before wiring that.** Also seen: after reattach the drive returned as `/dev/sdc` while the stable bind still recorded `/dev/sdb`, and the state read healthy. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 | CC | | **R-29** | **The green gates are not enforced anywhere — one was RED for 16 releases before anyone ran it.** This is the **class**, not an instance: a gate that exists, asserts something true, is red, and is invoked by nothing reads as coverage it is not providing. `controller/scripts/docker_run_volume_path_gate.py` failed continuously from **2026-07-14 (v0.129.0)** until R-7b's close-out ran it by hand at v0.145.0 — sixteen releases in which every REPORT said "green" | **READY (S for (a) / M for (b))** | — | **This item has existed at `ROADMAP.md:158` since before the register was rebuilt (2026-07-27) and was never carried across — that omission is itself part of the finding**, because it is an open item *about work not getting done* that then went missing from the page that decides what gets done. Two separable parts, per R-29's own analysis: **(a)** the `docker_run_volume_path_gate` finding is benign and the fix is a 3-line ALLOWLIST addition with its why — **not** a rewrite of the flagged call — and it gets its own reviewed diff, never bundled into a feature commit; **(b)** the systemic half, the real item: decide where gates run (pre-push hook, `build.sh` step, or CI) and make a red gate block the train the way the Go green gate does. **Two further orphans confirmed 2026-07-29** by repo-wide grep across all file types + sibling repos + `~/.claude` settings/skills/hooks + `.git/hooks` (none non-sample) + Makefile/justfile/Taskfile find (only `hub/Makefile`, zero `gate` occurrences) + CI-directory find (**this repo has no CI at all**) — every one of the 19 hits is a docstring, a code comment or prose, and **not one is an invocation**: `scripts/hostinstall_gates.py` — **RED today** (`hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0`, exit 1), the same finding as **R-94 leg (b)** — and `scripts/hub_confirm_gate.py`. Of the four gates in `scripts/`, only `site_gates.py` is mandated anywhere (`CLAUDE.md:153`) and `manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`. **In R-29's own words, carried forward deliberately: do not mint a new ID for a new instance** — the 2026-07-18 rehearsal independently re-raised this item and no second ID was minted then either | CC | | **R-95** | restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only | **READY** | — | Root exposure still open. Mitigation now ARMED — split prune off-box or move to REST `--append-only` | CC | | **R-86** | Restore-tests are interval-scheduled, not backup-aligned | **READY** | R-90 (ep0 headroom) informs cadence | Trigger a tier ~24 h after **its own** newest archive | CC | @@ -48,7 +49,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **F-REBOOT** | ~~A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal; 9m47s total appliance outage with every alarm silent~~ | **SHIPPED + PROVEN-LIVE** (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; `onboot` is the deliberate-stop discriminator (already the stale-lock path's, and what `pve-guests` consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: **120 s unattended** vs the incident's 587 s with a human; Scenario B proven (an `onboot:0` guest left stopped) | — | | **F-LEAK** | ~~A failed restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); the 10-slot VMID band shrinks silently~~ | **SHIPPED + PROVEN-LIVE** (agent v0.110.0 + host-install v1.21.0, 2026-07-28) | — | **Three attempts, two refuted live.** (1) Pool adoption: `PUT /pools/{pool}` also needs `VM.Allocate` on the VM — membership cannot bootstrap its own authority. (2) Per-path `/vms/990000..990009` ACLs: work, but PVE's destroy calls `remove_vm_access` (`LXC.pm:906`) which deletes every ACL at `/vms/` — **consumed by the op it authorises**, one use per slot. (3) SHIPPED: 4th root-fenced exception, band enforced in sudoers **literally** (`pct destroy 99000[0-9] --purge`) + in code + at the caller; API destroy still tried first. Live: band PERMITTED, `9201`/`9100`/`9999`/`990010`/`1` REFUSED, and `pct start 990000` REFUSED too | — | | **F-OBS** | ~~`deadapp-check` leaves NO positive observable on a default (info-level) box — "no alarms" was indistinguishable from "never ran"~~ | **SHIPPED + PROVEN-LIVE** (controller v0.180.0 + agent v0.109.0, 2026-07-28) | — | INFO summary every 20th scan carrying scans/evaluated/down. **Agent v0.109.0 fixes the same shape in the guest-power watchdog shipped hours earlier in v0.107.0** — it logged only at startup and when it acted, so its health could be read only from absence | — | -| **E-2** | ~~Drive-role machinery around the moved vzdump target~~ | **SHIPPED** (hub 0.81.0, agent 0.113.0, controller 0.185.1, host-install 1.22.0 — 2026-07-29) | — | Parts 1–5 complete. Role model + offer-only assignment + Hungarian degraded banner + absent-target signal + installer Case A/B. **NOT yet live-proven:** the DEGRADED banner and the offer acceptance (both demo boxes are healthy, so neither state occurs naturally) and `backup_target_absent` end-to-end. Installer is ~~installer-logic-tested, not install-tested~~ — **INSTALL-TESTED 2026-07-29** on a fresh nested box via the real ISO/PAIRING route, rc=0 (`audits/E2D-fresh-vm-2026-07-29.md` §3). **Of the "NOT yet live-proven" list: Case B + the degraded state are now PROVEN at the installer and API level; the OFFER ACCEPTANCE is PROVEN at the API level** (decline path, `restart_required:true`, E-2a wrapper, healthy-renders-nothing). **Still NOT proven, and now known to be BROKEN rather than merely untested:** the banner/offer never reach a customer (**R-112**) and `backup_target_absent` cannot fire on device loss (**R-113**), with the absent-state message itself wrong (**R-114**) | CC | +| **E-2** | ~~Drive-role machinery around the moved vzdump target~~ | **CLOSED — PARTIALLY PROVEN** (Session C, 2026-07-29) | — | **CLOSED by `audits/SESSION-C-2026-07-29.md`.** C1/C2 proven in E-2d; **C3 and C4 PROVEN LIVE** this session (R-114, R-112); **C5 FAILED** — the gate fires and an alarm reaches the hub, but it is the generic event, not `backup_target_absent` (→ **R-116**, the one named open leg). Per the runbook's §9, decided in advance: a failed claim closes E-2 as partially proven with a named leg rather than re-running. The arc's stated definition of done is R-106+R-109, R-108 and D5 — none of which this detour touched. Parts 1–5 complete. Role model + offer-only assignment + Hungarian degraded banner + absent-target signal + installer Case A/B. **NOT yet live-proven:** the DEGRADED banner and the offer acceptance (both demo boxes are healthy, so neither state occurs naturally) and `backup_target_absent` end-to-end. Installer is ~~installer-logic-tested, not install-tested~~ — **INSTALL-TESTED 2026-07-29** on a fresh nested box via the real ISO/PAIRING route, rc=0 (`audits/E2D-fresh-vm-2026-07-29.md` §3). **Of the "NOT yet live-proven" list: Case B + the degraded state are now PROVEN at the installer and API level; the OFFER ACCEPTANCE is PROVEN at the API level** (decline path, `restart_required:true`, E-2a wrapper, healthy-renders-nothing). **Still NOT proven, and now known to be BROKEN rather than merely untested:** the banner/offer never reach a customer (**R-112**) and `backup_target_absent` cannot fire on device loss (**R-113**), with the absent-state message itself wrong (**R-114**) | CC | | **E-2a** | ~~The target move needs a root-fenced wrapper — the agent cannot do it~~ | **SHIPPED + PROVEN-LIVE** (agent v0.113.0 + host-install v1.22.0, 2026-07-29) | — | `felhom-backup-target-apply` behind a literal `FELHOM_BACKUPTARGET` sudoers alias; the agent's PVE role was NOT widened. Enforces F-1 (`mountpoint -q`) and F-2 (`is_mountpoint 1` hardcoded), refuses a root-device target, has NO storage-removal path (grep-assertable), is idempotent and refuses to repoint. All five laws proven live as root on demo-hp with 0 stray storages | — | | **E-2b** | ~~`NotifyStorageDisconnected`/`Reconnected` defined and called NOWHERE — a drive going absent emitted no event on any channel~~ | **SHIPPED + PROVEN-LIVE** (controller v0.184.1 + agent v0.112.0 + hub v0.81.0, 2026-07-29) | — | Seam wired in `ReconcileDriveGates`; a target drive raises the specific `backup_target_absent` instead. **A keying bug was caught before deploy:** `a.Path` is the registered GUEST path, not the agent's host `MountPath`, so the target branch was unreachable — every absent drive, target included, fell through to the generic event (v0.184.1). Tests observe the WIRE (httptest hub), not a mock | — | | **E-2c** | ~~E-1 put the whole-guest backups on a drive `POST /disks/eject` would eject~~ | **SHIPPED + PROVEN-LIVE** (agent v0.112.0, 2026-07-29) | — | Eject + decommission refuse 409 on the backup-target mount, naming the storage and the remedy. **Live on BOTH boxes:** demo-hp `/mnt/nvme-1tb` and demo-felhom `/mnt/hdd_1` both refused, drives unmoved. NOT a role reclassification — `RoleForStorage` untouched, because on both boxes that drive is ALSO the enrolled user-data drive; `TestEjectStillAllowedOnANonTargetDrive` pins the non-over-correction and `/var/lib/vz` is still refused by the PRE-EXISTING role gate, not this one | — | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index d3da121..61cef50 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -21,6 +21,7 @@ | ID | Item | Size | Status | Notes / map rows flipped | |----|------|------|--------|--------------------------| | R-115 | **Publishing is a remembered step — forgotten within eight hours of being documented as forgettable** | M | idea — **WAITING-ON-OPERATOR**, 2026-07-29 | A box installs the agent from a Gitea generic package the hub explicitly vouches, never from git. Nothing in the build, deploy or session-end path publishes or checks that a version was published, so **"deployed" and "installable" are independent states that drift silently**. **Instance 1 — R-111** (morning): 17 agent releases v0.97.0–v0.113.0 stranded; a new customer would have installed without the whole R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT. Found only because the E-2d Phase 0 gate happened to look. **Instance 2 — agent 0.114.0** (same afternoon): the R-113 fix, built and pushed at `b58d7bc`, deployed to felhom-pve, never published — which blocked Session C, since a fresh drill box would have installed 0.113.0 and proven the bug rather than the fix. **The finding is the RECURRENCE, not either instance** — both are fixed. R-111 named this leg in its own text (*"decide whether publishing joins the release train rather than staying a remembered step"*) and closed SHIPPED without it; it recurred the same day, which is the evidence that **a note is not a mechanism**. **Class: → R-29, one layer up** (a control that exists and is never walked) — deliberately NOT given a second ID. **Filed as its own item rather than reopening R-111** because R-111's finding (the channel *was* stale) is closed and verified end-to-end by the E-2d install, while the process defect that caused it is a distinct problem with a distinct fix and a distinct owner. **Operator's decision, mechanisms first:** (a) publish as a step in the build/release path so deployed and installable cannot diverge; (b) a gate that refuses to deploy an unpublished+unvouched version — strongest, fails closed; (c) a session-end checklist entry; (d) accept manual + a pre-Session-C verification. **(a)/(b) are mechanisms, (c)/(d) are reminders — and R-29's whole finding is that reminders do not hold.** No code written when filed, by design | +| R-116 | **The drive-absent alarm and its recovery are a mismatched pair — generic on the way out, specific on the way back** | S | idea — **PROVEN LIVE 2026-07-29** | Absent fires `storage_disconnected`; return fires `backup_target_restored`. `backup_target_absent` never fires at all (count 0 across a full Session-C run), so an operator gets an alarm they cannot match to its recovery — exactly what `notifyDriveReturned`'s own comment forbids. Root cause: `notifyDriveAbsent` (`intermediary.go:635-646`) branches on `isTarget[a.Path]` with `a.Path` the GUEST path, and `driveTargetByPath` (`:602-616`) builds it as `out[GuestPath] = d.BackupTarget` — but **the drive is TWO `/disks` rows and the flag and the guest path sit on different ones**: the `felhom-backup` storage row has `BackupTarget: true` (`felhom-agent/internal/localapi/disks.go:211`) and gets a guest path only while classified user-data, while the registry union row has the guest path and **never assigns `BackupTarget`** (`disks.go:265-267`). Absent ⇒ the flagged row loses its guest path ⇒ the union row writes `false` ⇒ generic. On return the rows rejoin ⇒ specific. v0.184.1 fixed the KEYING, not this. **Only reachable because R-113 made the gate fire at all.** Fix likely agent-side; decide the repo first. Blocks E-2's C5. Evidence: `audits/SESSION-C-2026-07-29.md` §5 | | R-113 | **The drive-absent gate cannot fire on device loss — E-2b's alarm is wired to an unreachable condition** | M | idea — **PROVEN LIVE 2026-07-29** | `planDriveGates` (`felhom-controller/internal/web/intermediary.go:216-262`) treats a path as present by OR-ing in `d.BoundUnderParent`, which the agent derives from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc//mountinfo`"* (`internal/localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is not device-bound and its mountinfo entry outlives the device**, so the gate reads it as present and `notifyDriveAbsent` is never called. Live on a fresh box: target drive hot-detached, agent said `enrolled drive absent by UUID` every 20 s for 4½ min, controller logged **0** `[gate]` lines, hub received **zero** events — neither `backup_target_absent` nor the generic `storage_disconnected`. Not a virtualisation artefact (device-bound-mount vs manual-bind is the same on metal); caveat: SCSI hot-detach, physical unplug not staged. **Sixth instance of seam-built-but-never-wired — E-2b wired the seam to a condition that cannot occur.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 | | R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints** — `backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | | R-114 | **On target-drive loss the customer is told the wrong story and offered the drive that vanished** | S | idea — **PROVEN LIVE 2026-07-29** | With the assigned target absent the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a missing drive, not the system disk — **and** an `offer_path` pointing at the drive that just disappeared. `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing *never configured* from *configured and now missing*. Shares R-113's root cause (two disagreeing presence signals), different code path and fix. **Invisible today only because of R-112.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 |