R-113 code shipped (agent v0.114.0) — NOT live-validated, awaiting Session C

BoundUnderParent is now a CONJUNCTION: bound under the parent AND the drive's
raw host mount still mounted. The raw mount is the device-bound systemd unit
that dies with the device; the agent's own bind is not, which is why the bind
outlived the device and the gate could never fire.

Conjunction deliberately, not replacement: the device half alone would regress
boot ordering (raw mounts early, bind lands ~18s later — that window must keep
reading absent), so existing behaviour is byte-identical and only the
unreachable case is closed. Unknown is never absent.

Controller UNCHANGED, no MinAgent bump — BoundUnderParent has exactly one
functional consumer (planDriveGates:226). A new DevicePresent bool was rejected:
absent-from-JSON decodes to false, so every drive on an older agent would have
read ABSENT and stopped its apps.

+6 tests (208->214), 4 red-proofs run and reverted. Deployed to demo-felhom and
the over-correction guard verified in production: raw mount present, drive still
reads present, 10/10 apps untouched, no gate action, no false alarm. demo-hp
deliberately left on 0.113.0 (the spec scoped deploy to felhom-pve).

SESSION C BLOCKER recorded on the row: the hub Day-0 manifest vouches agent
0.113.0, so a fresh drill box would install WITHOUT this fix and validate
nothing. Publish + vouch 0.114.0 first — R-111's trap in the same shape.
This commit is contained in:
2026-07-29 17:23:09 +02:00
parent d839ddcb60
commit ca4c8b3afc
+1 -1
View File
@@ -15,7 +15,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha
| **R-94** | **A hand-synced version constant drifts, and the gate that would catch it is never run**`hub/internal/web/configs.go:28` pins `hostInstallVersion = "1.19.0"` while `scripts/felhom-host-install.sh:187` is `SCRIPT_VERSION="1.22.0"` | **READY (XS)** | — | **CORRECTED 2026-07-29 — the earlier framing of this row was false and is retracted.** The constant selects no script: its only consumers are `configs.go:487` (`ScriptVersion`) and `render_test.go:219`, and it renders as a label at `customer_unified.html:494`. The install command beneath that label fetches `https://felhom.eu/scripts/felhom-host-install.sh` (`customer_unified.html:563`, `:1262`), which the website git-syncs from `main` on a 30 s period (`manifests/webpage.yaml`) — so **1.22.0 is what every install already gets** (live fetch, 2026-07-29). Every flag the generator emits is parsed by 1.22.0 (`customer_unified.html`~`:1210``:1238` vs `felhom-host-install.sh:1177``:1210`): **no functional gap, only a wrong number on the operator's screen.** Three legs, all XS: **(a)** derive the label from `SCRIPT_VERSION` rather than hand-syncing it, or delete it; **(b)** `scripts/hostinstall_gates.py` **fails today** and is invoked by no Makefile, hook or `CLAUDE.md` — wire it next to `site_gates.py` or delete it, because a gate nobody runs reads as coverage it is not providing (**this leg is one instance of → R-29**, which is the class: gates are enforced nowhere, and the enforcement decision belongs there, not here); **(c)** `render_test.go:219` compares the constant to itself and passes at any value — replace it with the cross-file assertion. **No longer blocked on E-2d** — it never gated anything. **2026-07-29: a real 1.22.0 install has now happened** (`audits/E2D-fresh-vm-2026-07-29.md`), so even the original (retracted) precaution is discharged — nothing stands in front of this row | CC |
| **R-110** | **`main` is the installer's publish channel — there is no staging.** `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a 30 s period and nginx serves that working tree directly (`location /scripts/`, `root …/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as **root on a virgin box**, the single most privileged thing Felhom ships | **WAITING-ON-OPERATOR (S)** | operator ruling | **Two consequences worth stating:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" **was never available to take**. **Open question for the operator, not a defect to fix blind:** whether `/scripts/` should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether `main`-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now. **SECOND INSTANCE, found 2026-07-29 by the E-2d run and filed here rather than as a new ID:** `felhom-host-install.sh` fetches **nine** files from `raw/branch/main` (`:2072``:2206`) and the hub manifest vouches a sha for exactly **one** (`wrapper_sha256``felhom-pbs-apply`; re-checked this run, no drift). E-2a's `felhom-backup-target-apply` (`:2116`) is installed **0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n` — a root-executed artifact taken from `main` with no pinned integrity, which is this row's class exactly | CC |
| **R-111** | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`.**~~ `felhom-host-install.sh` does not use `main`: it reads the hub-vouched manifest (`:423-436`) and fetches Gitea generic packages (agent `:1945`, golden `:2573`). Gitea's newest are **agent 0.96.0** and **golden 0.161.0**, and the hub's manifest selects exactly those — so a fresh box lands on **agent 0.96.0 + controller 0.161.0** (global floor `v0.156.0` < the golden's 0.161.0, so no self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and was never published** | **SHIPPED 2026-07-29 — the channel now serves agent 0.113.0 + golden 0.185.1** | — | **FIXED the same day it was found.** Agent **0.113.0** built from the clean tree @ `58b598b` and published (`scripts/publish-agent.sh`), sha `5f3247f756cb658e…`, round-trip GET verified. Golden **0.185.1** baked on the nested drill VM embedding controller `0.185.1`, published, sha `dba00f3e845c415e…` — bake clean: `Result=success`, overlay2, **all 3 mounts included** (rootfs+mp0+mp1), 0 FATAL/exclusions, upload HTTP 201, token-leak grep 0; log `drill/bake-0.185.1.log`; GL-1 teardown done (guest 9100 purged, secrets shredded, disk restored to `virgin`). Hub Day-0 manifest moved **both together in one POST** so it never vouched a new agent against an old golden; `min_agent` **0.93.0 → 0.113.0**, which is what controller v0.185.0 declares (`felhom-controller/CHANGELOG.md:15`) — **zero fleet impact, verified: all three enrolled hosts already run agent 0.113.0, so no box is held.** `wrapper_sha256` preserved verbatim (re-checked against `configs/felhom-pbs-apply` — no drift). **The global controller floor was deliberately NOT raised**: the golden now bakes 0.185.1, so a fresh box needs no self-update, and raising it would have been an unnecessary fleet-wide write. Original finding follows. **Found 2026-07-29 by the E-2d Phase 0 gate, which stopped the run before a VM was created.** 17 unpublished releases (v0.97.0v0.113.0) strand the **entire R-82 tiered-backup arc** plus **F-CRIT-2** (a failed backup looking fresh — 7 days silent) and **F-REBOOT** (a guest rebooted mid-backup never returns): a new customer's box would install without them. **Blocks E-2d's C3/C4/C5** — those test endpoints and events that do not exist in 0.96.0/0.161.0. The **controller is fine** (registry has 0.185.1, floor-driven self-update), so the gap is specific to the two Gitea-generic artifacts. **Mirror of R-110, not a duplicate:** R-110 = the installer publishes instantly with no staging; R-111 = the agent/golden publish gate exists and was never walked. Fix should decide whether publishing joins the release train rather than staying a remembered step (R-29's shape, one layer up). Evidence: `audits/E2D-fresh-vm-2026-07-29.md` | CC |
| **R-113** | **The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.** `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()`*"is this path a mount target in the guest's `/proc/<pid>/mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **READY (M) — P1** | — | **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 | CC |
| **R-113** | ~~**The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.**~~ `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()`*"is this path a mount target in the guest's `/proc/<pid>/mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **CODE SHIPPED (agent v0.114.0, `b58d7bc`) — NOT LIVE-VALIDATED** | Session C (the drill-VM rebuild) | **FIX: `BoundUnderParent` is now a CONJUNCTION — bound under the parent AND the drive's raw host mount still mounted** (`devicePresent`, new `deviceCheck` seam), at BOTH `/disks` construction sites. The raw mount is the device-bound systemd unit that dies with the device; the agent's bind is not — so the raw mount IS the device signal, grounded in E-2d's measurement rather than inference. **Conjunction, deliberately:** the device half alone would regress boot ordering (raw mounts early, bind lands ~18 s later — that window must keep reading absent), so existing behaviour is byte-identical and only the unreachable case is closed. **Unknown is never absent** (`devicePresent("")` = true) — a false absent stops a working customer's apps. **Controller UNCHANGED, no MinAgent bump:** `BoundUnderParent` has exactly one functional consumer (`planDriveGates:226`); a new `DevicePresent` bool was rejected because absent-from-JSON decodes to `false`, which would have made every drive on an older agent read ABSENT. +6 tests (208→214), 4 red-proofs run and reverted. **Deployed to demo-felhom; over-correction guard verified in production** — raw mount present, drive still reads present, 10/10 apps untouched, no gate action, no false alarm. **demo-hp deliberately left on 0.113.0** (the spec scoped deploy to felhom-pve) — it still carries the bug. **⚠️ SESSION C BLOCKER: the hub Day-0 manifest vouches agent 0.113.0**, so a fresh drill box would install WITHOUT this fix and validate nothing — publish + vouch 0.114.0 first (R-111's trap, same shape). **Awaiting live proof of exactly one leg:** device loss → gate `Stop``SetDisconnected``backup_target_absent` on the wire. Original finding: **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 | CC |
| **R-112** | **E-2's degraded banner and offer have NO UI CONSUMER — the endpoint is correct and the customer never sees it.** `GET /api/storage/backup-target` returns byte-exact copy (proven live), and **nothing fetches it**: `grep 'backup-target'` across all `*.html`/`*.js`/`*.css`**0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` + `degradedMessageFor` are consumed **only** by the JSON handler — **no page handler injects the state** | **READY (S) — P1** | — | **The decisive contrast: templates fetch 18 distinct `/api/storage/*` endpoints; `backup-target` and `backup-target/assign` are the only two with zero references.** The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"* — an invariant comment asserting a consumer that does not exist (7th instance of that class). v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* — it fixed the **router mount** and stopped one layer short of the **render**; its test `TestBackupTargetRoutesLiveUnderTheStorageAPIMount` pins dispatch, not reachability, which is exactly what `CLAUDE.md`'s seam rule warns about. **Fifth instance of seam-built-but-never-wired.** **Fix R-114 FIRST** — wiring this alone would start showing customers the wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | CC |
| **R-114** | **On target-drive loss the customer is told the wrong story and offered the drive that just vanished.** With the assigned target absent, the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"a rendszermentés ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a drive that has disappeared, not the system disk — **and** `offer_path` pointing at the missing drive as the remedy | **READY (S)** | — | `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing **never configured** from **configured and now missing**. Shares R-113's root cause — two disagreeing presence signals — but is a different code path with a different fix. **Currently invisible ONLY because of R-112; fix this before wiring that.** Also seen: after reattach the drive returned as `/dev/sdc` while the stable bind still recorded `/dev/sdb`, and the state read healthy. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 | CC |
| **R-29** | **The green gates are not enforced anywhere — one was RED for 16 releases before anyone ran it.** This is the **class**, not an instance: a gate that exists, asserts something true, is red, and is invoked by nothing reads as coverage it is not providing. `controller/scripts/docker_run_volume_path_gate.py` failed continuously from **2026-07-14 (v0.129.0)** until R-7b's close-out ran it by hand at v0.145.0 — sixteen releases in which every REPORT said "green" | **READY (S for (a) / M for (b))** | — | **This item has existed at `ROADMAP.md:158` since before the register was rebuilt (2026-07-27) and was never carried across — that omission is itself part of the finding**, because it is an open item *about work not getting done* that then went missing from the page that decides what gets done. Two separable parts, per R-29's own analysis: **(a)** the `docker_run_volume_path_gate` finding is benign and the fix is a 3-line ALLOWLIST addition with its why — **not** a rewrite of the flagged call — and it gets its own reviewed diff, never bundled into a feature commit; **(b)** the systemic half, the real item: decide where gates run (pre-push hook, `build.sh` step, or CI) and make a red gate block the train the way the Go green gate does. **Two further orphans confirmed 2026-07-29** by repo-wide grep across all file types + sibling repos + `~/.claude` settings/skills/hooks + `.git/hooks` (none non-sample) + Makefile/justfile/Taskfile find (only `hub/Makefile`, zero `gate` occurrences) + CI-directory find (**this repo has no CI at all**) — every one of the 19 hits is a docstring, a code comment or prose, and **not one is an invocation**: `scripts/hostinstall_gates.py`**RED today** (`hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0`, exit 1), the same finding as **R-94 leg (b)** — and `scripts/hub_confirm_gate.py`. Of the four gates in `scripts/`, only `site_gates.py` is mandated anywhere (`CLAUDE.md:153`) and `manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`. **In R-29's own words, carried forward deliberately: do not mint a new ID for a new instance** — the 2026-07-18 rehearsal independently re-raised this item and no second ID was minted then either | CC |