diff --git a/REPORT-e2d.md b/REPORT-e2d.md index 7ce35a6..16ce52e 100644 --- a/REPORT-e2d.md +++ b/REPORT-e2d.md @@ -1,58 +1,87 @@ -# REPORT — E-2d fresh-VM run: STOPPED at Phase 0 (2026-07-29) +# REPORT — R-111 fixed, then E-2 proven on a fresh box (2026-07-29) -`RUNBOOK-e2d-fresh-vm-2026-07-29.md`, executed by CC on DooPlex. Full evidence: -`documentation/audits/E2D-fresh-vm-2026-07-29.md`. Written as `REPORT-e2d.md` per the runbook §8.5; -root `REPORT.md` untouched. +Two phases in one session. Full evidence: `documentation/audits/E2D-fresh-vm-2026-07-29.md`. +Root `REPORT.md` untouched. -**Outcome: no VM created, no install run, no box touched, no teardown needed.** The run stopped at the -Phase 0 gate, per §3 (*"If any gate fails, STOP and report — do not adapt around it"*). +## Phase 1 — R-111: the Day-0 channel now serves the current software -## Why +A Phase 0 gate earlier the same day stopped the E-2d run before any VM existed: a fresh box would +have installed **agent 0.96.0 + controller 0.161.0**, ~17 and ~24 releases behind `main`. -`felhom-host-install.sh` does not install what is on `main`. It resolves the hub-vouched manifest -(`:423-436`) and fetches **Gitea generic packages** (agent `:1945`, golden `:2573`). Gitea's newest are -**agent `0.96.0`** and **golden `0.161.0`**; the hub's saved manifest selects exactly those; the global -controller floor is `v0.156.0`, below the golden's 0.161.0, so no self-update follows. - -| Component | Fresh install gets | `main` / demo boxes | +| | Before | Now | |---|---|---| -| host-install | **1.22.0** | 1.22.0 | -| agent | **0.96.0** | 0.113.0 — **never published**, direct-deployed | -| controller | **0.161.0** | 0.185.1 | +| agent (Gitea generic) | 0.96.0 | **0.113.0**, sha `5f3247f7…`, round-trip verified | +| golden (Gitea generic) | 0.161.0 | **0.185.1**, sha `dba00f3e…`, embeds controller 0.185.1 | +| hub `min_agent` | 0.93.0 | **0.113.0** (what controller v0.185.0 declares) | -## Claims +Bake clean on every marker: `Result=success`, overlay2, **all three mounts in the archive**, 0 +FATAL/exclusions, HTTP 201, token-leak grep 0. GL-1 teardown: guest 9100 purged, secrets shredded, +drill disk restored to `virgin`. Agent + golden moved in **one** manifest POST so it never vouched a +new agent against an old golden. `min_agent` verified zero-impact first (all three enrolled hosts +already at 0.113.0). Global floor deliberately **not** raised — the golden now bakes 0.185.1. -| Claim | Verdict | Pinned to | -|---|---|---| -| C1 — real rc=0 install of 1.22.0 | **ACHIEVABLE, not run** | installer-side | -| C2 — Case B fires naturally | **ACHIEVABLE, not run** | `felhom-host-install.sh:627`, `:653-655` | -| C3 — degraded banner renders | **BLOCKED** | controller v0.185.1 `cdaeb36`; copy v0.185.0 `3f7cf2a` | -| C4 — offer moves the target | **BLOCKED** | controller v0.185.1 + agent v0.113.0 `58b598b` | -| C5 — `backup_target_absent` e2e | **BLOCKED** | controller v0.184.0 `c1a63de` + agent v0.112.0 | +Commit `3dff357`. -C3 unblocks by raising the hub floor to ≥0.185.0 (controller 0.185.1 **is** in the registry); measured -fleet impact nil — both demo boxes already run 0.185.1, Peti is DOWN 14 d and already below the current -floor. C4/C5 need agent 0.112.0/0.113.0 **published**, which runbook §0 forbids this run from doing. +## Phase 2 — the E-2d run, full ISO/PAIRING route -## Filed +Nested PVE VM on demo-hp, one disk, outside the `felhom` pool. Bind → running controller in +**3 m 35 s**. The install fetched exactly the artifacts published an hour earlier and restored +`vzdump-lxc-9100-2026_07_29-12_37_56` — the golden baked 20 minutes before. The publish train is +proven end to end on a real install. -**R-111 (P1)** — the Day-0 artifact channel is 17 agent releases stale. v0.97.0–v0.113.0 unpublished, -stranding the entire R-82 tiered-backup arc plus **F-CRIT-2** and **F-REBOOT**. A new customer's box -installs without them. Mirror of R-110, not a duplicate: R-110 = publishes instantly with no staging; -R-111 = the publish gate exists and was never walked. +| Claim | Verdict | +|---|---| +| **C1** host-install 1.22.0 completes a real install, rc=0 | ✅ **PROVEN** | +| **C2** Case B fires naturally | ✅ **PROVEN** — both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort | +| **C3** degraded banner renders **to a customer** | ⚠️ **PARTIAL** — API byte-exact; **no UI consumer exists** → **R-112** | +| **C4** offer appears and moves the target | ⚠️ **PARTIAL** — decline path, `restart_required:true`, no self-restart, E-2a wrapper, healthy-renders-nothing all PROVEN at API level; offer equally invisible → **R-112** | +| **C5** `backup_target_absent` end to end | ❌ **FAILED** — zero events on any channel → **R-113** | + +## The three findings + +**R-112 (P1)** — E-2's banner and offer have **no UI consumer**. The endpoint returns byte-exact copy; +`grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**, and no page handler injects +the state. Decisive contrast: templates fetch **18** distinct `/api/storage/*` endpoints; +`backup-target` and `backup-target/assign` are the only two with zero references. v0.185.1 fixed the +router mount and stopped one layer short of the render. Fifth instance of seam-built-but-never-wired. + +**R-113 (P1)** — the drive-absent gate **cannot fire on device loss**. `planDriveGates` reads presence +from `BoundUnderParent` = "is this path in the guest's mountinfo". The raw mount is a device-bound +systemd unit and dies; **the agent's own bind is not device-bound and outlives the device**, so the +gate sees "present" forever. Live: agent said `enrolled drive absent by UUID` every 20 s for 4½ +minutes, controller logged **0** `[gate]` lines, hub got **zero** events — neither the specific nor the +generic one. Sixth instance of the class, one layer deeper: E-2b wired the seam to a condition that +cannot occur. + +**R-114** — on target-drive loss the message says the backup is *"on the same disk as the system"* +(false) and offers **the drive that just vanished**. Invisible today only because of R-112 — so +**R-114 must be fixed before R-112 is wired.** + +Also filed as a **second instance under R-110** (not a new ID): host-install fetches **nine** files +from `raw/branch/main` and the hub vouches a sha for **one**; E-2a's wrapper is installed 0755 to +`/usr/local/sbin`, root-fenced in sudoers, validated only by `bash -n`. ## Record -- `OPEN-ITEMS.md` — **R-111 opened** (READY (M), P1). **E-2d re-stated**, not closed: the attempt, the - blocker, the C1/C2-vs-C3/C4/C5 split, and every Phase 0 answer so a resumed run does not re-derive them. -- `ROADMAP.md` — R-111 filed under **P1 — closed-alpha blockers**. -- `architecture/00-capability-map.md` — **not touched.** Nothing was proven live; no row qualifies. +- `OPEN-ITEMS.md` — **R-112/R-113/R-114 opened**; E-2d re-stated with results and left open for the + residue; E-2's "NOT yet live-proven" list resolved into proven / known-broken; R-94 fully unblocked; + R-110 extended; a WATCHING row for the drill cleanup. +- `ROADMAP.md` — R-112/R-113/R-114 under P1; R-111 marked SHIPPED. +- **`architecture/00-capability-map.md` not touched** — for two reasons: the customer-facing legs are + broken rather than proven, and the map has **no E-2 / backup-target rows at all** (worth noting + against the ROADMAP's coupling rule). -## Not done +## Teardown -C1–C5 all unproven. No VM, no ISO boot, no appliance registered/bound/discarded, no storage created or -modified on demo-hp, no drive attached or detached, no hub setting written (the floor was read only), -no code written. +VM destroyed, scratch storage removed, **`pvesm status` after == before** (`local-lvm` 38.77 %, +byte-identical), guest 9201 and drill-r50 untouched. **Hub records for `e2d-fresh` remain** — the +delete was attempted and correctly refused at four gates, finally `host … is ONLINE`; it becomes +deletable once the destroyed host ages to DOWN (>1 h). Command recorded in `OPEN-ITEMS.md`. The stale +2026-07-25 appliance (`206c8838…`) was left alone — not mine. -The runbook's **§5.1a operator STOP is retired**: `HUB_PW` is in `~/.config/credentials` and hub auth -was verified working, so CC can perform the bind itself on a resumed run. +## One human step, and a premise correction + +The runbook's §5.1a operator STOP (the bind) is **retired** — CC did it. But E-2d's premise that a +fresh install yields a CC-drivable claimable customer is **wrong**: the claim code is bcrypt-hashed and +email-only, and the gate covers everything except `/claim`, `/api/health`, `/static/`. One operator +relay of the emailed code was required — which also proved the claim flow end to end. diff --git a/documentation/audits/E2D-fresh-vm-2026-07-29.md b/documentation/audits/E2D-fresh-vm-2026-07-29.md index 878640b..3dc2d08 100644 --- a/documentation/audits/E2D-fresh-vm-2026-07-29.md +++ b/documentation/audits/E2D-fresh-vm-2026-07-29.md @@ -1,198 +1,258 @@ -# E2D-fresh-vm-2026-07-29 — Phase 0 STOP: the Day-0 artifact channel cannot deliver the code under test +# E2D-fresh-vm-2026-07-29 — E-2 proven on a fresh box: C1/C2 pass, C3/C4 partial, C5 FAILS **Run:** `RUNBOOK-e2d-fresh-vm-2026-07-29.md`, executed by CC on DooPlex, 2026-07-29. -**Outcome:** **STOPPED at Phase 0, before any VM was created.** No VM provisioned, no install run, no -box touched, no teardown required. Per §3: *"If any gate fails, STOP and report — do not adapt around it."* +**Preceded by:** a Phase 0 STOP earlier the same day (R-111 — the Day-0 channel was 17 agent releases +stale). R-111 was fixed first; this run then proceeded on the real customer path. -**The finding in one sentence:** a box installed today through the real customer chain receives -**agent 0.96.0** and **controller 0.161.0**, because those are the newest artifacts ever published to -the Day-0 channel — so three of the five claims (C3, C4, C5) test endpoints and events that do not -exist in the software a fresh box actually runs. +**Headline:** the installer and its Case B are proven on a real install. **The two customer-facing +halves of E-2 are not reachable by a customer at all**, and the drive-absent alarm **cannot fire on +device loss**. Both were invisible to a green unit suite and to an API-level check; only the live run +found them. ---- - -## 1. Baselines confirmed - -| Artifact | Runbook §1 | Confirmed | Source | -|---|---|---|---| -| hub | 0.81.0 | **0.81.0** | `manifests/hub.yaml:128`; `hub/CHANGELOG.md:1`; live deploy image | -| agent | 0.113.0 | **0.113.0** on `main` @ `58b598b` | `felhom-agent` HEAD; live `felhom-agent -version` on demo-hp | -| controller | 0.185.1 | **0.185.1** on `main` @ `cdaeb36` | `felhom-controller` HEAD; live image on guest 9201 | -| host-install | 1.22.0 | **1.22.0** | `scripts/felhom-host-install.sh:187` | -| felhom.eu | `91a1dad` | **`91a1dad`**, clean, == `origin/main` | `git rev-parse` | - -These are the versions **on `main` and on the demo boxes**. They are not the versions a fresh install -receives — that distinction is the whole finding. - ---- - -## 2. THE BLOCKER — the Day-0 artifact channel is 17 agent releases stale - -### 2.1 The chain, read at source - -`felhom-host-install.sh` does not use `main`. It resolves a hub-vouched manifest and fetches versioned -packages from Gitea: - -- `resolve_artifacts()` (`:423-436`) → `GET $HUB_URL/api/v1/artifacts/$CUSTOMER_ID` → `agent.version`, - `golden.version` (served by `hub/internal/api/handler.go:2120` `handleArtifactManifest`). -- agent binary ← `$GITEA_BASE/api/packages/admin/generic/felhom-agent/$ART_AGENT_VER/felhom-agent` (`:1945`) -- golden ← `$GITEA_BASE/api/packages/admin/generic/felhom-golden/$ART_GOLDEN_VER/golden.tar.zst` (`:2573`) - -### 2.2 What that channel actually holds (Gitea API, authenticated, 2026-07-29) - -``` -felhom-agent : newest = 0.96.0 - all = 0.79.0 0.80.0 0.81.0 0.84.0 0.85.0 0.86.0 0.87.0 0.88.0 0.89.0 - 0.90.0 0.91.0 0.91.1 0.91.2 0.92.0 0.92.1 0.93.0 0.96.0 -felhom-golden: newest = 0.161.0 - all = 0.136.0 0.143.0 0.146.0 0.153.0 0.161.0 -``` - -Hub's saved Day-0 manifest (`/configuration`, selected options): **agent `0.96.0`** -(sha `af938601…`), **golden `0.161.0`** (sha `77624408…`), min-agent `0.93.0`. -Hub global controller floor: **v0.156.0** (DB override; env fallback v0.120.0). - -### 2.3 Therefore a fresh box lands on - -| Component | Fresh install gets | `main` / demo boxes | Gap | -|---|---|---|---| -| host-install | **1.22.0** (website git-sync from `main`) | 1.22.0 | none | -| agent | **0.96.0** | 0.113.0 | **17 releases** | -| controller | **0.161.0** (golden bake; floor 0.156.0 < 0.161.0 ⇒ no self-update) | 0.185.1 | **24 releases** | - -**Agent 0.113.0 is not in the channel at all** — it reached both demo boxes by direct deploy, never -through publish. Confirmed live: demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. - -### 2.4 What that strands — 17 unpublished agent releases (`felhom-agent/CHANGELOG.md`) - -| Version | What it carries | +| Claim | Verdict | |---|---| -| v0.97.0–v0.104.0 | **the entire R-82 per-target backup-tier arc** (local daily + PBS weekly), incl. v0.98.0's 30-minute false-failure bound, v0.101.0/0.102.0 scratch-guest + defer fixes, v0.103.0 R-84, v0.104.0 R-85 restore-test | -| v0.105.0 | R-88 Part 2 — the agent can say `unknown` | -| v0.106.0 | **F-CRIT-2** — a failed backup must not look like a fresh one (the phantom snapshot that reset the freshness clock, 7 days silent) | -| v0.107.0 | **F-REBOOT** — a guest rebooted during its backup never comes back | -| v0.108.0, v0.110.0 | F-LEAK — scratch-guest VMID band leak | -| v0.109.0 | F-OBS — the guest-power watchdog's positive observable | -| v0.111.0 | E-2c — the backup-target drive can no longer be ejected out from under the backup | -| v0.112.0 | **E-2b** — `GET /disks` flags the backup-target drive | -| v0.113.0 | **E-2a** — the guarded wrapper + `POST /backup/target` | - -**A new customer box installed today therefore runs an agent that predates the whole tiered-backup -model and lacks F-CRIT-2 and F-REBOOT** — two customer-impacting silent-failure fixes. That is a -larger finding than E-2d itself and is filed as **R-111**. +| **C1** host-install 1.22.0 completes a real install, rc=0 | ✅ **PROVEN** | +| **C2** Case B fires naturally on a single-drive box | ✅ **PROVEN** | +| **C3** the degraded banner renders **to a customer** | ⚠️ **PARTIAL — API exact, NO UI CONSUMER** (R-112) | +| **C4** the offer appears and moves the target when accepted | ⚠️ **PARTIAL — full API flow proven; offer equally invisible** (R-112) | +| **C5** `backup_target_absent` fires end to end | ❌ **FAILED** — no event on any channel (R-113) | --- -## 3. Claim-by-claim impact, each pinned to its introducing commit +## 1. Baselines as actually confirmed -| Claim | Needs | Introduced in | Fresh box has | Verdict | -|---|---|---|---|---| -| **C1** rc=0 real install, banner names 1.22.0 | host-install 1.22.0 | website ← `main` | **1.22.0** | **ACHIEVABLE** | -| **C2** Case B DEGRADED lines + `resolved local` | host-install `configure_backup_target()` `:627`, Case B `:653-655` | host-install 1.22.0 | **1.22.0** | **ACHIEVABLE** — installer-side only | -| **C3** degraded banner via `GET /api/storage/backup-target` | controller **v0.185.1** (`cdaeb36`); copy „lemezhiba ellen nem" **v0.185.0** (`3f7cf2a`) | — | controller 0.161.0 | **BLOCKED** — endpoint and copy do not exist | -| **C4** offer → `POST …/assign` → `restart_required` | controller **v0.185.1** (`cdaeb36`) + agent **v0.113.0** `POST /backup/target` (`58b598b`) | — | 0.161.0 / 0.96.0 | **BLOCKED** | -| **C5** `backup_target_absent` / `_restored` | controller **v0.184.0** (`c1a63de`) + agent **v0.112.0** | — | 0.161.0 / 0.96.0 | **BLOCKED** | - -**C3 has a cheap unblock; C4 and C5 do not.** C3 needs only the hub's global controller floor raised -0.156.0 → ≥0.185.0, because controller **0.185.1 IS published** to the registry (60 tags, newest -0.185.1) — a fresh box would self-update on its first report. C4/C5 need agent 0.112.0/0.113.0 -*published as Gitea generic packages*, which §0 of the runbook explicitly forbids this run from doing -(*"Does not touch: … any published artifact"*). - -Fleet impact of raising the floor, measured rather than assumed (hub dashboard, 2026-07-29): - -| Customer | Controller | Effect of floor → 0.185.1 | +| Artifact | Confirmed | Source | |---|---|---| -| Demo Ügyfél | 0.185.1 | none — already at it | -| Demo HP | 0.185.1 | none — already at it | -| Peti Proxmox | 0.115.0, **DOWN 14d** | already below the current 0.156.0 floor, so it updates on return either way; only the target version changes | +| hub | 0.81.0 | `manifests/hub.yaml:128`; live deploy image | +| agent | 0.113.0 | `main` @ `58b598b`; **published this run**, sha `5f3247f7…` | +| golden | **0.185.1** | baked this run, sha `dba00f3e…`, embeds controller 0.185.1 | +| controller | 0.185.1 | `main` @ `cdaeb36`; live in the guest | +| host-install | 1.22.0 | `scripts/felhom-host-install.sh:187`; fetched from the website at run time | +| felhom.eu | `3dff357` | `main` HEAD at run time | ---- +**Route: ISO/PAIRING (the real customer chain).** ISO +`felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso`. Every C1–C5 result is on the real route; no +manual-installer fallback was used. -## 4. Phase 0 gate answers (all four completed before the stop) +**Operator STOP: not required and now retired.** `HUB_PW` is in `~/.config/credentials`; CC created +the customer and performed the bind itself. The one human step that *was* needed is new — see §6. -### 4.1 Storage placement — gate PASSES, fence confirmed +## 2. Timeline (VM 9300 `e2d-fresh` on demo-hp, nested PVE) -`pvesm status` on demo-hp, 2026-07-29 (BEFORE; there is no AFTER — no VM was created): +| UTC | Event | +|---|---| +| 10:29 | VM created — q35/OVMF SB-off, 4c/8G, **one** 160 G disk, `hotplug disk`, outside the `felhom` pool | +| 10:34:52 | PVE auto-install done, first boot, DHCP `192.168.0.125` (no R-59 gate trip) | +| 10:34:53 | registered as unclaimed appliance, pairing code `SB4-7ZK`, console banner rendered | +| 10:36:52 | **bound** to customer `e2d-fresh` by CC; credentials delivered 2 s later | +| 10:37:31 | host enrolled `e2d-fresh-ac9f09`; break-glass root credential vaulted | +| 10:37:33 | **artifact manifest served: agent=0.113.0 golden=0.185.1** | +| 10:40:27 | `controller_started (0.185.1)` — **bind → running controller in 3 m 35 s** | +| 10:56 | second 100 G disk hot-attached; wizard init→mount→register | +| 10:57:08 | offer accepted → `restart_required:true`; agent restarted at 10:57:36 | +| 10:58:37 | target drive **hot-detached** (volume survives as `unused0`) | +| 10:58:37–11:03 | **no event on any channel** for 4½ minutes (budget was 60 s) | +| 11:03:31 | reattached; state returns healthy; **still no event** | + +## 3. C1 — PROVEN + +From the `felhom-bootstrap.service` journal on the box: ``` -Name Type Status Total (KiB) Used (KiB) Available (KiB) % -felhom-backup dir active 983379700 2293856 931059232 0.23% -felhom-pbs pbs active 0 0 0 0.00% -local dir active 40516856 14721460 23705004 36.33% -local-lvm lvmthin active 56545280 21922605 34622674 38.77% +felhom-bootstrap: fetching host-install: https://felhom.eu/scripts/felhom-host-install.sh +[INFO] felhom-host-install v1.22.0 — mode=appliance customer=e2d-fresh vmid=9201 +... +[OK] controller: Up 21 seconds (healthy) +[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.185.1 +[OK] Day-0 provision SUCCESS — vmid=9201 host_id=e2d-fresh-ac9f09 customer=e2d-fresh + golden=local:backup/vzdump-lxc-9100-2026_07_29-12_37_56.tar.zst +felhom-bootstrap: host-install SUCCESS — writing done-flag, disabling unit, scrubbing secrets ``` -`local-lvm` is thin, pool `<53.93g`, **data 38.77 %** — matching the runbook's "~54 GB pool, 38.8 %" -to 0.03 pp, so the picture has **not** materially changed. Allocated LVs on it: `vm-9201-disk-{0,1,2}` -(32+50+20 G, the **live customer guest**), `vm-300-disk-{0,1}` (drill-r50, 32 G) and its `r50pre` -snapshots (32 G) ⇒ ~166 G allocated against 53.93 G real. `vgs` shows only 14.75 G VFree. -**The fence holds: nothing goes on `local-lvm`.** +rc=0 is corroborated structurally: the unit wrote its done-flag, **self-disabled**, and +`Deactivated successfully`. `pct list` showed guest **9201 `e2d-fresh` running**. -`/mnt/nvme-1tb` = `/dev/nvme0n1`, ext4, 938 G, **888 G free**. ≥100 G requirement satisfied. +**The publish train is proven end to end:** the golden restored is +`vzdump-lxc-9100-2026_07_29-12_37_56` — the golden baked ~20 minutes earlier in the same session. +This retires E-2's *"installer-logic-tested, not install-tested"*. -**Placement decision (recorded deliberately, per §3.1):** the only storage with `content images` is -`local-lvm` (forbidden). `felhom-backup` is a `dir` at exactly `/mnt/nvme-1tb` but is `content backup` -only **and is demo-hp's live backup target** — widening its content set would mutate the very storage -E-2c/E-2 role logic keys on, which §0/§9 forbid. So a *new* dir storage would have been required. -Both remaining shapes carry a cost and the choice was **not** forced this run because the run stopped: -a second storage at the same mountpoint risks perturbing the agent's role resolution on a live box; -a storage at a **subdirectory** fails `exactMount` and reports `disconnected` in the host report, -which is not purely cosmetic since it can reach the hub's storage monitor. **This decision is left -open and is an input to any resumed run.** +## 4. C2 — PROVEN -### 4.2 Drive-gate cadence — answered +Both required warning lines, verbatim, ANSI-stripped: -- Symbol: `driveGateLoop` → `s.ReconcileDriveGates()` -- Definition: `controller/internal/web/intermediary.go:328` (gate itself at `:267`) -- Registration: `controller/internal/web/server.go:217` — `go s.driveGateLoop()` -- **Interval: `time.NewTicker(30 * time.Second)`** (`intermediary.go:337`) +``` +[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive. +[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard. +``` -⇒ Stage 7's derived budget would be **two cycles = 60 s** of controller-side reconcile, *plus* the -agent's own `/disks` refresh, since `ReconcileDriveGates` consumes `resp.Disks` (`:281`). Recorded so -a resumed run does not have to re-derive it. +- `agent.json` → `backup.local_backup_target = 'local'` (the "resolved local" observable) +- no `felhom-backup` storage created at install +- **the install did not abort** — a single-drive appliance is a valid product -### 4.3 Hot-detach — feasible, not exercised +## 5. THE FINDINGS -demo-hp: AMD `svm` present, `/sys/module/kvm_amd/parameters/nested = 1`, 8 cores, 29 GB RAM (25 GB -available). The working nested-PVE reference is VM 300 (`drill-r50`): `bios: ovmf`, -`efitype=4m,pre-enrolled-keys=0` (Secure Boot OFF, as the mkimage loader requires), `machine: q35`, -`scsihw: virtio-scsi-single`, `cpu: host`. `virtio-scsi-single` + PVE's default -`hotplug: disk,network,usb` supports SCSI hot-detach, so the **live-transition** variant of Stage 7 -was available. Not exercised. +### 5.1 R-112 — the degraded banner and the offer have NO UI consumer (customer-invisible) -### 4.4 Install route — ISO available, route selected, not used +The endpoint is **correct**. Authenticated `GET /api/storage/backup-target` returned, exactly: -Two generic (pairing) ISOs are present on demo-hp `local`: -`felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso` (**preferred**, the v1.25.0 train, mkimage -loader for nested VMs) and `felhom-pve-9.2-1-v1.22.0-nested-canary-generic.iso`. -The **ISO/PAIRING route was selected**. It reaches the same installer invocation — `run_pairing` -(`felhom-bootstrap.sh:441`) falls through to `run_direct` (`:495-499`) in the same invocation, which -fetches `$INSTALL_URL` (`:322-330`) and runs it (`:343`). Not used — the run stopped first. +```json +{"degraded":true,"known":true,"target":"local", + "message":"A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer — így hibás fájlok + ellen véd, lemezhiba ellen nem. Csatlakoztass egy második meghajtót a teljes védelemhez."} +``` -**Operator STOP: not required.** `HUB_PW` is present in `~/.config/credentials` and hub auth was -verified working (`http://10.43.52.34:8080/` → 200 with `curl -u ":$HUB_PW"`; public ingress also 200), -so CC could have performed the bind itself. The runbook's §5.1a STOP is retired for future runs. +byte-identical to the runbook's required copy. `known` is a field **separate** from `degraded`, so +UNKNOWN genuinely cannot render as degraded (R-88 Part 2's lesson held). ---- +**And nothing in the product ever asks for it.** Negative claims with their search scope: -## 5. Incidental observations (filed, not acted on) +| Search | Result | +|---|---| +| `grep -rn 'backup-target' --include='*.html' --include='*.js' --include='*.css' controller/` | **0 hits** | +| `grep -rn 'kijelölheted\|ugyanazon a lemezen\|OfferPath\|OfferLabel\|Degraded' controller/internal/web/templates/` | **0 hits** | +| consumers of `backupTargetDegradedText` / `backupTargetOfferText` | only `degradedMessageFor` (`:136`) and the JSON handler (`:153`) — both inside `backup_target_offer.go` | +| consumers of `resolveBackupTargetState` / `degradedMessageFor` across all Go | **only** the API handler. **No page handler injects the state.** | -1. **A stale unclaimed appliance is sitting in the hub**: `206c8838-7755-4751-8a16-a842853d718f`, - pairing code `QWA-WJE`, MAC `bc:24:11:ea:55:d0`, "Standard PC (Q35 + ICH9, 2009)" on an - AMD V1756B with 7.7 GB — i.e. a **nested VM on demo-hp**, first and last seen **5 d ago**. It is a - leftover from the 2026-07-25 ISO work and has never been bound or discarded. A resumed run must - distinguish its own appliance from this one, or discard it first. -2. **The agent has no container image** (`/v2/admin/felhom-agent/tags/list` → 0 tags). It ships only - as a Gitea generic package, which is why the publish gap is invisible from the registry. -3. `drill-r50` (VM 300) still holds a `r50pre` snapshot pair consuming ~32 G of allocation on the - over-subscribed `local-lvm`. Untouched per §9; noted because it is part of why the pool is tight. +**The decisive contrast:** the templates fetch **18 distinct `/api/storage/*` endpoints`**. +`backup-target` and `backup-target/assign` are the **only two** referenced by zero templates. ---- +The handler's own doc comment reads *"serves GET /api/storage/backup-target — **the dashboard's source +for the degraded banner and the offer**"* — an invariant comment asserting a consumer that does not +exist (`CLAUDE.md`'s "a comment asserting an invariant needs a test pinning it, or it is a wish", +instance #7). And v0.185.1's own test, `TestBackupTargetRoutesLiveUnderTheStorageAPIMount`, pins that +the **router dispatches** the path — not that anything renders it. v0.185.1 shipped as *"the offer +endpoints were mounted where nothing routed to them"*: it fixed the mount and stopped one layer short. -## 6. What did not happen +**Fifth instance of the seam-built-but-never-wired class.** `CLAUDE.md`'s seam rule names exactly this: +*"a feature is not shipped until its entry point is reachable… handler tests that POST directly prove +nothing about reachability."* -No VM was created. No ISO booted. No install ran. No appliance registered, bound or discarded. No -storage was created or modified on demo-hp. No drive attached or detached. No hub setting changed -(the floor was **read**, not written). No teardown was needed. **C1–C5 are all UNPROVEN**, and the -E-2 / E-2d rows are unchanged except to record this blocker. +### 5.2 R-113 — the drive-absent gate CANNOT fire on device loss (E-2b's alarm is unreachable) + +Detached the assigned target drive at 10:58:37Z under a running agent. Over the next 4½ minutes: + +- **agent**, every 20 s: `storage: enrolled drive absent by UUID — not re-asserting` / + `reconcile: enrolled drive not present (durable-id absent) — skipping` — **the agent knows** +- **controller**: `docker logs | grep -c '\[gate\]'` → **0**. The gate never acted, ABSENT or RETURNED +- **hub**: zero events for the customer across the whole window — no `backup_target_absent`, + **and no generic `storage_disconnected` either** + +**Root cause, established at source and confirmed live.** `planDriveGates` +(`intermediary.go:216-262`) computes presence as +`present[GuestPath] = present[GuestPath] || d.BoundUnderParent`, and the agent derives +`BoundUnderParent` from `GuestSeesMount()` — *"does the guest's `/proc//mountinfo` list this path +as a mount target"* (`localapi/disks.go:210`, `localapi/intermediary.go` `GuestSeesMount`). + +Measured on the box with the device removed: + +``` +-- raw mount /mnt/mentes2 -- NOT mounted <- systemd device-bound unit, died with the device +-- stable bind /mnt/felhom-drives/mentes2 -- /dev/sdb[/felhom-data] ext4 <- the agent's MANUAL bind, SURVIVES +``` + +The raw mount is a device-bound systemd unit and dies correctly; **the agent's own bind under the +shared parent is not device-bound, so its mountinfo entry outlives the device.** The gate reads that +surviving entry as "present" ⇒ `!present[...]` is never true ⇒ no `Stop` action ⇒ `notifyDriveAbsent` +is never called. + +This is not a virtualisation artefact: the asymmetry is between a device-bound mount and a manual +bind, which is identical on physical hardware. **Caveat kept honest:** proven on a SCSI hot-detach; a +physical USB unplug was not staged. + +**Consequence.** E-2b's celebrated fix — *"THE SEAM THAT WAS NEVER WIRED… a drive that is ONLY a +backup target has no apps to stop, so it was silent twice over"* (`intermediary.go:288-296`) — wired +the notify to a branch that **cannot execute on device loss**. The seam is wired; the condition is +unreachable. Sixth instance of the class, one layer deeper than the fifth. + +**Mirror scenario:** not separately staged, and it does not need to be — both the specific and the +generic event are emitted from the same `a.Stop` branch, which never executed. The generic +`storage_disconnected` is equally unreachable by this path. Recorded as *reasoned, not observed*. + +### 5.3 R-114 — on target-drive loss the customer is told the wrong story and offered the missing drive + +While the target drive was absent, the endpoint returned: + +```json +{"degraded":true,"target":"felhom-backup", + "message":"A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer …", + "offer_path":"/mnt/felhom-drives/mentes2","offer_label":"Mentés meghajtó"} +``` + +Two defects in one payload: the message claims the backup is **on the system disk**, which is false — +the target is `felhom-backup` on a drive that has vanished; and the remedy offered is **the drive that +just disappeared**. `resolveBackupTargetState` falls through to the generic degraded branch whenever no +disk satisfies `d.BackupTarget && d.MountPath != ""`, without distinguishing *never configured* from +*configured and now missing*. + +**Interaction worth stating: R-114 is currently invisible only because of R-112.** Fixing R-112 alone — +wiring the banner — would immediately start showing customers this wrong message. **They must be fixed +together, R-114 first.** + +**Also observed:** after reattach the drive returned as `/dev/sdc`, while the stable bind still +recorded `/dev/sdb[/felhom-data]`. The state read healthy (`degraded:false`) with the guest-visible +bind still naming the dead device node. Not chased further; recorded as part of R-113's shape. + +### 5.4 Smaller findings (recorded, not filed as their own IDs) + +1. **A "hard min" that only warns.** `[WARN] local-lvm free ~83 GiB < hard min 120 GiB` — the installer + names a hard minimum and proceeds. Either it is not hard, or the wording is wrong. +2. **`felhom-backup-target-apply` is fetched unvouched.** host-install pulls **nine** files from + `raw/branch/main` (`:2072`–`:2206`); the hub manifest vouches a sha for exactly one + (`wrapper_sha256` → `felhom-pbs-apply`, verified this run: no drift). E-2a's wrapper is installed + **0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n`. + **Filed as a second instance under R-110**, not a new ID — same class (a root-executed artifact + taken from `main` with no pinned integrity). + +## 6. C4 — what IS proven (API level) + +Everything except customer reachability: + +- **The offer** appeared with all three fields: `offer_path=/mnt/felhom-drives/mentes2`, + `offer_label=Mentés meghajtó`, `offer_message=Ezt a meghajtót kijelölheted a rendszermentés helyéül…` +- **The decline path (§6.4) — PROVEN.** After registering the drive and *not* accepting: + `target` still `local`, no `felhom-backup` storage, `agent.json` unchanged. **Registration does not + confer a role** — E-2 §3's central invariant, live. +- **Accept** → `{"assigned":"/mnt/felhom-drives/mentes2","restart_required":true}` +- **The agent did NOT self-restart** — `ActiveEnterTimestamp` unchanged at 12:37:53 CEST twenty + minutes later. **In-flight check performed and recorded before restarting:** 0 running PVE tasks, + no `vzdump` process, 0 backup lines in the agent journal. +- **The E-2a root-fenced wrapper worked on a fresh box:** created `dir: felhom-backup`, + `path /mnt/mentes2`, `is_mountpoint 1` — the drive's **own** mountpoint (law F-2). +- **Healthy renders nothing:** after the restart the payload is + `{"degraded":false,"known":true,"label":"Mentés meghajtó","target":"felhom-backup"}` — **no + `message` field at all**. No "backup protected" reassurance (E-2 Scenario E). + +## 7. E-2d's own premise needs amending + +The runbook assumed a fresh install yields a claimable customer CC can then drive. It does not: +**the claim code is bcrypt-hashed in the hub and only ever emailed** (`claim/engine.go:58-95`), and the +gate covers everything except `/claim`, `/claim/request-new-code`, `/api/health`, `/static/` +(`claim.go:222-229`). C3/C4/C5 all sit behind it. This run cleared it by registering an operator email, +resending, and having the operator relay the code — **the one genuine human step**, and it also proved +the claim flow end to end (code → password → `401 "dashboard not yet claimed"` becoming +`401 "authentication required"`, a positive discriminator). + +## 8. Teardown + +- VM 9300 destroyed `--purge`; `e2d-images` storage removed; scratch dir removed. +- **`pvesm status` after == before**: `local-lvm` **38.77 %**, `felhom-backup` 931059224 KiB available + — byte-identical to the pre-run measurement. Freed space returned. +- **Guest 9201 (live customer) untouched and running; `drill-r50` VM 300 untouched.** +- Drill VM (golden bake) torn down per GL-1 earlier: guest 9100 purged, secrets shredded, disk restored + to `virgin`, token-leak grep 0. + +**Hub records NOT yet removed — stated, not silent.** `e2d-fresh` + host `e2d-fresh-ac9f09` remain. +The delete was attempted and **correctly refused** at four successive gates: acknowledgements → typed +`confirm_id` → `expect_hosts` stale-preview → finally `host e2d-fresh-ac9f09 is ONLINE`. The host is +online only because its last report is recent; the VM is destroyed, so it ages OK → WARN (30 m) → DOWN +(>1 h) and is then deletable. **Cleanup command, once it reads DOWN:** + +``` +POST /configs/e2d-fresh/delete ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=e2d-fresh expect_hosts=1 +``` + +Tracked in `OPEN-ITEMS.md`. Deleting rather than keeping is deliberate — R-93 records what a +half-real fixture costs. + +**Also still present:** the stale unclaimed appliance from 2026-07-25 (`206c8838…`, code `QWA-WJE`). +Not mine; not discarded. A future run must distinguish its own appliance from it. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 44b440b..91484c8 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -11,10 +11,14 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha |---|---|---|---|---|---| | **R-88a** | ~~Failing backup re-quiesces every 5 min, no backoff~~ | **SHIPPED** (controller v0.176.0, 2026-07-27) | — | Live on both boxes; breaker 15m→4h, per-tier, never permanent | — | | **R-88b** | ~~`/backup/due` cannot say *unknown*~~ | **SHIPPED + PROVEN-LIVE** (agent v0.105.0 + controller v0.178.0, 2026-07-27) | — | `age_state=unknown` captured on real hardware during a deliberate ep0 outage; controller deferred, **zero app stacks stopped** | — | -| **E-2d** | **Prove E-2 on a fresh VM on the t740** — the only remaining route to four unproven items: a real `felhom-host-install.sh` **1.22.0** run (never done), Case B naturally (single-drive install renders the degraded banner without degrading a live box), a **claimable** customer so the three claim-gated items stop being gated, then add a drive (the offer) and unplug it (`backup_target_absent` end-to-end) | **READY (M)** | — | **Space checked 2026-07-29 — NOT a blocker, with one constraint: the VM disk must NOT go on `local-lvm`.** That thin pool is over-subscribed (144 GB allocated against 54 GB, 38.8% used) on a box running a live customer guest, and a full thin pool corrupts every guest on it. `local` has only 23.7 GB and sits on `pve-root`. **Use `/mnt/nvme-1tb` (888 GB free).** Caveat found: a dir storage at a SUBDIRECTORY there will fail the agent's `exactMount` check and report `disconnected` in the host report — cosmetic, but decide the placement deliberately. **Do NOT unblock drill-r50** (deliberately blocked; unblocking it means the fixture stops representing anything real). **CORRECTED 2026-07-29 — the ISO is not an obstacle to the 1.22.0 proof, it is the best route to it.** `felhom-bootstrap.sh:96` fetches from `https://felhom.eu/scripts/felhom-host-install.sh`, not the hub, and that URL serves **1.22.0** (git-sync from `main`, ≤30 s — live fetch confirmed). A fresh ISO install therefore exercises 1.22.0 **automatically** — which makes the ISO leg the *stronger* proof, because it is the real customer path rather than a proxy for it, and it retires the "run 1.22.0 manually" step. **The Phase 0 question that decides the shape is ANSWERED — it was a source read, and the answer is yes.** A fresh VM with no baked customer-id lands in **PAIRING** mode (`felhom-bootstrap.sh:537-541`), not **DIRECT** (`:312`), and only DIRECT passes `--customer-id / --mode / --passphrase-file`. But on a 200 from `/api/v1/appliance/poll` the pairing loop writes the hub-delivered `FELHOM_CUSTOMER_ID` + `FELHOM_RETRIEVAL_PASSPHRASE` into the 0600 env, re-sources it and calls `run_direct` **in the same invocation** (`:495-499`) — so pairing reaches the identical installer invocation (`:322-343` — the single `$INSTALL_URL` fetch at `:322-330`, the `--customer-id/--mode/--hub-url/--passphrase-file` args array at `:334`, and the `bash "$SCRIPT_TMP" "${args[@]}"` call itself at `:343`) and the customer it yields is the one the operator bound, i.e. claimable. **So the ISO leg is the spine**; a manual 1.22.0 run is not needed as a separate scenario. **RUN ATTEMPTED 2026-07-29 — STOPPED AT PHASE 0, no VM created, nothing touched (`audits/E2D-fresh-vm-2026-07-29.md`).** The blocker is **R-111**: a fresh box installs **agent 0.96.0 + controller 0.161.0**, not `main`'s 0.113.0/0.185.1, so **C3/C4/C5 test surfaces that do not exist on it** — the degraded banner + `GET /api/storage/backup-target` landed in controller **v0.185.1** (`cdaeb36`) with the copy in **v0.185.0** (`3f7cf2a`); `backup_target_absent` in **v0.184.0** (`c1a63de`); the offer's apply needs agent **v0.113.0** (`58b598b`). **C1 (a real rc=0 1.22.0 install) and C2 (Case B natural, `configure_backup_target()` `felhom-host-install.sh:627`, warnings `:653-655`) remain ACHIEVABLE TODAY** — both are installer-side and host-install is served at 1.22.0. **C3 unblocks cheaply** by raising the hub global floor 0.156.0 → ≥0.185.0 (controller 0.185.1 IS in the registry): measured fleet impact is **nil** — both demo boxes already run 0.185.1, and Peti is DOWN 14 d and already below the current floor. **C4/C5 need R-111 first.** Phase 0 answers are all recorded in the audit, so a resumed run does not re-derive them: drive-gate cadence **30 s** (`intermediary.go:337`, registered `server.go:217`) ⇒ a 60 s two-cycle budget; hot-detach available (`virtio-scsi-single` + default hotplug, VM 300 is the working reference); ISO present (`…v1.25.0-nested-vm-generic-mkimage.iso`); `/mnt/nvme-1tb` 888 G free and the `local-lvm` fence re-measured (38.77 %, unchanged). **The §5.1a operator STOP is retired** — `HUB_PW` is in `~/.config/credentials` and hub auth was verified, so CC can bind. **One open decision carried forward:** where the VM disk's dir storage goes, since `local-lvm` is forbidden and `felhom-backup` is the live backup target — see the audit §4.1 | CC | -| **R-94** | **A hand-synced version constant drifts, and the gate that would catch it is never run** — `hub/internal/web/configs.go:28` pins `hostInstallVersion = "1.19.0"` while `scripts/felhom-host-install.sh:187` is `SCRIPT_VERSION="1.22.0"` | **READY (XS)** | — | **CORRECTED 2026-07-29 — the earlier framing of this row was false and is retracted.** The constant selects no script: its only consumers are `configs.go:487` (`ScriptVersion`) and `render_test.go:219`, and it renders as a label at `customer_unified.html:494`. The install command beneath that label fetches `https://felhom.eu/scripts/felhom-host-install.sh` (`customer_unified.html:563`, `:1262`), which the website git-syncs from `main` on a 30 s period (`manifests/webpage.yaml`) — so **1.22.0 is what every install already gets** (live fetch, 2026-07-29). Every flag the generator emits is parsed by 1.22.0 (`customer_unified.html`~`:1210`–`:1238` vs `felhom-host-install.sh:1177`–`:1210`): **no functional gap, only a wrong number on the operator's screen.** Three legs, all XS: **(a)** derive the label from `SCRIPT_VERSION` rather than hand-syncing it, or delete it; **(b)** `scripts/hostinstall_gates.py` **fails today** and is invoked by no Makefile, hook or `CLAUDE.md` — wire it next to `site_gates.py` or delete it, because a gate nobody runs reads as coverage it is not providing (**this leg is one instance of → R-29**, which is the class: gates are enforced nowhere, and the enforcement decision belongs there, not here); **(c)** `render_test.go:219` compares the constant to itself and passes at any value — replace it with the cross-file assertion. **No longer blocked on E-2d** — it never gated anything | CC | -| **R-110** | **`main` is the installer's publish channel — there is no staging.** `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a 30 s period and nginx serves that working tree directly (`location /scripts/`, `root …/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as **root on a virgin box**, the single most privileged thing Felhom ships | **WAITING-ON-OPERATOR (S)** | operator ruling | **Two consequences worth stating:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" **was never available to take**. **Open question for the operator, not a defect to fix blind:** whether `/scripts/` should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether `main`-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now | CC | +| **E-2d** | **Prove E-2 on a fresh VM on the t740** — the only remaining route to four unproven items: a real `felhom-host-install.sh` **1.22.0** run (never done), Case B naturally (single-drive install renders the degraded banner without degrading a live box), a **claimable** customer so the three claim-gated items stop being gated, then add a drive (the offer) and unplug it (`backup_target_absent` end-to-end) | **READY (M)** | — | **Space checked 2026-07-29 — NOT a blocker, with one constraint: the VM disk must NOT go on `local-lvm`.** That thin pool is over-subscribed (144 GB allocated against 54 GB, 38.8% used) on a box running a live customer guest, and a full thin pool corrupts every guest on it. `local` has only 23.7 GB and sits on `pve-root`. **Use `/mnt/nvme-1tb` (888 GB free).** Caveat found: a dir storage at a SUBDIRECTORY there will fail the agent's `exactMount` check and report `disconnected` in the host report — cosmetic, but decide the placement deliberately. **Do NOT unblock drill-r50** (deliberately blocked; unblocking it means the fixture stops representing anything real). **CORRECTED 2026-07-29 — the ISO is not an obstacle to the 1.22.0 proof, it is the best route to it.** `felhom-bootstrap.sh:96` fetches from `https://felhom.eu/scripts/felhom-host-install.sh`, not the hub, and that URL serves **1.22.0** (git-sync from `main`, ≤30 s — live fetch confirmed). A fresh ISO install therefore exercises 1.22.0 **automatically** — which makes the ISO leg the *stronger* proof, because it is the real customer path rather than a proxy for it, and it retires the "run 1.22.0 manually" step. **The Phase 0 question that decides the shape is ANSWERED — it was a source read, and the answer is yes.** A fresh VM with no baked customer-id lands in **PAIRING** mode (`felhom-bootstrap.sh:537-541`), not **DIRECT** (`:312`), and only DIRECT passes `--customer-id / --mode / --passphrase-file`. But on a 200 from `/api/v1/appliance/poll` the pairing loop writes the hub-delivered `FELHOM_CUSTOMER_ID` + `FELHOM_RETRIEVAL_PASSPHRASE` into the 0600 env, re-sources it and calls `run_direct` **in the same invocation** (`:495-499`) — so pairing reaches the identical installer invocation (`:322-343` — the single `$INSTALL_URL` fetch at `:322-330`, the `--customer-id/--mode/--hub-url/--passphrase-file` args array at `:334`, and the `bash "$SCRIPT_TMP" "${args[@]}"` call itself at `:343`) and the customer it yields is the one the operator bound, i.e. claimable. **So the ISO leg is the spine**; a manual 1.22.0 run is not needed as a separate scenario. **RUN ATTEMPTED 2026-07-29 — STOPPED AT PHASE 0, no VM created, nothing touched (`audits/E2D-fresh-vm-2026-07-29.md`).** The blocker is **R-111**: a fresh box installs **agent 0.96.0 + controller 0.161.0**, not `main`'s 0.113.0/0.185.1, so **C3/C4/C5 test surfaces that do not exist on it** — the degraded banner + `GET /api/storage/backup-target` landed in controller **v0.185.1** (`cdaeb36`) with the copy in **v0.185.0** (`3f7cf2a`); `backup_target_absent` in **v0.184.0** (`c1a63de`); the offer's apply needs agent **v0.113.0** (`58b598b`). **C1 (a real rc=0 1.22.0 install) and C2 (Case B natural, `configure_backup_target()` `felhom-host-install.sh:627`, warnings `:653-655`) remain ACHIEVABLE TODAY** — both are installer-side and host-install is served at 1.22.0. **C3 unblocks cheaply** by raising the hub global floor 0.156.0 → ≥0.185.0 (controller 0.185.1 IS in the registry): measured fleet impact is **nil** — both demo boxes already run 0.185.1, and Peti is DOWN 14 d and already below the current floor. **C4/C5 need R-111 first.** Phase 0 answers are all recorded in the audit, so a resumed run does not re-derive them: drive-gate cadence **30 s** (`intermediary.go:337`, registered `server.go:217`) ⇒ a 60 s two-cycle budget; hot-detach available (`virtio-scsi-single` + default hotplug, VM 300 is the working reference); ISO present (`…v1.25.0-nested-vm-generic-mkimage.iso`); `/mnt/nvme-1tb` 888 G free and the `local-lvm` fence re-measured (38.77 %, unchanged). **The §5.1a operator STOP is retired** — `HUB_PW` is in `~/.config/credentials` and hub auth was verified, so CC can bind. **One open decision carried forward:** where the VM disk's dir storage goes, since `local-lvm` is forbidden and `felhom-backup` is the live backup target — see the audit §4.1. **RUN EXECUTED 2026-07-29 after R-111 was fixed — `audits/E2D-fresh-vm-2026-07-29.md`.** Full ISO/PAIRING route on a nested VM on demo-hp; bind → running controller in **3 m 35 s**; teardown clean (`pvesm status` after == before, `local-lvm` 38.77 %, guest 9201 untouched). **C1 PROVEN** (`felhom-host-install v1.22.0`, `Day-0 provision SUCCESS`, guest 9201 running, golden = the one baked 20 min earlier). **C2 PROVEN** (both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort). **C3/C4 PARTIAL — API exact and complete, but customer-invisible → R-112.** **C5 FAILED → R-113.** C4's decline path PROVEN (registration confers no role), `restart_required:true` PROVEN, agent did **not** self-restart, E-2a wrapper created the storage at the drive's own mountpoint, healthy renders nothing. **E-2d's own premise needed amending:** a fresh install does NOT yield a CC-drivable claimable customer — the claim code is bcrypt-hashed and email-only, so one operator relay was required (and the claim flow is now proven end to end). **This row stays OPEN only for the residue:** C5 re-test after R-113, and the C3/C4 UI legs after R-112 | CC | +| **R-94** | **A hand-synced version constant drifts, and the gate that would catch it is never run** — `hub/internal/web/configs.go:28` pins `hostInstallVersion = "1.19.0"` while `scripts/felhom-host-install.sh:187` is `SCRIPT_VERSION="1.22.0"` | **READY (XS)** | — | **CORRECTED 2026-07-29 — the earlier framing of this row was false and is retracted.** The constant selects no script: its only consumers are `configs.go:487` (`ScriptVersion`) and `render_test.go:219`, and it renders as a label at `customer_unified.html:494`. The install command beneath that label fetches `https://felhom.eu/scripts/felhom-host-install.sh` (`customer_unified.html:563`, `:1262`), which the website git-syncs from `main` on a 30 s period (`manifests/webpage.yaml`) — so **1.22.0 is what every install already gets** (live fetch, 2026-07-29). Every flag the generator emits is parsed by 1.22.0 (`customer_unified.html`~`:1210`–`:1238` vs `felhom-host-install.sh:1177`–`:1210`): **no functional gap, only a wrong number on the operator's screen.** Three legs, all XS: **(a)** derive the label from `SCRIPT_VERSION` rather than hand-syncing it, or delete it; **(b)** `scripts/hostinstall_gates.py` **fails today** and is invoked by no Makefile, hook or `CLAUDE.md` — wire it next to `site_gates.py` or delete it, because a gate nobody runs reads as coverage it is not providing (**this leg is one instance of → R-29**, which is the class: gates are enforced nowhere, and the enforcement decision belongs there, not here); **(c)** `render_test.go:219` compares the constant to itself and passes at any value — replace it with the cross-file assertion. **No longer blocked on E-2d** — it never gated anything. **2026-07-29: a real 1.22.0 install has now happened** (`audits/E2D-fresh-vm-2026-07-29.md`), so even the original (retracted) precaution is discharged — nothing stands in front of this row | CC | +| **R-110** | **`main` is the installer's publish channel — there is no staging.** `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a 30 s period and nginx serves that working tree directly (`location /scripts/`, `root …/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as **root on a virgin box**, the single most privileged thing Felhom ships | **WAITING-ON-OPERATOR (S)** | operator ruling | **Two consequences worth stating:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" **was never available to take**. **Open question for the operator, not a defect to fix blind:** whether `/scripts/` should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether `main`-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now. **SECOND INSTANCE, found 2026-07-29 by the E-2d run and filed here rather than as a new ID:** `felhom-host-install.sh` fetches **nine** files from `raw/branch/main` (`:2072`–`:2206`) and the hub manifest vouches a sha for exactly **one** (`wrapper_sha256` → `felhom-pbs-apply`; re-checked this run, no drift). E-2a's `felhom-backup-target-apply` (`:2116`) is installed **0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n` — a root-executed artifact taken from `main` with no pinned integrity, which is this row's class exactly | CC | | **R-111** | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`.**~~ `felhom-host-install.sh` does not use `main`: it reads the hub-vouched manifest (`:423-436`) and fetches Gitea generic packages (agent `:1945`, golden `:2573`). Gitea's newest are **agent 0.96.0** and **golden 0.161.0**, and the hub's manifest selects exactly those — so a fresh box lands on **agent 0.96.0 + controller 0.161.0** (global floor `v0.156.0` < the golden's 0.161.0, so no self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and was never published** | **SHIPPED 2026-07-29 — the channel now serves agent 0.113.0 + golden 0.185.1** | — | **FIXED the same day it was found.** Agent **0.113.0** built from the clean tree @ `58b598b` and published (`scripts/publish-agent.sh`), sha `5f3247f756cb658e…`, round-trip GET verified. Golden **0.185.1** baked on the nested drill VM embedding controller `0.185.1`, published, sha `dba00f3e845c415e…` — bake clean: `Result=success`, overlay2, **all 3 mounts included** (rootfs+mp0+mp1), 0 FATAL/exclusions, upload HTTP 201, token-leak grep 0; log `drill/bake-0.185.1.log`; GL-1 teardown done (guest 9100 purged, secrets shredded, disk restored to `virgin`). Hub Day-0 manifest moved **both together in one POST** so it never vouched a new agent against an old golden; `min_agent` **0.93.0 → 0.113.0**, which is what controller v0.185.0 declares (`felhom-controller/CHANGELOG.md:15`) — **zero fleet impact, verified: all three enrolled hosts already run agent 0.113.0, so no box is held.** `wrapper_sha256` preserved verbatim (re-checked against `configs/felhom-pbs-apply` — no drift). **The global controller floor was deliberately NOT raised**: the golden now bakes 0.185.1, so a fresh box needs no self-update, and raising it would have been an unnecessary fleet-wide write. Original finding follows. **Found 2026-07-29 by the E-2d Phase 0 gate, which stopped the run before a VM was created.** 17 unpublished releases (v0.97.0–v0.113.0) strand the **entire R-82 tiered-backup arc** plus **F-CRIT-2** (a failed backup looking fresh — 7 days silent) and **F-REBOOT** (a guest rebooted mid-backup never returns): a new customer's box would install without them. **Blocks E-2d's C3/C4/C5** — those test endpoints and events that do not exist in 0.96.0/0.161.0. The **controller is fine** (registry has 0.185.1, floor-driven self-update), so the gap is specific to the two Gitea-generic artifacts. **Mirror of R-110, not a duplicate:** R-110 = the installer publishes instantly with no staging; R-111 = the agent/golden publish gate exists and was never walked. Fix should decide whether publishing joins the release train rather than staying a remembered step (R-29's shape, one layer up). Evidence: `audits/E2D-fresh-vm-2026-07-29.md` | CC | +| **R-113** | **The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.** `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc//mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **READY (M) — P1** | — | **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 | CC | +| **R-112** | **E-2's degraded banner and offer have NO UI CONSUMER — the endpoint is correct and the customer never sees it.** `GET /api/storage/backup-target` returns byte-exact copy (proven live), and **nothing fetches it**: `grep 'backup-target'` across all `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` + `degradedMessageFor` are consumed **only** by the JSON handler — **no page handler injects the state** | **READY (S) — P1** | — | **The decisive contrast: templates fetch 18 distinct `/api/storage/*` endpoints; `backup-target` and `backup-target/assign` are the only two with zero references.** The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"* — an invariant comment asserting a consumer that does not exist (7th instance of that class). v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* — it fixed the **router mount** and stopped one layer short of the **render**; its test `TestBackupTargetRoutesLiveUnderTheStorageAPIMount` pins dispatch, not reachability, which is exactly what `CLAUDE.md`'s seam rule warns about. **Fifth instance of seam-built-but-never-wired.** **Fix R-114 FIRST** — wiring this alone would start showing customers the wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | CC | +| **R-114** | **On target-drive loss the customer is told the wrong story and offered the drive that just vanished.** With the assigned target absent, the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"a rendszermentés ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a drive that has disappeared, not the system disk — **and** `offer_path` pointing at the missing drive as the remedy | **READY (S)** | — | `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing **never configured** from **configured and now missing**. Shares R-113's root cause — two disagreeing presence signals — but is a different code path with a different fix. **Currently invisible ONLY because of R-112; fix this before wiring that.** Also seen: after reattach the drive returned as `/dev/sdc` while the stable bind still recorded `/dev/sdb`, and the state read healthy. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 | CC | +| — | **Drill cleanup: hub records for `e2d-fresh` + host `e2d-fresh-ac9f09` still present** | WATCHING | the host aging OK → WARN (30 m) → DOWN (>1 h); the VM is already destroyed so no further reports come | Delete was attempted and **correctly refused** at four gates, finally `host … is ONLINE`. Once it reads DOWN: `POST /configs/e2d-fresh/delete ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=e2d-fresh expect_hosts=1`. Deleting rather than keeping is deliberate — R-93 records what a half-real fixture costs. Also still present: the **stale 2026-07-25 unclaimed appliance** `206c8838…` (code `QWA-WJE`), not mine, not discarded | CC | | **R-29** | **The green gates are not enforced anywhere — one was RED for 16 releases before anyone ran it.** This is the **class**, not an instance: a gate that exists, asserts something true, is red, and is invoked by nothing reads as coverage it is not providing. `controller/scripts/docker_run_volume_path_gate.py` failed continuously from **2026-07-14 (v0.129.0)** until R-7b's close-out ran it by hand at v0.145.0 — sixteen releases in which every REPORT said "green" | **READY (S for (a) / M for (b))** | — | **This item has existed at `ROADMAP.md:158` since before the register was rebuilt (2026-07-27) and was never carried across — that omission is itself part of the finding**, because it is an open item *about work not getting done* that then went missing from the page that decides what gets done. Two separable parts, per R-29's own analysis: **(a)** the `docker_run_volume_path_gate` finding is benign and the fix is a 3-line ALLOWLIST addition with its why — **not** a rewrite of the flagged call — and it gets its own reviewed diff, never bundled into a feature commit; **(b)** the systemic half, the real item: decide where gates run (pre-push hook, `build.sh` step, or CI) and make a red gate block the train the way the Go green gate does. **Two further orphans confirmed 2026-07-29** by repo-wide grep across all file types + sibling repos + `~/.claude` settings/skills/hooks + `.git/hooks` (none non-sample) + Makefile/justfile/Taskfile find (only `hub/Makefile`, zero `gate` occurrences) + CI-directory find (**this repo has no CI at all**) — every one of the 19 hits is a docstring, a code comment or prose, and **not one is an invocation**: `scripts/hostinstall_gates.py` — **RED today** (`hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0`, exit 1), the same finding as **R-94 leg (b)** — and `scripts/hub_confirm_gate.py`. Of the four gates in `scripts/`, only `site_gates.py` is mandated anywhere (`CLAUDE.md:153`) and `manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`. **In R-29's own words, carried forward deliberately: do not mint a new ID for a new instance** — the 2026-07-18 rehearsal independently re-raised this item and no second ID was minted then either | CC | | **R-95** | restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only | **READY** | — | Root exposure still open. Mitigation now ARMED — split prune off-box or move to REST `--append-only` | CC | | **R-86** | Restore-tests are interval-scheduled, not backup-aligned | **READY** | R-90 (ep0 headroom) informs cadence | Trigger a tier ~24 h after **its own** newest archive | CC | @@ -44,7 +48,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **F-REBOOT** | ~~A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal; 9m47s total appliance outage with every alarm silent~~ | **SHIPPED + PROVEN-LIVE** (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; `onboot` is the deliberate-stop discriminator (already the stale-lock path's, and what `pve-guests` consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: **120 s unattended** vs the incident's 587 s with a human; Scenario B proven (an `onboot:0` guest left stopped) | — | | **F-LEAK** | ~~A failed restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); the 10-slot VMID band shrinks silently~~ | **SHIPPED + PROVEN-LIVE** (agent v0.110.0 + host-install v1.21.0, 2026-07-28) | — | **Three attempts, two refuted live.** (1) Pool adoption: `PUT /pools/{pool}` also needs `VM.Allocate` on the VM — membership cannot bootstrap its own authority. (2) Per-path `/vms/990000..990009` ACLs: work, but PVE's destroy calls `remove_vm_access` (`LXC.pm:906`) which deletes every ACL at `/vms/` — **consumed by the op it authorises**, one use per slot. (3) SHIPPED: 4th root-fenced exception, band enforced in sudoers **literally** (`pct destroy 99000[0-9] --purge`) + in code + at the caller; API destroy still tried first. Live: band PERMITTED, `9201`/`9100`/`9999`/`990010`/`1` REFUSED, and `pct start 990000` REFUSED too | — | | **F-OBS** | ~~`deadapp-check` leaves NO positive observable on a default (info-level) box — "no alarms" was indistinguishable from "never ran"~~ | **SHIPPED + PROVEN-LIVE** (controller v0.180.0 + agent v0.109.0, 2026-07-28) | — | INFO summary every 20th scan carrying scans/evaluated/down. **Agent v0.109.0 fixes the same shape in the guest-power watchdog shipped hours earlier in v0.107.0** — it logged only at startup and when it acted, so its health could be read only from absence | — | -| **E-2** | ~~Drive-role machinery around the moved vzdump target~~ | **SHIPPED** (hub 0.81.0, agent 0.113.0, controller 0.185.1, host-install 1.22.0 — 2026-07-29) | — | Parts 1–5 complete. Role model + offer-only assignment + Hungarian degraded banner + absent-target signal + installer Case A/B. **NOT yet live-proven:** the DEGRADED banner and the offer acceptance (both demo boxes are healthy, so neither state occurs naturally) and `backup_target_absent` end-to-end. Installer is **installer-logic-tested, not install-tested** — no reinstall was performed | CC | +| **E-2** | ~~Drive-role machinery around the moved vzdump target~~ | **SHIPPED** (hub 0.81.0, agent 0.113.0, controller 0.185.1, host-install 1.22.0 — 2026-07-29) | — | Parts 1–5 complete. Role model + offer-only assignment + Hungarian degraded banner + absent-target signal + installer Case A/B. **NOT yet live-proven:** the DEGRADED banner and the offer acceptance (both demo boxes are healthy, so neither state occurs naturally) and `backup_target_absent` end-to-end. Installer is ~~installer-logic-tested, not install-tested~~ — **INSTALL-TESTED 2026-07-29** on a fresh nested box via the real ISO/PAIRING route, rc=0 (`audits/E2D-fresh-vm-2026-07-29.md` §3). **Of the "NOT yet live-proven" list: Case B + the degraded state are now PROVEN at the installer and API level; the OFFER ACCEPTANCE is PROVEN at the API level** (decline path, `restart_required:true`, E-2a wrapper, healthy-renders-nothing). **Still NOT proven, and now known to be BROKEN rather than merely untested:** the banner/offer never reach a customer (**R-112**) and `backup_target_absent` cannot fire on device loss (**R-113**), with the absent-state message itself wrong (**R-114**) | CC | | **E-2a** | ~~The target move needs a root-fenced wrapper — the agent cannot do it~~ | **SHIPPED + PROVEN-LIVE** (agent v0.113.0 + host-install v1.22.0, 2026-07-29) | — | `felhom-backup-target-apply` behind a literal `FELHOM_BACKUPTARGET` sudoers alias; the agent's PVE role was NOT widened. Enforces F-1 (`mountpoint -q`) and F-2 (`is_mountpoint 1` hardcoded), refuses a root-device target, has NO storage-removal path (grep-assertable), is idempotent and refuses to repoint. All five laws proven live as root on demo-hp with 0 stray storages | — | | **E-2b** | ~~`NotifyStorageDisconnected`/`Reconnected` defined and called NOWHERE — a drive going absent emitted no event on any channel~~ | **SHIPPED + PROVEN-LIVE** (controller v0.184.1 + agent v0.112.0 + hub v0.81.0, 2026-07-29) | — | Seam wired in `ReconcileDriveGates`; a target drive raises the specific `backup_target_absent` instead. **A keying bug was caught before deploy:** `a.Path` is the registered GUEST path, not the agent's host `MountPath`, so the target branch was unreachable — every absent drive, target included, fell through to the generic event (v0.184.1). Tests observe the WIRE (httptest hub), not a mock | — | | **E-2c** | ~~E-1 put the whole-guest backups on a drive `POST /disks/eject` would eject~~ | **SHIPPED + PROVEN-LIVE** (agent v0.112.0, 2026-07-29) | — | Eject + decommission refuse 409 on the backup-target mount, naming the storage and the remedy. **Live on BOTH boxes:** demo-hp `/mnt/nvme-1tb` and demo-felhom `/mnt/hdd_1` both refused, drives unmoved. NOT a role reclassification — `RoleForStorage` untouched, because on both boxes that drive is ALSO the enrolled user-data drive; `TestEjectStillAllowedOnANonTargetDrive` pins the non-over-correction and `/var/lib/vz` is still refused by the PRE-EXISTING role gate, not this one | — | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index e1f5bba..ec45175 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -20,7 +20,10 @@ | ID | Item | Size | Status | Notes / map rows flipped | |----|------|------|--------|--------------------------| -| R-111 | **The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`** | M | idea — found 2026-07-29 by the E-2d Phase 0 gate | **The fleet's live versions are not the fleet's INSTALLABLE versions, and only the first were ever checked.** `felhom-host-install.sh` does not use `main`: `resolve_artifacts()` (`:423-436`) reads the hub-vouched manifest (`GET /api/v1/artifacts/`, `hub/internal/api/handler.go:2120`) and fetches versioned **Gitea generic packages** — agent from `:1945`, golden from `:2573`. Gitea holds **`felhom-agent` newest `0.96.0`** and **`felhom-golden` newest `0.161.0`**; the hub's saved manifest selects exactly those. So a fresh box lands on **agent 0.96.0 + controller 0.161.0** (golden bake; the global floor is `v0.156.0` < 0.161.0, so it does not self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and is not in the channel at all** — demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. **17 unpublished releases (`felhom-agent/CHANGELOG.md` v0.97.0–v0.113.0)**, including the ENTIRE R-82 per-target backup-tier arc (v0.97.0–v0.104.0), **F-CRIT-2** (v0.106.0 — a failed backup looking fresh, 7 days silent), **F-REBOOT** (v0.107.0 — a guest rebooted mid-backup never returns), F-LEAK (v0.108.0/0.110.0), F-OBS (v0.109.0), E-2c (v0.111.0), E-2b (v0.112.0), E-2a (v0.113.0). **P1 because it gates the first remote tester:** their box would install an agent predating the tiered-backup model and both silent-failure fixes. **Mirror of R-110, not a duplicate:** R-110 is *the installer publishes instantly with no staging*; this is *the agent and golden have a deliberate publish+vouch gate and it was never walked* — opposite failure modes of one subject, different fixes. Contrast worth keeping: the **controller** is fine (registry has 0.185.1; it self-updates from the floor), so the gap is specific to the two Gitea-generic artifacts. **Decide as part of the fix:** whether publishing becomes part of the release train rather than a separate remembered step — this is R-29's shape (a gate that exists and is never walked) one layer up. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §2 | +| R-113 | **The drive-absent gate cannot fire on device loss — E-2b's alarm is wired to an unreachable condition** | M | idea — **PROVEN LIVE 2026-07-29** | `planDriveGates` (`felhom-controller/internal/web/intermediary.go:216-262`) treats a path as present by OR-ing in `d.BoundUnderParent`, which the agent derives from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc//mountinfo`"* (`internal/localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is not device-bound and its mountinfo entry outlives the device**, so the gate reads it as present and `notifyDriveAbsent` is never called. Live on a fresh box: target drive hot-detached, agent said `enrolled drive absent by UUID` every 20 s for 4½ min, controller logged **0** `[gate]` lines, hub received **zero** events — neither `backup_target_absent` nor the generic `storage_disconnected`. Not a virtualisation artefact (device-bound-mount vs manual-bind is the same on metal); caveat: SCSI hot-detach, physical unplug not staged. **Sixth instance of seam-built-but-never-wired — E-2b wired the seam to a condition that cannot occur.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 | +| R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints** — `backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | +| R-114 | **On target-drive loss the customer is told the wrong story and offered the drive that vanished** | S | idea — **PROVEN LIVE 2026-07-29** | With the assigned target absent the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a missing drive, not the system disk — **and** an `offer_path` pointing at the drive that just disappeared. `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing *never configured* from *configured and now missing*. Shares R-113's root cause (two disagreeing presence signals), different code path and fix. **Invisible today only because of R-112.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 | +| R-111 | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`**~~ | M | **SHIPPED 2026-07-29** — agent 0.113.0 published (sha `5f3247f7…`, round-trip verified) + golden **0.185.1** baked and published (sha `dba00f3e…`, embeds controller 0.185.1); hub Day-0 manifest moved both together with `min_agent` 0.93.0→0.113.0 (zero fleet impact — all three enrolled hosts already at 0.113.0). **Proven end-to-end the same day:** the E-2d fresh install fetched exactly these and restored the golden baked 20 minutes earlier. Original finding: | **The fleet's live versions are not the fleet's INSTALLABLE versions, and only the first were ever checked.** `felhom-host-install.sh` does not use `main`: `resolve_artifacts()` (`:423-436`) reads the hub-vouched manifest (`GET /api/v1/artifacts/`, `hub/internal/api/handler.go:2120`) and fetches versioned **Gitea generic packages** — agent from `:1945`, golden from `:2573`. Gitea holds **`felhom-agent` newest `0.96.0`** and **`felhom-golden` newest `0.161.0`**; the hub's saved manifest selects exactly those. So a fresh box lands on **agent 0.96.0 + controller 0.161.0** (golden bake; the global floor is `v0.156.0` < 0.161.0, so it does not self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and is not in the channel at all** — demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. **17 unpublished releases (`felhom-agent/CHANGELOG.md` v0.97.0–v0.113.0)**, including the ENTIRE R-82 per-target backup-tier arc (v0.97.0–v0.104.0), **F-CRIT-2** (v0.106.0 — a failed backup looking fresh, 7 days silent), **F-REBOOT** (v0.107.0 — a guest rebooted mid-backup never returns), F-LEAK (v0.108.0/0.110.0), F-OBS (v0.109.0), E-2c (v0.111.0), E-2b (v0.112.0), E-2a (v0.113.0). **P1 because it gates the first remote tester:** their box would install an agent predating the tiered-backup model and both silent-failure fixes. **Mirror of R-110, not a duplicate:** R-110 is *the installer publishes instantly with no staging*; this is *the agent and golden have a deliberate publish+vouch gate and it was never walked* — opposite failure modes of one subject, different fixes. Contrast worth keeping: the **controller** is fine (registry has 0.185.1; it self-updates from the floor), so the gap is specific to the two Gitea-generic artifacts. **Decide as part of the fix:** whether publishing becomes part of the release train rather than a separate remembered step — this is R-29's shape (a gate that exists and is never walked) one layer up. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §2 | | R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path | | R-2 | ~~Resolve ~215 lines of foreign WIP in felhom.eu clone (`hub/internal/notify/`, `store.go`, `hub/internal/claim/`)~~ | S | **killed** (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — `notify/`+`claim/engine.go` in `6b40eb8` (v0.50.0), `store.go` in `a1d0450` (v0.54.0), plus follow-up `e205a2d`; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) | | R-3 | Friend-alpha onboarding runbook (generalized from `pilot/RUNBOOK-peti-return-2026-07-13`): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step | M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of `PETI-tester-agreement.md`. **Next from-scratch rehearsal to include customer DELETE + re-create** — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). **S6b (manual stale-host delete before re-enroll) is OBSOLETE** — re-enrollment upserts the existing host row cleanly (`store.UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. **NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE):** one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). **Remaining consistency gap:** the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b |