From f3975cf5bc4722707a23cf68466518440b2227b0 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Wed, 29 Jul 2026 13:13:56 +0200 Subject: [PATCH] =?UTF-8?q?E-2d=20executed=20on=20a=20fresh=20box:=20C1/C2?= =?UTF-8?q?=20proven,=20C3/C4=20partial,=20C5=20FAILS=20=E2=80=94=20R-112/?= =?UTF-8?q?113/114?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Full ISO/PAIRING route on a nested PVE VM on demo-hp, after R-111 was fixed earlier in the session. Bind -> running controller in 3m35s. The install fetched the artifacts published an hour before and restored the golden baked 20 minutes before, so the publish train is proven end to end on a real install. C1 PROVEN: "felhom-host-install v1.22.0", "Day-0 provision SUCCESS", guest 9201 running, bootstrap unit wrote its done-flag and self-disabled. This retires E-2's "installer-logic-tested, not install-tested". C2 PROVEN: both DEGRADED warning lines verbatim, backup.local_backup_target=local, no felhom-backup storage created, and the install did not abort. C3/C4 PARTIAL and C5 FAILED — three findings, none fixed: R-112 (P1): E-2's degraded banner and offer have NO UI CONSUMER. The endpoint returns byte-exact copy; grep 'backup-target' across every html/js/css is 0 hits and no page handler injects the state. Templates fetch 18 distinct /api/storage/* endpoints; these two are the only ones with zero references. v0.185.1 fixed the router mount and stopped one layer short of the render. Fifth instance of seam-built-but-never-wired. R-113 (P1): the drive-absent gate CANNOT FIRE on device loss. planDriveGates reads presence from BoundUnderParent = "is this path in the guest's mountinfo". The raw mount is a device-bound systemd unit and dies with the device; the agent's own bind is not device-bound and outlives it, so the gate sees "present" forever. Live: agent reported the drive absent every 20s for 4.5 minutes, the controller logged 0 [gate] lines, the hub received zero events -- neither backup_target_absent nor the generic storage_disconnected. Sixth instance of the class: E-2b wired the seam to a condition that cannot occur. R-114: on target-drive loss the message claims the backup is on the system disk (false) and offers the drive that just vanished. Invisible only because of R-112, so it must be fixed BEFORE R-112 is wired. Also filed as a second instance under R-110 rather than a new ID: host-install fetches nine files from raw/branch/main and the hub vouches a sha for one; E-2a's wrapper is installed 0755 to /usr/local/sbin, root-fenced in sudoers, validated only by bash -n. C4 is fully proven at API level: decline path (registration confers no role), restart_required:true, agent did NOT self-restart (in-flight check performed and recorded first), E-2a wrapper created the storage at the drive's own mountpoint, and healthy renders nothing. Teardown: VM destroyed, scratch storage removed, pvesm status after == before (local-lvm 38.77%), guest 9201 and drill-r50 untouched. Hub records for e2d-fresh remain -- delete correctly refused at four gates, finally "host is ONLINE"; deletable once it ages to DOWN. Command recorded in OPEN-ITEMS.md. capability-map NOT touched: the customer-facing legs are broken rather than proven, and the map has no E-2 rows at all. --- REPORT-e2d.md | 113 +++-- .../audits/E2D-fresh-vm-2026-07-29.md | 388 ++++++++++-------- documentation/backlog/OPEN-ITEMS.md | 12 +- documentation/backlog/ROADMAP.md | 5 +- 4 files changed, 307 insertions(+), 211 deletions(-) diff --git a/REPORT-e2d.md b/REPORT-e2d.md index 7ce35a6..16ce52e 100644 --- a/REPORT-e2d.md +++ b/REPORT-e2d.md @@ -1,58 +1,87 @@ -# REPORT — E-2d fresh-VM run: STOPPED at Phase 0 (2026-07-29) +# REPORT — R-111 fixed, then E-2 proven on a fresh box (2026-07-29) -`RUNBOOK-e2d-fresh-vm-2026-07-29.md`, executed by CC on DooPlex. Full evidence: -`documentation/audits/E2D-fresh-vm-2026-07-29.md`. Written as `REPORT-e2d.md` per the runbook §8.5; -root `REPORT.md` untouched. +Two phases in one session. Full evidence: `documentation/audits/E2D-fresh-vm-2026-07-29.md`. +Root `REPORT.md` untouched. -**Outcome: no VM created, no install run, no box touched, no teardown needed.** The run stopped at the -Phase 0 gate, per §3 (*"If any gate fails, STOP and report — do not adapt around it"*). +## Phase 1 — R-111: the Day-0 channel now serves the current software -## Why +A Phase 0 gate earlier the same day stopped the E-2d run before any VM existed: a fresh box would +have installed **agent 0.96.0 + controller 0.161.0**, ~17 and ~24 releases behind `main`. -`felhom-host-install.sh` does not install what is on `main`. It resolves the hub-vouched manifest -(`:423-436`) and fetches **Gitea generic packages** (agent `:1945`, golden `:2573`). Gitea's newest are -**agent `0.96.0`** and **golden `0.161.0`**; the hub's saved manifest selects exactly those; the global -controller floor is `v0.156.0`, below the golden's 0.161.0, so no self-update follows. - -| Component | Fresh install gets | `main` / demo boxes | +| | Before | Now | |---|---|---| -| host-install | **1.22.0** | 1.22.0 | -| agent | **0.96.0** | 0.113.0 — **never published**, direct-deployed | -| controller | **0.161.0** | 0.185.1 | +| agent (Gitea generic) | 0.96.0 | **0.113.0**, sha `5f3247f7…`, round-trip verified | +| golden (Gitea generic) | 0.161.0 | **0.185.1**, sha `dba00f3e…`, embeds controller 0.185.1 | +| hub `min_agent` | 0.93.0 | **0.113.0** (what controller v0.185.0 declares) | -## Claims +Bake clean on every marker: `Result=success`, overlay2, **all three mounts in the archive**, 0 +FATAL/exclusions, HTTP 201, token-leak grep 0. GL-1 teardown: guest 9100 purged, secrets shredded, +drill disk restored to `virgin`. Agent + golden moved in **one** manifest POST so it never vouched a +new agent against an old golden. `min_agent` verified zero-impact first (all three enrolled hosts +already at 0.113.0). Global floor deliberately **not** raised — the golden now bakes 0.185.1. -| Claim | Verdict | Pinned to | -|---|---|---| -| C1 — real rc=0 install of 1.22.0 | **ACHIEVABLE, not run** | installer-side | -| C2 — Case B fires naturally | **ACHIEVABLE, not run** | `felhom-host-install.sh:627`, `:653-655` | -| C3 — degraded banner renders | **BLOCKED** | controller v0.185.1 `cdaeb36`; copy v0.185.0 `3f7cf2a` | -| C4 — offer moves the target | **BLOCKED** | controller v0.185.1 + agent v0.113.0 `58b598b` | -| C5 — `backup_target_absent` e2e | **BLOCKED** | controller v0.184.0 `c1a63de` + agent v0.112.0 | +Commit `3dff357`. -C3 unblocks by raising the hub floor to ≥0.185.0 (controller 0.185.1 **is** in the registry); measured -fleet impact nil — both demo boxes already run 0.185.1, Peti is DOWN 14 d and already below the current -floor. C4/C5 need agent 0.112.0/0.113.0 **published**, which runbook §0 forbids this run from doing. +## Phase 2 — the E-2d run, full ISO/PAIRING route -## Filed +Nested PVE VM on demo-hp, one disk, outside the `felhom` pool. Bind → running controller in +**3 m 35 s**. The install fetched exactly the artifacts published an hour earlier and restored +`vzdump-lxc-9100-2026_07_29-12_37_56` — the golden baked 20 minutes before. The publish train is +proven end to end on a real install. -**R-111 (P1)** — the Day-0 artifact channel is 17 agent releases stale. v0.97.0–v0.113.0 unpublished, -stranding the entire R-82 tiered-backup arc plus **F-CRIT-2** and **F-REBOOT**. A new customer's box -installs without them. Mirror of R-110, not a duplicate: R-110 = publishes instantly with no staging; -R-111 = the publish gate exists and was never walked. +| Claim | Verdict | +|---|---| +| **C1** host-install 1.22.0 completes a real install, rc=0 | ✅ **PROVEN** | +| **C2** Case B fires naturally | ✅ **PROVEN** — both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort | +| **C3** degraded banner renders **to a customer** | ⚠️ **PARTIAL** — API byte-exact; **no UI consumer exists** → **R-112** | +| **C4** offer appears and moves the target | ⚠️ **PARTIAL** — decline path, `restart_required:true`, no self-restart, E-2a wrapper, healthy-renders-nothing all PROVEN at API level; offer equally invisible → **R-112** | +| **C5** `backup_target_absent` end to end | ❌ **FAILED** — zero events on any channel → **R-113** | + +## The three findings + +**R-112 (P1)** — E-2's banner and offer have **no UI consumer**. The endpoint returns byte-exact copy; +`grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**, and no page handler injects +the state. Decisive contrast: templates fetch **18** distinct `/api/storage/*` endpoints; +`backup-target` and `backup-target/assign` are the only two with zero references. v0.185.1 fixed the +router mount and stopped one layer short of the render. Fifth instance of seam-built-but-never-wired. + +**R-113 (P1)** — the drive-absent gate **cannot fire on device loss**. `planDriveGates` reads presence +from `BoundUnderParent` = "is this path in the guest's mountinfo". The raw mount is a device-bound +systemd unit and dies; **the agent's own bind is not device-bound and outlives the device**, so the +gate sees "present" forever. Live: agent said `enrolled drive absent by UUID` every 20 s for 4½ +minutes, controller logged **0** `[gate]` lines, hub got **zero** events — neither the specific nor the +generic one. Sixth instance of the class, one layer deeper: E-2b wired the seam to a condition that +cannot occur. + +**R-114** — on target-drive loss the message says the backup is *"on the same disk as the system"* +(false) and offers **the drive that just vanished**. Invisible today only because of R-112 — so +**R-114 must be fixed before R-112 is wired.** + +Also filed as a **second instance under R-110** (not a new ID): host-install fetches **nine** files +from `raw/branch/main` and the hub vouches a sha for **one**; E-2a's wrapper is installed 0755 to +`/usr/local/sbin`, root-fenced in sudoers, validated only by `bash -n`. ## Record -- `OPEN-ITEMS.md` — **R-111 opened** (READY (M), P1). **E-2d re-stated**, not closed: the attempt, the - blocker, the C1/C2-vs-C3/C4/C5 split, and every Phase 0 answer so a resumed run does not re-derive them. -- `ROADMAP.md` — R-111 filed under **P1 — closed-alpha blockers**. -- `architecture/00-capability-map.md` — **not touched.** Nothing was proven live; no row qualifies. +- `OPEN-ITEMS.md` — **R-112/R-113/R-114 opened**; E-2d re-stated with results and left open for the + residue; E-2's "NOT yet live-proven" list resolved into proven / known-broken; R-94 fully unblocked; + R-110 extended; a WATCHING row for the drill cleanup. +- `ROADMAP.md` — R-112/R-113/R-114 under P1; R-111 marked SHIPPED. +- **`architecture/00-capability-map.md` not touched** — for two reasons: the customer-facing legs are + broken rather than proven, and the map has **no E-2 / backup-target rows at all** (worth noting + against the ROADMAP's coupling rule). -## Not done +## Teardown -C1–C5 all unproven. No VM, no ISO boot, no appliance registered/bound/discarded, no storage created or -modified on demo-hp, no drive attached or detached, no hub setting written (the floor was read only), -no code written. +VM destroyed, scratch storage removed, **`pvesm status` after == before** (`local-lvm` 38.77 %, +byte-identical), guest 9201 and drill-r50 untouched. **Hub records for `e2d-fresh` remain** — the +delete was attempted and correctly refused at four gates, finally `host … is ONLINE`; it becomes +deletable once the destroyed host ages to DOWN (>1 h). Command recorded in `OPEN-ITEMS.md`. The stale +2026-07-25 appliance (`206c8838…`) was left alone — not mine. -The runbook's **§5.1a operator STOP is retired**: `HUB_PW` is in `~/.config/credentials` and hub auth -was verified working, so CC can perform the bind itself on a resumed run. +## One human step, and a premise correction + +The runbook's §5.1a operator STOP (the bind) is **retired** — CC did it. But E-2d's premise that a +fresh install yields a CC-drivable claimable customer is **wrong**: the claim code is bcrypt-hashed and +email-only, and the gate covers everything except `/claim`, `/api/health`, `/static/`. One operator +relay of the emailed code was required — which also proved the claim flow end to end. diff --git a/documentation/audits/E2D-fresh-vm-2026-07-29.md b/documentation/audits/E2D-fresh-vm-2026-07-29.md index 878640b..3dc2d08 100644 --- a/documentation/audits/E2D-fresh-vm-2026-07-29.md +++ b/documentation/audits/E2D-fresh-vm-2026-07-29.md @@ -1,198 +1,258 @@ -# E2D-fresh-vm-2026-07-29 — Phase 0 STOP: the Day-0 artifact channel cannot deliver the code under test +# E2D-fresh-vm-2026-07-29 — E-2 proven on a fresh box: C1/C2 pass, C3/C4 partial, C5 FAILS **Run:** `RUNBOOK-e2d-fresh-vm-2026-07-29.md`, executed by CC on DooPlex, 2026-07-29. -**Outcome:** **STOPPED at Phase 0, before any VM was created.** No VM provisioned, no install run, no -box touched, no teardown required. Per §3: *"If any gate fails, STOP and report — do not adapt around it."* +**Preceded by:** a Phase 0 STOP earlier the same day (R-111 — the Day-0 channel was 17 agent releases +stale). R-111 was fixed first; this run then proceeded on the real customer path. -**The finding in one sentence:** a box installed today through the real customer chain receives -**agent 0.96.0** and **controller 0.161.0**, because those are the newest artifacts ever published to -the Day-0 channel — so three of the five claims (C3, C4, C5) test endpoints and events that do not -exist in the software a fresh box actually runs. +**Headline:** the installer and its Case B are proven on a real install. **The two customer-facing +halves of E-2 are not reachable by a customer at all**, and the drive-absent alarm **cannot fire on +device loss**. Both were invisible to a green unit suite and to an API-level check; only the live run +found them. ---- - -## 1. Baselines confirmed - -| Artifact | Runbook §1 | Confirmed | Source | -|---|---|---|---| -| hub | 0.81.0 | **0.81.0** | `manifests/hub.yaml:128`; `hub/CHANGELOG.md:1`; live deploy image | -| agent | 0.113.0 | **0.113.0** on `main` @ `58b598b` | `felhom-agent` HEAD; live `felhom-agent -version` on demo-hp | -| controller | 0.185.1 | **0.185.1** on `main` @ `cdaeb36` | `felhom-controller` HEAD; live image on guest 9201 | -| host-install | 1.22.0 | **1.22.0** | `scripts/felhom-host-install.sh:187` | -| felhom.eu | `91a1dad` | **`91a1dad`**, clean, == `origin/main` | `git rev-parse` | - -These are the versions **on `main` and on the demo boxes**. They are not the versions a fresh install -receives — that distinction is the whole finding. - ---- - -## 2. THE BLOCKER — the Day-0 artifact channel is 17 agent releases stale - -### 2.1 The chain, read at source - -`felhom-host-install.sh` does not use `main`. It resolves a hub-vouched manifest and fetches versioned -packages from Gitea: - -- `resolve_artifacts()` (`:423-436`) → `GET $HUB_URL/api/v1/artifacts/$CUSTOMER_ID` → `agent.version`, - `golden.version` (served by `hub/internal/api/handler.go:2120` `handleArtifactManifest`). -- agent binary ← `$GITEA_BASE/api/packages/admin/generic/felhom-agent/$ART_AGENT_VER/felhom-agent` (`:1945`) -- golden ← `$GITEA_BASE/api/packages/admin/generic/felhom-golden/$ART_GOLDEN_VER/golden.tar.zst` (`:2573`) - -### 2.2 What that channel actually holds (Gitea API, authenticated, 2026-07-29) - -``` -felhom-agent : newest = 0.96.0 - all = 0.79.0 0.80.0 0.81.0 0.84.0 0.85.0 0.86.0 0.87.0 0.88.0 0.89.0 - 0.90.0 0.91.0 0.91.1 0.91.2 0.92.0 0.92.1 0.93.0 0.96.0 -felhom-golden: newest = 0.161.0 - all = 0.136.0 0.143.0 0.146.0 0.153.0 0.161.0 -``` - -Hub's saved Day-0 manifest (`/configuration`, selected options): **agent `0.96.0`** -(sha `af938601…`), **golden `0.161.0`** (sha `77624408…`), min-agent `0.93.0`. -Hub global controller floor: **v0.156.0** (DB override; env fallback v0.120.0). - -### 2.3 Therefore a fresh box lands on - -| Component | Fresh install gets | `main` / demo boxes | Gap | -|---|---|---|---| -| host-install | **1.22.0** (website git-sync from `main`) | 1.22.0 | none | -| agent | **0.96.0** | 0.113.0 | **17 releases** | -| controller | **0.161.0** (golden bake; floor 0.156.0 < 0.161.0 ⇒ no self-update) | 0.185.1 | **24 releases** | - -**Agent 0.113.0 is not in the channel at all** — it reached both demo boxes by direct deploy, never -through publish. Confirmed live: demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. - -### 2.4 What that strands — 17 unpublished agent releases (`felhom-agent/CHANGELOG.md`) - -| Version | What it carries | +| Claim | Verdict | |---|---| -| v0.97.0–v0.104.0 | **the entire R-82 per-target backup-tier arc** (local daily + PBS weekly), incl. v0.98.0's 30-minute false-failure bound, v0.101.0/0.102.0 scratch-guest + defer fixes, v0.103.0 R-84, v0.104.0 R-85 restore-test | -| v0.105.0 | R-88 Part 2 — the agent can say `unknown` | -| v0.106.0 | **F-CRIT-2** — a failed backup must not look like a fresh one (the phantom snapshot that reset the freshness clock, 7 days silent) | -| v0.107.0 | **F-REBOOT** — a guest rebooted during its backup never comes back | -| v0.108.0, v0.110.0 | F-LEAK — scratch-guest VMID band leak | -| v0.109.0 | F-OBS — the guest-power watchdog's positive observable | -| v0.111.0 | E-2c — the backup-target drive can no longer be ejected out from under the backup | -| v0.112.0 | **E-2b** — `GET /disks` flags the backup-target drive | -| v0.113.0 | **E-2a** — the guarded wrapper + `POST /backup/target` | - -**A new customer box installed today therefore runs an agent that predates the whole tiered-backup -model and lacks F-CRIT-2 and F-REBOOT** — two customer-impacting silent-failure fixes. That is a -larger finding than E-2d itself and is filed as **R-111**. +| **C1** host-install 1.22.0 completes a real install, rc=0 | ✅ **PROVEN** | +| **C2** Case B fires naturally on a single-drive box | ✅ **PROVEN** | +| **C3** the degraded banner renders **to a customer** | ⚠️ **PARTIAL — API exact, NO UI CONSUMER** (R-112) | +| **C4** the offer appears and moves the target when accepted | ⚠️ **PARTIAL — full API flow proven; offer equally invisible** (R-112) | +| **C5** `backup_target_absent` fires end to end | ❌ **FAILED** — no event on any channel (R-113) | --- -## 3. Claim-by-claim impact, each pinned to its introducing commit +## 1. Baselines as actually confirmed -| Claim | Needs | Introduced in | Fresh box has | Verdict | -|---|---|---|---|---| -| **C1** rc=0 real install, banner names 1.22.0 | host-install 1.22.0 | website ← `main` | **1.22.0** | **ACHIEVABLE** | -| **C2** Case B DEGRADED lines + `resolved local` | host-install `configure_backup_target()` `:627`, Case B `:653-655` | host-install 1.22.0 | **1.22.0** | **ACHIEVABLE** — installer-side only | -| **C3** degraded banner via `GET /api/storage/backup-target` | controller **v0.185.1** (`cdaeb36`); copy „lemezhiba ellen nem" **v0.185.0** (`3f7cf2a`) | — | controller 0.161.0 | **BLOCKED** — endpoint and copy do not exist | -| **C4** offer → `POST …/assign` → `restart_required` | controller **v0.185.1** (`cdaeb36`) + agent **v0.113.0** `POST /backup/target` (`58b598b`) | — | 0.161.0 / 0.96.0 | **BLOCKED** | -| **C5** `backup_target_absent` / `_restored` | controller **v0.184.0** (`c1a63de`) + agent **v0.112.0** | — | 0.161.0 / 0.96.0 | **BLOCKED** | - -**C3 has a cheap unblock; C4 and C5 do not.** C3 needs only the hub's global controller floor raised -0.156.0 → ≥0.185.0, because controller **0.185.1 IS published** to the registry (60 tags, newest -0.185.1) — a fresh box would self-update on its first report. C4/C5 need agent 0.112.0/0.113.0 -*published as Gitea generic packages*, which §0 of the runbook explicitly forbids this run from doing -(*"Does not touch: … any published artifact"*). - -Fleet impact of raising the floor, measured rather than assumed (hub dashboard, 2026-07-29): - -| Customer | Controller | Effect of floor → 0.185.1 | +| Artifact | Confirmed | Source | |---|---|---| -| Demo Ügyfél | 0.185.1 | none — already at it | -| Demo HP | 0.185.1 | none — already at it | -| Peti Proxmox | 0.115.0, **DOWN 14d** | already below the current 0.156.0 floor, so it updates on return either way; only the target version changes | +| hub | 0.81.0 | `manifests/hub.yaml:128`; live deploy image | +| agent | 0.113.0 | `main` @ `58b598b`; **published this run**, sha `5f3247f7…` | +| golden | **0.185.1** | baked this run, sha `dba00f3e…`, embeds controller 0.185.1 | +| controller | 0.185.1 | `main` @ `cdaeb36`; live in the guest | +| host-install | 1.22.0 | `scripts/felhom-host-install.sh:187`; fetched from the website at run time | +| felhom.eu | `3dff357` | `main` HEAD at run time | ---- +**Route: ISO/PAIRING (the real customer chain).** ISO +`felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso`. Every C1–C5 result is on the real route; no +manual-installer fallback was used. -## 4. Phase 0 gate answers (all four completed before the stop) +**Operator STOP: not required and now retired.** `HUB_PW` is in `~/.config/credentials`; CC created +the customer and performed the bind itself. The one human step that *was* needed is new — see §6. -### 4.1 Storage placement — gate PASSES, fence confirmed +## 2. Timeline (VM 9300 `e2d-fresh` on demo-hp, nested PVE) -`pvesm status` on demo-hp, 2026-07-29 (BEFORE; there is no AFTER — no VM was created): +| UTC | Event | +|---|---| +| 10:29 | VM created — q35/OVMF SB-off, 4c/8G, **one** 160 G disk, `hotplug disk`, outside the `felhom` pool | +| 10:34:52 | PVE auto-install done, first boot, DHCP `192.168.0.125` (no R-59 gate trip) | +| 10:34:53 | registered as unclaimed appliance, pairing code `SB4-7ZK`, console banner rendered | +| 10:36:52 | **bound** to customer `e2d-fresh` by CC; credentials delivered 2 s later | +| 10:37:31 | host enrolled `e2d-fresh-ac9f09`; break-glass root credential vaulted | +| 10:37:33 | **artifact manifest served: agent=0.113.0 golden=0.185.1** | +| 10:40:27 | `controller_started (0.185.1)` — **bind → running controller in 3 m 35 s** | +| 10:56 | second 100 G disk hot-attached; wizard init→mount→register | +| 10:57:08 | offer accepted → `restart_required:true`; agent restarted at 10:57:36 | +| 10:58:37 | target drive **hot-detached** (volume survives as `unused0`) | +| 10:58:37–11:03 | **no event on any channel** for 4½ minutes (budget was 60 s) | +| 11:03:31 | reattached; state returns healthy; **still no event** | + +## 3. C1 — PROVEN + +From the `felhom-bootstrap.service` journal on the box: ``` -Name Type Status Total (KiB) Used (KiB) Available (KiB) % -felhom-backup dir active 983379700 2293856 931059232 0.23% -felhom-pbs pbs active 0 0 0 0.00% -local dir active 40516856 14721460 23705004 36.33% -local-lvm lvmthin active 56545280 21922605 34622674 38.77% +felhom-bootstrap: fetching host-install: https://felhom.eu/scripts/felhom-host-install.sh +[INFO] felhom-host-install v1.22.0 — mode=appliance customer=e2d-fresh vmid=9201 +... +[OK] controller: Up 21 seconds (healthy) +[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.185.1 +[OK] Day-0 provision SUCCESS — vmid=9201 host_id=e2d-fresh-ac9f09 customer=e2d-fresh + golden=local:backup/vzdump-lxc-9100-2026_07_29-12_37_56.tar.zst +felhom-bootstrap: host-install SUCCESS — writing done-flag, disabling unit, scrubbing secrets ``` -`local-lvm` is thin, pool `<53.93g`, **data 38.77 %** — matching the runbook's "~54 GB pool, 38.8 %" -to 0.03 pp, so the picture has **not** materially changed. Allocated LVs on it: `vm-9201-disk-{0,1,2}` -(32+50+20 G, the **live customer guest**), `vm-300-disk-{0,1}` (drill-r50, 32 G) and its `r50pre` -snapshots (32 G) ⇒ ~166 G allocated against 53.93 G real. `vgs` shows only 14.75 G VFree. -**The fence holds: nothing goes on `local-lvm`.** +rc=0 is corroborated structurally: the unit wrote its done-flag, **self-disabled**, and +`Deactivated successfully`. `pct list` showed guest **9201 `e2d-fresh` running**. -`/mnt/nvme-1tb` = `/dev/nvme0n1`, ext4, 938 G, **888 G free**. ≥100 G requirement satisfied. +**The publish train is proven end to end:** the golden restored is +`vzdump-lxc-9100-2026_07_29-12_37_56` — the golden baked ~20 minutes earlier in the same session. +This retires E-2's *"installer-logic-tested, not install-tested"*. -**Placement decision (recorded deliberately, per §3.1):** the only storage with `content images` is -`local-lvm` (forbidden). `felhom-backup` is a `dir` at exactly `/mnt/nvme-1tb` but is `content backup` -only **and is demo-hp's live backup target** — widening its content set would mutate the very storage -E-2c/E-2 role logic keys on, which §0/§9 forbid. So a *new* dir storage would have been required. -Both remaining shapes carry a cost and the choice was **not** forced this run because the run stopped: -a second storage at the same mountpoint risks perturbing the agent's role resolution on a live box; -a storage at a **subdirectory** fails `exactMount` and reports `disconnected` in the host report, -which is not purely cosmetic since it can reach the hub's storage monitor. **This decision is left -open and is an input to any resumed run.** +## 4. C2 — PROVEN -### 4.2 Drive-gate cadence — answered +Both required warning lines, verbatim, ANSI-stripped: -- Symbol: `driveGateLoop` → `s.ReconcileDriveGates()` -- Definition: `controller/internal/web/intermediary.go:328` (gate itself at `:267`) -- Registration: `controller/internal/web/server.go:217` — `go s.driveGateLoop()` -- **Interval: `time.NewTicker(30 * time.Second)`** (`intermediary.go:337`) +``` +[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive. +[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard. +``` -⇒ Stage 7's derived budget would be **two cycles = 60 s** of controller-side reconcile, *plus* the -agent's own `/disks` refresh, since `ReconcileDriveGates` consumes `resp.Disks` (`:281`). Recorded so -a resumed run does not have to re-derive it. +- `agent.json` → `backup.local_backup_target = 'local'` (the "resolved local" observable) +- no `felhom-backup` storage created at install +- **the install did not abort** — a single-drive appliance is a valid product -### 4.3 Hot-detach — feasible, not exercised +## 5. THE FINDINGS -demo-hp: AMD `svm` present, `/sys/module/kvm_amd/parameters/nested = 1`, 8 cores, 29 GB RAM (25 GB -available). The working nested-PVE reference is VM 300 (`drill-r50`): `bios: ovmf`, -`efitype=4m,pre-enrolled-keys=0` (Secure Boot OFF, as the mkimage loader requires), `machine: q35`, -`scsihw: virtio-scsi-single`, `cpu: host`. `virtio-scsi-single` + PVE's default -`hotplug: disk,network,usb` supports SCSI hot-detach, so the **live-transition** variant of Stage 7 -was available. Not exercised. +### 5.1 R-112 — the degraded banner and the offer have NO UI consumer (customer-invisible) -### 4.4 Install route — ISO available, route selected, not used +The endpoint is **correct**. Authenticated `GET /api/storage/backup-target` returned, exactly: -Two generic (pairing) ISOs are present on demo-hp `local`: -`felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso` (**preferred**, the v1.25.0 train, mkimage -loader for nested VMs) and `felhom-pve-9.2-1-v1.22.0-nested-canary-generic.iso`. -The **ISO/PAIRING route was selected**. It reaches the same installer invocation — `run_pairing` -(`felhom-bootstrap.sh:441`) falls through to `run_direct` (`:495-499`) in the same invocation, which -fetches `$INSTALL_URL` (`:322-330`) and runs it (`:343`). Not used — the run stopped first. +```json +{"degraded":true,"known":true,"target":"local", + "message":"A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer — így hibás fájlok + ellen véd, lemezhiba ellen nem. Csatlakoztass egy második meghajtót a teljes védelemhez."} +``` -**Operator STOP: not required.** `HUB_PW` is present in `~/.config/credentials` and hub auth was -verified working (`http://10.43.52.34:8080/` → 200 with `curl -u ":$HUB_PW"`; public ingress also 200), -so CC could have performed the bind itself. The runbook's §5.1a STOP is retired for future runs. +byte-identical to the runbook's required copy. `known` is a field **separate** from `degraded`, so +UNKNOWN genuinely cannot render as degraded (R-88 Part 2's lesson held). ---- +**And nothing in the product ever asks for it.** Negative claims with their search scope: -## 5. Incidental observations (filed, not acted on) +| Search | Result | +|---|---| +| `grep -rn 'backup-target' --include='*.html' --include='*.js' --include='*.css' controller/` | **0 hits** | +| `grep -rn 'kijelölheted\|ugyanazon a lemezen\|OfferPath\|OfferLabel\|Degraded' controller/internal/web/templates/` | **0 hits** | +| consumers of `backupTargetDegradedText` / `backupTargetOfferText` | only `degradedMessageFor` (`:136`) and the JSON handler (`:153`) — both inside `backup_target_offer.go` | +| consumers of `resolveBackupTargetState` / `degradedMessageFor` across all Go | **only** the API handler. **No page handler injects the state.** | -1. **A stale unclaimed appliance is sitting in the hub**: `206c8838-7755-4751-8a16-a842853d718f`, - pairing code `QWA-WJE`, MAC `bc:24:11:ea:55:d0`, "Standard PC (Q35 + ICH9, 2009)" on an - AMD V1756B with 7.7 GB — i.e. a **nested VM on demo-hp**, first and last seen **5 d ago**. It is a - leftover from the 2026-07-25 ISO work and has never been bound or discarded. A resumed run must - distinguish its own appliance from this one, or discard it first. -2. **The agent has no container image** (`/v2/admin/felhom-agent/tags/list` → 0 tags). It ships only - as a Gitea generic package, which is why the publish gap is invisible from the registry. -3. `drill-r50` (VM 300) still holds a `r50pre` snapshot pair consuming ~32 G of allocation on the - over-subscribed `local-lvm`. Untouched per §9; noted because it is part of why the pool is tight. +**The decisive contrast:** the templates fetch **18 distinct `/api/storage/*` endpoints`**. +`backup-target` and `backup-target/assign` are the **only two** referenced by zero templates. ---- +The handler's own doc comment reads *"serves GET /api/storage/backup-target — **the dashboard's source +for the degraded banner and the offer**"* — an invariant comment asserting a consumer that does not +exist (`CLAUDE.md`'s "a comment asserting an invariant needs a test pinning it, or it is a wish", +instance #7). And v0.185.1's own test, `TestBackupTargetRoutesLiveUnderTheStorageAPIMount`, pins that +the **router dispatches** the path — not that anything renders it. v0.185.1 shipped as *"the offer +endpoints were mounted where nothing routed to them"*: it fixed the mount and stopped one layer short. -## 6. What did not happen +**Fifth instance of the seam-built-but-never-wired class.** `CLAUDE.md`'s seam rule names exactly this: +*"a feature is not shipped until its entry point is reachable… handler tests that POST directly prove +nothing about reachability."* -No VM was created. No ISO booted. No install ran. No appliance registered, bound or discarded. No -storage was created or modified on demo-hp. No drive attached or detached. No hub setting changed -(the floor was **read**, not written). No teardown was needed. **C1–C5 are all UNPROVEN**, and the -E-2 / E-2d rows are unchanged except to record this blocker. +### 5.2 R-113 — the drive-absent gate CANNOT fire on device loss (E-2b's alarm is unreachable) + +Detached the assigned target drive at 10:58:37Z under a running agent. Over the next 4½ minutes: + +- **agent**, every 20 s: `storage: enrolled drive absent by UUID — not re-asserting` / + `reconcile: enrolled drive not present (durable-id absent) — skipping` — **the agent knows** +- **controller**: `docker logs | grep -c '\[gate\]'` → **0**. The gate never acted, ABSENT or RETURNED +- **hub**: zero events for the customer across the whole window — no `backup_target_absent`, + **and no generic `storage_disconnected` either** + +**Root cause, established at source and confirmed live.** `planDriveGates` +(`intermediary.go:216-262`) computes presence as +`present[GuestPath] = present[GuestPath] || d.BoundUnderParent`, and the agent derives +`BoundUnderParent` from `GuestSeesMount()` — *"does the guest's `/proc//mountinfo` list this path +as a mount target"* (`localapi/disks.go:210`, `localapi/intermediary.go` `GuestSeesMount`). + +Measured on the box with the device removed: + +``` +-- raw mount /mnt/mentes2 -- NOT mounted <- systemd device-bound unit, died with the device +-- stable bind /mnt/felhom-drives/mentes2 -- /dev/sdb[/felhom-data] ext4 <- the agent's MANUAL bind, SURVIVES +``` + +The raw mount is a device-bound systemd unit and dies correctly; **the agent's own bind under the +shared parent is not device-bound, so its mountinfo entry outlives the device.** The gate reads that +surviving entry as "present" ⇒ `!present[...]` is never true ⇒ no `Stop` action ⇒ `notifyDriveAbsent` +is never called. + +This is not a virtualisation artefact: the asymmetry is between a device-bound mount and a manual +bind, which is identical on physical hardware. **Caveat kept honest:** proven on a SCSI hot-detach; a +physical USB unplug was not staged. + +**Consequence.** E-2b's celebrated fix — *"THE SEAM THAT WAS NEVER WIRED… a drive that is ONLY a +backup target has no apps to stop, so it was silent twice over"* (`intermediary.go:288-296`) — wired +the notify to a branch that **cannot execute on device loss**. The seam is wired; the condition is +unreachable. Sixth instance of the class, one layer deeper than the fifth. + +**Mirror scenario:** not separately staged, and it does not need to be — both the specific and the +generic event are emitted from the same `a.Stop` branch, which never executed. The generic +`storage_disconnected` is equally unreachable by this path. Recorded as *reasoned, not observed*. + +### 5.3 R-114 — on target-drive loss the customer is told the wrong story and offered the missing drive + +While the target drive was absent, the endpoint returned: + +```json +{"degraded":true,"target":"felhom-backup", + "message":"A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer …", + "offer_path":"/mnt/felhom-drives/mentes2","offer_label":"Mentés meghajtó"} +``` + +Two defects in one payload: the message claims the backup is **on the system disk**, which is false — +the target is `felhom-backup` on a drive that has vanished; and the remedy offered is **the drive that +just disappeared**. `resolveBackupTargetState` falls through to the generic degraded branch whenever no +disk satisfies `d.BackupTarget && d.MountPath != ""`, without distinguishing *never configured* from +*configured and now missing*. + +**Interaction worth stating: R-114 is currently invisible only because of R-112.** Fixing R-112 alone — +wiring the banner — would immediately start showing customers this wrong message. **They must be fixed +together, R-114 first.** + +**Also observed:** after reattach the drive returned as `/dev/sdc`, while the stable bind still +recorded `/dev/sdb[/felhom-data]`. The state read healthy (`degraded:false`) with the guest-visible +bind still naming the dead device node. Not chased further; recorded as part of R-113's shape. + +### 5.4 Smaller findings (recorded, not filed as their own IDs) + +1. **A "hard min" that only warns.** `[WARN] local-lvm free ~83 GiB < hard min 120 GiB` — the installer + names a hard minimum and proceeds. Either it is not hard, or the wording is wrong. +2. **`felhom-backup-target-apply` is fetched unvouched.** host-install pulls **nine** files from + `raw/branch/main` (`:2072`–`:2206`); the hub manifest vouches a sha for exactly one + (`wrapper_sha256` → `felhom-pbs-apply`, verified this run: no drift). E-2a's wrapper is installed + **0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n`. + **Filed as a second instance under R-110**, not a new ID — same class (a root-executed artifact + taken from `main` with no pinned integrity). + +## 6. C4 — what IS proven (API level) + +Everything except customer reachability: + +- **The offer** appeared with all three fields: `offer_path=/mnt/felhom-drives/mentes2`, + `offer_label=Mentés meghajtó`, `offer_message=Ezt a meghajtót kijelölheted a rendszermentés helyéül…` +- **The decline path (§6.4) — PROVEN.** After registering the drive and *not* accepting: + `target` still `local`, no `felhom-backup` storage, `agent.json` unchanged. **Registration does not + confer a role** — E-2 §3's central invariant, live. +- **Accept** → `{"assigned":"/mnt/felhom-drives/mentes2","restart_required":true}` +- **The agent did NOT self-restart** — `ActiveEnterTimestamp` unchanged at 12:37:53 CEST twenty + minutes later. **In-flight check performed and recorded before restarting:** 0 running PVE tasks, + no `vzdump` process, 0 backup lines in the agent journal. +- **The E-2a root-fenced wrapper worked on a fresh box:** created `dir: felhom-backup`, + `path /mnt/mentes2`, `is_mountpoint 1` — the drive's **own** mountpoint (law F-2). +- **Healthy renders nothing:** after the restart the payload is + `{"degraded":false,"known":true,"label":"Mentés meghajtó","target":"felhom-backup"}` — **no + `message` field at all**. No "backup protected" reassurance (E-2 Scenario E). + +## 7. E-2d's own premise needs amending + +The runbook assumed a fresh install yields a claimable customer CC can then drive. It does not: +**the claim code is bcrypt-hashed in the hub and only ever emailed** (`claim/engine.go:58-95`), and the +gate covers everything except `/claim`, `/claim/request-new-code`, `/api/health`, `/static/` +(`claim.go:222-229`). C3/C4/C5 all sit behind it. This run cleared it by registering an operator email, +resending, and having the operator relay the code — **the one genuine human step**, and it also proved +the claim flow end to end (code → password → `401 "dashboard not yet claimed"` becoming +`401 "authentication required"`, a positive discriminator). + +## 8. Teardown + +- VM 9300 destroyed `--purge`; `e2d-images` storage removed; scratch dir removed. +- **`pvesm status` after == before**: `local-lvm` **38.77 %**, `felhom-backup` 931059224 KiB available + — byte-identical to the pre-run measurement. Freed space returned. +- **Guest 9201 (live customer) untouched and running; `drill-r50` VM 300 untouched.** +- Drill VM (golden bake) torn down per GL-1 earlier: guest 9100 purged, secrets shredded, disk restored + to `virgin`, token-leak grep 0. + +**Hub records NOT yet removed — stated, not silent.** `e2d-fresh` + host `e2d-fresh-ac9f09` remain. +The delete was attempted and **correctly refused** at four successive gates: acknowledgements → typed +`confirm_id` → `expect_hosts` stale-preview → finally `host e2d-fresh-ac9f09 is ONLINE`. The host is +online only because its last report is recent; the VM is destroyed, so it ages OK → WARN (30 m) → DOWN +(>1 h) and is then deletable. **Cleanup command, once it reads DOWN:** + +``` +POST /configs/e2d-fresh/delete ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=e2d-fresh expect_hosts=1 +``` + +Tracked in `OPEN-ITEMS.md`. Deleting rather than keeping is deliberate — R-93 records what a +half-real fixture costs. + +**Also still present:** the stale unclaimed appliance from 2026-07-25 (`206c8838…`, code `QWA-WJE`). +Not mine; not discarded. A future run must distinguish its own appliance from it. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 44b440b..91484c8 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -11,10 +11,14 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha |---|---|---|---|---|---| | **R-88a** | ~~Failing backup re-quiesces every 5 min, no backoff~~ | **SHIPPED** (controller v0.176.0, 2026-07-27) | — | Live on both boxes; breaker 15m→4h, per-tier, never permanent | — | | **R-88b** | ~~`/backup/due` cannot say *unknown*~~ | **SHIPPED + PROVEN-LIVE** (agent v0.105.0 + controller v0.178.0, 2026-07-27) | — | `age_state=unknown` captured on real hardware during a deliberate ep0 outage; controller deferred, **zero app stacks stopped** | — | -| **E-2d** | **Prove E-2 on a fresh VM on the t740** — the only remaining route to four unproven items: a real `felhom-host-install.sh` **1.22.0** run (never done), Case B naturally (single-drive install renders the degraded banner without degrading a live box), a **claimable** customer so the three claim-gated items stop being gated, then add a drive (the offer) and unplug it (`backup_target_absent` end-to-end) | **READY (M)** | — | **Space checked 2026-07-29 — NOT a blocker, with one constraint: the VM disk must NOT go on `local-lvm`.** That thin pool is over-subscribed (144 GB allocated against 54 GB, 38.8% used) on a box running a live customer guest, and a full thin pool corrupts every guest on it. `local` has only 23.7 GB and sits on `pve-root`. **Use `/mnt/nvme-1tb` (888 GB free).** Caveat found: a dir storage at a SUBDIRECTORY there will fail the agent's `exactMount` check and report `disconnected` in the host report — cosmetic, but decide the placement deliberately. **Do NOT unblock drill-r50** (deliberately blocked; unblocking it means the fixture stops representing anything real). **CORRECTED 2026-07-29 — the ISO is not an obstacle to the 1.22.0 proof, it is the best route to it.** `felhom-bootstrap.sh:96` fetches from `https://felhom.eu/scripts/felhom-host-install.sh`, not the hub, and that URL serves **1.22.0** (git-sync from `main`, ≤30 s — live fetch confirmed). A fresh ISO install therefore exercises 1.22.0 **automatically** — which makes the ISO leg the *stronger* proof, because it is the real customer path rather than a proxy for it, and it retires the "run 1.22.0 manually" step. **The Phase 0 question that decides the shape is ANSWERED — it was a source read, and the answer is yes.** A fresh VM with no baked customer-id lands in **PAIRING** mode (`felhom-bootstrap.sh:537-541`), not **DIRECT** (`:312`), and only DIRECT passes `--customer-id / --mode / --passphrase-file`. But on a 200 from `/api/v1/appliance/poll` the pairing loop writes the hub-delivered `FELHOM_CUSTOMER_ID` + `FELHOM_RETRIEVAL_PASSPHRASE` into the 0600 env, re-sources it and calls `run_direct` **in the same invocation** (`:495-499`) — so pairing reaches the identical installer invocation (`:322-343` — the single `$INSTALL_URL` fetch at `:322-330`, the `--customer-id/--mode/--hub-url/--passphrase-file` args array at `:334`, and the `bash "$SCRIPT_TMP" "${args[@]}"` call itself at `:343`) and the customer it yields is the one the operator bound, i.e. claimable. **So the ISO leg is the spine**; a manual 1.22.0 run is not needed as a separate scenario. **RUN ATTEMPTED 2026-07-29 — STOPPED AT PHASE 0, no VM created, nothing touched (`audits/E2D-fresh-vm-2026-07-29.md`).** The blocker is **R-111**: a fresh box installs **agent 0.96.0 + controller 0.161.0**, not `main`'s 0.113.0/0.185.1, so **C3/C4/C5 test surfaces that do not exist on it** — the degraded banner + `GET /api/storage/backup-target` landed in controller **v0.185.1** (`cdaeb36`) with the copy in **v0.185.0** (`3f7cf2a`); `backup_target_absent` in **v0.184.0** (`c1a63de`); the offer's apply needs agent **v0.113.0** (`58b598b`). **C1 (a real rc=0 1.22.0 install) and C2 (Case B natural, `configure_backup_target()` `felhom-host-install.sh:627`, warnings `:653-655`) remain ACHIEVABLE TODAY** — both are installer-side and host-install is served at 1.22.0. **C3 unblocks cheaply** by raising the hub global floor 0.156.0 → ≥0.185.0 (controller 0.185.1 IS in the registry): measured fleet impact is **nil** — both demo boxes already run 0.185.1, and Peti is DOWN 14 d and already below the current floor. **C4/C5 need R-111 first.** Phase 0 answers are all recorded in the audit, so a resumed run does not re-derive them: drive-gate cadence **30 s** (`intermediary.go:337`, registered `server.go:217`) ⇒ a 60 s two-cycle budget; hot-detach available (`virtio-scsi-single` + default hotplug, VM 300 is the working reference); ISO present (`…v1.25.0-nested-vm-generic-mkimage.iso`); `/mnt/nvme-1tb` 888 G free and the `local-lvm` fence re-measured (38.77 %, unchanged). **The §5.1a operator STOP is retired** — `HUB_PW` is in `~/.config/credentials` and hub auth was verified, so CC can bind. **One open decision carried forward:** where the VM disk's dir storage goes, since `local-lvm` is forbidden and `felhom-backup` is the live backup target — see the audit §4.1 | CC | -| **R-94** | **A hand-synced version constant drifts, and the gate that would catch it is never run** — `hub/internal/web/configs.go:28` pins `hostInstallVersion = "1.19.0"` while `scripts/felhom-host-install.sh:187` is `SCRIPT_VERSION="1.22.0"` | **READY (XS)** | — | **CORRECTED 2026-07-29 — the earlier framing of this row was false and is retracted.** The constant selects no script: its only consumers are `configs.go:487` (`ScriptVersion`) and `render_test.go:219`, and it renders as a label at `customer_unified.html:494`. The install command beneath that label fetches `https://felhom.eu/scripts/felhom-host-install.sh` (`customer_unified.html:563`, `:1262`), which the website git-syncs from `main` on a 30 s period (`manifests/webpage.yaml`) — so **1.22.0 is what every install already gets** (live fetch, 2026-07-29). Every flag the generator emits is parsed by 1.22.0 (`customer_unified.html`~`:1210`–`:1238` vs `felhom-host-install.sh:1177`–`:1210`): **no functional gap, only a wrong number on the operator's screen.** Three legs, all XS: **(a)** derive the label from `SCRIPT_VERSION` rather than hand-syncing it, or delete it; **(b)** `scripts/hostinstall_gates.py` **fails today** and is invoked by no Makefile, hook or `CLAUDE.md` — wire it next to `site_gates.py` or delete it, because a gate nobody runs reads as coverage it is not providing (**this leg is one instance of → R-29**, which is the class: gates are enforced nowhere, and the enforcement decision belongs there, not here); **(c)** `render_test.go:219` compares the constant to itself and passes at any value — replace it with the cross-file assertion. **No longer blocked on E-2d** — it never gated anything | CC | -| **R-110** | **`main` is the installer's publish channel — there is no staging.** `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a 30 s period and nginx serves that working tree directly (`location /scripts/`, `root …/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as **root on a virgin box**, the single most privileged thing Felhom ships | **WAITING-ON-OPERATOR (S)** | operator ruling | **Two consequences worth stating:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" **was never available to take**. **Open question for the operator, not a defect to fix blind:** whether `/scripts/` should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether `main`-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now | CC | +| **E-2d** | **Prove E-2 on a fresh VM on the t740** — the only remaining route to four unproven items: a real `felhom-host-install.sh` **1.22.0** run (never done), Case B naturally (single-drive install renders the degraded banner without degrading a live box), a **claimable** customer so the three claim-gated items stop being gated, then add a drive (the offer) and unplug it (`backup_target_absent` end-to-end) | **READY (M)** | — | **Space checked 2026-07-29 — NOT a blocker, with one constraint: the VM disk must NOT go on `local-lvm`.** That thin pool is over-subscribed (144 GB allocated against 54 GB, 38.8% used) on a box running a live customer guest, and a full thin pool corrupts every guest on it. `local` has only 23.7 GB and sits on `pve-root`. **Use `/mnt/nvme-1tb` (888 GB free).** Caveat found: a dir storage at a SUBDIRECTORY there will fail the agent's `exactMount` check and report `disconnected` in the host report — cosmetic, but decide the placement deliberately. **Do NOT unblock drill-r50** (deliberately blocked; unblocking it means the fixture stops representing anything real). **CORRECTED 2026-07-29 — the ISO is not an obstacle to the 1.22.0 proof, it is the best route to it.** `felhom-bootstrap.sh:96` fetches from `https://felhom.eu/scripts/felhom-host-install.sh`, not the hub, and that URL serves **1.22.0** (git-sync from `main`, ≤30 s — live fetch confirmed). A fresh ISO install therefore exercises 1.22.0 **automatically** — which makes the ISO leg the *stronger* proof, because it is the real customer path rather than a proxy for it, and it retires the "run 1.22.0 manually" step. **The Phase 0 question that decides the shape is ANSWERED — it was a source read, and the answer is yes.** A fresh VM with no baked customer-id lands in **PAIRING** mode (`felhom-bootstrap.sh:537-541`), not **DIRECT** (`:312`), and only DIRECT passes `--customer-id / --mode / --passphrase-file`. But on a 200 from `/api/v1/appliance/poll` the pairing loop writes the hub-delivered `FELHOM_CUSTOMER_ID` + `FELHOM_RETRIEVAL_PASSPHRASE` into the 0600 env, re-sources it and calls `run_direct` **in the same invocation** (`:495-499`) — so pairing reaches the identical installer invocation (`:322-343` — the single `$INSTALL_URL` fetch at `:322-330`, the `--customer-id/--mode/--hub-url/--passphrase-file` args array at `:334`, and the `bash "$SCRIPT_TMP" "${args[@]}"` call itself at `:343`) and the customer it yields is the one the operator bound, i.e. claimable. **So the ISO leg is the spine**; a manual 1.22.0 run is not needed as a separate scenario. **RUN ATTEMPTED 2026-07-29 — STOPPED AT PHASE 0, no VM created, nothing touched (`audits/E2D-fresh-vm-2026-07-29.md`).** The blocker is **R-111**: a fresh box installs **agent 0.96.0 + controller 0.161.0**, not `main`'s 0.113.0/0.185.1, so **C3/C4/C5 test surfaces that do not exist on it** — the degraded banner + `GET /api/storage/backup-target` landed in controller **v0.185.1** (`cdaeb36`) with the copy in **v0.185.0** (`3f7cf2a`); `backup_target_absent` in **v0.184.0** (`c1a63de`); the offer's apply needs agent **v0.113.0** (`58b598b`). **C1 (a real rc=0 1.22.0 install) and C2 (Case B natural, `configure_backup_target()` `felhom-host-install.sh:627`, warnings `:653-655`) remain ACHIEVABLE TODAY** — both are installer-side and host-install is served at 1.22.0. **C3 unblocks cheaply** by raising the hub global floor 0.156.0 → ≥0.185.0 (controller 0.185.1 IS in the registry): measured fleet impact is **nil** — both demo boxes already run 0.185.1, and Peti is DOWN 14 d and already below the current floor. **C4/C5 need R-111 first.** Phase 0 answers are all recorded in the audit, so a resumed run does not re-derive them: drive-gate cadence **30 s** (`intermediary.go:337`, registered `server.go:217`) ⇒ a 60 s two-cycle budget; hot-detach available (`virtio-scsi-single` + default hotplug, VM 300 is the working reference); ISO present (`…v1.25.0-nested-vm-generic-mkimage.iso`); `/mnt/nvme-1tb` 888 G free and the `local-lvm` fence re-measured (38.77 %, unchanged). **The §5.1a operator STOP is retired** — `HUB_PW` is in `~/.config/credentials` and hub auth was verified, so CC can bind. **One open decision carried forward:** where the VM disk's dir storage goes, since `local-lvm` is forbidden and `felhom-backup` is the live backup target — see the audit §4.1. **RUN EXECUTED 2026-07-29 after R-111 was fixed — `audits/E2D-fresh-vm-2026-07-29.md`.** Full ISO/PAIRING route on a nested VM on demo-hp; bind → running controller in **3 m 35 s**; teardown clean (`pvesm status` after == before, `local-lvm` 38.77 %, guest 9201 untouched). **C1 PROVEN** (`felhom-host-install v1.22.0`, `Day-0 provision SUCCESS`, guest 9201 running, golden = the one baked 20 min earlier). **C2 PROVEN** (both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort). **C3/C4 PARTIAL — API exact and complete, but customer-invisible → R-112.** **C5 FAILED → R-113.** C4's decline path PROVEN (registration confers no role), `restart_required:true` PROVEN, agent did **not** self-restart, E-2a wrapper created the storage at the drive's own mountpoint, healthy renders nothing. **E-2d's own premise needed amending:** a fresh install does NOT yield a CC-drivable claimable customer — the claim code is bcrypt-hashed and email-only, so one operator relay was required (and the claim flow is now proven end to end). **This row stays OPEN only for the residue:** C5 re-test after R-113, and the C3/C4 UI legs after R-112 | CC | +| **R-94** | **A hand-synced version constant drifts, and the gate that would catch it is never run** — `hub/internal/web/configs.go:28` pins `hostInstallVersion = "1.19.0"` while `scripts/felhom-host-install.sh:187` is `SCRIPT_VERSION="1.22.0"` | **READY (XS)** | — | **CORRECTED 2026-07-29 — the earlier framing of this row was false and is retracted.** The constant selects no script: its only consumers are `configs.go:487` (`ScriptVersion`) and `render_test.go:219`, and it renders as a label at `customer_unified.html:494`. The install command beneath that label fetches `https://felhom.eu/scripts/felhom-host-install.sh` (`customer_unified.html:563`, `:1262`), which the website git-syncs from `main` on a 30 s period (`manifests/webpage.yaml`) — so **1.22.0 is what every install already gets** (live fetch, 2026-07-29). Every flag the generator emits is parsed by 1.22.0 (`customer_unified.html`~`:1210`–`:1238` vs `felhom-host-install.sh:1177`–`:1210`): **no functional gap, only a wrong number on the operator's screen.** Three legs, all XS: **(a)** derive the label from `SCRIPT_VERSION` rather than hand-syncing it, or delete it; **(b)** `scripts/hostinstall_gates.py` **fails today** and is invoked by no Makefile, hook or `CLAUDE.md` — wire it next to `site_gates.py` or delete it, because a gate nobody runs reads as coverage it is not providing (**this leg is one instance of → R-29**, which is the class: gates are enforced nowhere, and the enforcement decision belongs there, not here); **(c)** `render_test.go:219` compares the constant to itself and passes at any value — replace it with the cross-file assertion. **No longer blocked on E-2d** — it never gated anything. **2026-07-29: a real 1.22.0 install has now happened** (`audits/E2D-fresh-vm-2026-07-29.md`), so even the original (retracted) precaution is discharged — nothing stands in front of this row | CC | +| **R-110** | **`main` is the installer's publish channel — there is no staging.** `manifests/webpage.yaml` git-syncs `/scripts/` from `--branch=main` on a 30 s period and nginx serves that working tree directly (`location /scripts/`, `root …/current`). So pushing `scripts/felhom-host-install.sh` **is** publishing it: within thirty seconds it is what every subsequent `felhom-bootstrap.sh` fetch (`scripts/iso/felhom-bootstrap.sh:96`) and every operator-run day-0 command (`customer_unified.html:563`) receives. There is no tag, no pinned-version path, no staging copy and no rollback other than another push — for the artifact that runs as **root on a virgin box**, the single most privileged thing Felhom ships | **WAITING-ON-OPERATOR (S)** | operator ruling | **Two consequences worth stating:** E-2d is not a gate *before* exposure — 1.22.0 has been the live installer since it hit `main` on 2026-07-29 — and the precaution recorded on the old R-94 row as "do not point every new box at an installer that has never run" **was never available to take**. **Open question for the operator, not a defect to fix blind:** whether `/scripts/` should serve a pinned release (tag-tracked path, or a versioned directory with the customer command naming a version) or whether `main`-tracking is the accepted shape for a one-operator product. Exposure today is zero — there are no boxes installing — which is exactly why it is cheap to decide now. **SECOND INSTANCE, found 2026-07-29 by the E-2d run and filed here rather than as a new ID:** `felhom-host-install.sh` fetches **nine** files from `raw/branch/main` (`:2072`–`:2206`) and the hub manifest vouches a sha for exactly **one** (`wrapper_sha256` → `felhom-pbs-apply`; re-checked this run, no drift). E-2a's `felhom-backup-target-apply` (`:2116`) is installed **0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n` — a root-executed artifact taken from `main` with no pinned integrity, which is this row's class exactly | CC | | **R-111** | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`.**~~ `felhom-host-install.sh` does not use `main`: it reads the hub-vouched manifest (`:423-436`) and fetches Gitea generic packages (agent `:1945`, golden `:2573`). Gitea's newest are **agent 0.96.0** and **golden 0.161.0**, and the hub's manifest selects exactly those — so a fresh box lands on **agent 0.96.0 + controller 0.161.0** (global floor `v0.156.0` < the golden's 0.161.0, so no self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and was never published** | **SHIPPED 2026-07-29 — the channel now serves agent 0.113.0 + golden 0.185.1** | — | **FIXED the same day it was found.** Agent **0.113.0** built from the clean tree @ `58b598b` and published (`scripts/publish-agent.sh`), sha `5f3247f756cb658e…`, round-trip GET verified. Golden **0.185.1** baked on the nested drill VM embedding controller `0.185.1`, published, sha `dba00f3e845c415e…` — bake clean: `Result=success`, overlay2, **all 3 mounts included** (rootfs+mp0+mp1), 0 FATAL/exclusions, upload HTTP 201, token-leak grep 0; log `drill/bake-0.185.1.log`; GL-1 teardown done (guest 9100 purged, secrets shredded, disk restored to `virgin`). Hub Day-0 manifest moved **both together in one POST** so it never vouched a new agent against an old golden; `min_agent` **0.93.0 → 0.113.0**, which is what controller v0.185.0 declares (`felhom-controller/CHANGELOG.md:15`) — **zero fleet impact, verified: all three enrolled hosts already run agent 0.113.0, so no box is held.** `wrapper_sha256` preserved verbatim (re-checked against `configs/felhom-pbs-apply` — no drift). **The global controller floor was deliberately NOT raised**: the golden now bakes 0.185.1, so a fresh box needs no self-update, and raising it would have been an unnecessary fleet-wide write. Original finding follows. **Found 2026-07-29 by the E-2d Phase 0 gate, which stopped the run before a VM was created.** 17 unpublished releases (v0.97.0–v0.113.0) strand the **entire R-82 tiered-backup arc** plus **F-CRIT-2** (a failed backup looking fresh — 7 days silent) and **F-REBOOT** (a guest rebooted mid-backup never returns): a new customer's box would install without them. **Blocks E-2d's C3/C4/C5** — those test endpoints and events that do not exist in 0.96.0/0.161.0. The **controller is fine** (registry has 0.185.1, floor-driven self-update), so the gap is specific to the two Gitea-generic artifacts. **Mirror of R-110, not a duplicate:** R-110 = the installer publishes instantly with no staging; R-111 = the agent/golden publish gate exists and was never walked. Fix should decide whether publishing joins the release train rather than staying a remembered step (R-29's shape, one layer up). Evidence: `audits/E2D-fresh-vm-2026-07-29.md` | CC | +| **R-113** | **The drive-absent gate CANNOT FIRE on device loss — E-2b's alarm is wired to an unreachable condition.** `planDriveGates` (`intermediary.go:216-262`) computes presence by OR-ing `d.BoundUnderParent` into `present[GuestPath]`, and the agent derives `BoundUnderParent` from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc//mountinfo`"* (`localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is NOT device-bound, so its mountinfo entry outlives the device**. The gate reads the surviving bind as "present" ⇒ no `Stop` action ⇒ `notifyDriveAbsent` never called | **READY (M) — P1** | — | **PROVEN LIVE 2026-07-29 on a fresh box.** Target drive hot-detached under a running agent; over 4½ min (budget was 60 s): the **agent** said `enrolled drive absent by UUID` every 20 s, the controller logged **0** `[gate]` lines, and the hub received **zero** events — no `backup_target_absent` **and no generic `storage_disconnected`**. Measured with the device gone: `/mnt/mentes2` NOT mounted, `/mnt/felhom-drives/mentes2` still `/dev/sdb[/felhom-data]`. **Not a virtualisation artefact** — the asymmetry is device-bound-mount vs manual-bind, identical on metal (caveat: proven on SCSI hot-detach; physical unplug not staged). **Mirror scenario reasoned, not observed:** both the specific and generic events come from the same `a.Stop` branch, so the generic one is equally unreachable. **Sixth instance of seam-built-but-never-wired** — E-2b wired the seam, to a condition that cannot occur. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 | CC | +| **R-112** | **E-2's degraded banner and offer have NO UI CONSUMER — the endpoint is correct and the customer never sees it.** `GET /api/storage/backup-target` returns byte-exact copy (proven live), and **nothing fetches it**: `grep 'backup-target'` across all `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` + `degradedMessageFor` are consumed **only** by the JSON handler — **no page handler injects the state** | **READY (S) — P1** | — | **The decisive contrast: templates fetch 18 distinct `/api/storage/*` endpoints; `backup-target` and `backup-target/assign` are the only two with zero references.** The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"* — an invariant comment asserting a consumer that does not exist (7th instance of that class). v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* — it fixed the **router mount** and stopped one layer short of the **render**; its test `TestBackupTargetRoutesLiveUnderTheStorageAPIMount` pins dispatch, not reachability, which is exactly what `CLAUDE.md`'s seam rule warns about. **Fifth instance of seam-built-but-never-wired.** **Fix R-114 FIRST** — wiring this alone would start showing customers the wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | CC | +| **R-114** | **On target-drive loss the customer is told the wrong story and offered the drive that just vanished.** With the assigned target absent, the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"a rendszermentés ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a drive that has disappeared, not the system disk — **and** `offer_path` pointing at the missing drive as the remedy | **READY (S)** | — | `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing **never configured** from **configured and now missing**. Shares R-113's root cause — two disagreeing presence signals — but is a different code path with a different fix. **Currently invisible ONLY because of R-112; fix this before wiring that.** Also seen: after reattach the drive returned as `/dev/sdc` while the stable bind still recorded `/dev/sdb`, and the state read healthy. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 | CC | +| — | **Drill cleanup: hub records for `e2d-fresh` + host `e2d-fresh-ac9f09` still present** | WATCHING | the host aging OK → WARN (30 m) → DOWN (>1 h); the VM is already destroyed so no further reports come | Delete was attempted and **correctly refused** at four gates, finally `host … is ONLINE`. Once it reads DOWN: `POST /configs/e2d-fresh/delete ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=e2d-fresh expect_hosts=1`. Deleting rather than keeping is deliberate — R-93 records what a half-real fixture costs. Also still present: the **stale 2026-07-25 unclaimed appliance** `206c8838…` (code `QWA-WJE`), not mine, not discarded | CC | | **R-29** | **The green gates are not enforced anywhere — one was RED for 16 releases before anyone ran it.** This is the **class**, not an instance: a gate that exists, asserts something true, is red, and is invoked by nothing reads as coverage it is not providing. `controller/scripts/docker_run_volume_path_gate.py` failed continuously from **2026-07-14 (v0.129.0)** until R-7b's close-out ran it by hand at v0.145.0 — sixteen releases in which every REPORT said "green" | **READY (S for (a) / M for (b))** | — | **This item has existed at `ROADMAP.md:158` since before the register was rebuilt (2026-07-27) and was never carried across — that omission is itself part of the finding**, because it is an open item *about work not getting done* that then went missing from the page that decides what gets done. Two separable parts, per R-29's own analysis: **(a)** the `docker_run_volume_path_gate` finding is benign and the fix is a 3-line ALLOWLIST addition with its why — **not** a rewrite of the flagged call — and it gets its own reviewed diff, never bundled into a feature commit; **(b)** the systemic half, the real item: decide where gates run (pre-push hook, `build.sh` step, or CI) and make a red gate block the train the way the Go green gate does. **Two further orphans confirmed 2026-07-29** by repo-wide grep across all file types + sibling repos + `~/.claude` settings/skills/hooks + `.git/hooks` (none non-sample) + Makefile/justfile/Taskfile find (only `hub/Makefile`, zero `gate` occurrences) + CI-directory find (**this repo has no CI at all**) — every one of the 19 hits is a docstring, a code comment or prose, and **not one is an invocation**: `scripts/hostinstall_gates.py` — **RED today** (`hub Setup-tab hostInstallVersion=1.19.0 != SCRIPT_VERSION=1.22.0`, exit 1), the same finding as **R-94 leg (b)** — and `scripts/hub_confirm_gate.py`. Of the four gates in `scripts/`, only `site_gates.py` is mandated anywhere (`CLAUDE.md:153`) and `manifest_bearer_gate.py` is named in `runbooks/secrets.md:76`. **In R-29's own words, carried forward deliberately: do not mint a new ID for a new instance** — the 2026-07-18 rehearsal independently re-raised this item and no second ID was minted then either | CC | | **R-95** | restic offsite credential **can delete** (`readonly=False`, `forget --prune` runs from the box); SFTP cannot express append-only | **READY** | — | Root exposure still open. Mitigation now ARMED — split prune off-box or move to REST `--append-only` | CC | | **R-86** | Restore-tests are interval-scheduled, not backup-aligned | **READY** | R-90 (ep0 headroom) informs cadence | Trigger a tier ~24 h after **its own** newest archive | CC | @@ -44,7 +48,7 @@ State: `BLOCKED` · `READY` · `WAITING-ON-OPERATOR` · `WATCHING`. Every row ha | **F-REBOOT** | ~~A guest rebooted during its backup does not come back — shutdown completes, start never happens, no self-heal; 9m47s total appliance outage with every alarm silent~~ | **SHIPPED + PROVEN-LIVE** (agent v0.107.0, 2026-07-28) | — | 60 s guest-power watchdog; `onboot` is the deliberate-stop discriminator (already the stale-lock path's, and what `pve-guests` consults), retry bounded 3x/1m-2m-4m then escalates once. Live on demo-hp: **120 s unattended** vs the incident's 587 s with a human; Scenario B proven (an `onboot:0` guest left stopped) | — | | **F-LEAK** | ~~A failed restore-test cannot destroy its own scratch guest (403 `VM.Allocate`); the 10-slot VMID band shrinks silently~~ | **SHIPPED + PROVEN-LIVE** (agent v0.110.0 + host-install v1.21.0, 2026-07-28) | — | **Three attempts, two refuted live.** (1) Pool adoption: `PUT /pools/{pool}` also needs `VM.Allocate` on the VM — membership cannot bootstrap its own authority. (2) Per-path `/vms/990000..990009` ACLs: work, but PVE's destroy calls `remove_vm_access` (`LXC.pm:906`) which deletes every ACL at `/vms/` — **consumed by the op it authorises**, one use per slot. (3) SHIPPED: 4th root-fenced exception, band enforced in sudoers **literally** (`pct destroy 99000[0-9] --purge`) + in code + at the caller; API destroy still tried first. Live: band PERMITTED, `9201`/`9100`/`9999`/`990010`/`1` REFUSED, and `pct start 990000` REFUSED too | — | | **F-OBS** | ~~`deadapp-check` leaves NO positive observable on a default (info-level) box — "no alarms" was indistinguishable from "never ran"~~ | **SHIPPED + PROVEN-LIVE** (controller v0.180.0 + agent v0.109.0, 2026-07-28) | — | INFO summary every 20th scan carrying scans/evaluated/down. **Agent v0.109.0 fixes the same shape in the guest-power watchdog shipped hours earlier in v0.107.0** — it logged only at startup and when it acted, so its health could be read only from absence | — | -| **E-2** | ~~Drive-role machinery around the moved vzdump target~~ | **SHIPPED** (hub 0.81.0, agent 0.113.0, controller 0.185.1, host-install 1.22.0 — 2026-07-29) | — | Parts 1–5 complete. Role model + offer-only assignment + Hungarian degraded banner + absent-target signal + installer Case A/B. **NOT yet live-proven:** the DEGRADED banner and the offer acceptance (both demo boxes are healthy, so neither state occurs naturally) and `backup_target_absent` end-to-end. Installer is **installer-logic-tested, not install-tested** — no reinstall was performed | CC | +| **E-2** | ~~Drive-role machinery around the moved vzdump target~~ | **SHIPPED** (hub 0.81.0, agent 0.113.0, controller 0.185.1, host-install 1.22.0 — 2026-07-29) | — | Parts 1–5 complete. Role model + offer-only assignment + Hungarian degraded banner + absent-target signal + installer Case A/B. **NOT yet live-proven:** the DEGRADED banner and the offer acceptance (both demo boxes are healthy, so neither state occurs naturally) and `backup_target_absent` end-to-end. Installer is ~~installer-logic-tested, not install-tested~~ — **INSTALL-TESTED 2026-07-29** on a fresh nested box via the real ISO/PAIRING route, rc=0 (`audits/E2D-fresh-vm-2026-07-29.md` §3). **Of the "NOT yet live-proven" list: Case B + the degraded state are now PROVEN at the installer and API level; the OFFER ACCEPTANCE is PROVEN at the API level** (decline path, `restart_required:true`, E-2a wrapper, healthy-renders-nothing). **Still NOT proven, and now known to be BROKEN rather than merely untested:** the banner/offer never reach a customer (**R-112**) and `backup_target_absent` cannot fire on device loss (**R-113**), with the absent-state message itself wrong (**R-114**) | CC | | **E-2a** | ~~The target move needs a root-fenced wrapper — the agent cannot do it~~ | **SHIPPED + PROVEN-LIVE** (agent v0.113.0 + host-install v1.22.0, 2026-07-29) | — | `felhom-backup-target-apply` behind a literal `FELHOM_BACKUPTARGET` sudoers alias; the agent's PVE role was NOT widened. Enforces F-1 (`mountpoint -q`) and F-2 (`is_mountpoint 1` hardcoded), refuses a root-device target, has NO storage-removal path (grep-assertable), is idempotent and refuses to repoint. All five laws proven live as root on demo-hp with 0 stray storages | — | | **E-2b** | ~~`NotifyStorageDisconnected`/`Reconnected` defined and called NOWHERE — a drive going absent emitted no event on any channel~~ | **SHIPPED + PROVEN-LIVE** (controller v0.184.1 + agent v0.112.0 + hub v0.81.0, 2026-07-29) | — | Seam wired in `ReconcileDriveGates`; a target drive raises the specific `backup_target_absent` instead. **A keying bug was caught before deploy:** `a.Path` is the registered GUEST path, not the agent's host `MountPath`, so the target branch was unreachable — every absent drive, target included, fell through to the generic event (v0.184.1). Tests observe the WIRE (httptest hub), not a mock | — | | **E-2c** | ~~E-1 put the whole-guest backups on a drive `POST /disks/eject` would eject~~ | **SHIPPED + PROVEN-LIVE** (agent v0.112.0, 2026-07-29) | — | Eject + decommission refuse 409 on the backup-target mount, naming the storage and the remedy. **Live on BOTH boxes:** demo-hp `/mnt/nvme-1tb` and demo-felhom `/mnt/hdd_1` both refused, drives unmoved. NOT a role reclassification — `RoleForStorage` untouched, because on both boxes that drive is ALSO the enrolled user-data drive; `TestEjectStillAllowedOnANonTargetDrive` pins the non-over-correction and `/var/lib/vz` is still refused by the PRE-EXISTING role gate, not this one | — | diff --git a/documentation/backlog/ROADMAP.md b/documentation/backlog/ROADMAP.md index e1f5bba..ec45175 100644 --- a/documentation/backlog/ROADMAP.md +++ b/documentation/backlog/ROADMAP.md @@ -20,7 +20,10 @@ | ID | Item | Size | Status | Notes / map rows flipped | |----|------|------|--------|--------------------------| -| R-111 | **The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`** | M | idea — found 2026-07-29 by the E-2d Phase 0 gate | **The fleet's live versions are not the fleet's INSTALLABLE versions, and only the first were ever checked.** `felhom-host-install.sh` does not use `main`: `resolve_artifacts()` (`:423-436`) reads the hub-vouched manifest (`GET /api/v1/artifacts/`, `hub/internal/api/handler.go:2120`) and fetches versioned **Gitea generic packages** — agent from `:1945`, golden from `:2573`. Gitea holds **`felhom-agent` newest `0.96.0`** and **`felhom-golden` newest `0.161.0`**; the hub's saved manifest selects exactly those. So a fresh box lands on **agent 0.96.0 + controller 0.161.0** (golden bake; the global floor is `v0.156.0` < 0.161.0, so it does not self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and is not in the channel at all** — demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. **17 unpublished releases (`felhom-agent/CHANGELOG.md` v0.97.0–v0.113.0)**, including the ENTIRE R-82 per-target backup-tier arc (v0.97.0–v0.104.0), **F-CRIT-2** (v0.106.0 — a failed backup looking fresh, 7 days silent), **F-REBOOT** (v0.107.0 — a guest rebooted mid-backup never returns), F-LEAK (v0.108.0/0.110.0), F-OBS (v0.109.0), E-2c (v0.111.0), E-2b (v0.112.0), E-2a (v0.113.0). **P1 because it gates the first remote tester:** their box would install an agent predating the tiered-backup model and both silent-failure fixes. **Mirror of R-110, not a duplicate:** R-110 is *the installer publishes instantly with no staging*; this is *the agent and golden have a deliberate publish+vouch gate and it was never walked* — opposite failure modes of one subject, different fixes. Contrast worth keeping: the **controller** is fine (registry has 0.185.1; it self-updates from the floor), so the gap is specific to the two Gitea-generic artifacts. **Decide as part of the fix:** whether publishing becomes part of the release train rather than a separate remembered step — this is R-29's shape (a gate that exists and is never walked) one layer up. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §2 | +| R-113 | **The drive-absent gate cannot fire on device loss — E-2b's alarm is wired to an unreachable condition** | M | idea — **PROVEN LIVE 2026-07-29** | `planDriveGates` (`felhom-controller/internal/web/intermediary.go:216-262`) treats a path as present by OR-ing in `d.BoundUnderParent`, which the agent derives from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc//mountinfo`"* (`internal/localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is not device-bound and its mountinfo entry outlives the device**, so the gate reads it as present and `notifyDriveAbsent` is never called. Live on a fresh box: target drive hot-detached, agent said `enrolled drive absent by UUID` every 20 s for 4½ min, controller logged **0** `[gate]` lines, hub received **zero** events — neither `backup_target_absent` nor the generic `storage_disconnected`. Not a virtualisation artefact (device-bound-mount vs manual-bind is the same on metal); caveat: SCSI hot-detach, physical unplug not staged. **Sixth instance of seam-built-but-never-wired — E-2b wired the seam to a condition that cannot occur.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 | +| R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints** — `backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 | +| R-114 | **On target-drive loss the customer is told the wrong story and offered the drive that vanished** | S | idea — **PROVEN LIVE 2026-07-29** | With the assigned target absent the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a missing drive, not the system disk — **and** an `offer_path` pointing at the drive that just disappeared. `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing *never configured* from *configured and now missing*. Shares R-113's root cause (two disagreeing presence signals), different code path and fix. **Invisible today only because of R-112.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 | +| R-111 | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`**~~ | M | **SHIPPED 2026-07-29** — agent 0.113.0 published (sha `5f3247f7…`, round-trip verified) + golden **0.185.1** baked and published (sha `dba00f3e…`, embeds controller 0.185.1); hub Day-0 manifest moved both together with `min_agent` 0.93.0→0.113.0 (zero fleet impact — all three enrolled hosts already at 0.113.0). **Proven end-to-end the same day:** the E-2d fresh install fetched exactly these and restored the golden baked 20 minutes earlier. Original finding: | **The fleet's live versions are not the fleet's INSTALLABLE versions, and only the first were ever checked.** `felhom-host-install.sh` does not use `main`: `resolve_artifacts()` (`:423-436`) reads the hub-vouched manifest (`GET /api/v1/artifacts/`, `hub/internal/api/handler.go:2120`) and fetches versioned **Gitea generic packages** — agent from `:1945`, golden from `:2573`. Gitea holds **`felhom-agent` newest `0.96.0`** and **`felhom-golden` newest `0.161.0`**; the hub's saved manifest selects exactly those. So a fresh box lands on **agent 0.96.0 + controller 0.161.0** (golden bake; the global floor is `v0.156.0` < 0.161.0, so it does not self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and is not in the channel at all** — demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. **17 unpublished releases (`felhom-agent/CHANGELOG.md` v0.97.0–v0.113.0)**, including the ENTIRE R-82 per-target backup-tier arc (v0.97.0–v0.104.0), **F-CRIT-2** (v0.106.0 — a failed backup looking fresh, 7 days silent), **F-REBOOT** (v0.107.0 — a guest rebooted mid-backup never returns), F-LEAK (v0.108.0/0.110.0), F-OBS (v0.109.0), E-2c (v0.111.0), E-2b (v0.112.0), E-2a (v0.113.0). **P1 because it gates the first remote tester:** their box would install an agent predating the tiered-backup model and both silent-failure fixes. **Mirror of R-110, not a duplicate:** R-110 is *the installer publishes instantly with no staging*; this is *the agent and golden have a deliberate publish+vouch gate and it was never walked* — opposite failure modes of one subject, different fixes. Contrast worth keeping: the **controller** is fine (registry has 0.185.1; it self-updates from the floor), so the gap is specific to the two Gitea-generic artifacts. **Decide as part of the fix:** whether publishing becomes part of the release train rather than a separate remembered step — this is R-29's shape (a gate that exists and is never walked) one layer up. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §2 | | R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path | | R-2 | ~~Resolve ~215 lines of foreign WIP in felhom.eu clone (`hub/internal/notify/`, `store.go`, `hub/internal/claim/`)~~ | S | **killed** (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — `notify/`+`claim/engine.go` in `6b40eb8` (v0.50.0), `store.go` in `a1d0450` (v0.54.0), plus follow-up `e205a2d`; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) | | R-3 | Friend-alpha onboarding runbook (generalized from `pilot/RUNBOOK-peti-return-2026-07-13`): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step | M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of `PETI-tester-agreement.md`. **Next from-scratch rehearsal to include customer DELETE + re-create** — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). **S6b (manual stale-host delete before re-enroll) is OBSOLETE** — re-enrollment upserts the existing host row cleanly (`store.UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. **NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE):** one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). **Remaining consistency gap:** the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b |