admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

E-2d executed on a fresh box: C1/C2 proven, C3/C4 partial, C5 FAILS — R-112/113/114

Browse Source 
Full ISO/PAIRING route on a nested PVE VM on demo-hp, after R-111 was fixed
earlier in the session. Bind -> running controller in 3m35s. The install fetched
the artifacts published an hour before and restored the golden baked 20 minutes
before, so the publish train is proven end to end on a real install.

C1 PROVEN: "felhom-host-install v1.22.0", "Day-0 provision SUCCESS", guest 9201
running, bootstrap unit wrote its done-flag and self-disabled. This retires E-2's
"installer-logic-tested, not install-tested".

C2 PROVEN: both DEGRADED warning lines verbatim, backup.local_backup_target=local,
no felhom-backup storage created, and the install did not abort.

C3/C4 PARTIAL and C5 FAILED — three findings, none fixed:

R-112 (P1): E-2's degraded banner and offer have NO UI CONSUMER. The endpoint
returns byte-exact copy; grep 'backup-target' across every html/js/css is 0 hits
and no page handler injects the state. Templates fetch 18 distinct /api/storage/*
endpoints; these two are the only ones with zero references. v0.185.1 fixed the
router mount and stopped one layer short of the render. Fifth instance of
seam-built-but-never-wired.

R-113 (P1): the drive-absent gate CANNOT FIRE on device loss. planDriveGates
reads presence from BoundUnderParent = "is this path in the guest's mountinfo".
The raw mount is a device-bound systemd unit and dies with the device; the
agent's own bind is not device-bound and outlives it, so the gate sees "present"
forever. Live: agent reported the drive absent every 20s for 4.5 minutes, the
controller logged 0 [gate] lines, the hub received zero events -- neither
backup_target_absent nor the generic storage_disconnected. Sixth instance of the
class: E-2b wired the seam to a condition that cannot occur.

R-114: on target-drive loss the message claims the backup is on the system disk
(false) and offers the drive that just vanished. Invisible only because of R-112,
so it must be fixed BEFORE R-112 is wired.

Also filed as a second instance under R-110 rather than a new ID: host-install
fetches nine files from raw/branch/main and the hub vouches a sha for one;
E-2a's wrapper is installed 0755 to /usr/local/sbin, root-fenced in sudoers,
validated only by bash -n.

C4 is fully proven at API level: decline path (registration confers no role),
restart_required:true, agent did NOT self-restart (in-flight check performed and
recorded first), E-2a wrapper created the storage at the drive's own mountpoint,
and healthy renders nothing.

Teardown: VM destroyed, scratch storage removed, pvesm status after == before
(local-lvm 38.77%), guest 9201 and drill-r50 untouched. Hub records for e2d-fresh
remain -- delete correctly refused at four gates, finally "host is ONLINE";
deletable once it ages to DOWN. Command recorded in OPEN-ITEMS.md.

capability-map NOT touched: the customer-facing legs are broken rather than
proven, and the map has no E-2 rows at all.
This commit is contained in:
authentik Default Admin
2026-07-29 13:13:56 +02:00
parent 3dff3573f7
commit f3975cf5bc
4 changed files with 307 additions and 211 deletions
Download Patch File Download Diff File Expand all files Collapse all files
REPORT-e2d.md
+71 -42
View File
@@ -1,58 +1,87 @@
# REPORT — E-2d fresh-VM run: STOPPED at Phase 0 (2026-07-29)
# REPORT — R-111 fixed, then E-2 proven on a fresh box (2026-07-29)
`RUNBOOK-e2d-fresh-vm-2026-07-29.md`, executed by CC on DooPlex. Full evidence:
`documentation/audits/E2D-fresh-vm-2026-07-29.md`. Written as `REPORT-e2d.md` per the runbook §8.5;
root `REPORT.md` untouched.
Two phases in one session. Full evidence: `documentation/audits/E2D-fresh-vm-2026-07-29.md`.
Root `REPORT.md` untouched.
**Outcome: no VM created, no install run, no box touched, no teardown needed.** The run stopped at the
Phase 0 gate, per §3 (*"If any gate fails, STOP and report — do not adapt around it"*).
## Phase 1 — R-111: the Day-0 channel now serves the current software
## Why
A Phase 0 gate earlier the same day stopped the E-2d run before any VM existed: a fresh box would
have installed **agent 0.96.0 + controller 0.161.0**, ~17 and ~24 releases behind `main`.
`felhom-host-install.sh` does not install what is on `main`. It resolves the hub-vouched manifest
(`:423-436`) and fetches **Gitea generic packages** (agent `:1945`, golden `:2573`). Gitea's newest are
**agent `0.96.0`** and **golden `0.161.0`**; the hub's saved manifest selects exactly those; the global
controller floor is `v0.156.0`, below the golden's 0.161.0, so no self-update follows.
| Component | Fresh install gets | `main` / demo boxes |
| | Before | Now |
|---|---|---|
| host-install | **1.22.0** | 1.22.0 |
| agent | **0.96.0** | 0.113.0 — **never published**, direct-deployed |
| controller | **0.161.0** | 0.185.1 |
| agent (Gitea generic) | 0.96.0 | **0.113.0**, sha `5f3247f7…`, round-trip verified |
| golden (Gitea generic) | 0.161.0 | **0.185.1**, sha `dba00f3e…`, embeds controller 0.185.1 |
| hub `min_agent` | 0.93.0 | **0.113.0** (what controller v0.185.0 declares) |
## Claims
Bake clean on every marker: `Result=success`, overlay2, **all three mounts in the archive**, 0
FATAL/exclusions, HTTP 201, token-leak grep 0. GL-1 teardown: guest 9100 purged, secrets shredded,
drill disk restored to `virgin`. Agent + golden moved in **one** manifest POST so it never vouched a
new agent against an old golden. `min_agent` verified zero-impact first (all three enrolled hosts
already at 0.113.0). Global floor deliberately **not** raised — the golden now bakes 0.185.1.
| Claim | Verdict | Pinned to |
|---|---|---|
| C1 — real rc=0 install of 1.22.0 | **ACHIEVABLE, not run** | installer-side |
| C2 — Case B fires naturally | **ACHIEVABLE, not run** | `felhom-host-install.sh:627`, `:653-655` |
| C3 — degraded banner renders | **BLOCKED** | controller v0.185.1 `cdaeb36`; copy v0.185.0 `3f7cf2a` |
| C4 — offer moves the target | **BLOCKED** | controller v0.185.1 + agent v0.113.0 `58b598b` |
| C5 — `backup_target_absent` e2e | **BLOCKED** | controller v0.184.0 `c1a63de` + agent v0.112.0 |
Commit `3dff357`.
C3 unblocks by raising the hub floor to ≥0.185.0 (controller 0.185.1 **is** in the registry); measured
fleet impact nil — both demo boxes already run 0.185.1, Peti is DOWN 14 d and already below the current
floor. C4/C5 need agent 0.112.0/0.113.0 **published**, which runbook §0 forbids this run from doing.
## Phase 2 — the E-2d run, full ISO/PAIRING route
## Filed
Nested PVE VM on demo-hp, one disk, outside the `felhom` pool. Bind → running controller in
**3 m 35 s**. The install fetched exactly the artifacts published an hour earlier and restored
`vzdump-lxc-9100-2026_07_29-12_37_56` — the golden baked 20 minutes before. The publish train is
proven end to end on a real install.
**R-111 (P1)** — the Day-0 artifact channel is 17 agent releases stale. v0.97.0–v0.113.0 unpublished,
stranding the entire R-82 tiered-backup arc plus **F-CRIT-2** and **F-REBOOT**. A new customer's box
installs without them. Mirror of R-110, not a duplicate: R-110 = publishes instantly with no staging;
R-111 = the publish gate exists and was never walked.
| Claim | Verdict |
|---|---|
| **C1** host-install 1.22.0 completes a real install, rc=0 | ✅ **PROVEN** |
| **C2** Case B fires naturally | ✅ **PROVEN** — both DEGRADED lines verbatim, `local_backup_target=local`, install did not abort |
| **C3** degraded banner renders **to a customer** | ⚠️ **PARTIAL** — API byte-exact; **no UI consumer exists** → **R-112** |
| **C4** offer appears and moves the target | ⚠️ **PARTIAL** — decline path, `restart_required:true`, no self-restart, E-2a wrapper, healthy-renders-nothing all PROVEN at API level; offer equally invisible → **R-112** |
| **C5** `backup_target_absent` end to end | ❌ **FAILED** — zero events on any channel → **R-113** |
## The three findings
**R-112 (P1)** — E-2's banner and offer have **no UI consumer**. The endpoint returns byte-exact copy;
`grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**, and no page handler injects
the state. Decisive contrast: templates fetch **18** distinct `/api/storage/*` endpoints;
`backup-target` and `backup-target/assign` are the only two with zero references. v0.185.1 fixed the
router mount and stopped one layer short of the render. Fifth instance of seam-built-but-never-wired.
**R-113 (P1)** — the drive-absent gate **cannot fire on device loss**. `planDriveGates` reads presence
from `BoundUnderParent` = "is this path in the guest's mountinfo". The raw mount is a device-bound
systemd unit and dies; **the agent's own bind is not device-bound and outlives the device**, so the
gate sees "present" forever. Live: agent said `enrolled drive absent by UUID` every 20 s for 4½
minutes, controller logged **0** `[gate]` lines, hub got **zero** events — neither the specific nor the
generic one. Sixth instance of the class, one layer deeper: E-2b wired the seam to a condition that
cannot occur.
**R-114** — on target-drive loss the message says the backup is *"on the same disk as the system"*
(false) and offers **the drive that just vanished**. Invisible today only because of R-112 — so
**R-114 must be fixed before R-112 is wired.**
Also filed as a **second instance under R-110** (not a new ID): host-install fetches **nine** files
from `raw/branch/main` and the hub vouches a sha for **one**; E-2a's wrapper is installed 0755 to
`/usr/local/sbin`, root-fenced in sudoers, validated only by `bash -n`.
## Record
- `OPEN-ITEMS.md` — **R-111 opened** (READY (M), P1). **E-2d re-stated**, not closed: the attempt, the
blocker, the C1/C2-vs-C3/C4/C5 split, and every Phase 0 answer so a resumed run does not re-derive them.
- `ROADMAP.md` — R-111 filed under **P1 — closed-alpha blockers**.
- `architecture/00-capability-map.md` — **not touched.** Nothing was proven live; no row qualifies.
- `OPEN-ITEMS.md` — **R-112/R-113/R-114 opened**; E-2d re-stated with results and left open for the
residue; E-2's "NOT yet live-proven" list resolved into proven / known-broken; R-94 fully unblocked;
R-110 extended; a WATCHING row for the drill cleanup.
- `ROADMAP.md` — R-112/R-113/R-114 under P1; R-111 marked SHIPPED.
- **`architecture/00-capability-map.md` not touched** — for two reasons: the customer-facing legs are
broken rather than proven, and the map has **no E-2 / backup-target rows at all** (worth noting
against the ROADMAP's coupling rule).
## Not done
## Teardown
C1–C5 all unproven. No VM, no ISO boot, no appliance registered/bound/discarded, no storage created or
modified on demo-hp, no drive attached or detached, no hub setting written (the floor was read only),
no code written.
VM destroyed, scratch storage removed, **`pvesm status` after == before** (`local-lvm` 38.77 %,
byte-identical), guest 9201 and drill-r50 untouched. **Hub records for `e2d-fresh` remain** — the
delete was attempted and correctly refused at four gates, finally `host … is ONLINE`; it becomes
deletable once the destroyed host ages to DOWN (>1 h). Command recorded in `OPEN-ITEMS.md`. The stale
2026-07-25 appliance (`206c8838…`) was left alone — not mine.
The runbook's **§5.1a operator STOP is retired**: `HUB_PW` is in `~/.config/credentials` and hub auth
was verified working, so CC can perform the bind itself on a resumed run.
## One human step, and a premise correction
The runbook's §5.1a operator STOP (the bind) is **retired** — CC did it. But E-2d's premise that a
fresh install yields a CC-drivable claimable customer is **wrong**: the claim code is bcrypt-hashed and
email-only, and the gate covers everything except `/claim`, `/api/health`, `/static/`. One operator
relay of the emailed code was required — which also proved the claim flow end to end.
documentation/audits/E2D-fresh-vm-2026-07-29.md
+224 -164
View File
@@ -1,198 +1,258 @@
# E2D-fresh-vm-2026-07-29 — Phase 0 STOP: the Day-0 artifact channel cannot deliver the code under test
# E2D-fresh-vm-2026-07-29 — E-2 proven on a fresh box: C1/C2 pass, C3/C4 partial, C5 FAILS
**Run:** `RUNBOOK-e2d-fresh-vm-2026-07-29.md`, executed by CC on DooPlex, 2026-07-29.
**Outcome:** **STOPPED at Phase 0, before any VM was created.** No VM provisioned, no install run, no
box touched, no teardown required. Per §3: *"If any gate fails, STOP and report — do not adapt around it."*
**Preceded by:** a Phase 0 STOP earlier the same day (R-111 — the Day-0 channel was 17 agent releases
stale). R-111 was fixed first; this run then proceeded on the real customer path.
**The finding in one sentence:** a box installed today through the real customer chain receives
**agent 0.96.0** and **controller 0.161.0**, because those are the newest artifacts ever published to
the Day-0 channel — so three of the five claims (C3, C4, C5) test endpoints and events that do not
exist in the software a fresh box actually runs.
**Headline:** the installer and its Case B are proven on a real install. **The two customer-facing
halves of E-2 are not reachable by a customer at all**, and the drive-absent alarm **cannot fire on
device loss**. Both were invisible to a green unit suite and to an API-level check; only the live run
found them.
---
## 1. Baselines confirmed
| Artifact | Runbook §1 | Confirmed | Source |
|---|---|---|---|
| hub | 0.81.0 | **0.81.0** | `manifests/hub.yaml:128`; `hub/CHANGELOG.md:1`; live deploy image |
| agent | 0.113.0 | **0.113.0** on `main` @ `58b598b` | `felhom-agent` HEAD; live `felhom-agent -version` on demo-hp |
| controller | 0.185.1 | **0.185.1** on `main` @ `cdaeb36` | `felhom-controller` HEAD; live image on guest 9201 |
| host-install | 1.22.0 | **1.22.0** | `scripts/felhom-host-install.sh:187` |
| felhom.eu | `91a1dad` | **`91a1dad`**, clean, == `origin/main` | `git rev-parse` |
These are the versions **on `main` and on the demo boxes**. They are not the versions a fresh install
receives — that distinction is the whole finding.
---
## 2. THE BLOCKER — the Day-0 artifact channel is 17 agent releases stale
### 2.1 The chain, read at source
`felhom-host-install.sh` does not use `main`. It resolves a hub-vouched manifest and fetches versioned
packages from Gitea:
- `resolve_artifacts()` (`:423-436`) → `GET $HUB_URL/api/v1/artifacts/$CUSTOMER_ID` → `agent.version`,
`golden.version` (served by `hub/internal/api/handler.go:2120` `handleArtifactManifest`).
- agent binary ← `$GITEA_BASE/api/packages/admin/generic/felhom-agent/$ART_AGENT_VER/felhom-agent` (`:1945`)
- golden ← `$GITEA_BASE/api/packages/admin/generic/felhom-golden/$ART_GOLDEN_VER/golden.tar.zst` (`:2573`)
### 2.2 What that channel actually holds (Gitea API, authenticated, 2026-07-29)
```
felhom-agent : newest = 0.96.0
all = 0.79.0 0.80.0 0.81.0 0.84.0 0.85.0 0.86.0 0.87.0 0.88.0 0.89.0
0.90.0 0.91.0 0.91.1 0.91.2 0.92.0 0.92.1 0.93.0 0.96.0
felhom-golden: newest = 0.161.0
all = 0.136.0 0.143.0 0.146.0 0.153.0 0.161.0
```
Hub's saved Day-0 manifest (`/configuration`, selected options): **agent `0.96.0`**
(sha `af938601…`), **golden `0.161.0`** (sha `77624408…`), min-agent `0.93.0`.
Hub global controller floor: **v0.156.0** (DB override; env fallback v0.120.0).
### 2.3 Therefore a fresh box lands on
| Component | Fresh install gets | `main` / demo boxes | Gap |
|---|---|---|---|
| host-install | **1.22.0** (website git-sync from `main`) | 1.22.0 | none |
| agent | **0.96.0** | 0.113.0 | **17 releases** |
| controller | **0.161.0** (golden bake; floor 0.156.0 < 0.161.0 ⇒ no self-update) | 0.185.1 | **24 releases** |
**Agent 0.113.0 is not in the channel at all** — it reached both demo boxes by direct deploy, never
through publish. Confirmed live: demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0.
### 2.4 What that strands — 17 unpublished agent releases (`felhom-agent/CHANGELOG.md`)
| Version | What it carries |
| Claim | Verdict |
|---|---|
| v0.97.0–v0.104.0 | **the entire R-82 per-target backup-tier arc** (local daily + PBS weekly), incl. v0.98.0's 30-minute false-failure bound, v0.101.0/0.102.0 scratch-guest + defer fixes, v0.103.0 R-84, v0.104.0 R-85 restore-test |
| v0.105.0 | R-88 Part 2 — the agent can say `unknown` |
| v0.106.0 | **F-CRIT-2** — a failed backup must not look like a fresh one (the phantom snapshot that reset the freshness clock, 7 days silent) |
| v0.107.0 | **F-REBOOT** — a guest rebooted during its backup never comes back |
| v0.108.0, v0.110.0 | F-LEAK — scratch-guest VMID band leak |
| v0.109.0 | F-OBS — the guest-power watchdog's positive observable |
| v0.111.0 | E-2c — the backup-target drive can no longer be ejected out from under the backup |
| v0.112.0 | **E-2b** — `GET /disks` flags the backup-target drive |
| v0.113.0 | **E-2a** — the guarded wrapper + `POST /backup/target` |
**A new customer box installed today therefore runs an agent that predates the whole tiered-backup
model and lacks F-CRIT-2 and F-REBOOT** — two customer-impacting silent-failure fixes. That is a
larger finding than E-2d itself and is filed as **R-111**.
| **C1** host-install 1.22.0 completes a real install, rc=0 | ✅ **PROVEN** |
| **C2** Case B fires naturally on a single-drive box | ✅ **PROVEN** |
| **C3** the degraded banner renders **to a customer** | ⚠️ **PARTIAL — API exact, NO UI CONSUMER** (R-112) |
| **C4** the offer appears and moves the target when accepted | ⚠️ **PARTIAL — full API flow proven; offer equally invisible** (R-112) |
| **C5** `backup_target_absent` fires end to end | ❌ **FAILED** — no event on any channel (R-113) |
---
## 3. Claim-by-claim impact, each pinned to its introducing commit
## 1. Baselines as actually confirmed
| Claim | Needs | Introduced in | Fresh box has | Verdict |
|---|---|---|---|---|
| **C1** rc=0 real install, banner names 1.22.0 | host-install 1.22.0 | website ← `main` | **1.22.0** | **ACHIEVABLE** |
| **C2** Case B DEGRADED lines + `resolved local` | host-install `configure_backup_target()` `:627`, Case B `:653-655` | host-install 1.22.0 | **1.22.0** | **ACHIEVABLE** — installer-side only |
| **C3** degraded banner via `GET /api/storage/backup-target` | controller **v0.185.1** (`cdaeb36`); copy „lemezhiba ellen nem" **v0.185.0** (`3f7cf2a`) | — | controller 0.161.0 | **BLOCKED** — endpoint and copy do not exist |
| **C4** offer → `POST …/assign` → `restart_required` | controller **v0.185.1** (`cdaeb36`) + agent **v0.113.0** `POST /backup/target` (`58b598b`) | — | 0.161.0 / 0.96.0 | **BLOCKED** |
| **C5** `backup_target_absent` / `_restored` | controller **v0.184.0** (`c1a63de`) + agent **v0.112.0** | — | 0.161.0 / 0.96.0 | **BLOCKED** |
**C3 has a cheap unblock; C4 and C5 do not.** C3 needs only the hub's global controller floor raised
0.156.0 → ≥0.185.0, because controller **0.185.1 IS published** to the registry (60 tags, newest
0.185.1) — a fresh box would self-update on its first report. C4/C5 need agent 0.112.0/0.113.0
*published as Gitea generic packages*, which §0 of the runbook explicitly forbids this run from doing
(*"Does not touch: … any published artifact"*).
Fleet impact of raising the floor, measured rather than assumed (hub dashboard, 2026-07-29):
| Customer | Controller | Effect of floor → 0.185.1 |
| Artifact | Confirmed | Source |
|---|---|---|
| Demo Ügyfél | 0.185.1 | none — already at it |
| Demo HP | 0.185.1 | none — already at it |
| Peti Proxmox | 0.115.0, **DOWN 14d** | already below the current 0.156.0 floor, so it updates on return either way; only the target version changes |
| hub | 0.81.0 | `manifests/hub.yaml:128`; live deploy image |
| agent | 0.113.0 | `main` @ `58b598b`; **published this run**, sha `5f3247f7…` |
| golden | **0.185.1** | baked this run, sha `dba00f3e…`, embeds controller 0.185.1 |
| controller | 0.185.1 | `main` @ `cdaeb36`; live in the guest |
| host-install | 1.22.0 | `scripts/felhom-host-install.sh:187`; fetched from the website at run time |
| felhom.eu | `3dff357` | `main` HEAD at run time |
---
**Route: ISO/PAIRING (the real customer chain).** ISO
`felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso`. Every C1–C5 result is on the real route; no
manual-installer fallback was used.
## 4. Phase 0 gate answers (all four completed before the stop)
**Operator STOP: not required and now retired.** `HUB_PW` is in `~/.config/credentials`; CC created
the customer and performed the bind itself. The one human step that *was* needed is new — see §6.
### 4.1 Storage placement — gate PASSES, fence confirmed
## 2. Timeline (VM 9300 `e2d-fresh` on demo-hp, nested PVE)
`pvesm status` on demo-hp, 2026-07-29 (BEFORE; there is no AFTER — no VM was created):
| UTC | Event |
|---|---|
| 10:29 | VM created — q35/OVMF SB-off, 4c/8G, **one** 160 G disk, `hotplug disk`, outside the `felhom` pool |
| 10:34:52 | PVE auto-install done, first boot, DHCP `192.168.0.125` (no R-59 gate trip) |
| 10:34:53 | registered as unclaimed appliance, pairing code `SB4-7ZK`, console banner rendered |
| 10:36:52 | **bound** to customer `e2d-fresh` by CC; credentials delivered 2 s later |
| 10:37:31 | host enrolled `e2d-fresh-ac9f09`; break-glass root credential vaulted |
| 10:37:33 | **artifact manifest served: agent=0.113.0 golden=0.185.1** |
| 10:40:27 | `controller_started (0.185.1)` — **bind → running controller in 3 m 35 s** |
| 10:56 | second 100 G disk hot-attached; wizard init→mount→register |
| 10:57:08 | offer accepted → `restart_required:true`; agent restarted at 10:57:36 |
| 10:58:37 | target drive **hot-detached** (volume survives as `unused0`) |
| 10:58:37–11:03 | **no event on any channel** for 4½ minutes (budget was 60 s) |
| 11:03:31 | reattached; state returns healthy; **still no event** |
## 3. C1 — PROVEN
From the `felhom-bootstrap.service` journal on the box:
```
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
felhom-backup dir active 983379700 2293856 931059232 0.23%
felhom-pbs pbs active 0 0 0 0.00%
local dir active 40516856 14721460 23705004 36.33%
local-lvm lvmthin active 56545280 21922605 34622674 38.77%
felhom-bootstrap: fetching host-install: https://felhom.eu/scripts/felhom-host-install.sh
[INFO] felhom-host-install v1.22.0 — mode=appliance customer=e2d-fresh vmid=9201
...
[OK] controller: Up 21 seconds (healthy)
[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.185.1
[OK] Day-0 provision SUCCESS — vmid=9201 host_id=e2d-fresh-ac9f09 customer=e2d-fresh
golden=local:backup/vzdump-lxc-9100-2026_07_29-12_37_56.tar.zst
felhom-bootstrap: host-install SUCCESS — writing done-flag, disabling unit, scrubbing secrets
```
`local-lvm` is thin, pool `<53.93g`, **data 38.77 %** — matching the runbook's "~54 GB pool, 38.8 %"
to 0.03 pp, so the picture has **not** materially changed. Allocated LVs on it: `vm-9201-disk-{0,1,2}`
(32+50+20 G, the **live customer guest**), `vm-300-disk-{0,1}` (drill-r50, 32 G) and its `r50pre`
snapshots (32 G) ⇒ ~166 G allocated against 53.93 G real. `vgs` shows only 14.75 G VFree.
**The fence holds: nothing goes on `local-lvm`.**
rc=0 is corroborated structurally: the unit wrote its done-flag, **self-disabled**, and
`Deactivated successfully`. `pct list` showed guest **9201 `e2d-fresh` running**.
`/mnt/nvme-1tb` = `/dev/nvme0n1`, ext4, 938 G, **888 G free**. ≥100 G requirement satisfied.
**The publish train is proven end to end:** the golden restored is
`vzdump-lxc-9100-2026_07_29-12_37_56` — the golden baked ~20 minutes earlier in the same session.
This retires E-2's *"installer-logic-tested, not install-tested"*.
**Placement decision (recorded deliberately, per §3.1):** the only storage with `content images` is
`local-lvm` (forbidden). `felhom-backup` is a `dir` at exactly `/mnt/nvme-1tb` but is `content backup`
only **and is demo-hp's live backup target** — widening its content set would mutate the very storage
E-2c/E-2 role logic keys on, which §0/§9 forbid. So a *new* dir storage would have been required.
Both remaining shapes carry a cost and the choice was **not** forced this run because the run stopped:
a second storage at the same mountpoint risks perturbing the agent's role resolution on a live box;
a storage at a **subdirectory** fails `exactMount` and reports `disconnected` in the host report,
which is not purely cosmetic since it can reach the hub's storage monitor. **This decision is left
open and is an input to any resumed run.**
## 4. C2 — PROVEN
### 4.2 Drive-gate cadence — answered
Both required warning lines, verbatim, ANSI-stripped:
- Symbol: `driveGateLoop` → `s.ReconcileDriveGates()`
- Definition: `controller/internal/web/intermediary.go:328` (gate itself at `:267`)
- Registration: `controller/internal/web/server.go:217` — `go s.driveGateLoop()`
- **Interval: `time.NewTicker(30 * time.Second)`** (`intermediary.go:337`)
```
[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive.
[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard.
```
⇒ Stage 7's derived budget would be **two cycles = 60 s** of controller-side reconcile, *plus* the
agent's own `/disks` refresh, since `ReconcileDriveGates` consumes `resp.Disks` (`:281`). Recorded so
a resumed run does not have to re-derive it.
- `agent.json` → `backup.local_backup_target = 'local'` (the "resolved local" observable)
- no `felhom-backup` storage created at install
- **the install did not abort** — a single-drive appliance is a valid product
### 4.3 Hot-detach — feasible, not exercised
## 5. THE FINDINGS
demo-hp: AMD `svm` present, `/sys/module/kvm_amd/parameters/nested = 1`, 8 cores, 29 GB RAM (25 GB
available). The working nested-PVE reference is VM 300 (`drill-r50`): `bios: ovmf`,
`efitype=4m,pre-enrolled-keys=0` (Secure Boot OFF, as the mkimage loader requires), `machine: q35`,
`scsihw: virtio-scsi-single`, `cpu: host`. `virtio-scsi-single` + PVE's default
`hotplug: disk,network,usb` supports SCSI hot-detach, so the **live-transition** variant of Stage 7
was available. Not exercised.
### 5.1 R-112 — the degraded banner and the offer have NO UI consumer (customer-invisible)
### 4.4 Install route — ISO available, route selected, not used
The endpoint is **correct**. Authenticated `GET /api/storage/backup-target` returned, exactly:
Two generic (pairing) ISOs are present on demo-hp `local`:
`felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso` (**preferred**, the v1.25.0 train, mkimage
loader for nested VMs) and `felhom-pve-9.2-1-v1.22.0-nested-canary-generic.iso`.
The **ISO/PAIRING route was selected**. It reaches the same installer invocation — `run_pairing`
(`felhom-bootstrap.sh:441`) falls through to `run_direct` (`:495-499`) in the same invocation, which
fetches `$INSTALL_URL` (`:322-330`) and runs it (`:343`). Not used — the run stopped first.
```json
{"degraded":true,"known":true,"target":"local",
"message":"A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer — így hibás fájlok
ellen véd, lemezhiba ellen nem. Csatlakoztass egy második meghajtót a teljes védelemhez."}
```
**Operator STOP: not required.** `HUB_PW` is present in `~/.config/credentials` and hub auth was
verified working (`http://10.43.52.34:8080/` → 200 with `curl -u ":$HUB_PW"`; public ingress also 200),
so CC could have performed the bind itself. The runbook's §5.1a STOP is retired for future runs.
byte-identical to the runbook's required copy. `known` is a field **separate** from `degraded`, so
UNKNOWN genuinely cannot render as degraded (R-88 Part 2's lesson held).
---
**And nothing in the product ever asks for it.** Negative claims with their search scope:
## 5. Incidental observations (filed, not acted on)
| Search | Result |
|---|---|
| `grep -rn 'backup-target' --include='*.html' --include='*.js' --include='*.css' controller/` | **0 hits** |
| `grep -rn 'kijelölheted\|ugyanazon a lemezen\|OfferPath\|OfferLabel\|Degraded' controller/internal/web/templates/` | **0 hits** |
| consumers of `backupTargetDegradedText` / `backupTargetOfferText` | only `degradedMessageFor` (`:136`) and the JSON handler (`:153`) — both inside `backup_target_offer.go` |
| consumers of `resolveBackupTargetState` / `degradedMessageFor` across all Go | **only** the API handler. **No page handler injects the state.** |
1. **A stale unclaimed appliance is sitting in the hub**: `206c8838-7755-4751-8a16-a842853d718f`,
pairing code `QWA-WJE`, MAC `bc:24:11:ea:55:d0`, "Standard PC (Q35 + ICH9, 2009)" on an
AMD V1756B with 7.7 GB — i.e. a **nested VM on demo-hp**, first and last seen **5 d ago**. It is a
leftover from the 2026-07-25 ISO work and has never been bound or discarded. A resumed run must
distinguish its own appliance from this one, or discard it first.
2. **The agent has no container image** (`/v2/admin/felhom-agent/tags/list` → 0 tags). It ships only
as a Gitea generic package, which is why the publish gap is invisible from the registry.
3. `drill-r50` (VM 300) still holds a `r50pre` snapshot pair consuming ~32 G of allocation on the
over-subscribed `local-lvm`. Untouched per §9; noted because it is part of why the pool is tight.
**The decisive contrast:** the templates fetch **18 distinct `/api/storage/*` endpoints`**.
`backup-target` and `backup-target/assign` are the **only two** referenced by zero templates.
---
The handler's own doc comment reads *"serves GET /api/storage/backup-target — **the dashboard's source
for the degraded banner and the offer**"* — an invariant comment asserting a consumer that does not
exist (`CLAUDE.md`'s "a comment asserting an invariant needs a test pinning it, or it is a wish",
instance #7). And v0.185.1's own test, `TestBackupTargetRoutesLiveUnderTheStorageAPIMount`, pins that
the **router dispatches** the path — not that anything renders it. v0.185.1 shipped as *"the offer
endpoints were mounted where nothing routed to them"*: it fixed the mount and stopped one layer short.
## 6. What did not happen
**Fifth instance of the seam-built-but-never-wired class.** `CLAUDE.md`'s seam rule names exactly this:
*"a feature is not shipped until its entry point is reachable… handler tests that POST directly prove
nothing about reachability."*
No VM was created. No ISO booted. No install ran. No appliance registered, bound or discarded. No
storage was created or modified on demo-hp. No drive attached or detached. No hub setting changed
(the floor was **read**, not written). No teardown was needed. **C1–C5 are all UNPROVEN**, and the
E-2 / E-2d rows are unchanged except to record this blocker.
### 5.2 R-113 — the drive-absent gate CANNOT fire on device loss (E-2b's alarm is unreachable)
Detached the assigned target drive at 10:58:37Z under a running agent. Over the next 4½ minutes:
- **agent**, every 20 s: `storage: enrolled drive absent by UUID — not re-asserting` /
`reconcile: enrolled drive not present (durable-id absent) — skipping` — **the agent knows**
- **controller**: `docker logs | grep -c '\[gate\]'` → **0**. The gate never acted, ABSENT or RETURNED
- **hub**: zero events for the customer across the whole window — no `backup_target_absent`,
**and no generic `storage_disconnected` either**
**Root cause, established at source and confirmed live.** `planDriveGates`
(`intermediary.go:216-262`) computes presence as
`present[GuestPath] = present[GuestPath] || d.BoundUnderParent`, and the agent derives
`BoundUnderParent` from `GuestSeesMount()` — *"does the guest's `/proc/<pid>/mountinfo` list this path
as a mount target"* (`localapi/disks.go:210`, `localapi/intermediary.go` `GuestSeesMount`).
Measured on the box with the device removed:
```
-- raw mount /mnt/mentes2 -- NOT mounted <- systemd device-bound unit, died with the device
-- stable bind /mnt/felhom-drives/mentes2 -- /dev/sdb[/felhom-data] ext4 <- the agent's MANUAL bind, SURVIVES
```
The raw mount is a device-bound systemd unit and dies correctly; **the agent's own bind under the
shared parent is not device-bound, so its mountinfo entry outlives the device.** The gate reads that
surviving entry as "present" ⇒ `!present[...]` is never true ⇒ no `Stop` action ⇒ `notifyDriveAbsent`
is never called.
This is not a virtualisation artefact: the asymmetry is between a device-bound mount and a manual
bind, which is identical on physical hardware. **Caveat kept honest:** proven on a SCSI hot-detach; a
physical USB unplug was not staged.
**Consequence.** E-2b's celebrated fix — *"THE SEAM THAT WAS NEVER WIRED… a drive that is ONLY a
backup target has no apps to stop, so it was silent twice over"* (`intermediary.go:288-296`) — wired
the notify to a branch that **cannot execute on device loss**. The seam is wired; the condition is
unreachable. Sixth instance of the class, one layer deeper than the fifth.
**Mirror scenario:** not separately staged, and it does not need to be — both the specific and the
generic event are emitted from the same `a.Stop` branch, which never executed. The generic
`storage_disconnected` is equally unreachable by this path. Recorded as *reasoned, not observed*.
### 5.3 R-114 — on target-drive loss the customer is told the wrong story and offered the missing drive
While the target drive was absent, the endpoint returned:
```json
{"degraded":true,"target":"felhom-backup",
"message":"A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer …",
"offer_path":"/mnt/felhom-drives/mentes2","offer_label":"Mentés meghajtó"}
```
Two defects in one payload: the message claims the backup is **on the system disk**, which is false —
the target is `felhom-backup` on a drive that has vanished; and the remedy offered is **the drive that
just disappeared**. `resolveBackupTargetState` falls through to the generic degraded branch whenever no
disk satisfies `d.BackupTarget && d.MountPath != ""`, without distinguishing *never configured* from
*configured and now missing*.
**Interaction worth stating: R-114 is currently invisible only because of R-112.** Fixing R-112 alone —
wiring the banner — would immediately start showing customers this wrong message. **They must be fixed
together, R-114 first.**
**Also observed:** after reattach the drive returned as `/dev/sdc`, while the stable bind still
recorded `/dev/sdb[/felhom-data]`. The state read healthy (`degraded:false`) with the guest-visible
bind still naming the dead device node. Not chased further; recorded as part of R-113's shape.
### 5.4 Smaller findings (recorded, not filed as their own IDs)
1. **A "hard min" that only warns.** `[WARN] local-lvm free ~83 GiB < hard min 120 GiB` — the installer
names a hard minimum and proceeds. Either it is not hard, or the wording is wrong.
2. **`felhom-backup-target-apply` is fetched unvouched.** host-install pulls **nine** files from
`raw/branch/main` (`:2072`–`:2206`); the hub manifest vouches a sha for exactly one
(`wrapper_sha256` → `felhom-pbs-apply`, verified this run: no drift). E-2a's wrapper is installed
**0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n`.
**Filed as a second instance under R-110**, not a new ID — same class (a root-executed artifact
taken from `main` with no pinned integrity).
## 6. C4 — what IS proven (API level)
Everything except customer reachability:
- **The offer** appeared with all three fields: `offer_path=/mnt/felhom-drives/mentes2`,
`offer_label=Mentés meghajtó`, `offer_message=Ezt a meghajtót kijelölheted a rendszermentés helyéül…`
- **The decline path (§6.4) — PROVEN.** After registering the drive and *not* accepting:
`target` still `local`, no `felhom-backup` storage, `agent.json` unchanged. **Registration does not
confer a role** — E-2 §3's central invariant, live.
- **Accept** → `{"assigned":"/mnt/felhom-drives/mentes2","restart_required":true}`
- **The agent did NOT self-restart** — `ActiveEnterTimestamp` unchanged at 12:37:53 CEST twenty
minutes later. **In-flight check performed and recorded before restarting:** 0 running PVE tasks,
no `vzdump` process, 0 backup lines in the agent journal.
- **The E-2a root-fenced wrapper worked on a fresh box:** created `dir: felhom-backup`,
`path /mnt/mentes2`, `is_mountpoint 1` — the drive's **own** mountpoint (law F-2).
- **Healthy renders nothing:** after the restart the payload is
`{"degraded":false,"known":true,"label":"Mentés meghajtó","target":"felhom-backup"}` — **no
`message` field at all**. No "backup protected" reassurance (E-2 Scenario E).
## 7. E-2d's own premise needs amending
The runbook assumed a fresh install yields a claimable customer CC can then drive. It does not:
**the claim code is bcrypt-hashed in the hub and only ever emailed** (`claim/engine.go:58-95`), and the
gate covers everything except `/claim`, `/claim/request-new-code`, `/api/health`, `/static/`
(`claim.go:222-229`). C3/C4/C5 all sit behind it. This run cleared it by registering an operator email,
resending, and having the operator relay the code — **the one genuine human step**, and it also proved
the claim flow end to end (code → password → `401 "dashboard not yet claimed"` becoming
`401 "authentication required"`, a positive discriminator).
## 8. Teardown
- VM 9300 destroyed `--purge`; `e2d-images` storage removed; scratch dir removed.
- **`pvesm status` after == before**: `local-lvm` **38.77 %**, `felhom-backup` 931059224 KiB available
— byte-identical to the pre-run measurement. Freed space returned.
- **Guest 9201 (live customer) untouched and running; `drill-r50` VM 300 untouched.**
- Drill VM (golden bake) torn down per GL-1 earlier: guest 9100 purged, secrets shredded, disk restored
to `virgin`, token-leak grep 0.
**Hub records NOT yet removed — stated, not silent.** `e2d-fresh` + host `e2d-fresh-ac9f09` remain.
The delete was attempted and **correctly refused** at four successive gates: acknowledgements → typed
`confirm_id` → `expect_hosts` stale-preview → finally `host e2d-fresh-ac9f09 is ONLINE`. The host is
online only because its last report is recent; the VM is destroyed, so it ages OK → WARN (30 m) → DOWN
(>1 h) and is then deletable. **Cleanup command, once it reads DOWN:**
```
POST /configs/e2d-fresh/delete ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=e2d-fresh expect_hosts=1
```
Tracked in `OPEN-ITEMS.md`. Deleting rather than keeping is deliberate — R-93 records what a
half-real fixture costs.
**Also still present:** the stale unclaimed appliance from 2026-07-25 (`206c8838…`, code `QWA-WJE`).
Not mine; not discarded. A future run must distinguish its own appliance from it.
documentation/backlog/OPEN-ITEMS.md
+8 -4
View File
File diff suppressed because one or more lines are too long
documentation/backlog/ROADMAP.md
+4 -1
View File
@@ -20,7 +20,10 @@
| ID | Item | Size | Status | Notes / map rows flipped |
|----|------|------|--------|--------------------------|
| R-111 | **The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`** | M | idea — found 2026-07-29 by the E-2d Phase 0 gate | **The fleet's live versions are not the fleet's INSTALLABLE versions, and only the first were ever checked.** `felhom-host-install.sh` does not use `main`: `resolve_artifacts()` (`:423-436`) reads the hub-vouched manifest (`GET /api/v1/artifacts/<customer>`, `hub/internal/api/handler.go:2120`) and fetches versioned **Gitea generic packages** — agent from `:1945`, golden from `:2573`. Gitea holds **`felhom-agent` newest `0.96.0`** and **`felhom-golden` newest `0.161.0`**; the hub's saved manifest selects exactly those. So a fresh box lands on **agent 0.96.0 + controller 0.161.0** (golden bake; the global floor is `v0.156.0` < 0.161.0, so it does not self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and is not in the channel at all** — demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. **17 unpublished releases (`felhom-agent/CHANGELOG.md` v0.97.0–v0.113.0)**, including the ENTIRE R-82 per-target backup-tier arc (v0.97.0–v0.104.0), **F-CRIT-2** (v0.106.0 — a failed backup looking fresh, 7 days silent), **F-REBOOT** (v0.107.0 — a guest rebooted mid-backup never returns), F-LEAK (v0.108.0/0.110.0), F-OBS (v0.109.0), E-2c (v0.111.0), E-2b (v0.112.0), E-2a (v0.113.0). **P1 because it gates the first remote tester:** their box would install an agent predating the tiered-backup model and both silent-failure fixes. **Mirror of R-110, not a duplicate:** R-110 is *the installer publishes instantly with no staging*; this is *the agent and golden have a deliberate publish+vouch gate and it was never walked* — opposite failure modes of one subject, different fixes. Contrast worth keeping: the **controller** is fine (registry has 0.185.1; it self-updates from the floor), so the gap is specific to the two Gitea-generic artifacts. **Decide as part of the fix:** whether publishing becomes part of the release train rather than a separate remembered step — this is R-29's shape (a gate that exists and is never walked) one layer up. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §2 |
| R-113 | **The drive-absent gate cannot fire on device loss — E-2b's alarm is wired to an unreachable condition** | M | idea — **PROVEN LIVE 2026-07-29** | `planDriveGates` (`felhom-controller/internal/web/intermediary.go:216-262`) treats a path as present by OR-ing in `d.BoundUnderParent`, which the agent derives from `GuestSeesMount()` — *"is this path a mount target in the guest's `/proc/<pid>/mountinfo`"* (`internal/localapi/disks.go:210`). The raw drive mount is a **device-bound systemd unit** and dies with the device; **the agent's own bind under the shared parent is not device-bound and its mountinfo entry outlives the device**, so the gate reads it as present and `notifyDriveAbsent` is never called. Live on a fresh box: target drive hot-detached, agent said `enrolled drive absent by UUID` every 20 s for 4½ min, controller logged **0** `[gate]` lines, hub received **zero** events — neither `backup_target_absent` nor the generic `storage_disconnected`. Not a virtualisation artefact (device-bound-mount vs manual-bind is the same on metal); caveat: SCSI hot-detach, physical unplug not staged. **Sixth instance of seam-built-but-never-wired — E-2b wired the seam to a condition that cannot occur.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.2 |
| R-112 | **E-2's degraded banner and offer have no UI consumer — correct endpoint, invisible to the customer** | S | idea — **PROVEN LIVE 2026-07-29** | `GET /api/storage/backup-target` returns byte-exact Hungarian copy (verified on a live box), and nothing in the product asks for it: `grep 'backup-target'` across every `*.html`/`*.js`/`*.css` → **0 hits**; no template references `OfferPath`/`Degraded`/the copy; `resolveBackupTargetState` and `degradedMessageFor` are consumed **only** by the JSON handler, with **no page handler injecting the state**. Decisive contrast: the templates fetch **18 distinct `/api/storage/*` endpoints** — `backup-target` and `backup-target/assign` are the only two with zero references. The handler's own comment calls itself *"the dashboard's source for the degraded banner and the offer"*. v0.185.1 shipped as *"the offer endpoints were mounted where nothing routed to them"* and fixed the **mount**, stopping one layer short of the **render**; its test pins dispatch, not reachability. **Fifth instance of the class. Fix R-114 first** — wiring this alone starts showing customers a wrong message. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.1 |
| R-114 | **On target-drive loss the customer is told the wrong story and offered the drive that vanished** | S | idea — **PROVEN LIVE 2026-07-29** | With the assigned target absent the endpoint returned `degraded:true, target:"felhom-backup"` **plus** the *"ugyanazon a lemezen van, mint a rendszer"* message — false, the target is a missing drive, not the system disk — **and** an `offer_path` pointing at the drive that just disappeared. `resolveBackupTargetState` falls through to the generic degraded branch whenever no disk satisfies `d.BackupTarget && d.MountPath != ""`, never distinguishing *never configured* from *configured and now missing*. Shares R-113's root cause (two disagreeing presence signals), different code path and fix. **Invisible today only because of R-112.** Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §5.3 |
| R-111 | ~~**The Day-0 artifact channel is 17 agent releases stale — a box installed today gets agent `0.96.0`, not `0.113.0`**~~ | M | **SHIPPED 2026-07-29** — agent 0.113.0 published (sha `5f3247f7…`, round-trip verified) + golden **0.185.1** baked and published (sha `dba00f3e…`, embeds controller 0.185.1); hub Day-0 manifest moved both together with `min_agent` 0.93.0→0.113.0 (zero fleet impact — all three enrolled hosts already at 0.113.0). **Proven end-to-end the same day:** the E-2d fresh install fetched exactly these and restored the golden baked 20 minutes earlier. Original finding: | **The fleet's live versions are not the fleet's INSTALLABLE versions, and only the first were ever checked.** `felhom-host-install.sh` does not use `main`: `resolve_artifacts()` (`:423-436`) reads the hub-vouched manifest (`GET /api/v1/artifacts/<customer>`, `hub/internal/api/handler.go:2120`) and fetches versioned **Gitea generic packages** — agent from `:1945`, golden from `:2573`. Gitea holds **`felhom-agent` newest `0.96.0`** and **`felhom-golden` newest `0.161.0`**; the hub's saved manifest selects exactly those. So a fresh box lands on **agent 0.96.0 + controller 0.161.0** (golden bake; the global floor is `v0.156.0` < 0.161.0, so it does not self-update) against `main`'s 0.113.0 / 0.185.1. Agent 0.113.0 reached both demo boxes by **direct deploy and is not in the channel at all** — demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0. **17 unpublished releases (`felhom-agent/CHANGELOG.md` v0.97.0–v0.113.0)**, including the ENTIRE R-82 per-target backup-tier arc (v0.97.0–v0.104.0), **F-CRIT-2** (v0.106.0 — a failed backup looking fresh, 7 days silent), **F-REBOOT** (v0.107.0 — a guest rebooted mid-backup never returns), F-LEAK (v0.108.0/0.110.0), F-OBS (v0.109.0), E-2c (v0.111.0), E-2b (v0.112.0), E-2a (v0.113.0). **P1 because it gates the first remote tester:** their box would install an agent predating the tiered-backup model and both silent-failure fixes. **Mirror of R-110, not a duplicate:** R-110 is *the installer publishes instantly with no staging*; this is *the agent and golden have a deliberate publish+vouch gate and it was never walked* — opposite failure modes of one subject, different fixes. Contrast worth keeping: the **controller** is fine (registry has 0.185.1; it self-updates from the floor), so the gap is specific to the two Gitea-generic artifacts. **Decide as part of the fix:** whether publishing becomes part of the release train rather than a separate remembered step — this is R-29's shape (a gate that exists and is never walked) one layer up. Evidence: `audits/E2D-fresh-vm-2026-07-29.md` §2 |
| R-1 | **Peti convergence** — *the appliance half is DONE; this item is now Peti-only.* **Rehearsal EXECUTED 2026-07-18** (`tests/VALIDATION-n100-rehearsal-2026-07-18.md`): the full final-product flow ran on real metal in one pass (RESET → generic ISO → **customer self-bind** → day-0 → floor lift → escrow ceremony → offsite snapshots), which retires the "supervised rehearsal" dependency that R-13/R-21/R-23/R-24/R-27/R-28 were all parked behind. **Surviving half: Peti's clean-slate proxmox2 reinstall + the parked publish trains on a REAL REMOTE customer** — the one thing a demo box on the operator's own LAN can never prove. | L | **rehearsal DONE; Peti half open** | Flips: publish train PARTIAL→PROVEN-LIVE; appliance/BYO/day-0 "real customer" notes; escrow ceremony. The single biggest unproven surface — an alpha where fixes can't ship remotely is dead. **Reinstall arc SHIPPED hub v0.57.0 (2026-07-16):** the clean-slate reinstall-of-existing-customer path is now first-class — claim re-issue (F2), offsite re-issue (F3), escrow-honesty-on-re-issue (2.3) all auto-fire on re-enrollment. Peti's proxmox2 clean-slate now walks a supported path |
| R-2 | ~~Resolve ~215 lines of foreign WIP in felhom.eu clone (`hub/internal/notify/`, `store.go`, `hub/internal/claim/`)~~ | S | **killed** (2026-07-16) | Not a real issue: the "foreign WIP" was in-flight code from a concurrent CC session on the customer-claim arc, snapshotted before it committed. All of it landed cleanly — `notify/`+`claim/engine.go` in `6b40eb8` (v0.50.0), `store.go` in `a1d0450` (v0.54.0), plus follow-up `e205a2d`; v0.55.0 shipped. Working tree is clean, no stashes. Lesson already codified: never run two writing sessions on one felhom.eu clone (CLAUDE.md §git add -A) |
| R-3 | Friend-alpha onboarding runbook (generalized from `pilot/RUNBOOK-peti-return-2026-07-13`): hardware prep → golden → install → claim → ceremony → "first restore by the customer" scripted step | M | idea | Flips: "customer performs a restore" MISSING row; produces the tester-agreement sibling of `PETI-tester-agreement.md`. **Next from-scratch rehearsal to include customer DELETE + re-create** — the ESCROW cascade is now DEFINED (hub v0.60.1): host delete DEMOTES escrow to retained custody (never destroys), customer Danger-zone delete PURGES it (the one true purge point). **S6b (manual stale-host delete before re-enroll) is OBSOLETE** — re-enrollment upserts the existing host row cleanly (`store.UpsertHost` ON CONFLICT DO UPDATE; `handleAdminCreateHost` no duplicate refusal) + the v0.57.0 arc auto-fires the re-issues; the rehearsal live-confirms it. **NON-escrow offboarding NOW ANSWERED by the middle-tier Customer RESET (hub v0.61.0, LIVE):** one operator action deprovisions the Hetzner sub-account/box (repo data destroyed), destroys the PBS namespace + backup groups + token, clears the DR recipe / one-time secret / claim state / retained escrow custody (separate ack) — identity + basic config survive. WG peer release rides host delete (peers are host-scoped, gone before RESET runs — RESET refuses while any host row exists). **Remaining consistency gap:** the customer Danger-zone DELETE still leaves host rows and does NOT run the offsite/PBS teardown (RESET is the teardown path; DELETE is escrow-purge + config-drop). Decide whether DELETE should require a prior RESET (or subsume it) — new item R-25b |