admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

E-2d executed on a fresh box: C1/C2 proven, C3/C4 partial, C5 FAILS — R-112/113/114

Browse Source 
Full ISO/PAIRING route on a nested PVE VM on demo-hp, after R-111 was fixed
earlier in the session. Bind -> running controller in 3m35s. The install fetched
the artifacts published an hour before and restored the golden baked 20 minutes
before, so the publish train is proven end to end on a real install.

C1 PROVEN: "felhom-host-install v1.22.0", "Day-0 provision SUCCESS", guest 9201
running, bootstrap unit wrote its done-flag and self-disabled. This retires E-2's
"installer-logic-tested, not install-tested".

C2 PROVEN: both DEGRADED warning lines verbatim, backup.local_backup_target=local,
no felhom-backup storage created, and the install did not abort.

C3/C4 PARTIAL and C5 FAILED — three findings, none fixed:

R-112 (P1): E-2's degraded banner and offer have NO UI CONSUMER. The endpoint
returns byte-exact copy; grep 'backup-target' across every html/js/css is 0 hits
and no page handler injects the state. Templates fetch 18 distinct /api/storage/*
endpoints; these two are the only ones with zero references. v0.185.1 fixed the
router mount and stopped one layer short of the render. Fifth instance of
seam-built-but-never-wired.

R-113 (P1): the drive-absent gate CANNOT FIRE on device loss. planDriveGates
reads presence from BoundUnderParent = "is this path in the guest's mountinfo".
The raw mount is a device-bound systemd unit and dies with the device; the
agent's own bind is not device-bound and outlives it, so the gate sees "present"
forever. Live: agent reported the drive absent every 20s for 4.5 minutes, the
controller logged 0 [gate] lines, the hub received zero events -- neither
backup_target_absent nor the generic storage_disconnected. Sixth instance of the
class: E-2b wired the seam to a condition that cannot occur.

R-114: on target-drive loss the message claims the backup is on the system disk
(false) and offers the drive that just vanished. Invisible only because of R-112,
so it must be fixed BEFORE R-112 is wired.

Also filed as a second instance under R-110 rather than a new ID: host-install
fetches nine files from raw/branch/main and the hub vouches a sha for one;
E-2a's wrapper is installed 0755 to /usr/local/sbin, root-fenced in sudoers,
validated only by bash -n.

C4 is fully proven at API level: decline path (registration confers no role),
restart_required:true, agent did NOT self-restart (in-flight check performed and
recorded first), E-2a wrapper created the storage at the drive's own mountpoint,
and healthy renders nothing.

Teardown: VM destroyed, scratch storage removed, pvesm status after == before
(local-lvm 38.77%), guest 9201 and drill-r50 untouched. Hub records for e2d-fresh
remain -- delete correctly refused at four gates, finally "host is ONLINE";
deletable once it ages to DOWN. Command recorded in OPEN-ITEMS.md.

capability-map NOT touched: the customer-facing legs are broken rather than
proven, and the map has no E-2 rows at all.
This commit is contained in:
authentik Default Admin
2026-07-29 13:13:56 +02:00
parent 3dff3573f7
commit f3975cf5bc
4 changed files with 307 additions and 211 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/E2D-fresh-vm-2026-07-29.md
+224 -164
View File
@@ -1,198 +1,258 @@
# E2D-fresh-vm-2026-07-29 — Phase 0 STOP: the Day-0 artifact channel cannot deliver the code under test
# E2D-fresh-vm-2026-07-29 — E-2 proven on a fresh box: C1/C2 pass, C3/C4 partial, C5 FAILS
**Run:** `RUNBOOK-e2d-fresh-vm-2026-07-29.md`, executed by CC on DooPlex, 2026-07-29.
**Outcome:** **STOPPED at Phase 0, before any VM was created.** No VM provisioned, no install run, no
box touched, no teardown required. Per §3: *"If any gate fails, STOP and report — do not adapt around it."*
**Preceded by:** a Phase 0 STOP earlier the same day (R-111 — the Day-0 channel was 17 agent releases
stale). R-111 was fixed first; this run then proceeded on the real customer path.
**The finding in one sentence:** a box installed today through the real customer chain receives
**agent 0.96.0** and **controller 0.161.0**, because those are the newest artifacts ever published to
the Day-0 channel — so three of the five claims (C3, C4, C5) test endpoints and events that do not
exist in the software a fresh box actually runs.
**Headline:** the installer and its Case B are proven on a real install. **The two customer-facing
halves of E-2 are not reachable by a customer at all**, and the drive-absent alarm **cannot fire on
device loss**. Both were invisible to a green unit suite and to an API-level check; only the live run
found them.
---
## 1. Baselines confirmed
| Artifact | Runbook §1 | Confirmed | Source |
|---|---|---|---|
| hub | 0.81.0 | **0.81.0** | `manifests/hub.yaml:128`; `hub/CHANGELOG.md:1`; live deploy image |
| agent | 0.113.0 | **0.113.0** on `main` @ `58b598b` | `felhom-agent` HEAD; live `felhom-agent -version` on demo-hp |
| controller | 0.185.1 | **0.185.1** on `main` @ `cdaeb36` | `felhom-controller` HEAD; live image on guest 9201 |
| host-install | 1.22.0 | **1.22.0** | `scripts/felhom-host-install.sh:187` |
| felhom.eu | `91a1dad` | **`91a1dad`**, clean, == `origin/main` | `git rev-parse` |
These are the versions **on `main` and on the demo boxes**. They are not the versions a fresh install
receives — that distinction is the whole finding.
---
## 2. THE BLOCKER — the Day-0 artifact channel is 17 agent releases stale
### 2.1 The chain, read at source
`felhom-host-install.sh` does not use `main`. It resolves a hub-vouched manifest and fetches versioned
packages from Gitea:
- `resolve_artifacts()` (`:423-436`) → `GET $HUB_URL/api/v1/artifacts/$CUSTOMER_ID` → `agent.version`,
`golden.version` (served by `hub/internal/api/handler.go:2120` `handleArtifactManifest`).
- agent binary ← `$GITEA_BASE/api/packages/admin/generic/felhom-agent/$ART_AGENT_VER/felhom-agent` (`:1945`)
- golden ← `$GITEA_BASE/api/packages/admin/generic/felhom-golden/$ART_GOLDEN_VER/golden.tar.zst` (`:2573`)
### 2.2 What that channel actually holds (Gitea API, authenticated, 2026-07-29)
```
felhom-agent : newest = 0.96.0
all = 0.79.0 0.80.0 0.81.0 0.84.0 0.85.0 0.86.0 0.87.0 0.88.0 0.89.0
0.90.0 0.91.0 0.91.1 0.91.2 0.92.0 0.92.1 0.93.0 0.96.0
felhom-golden: newest = 0.161.0
all = 0.136.0 0.143.0 0.146.0 0.153.0 0.161.0
```
Hub's saved Day-0 manifest (`/configuration`, selected options): **agent `0.96.0`**
(sha `af938601…`), **golden `0.161.0`** (sha `77624408…`), min-agent `0.93.0`.
Hub global controller floor: **v0.156.0** (DB override; env fallback v0.120.0).
### 2.3 Therefore a fresh box lands on
| Component | Fresh install gets | `main` / demo boxes | Gap |
|---|---|---|---|
| host-install | **1.22.0** (website git-sync from `main`) | 1.22.0 | none |
| agent | **0.96.0** | 0.113.0 | **17 releases** |
| controller | **0.161.0** (golden bake; floor 0.156.0 < 0.161.0 ⇒ no self-update) | 0.185.1 | **24 releases** |
**Agent 0.113.0 is not in the channel at all** — it reached both demo boxes by direct deploy, never
through publish. Confirmed live: demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0.
### 2.4 What that strands — 17 unpublished agent releases (`felhom-agent/CHANGELOG.md`)
| Version | What it carries |
| Claim | Verdict |
|---|---|
| v0.97.0–v0.104.0 | **the entire R-82 per-target backup-tier arc** (local daily + PBS weekly), incl. v0.98.0's 30-minute false-failure bound, v0.101.0/0.102.0 scratch-guest + defer fixes, v0.103.0 R-84, v0.104.0 R-85 restore-test |
| v0.105.0 | R-88 Part 2 — the agent can say `unknown` |
| v0.106.0 | **F-CRIT-2** — a failed backup must not look like a fresh one (the phantom snapshot that reset the freshness clock, 7 days silent) |
| v0.107.0 | **F-REBOOT** — a guest rebooted during its backup never comes back |
| v0.108.0, v0.110.0 | F-LEAK — scratch-guest VMID band leak |
| v0.109.0 | F-OBS — the guest-power watchdog's positive observable |
| v0.111.0 | E-2c — the backup-target drive can no longer be ejected out from under the backup |
| v0.112.0 | **E-2b** — `GET /disks` flags the backup-target drive |
| v0.113.0 | **E-2a** — the guarded wrapper + `POST /backup/target` |
**A new customer box installed today therefore runs an agent that predates the whole tiered-backup
model and lacks F-CRIT-2 and F-REBOOT** — two customer-impacting silent-failure fixes. That is a
larger finding than E-2d itself and is filed as **R-111**.
| **C1** host-install 1.22.0 completes a real install, rc=0 | ✅ **PROVEN** |
| **C2** Case B fires naturally on a single-drive box | ✅ **PROVEN** |
| **C3** the degraded banner renders **to a customer** | ⚠️ **PARTIAL — API exact, NO UI CONSUMER** (R-112) |
| **C4** the offer appears and moves the target when accepted | ⚠️ **PARTIAL — full API flow proven; offer equally invisible** (R-112) |
| **C5** `backup_target_absent` fires end to end | ❌ **FAILED** — no event on any channel (R-113) |
---
## 3. Claim-by-claim impact, each pinned to its introducing commit
## 1. Baselines as actually confirmed
| Claim | Needs | Introduced in | Fresh box has | Verdict |
|---|---|---|---|---|
| **C1** rc=0 real install, banner names 1.22.0 | host-install 1.22.0 | website ← `main` | **1.22.0** | **ACHIEVABLE** |
| **C2** Case B DEGRADED lines + `resolved local` | host-install `configure_backup_target()` `:627`, Case B `:653-655` | host-install 1.22.0 | **1.22.0** | **ACHIEVABLE** — installer-side only |
| **C3** degraded banner via `GET /api/storage/backup-target` | controller **v0.185.1** (`cdaeb36`); copy „lemezhiba ellen nem" **v0.185.0** (`3f7cf2a`) | — | controller 0.161.0 | **BLOCKED** — endpoint and copy do not exist |
| **C4** offer → `POST …/assign` → `restart_required` | controller **v0.185.1** (`cdaeb36`) + agent **v0.113.0** `POST /backup/target` (`58b598b`) | — | 0.161.0 / 0.96.0 | **BLOCKED** |
| **C5** `backup_target_absent` / `_restored` | controller **v0.184.0** (`c1a63de`) + agent **v0.112.0** | — | 0.161.0 / 0.96.0 | **BLOCKED** |
**C3 has a cheap unblock; C4 and C5 do not.** C3 needs only the hub's global controller floor raised
0.156.0 → ≥0.185.0, because controller **0.185.1 IS published** to the registry (60 tags, newest
0.185.1) — a fresh box would self-update on its first report. C4/C5 need agent 0.112.0/0.113.0
*published as Gitea generic packages*, which §0 of the runbook explicitly forbids this run from doing
(*"Does not touch: … any published artifact"*).
Fleet impact of raising the floor, measured rather than assumed (hub dashboard, 2026-07-29):
| Customer | Controller | Effect of floor → 0.185.1 |
| Artifact | Confirmed | Source |
|---|---|---|
| Demo Ügyfél | 0.185.1 | none — already at it |
| Demo HP | 0.185.1 | none — already at it |
| Peti Proxmox | 0.115.0, **DOWN 14d** | already below the current 0.156.0 floor, so it updates on return either way; only the target version changes |
| hub | 0.81.0 | `manifests/hub.yaml:128`; live deploy image |
| agent | 0.113.0 | `main` @ `58b598b`; **published this run**, sha `5f3247f7…` |
| golden | **0.185.1** | baked this run, sha `dba00f3e…`, embeds controller 0.185.1 |
| controller | 0.185.1 | `main` @ `cdaeb36`; live in the guest |
| host-install | 1.22.0 | `scripts/felhom-host-install.sh:187`; fetched from the website at run time |
| felhom.eu | `3dff357` | `main` HEAD at run time |
---
**Route: ISO/PAIRING (the real customer chain).** ISO
`felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso`. Every C1–C5 result is on the real route; no
manual-installer fallback was used.
## 4. Phase 0 gate answers (all four completed before the stop)
**Operator STOP: not required and now retired.** `HUB_PW` is in `~/.config/credentials`; CC created
the customer and performed the bind itself. The one human step that *was* needed is new — see §6.
### 4.1 Storage placement — gate PASSES, fence confirmed
## 2. Timeline (VM 9300 `e2d-fresh` on demo-hp, nested PVE)
`pvesm status` on demo-hp, 2026-07-29 (BEFORE; there is no AFTER — no VM was created):
| UTC | Event |
|---|---|
| 10:29 | VM created — q35/OVMF SB-off, 4c/8G, **one** 160 G disk, `hotplug disk`, outside the `felhom` pool |
| 10:34:52 | PVE auto-install done, first boot, DHCP `192.168.0.125` (no R-59 gate trip) |
| 10:34:53 | registered as unclaimed appliance, pairing code `SB4-7ZK`, console banner rendered |
| 10:36:52 | **bound** to customer `e2d-fresh` by CC; credentials delivered 2 s later |
| 10:37:31 | host enrolled `e2d-fresh-ac9f09`; break-glass root credential vaulted |
| 10:37:33 | **artifact manifest served: agent=0.113.0 golden=0.185.1** |
| 10:40:27 | `controller_started (0.185.1)` — **bind → running controller in 3 m 35 s** |
| 10:56 | second 100 G disk hot-attached; wizard init→mount→register |
| 10:57:08 | offer accepted → `restart_required:true`; agent restarted at 10:57:36 |
| 10:58:37 | target drive **hot-detached** (volume survives as `unused0`) |
| 10:58:37–11:03 | **no event on any channel** for 4½ minutes (budget was 60 s) |
| 11:03:31 | reattached; state returns healthy; **still no event** |
## 3. C1 — PROVEN
From the `felhom-bootstrap.service` journal on the box:
```
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
felhom-backup dir active 983379700 2293856 931059232 0.23%
felhom-pbs pbs active 0 0 0 0.00%
local dir active 40516856 14721460 23705004 36.33%
local-lvm lvmthin active 56545280 21922605 34622674 38.77%
felhom-bootstrap: fetching host-install: https://felhom.eu/scripts/felhom-host-install.sh
[INFO] felhom-host-install v1.22.0 — mode=appliance customer=e2d-fresh vmid=9201
...
[OK] controller: Up 21 seconds (healthy)
[INFO] controller image: gitea.dooplex.hu/admin/felhom-controller:0.185.1
[OK] Day-0 provision SUCCESS — vmid=9201 host_id=e2d-fresh-ac9f09 customer=e2d-fresh
golden=local:backup/vzdump-lxc-9100-2026_07_29-12_37_56.tar.zst
felhom-bootstrap: host-install SUCCESS — writing done-flag, disabling unit, scrubbing secrets
```
`local-lvm` is thin, pool `<53.93g`, **data 38.77 %** — matching the runbook's "~54 GB pool, 38.8 %"
to 0.03 pp, so the picture has **not** materially changed. Allocated LVs on it: `vm-9201-disk-{0,1,2}`
(32+50+20 G, the **live customer guest**), `vm-300-disk-{0,1}` (drill-r50, 32 G) and its `r50pre`
snapshots (32 G) ⇒ ~166 G allocated against 53.93 G real. `vgs` shows only 14.75 G VFree.
**The fence holds: nothing goes on `local-lvm`.**
rc=0 is corroborated structurally: the unit wrote its done-flag, **self-disabled**, and
`Deactivated successfully`. `pct list` showed guest **9201 `e2d-fresh` running**.
`/mnt/nvme-1tb` = `/dev/nvme0n1`, ext4, 938 G, **888 G free**. ≥100 G requirement satisfied.
**The publish train is proven end to end:** the golden restored is
`vzdump-lxc-9100-2026_07_29-12_37_56` — the golden baked ~20 minutes earlier in the same session.
This retires E-2's *"installer-logic-tested, not install-tested"*.
**Placement decision (recorded deliberately, per §3.1):** the only storage with `content images` is
`local-lvm` (forbidden). `felhom-backup` is a `dir` at exactly `/mnt/nvme-1tb` but is `content backup`
only **and is demo-hp's live backup target** — widening its content set would mutate the very storage
E-2c/E-2 role logic keys on, which §0/§9 forbid. So a *new* dir storage would have been required.
Both remaining shapes carry a cost and the choice was **not** forced this run because the run stopped:
a second storage at the same mountpoint risks perturbing the agent's role resolution on a live box;
a storage at a **subdirectory** fails `exactMount` and reports `disconnected` in the host report,
which is not purely cosmetic since it can reach the hub's storage monitor. **This decision is left
open and is an input to any resumed run.**
## 4. C2 — PROVEN
### 4.2 Drive-gate cadence — answered
Both required warning lines, verbatim, ANSI-stripped:
- Symbol: `driveGateLoop` → `s.ReconcileDriveGates()`
- Definition: `controller/internal/web/intermediary.go:328` (gate itself at `:267`)
- Registration: `controller/internal/web/server.go:217` — `go s.driveGateLoop()`
- **Interval: `time.NewTicker(30 * time.Second)`** (`intermediary.go:337`)
```
[WARN] backup target: DEGRADED — no eligible second drive, so the whole-system backup stays on the SYSTEM drive.
[WARN] It protects against file corruption but NOT against a disk failure. Attach a second drive and assign it in the dashboard.
```
⇒ Stage 7's derived budget would be **two cycles = 60 s** of controller-side reconcile, *plus* the
agent's own `/disks` refresh, since `ReconcileDriveGates` consumes `resp.Disks` (`:281`). Recorded so
a resumed run does not have to re-derive it.
- `agent.json` → `backup.local_backup_target = 'local'` (the "resolved local" observable)
- no `felhom-backup` storage created at install
- **the install did not abort** — a single-drive appliance is a valid product
### 4.3 Hot-detach — feasible, not exercised
## 5. THE FINDINGS
demo-hp: AMD `svm` present, `/sys/module/kvm_amd/parameters/nested = 1`, 8 cores, 29 GB RAM (25 GB
available). The working nested-PVE reference is VM 300 (`drill-r50`): `bios: ovmf`,
`efitype=4m,pre-enrolled-keys=0` (Secure Boot OFF, as the mkimage loader requires), `machine: q35`,
`scsihw: virtio-scsi-single`, `cpu: host`. `virtio-scsi-single` + PVE's default
`hotplug: disk,network,usb` supports SCSI hot-detach, so the **live-transition** variant of Stage 7
was available. Not exercised.
### 5.1 R-112 — the degraded banner and the offer have NO UI consumer (customer-invisible)
### 4.4 Install route — ISO available, route selected, not used
The endpoint is **correct**. Authenticated `GET /api/storage/backup-target` returned, exactly:
Two generic (pairing) ISOs are present on demo-hp `local`:
`felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso` (**preferred**, the v1.25.0 train, mkimage
loader for nested VMs) and `felhom-pve-9.2-1-v1.22.0-nested-canary-generic.iso`.
The **ISO/PAIRING route was selected**. It reaches the same installer invocation — `run_pairing`
(`felhom-bootstrap.sh:441`) falls through to `run_direct` (`:495-499`) in the same invocation, which
fetches `$INSTALL_URL` (`:322-330`) and runs it (`:343`). Not used — the run stopped first.
```json
{"degraded":true,"known":true,"target":"local",
"message":"A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer — így hibás fájlok
ellen véd, lemezhiba ellen nem. Csatlakoztass egy második meghajtót a teljes védelemhez."}
```
**Operator STOP: not required.** `HUB_PW` is present in `~/.config/credentials` and hub auth was
verified working (`http://10.43.52.34:8080/` → 200 with `curl -u ":$HUB_PW"`; public ingress also 200),
so CC could have performed the bind itself. The runbook's §5.1a STOP is retired for future runs.
byte-identical to the runbook's required copy. `known` is a field **separate** from `degraded`, so
UNKNOWN genuinely cannot render as degraded (R-88 Part 2's lesson held).
---
**And nothing in the product ever asks for it.** Negative claims with their search scope:
## 5. Incidental observations (filed, not acted on)
| Search | Result |
|---|---|
| `grep -rn 'backup-target' --include='*.html' --include='*.js' --include='*.css' controller/` | **0 hits** |
| `grep -rn 'kijelölheted\|ugyanazon a lemezen\|OfferPath\|OfferLabel\|Degraded' controller/internal/web/templates/` | **0 hits** |
| consumers of `backupTargetDegradedText` / `backupTargetOfferText` | only `degradedMessageFor` (`:136`) and the JSON handler (`:153`) — both inside `backup_target_offer.go` |
| consumers of `resolveBackupTargetState` / `degradedMessageFor` across all Go | **only** the API handler. **No page handler injects the state.** |
1. **A stale unclaimed appliance is sitting in the hub**: `206c8838-7755-4751-8a16-a842853d718f`,
pairing code `QWA-WJE`, MAC `bc:24:11:ea:55:d0`, "Standard PC (Q35 + ICH9, 2009)" on an
AMD V1756B with 7.7 GB — i.e. a **nested VM on demo-hp**, first and last seen **5 d ago**. It is a
leftover from the 2026-07-25 ISO work and has never been bound or discarded. A resumed run must
distinguish its own appliance from this one, or discard it first.
2. **The agent has no container image** (`/v2/admin/felhom-agent/tags/list` → 0 tags). It ships only
as a Gitea generic package, which is why the publish gap is invisible from the registry.
3. `drill-r50` (VM 300) still holds a `r50pre` snapshot pair consuming ~32 G of allocation on the
over-subscribed `local-lvm`. Untouched per §9; noted because it is part of why the pool is tight.
**The decisive contrast:** the templates fetch **18 distinct `/api/storage/*` endpoints`**.
`backup-target` and `backup-target/assign` are the **only two** referenced by zero templates.
---
The handler's own doc comment reads *"serves GET /api/storage/backup-target — **the dashboard's source
for the degraded banner and the offer**"* — an invariant comment asserting a consumer that does not
exist (`CLAUDE.md`'s "a comment asserting an invariant needs a test pinning it, or it is a wish",
instance #7). And v0.185.1's own test, `TestBackupTargetRoutesLiveUnderTheStorageAPIMount`, pins that
the **router dispatches** the path — not that anything renders it. v0.185.1 shipped as *"the offer
endpoints were mounted where nothing routed to them"*: it fixed the mount and stopped one layer short.
## 6. What did not happen
**Fifth instance of the seam-built-but-never-wired class.** `CLAUDE.md`'s seam rule names exactly this:
*"a feature is not shipped until its entry point is reachable… handler tests that POST directly prove
nothing about reachability."*
No VM was created. No ISO booted. No install ran. No appliance registered, bound or discarded. No
storage was created or modified on demo-hp. No drive attached or detached. No hub setting changed
(the floor was **read**, not written). No teardown was needed. **C1–C5 are all UNPROVEN**, and the
E-2 / E-2d rows are unchanged except to record this blocker.
### 5.2 R-113 — the drive-absent gate CANNOT fire on device loss (E-2b's alarm is unreachable)
Detached the assigned target drive at 10:58:37Z under a running agent. Over the next 4½ minutes:
- **agent**, every 20 s: `storage: enrolled drive absent by UUID — not re-asserting` /
`reconcile: enrolled drive not present (durable-id absent) — skipping` — **the agent knows**
- **controller**: `docker logs | grep -c '\[gate\]'` → **0**. The gate never acted, ABSENT or RETURNED
- **hub**: zero events for the customer across the whole window — no `backup_target_absent`,
**and no generic `storage_disconnected` either**
**Root cause, established at source and confirmed live.** `planDriveGates`
(`intermediary.go:216-262`) computes presence as
`present[GuestPath] = present[GuestPath] || d.BoundUnderParent`, and the agent derives
`BoundUnderParent` from `GuestSeesMount()` — *"does the guest's `/proc/<pid>/mountinfo` list this path
as a mount target"* (`localapi/disks.go:210`, `localapi/intermediary.go` `GuestSeesMount`).
Measured on the box with the device removed:
```
-- raw mount /mnt/mentes2 -- NOT mounted <- systemd device-bound unit, died with the device
-- stable bind /mnt/felhom-drives/mentes2 -- /dev/sdb[/felhom-data] ext4 <- the agent's MANUAL bind, SURVIVES
```
The raw mount is a device-bound systemd unit and dies correctly; **the agent's own bind under the
shared parent is not device-bound, so its mountinfo entry outlives the device.** The gate reads that
surviving entry as "present" ⇒ `!present[...]` is never true ⇒ no `Stop` action ⇒ `notifyDriveAbsent`
is never called.
This is not a virtualisation artefact: the asymmetry is between a device-bound mount and a manual
bind, which is identical on physical hardware. **Caveat kept honest:** proven on a SCSI hot-detach; a
physical USB unplug was not staged.
**Consequence.** E-2b's celebrated fix — *"THE SEAM THAT WAS NEVER WIRED… a drive that is ONLY a
backup target has no apps to stop, so it was silent twice over"* (`intermediary.go:288-296`) — wired
the notify to a branch that **cannot execute on device loss**. The seam is wired; the condition is
unreachable. Sixth instance of the class, one layer deeper than the fifth.
**Mirror scenario:** not separately staged, and it does not need to be — both the specific and the
generic event are emitted from the same `a.Stop` branch, which never executed. The generic
`storage_disconnected` is equally unreachable by this path. Recorded as *reasoned, not observed*.
### 5.3 R-114 — on target-drive loss the customer is told the wrong story and offered the missing drive
While the target drive was absent, the endpoint returned:
```json
{"degraded":true,"target":"felhom-backup",
"message":"A rendszermentés jelenleg ugyanazon a lemezen van, mint a rendszer …",
"offer_path":"/mnt/felhom-drives/mentes2","offer_label":"Mentés meghajtó"}
```
Two defects in one payload: the message claims the backup is **on the system disk**, which is false —
the target is `felhom-backup` on a drive that has vanished; and the remedy offered is **the drive that
just disappeared**. `resolveBackupTargetState` falls through to the generic degraded branch whenever no
disk satisfies `d.BackupTarget && d.MountPath != ""`, without distinguishing *never configured* from
*configured and now missing*.
**Interaction worth stating: R-114 is currently invisible only because of R-112.** Fixing R-112 alone —
wiring the banner — would immediately start showing customers this wrong message. **They must be fixed
together, R-114 first.**
**Also observed:** after reattach the drive returned as `/dev/sdc`, while the stable bind still
recorded `/dev/sdb[/felhom-data]`. The state read healthy (`degraded:false`) with the guest-visible
bind still naming the dead device node. Not chased further; recorded as part of R-113's shape.
### 5.4 Smaller findings (recorded, not filed as their own IDs)
1. **A "hard min" that only warns.** `[WARN] local-lvm free ~83 GiB < hard min 120 GiB` — the installer
names a hard minimum and proceeds. Either it is not hard, or the wording is wrong.
2. **`felhom-backup-target-apply` is fetched unvouched.** host-install pulls **nine** files from
`raw/branch/main` (`:2072`–`:2206`); the hub manifest vouches a sha for exactly one
(`wrapper_sha256` → `felhom-pbs-apply`, verified this run: no drift). E-2a's wrapper is installed
**0755 to `/usr/local/sbin` and root-fenced in sudoers**, validated only by `bash -n`.
**Filed as a second instance under R-110**, not a new ID — same class (a root-executed artifact
taken from `main` with no pinned integrity).
## 6. C4 — what IS proven (API level)
Everything except customer reachability:
- **The offer** appeared with all three fields: `offer_path=/mnt/felhom-drives/mentes2`,
`offer_label=Mentés meghajtó`, `offer_message=Ezt a meghajtót kijelölheted a rendszermentés helyéül…`
- **The decline path (§6.4) — PROVEN.** After registering the drive and *not* accepting:
`target` still `local`, no `felhom-backup` storage, `agent.json` unchanged. **Registration does not
confer a role** — E-2 §3's central invariant, live.
- **Accept** → `{"assigned":"/mnt/felhom-drives/mentes2","restart_required":true}`
- **The agent did NOT self-restart** — `ActiveEnterTimestamp` unchanged at 12:37:53 CEST twenty
minutes later. **In-flight check performed and recorded before restarting:** 0 running PVE tasks,
no `vzdump` process, 0 backup lines in the agent journal.
- **The E-2a root-fenced wrapper worked on a fresh box:** created `dir: felhom-backup`,
`path /mnt/mentes2`, `is_mountpoint 1` — the drive's **own** mountpoint (law F-2).
- **Healthy renders nothing:** after the restart the payload is
`{"degraded":false,"known":true,"label":"Mentés meghajtó","target":"felhom-backup"}` — **no
`message` field at all**. No "backup protected" reassurance (E-2 Scenario E).
## 7. E-2d's own premise needs amending
The runbook assumed a fresh install yields a claimable customer CC can then drive. It does not:
**the claim code is bcrypt-hashed in the hub and only ever emailed** (`claim/engine.go:58-95`), and the
gate covers everything except `/claim`, `/claim/request-new-code`, `/api/health`, `/static/`
(`claim.go:222-229`). C3/C4/C5 all sit behind it. This run cleared it by registering an operator email,
resending, and having the operator relay the code — **the one genuine human step**, and it also proved
the claim flow end to end (code → password → `401 "dashboard not yet claimed"` becoming
`401 "authentication required"`, a positive discriminator).
## 8. Teardown
- VM 9300 destroyed `--purge`; `e2d-images` storage removed; scratch dir removed.
- **`pvesm status` after == before**: `local-lvm` **38.77 %**, `felhom-backup` 931059224 KiB available
— byte-identical to the pre-run measurement. Freed space returned.
- **Guest 9201 (live customer) untouched and running; `drill-r50` VM 300 untouched.**
- Drill VM (golden bake) torn down per GL-1 earlier: guest 9100 purged, secrets shredded, disk restored
to `virgin`, token-leak grep 0.
**Hub records NOT yet removed — stated, not silent.** `e2d-fresh` + host `e2d-fresh-ac9f09` remain.
The delete was attempted and **correctly refused** at four successive gates: acknowledgements → typed
`confirm_id` → `expect_hosts` stale-preview → finally `host e2d-fresh-ac9f09 is ONLINE`. The host is
online only because its last report is recent; the VM is destroyed, so it ages OK → WARN (30 m) → DOWN
(>1 h) and is then deletable. **Cleanup command, once it reads DOWN:**
```
POST /configs/e2d-fresh/delete ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=e2d-fresh expect_hosts=1
```
Tracked in `OPEN-ITEMS.md`. Deleting rather than keeping is deliberate — R-93 records what a
half-real fixture costs.
**Also still present:** the stale unclaimed appliance from 2026-07-25 (`206c8838…`, code `QWA-WJE`).
Not mine; not discarded. A future run must distinguish its own appliance from it.