felhom.eu/documentation/audits/E2D-fresh-vm-2026-07-29.md

E2D-fresh-vm-2026-07-29 — Phase 0 STOP: the Day-0 artifact channel cannot deliver the code under test

Run: RUNBOOK-e2d-fresh-vm-2026-07-29.md, executed by CC on DooPlex, 2026-07-29. Outcome: STOPPED at Phase 0, before any VM was created. No VM provisioned, no install run, no box touched, no teardown required. Per §3: "If any gate fails, STOP and report — do not adapt around it."

The finding in one sentence: a box installed today through the real customer chain receives agent 0.96.0 and controller 0.161.0, because those are the newest artifacts ever published to the Day-0 channel — so three of the five claims (C3, C4, C5) test endpoints and events that do not exist in the software a fresh box actually runs.

---

1. Baselines confirmed

Artifact	Runbook §1	Confirmed	Source
hub	0.81.0	0.81.0	manifests/hub.yaml:128; hub/CHANGELOG.md:1; live deploy image
agent	0.113.0	0.113.0 on main @ 58b598b	felhom-agent HEAD; live felhom-agent -version on demo-hp
controller	0.185.1	0.185.1 on main @ cdaeb36	felhom-controller HEAD; live image on guest 9201
host-install	1.22.0	1.22.0	scripts/felhom-host-install.sh:187
felhom.eu	91a1dad	91a1dad, clean, == origin/main	git rev-parse

These are the versions on main and on the demo boxes. They are not the versions a fresh install receives — that distinction is the whole finding.

---

2. THE BLOCKER — the Day-0 artifact channel is 17 agent releases stale

2.1 The chain, read at source

felhom-host-install.sh does not use main. It resolves a hub-vouched manifest and fetches versioned packages from Gitea:

• resolve_artifacts() (:423-436) → GET $HUB_URL/api/v1/artifacts/$CUSTOMER_ID → agent.version, golden.version (served by hub/internal/api/handler.go:2120 handleArtifactManifest).
• agent binary ← $GITEA_BASE/api/packages/admin/generic/felhom-agent/$ART_AGENT_VER/felhom-agent (:1945)
• golden ← $GITEA_BASE/api/packages/admin/generic/felhom-golden/$ART_GOLDEN_VER/golden.tar.zst (:2573)

2.2 What that channel actually holds (Gitea API, authenticated, 2026-07-29)

felhom-agent : newest = 0.96.0
  all = 0.79.0 0.80.0 0.81.0 0.84.0 0.85.0 0.86.0 0.87.0 0.88.0 0.89.0
        0.90.0 0.91.0 0.91.1 0.91.2 0.92.0 0.92.1 0.93.0 0.96.0
felhom-golden: newest = 0.161.0
  all = 0.136.0 0.143.0 0.146.0 0.153.0 0.161.0

Hub's saved Day-0 manifest (/configuration, selected options): agent 0.96.0 (sha af938601…), golden 0.161.0 (sha 77624408…), min-agent 0.93.0. Hub global controller floor: v0.156.0 (DB override; env fallback v0.120.0).

2.3 Therefore a fresh box lands on

Component	Fresh install gets	main / demo boxes	Gap
host-install	1.22.0 (website git-sync from main)	1.22.0	none
agent	0.96.0	0.113.0	17 releases
controller	0.161.0 (golden bake; floor 0.156.0 < 0.161.0 ⇒ no self-update)	0.185.1	24 releases

Agent 0.113.0 is not in the channel at all — it reached both demo boxes by direct deploy, never through publish. Confirmed live: demo-hp reports felhom-agent 0.113.0 while Gitea's newest is 0.96.0.

2.4 What that strands — 17 unpublished agent releases (felhom-agent/CHANGELOG.md)

Version	What it carries
v0.97.0–v0.104.0	the entire R-82 per-target backup-tier arc (local daily + PBS weekly), incl. v0.98.0's 30-minute false-failure bound, v0.101.0/0.102.0 scratch-guest + defer fixes, v0.103.0 R-84, v0.104.0 R-85 restore-test
v0.105.0	R-88 Part 2 — the agent can say unknown
v0.106.0	F-CRIT-2 — a failed backup must not look like a fresh one (the phantom snapshot that reset the freshness clock, 7 days silent)
v0.107.0	F-REBOOT — a guest rebooted during its backup never comes back
v0.108.0, v0.110.0	F-LEAK — scratch-guest VMID band leak
v0.109.0	F-OBS — the guest-power watchdog's positive observable
v0.111.0	E-2c — the backup-target drive can no longer be ejected out from under the backup
v0.112.0	E-2b — GET /disks flags the backup-target drive
v0.113.0	E-2a — the guarded wrapper + POST /backup/target

A new customer box installed today therefore runs an agent that predates the whole tiered-backup model and lacks F-CRIT-2 and F-REBOOT — two customer-impacting silent-failure fixes. That is a larger finding than E-2d itself and is filed as R-111.

---

3. Claim-by-claim impact, each pinned to its introducing commit

Claim	Needs	Introduced in	Fresh box has	Verdict
C1 rc=0 real install, banner names 1.22.0	host-install 1.22.0	website ← main	1.22.0	ACHIEVABLE
C2 Case B DEGRADED lines + resolved local	host-install configure_backup_target() :627, Case B :653-655	host-install 1.22.0	1.22.0	ACHIEVABLE — installer-side only
C3 degraded banner via GET /api/storage/backup-target	controller v0.185.1 (cdaeb36); copy „lemezhiba ellen nem" v0.185.0 (3f7cf2a)	—	controller 0.161.0	BLOCKED — endpoint and copy do not exist
C4 offer → POST …/assign → restart_required	controller v0.185.1 (cdaeb36) + agent v0.113.0 POST /backup/target (58b598b)	—	0.161.0 / 0.96.0	BLOCKED
C5 backup_target_absent / _restored	controller v0.184.0 (c1a63de) + agent v0.112.0	—	0.161.0 / 0.96.0	BLOCKED

C3 has a cheap unblock; C4 and C5 do not. C3 needs only the hub's global controller floor raised 0.156.0 → ≥0.185.0, because controller 0.185.1 IS published to the registry (60 tags, newest 0.185.1) — a fresh box would self-update on its first report. C4/C5 need agent 0.112.0/0.113.0 published as Gitea generic packages, which §0 of the runbook explicitly forbids this run from doing ("Does not touch: … any published artifact").

Fleet impact of raising the floor, measured rather than assumed (hub dashboard, 2026-07-29):

Customer	Controller	Effect of floor → 0.185.1
Demo Ügyfél	0.185.1	none — already at it
Demo HP	0.185.1	none — already at it
Peti Proxmox	0.115.0, DOWN 14d	already below the current 0.156.0 floor, so it updates on return either way; only the target version changes

---

4. Phase 0 gate answers (all four completed before the stop)

4.1 Storage placement — gate PASSES, fence confirmed

pvesm status on demo-hp, 2026-07-29 (BEFORE; there is no AFTER — no VM was created):

Name                 Type     Status     Total (KiB)      Used (KiB) Available (KiB)        %
felhom-backup         dir     active       983379700         2293856       931059232    0.23%
felhom-pbs            pbs     active               0               0               0    0.00%
local                 dir     active        40516856        14721460        23705004   36.33%
local-lvm         lvmthin     active        56545280        21922605        34622674   38.77%

local-lvm is thin, pool <53.93g, data 38.77 % — matching the runbook's "~54 GB pool, 38.8 %" to 0.03 pp, so the picture has not materially changed. Allocated LVs on it: vm-9201-disk-{0,1,2} (32+50+20 G, the live customer guest), vm-300-disk-{0,1} (drill-r50, 32 G) and its r50pre snapshots (32 G) ⇒ ~166 G allocated against 53.93 G real. vgs shows only 14.75 G VFree. The fence holds: nothing goes on local-lvm.

/mnt/nvme-1tb = /dev/nvme0n1, ext4, 938 G, 888 G free. ≥100 G requirement satisfied.

Placement decision (recorded deliberately, per §3.1): the only storage with content images is local-lvm (forbidden). felhom-backup is a dir at exactly /mnt/nvme-1tb but is content backup only and is demo-hp's live backup target — widening its content set would mutate the very storage E-2c/E-2 role logic keys on, which §0/§9 forbid. So a new dir storage would have been required. Both remaining shapes carry a cost and the choice was not forced this run because the run stopped: a second storage at the same mountpoint risks perturbing the agent's role resolution on a live box; a storage at a subdirectory fails exactMount and reports disconnected in the host report, which is not purely cosmetic since it can reach the hub's storage monitor. This decision is left open and is an input to any resumed run.

4.2 Drive-gate cadence — answered

• Symbol: driveGateLoop → s.ReconcileDriveGates()
• Definition: controller/internal/web/intermediary.go:328 (gate itself at :267)
• Registration: controller/internal/web/server.go:217 — go s.driveGateLoop()
• Interval: time.NewTicker(30 * time.Second) (intermediary.go:337)

⇒ Stage 7's derived budget would be two cycles = 60 s of controller-side reconcile, plus the agent's own /disks refresh, since ReconcileDriveGates consumes resp.Disks (:281). Recorded so a resumed run does not have to re-derive it.

4.3 Hot-detach — feasible, not exercised

demo-hp: AMD svm present, /sys/module/kvm_amd/parameters/nested = 1, 8 cores, 29 GB RAM (25 GB available). The working nested-PVE reference is VM 300 (drill-r50): bios: ovmf, efitype=4m,pre-enrolled-keys=0 (Secure Boot OFF, as the mkimage loader requires), machine: q35, scsihw: virtio-scsi-single, cpu: host. virtio-scsi-single + PVE's default hotplug: disk,network,usb supports SCSI hot-detach, so the live-transition variant of Stage 7 was available. Not exercised.

4.4 Install route — ISO available, route selected, not used

Two generic (pairing) ISOs are present on demo-hp local: felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso (preferred, the v1.25.0 train, mkimage loader for nested VMs) and felhom-pve-9.2-1-v1.22.0-nested-canary-generic.iso. The ISO/PAIRING route was selected. It reaches the same installer invocation — run_pairing (felhom-bootstrap.sh:441) falls through to run_direct (:495-499) in the same invocation, which fetches $INSTALL_URL (:322-330) and runs it (:343). Not used — the run stopped first.

Operator STOP: not required. HUB_PW is present in ~/.config/credentials and hub auth was verified working (http://10.43.52.34:8080/ → 200 with curl -u ":$HUB_PW"; public ingress also 200), so CC could have performed the bind itself. The runbook's §5.1a STOP is retired for future runs.

---

5. Incidental observations (filed, not acted on)

1. A stale unclaimed appliance is sitting in the hub: 206c8838-7755-4751-8a16-a842853d718f, pairing code QWA-WJE, MAC bc:24:11:ea:55:d0, "Standard PC (Q35 + ICH9, 2009)" on an AMD V1756B with 7.7 GB — i.e. a nested VM on demo-hp, first and last seen 5 d ago. It is a leftover from the 2026-07-25 ISO work and has never been bound or discarded. A resumed run must distinguish its own appliance from this one, or discard it first.
2. The agent has no container image (/v2/admin/felhom-agent/tags/list → 0 tags). It ships only as a Gitea generic package, which is why the publish gap is invisible from the registry.
3. drill-r50 (VM 300) still holds a r50pre snapshot pair consuming ~32 G of allocation on the over-subscribed local-lvm. Untouched per §9; noted because it is part of why the pool is tight.

---

6. What did not happen

No VM was created. No ISO booted. No install ran. No appliance registered, bound or discarded. No storage was created or modified on demo-hp. No drive attached or detached. No hub setting changed (the floor was read, not written). No teardown was needed. C1–C5 are all UNPROVEN, and the E-2 / E-2d rows are unchanged except to record this blocker.