admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

E-2d: Phase 0 STOP — the Day-0 artifact channel cannot deliver the code under test

Browse Source 
No VM created, no install run, no box touched. The run stopped at the Phase 0
gate per runbook §3, before provisioning.

felhom-host-install.sh does not install what is on main. resolve_artifacts()
(:423-436) reads the hub-vouched manifest and fetches Gitea GENERIC PACKAGES
(agent :1945, golden :2573). Gitea's newest are agent 0.96.0 and golden 0.161.0;
the hub manifest selects exactly those; the global floor v0.156.0 is below the
golden's 0.161.0 so nothing self-updates. A fresh box therefore lands on agent
0.96.0 + controller 0.161.0 against main's 0.113.0 / 0.185.1. Agent 0.113.0
reached both demo boxes by direct deploy and is not in the channel at all.

Claim impact, each pinned to its introducing commit:
- C1 (real rc=0 1.22.0 install) and C2 (Case B natural) — ACHIEVABLE, not run;
  both are installer-side and host-install is served at 1.22.0.
- C3 — BLOCKED: banner + GET /api/storage/backup-target are controller v0.185.1
  (cdaeb36), copy v0.185.0 (3f7cf2a). Unblocks cheaply by raising the hub floor
  to >=0.185.0; measured fleet impact nil (both demo boxes already 0.185.1).
- C4 — BLOCKED: needs controller v0.185.1 + agent v0.113.0 (58b598b).
- C5 — BLOCKED: needs controller v0.184.0 (c1a63de) + agent v0.112.0.

Filed R-111 (P1): 17 unpublished agent releases (v0.97.0-v0.113.0) strand the
entire R-82 tiered-backup arc plus F-CRIT-2 and F-REBOOT, so a new customer's
box installs without them. Mirror of R-110, not a duplicate.

- audits/E2D-fresh-vm-2026-07-29.md — all four Phase 0 answers recorded so a
  resumed run does not re-derive them (cadence 30s; hot-detach available; ISO
  present; local-lvm fence re-measured at 38.77%, unchanged).
- OPEN-ITEMS.md — R-111 opened; E-2d re-stated, NOT closed.
- ROADMAP.md — R-111 under P1.
- capability map NOT touched: nothing was proven live.

The §5.1a operator STOP is retired — HUB_PW is in ~/.config/credentials and hub
auth was verified, so CC can bind on a resumed run.
This commit is contained in:
authentik Default Admin
2026-07-29 11:43:00 +02:00
parent 91a1dad0f3
commit f3f0d58844
4 changed files with 259 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/audits/E2D-fresh-vm-2026-07-29.md
+198
View File
@@ -0,0 +1,198 @@
# E2D-fresh-vm-2026-07-29 — Phase 0 STOP: the Day-0 artifact channel cannot deliver the code under test
**Run:** `RUNBOOK-e2d-fresh-vm-2026-07-29.md`, executed by CC on DooPlex, 2026-07-29.
**Outcome:** **STOPPED at Phase 0, before any VM was created.** No VM provisioned, no install run, no
box touched, no teardown required. Per §3: *"If any gate fails, STOP and report — do not adapt around it."*
**The finding in one sentence:** a box installed today through the real customer chain receives
**agent 0.96.0** and **controller 0.161.0**, because those are the newest artifacts ever published to
the Day-0 channel — so three of the five claims (C3, C4, C5) test endpoints and events that do not
exist in the software a fresh box actually runs.
---
## 1. Baselines confirmed
| Artifact | Runbook §1 | Confirmed | Source |
|---|---|---|---|
| hub | 0.81.0 | **0.81.0** | `manifests/hub.yaml:128`; `hub/CHANGELOG.md:1`; live deploy image |
| agent | 0.113.0 | **0.113.0** on `main` @ `58b598b` | `felhom-agent` HEAD; live `felhom-agent -version` on demo-hp |
| controller | 0.185.1 | **0.185.1** on `main` @ `cdaeb36` | `felhom-controller` HEAD; live image on guest 9201 |
| host-install | 1.22.0 | **1.22.0** | `scripts/felhom-host-install.sh:187` |
| felhom.eu | `91a1dad` | **`91a1dad`**, clean, == `origin/main` | `git rev-parse` |
These are the versions **on `main` and on the demo boxes**. They are not the versions a fresh install
receives — that distinction is the whole finding.
---
## 2. THE BLOCKER — the Day-0 artifact channel is 17 agent releases stale
### 2.1 The chain, read at source
`felhom-host-install.sh` does not use `main`. It resolves a hub-vouched manifest and fetches versioned
packages from Gitea:
- `resolve_artifacts()` (`:423-436`) → `GET $HUB_URL/api/v1/artifacts/$CUSTOMER_ID` → `agent.version`,
`golden.version` (served by `hub/internal/api/handler.go:2120` `handleArtifactManifest`).
- agent binary ← `$GITEA_BASE/api/packages/admin/generic/felhom-agent/$ART_AGENT_VER/felhom-agent` (`:1945`)
- golden ← `$GITEA_BASE/api/packages/admin/generic/felhom-golden/$ART_GOLDEN_VER/golden.tar.zst` (`:2573`)
### 2.2 What that channel actually holds (Gitea API, authenticated, 2026-07-29)
```
felhom-agent : newest = 0.96.0
all = 0.79.0 0.80.0 0.81.0 0.84.0 0.85.0 0.86.0 0.87.0 0.88.0 0.89.0
0.90.0 0.91.0 0.91.1 0.91.2 0.92.0 0.92.1 0.93.0 0.96.0
felhom-golden: newest = 0.161.0
all = 0.136.0 0.143.0 0.146.0 0.153.0 0.161.0
```
Hub's saved Day-0 manifest (`/configuration`, selected options): **agent `0.96.0`**
(sha `af938601…`), **golden `0.161.0`** (sha `77624408…`), min-agent `0.93.0`.
Hub global controller floor: **v0.156.0** (DB override; env fallback v0.120.0).
### 2.3 Therefore a fresh box lands on
| Component | Fresh install gets | `main` / demo boxes | Gap |
|---|---|---|---|
| host-install | **1.22.0** (website git-sync from `main`) | 1.22.0 | none |
| agent | **0.96.0** | 0.113.0 | **17 releases** |
| controller | **0.161.0** (golden bake; floor 0.156.0 < 0.161.0 ⇒ no self-update) | 0.185.1 | **24 releases** |
**Agent 0.113.0 is not in the channel at all** — it reached both demo boxes by direct deploy, never
through publish. Confirmed live: demo-hp reports `felhom-agent 0.113.0` while Gitea's newest is 0.96.0.
### 2.4 What that strands — 17 unpublished agent releases (`felhom-agent/CHANGELOG.md`)
| Version | What it carries |
|---|---|
| v0.97.0–v0.104.0 | **the entire R-82 per-target backup-tier arc** (local daily + PBS weekly), incl. v0.98.0's 30-minute false-failure bound, v0.101.0/0.102.0 scratch-guest + defer fixes, v0.103.0 R-84, v0.104.0 R-85 restore-test |
| v0.105.0 | R-88 Part 2 — the agent can say `unknown` |
| v0.106.0 | **F-CRIT-2** — a failed backup must not look like a fresh one (the phantom snapshot that reset the freshness clock, 7 days silent) |
| v0.107.0 | **F-REBOOT** — a guest rebooted during its backup never comes back |
| v0.108.0, v0.110.0 | F-LEAK — scratch-guest VMID band leak |
| v0.109.0 | F-OBS — the guest-power watchdog's positive observable |
| v0.111.0 | E-2c — the backup-target drive can no longer be ejected out from under the backup |
| v0.112.0 | **E-2b** — `GET /disks` flags the backup-target drive |
| v0.113.0 | **E-2a** — the guarded wrapper + `POST /backup/target` |
**A new customer box installed today therefore runs an agent that predates the whole tiered-backup
model and lacks F-CRIT-2 and F-REBOOT** — two customer-impacting silent-failure fixes. That is a
larger finding than E-2d itself and is filed as **R-111**.
---
## 3. Claim-by-claim impact, each pinned to its introducing commit
| Claim | Needs | Introduced in | Fresh box has | Verdict |
|---|---|---|---|---|
| **C1** rc=0 real install, banner names 1.22.0 | host-install 1.22.0 | website ← `main` | **1.22.0** | **ACHIEVABLE** |
| **C2** Case B DEGRADED lines + `resolved local` | host-install `configure_backup_target()` `:627`, Case B `:653-655` | host-install 1.22.0 | **1.22.0** | **ACHIEVABLE** — installer-side only |
| **C3** degraded banner via `GET /api/storage/backup-target` | controller **v0.185.1** (`cdaeb36`); copy „lemezhiba ellen nem" **v0.185.0** (`3f7cf2a`) | — | controller 0.161.0 | **BLOCKED** — endpoint and copy do not exist |
| **C4** offer → `POST …/assign` → `restart_required` | controller **v0.185.1** (`cdaeb36`) + agent **v0.113.0** `POST /backup/target` (`58b598b`) | — | 0.161.0 / 0.96.0 | **BLOCKED** |
| **C5** `backup_target_absent` / `_restored` | controller **v0.184.0** (`c1a63de`) + agent **v0.112.0** | — | 0.161.0 / 0.96.0 | **BLOCKED** |
**C3 has a cheap unblock; C4 and C5 do not.** C3 needs only the hub's global controller floor raised
0.156.0 → ≥0.185.0, because controller **0.185.1 IS published** to the registry (60 tags, newest
0.185.1) — a fresh box would self-update on its first report. C4/C5 need agent 0.112.0/0.113.0
*published as Gitea generic packages*, which §0 of the runbook explicitly forbids this run from doing
(*"Does not touch: … any published artifact"*).
Fleet impact of raising the floor, measured rather than assumed (hub dashboard, 2026-07-29):
| Customer | Controller | Effect of floor → 0.185.1 |
|---|---|---|
| Demo Ügyfél | 0.185.1 | none — already at it |
| Demo HP | 0.185.1 | none — already at it |
| Peti Proxmox | 0.115.0, **DOWN 14d** | already below the current 0.156.0 floor, so it updates on return either way; only the target version changes |
---
## 4. Phase 0 gate answers (all four completed before the stop)
### 4.1 Storage placement — gate PASSES, fence confirmed
`pvesm status` on demo-hp, 2026-07-29 (BEFORE; there is no AFTER — no VM was created):
```
Name Type Status Total (KiB) Used (KiB) Available (KiB) %
felhom-backup dir active 983379700 2293856 931059232 0.23%
felhom-pbs pbs active 0 0 0 0.00%
local dir active 40516856 14721460 23705004 36.33%
local-lvm lvmthin active 56545280 21922605 34622674 38.77%
```
`local-lvm` is thin, pool `<53.93g`, **data 38.77 %** — matching the runbook's "~54 GB pool, 38.8 %"
to 0.03 pp, so the picture has **not** materially changed. Allocated LVs on it: `vm-9201-disk-{0,1,2}`
(32+50+20 G, the **live customer guest**), `vm-300-disk-{0,1}` (drill-r50, 32 G) and its `r50pre`
snapshots (32 G) ⇒ ~166 G allocated against 53.93 G real. `vgs` shows only 14.75 G VFree.
**The fence holds: nothing goes on `local-lvm`.**
`/mnt/nvme-1tb` = `/dev/nvme0n1`, ext4, 938 G, **888 G free**. ≥100 G requirement satisfied.
**Placement decision (recorded deliberately, per §3.1):** the only storage with `content images` is
`local-lvm` (forbidden). `felhom-backup` is a `dir` at exactly `/mnt/nvme-1tb` but is `content backup`
only **and is demo-hp's live backup target** — widening its content set would mutate the very storage
E-2c/E-2 role logic keys on, which §0/§9 forbid. So a *new* dir storage would have been required.
Both remaining shapes carry a cost and the choice was **not** forced this run because the run stopped:
a second storage at the same mountpoint risks perturbing the agent's role resolution on a live box;
a storage at a **subdirectory** fails `exactMount` and reports `disconnected` in the host report,
which is not purely cosmetic since it can reach the hub's storage monitor. **This decision is left
open and is an input to any resumed run.**
### 4.2 Drive-gate cadence — answered
- Symbol: `driveGateLoop` → `s.ReconcileDriveGates()`
- Definition: `controller/internal/web/intermediary.go:328` (gate itself at `:267`)
- Registration: `controller/internal/web/server.go:217` — `go s.driveGateLoop()`
- **Interval: `time.NewTicker(30 * time.Second)`** (`intermediary.go:337`)
⇒ Stage 7's derived budget would be **two cycles = 60 s** of controller-side reconcile, *plus* the
agent's own `/disks` refresh, since `ReconcileDriveGates` consumes `resp.Disks` (`:281`). Recorded so
a resumed run does not have to re-derive it.
### 4.3 Hot-detach — feasible, not exercised
demo-hp: AMD `svm` present, `/sys/module/kvm_amd/parameters/nested = 1`, 8 cores, 29 GB RAM (25 GB
available). The working nested-PVE reference is VM 300 (`drill-r50`): `bios: ovmf`,
`efitype=4m,pre-enrolled-keys=0` (Secure Boot OFF, as the mkimage loader requires), `machine: q35`,
`scsihw: virtio-scsi-single`, `cpu: host`. `virtio-scsi-single` + PVE's default
`hotplug: disk,network,usb` supports SCSI hot-detach, so the **live-transition** variant of Stage 7
was available. Not exercised.
### 4.4 Install route — ISO available, route selected, not used
Two generic (pairing) ISOs are present on demo-hp `local`:
`felhom-pve-9.2-1-v1.25.0-nested-vm-generic-mkimage.iso` (**preferred**, the v1.25.0 train, mkimage
loader for nested VMs) and `felhom-pve-9.2-1-v1.22.0-nested-canary-generic.iso`.
The **ISO/PAIRING route was selected**. It reaches the same installer invocation — `run_pairing`
(`felhom-bootstrap.sh:441`) falls through to `run_direct` (`:495-499`) in the same invocation, which
fetches `$INSTALL_URL` (`:322-330`) and runs it (`:343`). Not used — the run stopped first.
**Operator STOP: not required.** `HUB_PW` is present in `~/.config/credentials` and hub auth was
verified working (`http://10.43.52.34:8080/` → 200 with `curl -u ":$HUB_PW"`; public ingress also 200),
so CC could have performed the bind itself. The runbook's §5.1a STOP is retired for future runs.
---
## 5. Incidental observations (filed, not acted on)
1. **A stale unclaimed appliance is sitting in the hub**: `206c8838-7755-4751-8a16-a842853d718f`,
pairing code `QWA-WJE`, MAC `bc:24:11:ea:55:d0`, "Standard PC (Q35 + ICH9, 2009)" on an
AMD V1756B with 7.7 GB — i.e. a **nested VM on demo-hp**, first and last seen **5 d ago**. It is a
leftover from the 2026-07-25 ISO work and has never been bound or discarded. A resumed run must
distinguish its own appliance from this one, or discard it first.
2. **The agent has no container image** (`/v2/admin/felhom-agent/tags/list` → 0 tags). It ships only
as a Gitea generic package, which is why the publish gap is invisible from the registry.
3. `drill-r50` (VM 300) still holds a `r50pre` snapshot pair consuming ~32 G of allocation on the
over-subscribed `local-lvm`. Untouched per §9; noted because it is part of why the pool is tight.
---
## 6. What did not happen
No VM was created. No ISO booted. No install ran. No appliance registered, bound or discarded. No
storage was created or modified on demo-hp. No drive attached or detached. No hub setting changed
(the floor was **read**, not written). No teardown was needed. **C1–C5 are all UNPROVEN**, and the
E-2 / E-2d rows are unchanged except to record this blocker.