From b4edc087fa9afe5820c98d4b653a2fdbf5607ae8 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Fri, 31 Jul 2026 08:27:36 +0200 Subject: [PATCH] =?UTF-8?q?Tester=20gate:=20golden=20re-baked=20to=200.188?= =?UTF-8?q?.0,=20fresh-install=20proof=20PASSED=20=E2=80=94=20a=20fresh=20?= =?UTF-8?q?box=20is=20safe=20to=20hand=20to=20a=20tester?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit §7.2 answer: YES. A real day-0 from the existing v1.25.0 ISO reached a claimable, app-serving box in ~10 minutes unattended, and an app's data came back from the drive with the guest's app.yaml gone — proven readable by the application over its own TCP path, with a discriminator (PRE-BACKUP row = 1, POST-BACKUP row = 0). Part 0: NO ISO rebuild needed, verified against the ISO on disk rather than from source. It bakes only felhom-bootstrap.sh, its unit and the secret-free pairing env (full-base64 match, 1 hit each) and 0 hits for any installer, controller or golden marker. The installer is fetched at run time; the live URL is byte-identical to repo HEAD (v1.22.0, six days newer than the ISO) and the fresh box ran it. Part 1: baked 0.188.0 rather than the brief's 0.187.0 — 0.187.0 lacks D5, which is the very claim Part 2 step 6 tests. Published (404 pre-gate with a 200 control; anonymous download, 649310288 bytes, sha match), vouched, and consumed by a real box. R-120's gate exercised BOTH ways: 0.185.1 refused with no write, 0.188.0 allowed — evaluated, not silently skipped. Part 3: RUNBOOK-manual-build.md cited a "RECORDED" qemu line that is itself labelled reconstructed and whose source says it was never saved. The real invocation is now captured from this bake as §4.0, with the bake/publish/teardown steps; the old entry is marked SUPERSEDED. Teardown all three layers, hub disposition stated: VM destroyed, scratch storage removed with space returned exactly, customer sess-g DELETED via full cascade. sess-f deliberately left (R-131) with its command recorded. Filed, none fixed: R-128 (false ISO_VERSION invariant comment), R-129 (demo-hp's "no baked SSH key" is stale — key auth works), R-130 (HARD_MIN_LVM_GIB warns and proceeds), R-131 (fourth orphaned scratch customer), R-132 (curl's %{redirect_url} printed the hub operator password into a transcript — HUB_PW needs rotating). --- REPORT-tester-gate-2026-07-31.md | 98 +++++ .../tester-gate-golden-0.188.0-2026-07-31.md | 402 ++++++++++++++++++ documentation/backlog/OPEN-ITEMS.md | 5 + .../RUNBOOK-publish-0.79-0.110-2026-07-10.md | 4 + .../runbooks/RUNBOOK-manual-build.md | 69 ++- 5 files changed, 568 insertions(+), 10 deletions(-) create mode 100644 REPORT-tester-gate-2026-07-31.md create mode 100644 documentation/audits/tester-gate-golden-0.188.0-2026-07-31.md diff --git a/REPORT-tester-gate-2026-07-31.md b/REPORT-tester-gate-2026-07-31.md new file mode 100644 index 0000000..0d7a1a5 --- /dev/null +++ b/REPORT-tester-gate-2026-07-31.md @@ -0,0 +1,98 @@ +# REPORT — tester gate: golden re-baked to 0.188.0, fresh-install proof PASSED (2026-07-31) + +Written as `REPORT-.md` per `CLAUDE.md:82-87` so the shared `REPORT.md` (E-2 increment 1) is +not clobbered. Full record with every observable: `documentation/audits/tester-gate-golden-0.188.0-2026-07-31.md`. + +## Outcome + +**§7.2 — YES: a fresh install is safe to hand to an external tester.** ISO boot → claimable, +app-serving box in ~10 minutes unattended, and an app's data restored **from the drive with the +guest's `app.yaml` gone**, proven readable by the application over its own TCP path. + +**Golden 0.186.0 → 0.188.0** baked, published, vouched. **No ISO rebuilt** — Part 0 proved none was +needed. No existing box changed; floor still v0.156.0, MinAgent still 0.113.0. + +## Part 0 — the ISO does not need rebuilding + +Verified against the ISO **on disk**, not from source. It bakes exactly three Felhom payloads +(`felhom-bootstrap.sh`, its unit, the secret-free pairing env) — full-base64 match, 1 hit each — and +**0** hits for `SCRIPT_VERSION="1.2`, `felhom-controller`, `vzdump-lxc-9100`. The installer is fetched +at run time (`felhom-bootstrap.sh:96`) and the live URL is byte-identical to repo HEAD +(sha `ab8b283e…`, v1.22.0, committed six days *after* the ISO). The golden arrives via the hub-vouched +artifact manifest (`felhom-host-install.sh:423-433`). The one genuinely baked, drift-capable thing is +`felhom-bootstrap.sh` itself — currently at repo HEAD. + +**Proven live**, not just argued: the fresh box ran `felhom-host-install v1.22.0` and fetched golden +**v0.188.0**, sha-verified. + +## Part 1 — bake / publish / vouch + +Baked **0.188.0**, not the brief's 0.187.0: 0.187.0 lacks D5, and Part 2 step 6 *is* the D5 claim, so +that golden could not have passed the proof this task exists for. 0.188.0 satisfies R-120 anyway. +Stated rather than absorbed, per standing rule 4. + + GOLDEN_VERSION=0.188.0 + GOLDEN_SHA256=7353d8beb63641f87a848e45f8aa12e465647e1190ad164a65b32ad01fc3d299 + +Three observables: 404 pre-gate (with a 200 control on 0.186.0 so it is not vacuous), then an +**anonymous** download returning `http=200 bytes=649310288` and a matching sha; the manifest read back +showing `0.188.0` selected; and the consumer call `GET https://hub.felhom.eu/api/v1/artifacts/sess-g` +returning the pair. Plus a fourth: a real fresh box fetched and sha-verified it. + +**R-120's gate evaluated and allowed.** Exercised both ways rather than inferred from silence — +vouching 0.185.1 first produced `flash=golden_behind_fleet`, the logged `artifact vouch REFUSED`, and +**no write** (the manifest still read 0.186.0); then 0.188.0 produced +`Artifact manifest set: agent=0.118.1 golden=0.188.0`. + +## Part 2 — the clean-install proof, on demo-hp + +All seven steps PASS. Venue was demo-hp (Tier 0, the designated drill host) using the scratch dir +storage at `/mnt/nvme-1tb` that `target-selection.md:38-40` names; `local-lvm`, `drill-r50` and both +9201s untouched. + +Highlights: real day-0 pairing → bind → install; a **real** claim (the code is emailed-only, R-119 — +the operator relayed it), with the gate flipping `dashboard not yet claimed` → `authentication +required`; controller **0.188.0** confirmed *from the box*; **rallly** (postgres) + **homebox** +deployed through the real endpoints. + +The D5 leg: recovery unit `portable-carried=2/2, withheld=0`; the carried `DB_PASSWORD` matched the +live one **by fingerprint** (`14c8f515…`, never printed); guest `app.yaml` moved aside; restore +returned `secrets recovered=2/2`. Step 7 read the data from **rallly's own network namespace** over +TCP to `rallly-postgres` — not the localhost trust socket that produced D5's false pass — and the same +path with a wrong password returned `FATAL: password authentication failed`, proving the credential +does real work. **The discriminator held: PRE-BACKUP row = 1, POST-BACKUP row = 0.** + +## Part 3 — runbook integrity + +`RUNBOOK-manual-build.md` told the reader to use a "RECORDED" qemu line that is itself labelled +*reconstructed*, and whose source says it *"was never saved"*. The real invocation is now captured +from this bake and recorded as canonical in **§4.0**, alongside the bake/publish/teardown steps, the +template-rot warning and where the R-120 gate actually lives. The old runbook's deviation entry is +marked SUPERSEDED with a forward pointer. + +## Teardown — three layers + +1. VM 310 destroyed with `--purge --destroy-unreferenced-disks 1`; `/mnt/nvme-1tb/images/` empty. +2. `cc-scratch` removed, `storage.cfg` back to its original four entries; `felhom-backup` available + **926 492 284 KiB before and after** — space returned exactly. +3. Hub: **`sess-g` and its host record DELETED, full cascade** — `customer DELETE cascade COMPLETE + for sess-g (journal #8) — full teardown`, residue purged including `appliance_registrations=1`. + Verified positively: `/configs` and `/hosts` both loaded (10160 / 9880 bytes) with **0** hits for + `sess-g` and 0 for the appliance UUID. The gate refused twice first (409 host ONLINE, then 400 + missing acknowledgements) — the record cannot be deleted until the destroyed box ages out of + ONLINE, ~30 min. **`sess-f` deliberately NOT deleted** (R-131); its command is in the audit §7.1. + Secrets shredded in the guest and on the box. + +## Findings — filed, none fixed + +`R-128` ISO_VERSION/SCRIPT_VERSION comment is false · `R-129` demo-hp's "no baked SSH key" is stale +(key auth works) · `R-130` `HARD_MIN_LVM_GIB` warns and proceeds — a hard min that is not hard · +`R-131` `sess-f` is a fourth orphaned scratch customer · **`R-132` — `curl -w '%{redirect_url}'` +printed the hub operator password into a session transcript; `HUB_PW` needs rotating.** + +## Not done, deliberately + +No ISO built; no defect fixed; no golden deployed to an existing box; no floor or MinAgent change; +offsite/PBS-DR legs not exercised (the task forbids pointing anything at production PBS or the real +restic offsite, so `sess-g` ran DR-tier off); Campaign 10, the demo-hp repartition and subdomain +onboarding untouched. diff --git a/documentation/audits/tester-gate-golden-0.188.0-2026-07-31.md b/documentation/audits/tester-gate-golden-0.188.0-2026-07-31.md new file mode 100644 index 0000000..d9be883 --- /dev/null +++ b/documentation/audits/tester-gate-golden-0.188.0-2026-07-31.md @@ -0,0 +1,402 @@ +# Tester gate — re-bake the golden, and prove a fresh box is safe to hand to a tester (2026-07-31) + +**The sentence that matters (§7.2): YES** — a fresh install from the existing v1.25.0 ISO is safe to +hand to an external tester. Installed clean and unattended, registered, claimed, deployed two apps, +and restored an app's data with the guest's `app.yaml` gone — with the data proven readable by the +application over the path the application actually uses. Caveats in §8; none of them blocks a tester. + +Golden re-baked **0.186.0 → 0.188.0** (D5), published, and vouched. No ISO was rebuilt — Part 0 showed +none was needed. Nothing already installed changed. + +--- + +## 1. Baselines — established separately, at the start of this run + +| What | Value | How | +|---|---|---| +| controller `main` | **v0.188.0** (D5) | `HEAD`=`origin/main`=`4115e88`, tree clean | +| controller on `demo-felhom` (9201) | 0.187.0 | `pct exec 9201 -- cat /etc/felhom-controller-image` | +| controller on `demo-hp` | 0.187.0 | hub `/configs` row (the box's own report) | +| **golden's baked controller, before this run** | **0.186.0** (sha `b760ac6a…`) | hub manifest; `drill/bake-0.186.0.log` | +| agent published + vouched | 0.118.1 (sha `9a74c87d…`) | hub manifest; matches `felhom-agent --version` on felhom-pve | +| hub | 0.83.0 | `deploy/hub` image tag | +| ISO | v1.25.0, built 2026-07-23 | `build-felhom-iso.sh:44`; on-disk manifest | +| `felhom.eu` HEAD | `0a9bd38`, clean | `git rev-parse` | +| managed floor | v0.156.0 (DB override) | hub `/configuration` | + +The brief expected golden 0.186.0 and fleet 0.187.0 — both correct. It also assumed `main` was +0.187.0; it is **0.188.0**. See §3. + +## 2. Part 0 — does the ISO need rebuilding? **NO** + +**1. What the ISO contains.** Three Felhom payloads, base64-injected into the first-boot stub +(`scripts/iso/stub-first-boot.sh:22-38`, filled at `scripts/iso/build-felhom-iso.sh:324-329`): +`felhom-bootstrap.sh`, `felhom-bootstrap.service`, and the bootstrap env. Plus `answer.toml` (fqdn, +disk-setup, a per-build throwaway root hash) and the GRUB branding. **No controller image, no golden +archive, no installer copy.** + +Verified against the ISO **on disk**, not inferred from source — full-base64 match, one hit each: + +| Probe | Hits | +|---|---| +| base64 of repo-HEAD `felhom-bootstrap.sh` (36 692 chars) | **1** | +| base64 of repo-HEAD `felhom-bootstrap.service` | **1** | +| base64 of the reconstructed secret-free pairing env | **1** | +| `SCRIPT_VERSION="1.2` (host-install marker) | **0** | +| `felhom-controller` | **0** | +| `vzdump-lxc-9100` (golden marker) | **0** | + +**2. What the bootstrap fetches at run time.** `felhom-bootstrap.sh:96` — +`INSTALL_URL="${FELHOM_INSTALL_URL:-https://felhom.eu/scripts/felhom-host-install.sh}"`, fetched on +every attempt at `:322-326`. Served from the repo's `/scripts` by git-sync +(`manifests/webpage.yaml:71-75`). Live check: 200, and **byte-identical to repo HEAD** (both sha256 +`ab8b283e…`), SCRIPT_VERSION 1.22.0 — committed `f6aed82` on 2026-07-29, *six days after the ISO was +built*. **Proven live in §5: the fresh box ran v1.22.0.** + +**3. How the golden reaches a fresh box.** The hub-vouched artifact manifest. +`scripts/felhom-host-install.sh:423-433` `resolve_artifacts()` GETs +`$HUB_URL/api/v1/artifacts/$CUSTOMER_ID` authed with the retrieval passphrase, reads +`golden.version` + `golden.sha256`, then fetches from Gitea and verifies the sha. Nothing baked. + +**4. Version pins inside the ISO.** Four candidates; none pins the box's controller or golden: + +- `ISO_VERSION="1.25.0"` (`build-felhom-iso.sh:44`) — filename/manifest tag, read by nothing at run time. +- The baked `felhom-bootstrap.sh` — **this one IS frozen**; a bootstrap change would require a new ISO. + Currently identical to repo HEAD, so no drift today. This is the single thing that could ever force + an ISO rebuild, and it is the answer to "what is baked". +- `answer.toml`'s disk-setup + throwaway root hash — per-build, not versions. +- The env's two URLs — endpoints, not versions. + +**Verdict: no ISO work.** Re-baking + vouching the golden is sufficient, and that is what was done. + +## 3. Part 1 — bake, publish, vouch + +### 3.1 The version baked, and why not 0.187.0 + +Part 1 said re-baking to **0.187.0** would level R-120's gate. It would — but it would ship a golden +**without D5**, and Part 2 step 6 *is* the D5 claim, so the proof this task exists for could not have +passed. The operator page also says "rebuilt on the current controller", and current is **0.188.0**. +The brief's arithmetic predates D5 landing (2026-07-30). **Baked `felhom-controller:0.188.0`**, which +also satisfies R-120 (`configs.go:1166` refuses only `compareVersions(golden, newest) < 0`; 0.188.0 vs +the fleet's 0.187.0 is not `< 0`). Recorded here rather than absorbed silently, per standing rule 4. + +### 3.2 The bake + +Venue: the DooPlex golden-bake nested VM, the **accepted exception for bakes** +(`target-selection.md:59-61`). Reverted to `virgin` first, restored to `virgin` after. +`build-golden.sh v2.1.0`, template `debian-13-standard_13.6-1_amd64.tar.zst`. + +Pass markers, as returned: `docker OK (overlay2; data-root /var/lib/docker)`; `including mount point` +for **rootfs, mp0 and mp1** with **no** `excluding` and **no** `FATAL`; archive 619 MB; +`upload OK (HTTP 201)`. Infra images baked = **4**, resolved from the controller itself +(`--print-infra-images`), not the drift-prone fallback list: `traefik:v3.6.7`, +`cloudflare/cloudflared:2026.6.0`, `gtstef/filebrowser:1.3.3-stable`, +`gitea.dooplex.hu/admin/felhom-samba:1.1.0`. Literal-token leak grep on the saved log: **0**. + + GOLDEN_VERSION=0.188.0 + GOLDEN_SHA256=7353d8beb63641f87a848e45f8aa12e465647e1190ad164a65b32ad01fc3d299 + +### 3.3 The three observables, as returned + +**1 — published.** 404 pre-gate first, with a discriminating control so the check is not vacuous: + + golden 0.188.0 pre-gate: HTTP 404 + golden 0.186.0 (control): HTTP 200 + +Then an **anonymous** full download — what a fresh box does, no auth header: + + http=200 bytes=649310288 + 7353d8beb63641f87a848e45f8aa12e465647e1190ad164a65b32ad01fc3d299 golden.tar.zst + +Byte count and sha both identical to what the bake printed. + +**2 — the manifest vouches it, read back:** + +