felhom.eu/documentation/audits/tester-gate-golden-0.188.0-2026-07-31.md

Tester gate — re-bake the golden, and prove a fresh box is safe to hand to a tester (2026-07-31)

The sentence that matters (§7.2): YES — a fresh install from the existing v1.25.0 ISO is safe to hand to an external tester. Installed clean and unattended, registered, claimed, deployed two apps, and restored an app's data with the guest's app.yaml gone — with the data proven readable by the application over the path the application actually uses. Caveats in §8; none of them blocks a tester.

Golden re-baked 0.186.0 → 0.188.0 (D5), published, and vouched. No ISO was rebuilt — Part 0 showed none was needed. Nothing already installed changed.

---

1. Baselines — established separately, at the start of this run

What	Value	How
controller main	v0.188.0 (D5)	HEAD=origin/main=4115e88, tree clean
controller on demo-felhom (9201)	0.187.0	pct exec 9201 -- cat /etc/felhom-controller-image
controller on demo-hp	0.187.0	hub /configs row (the box's own report)
golden's baked controller, before this run	0.186.0 (sha b760ac6a…)	hub manifest; drill/bake-0.186.0.log
agent published + vouched	0.118.1 (sha 9a74c87d…)	hub manifest; matches felhom-agent --version on felhom-pve
hub	0.83.0	deploy/hub image tag
ISO	v1.25.0, built 2026-07-23	build-felhom-iso.sh:44; on-disk manifest
felhom.eu HEAD	0a9bd38, clean	git rev-parse
managed floor	v0.156.0 (DB override)	hub /configuration

The brief expected golden 0.186.0 and fleet 0.187.0 — both correct. It also assumed main was 0.187.0; it is 0.188.0. See §3.

2. Part 0 — does the ISO need rebuilding? NO

1. What the ISO contains. Three Felhom payloads, base64-injected into the first-boot stub (scripts/iso/stub-first-boot.sh:22-38, filled at scripts/iso/build-felhom-iso.sh:324-329): felhom-bootstrap.sh, felhom-bootstrap.service, and the bootstrap env. Plus answer.toml (fqdn, disk-setup, a per-build throwaway root hash) and the GRUB branding. No controller image, no golden archive, no installer copy.

Verified against the ISO on disk, not inferred from source — full-base64 match, one hit each:

Probe	Hits
base64 of repo-HEAD felhom-bootstrap.sh (36 692 chars)	1
base64 of repo-HEAD felhom-bootstrap.service	1
base64 of the reconstructed secret-free pairing env	1
SCRIPT_VERSION="1.2 (host-install marker)	0
felhom-controller	0
vzdump-lxc-9100 (golden marker)	0

2. What the bootstrap fetches at run time. felhom-bootstrap.sh:96 — INSTALL_URL="${FELHOM_INSTALL_URL:-https://felhom.eu/scripts/felhom-host-install.sh}", fetched on every attempt at :322-326. Served from the repo's /scripts by git-sync (manifests/webpage.yaml:71-75). Live check: 200, and byte-identical to repo HEAD (both sha256 ab8b283e…), SCRIPT_VERSION 1.22.0 — committed f6aed82 on 2026-07-29, six days after the ISO was built. Proven live in §5: the fresh box ran v1.22.0.

3. How the golden reaches a fresh box. The hub-vouched artifact manifest. scripts/felhom-host-install.sh:423-433 resolve_artifacts() GETs $HUB_URL/api/v1/artifacts/$CUSTOMER_ID authed with the retrieval passphrase, reads golden.version + golden.sha256, then fetches from Gitea and verifies the sha. Nothing baked.

4. Version pins inside the ISO. Four candidates; none pins the box's controller or golden:

• ISO_VERSION="1.25.0" (build-felhom-iso.sh:44) — filename/manifest tag, read by nothing at run time.
• The baked felhom-bootstrap.sh — this one IS frozen; a bootstrap change would require a new ISO. Currently identical to repo HEAD, so no drift today. This is the single thing that could ever force an ISO rebuild, and it is the answer to "what is baked".
• answer.toml's disk-setup + throwaway root hash — per-build, not versions.
• The env's two URLs — endpoints, not versions.

Verdict: no ISO work. Re-baking + vouching the golden is sufficient, and that is what was done.

3. Part 1 — bake, publish, vouch

3.1 The version baked, and why not 0.187.0

Part 1 said re-baking to 0.187.0 would level R-120's gate. It would — but it would ship a golden without D5, and Part 2 step 6 is the D5 claim, so the proof this task exists for could not have passed. The operator page also says "rebuilt on the current controller", and current is 0.188.0. The brief's arithmetic predates D5 landing (2026-07-30). Baked felhom-controller:0.188.0, which also satisfies R-120 (configs.go:1166 refuses only compareVersions(golden, newest) < 0; 0.188.0 vs the fleet's 0.187.0 is not < 0). Recorded here rather than absorbed silently, per standing rule 4.

3.2 The bake

Venue: the DooPlex golden-bake nested VM, the accepted exception for bakes (target-selection.md:59-61). Reverted to virgin first, restored to virgin after. build-golden.sh v2.1.0, template debian-13-standard_13.6-1_amd64.tar.zst.

Pass markers, as returned: docker OK (overlay2; data-root /var/lib/docker); including mount point for rootfs, mp0 and mp1 with no excluding and no FATAL; archive 619 MB; upload OK (HTTP 201). Infra images baked = 4, resolved from the controller itself (--print-infra-images), not the drift-prone fallback list: traefik:v3.6.7, cloudflare/cloudflared:2026.6.0, gtstef/filebrowser:1.3.3-stable, gitea.dooplex.hu/admin/felhom-samba:1.1.0. Literal-token leak grep on the saved log: 0.

GOLDEN_VERSION=0.188.0
GOLDEN_SHA256=7353d8beb63641f87a848e45f8aa12e465647e1190ad164a65b32ad01fc3d299

3.3 The three observables, as returned

1 — published. 404 pre-gate first, with a discriminating control so the check is not vacuous:

golden 0.188.0 pre-gate: HTTP 404
golden 0.186.0 (control): HTTP 200

Then an anonymous full download — what a fresh box does, no auth header:

http=200 bytes=649310288
7353d8beb63641f87a848e45f8aa12e465647e1190ad164a65b32ad01fc3d299  golden.tar.zst

Byte count and sha both identical to what the bake printed.

2 — the manifest vouches it, read back:

<option value="0.188.0" data-sha="7353d8be…" selected
name="golden_sha256" value="7353d8beb63641f87a848e45f8aa12e465647e1190ad164a65b32ad01fc3d299"

3 — a consumer resolves it. The exact call felhom-host-install.sh:425 makes, against the public hub, passphrase-authed:

GET https://hub.felhom.eu/api/v1/artifacts/sess-g  →  HTTP 200
{"agent":{"version":"0.118.1","sha256":"9a74c87d…"},
 "golden":{"version":"0.188.0","sha256":"7353d8be…"}}

4 (stronger, from §5) — a real fresh box consumed it:

[INFO]   manifest: agent v0.118.1 (sha 9a74c87dc97f259b…), golden v0.188.0
[INFO]   fetching golden v0.188.0 from Gitea → …/vzdump-lxc-9100-2026_07_31-07_40_20.tar.zst
[OK]     verified sha256 7353d8beb63641f8… matches the hub manifest

3.4 R-120's gate: it evaluated, and it allowed

An absent refusal is not evidence (standing rule 3), so the gate was exercised both ways against the live hub before the real vouch. Vouching golden 0.185.1 (below the fleet's 0.187.0) first:

HTTP/1.1 303 → /configuration?flash=golden_behind_fleet
[WARN] artifact vouch REFUSED: golden 0.185.1 is older than the newest controller the fleet
       reports (0.187.0) — a fresh install would land on stale application code (R-120)

and the manifest still read 0.186.0 afterwards — the refusal wrote nothing. Then the real vouch:

[INFO] Artifact manifest set: agent=0.118.1 golden=0.188.0 min_agent="0.113.0" wrapper_sha=true

Evaluated and allowed. For the record and not for action: the gate lives in the vouch form (hub/internal/web/configs.go:1165) and cannot fire on an image deploy, so it does not cover every drift path. Not widened here.

4. Part 2 — venue

demo-hp (t740), Tier 0, the designated drill+build host. The brief's venue caution is accurate about the current storage.cfg — local carries no images content type and local-lvm is fenced — but target-selection.md:38-40 names the fix: "Put VM disks on a dir storage at /mnt/nvme-1tb, at its root", and "remove scratch storages when done". A temporary cc-scratch dir storage was added at that path and removed at teardown (§7). felhom-pve was the alternative and was not chosen: 11 GB free RAM against demo-hp's 25, and it is the Tier 0 box that carries the PBS-DR/offsite tier. Nested virt confirmed on (kvm_amd/parameters/nested = 1). drill-r50 (VM 300), guest 9201 on both boxes, local-lvm, and both backup targets were untouched.

VM 310: q35/OVMF with pre-enrolled-keys=0 (the ISO is a -mkimage build — unsigned loader, Secure Boot must be off), 8 GB, 4 cores, 150 G qcow2, boot order=scsi0;ide2 so the empty disk falls through to the ISO and the installed disk wins afterwards. The ISO already present on demo-hp was verified byte-identical to the DooPlex build (99b38420…).

5. Part 2 — the proof, step by step

#	Step	Result
1	Install from the ISO, real day-0	PASS
2	Register and claim	PASS
3	Controller is the new one, from the box	PASS
4	Two apps deployed, one with a database	PASS
5	Discriminating data seeded	PASS
6	Back up, restore with app.yaml moved aside	PASS
7	Data readable by the app over its real path	PASS

Step 1 — install. Unattended from boot to provisioned in ~10 minutes. PVE 9.2.2 installed, first-boot stub laid down the retry unit, the box registered itself as an unclaimed appliance (pairing code HTG-N8D), was bound to customer sess-g on the hub, and the delivery landed on the next poll. The installer that actually ran: felhom-host-install v1.22.0 — fetched at run time, six days newer than the ISO. That is Part 0's claim, proven on a real box.

[OK] Day-0 provision SUCCESS — vmid=9201 host_id=sess-g-e687c3 customer=sess-g
     golden=local:backup/vzdump-lxc-9100-2026_07_31-07_40_20.tar.zst

Secrets scrubbed on success: /etc/felhom/ retained only .bootstrap-done and the non-secret pairing code; bootstrap.env and the appliance token were shredded.

Step 2 — register and claim. Host enrolled, wg peer registered, host-report accepted, DR-recipe stored. The claim gate was armed from first boot and the claim was a real one — the code exists only in the customer's email (R-119; the operator relayed it). The positive discriminator flipped:

before:  {"ok":false,"error":"dashboard not yet claimed"}
after:   {"ok":false,"error":"authentication required"}
hub:     [INFO] [claim] customer sess-g CLAIMED its dashboard (password set by the customer)

Step 3 — the controller, from the box, not from the manifest:

/etc/felhom-controller-image : gitea.dooplex.hu/admin/felhom-controller:0.188.0
docker ps                    : gitea.dooplex.hu/admin/felhom-controller:0.188.0 | Up (healthy)
felhom-agent --version       : felhom-agent 0.118.1

Step 4 — two apps, through the real endpoints (GET /stacks/{app}/deploy for the pre-generated values, then POST /api/stacks/{app}/deploy, the endpoint the UI invokes):

• rallly — lukevella/rallly:4.11.1 + postgres:16-alpine, the DB app. Chosen because its DB_PASSWORD is type: secret, i.e. exactly D5's travelling class, and its DATABASE_URL is a TCP URL to a different container: postgresql://rallly@rallly-postgres:5432/rallly.
• homebox — sqlite, no DB container.

Both healthy; rallly served 200 through traefik on its own route.

Step 5 — the discriminator. cc_proof table, one row before the backup (PRE-BACKUP-sess-g-2026-07-31), one after (POST-BACKUP-sess-g-MUST-NOT-RETURN). Without this pair a stale restore is indistinguishable from a good one.

The read path, established and red-proofed before it was trusted. All DB access ran from rallly's own network namespace (docker run --network container:rallly … psql -h rallly-postgres) — the path DATABASE_URL names, never 127.0.0.1 inside the postgres container, which postgres trusts and which produced D5's §1.2 false pass. Proven to be doing real work:

correct password : rallly|t|t                    (user=rallly, over_tcp=t, PostgreSQL 16)
wrong password   : FATAL: password authentication failed for user "rallly"

Step 6 — backup, then restore with the guest gone.

[backup] Recovery unit captured for rallly → /mnt/sys_drive/felhom-data/backups/primary/rallly
         (images=2, secrets-referenced=2, data_keys=0, portable-carried=2/2, withheld=0)
[backup] Volume dump: rallly/rallly_rallly_postgres_data → 65.0 MB

Unit compose/app.yaml: mode 0600, header naming the carried class, both secrets present. Manifest schema_version: 2, controller_version: 0.188.0. The carried DB_PASSWORD was verified by fingerprint, never printed, against the live one:

unit  DB_PASSWORD sha256=14c8f5152b205829
live  DB_PASSWORD sha256=14c8f5152b205829

and the guest's own copy was ENC: at rest. The guest's app.yaml was then moved aside — confirmed absent — and the restore run through POST /backup/restore:

[backup] Restoring rallly from recovery unit: images=2, secrets recovered=2/2, data_keys=0
[backup] Restored 1 Docker volume(s) for rallly
[backup] Restore rallly: replaying DB dump into rallly-postgres (postgres)
[backup] Restore-from-unit completed: rallly            (30.97 s)

2/2 with the guest's app.yaml gone.

Step 7 — the application reading its own data. The live app process was asked for its own credential, and it matches the pre-backup one — so the secret genuinely survived the guest:

host=rallly-postgres port=5432 user=rallly pw_sha256=14c8f5152b205829

Then the read, over that path:

current_user | server_ip     | over_tcp
rallly       | 172.19.0.2/32 | t

id | marker
1  | PRE-BACKUP-sess-g-2026-07-31

PRE-BACKUP rows  = 1
POST-BACKUP rows = 0

The discriminator held. 32 public tables intact; the guest's app.yaml was regenerated from the unit with both secrets re-encrypted at rest; rallly's own healthcheck returned 200, and traefik served poll.sess-g.test 200. Checked the data, not the exit code.

One difference from D5's adventurelog run worth recording: this unit carried both a .sql dump and the volume tar, and the restore did both (volume restored, then dump replayed). D5's case had db_dumps: None. The password mattered either way — the replay authenticates with it.

6. Part 3 — the runbook integrity fix

RUNBOOK-manual-build.md:104-106 told the reader to "use its RECORDED qemu launch line; do not reconstruct from memory", citing RUNBOOK-publish-0.79-0.110-2026-07-10.md:29 — a line labelled "PASS (reconstructed — DEVIATION)" whose own §Deviations says "the canonical qemu-system one-liner was never saved". The document forbade improvising and pointed at an improvisation.

Captured verbatim from this bake and recorded as canonical in RUNBOOK-manual-build.md §4.0:

qemu-img snapshot -a virgin /mnt/5_hdd/felhom.eu/drill/drill.qcow2
qemu-system-x86_64 -enable-kvm -cpu host -smp 4 -m 8192 \
  -drive file=/mnt/5_hdd/felhom.eu/drill/drill.qcow2,format=qcow2,if=virtio,cache=writeback \
  -netdev user,id=n0,dhcpstart=10.0.2.30,hostfwd=tcp::2222-10.0.2.15:22 \
  -device virtio-net-pci,netdev=n0 -display none -daemonize \
  -pidfile /mnt/5_hdd/felhom.eu/drill/qemu.pid

SSH on :2222 in ~40 s; pve-manager/9.2.2. §4 also now carries the bake/publish/teardown steps and the template-rot warning, and the old runbook's deviation entry is marked SUPERSEDED with a forward pointer so the reconstruction is never used as an authority again.

7. Teardown — all three layers

1. The machine. VM 310 stopped and qm destroy --purge --destroy-unreferenced-disks 1. qm list afterwards shows only drill-r50 (300), untouched; /mnt/nvme-1tb/images/ is empty.
2. The host. cc-scratch removed; storage.cfg back to its original four entries. felhom-backup available 926 492 284 KiB before the run and 926 492 284 KiB after — the space returned exactly.
3. The hub. sess-g and its host record are DELETED — the full cascade ran. See §7.1.

7.1 Layer 3, stated explicitly

The delete gate refused twice before it allowed, which is the gate working, and both refusals are recorded because "I deleted it" without them would hide that this needs a wait:

POST /hosts/sess-g-e687c3/delete           → 409 Conflict   (host ONLINE)
POST /configs/sess-g/delete (bare confirm) → 400            (all three acknowledgements required)

customer_delete.go:220-227 refuses while hostStatus(...) == "ok", so the record cannot be deleted until the destroyed box ages out of ONLINE (~30 min after its last report). It flipped ok → stale at 08:25, and the cascade then ran:

POST /configs/sess-g/delete
  ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-g expect_hosts=1
→ 303 /configs?flash=deleted

[INFO] customer DELETE cascade started for sess-g (journal #8, 1 host(s))
[INFO] delete sess-g: host sess-g-e687c3 deleted (escrow DEMOTED to retained custody)
[INFO] tenantsync: deprovision ok for sess-g (ns=sess-g, existed=false)
[INFO] delete sess-g: residue purged (reports=5 app_telemetry=7 notif_prefs=1
       selfbind_tokens=1 appliance_registrations=1)
[INFO] customer DELETE cascade COMPLETE for sess-g (journal #8) — full teardown

Verified positively, not by absence — both pages loaded with real content and returned zero hits, and the unclaimed-appliance row is gone by its own UUID:

/configs  loaded 10160 bytes, occurrences of 'sess-g' = 0
/hosts    loaded  9880 bytes, occurrences of 'sess-g' = 0
/hosts    occurrences of appliance uuid f9f5d7e0-…            = 0

A polling trap, recorded because it nearly produced a false result here. The first delete-gate poll loop broke on grep -q '"status":"ok"' || break — and an empty response satisfies that just as well as a status change. It fired on one transient empty reply and reported "GATE OPEN" while the host was still ONLINE. Re-run parsing the JSON and treating unparseable as retry, the host was ok for another four minutes. Standing rule 3, in miniature: the absent signal is not the positive one.

sess-f is NOT deleted — it belongs to the 2026-07-30 session, not this run (R-131). Command for whoever picks it up, once its host is not ONLINE:

POST /configs/sess-f/delete
  ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-f expect_hosts=<live host count>

Secrets. Shredded in the guest (rallly-dbpw, the moved-aside app.yaml, deploy bodies, session and CSRF files, claim artefacts) and on the box; helper scripts removed from both. The scratch customer's retrieval passphrase and the ISO's throwaway root password were handled file→file and never printed; both die with the session scratchpad and the destroyed VM.

8. Findings — filed, none fixed

• R-128 — build-felhom-iso.sh:44 claims ISO_VERSION "aligns with felhom-host-install SCRIPT_VERSION". ISO 1.25.0, host-install 1.22.0. A comment asserting an invariant nothing maintains. Cosmetic; it misleads a reader about which installer an ISO implies.
• R-129 — the docs say demo-hp has "no baked SSH key" and needs the G1 break-glass password (CLAUDE.md, target-selection.md:41-42, felhom-agent/CLAUDE.md). ssh -o BatchMode=yes demo-hp authenticated by key, first try. The prohibition is stale in a direction that costs a future session a needless vault round-trip.
• R-130 — a fresh box's local-lvm was ~75 GiB against HARD_MIN_LVM_GIB=120, and the installer logged [WARN] local-lvm free ~75 GiB < hard min 120 GiB and continued to a fully successful install. Either the hard minimum is not hard, or it is wrong. A "hard min" that warns is the R-29 shape.
• R-131 (housekeeping) — hub customer sess-f ("R-120 golden 0.186.0 proof", DOWN) is a scratch customer left behind by the 2026-07-30 session — the fourth instance of the accumulation target-selection.md:86-87 and PROMPT-TEMPLATE.md §13 warn about (drill-r50, sess-c, sess-d, now sess-f). Not deleted here: it is not this run's record. Command in §7.1.
• R-132 — curl -w '%{redirect_url}' reconstructs the request URL WITH its basic-auth credential. While red-proofing the R-120 gate, a call made correctly with -u ":$HUB_PW" (no credential in any URL) still printed the hub operator password into this session's transcript, because the write-out format rebuilds the URL from the credential curl was given. The request was safe; the reporting was not. HUB_PW should be rotated. Rule going forward: read a redirect from -D - and grep '^Location:', never %{redirect_url}, on any authenticated call.
• R-119 confirmed live, not re-filed: the claim code is emailed-only with no operator-side route, and it is the single human step in an otherwise unattended day-0. Budget for it before a run.

9. §7.2 — the sentence that matters

Yes — a fresh install is safe to hand to an external tester. From ISO boot to a claimable, app-serving box was ~10 minutes unattended; the box fetched the current installer and the newly vouched golden and verified both by sha; two apps deployed through the real endpoints; and an app's data came back from the drive with the guest's app.yaml gone, readable by the application over its own TCP path, with a discriminator proving it was a restore and not a no-op.

One thing the tester's host must do that the operator must plan for: the claim code arrives only by email (R-119), so the tester needs a working mailbox on the customer record before the box boots. Nothing else found here blocks a tester; R-128/R-129/R-130 are documentation and gate-hygiene issues, and R-131 is housekeeping.

What "safe" does not mean: bug-free. Two things this run did not cover — the offsite/PBS-DR legs (deliberately: the task forbids pointing anything at production PBS or the real restic offsite, so sess-g was created with DR tier off and offsite disabled), and any bare-metal ISO. Only a nested-vm-profile v1.25.0 ISO exists on disk; a tester on real hardware needs a generic- or hp-profile build from the same tooling, which is an ISO build and therefore its own task.

10. What did not happen, and why

• No ISO built — Part 0 said none was needed, and the brief says to stop and report if one were.
• No defect fixed — R-128/129/130/131 filed only; a fix mid-run would describe a version that never existed.
• No existing box changed — no golden deployed to 9201 on either demo host, no floor moved (still v0.156.0), no MinAgent change (still 0.113.0).
• Campaign 10, the demo-hp repartition, subdomain onboarding — out of scope, untouched.
• sess-f not deleted — see R-131.