felhom.eu/documentation/audits/tester-gate-golden-0.188.0-2026-07-31.md

# Tester gate — re-bake the golden, and prove a fresh box is safe to hand to a tester (2026-07-31)

**The sentence that matters (§7.2): YES** — a fresh install from the existing v1.25.0 ISO is safe to
hand to an external tester. Installed clean and unattended, registered, claimed, deployed two apps,
and restored an app's data with the guest's `app.yaml` gone — with the data proven readable by the
application over the path the application actually uses. Caveats in §8; none of them blocks a tester.

Golden re-baked **0.186.0 → 0.188.0** (D5), published, and vouched. No ISO was rebuilt — Part 0 showed
none was needed. Nothing already installed changed.

---

## 1. Baselines — established separately, at the start of this run

| What | Value | How |
|---|---|---|
| controller `main` | **v0.188.0** (D5) | `HEAD`=`origin/main`=`4115e88`, tree clean |
| controller on `demo-felhom` (9201) | 0.187.0 | `pct exec 9201 -- cat /etc/felhom-controller-image` |
| controller on `demo-hp` | 0.187.0 | hub `/configs` row (the box's own report) |
| **golden's baked controller, before this run** | **0.186.0** (sha `b760ac6a…`) | hub manifest; `drill/bake-0.186.0.log` |
| agent published + vouched | 0.118.1 (sha `9a74c87d…`) | hub manifest; matches `felhom-agent --version` on felhom-pve |
| hub | 0.83.0 | `deploy/hub` image tag |
| ISO | v1.25.0, built 2026-07-23 | `build-felhom-iso.sh:44`; on-disk manifest |
| `felhom.eu` HEAD | `0a9bd38`, clean | `git rev-parse` |
| managed floor | v0.156.0 (DB override) | hub `/configuration` |

The brief expected golden 0.186.0 and fleet 0.187.0 — both correct. It also assumed `main` was
0.187.0; it is **0.188.0**. See §3.

## 2. Part 0 — does the ISO need rebuilding? **NO**

**1. What the ISO contains.** Three Felhom payloads, base64-injected into the first-boot stub
(`scripts/iso/stub-first-boot.sh:22-38`, filled at `scripts/iso/build-felhom-iso.sh:324-329`):
`felhom-bootstrap.sh`, `felhom-bootstrap.service`, and the bootstrap env. Plus `answer.toml` (fqdn,
disk-setup, a per-build throwaway root hash) and the GRUB branding. **No controller image, no golden
archive, no installer copy.**

Verified against the ISO **on disk**, not inferred from source — full-base64 match, one hit each:

| Probe | Hits |
|---|---|
| base64 of repo-HEAD `felhom-bootstrap.sh` (36 692 chars) | **1** |
| base64 of repo-HEAD `felhom-bootstrap.service` | **1** |
| base64 of the reconstructed secret-free pairing env | **1** |
| `SCRIPT_VERSION="1.2` (host-install marker) | **0** |
| `felhom-controller` | **0** |
| `vzdump-lxc-9100` (golden marker) | **0** |

**2. What the bootstrap fetches at run time.** `felhom-bootstrap.sh:96` —
`INSTALL_URL="${FELHOM_INSTALL_URL:-https://felhom.eu/scripts/felhom-host-install.sh}"`, fetched on
every attempt at `:322-326`. Served from the repo's `/scripts` by git-sync
(`manifests/webpage.yaml:71-75`). Live check: 200, and **byte-identical to repo HEAD** (both sha256
`ab8b283e…`), SCRIPT_VERSION 1.22.0 — committed `f6aed82` on 2026-07-29, *six days after the ISO was
built*. **Proven live in §5: the fresh box ran v1.22.0.**

**3. How the golden reaches a fresh box.** The hub-vouched artifact manifest.
`scripts/felhom-host-install.sh:423-433` `resolve_artifacts()` GETs
`$HUB_URL/api/v1/artifacts/$CUSTOMER_ID` authed with the retrieval passphrase, reads
`golden.version` + `golden.sha256`, then fetches from Gitea and verifies the sha. Nothing baked.

**4. Version pins inside the ISO.** Four candidates; none pins the box's controller or golden:

- `ISO_VERSION="1.25.0"` (`build-felhom-iso.sh:44`) — filename/manifest tag, read by nothing at run time.
- The baked `felhom-bootstrap.sh` — **this one IS frozen**; a bootstrap change would require a new ISO.
  Currently identical to repo HEAD, so no drift today. This is the single thing that could ever force
  an ISO rebuild, and it is the answer to "what is baked".
- `answer.toml`'s disk-setup + throwaway root hash — per-build, not versions.
- The env's two URLs — endpoints, not versions.

**Verdict: no ISO work.** Re-baking + vouching the golden is sufficient, and that is what was done.

## 3. Part 1 — bake, publish, vouch

### 3.1 The version baked, and why not 0.187.0

Part 1 said re-baking to **0.187.0** would level R-120's gate. It would — but it would ship a golden
**without D5**, and Part 2 step 6 *is* the D5 claim, so the proof this task exists for could not have
passed. The operator page also says "rebuilt on the current controller", and current is **0.188.0**.
The brief's arithmetic predates D5 landing (2026-07-30). **Baked `felhom-controller:0.188.0`**, which
also satisfies R-120 (`configs.go:1166` refuses only `compareVersions(golden, newest) < 0`; 0.188.0 vs
the fleet's 0.187.0 is not `< 0`). Recorded here rather than absorbed silently, per standing rule 4.

### 3.2 The bake

Venue: the DooPlex golden-bake nested VM, the **accepted exception for bakes**
(`target-selection.md:59-61`). Reverted to `virgin` first, restored to `virgin` after.
`build-golden.sh v2.1.0`, template `debian-13-standard_13.6-1_amd64.tar.zst`.

Pass markers, as returned: `docker OK (overlay2; data-root /var/lib/docker)`; `including mount point`
for **rootfs, mp0 and mp1** with **no** `excluding` and **no** `FATAL`; archive 619 MB;
`upload OK (HTTP 201)`. Infra images baked = **4**, resolved from the controller itself
(`--print-infra-images`), not the drift-prone fallback list: `traefik:v3.6.7`,
`cloudflare/cloudflared:2026.6.0`, `gtstef/filebrowser:1.3.3-stable`,
`gitea.dooplex.hu/admin/felhom-samba:1.1.0`. Literal-token leak grep on the saved log: **0**.

    GOLDEN_VERSION=0.188.0
    GOLDEN_SHA256=7353d8beb63641f87a848e45f8aa12e465647e1190ad164a65b32ad01fc3d299

### 3.3 The three observables, as returned

**1 — published.** 404 pre-gate first, with a discriminating control so the check is not vacuous:

    golden 0.188.0 pre-gate: HTTP 404
    golden 0.186.0 (control): HTTP 200

Then an **anonymous** full download — what a fresh box does, no auth header:

    http=200 bytes=649310288
    7353d8beb63641f87a848e45f8aa12e465647e1190ad164a65b32ad01fc3d299  golden.tar.zst

Byte count and sha both identical to what the bake printed.

**2 — the manifest vouches it, read back:**

    <option value="0.188.0" data-sha="7353d8be…" selected
    name="golden_sha256" value="7353d8beb63641f87a848e45f8aa12e465647e1190ad164a65b32ad01fc3d299"

**3 — a consumer resolves it.** The exact call `felhom-host-install.sh:425` makes, against the
**public** hub, passphrase-authed:

    GET https://hub.felhom.eu/api/v1/artifacts/sess-g  →  HTTP 200
    {"agent":{"version":"0.118.1","sha256":"9a74c87d…"},
     "golden":{"version":"0.188.0","sha256":"7353d8be…"}}

**4 (stronger, from §5) — a real fresh box consumed it:**

    [INFO]   manifest: agent v0.118.1 (sha 9a74c87dc97f259b…), golden v0.188.0
    [INFO]   fetching golden v0.188.0 from Gitea → …/vzdump-lxc-9100-2026_07_31-07_40_20.tar.zst
    [OK]     verified sha256 7353d8beb63641f8… matches the hub manifest

### 3.4 R-120's gate: it evaluated, and it allowed

An absent refusal is not evidence (standing rule 3), so the gate was exercised **both ways** against
the live hub before the real vouch. Vouching golden **0.185.1** (below the fleet's 0.187.0) first:

    HTTP/1.1 303 → /configuration?flash=golden_behind_fleet
    [WARN] artifact vouch REFUSED: golden 0.185.1 is older than the newest controller the fleet
           reports (0.187.0) — a fresh install would land on stale application code (R-120)

and the manifest **still read 0.186.0** afterwards — the refusal wrote nothing. Then the real vouch:

    [INFO] Artifact manifest set: agent=0.118.1 golden=0.188.0 min_agent="0.113.0" wrapper_sha=true

**Evaluated and allowed.** For the record and not for action: the gate lives in the **vouch form**
(`hub/internal/web/configs.go:1165`) and cannot fire on an image deploy, so it does not cover every
drift path. Not widened here.

## 4. Part 2 — venue

**demo-hp (t740), Tier 0, the designated drill+build host.** The brief's venue caution is accurate
about the *current* `storage.cfg` — `local` carries no `images` content type and `local-lvm` is fenced
— but `target-selection.md:38-40` names the fix: *"Put VM disks on a dir storage at `/mnt/nvme-1tb`,
at its root"*, and *"remove scratch storages when done"*. A temporary `cc-scratch` dir storage was
added at that path and removed at teardown (§7). felhom-pve was the alternative and was not chosen:
11 GB free RAM against demo-hp's 25, and it is the Tier 0 box that carries the PBS-DR/offsite tier.
Nested virt confirmed on (`kvm_amd/parameters/nested = 1`). `drill-r50` (VM 300), guest 9201 on both
boxes, `local-lvm`, and both backup targets were untouched.

VM 310: q35/OVMF with `pre-enrolled-keys=0` (the ISO is a `-mkimage` build — **unsigned loader,
Secure Boot must be off**), 8 GB, 4 cores, 150 G qcow2, `boot order=scsi0;ide2` so the empty disk
falls through to the ISO and the installed disk wins afterwards. The ISO already present on demo-hp
was verified byte-identical to the DooPlex build (`99b38420…`).

## 5. Part 2 — the proof, step by step

| # | Step | Result |
|---|---|---|
| 1 | Install from the ISO, real day-0 | **PASS** |
| 2 | Register and claim | **PASS** |
| 3 | Controller is the new one, from the box | **PASS** |
| 4 | Two apps deployed, one with a database | **PASS** |
| 5 | Discriminating data seeded | **PASS** |
| 6 | Back up, restore with `app.yaml` moved aside | **PASS** |
| 7 | Data readable by the app over its real path | **PASS** |

**Step 1 — install.** Unattended from boot to provisioned in **~10 minutes**. PVE 9.2.2 installed,
first-boot stub laid down the retry unit, the box registered itself as an unclaimed appliance
(pairing code `HTG-N8D`), was bound to customer `sess-g` on the hub, and the delivery landed on the
next poll. **The installer that actually ran: `felhom-host-install v1.22.0`** — fetched at run time,
six days newer than the ISO. That is Part 0's claim, proven on a real box.

    [OK] Day-0 provision SUCCESS — vmid=9201 host_id=sess-g-e687c3 customer=sess-g
         golden=local:backup/vzdump-lxc-9100-2026_07_31-07_40_20.tar.zst

Secrets scrubbed on success: `/etc/felhom/` retained only `.bootstrap-done` and the non-secret
pairing code; `bootstrap.env` and the appliance token were shredded.

**Step 2 — register and claim.** Host enrolled, wg peer registered, host-report accepted, DR-recipe
stored. The claim gate was armed from first boot and the claim was a **real** one — the code exists
only in the customer's email (`R-119`; the operator relayed it). The positive discriminator flipped:

    before:  {"ok":false,"error":"dashboard not yet claimed"}
    after:   {"ok":false,"error":"authentication required"}
    hub:     [INFO] [claim] customer sess-g CLAIMED its dashboard (password set by the customer)

**Step 3 — the controller, from the box, not from the manifest:**

    /etc/felhom-controller-image : gitea.dooplex.hu/admin/felhom-controller:0.188.0
    docker ps                    : gitea.dooplex.hu/admin/felhom-controller:0.188.0 | Up (healthy)
    felhom-agent --version       : felhom-agent 0.118.1

**Step 4 — two apps, through the real endpoints** (`GET /stacks/{app}/deploy` for the pre-generated
values, then `POST /api/stacks/{app}/deploy`, the endpoint the UI invokes):

- **rallly** — `lukevella/rallly:4.11.1` + `postgres:16-alpine`, the DB app. Chosen because its
  `DB_PASSWORD` is `type: secret`, i.e. exactly D5's travelling class, and its `DATABASE_URL` is a
  TCP URL to a *different container*: `postgresql://rallly@rallly-postgres:5432/rallly`.
- **homebox** — sqlite, no DB container.

Both healthy; rallly served 200 through traefik on its own route.

**Step 5 — the discriminator.** `cc_proof` table, one row **before** the backup
(`PRE-BACKUP-sess-g-2026-07-31`), one **after** (`POST-BACKUP-sess-g-MUST-NOT-RETURN`). Without this
pair a stale restore is indistinguishable from a good one.

**The read path, established and red-proofed before it was trusted.** All DB access ran from
**rallly's own network namespace** (`docker run --network container:rallly … psql -h rallly-postgres`)
— the path `DATABASE_URL` names, never `127.0.0.1` inside the postgres container, which postgres
trusts and which produced D5's §1.2 false pass. Proven to be doing real work:

    correct password : rallly|t|t                    (user=rallly, over_tcp=t, PostgreSQL 16)
    wrong password   : FATAL: password authentication failed for user "rallly"

**Step 6 — backup, then restore with the guest gone.**

    [backup] Recovery unit captured for rallly → /mnt/sys_drive/felhom-data/backups/primary/rallly
             (images=2, secrets-referenced=2, data_keys=0, portable-carried=2/2, withheld=0)
    [backup] Volume dump: rallly/rallly_rallly_postgres_data → 65.0 MB

Unit `compose/app.yaml`: mode **0600**, header naming the carried class, both secrets present.
Manifest `schema_version: 2`, `controller_version: 0.188.0`. The carried `DB_PASSWORD` was verified
**by fingerprint, never printed**, against the live one:

    unit  DB_PASSWORD sha256=14c8f5152b205829
    live  DB_PASSWORD sha256=14c8f5152b205829

and the guest's own copy was `ENC:` at rest. The guest's `app.yaml` was then **moved aside** —
confirmed absent — and the restore run through `POST /backup/restore`:

    [backup] Restoring rallly from recovery unit: images=2, secrets recovered=2/2, data_keys=0
    [backup] Restored 1 Docker volume(s) for rallly
    [backup] Restore rallly: replaying DB dump into rallly-postgres (postgres)
    [backup] Restore-from-unit completed: rallly            (30.97 s)

**2/2 with the guest's app.yaml gone.**

**Step 7 — the application reading its own data.** The **live app process** was asked for its own
credential, and it matches the pre-backup one — so the secret genuinely survived the guest:

    host=rallly-postgres port=5432 user=rallly pw_sha256=14c8f5152b205829

Then the read, over that path:

    current_user | server_ip     | over_tcp
    rallly       | 172.19.0.2/32 | t

    id | marker
    1  | PRE-BACKUP-sess-g-2026-07-31

    PRE-BACKUP rows  = 1
    POST-BACKUP rows = 0

**The discriminator held.** 32 `public` tables intact; the guest's `app.yaml` was regenerated from the
unit with both secrets re-encrypted at rest; rallly's own healthcheck returned **200**, and traefik
served `poll.sess-g.test` **200**. Checked the data, not the exit code.

One difference from D5's adventurelog run worth recording: this unit carried **both** a `.sql` dump
and the volume tar, and the restore did both (volume restored, then dump replayed). D5's case had
`db_dumps: None`. The password mattered either way — the replay authenticates with it.

## 6. Part 3 — the runbook integrity fix

`RUNBOOK-manual-build.md:104-106` told the reader to *"use its RECORDED qemu launch line; do not
reconstruct from memory"*, citing `RUNBOOK-publish-0.79-0.110-2026-07-10.md:29` — a line labelled
**"PASS (reconstructed — DEVIATION)"** whose own §Deviations says *"the canonical `qemu-system`
one-liner was never saved"*. The document forbade improvising and pointed at an improvisation.

**Captured verbatim from this bake and recorded as canonical** in `RUNBOOK-manual-build.md` §4.0:

```bash
qemu-img snapshot -a virgin /mnt/5_hdd/felhom.eu/drill/drill.qcow2
qemu-system-x86_64 -enable-kvm -cpu host -smp 4 -m 8192 \
  -drive file=/mnt/5_hdd/felhom.eu/drill/drill.qcow2,format=qcow2,if=virtio,cache=writeback \
  -netdev user,id=n0,dhcpstart=10.0.2.30,hostfwd=tcp::2222-10.0.2.15:22 \
  -device virtio-net-pci,netdev=n0 -display none -daemonize \
  -pidfile /mnt/5_hdd/felhom.eu/drill/qemu.pid
```

SSH on `:2222` in ~40 s; `pve-manager/9.2.2`. §4 also now carries the bake/publish/teardown steps and
the template-rot warning, and the old runbook's deviation entry is marked SUPERSEDED with a forward
pointer so the reconstruction is never used as an authority again.

## 7. Teardown — all three layers

1. **The machine.** VM 310 stopped and `qm destroy --purge --destroy-unreferenced-disks 1`.
   `qm list` afterwards shows only `drill-r50` (300), untouched; `/mnt/nvme-1tb/images/` is empty.
2. **The host.** `cc-scratch` removed; `storage.cfg` back to its original four entries.
   `felhom-backup` available **926 492 284 KiB before the run and 926 492 284 KiB after** — the space
   returned exactly.
3. **The hub. `sess-g` and its host record are DELETED — the full cascade ran.** See §7.1.

### 7.1 Layer 3, stated explicitly

The delete gate **refused twice before it allowed**, which is the gate working, and both refusals are
recorded because "I deleted it" without them would hide that this needs a wait:

    POST /hosts/sess-g-e687c3/delete           → 409 Conflict   (host ONLINE)
    POST /configs/sess-g/delete (bare confirm) → 400            (all three acknowledgements required)

`customer_delete.go:220-227` refuses while `hostStatus(...) == "ok"`, so the record cannot be deleted
until the destroyed box ages out of ONLINE (~30 min after its last report). It flipped `ok → stale` at
08:25, and the cascade then ran:

    POST /configs/sess-g/delete
      ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-g expect_hosts=1
    → 303 /configs?flash=deleted

    [INFO] customer DELETE cascade started for sess-g (journal #8, 1 host(s))
    [INFO] delete sess-g: host sess-g-e687c3 deleted (escrow DEMOTED to retained custody)
    [INFO] tenantsync: deprovision ok for sess-g (ns=sess-g, existed=false)
    [INFO] delete sess-g: residue purged (reports=5 app_telemetry=7 notif_prefs=1
           selfbind_tokens=1 appliance_registrations=1)
    [INFO] customer DELETE cascade COMPLETE for sess-g (journal #8) — full teardown

**Verified positively, not by absence** — both pages loaded with real content and returned zero hits,
and the unclaimed-appliance row is gone by its own UUID:

    /configs  loaded 10160 bytes, occurrences of 'sess-g' = 0
    /hosts    loaded  9880 bytes, occurrences of 'sess-g' = 0
    /hosts    occurrences of appliance uuid f9f5d7e0-…            = 0

**A polling trap, recorded because it nearly produced a false result here.** The first delete-gate
poll loop broke on `grep -q '"status":"ok"' || break` — and an **empty** response satisfies that just
as well as a status change. It fired on one transient empty reply and reported "GATE OPEN" while the
host was still ONLINE. Re-run parsing the JSON and treating unparseable as *retry*, the host was
`ok` for another four minutes. Standing rule 3, in miniature: the absent signal is not the positive one.

**`sess-f` is NOT deleted** — it belongs to the 2026-07-30 session, not this run (R-131). Command for
whoever picks it up, once its host is not ONLINE:

    POST /configs/sess-f/delete
      ack_hosts=1 ack_reset=1 ack_purge=1 confirm_id=sess-f expect_hosts=<live host count>

**Secrets.** Shredded in the guest (`rallly-dbpw`, the moved-aside `app.yaml`, deploy bodies, session
and CSRF files, claim artefacts) and on the box; helper scripts removed from both. The scratch
customer's retrieval passphrase and the ISO's throwaway root password were handled file→file and
never printed; both die with the session scratchpad and the destroyed VM.

## 8. Findings — filed, none fixed

- **R-128** — `build-felhom-iso.sh:44` claims `ISO_VERSION` "aligns with felhom-host-install
  SCRIPT_VERSION". ISO 1.25.0, host-install 1.22.0. A comment asserting an invariant nothing
  maintains. Cosmetic; it misleads a reader about which installer an ISO implies.
- **R-129** — the docs say demo-hp has **"no baked SSH key"** and needs the G1 break-glass password
  (`CLAUDE.md`, `target-selection.md:41-42`, `felhom-agent/CLAUDE.md`). `ssh -o BatchMode=yes demo-hp`
  authenticated by **key**, first try. The prohibition is stale in a direction that costs a future
  session a needless vault round-trip.
- **R-130** — a fresh box's `local-lvm` was ~75 GiB against `HARD_MIN_LVM_GIB=120`, and the installer
  logged `[WARN] local-lvm free ~75 GiB < hard min 120 GiB` and **continued to a fully successful
  install**. Either the hard minimum is not hard, or it is wrong. A "hard min" that warns is the
  R-29 shape.
- **R-131 (housekeeping)** — hub customer **`sess-f`** ("R-120 golden 0.186.0 proof", DOWN) is a
  scratch customer left behind by the 2026-07-30 session — the fourth instance of the accumulation
  `target-selection.md:86-87` and `PROMPT-TEMPLATE.md` §13 warn about (`drill-r50`, `sess-c`,
  `sess-d`, now `sess-f`). Not deleted here: it is not this run's record. Command in §7.1.
- **R-132 — `curl -w '%{redirect_url}'` reconstructs the request URL WITH its basic-auth credential.**
  While red-proofing the R-120 gate, a call made correctly with `-u ":$HUB_PW"` (no credential in any
  URL) still printed the hub operator password into this session's transcript, because the *write-out
  format* rebuilds the URL from the credential curl was given. The request was safe; the reporting was
  not. **`HUB_PW` should be rotated.** Rule going forward: read a redirect from `-D -` and
  `grep '^Location:'`, never `%{redirect_url}`, on any authenticated call.
- **R-119 confirmed live**, not re-filed: the claim code is emailed-only with no operator-side route,
  and it is the single human step in an otherwise unattended day-0. Budget for it before a run.

## 9. §7.2 — the sentence that matters

**Yes — a fresh install is safe to hand to an external tester.** From ISO boot to a claimable,
app-serving box was ~10 minutes unattended; the box fetched the current installer and the newly
vouched golden and verified both by sha; two apps deployed through the real endpoints; and an app's
data came back from the drive with the guest's `app.yaml` gone, readable by the application over its
own TCP path, with a discriminator proving it was a restore and not a no-op.

**One thing the tester's host must do that the operator must plan for:** the claim code arrives only
by email (R-119), so the tester needs a working mailbox on the customer record before the box boots.
Nothing else found here blocks a tester; R-128/R-129/R-130 are documentation and gate-hygiene issues,
and R-131 is housekeeping.

**What "safe" does not mean:** bug-free. Two things this run did **not** cover — the offsite/PBS-DR
legs (deliberately: the task forbids pointing anything at production PBS or the real restic offsite,
so `sess-g` was created with DR tier off and offsite disabled), and any bare-metal ISO. Only a
`nested-vm`-profile v1.25.0 ISO exists on disk; a tester on real hardware needs a `generic`- or
`hp`-profile build from the same tooling, which is an ISO build and therefore its own task.

## 10. What did not happen, and why

- **No ISO built** — Part 0 said none was needed, and the brief says to stop and report if one were.
- **No defect fixed** — R-128/129/130/131 filed only; a fix mid-run would describe a version that
  never existed.
- **No existing box changed** — no golden deployed to 9201 on either demo host, no floor moved
  (still v0.156.0), no MinAgent change (still 0.113.0).
- **Campaign 10, the demo-hp repartition, subdomain onboarding** — out of scope, untouched.
- **`sess-f` not deleted** — see R-131.