admin
721297ed5e
gates / gates (push) Failing after 10m43s
Controller v0.206.0 shipped the R-241 fixes on 2026-08-07 while the vouched golden still carried 0.205.0, so a machine installed this morning would have received neither - and the next recovery walk would have measured the old behaviour and failed for a reason nothing to do with the walk. Same gap as R-239, one day after R-239 was closed. version 0.206.0 sha256 c85230b42f53baa9c1ee9986ac312c751d6cbc29fbe070d87bb2214429a9108e size 656,750,694 bytes (uncompressed 2,003,138,560) MinAgent 0.127.0 Round-trip verified rather than trusted: the published bytes were fetched back, re-hashed independently (match), zstd-tested, and ./etc/felhom-controller-image was read OUT of the download -> felhom-controller:0.206.0. That last step is the one that matters, because GOLDEN_VERSION is derived from the tag argument and could be right over stale content. All acceptance markers pass; unit Result=success ExecMainStatus=0. Secret hygiene: token file->file, in-VM runner so it never reached a command line (unit-property grep 0), literal-value leak grep on the COMMITTED log 0 - with a positive control proving the grep works before the 0 was believed. Bake VM torn down: CT 9100 purged, secrets shredded, qemu observed gone via ps -eo comm, drill.qcow2 reverted to virgin. THE GATE BUILT EARLIER THIS SESSION NOW PASSES. It was shown CONVICTED against the pre-bake state and is OK now - red to green on the same check, the same command, which is its proof that it measures something real. Note it went green on the BAKE, not the vouch: that is its stated limitation, and the vouch is still pending the operator. NOT VOUCHED - the operator's act. Only one field moves: golden_version 0.205.0 -> 0.206.0 (+ its derived sha). agent_version and min_agent both stay 0.127.0. wrapper_sha256 is unchanged but is CLEARED if omitted from the POST.
121 lines
6.7 KiB
Markdown
121 lines
6.7 KiB
Markdown
# Golden 0.206.0 — bake evidence (2026-08-08)
|
|
|
|
Baked per `runbooks/RUNBOOK-manual-build.md` §4.0–§4.1, in the drill VM on DooPlex
|
|
(`/mnt/5_hdd/felhom.eu/drill/drill.qcow2`, internal snapshot `virgin`) — the sanctioned bake exception
|
|
in `runbooks/target-selection.md`. **R-239 recurring / R-242.**
|
|
|
|
**Baked, published and verified. NOT vouched** — the hub's Day-0 artifacts were read but not written.
|
|
That is deliberate: this part ends at a STOP, and vouching is the operator's act.
|
|
|
|
> **WHY THIS BAKE EXISTS AT ALL.** Controller **v0.206.0** shipped the R-241 fixes on 2026-08-07 while
|
|
> the vouched golden still carried **0.205.0** — so a machine installed on the morning of 2026-08-08
|
|
> would have received neither, and the next recovery walk would have measured the old behaviour and
|
|
> failed for a reason nothing to do with the walk. **This is the same gap as R-239, one day after
|
|
> R-239 was closed**, which is what a rule without a mechanism does. `scripts/golden_currency_gate.py`
|
|
> (built in this session, and shown FAILING against exactly this state before anything was baked) is
|
|
> the mechanism.
|
|
|
|
## The artifact
|
|
|
|
| | |
|
|
|---|---|
|
|
| version | **0.206.0** |
|
|
| sha256 | **`c85230b42f53baa9c1ee9986ac312c751d6cbc29fbe070d87bb2214429a9108e`** |
|
|
| size | **656,750,694** bytes (uncompressed 2,003,138,560) |
|
|
| controller baked in | `gitea.dooplex.hu/admin/felhom-controller:0.206.0` |
|
|
| MinAgent (controller `CHANGELOG.md` header) | **0.127.0** |
|
|
| published at | `…/api/packages/admin/generic/felhom-golden/0.206.0/golden.tar.zst` |
|
|
| in-VM archive volid (destroyed with the CT) | `vzdump-lxc-9100-2026_08_07-12_59_46.tar.zst` |
|
|
| `build-golden.sh` | v3.0.0 (R-165 — one data volume; **there is no `mp1`**) |
|
|
|
|
## Preconditions, checked rather than assumed
|
|
|
|
- **Clean-tree gate on both source repos:** `felhom-controller` HEAD == `origin/main` ==
|
|
`3d3b4496f377` (the image source), `felhom-agent` HEAD == `origin/main` == `703db166e762` (which
|
|
owns `build-golden.sh`). Both trees clean.
|
|
- Disk headroom `df -h /mnt/5_hdd /` → **24%** and **51%**. (Abort threshold 90%.)
|
|
- `ps -eo comm | grep -c qemu-system-x86` → **0**. Not `pgrep -f`, which self-matches.
|
|
- `qemu-img snapshot -a virgin` succeeded — independently re-proving nothing held the qcow2.
|
|
- Template **listed, not assumed**: `pveam available` → `debian-13-standard_13.6-1_amd64.tar.zst`
|
|
(unchanged since 2026-07-31); `pveam download` printed `checksum verified`.
|
|
- CT 9100 **absent** before the run (`pct list` empty).
|
|
- Token validated against the Gitea API **before** starting (`/api/v1/user` → 200); file mode `0600`.
|
|
- **404 pre-gate** on the real publish URL → **404** (the R-233 URL shape:
|
|
`…/0.206.0/golden.tar.zst`, not `felhom-golden-<VER>.tar.zst`).
|
|
- Controller image `0.206.0` manifest → **200**, so the tag being baked demonstrably exists.
|
|
- Drill VM read `pve-manager/9.2.2` — unchanged from the previous two bakes.
|
|
|
|
## Acceptance markers
|
|
|
|
From `bake.log` in this directory.
|
|
|
|
| marker | count | required |
|
|
|---|---|---|
|
|
| `docker OK (overlay2` | 1 | ≥1 |
|
|
| `including mount point` | 2 — `rootfs ('/')` and `mp0 ('/var/lib/felhom')` | rootfs + mp0 |
|
|
| `upload OK (HTTP 201)` | 1 | ≥1 |
|
|
| `excluding` | 0 | 0 |
|
|
| `FATAL` | 0 | 0 |
|
|
| `ERROR:` / `WARN:` | 0 / 0 | 0 |
|
|
| unit result | `Result=success`, `ExecMainStatus=0` | success / 0 |
|
|
|
|
Infra images baked (4, from the controller's own declaration): `traefik:v3.6.7`,
|
|
`cloudflare/cloudflared:2026.6.0`, `gtstef/filebrowser:1.3.3-stable`,
|
|
`gitea.dooplex.hu/admin/felhom-samba:1.1.0`.
|
|
|
|
## The published bytes were verified by round trip, not trusted
|
|
|
|
The bake **prints** a hash; that is the baker marking its own homework.
|
|
|
|
1. `curl` the package → **HTTP 200**, **656,750,694** bytes.
|
|
2. `sha256sum` of the download → **`c85230b4…108e`** — **matches** the baked value; byte count matches.
|
|
3. `zstd -t` → decompresses clean to **2,003,138,560** bytes: a real archive, not an error page.
|
|
4. `tar -xO ./etc/felhom-controller-image` **out of the downloaded archive** →
|
|
`gitea.dooplex.hu/admin/felhom-controller:0.206.0`.
|
|
5. **19,358** entries under `var/lib/felhom/docker` — the baked images are on the data volume.
|
|
|
|
**Step 4 is the one that matters:** `GOLDEN_VERSION` is *derived from the tag argument*, so the label
|
|
could be right while the content was stale. It is not — **the archive says so itself**.
|
|
|
|
## Secret hygiene
|
|
|
|
- Token copied **file → file** (`scp`); never through a shell on either side.
|
|
- Invocation in an **in-VM runner script that reads the token itself**, so it never reached a command
|
|
line. Verified: `systemctl show golden-bake -p Environment -p ExecStart | grep -c -F <token>` → **0**.
|
|
- Token-leak grep by **literal value** on the committed log → **0**. A broad `[a-f0-9]{40}` pattern is
|
|
wrong here — it false-hits image shas.
|
|
- **Positive control**, because an absence proves nothing until the instrument is shown to work: the
|
|
token was appended to a throwaway copy → **1 hit**; the copy was `shred -u`'d; the committed log's
|
|
**0** is therefore a measurement. Run against the copy **in this directory**, the one that is
|
|
committed.
|
|
|
|
## Teardown
|
|
|
|
`pct destroy 9100 --purge` (both logical volumes removed, `pct list` empty) · `shred -u` of the token,
|
|
`build-golden.sh`, the runner and `bake.log` **after** copying the log out · `poweroff` · qemu observed
|
|
gone via `ps -eo comm` · `qemu-img snapshot -a virgin` (confirmed: `virgin`, 0 B VM_SIZE). The drill VM
|
|
is back at its baseline and DooPlex is unchanged apart from this evidence directory.
|
|
|
|
## What is NOT done here — the STOP
|
|
|
|
**Vouching.** The hub still serves golden **0.205.0** (sha `8f49b2e8…4ee8`), agent **0.127.0**,
|
|
`min_agent` **0.127.0**.
|
|
|
|
**It is a three-field change, and as on 2026-08-07 only one field moves** — read from the live
|
|
`hub_settings`, not assumed:
|
|
|
|
| field | now | after | changes? |
|
|
|---|---|---|---|
|
|
| `artifact_golden_version` | `0.205.0` | `0.206.0` | **YES** |
|
|
| `artifact_golden_sha256` | `8f49b2e8…4ee8` | `c85230b4…108e` | **YES** (derived from the above) |
|
|
| `artifact_agent_version` | `0.127.0` | `0.127.0` | **no** — already ≥ the new golden's MinAgent |
|
|
| `artifact_min_agent` | `0.127.0` | `0.127.0` | **no** — the new golden's MinAgent is also 0.127.0 |
|
|
| `artifact_wrapper_sha256` | `104db0a4…16b3` | unchanged | **no** — but it is **cleared if omitted** from the POST |
|
|
|
|
**Vouching is reversible**: re-select the previous values and Save. A bake never deletes an older
|
|
golden's package (the publish step's pre-delete targets only its own version), so rolling back is a
|
|
form submission, not a rebuild.
|
|
|
|
**After the vouch, `scripts/golden_currency_gate.py` should go green** — the same check, the same
|
|
command. That transition is the gate's proof that it measures something real.
|