diff --git a/documentation/tests/golden-0.205.0-2026-08-07/README.md b/documentation/tests/golden-0.205.0-2026-08-07/README.md new file mode 100644 index 0000000..81431ab --- /dev/null +++ b/documentation/tests/golden-0.205.0-2026-08-07/README.md @@ -0,0 +1,118 @@ +# Golden 0.205.0 — bake evidence (2026-08-07) + +Baked per `runbooks/RUNBOOK-manual-build.md` §4.0–§4.1, in the drill VM on DooPlex +(`/mnt/5_hdd/felhom.eu/drill/drill.qcow2`, internal snapshot `virgin`) — the sanctioned bake exception +in `runbooks/target-selection.md`. **R-239.** + +**Baked, published and verified. NOT vouched** — the hub's Day-0 artifacts were read but not written. +That is deliberate: this part ends at a STOP, and vouching is the operator's act. + +## The artifact + +| | | +|---|---| +| version | **0.205.0** | +| sha256 | `8f49b2e8ccbc86a49df821fee9fb00c07293758811d3d0f0512dd0cf5fd54ee8` | +| size | **656,937,561** bytes (uncompressed 2,003,343,360) | +| controller baked in | `gitea.dooplex.hu/admin/felhom-controller:0.205.0` | +| MinAgent (from the controller `CHANGELOG.md` header) | **0.127.0** | +| published at | `…/api/packages/admin/generic/felhom-golden/0.205.0/golden.tar.zst` | +| golden archive volid (in-VM, destroyed with the CT) | `vzdump-lxc-9100-2026_08_07-09_00_40.tar.zst` | +| `build-golden.sh` version | 3.0.0 (R-165 — one data volume, **there is no `mp1`**) | + +## Preconditions, checked rather than assumed + +- **Clean-tree gate**, both source repos: `felhom-controller` HEAD == `origin/main` == + `c6b69d888e2b`, tree clean; `felhom-agent` (which owns `build-golden.sh`) HEAD == `origin/main` == + `703db166e762`, tree clean. +- Disk headroom `df -h /mnt/5_hdd /` → **24%** and **50%**. (Abort threshold is 90%.) +- `ps -eo comm | grep -c qemu-system-x86` → **0**. Not `pgrep -f`, which self-matches and reports a + false "still running". +- `qemu-img snapshot -a virgin` **succeeded**, which independently re-proves nothing held the qcow2. +- Template **listed, not assumed**: `pveam available --section system | grep debian-13` → + `debian-13-standard_13.6-1_amd64.tar.zst` (unchanged since 2026-07-31), and `pveam download` printed + `checksum verified`. +- CT 9100 **absent** on `virgin` before the run; `pct list` empty. +- Token verified against the Gitea API **before** starting (`/api/v1/user` → 200), so a credential + problem could not surface mid-bake. Token file `0600`. +- **404 pre-gate** on the real publish URL → **404** (the R-233 URL shape: `…/0.205.0/golden.tar.zst`, + not `felhom-golden-.tar.zst`). +- Controller image `0.205.0` manifest → **200**, so the tag being baked demonstrably exists. +- Drill VM read `pve-manager/9.2.2` — unchanged from the 0.203.0 bake. + +## Acceptance markers + +From `bake.log` in this directory. The runbook's original marker list was broken (R-233); these are +the real strings. + +| marker | count | required | +|---|---|---| +| `docker OK (overlay2` | 1 | ≥1 | +| `including mount point` | 2 — `rootfs ('/')` and `mp0 ('/var/lib/felhom')` | rootfs + mp0 | +| `upload OK (HTTP 201)` | 1 | ≥1 | +| `excluding` | 0 | 0 | +| `FATAL` | 0 | 0 | +| `ERROR:` / `WARN:` | 0 / 0 | 0 | +| unit result | `Result=success`, `ExecMainStatus=0` | success / 0 | + +Infra images baked (4, from the controller's own declaration): `traefik:v3.6.7`, +`cloudflare/cloudflared:2026.6.0`, `gtstef/filebrowser:1.3.3-stable`, +`gitea.dooplex.hu/admin/felhom-samba:1.1.0`. + +## The published bytes were verified by round trip, not trusted + +The bake **prints** a hash; that is the baker marking its own homework. Recomputed on DooPlex from an +independent download, and the content then read out of that download: + +1. `curl` the package → **HTTP 200**, **656,937,561** bytes. +2. `sha256sum` → `8f49b2e8ccbc86a4…4ee8` — **matches** the baked value; byte count matches. +3. `zstd -t` → decompresses clean to **2,003,343,360** bytes: a real archive, not an error page. +4. `tar -xO ./etc/felhom-controller-image` **out of the published archive** → + `gitea.dooplex.hu/admin/felhom-controller:0.205.0`. +5. **19,358** entries under `var/lib/felhom/docker` — the baked images are on the data volume. + +**Step 4 is the one that matters:** `GOLDEN_VERSION` is *derived from the tag argument*, so the label +could be right while the content was stale. It is not — the archive says so itself. + +## Secret hygiene + +- Token copied **file → file** (`scp`), never through a shell on either side. +- Invocation in an **in-VM runner script that reads the token itself**, so the value never reached a + command line. Verified: `systemctl show golden-bake -p Environment -p ExecStart | grep -c -F ` + → **0**. +- Token-leak grep by **literal value** on the saved log → **0**. A broad `[a-f0-9]{40}` pattern is + wrong here: it false-hits image shas. +- **Positive control on that grep**, because an absence proves nothing until the instrument is shown to + work: the token was appended to a throwaway copy → **1 hit**; the copy was `shred -u`'d; the + committed log's **0** is therefore a measurement. Run against the copy **in this directory** — the + one that gets committed — not only the one in the VM. + +## Teardown + +`pct destroy 9100 --purge` (both logical volumes removed, `pct list` empty) · `shred -u` of the token, +`build-golden.sh`, the runner and `bake.log` **after** copying the log out · `poweroff` · qemu observed +gone via `ps -eo comm` · `qemu-img snapshot -a virgin` (confirmed: `virgin`, 0 B VM_SIZE). The drill VM +is back at its baseline and DooPlex is unchanged apart from this evidence directory. + +## What is NOT done here — the STOP + +**Vouching.** The hub still serves golden **0.203.0** (sha `3039c6ff…dc88`), agent **0.127.0**, +`min_agent` **0.127.0**. + +**It is a three-field change, and this time only one field moves** — read from the live +`hub_settings`, not assumed: + +| field | now | after | changes? | +|---|---|---|---| +| `artifact_golden_version` | `0.203.0` | `0.205.0` | **YES** | +| `artifact_golden_sha256` | `3039c6ff…dc88` | `8f49b2e8…4ee8` | **YES** (derived from the above) | +| `artifact_agent_version` | `0.127.0` | `0.127.0` | **no** — already ≥ the new golden's MinAgent | +| `artifact_min_agent` | `0.127.0` | `0.127.0` | **no** — the new golden's MinAgent is also 0.127.0 | + +**The R-120 gate will pass, and only just.** It refuses a golden older than the newest controller the +fleet reports; the newest reported is **0.205.0** (demo-hp, 2026-08-07 06:29:52). A 0.204.0 golden +would have been **refused**. + +**Vouching is reversible**: re-select the previous values and Save. A bake never deletes an older +golden's package (the publish step's pre-delete targets only its own version), so rolling back is a +form submission, not a rebuild. diff --git a/documentation/tests/golden-0.205.0-2026-08-07/bake.log b/documentation/tests/golden-0.205.0-2026-08-07/bake.log new file mode 100644 index 0000000..f389745 --- /dev/null +++ b/documentation/tests/golden-0.205.0-2026-08-07/bake.log @@ -0,0 +1,326 @@ +[golden] build-golden.sh v3.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.205.0 +[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) … + Logical volume "vm-9100-disk-0" created. + Logical volume pve/vm-9100-disk-0 changed. +Creating filesystem with 8388608 4k blocks and 2097152 inodes +Filesystem UUID: 6c3bbf8e-c1f6-4a3f-a65a-c51c20cb030a +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, + 4096000, 7962624 + Logical volume "vm-9100-disk-1" created. + Logical volume pve/vm-9100-disk-1 changed. +Creating filesystem with 6291456 4k blocks and 1572864 inodes +Filesystem UUID: 0ee4d9af-888b-436c-a88f-91182e4b32f6 +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, +extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst' +Total bytes read: 553512960 (528MiB, 158MiB/s) +Detected container architecture: amd64 +Creating SSH host key 'ssh_host_rsa_key' - this may take some time ... +done: SHA256:7Jbe+Jf04bN6BizuXu5tsmLRdS2/mWlY7RWXtAydAj8 root@felhom-golden +Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ... +done: SHA256:sA2Edp2ejriDHUZl/IPmri27t35Lwo7Y7gFaF7yKGe4 root@felhom-golden +Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ... +done: SHA256:Dz85PydhjxVQYSOHk4MLYfDn3h9HIKKaSVJsAc4U2/A root@felhom-golden +[golden] starting + installing Docker (official repo, trixie channel) … +apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = (unset), + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to the standard locale ("C"). +locale: Cannot set LC_CTYPE to default locale: No such file or directory +locale: Cannot set LC_MESSAGES to default locale: No such file or directory +locale: Cannot set LC_ALL to default locale: No such file or directory +apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = (unset), + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to the standard locale ("C"). +locale: Cannot set LC_CTYPE to default locale: No such file or directory +locale: Cannot set LC_MESSAGES to default locale: No such file or directory +locale: Cannot set LC_ALL to default locale: No such file or directory +[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation … +[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds … +[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) … +Unable to find image 'hello-world:latest' locally +latest: Pulling from library/hello-world +4f55086f7dd0: Pulling fs layer +4f55086f7dd0: Verifying Checksum +4f55086f7dd0: Download complete +4f55086f7dd0: Pull complete +Digest: sha256:7f4da0fc94bcece205a8c0b6f4d11c8196924654ffe5c4d1aa439b7f632048b2 +Status: Downloaded newer image for hello-world:latest + docker OK (overlay2; data-root /var/lib/docker) + /var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4 + /mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4 + both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576 +[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.205.0 (no registry cred at deploy) … + +WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'. +Configure a credential helper to remove this warning. See +https://docs.docker.com/go/credential-store/ + +0.205.0: Pulling from admin/felhom-controller +039e6f9f9752: Pulling fs layer +80fbb63a1167: Pulling fs layer +9d6c7a1331cb: Pulling fs layer +3f0ff4614786: Pulling fs layer +a55e015e5c97: Pulling fs layer +138d57035e40: Pulling fs layer +3f0ff4614786: Waiting +a55e015e5c97: Waiting +138d57035e40: Waiting +039e6f9f9752: Verifying Checksum +039e6f9f9752: Download complete +9d6c7a1331cb: Verifying Checksum +9d6c7a1331cb: Download complete +3f0ff4614786: Verifying Checksum +3f0ff4614786: Download complete +a55e015e5c97: Verifying Checksum +a55e015e5c97: Download complete +80fbb63a1167: Verifying Checksum +80fbb63a1167: Download complete +138d57035e40: Verifying Checksum +138d57035e40: Download complete +039e6f9f9752: Pull complete +80fbb63a1167: Pull complete +9d6c7a1331cb: Pull complete +3f0ff4614786: Pull complete +a55e015e5c97: Pull complete +138d57035e40: Pull complete +Digest: sha256:c9ef38a5de82a8bb25d82258d4558fea3337dc41832bfeadf56a9a8aa9a41538 +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.205.0 +gitea.dooplex.hu/admin/felhom-controller:0.205.0 +[golden] asking the controller which infra images it manages … +[golden] baking infra images (4): traefik:v3.6.7 cloudflare/cloudflared:2026.6.0 gtstef/filebrowser:1.3.3-stable gitea.dooplex.hu/admin/felhom-samba:1.1.0 … +v3.6.7: Pulling from library/traefik +589002ba0eae: Pulling fs layer +ef63511ea6cc: Pulling fs layer +0738e5cb835e: Pulling fs layer +3e6813f70c64: Pulling fs layer +3e6813f70c64: Waiting +589002ba0eae: Verifying Checksum +589002ba0eae: Download complete +3e6813f70c64: Verifying Checksum +3e6813f70c64: Download complete +589002ba0eae: Pull complete +ef63511ea6cc: Verifying Checksum +ef63511ea6cc: Download complete +ef63511ea6cc: Pull complete +0738e5cb835e: Verifying Checksum +0738e5cb835e: Download complete +0738e5cb835e: Pull complete +3e6813f70c64: Pull complete +Digest: sha256:a9890c898f379c1905ee5b28342f6b408dc863f08db2dab20e46c267d1ff463a +Status: Downloaded newer image for traefik:v3.6.7 +docker.io/library/traefik:v3.6.7 +2026.6.0: Pulling from cloudflare/cloudflared +47de5dd0b812: Pulling fs layer +c172f21841df: Pulling fs layer +99515e7b4d35: Pulling fs layer +99ba982a9142: Pulling fs layer +d6b1b89eccac: Pulling fs layer +2780920e5dbf: Pulling fs layer +7c12895b777b: Pulling fs layer +3214acf345c0: Pulling fs layer +52630fc75a18: Pulling fs layer +dd64bf2dd177: Pulling fs layer +b839dfae01f6: Pulling fs layer +ebddc55facdc: Pulling fs layer +bdfd7f7e5bf6: Pulling fs layer +2d4d7adf6272: Pulling fs layer +40008157d8d2: Pulling fs layer +bd8962e29291: Pulling fs layer +cac2ae0193cb: Pulling fs layer +74d1dac84ecc: Pulling fs layer +3214acf345c0: Waiting +52630fc75a18: Waiting +dd64bf2dd177: Waiting +b839dfae01f6: Waiting +ebddc55facdc: Waiting +bdfd7f7e5bf6: Waiting +2d4d7adf6272: Waiting +40008157d8d2: Waiting +bd8962e29291: Waiting +cac2ae0193cb: Waiting +74d1dac84ecc: Waiting +d6b1b89eccac: Waiting +2780920e5dbf: Waiting +7c12895b777b: Waiting +99ba982a9142: Waiting +c172f21841df: Download complete +47de5dd0b812: Verifying Checksum +47de5dd0b812: Download complete +99515e7b4d35: Verifying Checksum +99515e7b4d35: Download complete +99ba982a9142: Download complete +d6b1b89eccac: Verifying Checksum +d6b1b89eccac: Download complete +2780920e5dbf: Verifying Checksum +2780920e5dbf: Download complete +47de5dd0b812: Pull complete +7c12895b777b: Verifying Checksum +7c12895b777b: Download complete +3214acf345c0: Verifying Checksum +3214acf345c0: Download complete +52630fc75a18: Verifying Checksum +52630fc75a18: Download complete +dd64bf2dd177: Verifying Checksum +dd64bf2dd177: Download complete +b839dfae01f6: Verifying Checksum +b839dfae01f6: Download complete +ebddc55facdc: Verifying Checksum +ebddc55facdc: Download complete +bdfd7f7e5bf6: Download complete +c172f21841df: Pull complete +2d4d7adf6272: Verifying Checksum +2d4d7adf6272: Download complete +bd8962e29291: Verifying Checksum +bd8962e29291: Download complete +40008157d8d2: Verifying Checksum +40008157d8d2: Download complete +cac2ae0193cb: Verifying Checksum +cac2ae0193cb: Download complete +74d1dac84ecc: Verifying Checksum +74d1dac84ecc: Download complete +99515e7b4d35: Pull complete +99ba982a9142: Pull complete +d6b1b89eccac: Pull complete +2780920e5dbf: Pull complete +7c12895b777b: Pull complete +3214acf345c0: Pull complete +52630fc75a18: Pull complete +dd64bf2dd177: Pull complete +b839dfae01f6: Pull complete +ebddc55facdc: Pull complete +bdfd7f7e5bf6: Pull complete +2d4d7adf6272: Pull complete +40008157d8d2: Pull complete +bd8962e29291: Pull complete +cac2ae0193cb: Pull complete +74d1dac84ecc: Pull complete +Digest: sha256:ba461b8aa9c042156dbd39c38657fe7431bafa063220eab8d5330a523863da9f +Status: Downloaded newer image for cloudflare/cloudflared:2026.6.0 +docker.io/cloudflare/cloudflared:2026.6.0 +1.3.3-stable: Pulling from gtstef/filebrowser +6a0ac1617861: Pulling fs layer +ef8806083e82: Pulling fs layer +b74107c861c7: Pulling fs layer +adc935def003: Pulling fs layer +4f4fb700ef54: Pulling fs layer +18695ccc900a: Pulling fs layer +45d119d5c397: Pulling fs layer +dac52db4fc51: Pulling fs layer +6d598f86b2f2: Pulling fs layer +8aa349c8396c: Pulling fs layer +dac52db4fc51: Waiting +6d598f86b2f2: Waiting +8aa349c8396c: Waiting +4f4fb700ef54: Waiting +18695ccc900a: Waiting +45d119d5c397: Waiting +adc935def003: Waiting +6a0ac1617861: Verifying Checksum +6a0ac1617861: Download complete +adc935def003: Verifying Checksum +adc935def003: Download complete +4f4fb700ef54: Verifying Checksum +4f4fb700ef54: Download complete +ef8806083e82: Verifying Checksum +ef8806083e82: Download complete +b74107c861c7: Verifying Checksum +b74107c861c7: Download complete +45d119d5c397: Verifying Checksum +45d119d5c397: Download complete +dac52db4fc51: Verifying Checksum +dac52db4fc51: Download complete +6d598f86b2f2: Verifying Checksum +6d598f86b2f2: Download complete +18695ccc900a: Verifying Checksum +18695ccc900a: Download complete +6a0ac1617861: Pull complete +8aa349c8396c: Verifying Checksum +8aa349c8396c: Download complete +ef8806083e82: Pull complete +b74107c861c7: Pull complete +adc935def003: Pull complete +4f4fb700ef54: Pull complete +18695ccc900a: Pull complete +45d119d5c397: Pull complete +dac52db4fc51: Pull complete +6d598f86b2f2: Pull complete +8aa349c8396c: Pull complete +Digest: sha256:eb3733681db8757412632c61a99ad656f0d94ed6781bb2ea114b4d70babab78c +Status: Downloaded newer image for gtstef/filebrowser:1.3.3-stable +docker.io/gtstef/filebrowser:1.3.3-stable +1.1.0: Pulling from admin/felhom-samba +897d797d2723: Pulling fs layer +3051591aa250: Pulling fs layer +ce57a3f93416: Pulling fs layer +fb94eeec2fe1: Pulling fs layer +fb94eeec2fe1: Waiting +ce57a3f93416: Verifying Checksum +ce57a3f93416: Download complete +897d797d2723: Verifying Checksum +897d797d2723: Download complete +fb94eeec2fe1: Verifying Checksum +fb94eeec2fe1: Download complete +897d797d2723: Pull complete +3051591aa250: Verifying Checksum +3051591aa250: Download complete +3051591aa250: Pull complete +ce57a3f93416: Pull complete +fb94eeec2fe1: Pull complete +Digest: sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10 +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.1.0 +gitea.dooplex.hu/admin/felhom-samba:1.1.0 +[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'. +[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'. +[golden] baking the first-boot SSH host-key regeneration unit (F3) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'. +[golden] identity-clean + minimize … +[golden] stop + archive … +INFO: including mount point rootfs ('/') in backup +INFO: including mount point mp0 ('/var/lib/felhom') in backup +INFO: archive file size: 626MB +INFO: Finished Backup of VM 9100 (00:00:29) +[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_08_07-09_00_40.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive) +[golden] publishing golden (656937561 bytes, sha256 8f49b2e8ccbc86a4…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.205.0/golden.tar.zst +[golden] pre-delete existing: HTTP 404 (404/204 expected) +[golden] upload OK (HTTP 201) +GOLDEN_VERSION=0.205.0 +GOLDEN_SHA256=8f49b2e8ccbc86a49df821fee9fb00c07293758811d3d0f0512dd0cf5fd54ee8 +[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.205.0 / 8f49b2e8ccbc86a49df821fee9fb00c07293758811d3d0f0512dd0cf5fd54ee8 +[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge)