golden 0.216.0: baked, published, vouched — gates green again

Closes the two-release day-0 gap that has been convicting CI since 2026-08-14. Run against RUNBOOK-manual-build.md 4.0 + 4.1. GOLDEN_VERSION = 0.216.0 GOLDEN_SHA256 = ac004dc90d8cefccc5448377892f9cff3a4c3e1e27d0e11129120e38ac31c34b archive = 656,970,239 bytes, controller image 0.216.0 template = debian-13-standard_13.6-1_amd64.tar.zst (listed live, not reused) Baselines re-read on the machine and all four matched the sheet: controller v0.216.0, its MinAgent 0.129.0, agent v0.129.0, previous golden 0.214.0. The published agent artifact for the vouched agent_version was confirmed present in the package registry rather than inferred from a CHANGELOG, and the R-216 check passed on the machine: MinAgent is EQUAL to, not above, the newest published agent. Verified beyond the script's own claim: the artifact was downloaded back out of Gitea and hashed, and it matches GOLDEN_SHA256 exactly. A script printing a digest and the registry serving those bytes are two different claims. Pass markers (corrected post-R-233 list) all present, quoted with line numbers in pass-markers.txt; excluding/FATAL absent; there is no mp1. Token never reached a command line: copied file->file, read inside the VM by the runner. systemctl show grep = 0. Token-leak grep on the COMMITTED log run with its positive control FIRST -- seeded copy 1, real log 0 -- because a grep -c that matches nothing also returns 0. Teardown: guest destroyed and purged, token/runner/script/log shredded AFTER the log was copied out, qemu exit confirmed with ps -eo comm (not pgrep -f), disk reverted to virgin. Vouched by the operator; verified by reading the hub's own store: golden 0.216.0 / agent 0.129.0 / min_agent 0.129.0, and the hub's recorded sha256 matches the independently downloaded artifact. That check was necessary because golden_currency_gate.py says of itself that it checks the BAKE, not the vouch. repo_gates.py --fast now rc=0, all nine gates OK -- first fully green run since 2026-08-14. Capability map deliberately NOT changed: the day-0 row cites drill documents, and the map's only golden literal is a dated historical citation on the recovery-journey row which bumping would falsify. R-334 is closed in a follow-up commit quoting this push's CI run id, since closing it without one would leave the ambiguity a third time.
2026-08-18 13:04:37 +02:00
parent 1b4d005f80
commit 7d81681d6e
8 changed files with 641 additions and 2 deletions
@@ -0,0 +1,151 @@
 # REPORT — bake and vouch golden 0.216.0, closing R-334 (2026-08-18, afternoon)
 **Outcome: R-334 CLOSED.** Golden **0.216.0** baked, published, and **vouched by the operator**.
 `golden_currency_gate.py` is green for the first time since 2026-08-14, and `repo_gates.py` is
 **fully green — all nine gates, rc=0**.
 Run against the existing `documentation/runbooks/RUNBOOK-manual-build.md` §4.0 + §4.1; the run sheet
 pinned this run's numbers and the stop. Evidence:
 `documentation/tests/golden-0.216.0-2026-08-18/`.
 ---
 ## 1. Baselines, re-read on the machine
 | item | value | source |
 |---|---|---|
 | newest released controller | **v0.216.0** | `felhom-controller/CHANGELOG.md` head |
 | its floor | **`MinAgent: 0.129.0`** | second line of that header |
 | newest agent release | **v0.129.0** | `felhom-agent/CHANGELOG.md` head |
 | newest golden before this run | **0.214.0** | `documentation/tests/golden-0.214.0-2026-08-12` |
 **All four match the run sheet's §1 — no disagreement to report.** All three repos were clean with
 `HEAD == origin/main` before starting.
 ## 2. The published agent artifact exists
 Checked against the **package registry**, not inferred from a CHANGELOG:
 `generic felhom-agent 0.129.0` is published. Vouching `agent_version` at a version that was never
 published would point day-0 installs at a 404.
 **The R-216 check passed on the machine rather than on the coincidence.** `MinAgent` (0.129.0) is
 **equal to**, not above, the newest published agent (0.129.0). Had it read higher, hub v0.97.0 would
 hold the fleet against a version nobody has.
 ## 3. Identity, and a verification beyond what was asked
 ```
 GOLDEN_VERSION = 0.216.0
 GOLDEN_SHA256  = ac004dc90d8cefccc5448377892f9cff3a4c3e1e27d0e11129120e38ac31c34b
 URL            = https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.216.0/golden.tar.zst
 archive        = 656,970,239 bytes (rootfs 32G + ONE data volume 24G @ /var/lib/felhom)
 controller     = gitea.dooplex.hu/admin/felhom-controller:0.216.0
 template       = debian-13-standard_13.6-1_amd64.tar.zst  (listed live per §4.1 step 2, not reused)
 ```
 The URL resolves (HTTP 206 on a range request). **I did not stop at the script's printed hash**: the
 artifact was downloaded back out of Gitea and hashed, and it matches `GOLDEN_SHA256` exactly. The
 script reporting a digest and the registry serving those bytes are two different claims, and only the
 second one is what a new install actually receives.
 ## 4. Pass markers — the corrected list, quoted from the real log
 ```
 82 :   docker OK (overlay2; data-root /var/lib/docker)
 313 : INFO: including mount point rootfs ('/') in backup
 314 : INFO: including mount point mp0 ('/var/lib/felhom') in backup
 319 : [golden]   pre-delete existing: HTTP 404 (404/204 expected)
 320 : [golden]   upload OK (HTTP 201)
 ```
 `excluding` and `FATAL`: **absent**. There is no mp1 — R-165 collapsed the two data volumes into one,
 which is exactly why the pre-2026-08-06 marker list could never match and why R-233 rewrote it.
 ## 5. Token handling, and why the control is not ceremony
 Copied **file → file** by `scp`; the runner script inside the VM read it from `/root/.gitea-token`
 itself, so the value never reached a command line or a unit's properties:
 ```
 systemctl show golden-bake -p Environment -p ExecStart | grep -c -F "<token>"   =  0
 ```
 Token-leak grep on the **committed** log, positive control run **first**:
 ```
 seeded throwaway copy  = 1     ← proves the grep can see a token when one is present
 committed bake.log     = 0     ← the real measurement, now worth believing
 ```
 **A `grep -c` that matches nothing returns `0`, which is indistinguishable from a clean file.**
 Without the control, the `0` is an assumption wearing a number's clothes. Both figures are from the
 copy that is committed to the repository, not only the one inside the VM.
 ## 6. Teardown
 `pct destroy 9100 --purge` (both LVs removed, CT purged) → `shred -u` on the token, runner,
 build script and log **after** the log was copied out (standing rule 5) → all four confirmed absent
 → `poweroff` → waited for qemu to exit using `ps -eo comm` (**not** `pgrep -f`, which self-matches and
 reports a false "still running") → `qemu-img snapshot -a virgin`, disk reverted, snapshot list shows
 the single `virgin` entry.
 **Nothing was provisioned that outlives this run.**
 ## 7. The vouch, and its verification
 **Performed by the operator (Viktor)** in the hub, Configuration → Day-0 artifacts. Verified
 afterwards by reading the hub's own store rather than trusting the save:
 | field | value | recorded |
 |---|---|---|
 | `artifact_golden_version` | **0.216.0** | 2026-08-18 11:00:59 |
 | `artifact_agent_version` | **0.129.0** | 2026-08-18 11:00:59 |
 | `artifact_min_agent` | **0.129.0** | 2026-08-18 11:01:00 |
 | `artifact_golden_sha256` | `ac004dc9…c34b` | 2026-08-18 11:01:00 |
 The recorded sha256 **matches the artifact I downloaded and hashed independently** — so the hub is
 vouching the bytes that are actually published, not merely a matching version string.
 **This separate check was necessary, and the gate says so itself.** `golden_currency_gate.py`'s own
 pass line reads *"this checks the BAKE, not the vouch"*. A green gate on an unvouched bake is exactly
 the "baked-but-unvouched golden is worse than none" state R-334 warned about, so the gate alone could
 not have closed this row.
 ## 8. Gates
 ```
 golden_currency_gate.py                        rc=0
  newest released controller : 0.216.0
  newest golden baked        : 0.216.0
 repo_gates.py --fast                           rc=0
  site OK · hostinstall OK · hub-confirm OK · manifest-bearer OK · reuse-refs OK
  instructions OK · golden-currency OK · wire-contract OK · hub-copy OK
  all felhom.eu gates OK
 ```
 **This is the first fully green gate run since 2026-08-14**, and it is the point of the run: the
 CI failure mail that has been arriving since then should now stop.
 ## 9. Documentation not changed, deliberately
 **`documentation/architecture/00-capability-map.md` — no change, and the reason matters.** The run
 sheet said to update it *if the day-0 install row's evidence citation names the golden version*. It
 does not: that row cites `DRILL-day0-vm-2026-07-12` / `DRILL-day0-take2-2026-07-12`. The only golden
 version literal in the map is `tests/golden-0.205.0-2026-08-07` on the **recovery-journey** row,
 which is a **dated historical citation** of what a fresh install landed on during the 2026-08-07
 walk. Bumping it to 0.216.0 would falsify a record of what happened on a specific date — `docs.md`
 permits historical citations precisely because they cannot go stale.
 ## 10. Observations, not acted on
 - **`min_controller_version` in the hub still reads `0.214.0`** (last touched 2026-08-12). That is a
  different field from the three vouched here — it is the floor the fleet is held to, not the day-0
  golden — and it was outside this run's scope. But it is now two releases behind the golden a new
  box receives, and STATUS.md's "approved pair" line describes it. Worth a decision; **not** changed
  here, because widening scope past the three named fields is how a vouch goes wrong.
 - **`pveam available` still offers `debian-13-standard_13.6-1_amd64.tar.zst`** — the same point
  release the runbook recorded on 2026-07-31. Listed live rather than assumed, per §4.1 step 2; the
  instruction stands even when the answer happens not to have moved.
 - **The bake ran in ~5 minutes** (12:49 launch → 12:54:14 archive), well inside the drill VM's normal
  envelope; no timeout or retry was needed.
@@ -1,7 +1,7 @@
 # STATUS — what works, what's broken, what's next
-**Updated 2026-08-18 (midday — a listening socket that served nobody, then a rehearsal upgrade that changed nothing; off-site backups were down for
+**Updated 2026-08-18 (afternoon — a listening socket that served nobody, a rehearsal upgrade that changed
-9½ hours overnight and are back).**
+nothing, and a new install that is finally current).**
 > **A view, not a source.** `documentation/backlog/OPEN-ITEMS.md` is the authority; this page restates
 > part of it in plain words, and **nothing may exist only here**. **Items, not paragraphs. One screen.**
@@ -43,6 +43,13 @@ record with no machine** — created 13 August, no host, no backups, nothing to
 ## Shipped
 - **A new machine installed today finally gets today's software** (R-334, closed). The pre-built image
  had been two releases behind since the 14th — anyone installing would have received a version
  missing last week's disk-warning fix *and* the follow-up that corrected it. A fresh image was baked
  and published, and **you vouched it**, which was the half that could not be done without you. **The
  build system is green again for the first time since 14 August**, so the failure mail should stop.
  The running machines were not touched: this only ever affected *new* installs.
 - **Last night's two backup alarms were real, and are fixed** (R-336). Both machines failed their
  off-site backup at 04:30; **neither machine was at fault**. The off-site box in Germany had run out
  of one internal resource and, while looking perfectly healthy from outside, was accepting no
@@ -0,0 +1,79 @@
 # Golden bake 0.216.0 — 2026-08-18
 Run under `RUN SHEET — bake and vouch golden 0.216.0 (closes R-334)`, following
 `documentation/runbooks/RUNBOOK-manual-build.md` §4.0 + §4.1. Closes the two-release gap
 R-334 has been convicting CI on since 2026-08-14.
 ## Identity
    GOLDEN_VERSION = 0.216.0
    GOLDEN_SHA256  = ac004dc90d8cefccc5448377892f9cff3a4c3e1e27d0e11129120e38ac31c34b
    URL            = https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.216.0/golden.tar.zst
    archive        = 656,970,239 bytes (rootfs 32G + ONE data volume 24G @ /var/lib/felhom)
    controller     = gitea.dooplex.hu/admin/felhom-controller:0.216.0
    template       = debian-13-standard_13.6-1_amd64.tar.zst (listed live, not reused from the runbook)
 **The published bytes were verified, not just the script's claim:** the artifact was downloaded back
 from Gitea and hashed, and its sha256 matches the value the script printed, exactly.
 ## Baselines, re-read on the machine
 | item | value | source |
 |---|---|---|
 | newest released controller | **v0.216.0** | `felhom-controller/CHANGELOG.md` head |
 | its floor | **`MinAgent: 0.129.0`** | second line of that same header |
 | newest agent release | **v0.129.0** | `felhom-agent/CHANGELOG.md` head |
 | newest golden before this run | **0.214.0** | `documentation/tests/golden-0.214.0-2026-08-12` |
 All four match the run sheet's §1 — no disagreement to report. The agent artifact for the vouched
 `agent_version` was confirmed **published in Gitea packages** (`generic felhom-agent 0.129.0`), not
 inferred from the CHANGELOG.
 **The R-216 check passed:** `MinAgent` (0.129.0) is **equal to**, not above, the newest published
 agent (0.129.0). The coincidence the run sheet warned about was confirmed on the machine rather than
 assumed.
 ## Pass markers — the corrected list (post-2026-08-06, R-233)
    line  82 :   docker OK (overlay2; data-root /var/lib/docker)
    line 313 : INFO: including mount point rootfs ('/') in backup
    line 314 : INFO: including mount point mp0 ('/var/lib/felhom') in backup
    line 319 : [golden]   pre-delete existing: HTTP 404 (404/204 expected)
    line 320 : [golden]   upload OK (HTTP 201)
 `excluding` and `FATAL`: **absent**. There is no mp1 — R-165 collapsed the two data volumes into one,
 which is why the pre-2026-08-06 marker list could never match.
 ## Token handling
 Copied **file → file** by `scp`; never on a command line. The runner script inside the VM read it
 from `/root/.gitea-token` itself.
    systemctl show golden-bake -p Environment -p ExecStart | grep -c -F "<token>"   =  0
 Token-leak grep on the **committed** log, with its positive control run FIRST:
    seeded throwaway copy  = 1   (proves the grep can see the token)
    committed bake.log     = 0   (the real measurement, now trustworthy)
 The control is not ceremony: a `grep -c` that silently matches nothing returns `0`, which is
 indistinguishable from a clean file. Full transcript in `token-leak-grep.txt`.
 ## Teardown
 - `pct destroy 9100 --purge` — both logical volumes removed, CT purged from related configs.
 - `shred -u` on `/root/.gitea-token`, `/root/bake-run.sh`, `/root/build-golden.sh`, `/root/bake.log`
  — **after** `bake.log` was copied out to this directory (standing rule 5).
 - All four confirmed absent by `ls` afterwards.
 - `poweroff`, waited for qemu to exit (`ps -eo comm`, **not** `pgrep -f`, which self-matches).
 - `qemu-img snapshot -a virgin` — disk reverted; snapshot list shows the single `virgin` entry.
 Nothing was provisioned that outlives this run.
 ## Files
    bake.log                     the real bake log, token-grepped (0, with control)
    pass-markers.txt             the markers quoted from that log with line numbers
    token-leak-grep.txt          control=1 / real=0 transcript
    package-url-verification.txt URL resolution + downloaded-sha256 match
    teardown.txt                 destroy, shred, qemu exit, revert
@@ -0,0 +1,324 @@
 [golden] build-golden.sh v3.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.216.0
 [golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) …
  Logical volume "vm-9100-disk-0" created.
  Logical volume pve/vm-9100-disk-0 changed.
 Creating filesystem with 8388608 4k blocks and 2097152 inodes
 Filesystem UUID: 04b6626c-cbce-49f2-b370-642a325029df
 Superblock backups stored on blocks: 
 	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
 	4096000, 7962624
  Logical volume "vm-9100-disk-1" created.
  Logical volume pve/vm-9100-disk-1 changed.
 Creating filesystem with 6291456 4k blocks and 1572864 inodes
 Filesystem UUID: 77e27bed-bdca-4436-aca7-42fd4f4db9e0
 Superblock backups stored on blocks: 
 	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
 extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst'
 Total bytes read: 553512960 (528MiB, 173MiB/s)
 Detected container architecture: amd64
 Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
 done: SHA256:0U6qROh3MR0dZ936k1KUiGYhd4BEcaQlMKGzTYgi7kY root@felhom-golden
 Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
 done: SHA256:tYitIywg5r71mqw1sMLS1UxQyf00iRDV/xRhGhCj6BM root@felhom-golden
 Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
 done: SHA256:ZdV26t00BfeuIwbEi52UJH40Lps/128W1nuDl5HJ1Wg root@felhom-golden
 [golden] starting + installing Docker (official repo, trixie channel) …
 apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
 perl: warning: Setting locale failed.
 perl: warning: Please check that your locale settings:
 	LANGUAGE = (unset),
 	LC_ALL = (unset),
 	LC_CTYPE = (unset),
 	LC_NUMERIC = (unset),
 	LC_COLLATE = (unset),
 	LC_TIME = (unset),
 	LC_MESSAGES = (unset),
 	LC_MONETARY = (unset),
 	LC_ADDRESS = (unset),
 	LC_IDENTIFICATION = (unset),
 	LC_MEASUREMENT = (unset),
 	LC_PAPER = (unset),
 	LC_TELEPHONE = (unset),
 	LC_NAME = (unset),
 	LANG = "en_US.UTF-8"
    are supported and installed on your system.
 perl: warning: Falling back to the standard locale ("C").
 locale: Cannot set LC_CTYPE to default locale: No such file or directory
 locale: Cannot set LC_MESSAGES to default locale: No such file or directory
 locale: Cannot set LC_ALL to default locale: No such file or directory
 apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
 perl: warning: Setting locale failed.
 perl: warning: Please check that your locale settings:
 	LANGUAGE = (unset),
 	LC_ALL = (unset),
 	LC_CTYPE = (unset),
 	LC_NUMERIC = (unset),
 	LC_COLLATE = (unset),
 	LC_TIME = (unset),
 	LC_MESSAGES = (unset),
 	LC_MONETARY = (unset),
 	LC_ADDRESS = (unset),
 	LC_IDENTIFICATION = (unset),
 	LC_MEASUREMENT = (unset),
 	LC_PAPER = (unset),
 	LC_TELEPHONE = (unset),
 	LC_NAME = (unset),
 	LANG = "en_US.UTF-8"
    are supported and installed on your system.
 perl: warning: Falling back to the standard locale ("C").
 locale: Cannot set LC_CTYPE to default locale: No such file or directory
 locale: Cannot set LC_MESSAGES to default locale: No such file or directory
 locale: Cannot set LC_ALL to default locale: No such file or directory
 [golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …
 [golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds …
 [golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …
 Unable to find image 'hello-world:latest' locally
 latest: Pulling from library/hello-world
 4f55086f7dd0: Pulling fs layer
 4f55086f7dd0: Download complete
 4f55086f7dd0: Pull complete
 Digest: sha256:5dd0d3e6e255913fc30f90b9f2b1d359cc2cbdb48090cc4b65f1676e203243cc
 Status: Downloaded newer image for hello-world:latest
  docker OK (overlay2; data-root /var/lib/docker)
  /var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4
  /mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4
  both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576
 [golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.216.0 (no registry cred at deploy) …
 WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
 Configure a credential helper to remove this warning. See
 https://docs.docker.com/go/credential-store/
 0.216.0: Pulling from admin/felhom-controller
 039e6f9f9752: Pulling fs layer
 0094c3ac0914: Pulling fs layer
 deca1dac7403: Pulling fs layer
 11c19a33d1b8: Pulling fs layer
 44e9e14e4e05: Pulling fs layer
 7d8b3a282cc4: Pulling fs layer
 11c19a33d1b8: Waiting
 44e9e14e4e05: Waiting
 7d8b3a282cc4: Waiting
 deca1dac7403: Verifying Checksum
 deca1dac7403: Download complete
 11c19a33d1b8: Verifying Checksum
 11c19a33d1b8: Download complete
 44e9e14e4e05: Verifying Checksum
 44e9e14e4e05: Download complete
 7d8b3a282cc4: Verifying Checksum
 7d8b3a282cc4: Download complete
 039e6f9f9752: Verifying Checksum
 039e6f9f9752: Download complete
 0094c3ac0914: Verifying Checksum
 0094c3ac0914: Download complete
 039e6f9f9752: Pull complete
 0094c3ac0914: Pull complete
 deca1dac7403: Pull complete
 11c19a33d1b8: Pull complete
 44e9e14e4e05: Pull complete
 7d8b3a282cc4: Pull complete
 Digest: sha256:2d9551b8a67112a19738a3943a346737d323cf772d1e50acbc799c0a5cc64673
 Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.216.0
 gitea.dooplex.hu/admin/felhom-controller:0.216.0
 [golden] asking the controller which infra images it manages …
 [golden] baking infra images (4): traefik:v3.6.7 cloudflare/cloudflared:2026.6.0 gtstef/filebrowser:1.3.3-stable gitea.dooplex.hu/admin/felhom-samba:1.1.0 …
 v3.6.7: Pulling from library/traefik
 589002ba0eae: Pulling fs layer
 ef63511ea6cc: Pulling fs layer
 0738e5cb835e: Pulling fs layer
 3e6813f70c64: Pulling fs layer
 3e6813f70c64: Waiting
 ef63511ea6cc: Verifying Checksum
 ef63511ea6cc: Download complete
 589002ba0eae: Download complete
 3e6813f70c64: Verifying Checksum
 3e6813f70c64: Download complete
 0738e5cb835e: Verifying Checksum
 0738e5cb835e: Download complete
 589002ba0eae: Pull complete
 ef63511ea6cc: Pull complete
 0738e5cb835e: Pull complete
 3e6813f70c64: Pull complete
 Digest: sha256:a9890c898f379c1905ee5b28342f6b408dc863f08db2dab20e46c267d1ff463a
 Status: Downloaded newer image for traefik:v3.6.7
 docker.io/library/traefik:v3.6.7
 2026.6.0: Pulling from cloudflare/cloudflared
 47de5dd0b812: Pulling fs layer
 c172f21841df: Pulling fs layer
 99515e7b4d35: Pulling fs layer
 99ba982a9142: Pulling fs layer
 d6b1b89eccac: Pulling fs layer
 2780920e5dbf: Pulling fs layer
 7c12895b777b: Pulling fs layer
 3214acf345c0: Pulling fs layer
 52630fc75a18: Pulling fs layer
 dd64bf2dd177: Pulling fs layer
 b839dfae01f6: Pulling fs layer
 ebddc55facdc: Pulling fs layer
 bdfd7f7e5bf6: Pulling fs layer
 2d4d7adf6272: Pulling fs layer
 40008157d8d2: Pulling fs layer
 bd8962e29291: Pulling fs layer
 cac2ae0193cb: Pulling fs layer
 74d1dac84ecc: Pulling fs layer
 99ba982a9142: Waiting
 d6b1b89eccac: Waiting
 2780920e5dbf: Waiting
 7c12895b777b: Waiting
 3214acf345c0: Waiting
 52630fc75a18: Waiting
 dd64bf2dd177: Waiting
 b839dfae01f6: Waiting
 ebddc55facdc: Waiting
 bdfd7f7e5bf6: Waiting
 2d4d7adf6272: Waiting
 40008157d8d2: Waiting
 bd8962e29291: Waiting
 cac2ae0193cb: Waiting
 74d1dac84ecc: Waiting
 47de5dd0b812: Verifying Checksum
 47de5dd0b812: Download complete
 99515e7b4d35: Verifying Checksum
 99515e7b4d35: Download complete
 c172f21841df: Verifying Checksum
 c172f21841df: Download complete
 99ba982a9142: Download complete
 47de5dd0b812: Pull complete
 d6b1b89eccac: Verifying Checksum
 d6b1b89eccac: Download complete
 2780920e5dbf: Verifying Checksum
 2780920e5dbf: Download complete
 7c12895b777b: Download complete
 3214acf345c0: Download complete
 52630fc75a18: Verifying Checksum
 52630fc75a18: Download complete
 dd64bf2dd177: Verifying Checksum
 dd64bf2dd177: Download complete
 b839dfae01f6: Verifying Checksum
 b839dfae01f6: Download complete
 ebddc55facdc: Verifying Checksum
 ebddc55facdc: Download complete
 bdfd7f7e5bf6: Verifying Checksum
 bdfd7f7e5bf6: Download complete
 c172f21841df: Pull complete
 2d4d7adf6272: Verifying Checksum
 2d4d7adf6272: Download complete
 40008157d8d2: Verifying Checksum
 40008157d8d2: Download complete
 bd8962e29291: Verifying Checksum
 bd8962e29291: Download complete
 cac2ae0193cb: Verifying Checksum
 cac2ae0193cb: Download complete
 74d1dac84ecc: Verifying Checksum
 74d1dac84ecc: Download complete
 99515e7b4d35: Pull complete
 99ba982a9142: Pull complete
 d6b1b89eccac: Pull complete
 2780920e5dbf: Pull complete
 7c12895b777b: Pull complete
 3214acf345c0: Pull complete
 52630fc75a18: Pull complete
 dd64bf2dd177: Pull complete
 b839dfae01f6: Pull complete
 ebddc55facdc: Pull complete
 bdfd7f7e5bf6: Pull complete
 2d4d7adf6272: Pull complete
 40008157d8d2: Pull complete
 bd8962e29291: Pull complete
 cac2ae0193cb: Pull complete
 74d1dac84ecc: Pull complete
 Digest: sha256:ba461b8aa9c042156dbd39c38657fe7431bafa063220eab8d5330a523863da9f
 Status: Downloaded newer image for cloudflare/cloudflared:2026.6.0
 docker.io/cloudflare/cloudflared:2026.6.0
 1.3.3-stable: Pulling from gtstef/filebrowser
 6a0ac1617861: Pulling fs layer
 ef8806083e82: Pulling fs layer
 b74107c861c7: Pulling fs layer
 adc935def003: Pulling fs layer
 4f4fb700ef54: Pulling fs layer
 18695ccc900a: Pulling fs layer
 45d119d5c397: Pulling fs layer
 dac52db4fc51: Pulling fs layer
 6d598f86b2f2: Pulling fs layer
 8aa349c8396c: Pulling fs layer
 45d119d5c397: Waiting
 dac52db4fc51: Waiting
 6d598f86b2f2: Waiting
 8aa349c8396c: Waiting
 adc935def003: Waiting
 4f4fb700ef54: Waiting
 18695ccc900a: Waiting
 6a0ac1617861: Verifying Checksum
 6a0ac1617861: Download complete
 b74107c861c7: Verifying Checksum
 b74107c861c7: Download complete
 4f4fb700ef54: Verifying Checksum
 4f4fb700ef54: Download complete
 6a0ac1617861: Pull complete
 ef8806083e82: Verifying Checksum
 ef8806083e82: Download complete
 adc935def003: Verifying Checksum
 adc935def003: Download complete
 18695ccc900a: Verifying Checksum
 18695ccc900a: Download complete
 45d119d5c397: Verifying Checksum
 45d119d5c397: Download complete
 dac52db4fc51: Verifying Checksum
 dac52db4fc51: Download complete
 6d598f86b2f2: Verifying Checksum
 6d598f86b2f2: Download complete
 8aa349c8396c: Verifying Checksum
 8aa349c8396c: Download complete
 ef8806083e82: Pull complete
 b74107c861c7: Pull complete
 adc935def003: Pull complete
 4f4fb700ef54: Pull complete
 18695ccc900a: Pull complete
 45d119d5c397: Pull complete
 dac52db4fc51: Pull complete
 6d598f86b2f2: Pull complete
 8aa349c8396c: Pull complete
 Digest: sha256:eb3733681db8757412632c61a99ad656f0d94ed6781bb2ea114b4d70babab78c
 Status: Downloaded newer image for gtstef/filebrowser:1.3.3-stable
 docker.io/gtstef/filebrowser:1.3.3-stable
 1.1.0: Pulling from admin/felhom-samba
 897d797d2723: Pulling fs layer
 3051591aa250: Pulling fs layer
 ce57a3f93416: Pulling fs layer
 fb94eeec2fe1: Pulling fs layer
 fb94eeec2fe1: Waiting
 ce57a3f93416: Verifying Checksum
 ce57a3f93416: Download complete
 fb94eeec2fe1: Verifying Checksum
 fb94eeec2fe1: Download complete
 897d797d2723: Verifying Checksum
 897d797d2723: Download complete
 3051591aa250: Verifying Checksum
 3051591aa250: Download complete
 897d797d2723: Pull complete
 3051591aa250: Pull complete
 ce57a3f93416: Pull complete
 fb94eeec2fe1: Pull complete
 Digest: sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10
 Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.1.0
 gitea.dooplex.hu/admin/felhom-samba:1.1.0
 [golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) …
 Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'.
 [golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …
 Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'.
 [golden] baking the first-boot SSH host-key regeneration unit (F3) …
 Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'.
 [golden] identity-clean + minimize …
 [golden] stop + archive …
 INFO: including mount point rootfs ('/') in backup
 INFO: including mount point mp0 ('/var/lib/felhom') in backup
 INFO: archive file size: 626MB
 INFO: Finished Backup of VM 9100 (00:00:43)
 [golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_08_18-12_54_14.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive)
 [golden] publishing golden (656970239 bytes, sha256 ac004dc90d8cefcc…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.216.0/golden.tar.zst
 [golden]   pre-delete existing: HTTP 404 (404/204 expected)
 [golden]   upload OK (HTTP 201)
 GOLDEN_VERSION=0.216.0
 GOLDEN_SHA256=ac004dc90d8cefccc5448377892f9cff3a4c3e1e27d0e11129120e38ac31c34b
 [golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.216.0 / ac004dc90d8cefccc5448377892f9cff3a4c3e1e27d0e11129120e38ac31c34b
 [golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge)
@@ -0,0 +1,8 @@
 PACKAGE URL RESOLUTION
 URL: https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.216.0/golden.tar.zst
  HTTP=206  size=1 bytes
  (expected: 200/206 and NOT 404; log reported 656970239 bytes)
 SHA256 verification — download and hash, rather than trusting the script's own print:
  downloaded sha256 = ac004dc90d8cefccc5448377892f9cff3a4c3e1e27d0e11129120e38ac31c34b
  script reported   = ac004dc90d8cefccc5448377892f9cff3a4c3e1e27d0e11129120e38ac31c34b
@@ -0,0 +1,20 @@
 PASS MARKERS — the CORRECTED list (RUNBOOK-manual-build.md §4.1, post-2026-08-06 R-233)
 quoted from the real log: /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/tests/golden-0.216.0-2026-08-18/bake.log
 1) storage driver — literal 'docker OK (overlay2'
 82:  docker OK (overlay2; data-root /var/lib/docker)
 2) 'including mount point' for rootfs AND mp0 (there is NO mp1)
 313:INFO: including mount point rootfs ('/') in backup
 314:INFO: including mount point mp0 ('/var/lib/felhom') in backup
 3) negative markers — 'excluding' and 'FATAL' must NOT appear
   none present  OK
 4) 'upload OK (HTTP 201)'
 320:[golden]   upload OK (HTTP 201)
 5) published identity
 319:[golden]   pre-delete existing: HTTP 404 (404/204 expected)
 321:GOLDEN_VERSION=0.216.0
 322:GOLDEN_SHA256=ac004dc90d8cefccc5448377892f9cff3a4c3e1e27d0e11129120e38ac31c34b
@@ -0,0 +1,38 @@
 ### destroy build guest 9100
  Logical volume "vm-9100-disk-0" successfully removed.
  Logical volume "vm-9100-disk-1" successfully removed.
 purging CT 9100 from related configurations..
 ### guest list after destroy (9100 must be gone)
 perl: warning: Setting locale failed.
 perl: warning: Please check that your locale settings:
 	LANGUAGE = (unset),
 	LC_ALL = (unset),
 	LC_CTYPE = "UTF-8",
 	LC_NUMERIC = (unset),
 	LC_COLLATE = (unset),
 	LC_TIME = (unset),
 	LC_MESSAGES = (unset),
 	LC_MONETARY = (unset),
 	LC_ADDRESS = (unset),
 	LC_IDENTIFICATION = (unset),
 	LC_MEASUREMENT = (unset),
 	LC_PAPER = (unset),
 	LC_TELEPHONE = (unset),
 	LC_NAME = (unset),
 	LANG = "en_US.UTF-8"
    are supported and installed on your system.
 perl: warning: Falling back to a fallback locale ("en_US.UTF-8").
 ### shred token, runner, script, log (log already copied out)
 ### confirm gone
 ls: cannot access '/root/.gitea-token': No such file or directory
 ls: cannot access '/root/bake-run.sh': No such file or directory
 ls: cannot access '/root/build-golden.sh': No such file or directory
 ls: cannot access '/root/bake.log': No such file or directory
 ### qemu liveness after poweroff (ps -eo comm, NOT pgrep -f)
  none — qemu exited
 ### revert disk to virgin
  revert OK
 ### snapshot list
 Snapshot list:
 ID      TAG               VM_SIZE                DATE        VM_CLOCK     ICOUNT
 1       virgin                0 B 2026-07-03 14:12:15  0000:00:00.000          0
@@ -0,0 +1,12 @@
 TOKEN-LEAK GREP — on the COMMITTED log, with a positive control first.
 Target: /mnt/5_hdd/felhom.eu/git/felhom.eu/documentation/tests/golden-0.216.0-2026-08-18/bake.log
 Method: grep -c -F against the LITERAL token value (a broad [a-f0-9]{40} pattern false-hits image shas).
 -- POSITIVE CONTROL: seed a throwaway copy with the token, prove the grep can find it --
   seeded copy match count = 1   (MUST be 1 — otherwise the instrument is blind)
   seeded copy shredded
 -- THE REAL MEASUREMENT, now that the grep is shown to work --
   committed log match count = 0   (MUST be 0)
 VERDICT: PASS — the grep works AND the committed log is clean.