golden 0.244.0 baked and PUBLISHED — after three attempts, all three failures mine
gates / gates (push) Successful in 23s

sha256 18328a3c7579628b8a7e9639777043db48063c86d37a2e0c221a6ccba6d755a0, 653 609 190
bytes, registry serves it. Markers: overlay2, both mount points, upload OK, no FATAL,
no publish-SKIPPED. Token-leak control passed with a planted positive control.

The three failures are written up because each is a rule this project already has:
scp -p instead of -P (nothing copied); the publisher run without the GITEA_USER it
requires, then the archive destroyed BEFORE checking the outcome; and a rewrite that
dropped the chmod, where the unit reported Result=success while the script inside it
had died on Permission denied.

The fix that matters is the gate: teardown now happens only when the REGISTRY serves
the package — not on an exit code, not on a log sentence. It held: on the failed
attempts the VM and its archive were left in place.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
2026-09-16 17:32:18 +02:00
parent 87cb923390
commit f57aed9ab9
3 changed files with 396 additions and 2 deletions
@@ -1,2 +1,321 @@
cat: /root/.gitea-token: No such file or directory
/root/bake-run.sh: line 4: /root/build-golden.sh: No such file or directory
[golden] build-golden.sh v3.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.244.0
[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) …
Logical volume "vm-9100-disk-0" created.
Logical volume pve/vm-9100-disk-0 changed.
Creating filesystem with 8388608 4k blocks and 2097152 inodes
Filesystem UUID: eca15c16-96ef-4653-9843-dbf37cc255ce
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624
Logical volume "vm-9100-disk-1" created.
Logical volume pve/vm-9100-disk-1 changed.
Creating filesystem with 6291456 4k blocks and 1572864 inodes
Filesystem UUID: d3c47b86-d18c-4e94-b1b5-ba2e334e4eb0
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst'
Total bytes read: 553512960 (528MiB, 109MiB/s)
Detected container architecture: amd64
Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
done: SHA256:EOX2WLMMVRDCNoNwCZhDSPncStDmfAnMNLpTwBT13tM root@felhom-golden
Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
done: SHA256:OLKxkKrKoHe6UiQsoiraX7vYHxRkq7duh9G1ggdfApA root@felhom-golden
Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
done: SHA256:VIZqqTtAxssw3+dm5HpszeLNwkich3niJj52gbO8BnI root@felhom-golden
[golden] starting + installing Docker (official repo, trixie channel) …
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = (unset),
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = (unset),
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …
[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds …
[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
4f55086f7dd0: Pulling fs layer
4f55086f7dd0: Verifying Checksum
4f55086f7dd0: Download complete
4f55086f7dd0: Pull complete
Digest: sha256:5e23090353324d887c48ad5e5c56d294eab81588df9605b07d1afe895f9cc8f8
Status: Downloaded newer image for hello-world:latest
docker OK (overlay2; data-root /var/lib/docker)
/var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4
/mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4
both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576
[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.244.0 (no registry cred at deploy) …
0.244.0: Pulling from admin/felhom-controller
a8ac7f6c67ab: Pulling fs layer
bf30769d36e7: Pulling fs layer
044b66fbe46c: Pulling fs layer
b5c41a28e83f: Pulling fs layer
81b60ff19091: Pulling fs layer
428d65c7a3d1: Pulling fs layer
b5c41a28e83f: Waiting
81b60ff19091: Waiting
428d65c7a3d1: Waiting
a8ac7f6c67ab: Verifying Checksum
a8ac7f6c67ab: Download complete
b5c41a28e83f: Verifying Checksum
b5c41a28e83f: Download complete
044b66fbe46c: Verifying Checksum
044b66fbe46c: Download complete
428d65c7a3d1: Verifying Checksum
428d65c7a3d1: Download complete
81b60ff19091: Verifying Checksum
81b60ff19091: Download complete
bf30769d36e7: Verifying Checksum
bf30769d36e7: Download complete
a8ac7f6c67ab: Pull complete
bf30769d36e7: Pull complete
044b66fbe46c: Pull complete
b5c41a28e83f: Pull complete
81b60ff19091: Pull complete
428d65c7a3d1: Pull complete
Digest: sha256:aaa9150aefcfa1a4909c34524844fdef342fa472d7a518bb6f7890d56ad795e1
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.244.0
gitea.dooplex.hu/admin/felhom-controller:0.244.0
[golden] asking the controller which infra images it manages …
[golden] baking infra images (4): traefik:v3.6.7 cloudflare/cloudflared:2026.6.0 gtstef/filebrowser:1.3.3-stable gitea.dooplex.hu/admin/felhom-samba:1.1.0 …
v3.6.7: Pulling from library/traefik
589002ba0eae: Pulling fs layer
ef63511ea6cc: Pulling fs layer
0738e5cb835e: Pulling fs layer
3e6813f70c64: Pulling fs layer
3e6813f70c64: Waiting
589002ba0eae: Verifying Checksum
589002ba0eae: Download complete
ef63511ea6cc: Download complete
3e6813f70c64: Verifying Checksum
3e6813f70c64: Download complete
589002ba0eae: Pull complete
0738e5cb835e: Verifying Checksum
0738e5cb835e: Download complete
ef63511ea6cc: Pull complete
0738e5cb835e: Pull complete
3e6813f70c64: Pull complete
Digest: sha256:a9890c898f379c1905ee5b28342f6b408dc863f08db2dab20e46c267d1ff463a
Status: Downloaded newer image for traefik:v3.6.7
docker.io/library/traefik:v3.6.7
2026.6.0: Pulling from cloudflare/cloudflared
47de5dd0b812: Pulling fs layer
c172f21841df: Pulling fs layer
99515e7b4d35: Pulling fs layer
99ba982a9142: Pulling fs layer
d6b1b89eccac: Pulling fs layer
2780920e5dbf: Pulling fs layer
7c12895b777b: Pulling fs layer
3214acf345c0: Pulling fs layer
52630fc75a18: Pulling fs layer
dd64bf2dd177: Pulling fs layer
b839dfae01f6: Pulling fs layer
ebddc55facdc: Pulling fs layer
bdfd7f7e5bf6: Pulling fs layer
2d4d7adf6272: Pulling fs layer
40008157d8d2: Pulling fs layer
bd8962e29291: Pulling fs layer
cac2ae0193cb: Pulling fs layer
74d1dac84ecc: Pulling fs layer
99ba982a9142: Waiting
d6b1b89eccac: Waiting
2780920e5dbf: Waiting
7c12895b777b: Waiting
3214acf345c0: Waiting
52630fc75a18: Waiting
dd64bf2dd177: Waiting
b839dfae01f6: Waiting
ebddc55facdc: Waiting
bdfd7f7e5bf6: Waiting
2d4d7adf6272: Waiting
40008157d8d2: Waiting
bd8962e29291: Waiting
cac2ae0193cb: Waiting
74d1dac84ecc: Waiting
47de5dd0b812: Download complete
c172f21841df: Verifying Checksum
c172f21841df: Download complete
99515e7b4d35: Verifying Checksum
99515e7b4d35: Download complete
99ba982a9142: Verifying Checksum
99ba982a9142: Download complete
d6b1b89eccac: Verifying Checksum
d6b1b89eccac: Download complete
2780920e5dbf: Verifying Checksum
2780920e5dbf: Download complete
47de5dd0b812: Pull complete
7c12895b777b: Verifying Checksum
7c12895b777b: Download complete
3214acf345c0: Verifying Checksum
3214acf345c0: Download complete
52630fc75a18: Verifying Checksum
52630fc75a18: Download complete
dd64bf2dd177: Verifying Checksum
dd64bf2dd177: Download complete
c172f21841df: Pull complete
b839dfae01f6: Verifying Checksum
b839dfae01f6: Download complete
ebddc55facdc: Verifying Checksum
ebddc55facdc: Download complete
bdfd7f7e5bf6: Verifying Checksum
bdfd7f7e5bf6: Download complete
40008157d8d2: Verifying Checksum
40008157d8d2: Download complete
2d4d7adf6272: Verifying Checksum
2d4d7adf6272: Download complete
bd8962e29291: Verifying Checksum
bd8962e29291: Download complete
cac2ae0193cb: Verifying Checksum
cac2ae0193cb: Download complete
99515e7b4d35: Pull complete
74d1dac84ecc: Verifying Checksum
74d1dac84ecc: Download complete
99ba982a9142: Pull complete
d6b1b89eccac: Pull complete
2780920e5dbf: Pull complete
7c12895b777b: Pull complete
3214acf345c0: Pull complete
52630fc75a18: Pull complete
dd64bf2dd177: Pull complete
b839dfae01f6: Pull complete
ebddc55facdc: Pull complete
bdfd7f7e5bf6: Pull complete
2d4d7adf6272: Pull complete
40008157d8d2: Pull complete
bd8962e29291: Pull complete
cac2ae0193cb: Pull complete
74d1dac84ecc: Pull complete
Digest: sha256:ba461b8aa9c042156dbd39c38657fe7431bafa063220eab8d5330a523863da9f
Status: Downloaded newer image for cloudflare/cloudflared:2026.6.0
docker.io/cloudflare/cloudflared:2026.6.0
1.3.3-stable: Pulling from gtstef/filebrowser
6a0ac1617861: Pulling fs layer
ef8806083e82: Pulling fs layer
b74107c861c7: Pulling fs layer
adc935def003: Pulling fs layer
4f4fb700ef54: Pulling fs layer
18695ccc900a: Pulling fs layer
45d119d5c397: Pulling fs layer
dac52db4fc51: Pulling fs layer
6d598f86b2f2: Pulling fs layer
8aa349c8396c: Pulling fs layer
dac52db4fc51: Waiting
6d598f86b2f2: Waiting
8aa349c8396c: Waiting
4f4fb700ef54: Waiting
18695ccc900a: Waiting
45d119d5c397: Waiting
adc935def003: Waiting
6a0ac1617861: Verifying Checksum
6a0ac1617861: Download complete
adc935def003: Verifying Checksum
adc935def003: Download complete
4f4fb700ef54: Verifying Checksum
4f4fb700ef54: Download complete
b74107c861c7: Verifying Checksum
b74107c861c7: Download complete
45d119d5c397: Verifying Checksum
45d119d5c397: Download complete
18695ccc900a: Verifying Checksum
18695ccc900a: Download complete
dac52db4fc51: Verifying Checksum
dac52db4fc51: Download complete
6a0ac1617861: Pull complete
ef8806083e82: Verifying Checksum
ef8806083e82: Download complete
6d598f86b2f2: Verifying Checksum
6d598f86b2f2: Download complete
8aa349c8396c: Download complete
ef8806083e82: Pull complete
b74107c861c7: Pull complete
adc935def003: Pull complete
4f4fb700ef54: Pull complete
18695ccc900a: Pull complete
45d119d5c397: Pull complete
dac52db4fc51: Pull complete
6d598f86b2f2: Pull complete
8aa349c8396c: Pull complete
Digest: sha256:eb3733681db8757412632c61a99ad656f0d94ed6781bb2ea114b4d70babab78c
Status: Downloaded newer image for gtstef/filebrowser:1.3.3-stable
docker.io/gtstef/filebrowser:1.3.3-stable
1.1.0: Pulling from admin/felhom-samba
897d797d2723: Pulling fs layer
3051591aa250: Pulling fs layer
ce57a3f93416: Pulling fs layer
fb94eeec2fe1: Pulling fs layer
fb94eeec2fe1: Waiting
ce57a3f93416: Verifying Checksum
ce57a3f93416: Download complete
fb94eeec2fe1: Verifying Checksum
fb94eeec2fe1: Download complete
897d797d2723: Verifying Checksum
897d797d2723: Download complete
3051591aa250: Verifying Checksum
3051591aa250: Download complete
897d797d2723: Pull complete
3051591aa250: Pull complete
ce57a3f93416: Pull complete
fb94eeec2fe1: Pull complete
Digest: sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.1.0
gitea.dooplex.hu/admin/felhom-samba:1.1.0
[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) …
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'.
[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'.
[golden] baking the first-boot SSH host-key regeneration unit (F3) …
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'.
[golden] identity-clean + minimize …
[golden] stop + archive …
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/var/lib/felhom') in backup
INFO: archive file size: 623MB
INFO: Finished Backup of VM 9100 (00:00:31)
[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_09_16-17_29_41.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive)
[golden] publishing golden (653609190 bytes, sha256 18328a3c7579628b…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.244.0/golden.tar.zst
[golden] pre-delete existing: HTTP 404 (404/204 expected)
[golden] upload OK (HTTP 201)
GOLDEN_VERSION=0.244.0
GOLDEN_SHA256=18328a3c7579628b8a7e9639777043db48063c86d37a2e0c221a6ccba6d755a0
[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.244.0 / 18328a3c7579628b8a7e9639777043db48063c86d37a2e0c221a6ccba6d755a0
[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge)
@@ -29,3 +29,71 @@
script + token landed in the VM (both non-empty)
bake launched as a transient unit at 2026-09-16T15:15:37Z
token leak check on the unit (must be 0, and the token is non-empty so the grep cannot match everything): 0
## 2026-09-16T15:19Z — MY WATCHER WAS KILLED, THE BAKE WAS NOT. Recorded so the two are not confused.
## The system stopped my background watcher shell for low memory (DooPlex: 62 GB total, ~11 GB
## available, the largest consumer a 10.6 GB java service; the bake VM itself was 4.1 GB RSS).
## The VM is daemonized and the bake runs INSIDE it as a transient systemd unit, so it kept going:
## `systemctl is-active golden-bake` = active, 120 log lines, at „baking infra images (4)".
## Production was unaffected — the k3s pods stayed Running and the hub kept serving.
## CONSEQUENCE: the watcher's post-bake steps (copy the log out, markers, token-leak control,
## destroy 9100, poweroff, revert to virgin) did NOT run and are done by hand below.
unit finished: inactive at 2026-09-16T15:21:10Z
log copied off the machine FIRST (R-320): 316 lines
markers: overlay2=1 mountpoints=2 upload=0 FATAL=0 excluding=0
token-leak control — planted copy must be 1: 1; the COMMITTED log must be 0: 0
teardown: shredded
qemu processes now: 0
disk reverted to virgin
## bake finished 2026-09-16T15:21:37Z
## MY SECOND MISTAKE ON THIS BAKE, and it is the more serious one (15:21Z):
## The bake BUILT the golden and then printed „Gitea publish SKIPPED (set GITEA_USER+GITEA_TOKEN or
## REGISTRY_USER+REGISTRY_TOKEN to enable)". My runner exported GITEA_TOKEN only. The script needs
## BOTH (build-golden.sh:417-418 — PUB_USER="${GITEA_USER:-${REGISTRY_USER:-}}").
## Then I tore the guest down and reverted the disk BEFORE checking the outcome, so the archive
## (local:backup/vzdump-lxc-9100-2026_09_16-17_19_53.tar.zst, 623 MB) went with it.
## Registry check confirms the miss: golden 0.244.0 -> HTTP 404; newest published is still 0.243.0.
## TWO rules of this project were broken by one habit — I read the message the script prints
## ("DONE. golden archive volid: …") as the outcome, when the outcome is the UPLOAD:
## * "an absent log line is not evidence" — I checked markers AFTER destroying, not before;
## * "presence is not success" — a built archive is an attempt, the published package is the result.
## The re-run below sets both variables and REFUSES to tear anything down until the upload marker
## and the registry both say the golden exists.
## 2026-09-16T15:23:07Z RE-BAKE golden 0.244.0 — with GITEA_USER this time, and teardown gated on the OUTCOME
reverted to virgin
ssh up: pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
template: debian-13-standard_13.6-1_amd64.tar.zst
bake launched 2026-09-16T15:23:54Z; token leak check on the unit (must be 0): 0
unit state: inactive at 2026-09-16T15:24:10Z
log: 1 lines
markers: overlay2=0 mountpoints=0 upload=0 FATAL=0 skipped=0
token-leak control — planted must be 1: 1; committed log must be 0: 0
REGISTRY CHECK (the outcome, not the attempt): golden 0.244.0 -> http=404
NOT PUBLISHED — the VM and the archive are LEFT IN PLACE on purpose so the artifact is not lost again.
## re-bake done 2026-09-16T15:24:10Z
## 2026-09-16T15:25:31Z THIRD attempt — same running VM (template + files already there), fixed:
(a) build-golden.sh is made EXECUTABLE (it landed 0644 and died 'Permission denied');
(b) GITEA_USER is the ADMIN account, not the first credential line (the package namespace is admin).
VM still up: pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
script executable: yes; token non-empty: yes
template already on the box: debian-13-standard_13.6-1_amd64.tar.zst
launched 2026-09-16T15:25:35Z; leak check on the unit (must be 0): 0
## MY THIRD MISTAKE ON THIS BAKE (15:24Z) — two of them in one run, both mine, both recorded:
## (a) When I rewrote the bake script I DROPPED the `chmod 0700 /root/build-golden.sh` line the
## first version had. The file landed 0644 and the run died instantly:
## „/root/bake-run.sh: line 5: /root/build-golden.sh: Permission denied"
## (b) `GITEA_USER` was taken with `grep -m1` from the credentials file and resolved to `kisfenyo`,
## but the package namespace is `admin` (`/api/packages/admin/generic/…`). Even a runnable
## script would have authenticated as the wrong account.
## AND THE UNIT REPORTED SUCCESS: `systemctl show golden-bake -p Result` = `Result=success`,
## `ExecMainStatus=0` — because the UNIT ran fine; the script inside it failed. That is exactly the
## "exit codes that lie" class this project keeps re-learning, and it is why the teardown here is
## gated on the REGISTRY answering 200 for the package, not on any exit code or log sentence.
## Nothing was lost this time: the gate held the VM and the archive in place.
unit state: inactive at 2026-09-16T15:30:47Z
log: 321 lines; markers: overlay2=1 mountpoints=2 upload=1 FATAL=0 skipped=0
GOLDEN_VERSION=0.244.0
GOLDEN_SHA256=18328a3c7579628b8a7e9639777043db48063c86d37a2e0c221a6ccba6d755a0
token-leak control — planted must be 1: 1; committed log must be 0: 0
REGISTRY CHECK (the outcome): golden 0.244.0 -> http=200
PUBLISHED; guest destroyed, qemu exited, disk reverted to virgin
## attempt 3 done 2026-09-16T15:31:23Z
@@ -0,0 +1,7 @@
## 2026-09-16T15:31:52Z vouch + floor for golden 0.244.0
BEFORE: golden=0.243.0 agent=0.131.0 min_agent=0.131.0
vouch POST -> http=200
AFTER: golden=0.244.0 agent=0.131.0 min_agent=0.131.0
floor BEFORE: controller=0.242.0 agent=0.129.0
floor POST -> http=200
floor AFTER: controller=0.244.0 agent=0.131.0