R-379/R-380 docs: the failure ladder, the drill record, register housekeeping
gates / gates (push) Successful in 17s

07-backup-architecture.md 6.3 gains a dated [DESIGN] paragraph on replay ->
rollback -> hold, including why no engine flag closes it: --single-transaction
makes Postgres atomic, MariaDB DDL is not transactional, so the rollback is the
fix and the flag is a belt.

Drill record for the live walk, including the TWO defects the walk found in the
fix itself (a rollback into a re-created container; an operator route that
cleared the file while the running controller kept refusing) and the ONE
red-proof that PASSED, which is reported rather than omitted.

R-379..R-382 compressed into CLOSED-ITEMS.md. OPEN-ITEMS 330683 -> 325236 bytes.

STATUS.md restates the outcome and names the next operator step.
This commit is contained in:
2026-08-22 18:31:23 +02:00
parent 4e488321bf
commit a8caa0fdde
23 changed files with 744 additions and 75 deletions
@@ -0,0 +1,65 @@
# Golden bake — 0.220.2 (2026-08-22)
Baked in the drill VM on DooPlex per `documentation/runbooks/RUNBOOK-manual-build.md` §4.0/§4.1,
carrying controller **v0.220.2** (R-379/R-380/R-381/R-382 — the rollback, the hold, and the two
defects the live walk found in them).
| | |
|---|---|
| `GOLDEN_VERSION` | **0.220.2** |
| `GOLDEN_SHA256` | **cb439418c7005ce01bcb8126bb6385688c2f408c3c4649f6740001bc2c864ed5** |
| package | `https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.220.2/golden.tar.zst` |
| size | 657 271 965 B |
| controller image | `gitea.dooplex.hu/admin/felhom-controller:0.220.2` |
| template | `debian-13-standard_13.6-1_amd64.tar.zst` (**listed fresh**, checksum verified on download) |
| `build-golden.sh` | v3.0.0, from `felhom-agent` @ `40d857b52711` |
| `MinAgent` | **0.129.0** (from the controller CHANGELOG header — unchanged) |
## Why this bake happened in this session
The `golden-currency` gate refused the docs push: controller 0.220.2 was released with no golden
carrying it. **That block is not circular** — a golden needs the controller image, which was already
built and pushed, not the docs commit. So the gate was satisfied by doing the work it asked for,
rather than bypassed with `--no-verify`.
## Pass markers — each checked, with the negative controls
```
docker OK (overlay2 : 1 -> " docker OK (overlay2; data-root /var/lib/docker)"
including mount point : 2 -> rootfs ('/') and mp0 ('/var/lib/felhom') [there is no mp1]
upload OK (HTTP 201) : 1 -> pre-delete returned HTTP 404 (404/204 expected)
excluding : 0 <- negative control
FATAL : 0 <- negative control
```
**The 404 pre-gate was controlled before it was believed:** the same URL shape for **0.219.0 returned
HTTP 200** in the same minute, so the 404 on 0.220.2 means absent, not a wrong URL.
## Verified by ROUND TRIP
The published object was downloaded again — **HTTP 200, 657 271 965 bytes** — and its sha256
recomputed: `cb439418…` on both sides. What a machine receives is byte-identical to what was baked.
## Token hygiene
Copied **file → file** and read by a runner script inside the VM.
`systemctl show golden-bake -p Environment -p ExecStart | grep -c -F "$(cat /root/.gitea-token)"` → **0**,
and that grep was **proved able to convict first** (planted copy → **1**, copy shredded).
The same control was run on the **committed** `bake.log`: **0**, positive control **1**.
## Teardown
`pct destroy 9100 --purge`; token, runner, build script and in-VM log `shred -u`'d **after** `bake.log`
was copied out to this directory — all four confirmed absent; `poweroff`; waited for qemu to exit;
`qemu-img snapshot -a virgin`.
## NOT vouched
The hub's Day-0 artifact manifest was **not** changed and the floor was **not** raised. Both are the
operator's, and it is a **three-field** save:
| field | value |
|---|---|
| `golden_version` | **0.220.2** |
| `agent_version` | **0.130.0** |
| `min_agent` | **0.129.0** |
@@ -0,0 +1,325 @@
[golden] build-golden.sh v3.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.220.2
[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) …
Logical volume "vm-9100-disk-0" created.
Logical volume pve/vm-9100-disk-0 changed.
Creating filesystem with 8388608 4k blocks and 2097152 inodes
Filesystem UUID: 0d64b7d6-7aaa-48f0-af9b-affbb4f803aa
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
4096000, 7962624
Logical volume "vm-9100-disk-1" created.
Logical volume pve/vm-9100-disk-1 changed.
Creating filesystem with 6291456 4k blocks and 1572864 inodes
Filesystem UUID: ac639ff8-e9cf-4b7c-9938-e379acec78e9
Superblock backups stored on blocks:
32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208,
extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst'
Total bytes read: 553512960 (528MiB, 97MiB/s)
Detected container architecture: amd64
Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
done: SHA256:hwBeMD4w5PElp4dDVRT4eTbhhajRJBa3/Q1LLn9bT0g root@felhom-golden
Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
done: SHA256:QvTmAFW3Had7tj3Uoz+LoCBF/bLpfMjkRDmI9yNFeU8 root@felhom-golden
Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
done: SHA256:/dw8Ehg69wnYlNaGkDVtHV1qocB8JM4280GeTLkj6g4 root@felhom-golden
[golden] starting + installing Docker (official repo, trixie channel) …
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = (unset),
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LC_CTYPE = (unset),
LC_NUMERIC = (unset),
LC_COLLATE = (unset),
LC_TIME = (unset),
LC_MESSAGES = (unset),
LC_MONETARY = (unset),
LC_ADDRESS = (unset),
LC_IDENTIFICATION = (unset),
LC_MEASUREMENT = (unset),
LC_PAPER = (unset),
LC_TELEPHONE = (unset),
LC_NAME = (unset),
LANG = "en_US.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
locale: Cannot set LC_CTYPE to default locale: No such file or directory
locale: Cannot set LC_MESSAGES to default locale: No such file or directory
locale: Cannot set LC_ALL to default locale: No such file or directory
[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …
[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds …
[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
4f55086f7dd0: Pulling fs layer
4f55086f7dd0: Verifying Checksum
4f55086f7dd0: Download complete
4f55086f7dd0: Pull complete
Digest: sha256:5dd0d3e6e255913fc30f90b9f2b1d359cc2cbdb48090cc4b65f1676e203243cc
Status: Downloaded newer image for hello-world:latest
docker OK (overlay2; data-root /var/lib/docker)
/var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4
/mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4
both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576
[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.220.2 (no registry cred at deploy) …
WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
Configure a credential helper to remove this warning. See
https://docs.docker.com/go/credential-store/
0.220.2: Pulling from admin/felhom-controller
039e6f9f9752: Pulling fs layer
0094c3ac0914: Pulling fs layer
deca1dac7403: Pulling fs layer
11c19a33d1b8: Pulling fs layer
4f3e54e4eec5: Pulling fs layer
37c5f038fab8: Pulling fs layer
11c19a33d1b8: Waiting
4f3e54e4eec5: Waiting
37c5f038fab8: Waiting
deca1dac7403: Verifying Checksum
deca1dac7403: Download complete
11c19a33d1b8: Verifying Checksum
11c19a33d1b8: Download complete
4f3e54e4eec5: Verifying Checksum
4f3e54e4eec5: Download complete
37c5f038fab8: Verifying Checksum
37c5f038fab8: Download complete
0094c3ac0914: Verifying Checksum
0094c3ac0914: Download complete
039e6f9f9752: Verifying Checksum
039e6f9f9752: Download complete
039e6f9f9752: Pull complete
0094c3ac0914: Pull complete
deca1dac7403: Pull complete
11c19a33d1b8: Pull complete
4f3e54e4eec5: Pull complete
37c5f038fab8: Pull complete
Digest: sha256:3ad3862fff66746539781b686a78915525ab2438ea1b3c9fa4ee978b75ee7c69
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.220.2
gitea.dooplex.hu/admin/felhom-controller:0.220.2
[golden] asking the controller which infra images it manages …
[golden] baking infra images (4): traefik:v3.6.7 cloudflare/cloudflared:2026.6.0 gtstef/filebrowser:1.3.3-stable gitea.dooplex.hu/admin/felhom-samba:1.1.0 …
v3.6.7: Pulling from library/traefik
589002ba0eae: Pulling fs layer
ef63511ea6cc: Pulling fs layer
0738e5cb835e: Pulling fs layer
3e6813f70c64: Pulling fs layer
3e6813f70c64: Waiting
589002ba0eae: Verifying Checksum
589002ba0eae: Download complete
ef63511ea6cc: Verifying Checksum
ef63511ea6cc: Download complete
3e6813f70c64: Download complete
0738e5cb835e: Verifying Checksum
0738e5cb835e: Download complete
589002ba0eae: Pull complete
ef63511ea6cc: Pull complete
0738e5cb835e: Pull complete
3e6813f70c64: Pull complete
Digest: sha256:a9890c898f379c1905ee5b28342f6b408dc863f08db2dab20e46c267d1ff463a
Status: Downloaded newer image for traefik:v3.6.7
docker.io/library/traefik:v3.6.7
2026.6.0: Pulling from cloudflare/cloudflared
47de5dd0b812: Pulling fs layer
c172f21841df: Pulling fs layer
99515e7b4d35: Pulling fs layer
99ba982a9142: Pulling fs layer
d6b1b89eccac: Pulling fs layer
2780920e5dbf: Pulling fs layer
7c12895b777b: Pulling fs layer
3214acf345c0: Pulling fs layer
52630fc75a18: Pulling fs layer
dd64bf2dd177: Pulling fs layer
b839dfae01f6: Pulling fs layer
ebddc55facdc: Pulling fs layer
bdfd7f7e5bf6: Pulling fs layer
2d4d7adf6272: Pulling fs layer
40008157d8d2: Pulling fs layer
bd8962e29291: Pulling fs layer
cac2ae0193cb: Pulling fs layer
74d1dac84ecc: Pulling fs layer
dd64bf2dd177: Waiting
b839dfae01f6: Waiting
ebddc55facdc: Waiting
bdfd7f7e5bf6: Waiting
2d4d7adf6272: Waiting
99ba982a9142: Waiting
d6b1b89eccac: Waiting
7c12895b777b: Waiting
3214acf345c0: Waiting
52630fc75a18: Waiting
2780920e5dbf: Waiting
40008157d8d2: Waiting
bd8962e29291: Waiting
cac2ae0193cb: Waiting
74d1dac84ecc: Waiting
47de5dd0b812: Download complete
99515e7b4d35: Verifying Checksum
99515e7b4d35: Download complete
c172f21841df: Verifying Checksum
c172f21841df: Download complete
47de5dd0b812: Pull complete
d6b1b89eccac: Verifying Checksum
d6b1b89eccac: Download complete
2780920e5dbf: Verifying Checksum
2780920e5dbf: Download complete
99ba982a9142: Verifying Checksum
99ba982a9142: Download complete
c172f21841df: Pull complete
7c12895b777b: Verifying Checksum
7c12895b777b: Download complete
3214acf345c0: Verifying Checksum
3214acf345c0: Download complete
52630fc75a18: Download complete
dd64bf2dd177: Verifying Checksum
dd64bf2dd177: Download complete
b839dfae01f6: Verifying Checksum
b839dfae01f6: Download complete
ebddc55facdc: Verifying Checksum
ebddc55facdc: Download complete
bdfd7f7e5bf6: Verifying Checksum
bdfd7f7e5bf6: Download complete
40008157d8d2: Verifying Checksum
40008157d8d2: Download complete
bd8962e29291: Download complete
99515e7b4d35: Pull complete
2d4d7adf6272: Verifying Checksum
2d4d7adf6272: Download complete
cac2ae0193cb: Verifying Checksum
cac2ae0193cb: Download complete
74d1dac84ecc: Verifying Checksum
74d1dac84ecc: Download complete
99ba982a9142: Pull complete
d6b1b89eccac: Pull complete
2780920e5dbf: Pull complete
7c12895b777b: Pull complete
3214acf345c0: Pull complete
52630fc75a18: Pull complete
dd64bf2dd177: Pull complete
b839dfae01f6: Pull complete
ebddc55facdc: Pull complete
bdfd7f7e5bf6: Pull complete
2d4d7adf6272: Pull complete
40008157d8d2: Pull complete
bd8962e29291: Pull complete
cac2ae0193cb: Pull complete
74d1dac84ecc: Pull complete
Digest: sha256:ba461b8aa9c042156dbd39c38657fe7431bafa063220eab8d5330a523863da9f
Status: Downloaded newer image for cloudflare/cloudflared:2026.6.0
docker.io/cloudflare/cloudflared:2026.6.0
1.3.3-stable: Pulling from gtstef/filebrowser
6a0ac1617861: Pulling fs layer
ef8806083e82: Pulling fs layer
b74107c861c7: Pulling fs layer
adc935def003: Pulling fs layer
4f4fb700ef54: Pulling fs layer
18695ccc900a: Pulling fs layer
45d119d5c397: Pulling fs layer
dac52db4fc51: Pulling fs layer
6d598f86b2f2: Pulling fs layer
8aa349c8396c: Pulling fs layer
45d119d5c397: Waiting
dac52db4fc51: Waiting
6d598f86b2f2: Waiting
8aa349c8396c: Waiting
adc935def003: Waiting
4f4fb700ef54: Waiting
18695ccc900a: Waiting
6a0ac1617861: Verifying Checksum
6a0ac1617861: Download complete
b74107c861c7: Verifying Checksum
b74107c861c7: Download complete
adc935def003: Verifying Checksum
adc935def003: Download complete
4f4fb700ef54: Verifying Checksum
4f4fb700ef54: Download complete
45d119d5c397: Verifying Checksum
45d119d5c397: Download complete
18695ccc900a: Verifying Checksum
18695ccc900a: Download complete
6a0ac1617861: Pull complete
dac52db4fc51: Verifying Checksum
dac52db4fc51: Download complete
8aa349c8396c: Verifying Checksum
8aa349c8396c: Download complete
6d598f86b2f2: Verifying Checksum
6d598f86b2f2: Download complete
ef8806083e82: Verifying Checksum
ef8806083e82: Download complete
ef8806083e82: Pull complete
b74107c861c7: Pull complete
adc935def003: Pull complete
4f4fb700ef54: Pull complete
18695ccc900a: Pull complete
45d119d5c397: Pull complete
dac52db4fc51: Pull complete
6d598f86b2f2: Pull complete
8aa349c8396c: Pull complete
Digest: sha256:eb3733681db8757412632c61a99ad656f0d94ed6781bb2ea114b4d70babab78c
Status: Downloaded newer image for gtstef/filebrowser:1.3.3-stable
docker.io/gtstef/filebrowser:1.3.3-stable
1.1.0: Pulling from admin/felhom-samba
897d797d2723: Pulling fs layer
3051591aa250: Pulling fs layer
ce57a3f93416: Pulling fs layer
fb94eeec2fe1: Pulling fs layer
fb94eeec2fe1: Waiting
ce57a3f93416: Verifying Checksum
ce57a3f93416: Download complete
fb94eeec2fe1: Verifying Checksum
fb94eeec2fe1: Download complete
897d797d2723: Verifying Checksum
897d797d2723: Download complete
3051591aa250: Verifying Checksum
3051591aa250: Download complete
897d797d2723: Pull complete
3051591aa250: Pull complete
ce57a3f93416: Pull complete
fb94eeec2fe1: Pull complete
Digest: sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10
Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.1.0
gitea.dooplex.hu/admin/felhom-samba:1.1.0
[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) …
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'.
[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'.
[golden] baking the first-boot SSH host-key regeneration unit (F3) …
Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'.
[golden] identity-clean + minimize …
[golden] stop + archive …
INFO: including mount point rootfs ('/') in backup
INFO: including mount point mp0 ('/var/lib/felhom') in backup
INFO: archive file size: 626MB
INFO: Finished Backup of VM 9100 (00:00:30)
[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_08_22-18_37_25.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive)
[golden] publishing golden (657271965 bytes, sha256 cb439418c7005ce0…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.220.2/golden.tar.zst
[golden] pre-delete existing: HTTP 404 (404/204 expected)
[golden] upload OK (HTTP 201)
GOLDEN_VERSION=0.220.2
GOLDEN_SHA256=cb439418c7005ce01bcb8126bb6385688c2f408c3c4649f6740001bc2c864ed5
[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.220.2 / cb439418c7005ce01bcb8126bb6385688c2f408c3c4649f6740001bc2c864ed5
[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge)