diff --git a/documentation/audits/evidence-backup-promise-2026-09-16/phaseE-bake-full.log b/documentation/audits/evidence-backup-promise-2026-09-16/phaseE-bake-full.log index a593b09e..0aeb9d57 100644 --- a/documentation/audits/evidence-backup-promise-2026-09-16/phaseE-bake-full.log +++ b/documentation/audits/evidence-backup-promise-2026-09-16/phaseE-bake-full.log @@ -1,2 +1,321 @@ -cat: /root/.gitea-token: No such file or directory -/root/bake-run.sh: line 4: /root/build-golden.sh: No such file or directory +[golden] build-golden.sh v3.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.244.0 +[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) … + Logical volume "vm-9100-disk-0" created. + Logical volume pve/vm-9100-disk-0 changed. +Creating filesystem with 8388608 4k blocks and 2097152 inodes +Filesystem UUID: eca15c16-96ef-4653-9843-dbf37cc255ce +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, + 4096000, 7962624 + Logical volume "vm-9100-disk-1" created. + Logical volume pve/vm-9100-disk-1 changed. +Creating filesystem with 6291456 4k blocks and 1572864 inodes +Filesystem UUID: d3c47b86-d18c-4e94-b1b5-ba2e334e4eb0 +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, +extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst' +Total bytes read: 553512960 (528MiB, 109MiB/s) +Detected container architecture: amd64 +Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ... +done: SHA256:EOX2WLMMVRDCNoNwCZhDSPncStDmfAnMNLpTwBT13tM root@felhom-golden +Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ... +done: SHA256:OLKxkKrKoHe6UiQsoiraX7vYHxRkq7duh9G1ggdfApA root@felhom-golden +Creating SSH host key 'ssh_host_rsa_key' - this may take some time ... +done: SHA256:VIZqqTtAxssw3+dm5HpszeLNwkich3niJj52gbO8BnI root@felhom-golden +[golden] starting + installing Docker (official repo, trixie channel) … +apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = (unset), + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to the standard locale ("C"). +locale: Cannot set LC_CTYPE to default locale: No such file or directory +locale: Cannot set LC_MESSAGES to default locale: No such file or directory +locale: Cannot set LC_ALL to default locale: No such file or directory +apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = (unset), + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to the standard locale ("C"). +locale: Cannot set LC_CTYPE to default locale: No such file or directory +locale: Cannot set LC_MESSAGES to default locale: No such file or directory +locale: Cannot set LC_ALL to default locale: No such file or directory +[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation … +[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds … +[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) … +Unable to find image 'hello-world:latest' locally +latest: Pulling from library/hello-world +4f55086f7dd0: Pulling fs layer +4f55086f7dd0: Verifying Checksum +4f55086f7dd0: Download complete +4f55086f7dd0: Pull complete +Digest: sha256:5e23090353324d887c48ad5e5c56d294eab81588df9605b07d1afe895f9cc8f8 +Status: Downloaded newer image for hello-world:latest + docker OK (overlay2; data-root /var/lib/docker) + /var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4 + /mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4 + both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576 +[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.244.0 (no registry cred at deploy) … +0.244.0: Pulling from admin/felhom-controller +a8ac7f6c67ab: Pulling fs layer +bf30769d36e7: Pulling fs layer +044b66fbe46c: Pulling fs layer +b5c41a28e83f: Pulling fs layer +81b60ff19091: Pulling fs layer +428d65c7a3d1: Pulling fs layer +b5c41a28e83f: Waiting +81b60ff19091: Waiting +428d65c7a3d1: Waiting +a8ac7f6c67ab: Verifying Checksum +a8ac7f6c67ab: Download complete +b5c41a28e83f: Verifying Checksum +b5c41a28e83f: Download complete +044b66fbe46c: Verifying Checksum +044b66fbe46c: Download complete +428d65c7a3d1: Verifying Checksum +428d65c7a3d1: Download complete +81b60ff19091: Verifying Checksum +81b60ff19091: Download complete +bf30769d36e7: Verifying Checksum +bf30769d36e7: Download complete +a8ac7f6c67ab: Pull complete +bf30769d36e7: Pull complete +044b66fbe46c: Pull complete +b5c41a28e83f: Pull complete +81b60ff19091: Pull complete +428d65c7a3d1: Pull complete +Digest: sha256:aaa9150aefcfa1a4909c34524844fdef342fa472d7a518bb6f7890d56ad795e1 +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.244.0 +gitea.dooplex.hu/admin/felhom-controller:0.244.0 +[golden] asking the controller which infra images it manages … +[golden] baking infra images (4): traefik:v3.6.7 cloudflare/cloudflared:2026.6.0 gtstef/filebrowser:1.3.3-stable gitea.dooplex.hu/admin/felhom-samba:1.1.0 … +v3.6.7: Pulling from library/traefik +589002ba0eae: Pulling fs layer +ef63511ea6cc: Pulling fs layer +0738e5cb835e: Pulling fs layer +3e6813f70c64: Pulling fs layer +3e6813f70c64: Waiting +589002ba0eae: Verifying Checksum +589002ba0eae: Download complete +ef63511ea6cc: Download complete +3e6813f70c64: Verifying Checksum +3e6813f70c64: Download complete +589002ba0eae: Pull complete +0738e5cb835e: Verifying Checksum +0738e5cb835e: Download complete +ef63511ea6cc: Pull complete +0738e5cb835e: Pull complete +3e6813f70c64: Pull complete +Digest: sha256:a9890c898f379c1905ee5b28342f6b408dc863f08db2dab20e46c267d1ff463a +Status: Downloaded newer image for traefik:v3.6.7 +docker.io/library/traefik:v3.6.7 +2026.6.0: Pulling from cloudflare/cloudflared +47de5dd0b812: Pulling fs layer +c172f21841df: Pulling fs layer +99515e7b4d35: Pulling fs layer +99ba982a9142: Pulling fs layer +d6b1b89eccac: Pulling fs layer +2780920e5dbf: Pulling fs layer +7c12895b777b: Pulling fs layer +3214acf345c0: Pulling fs layer +52630fc75a18: Pulling fs layer +dd64bf2dd177: Pulling fs layer +b839dfae01f6: Pulling fs layer +ebddc55facdc: Pulling fs layer +bdfd7f7e5bf6: Pulling fs layer +2d4d7adf6272: Pulling fs layer +40008157d8d2: Pulling fs layer +bd8962e29291: Pulling fs layer +cac2ae0193cb: Pulling fs layer +74d1dac84ecc: Pulling fs layer +99ba982a9142: Waiting +d6b1b89eccac: Waiting +2780920e5dbf: Waiting +7c12895b777b: Waiting +3214acf345c0: Waiting +52630fc75a18: Waiting +dd64bf2dd177: Waiting +b839dfae01f6: Waiting +ebddc55facdc: Waiting +bdfd7f7e5bf6: Waiting +2d4d7adf6272: Waiting +40008157d8d2: Waiting +bd8962e29291: Waiting +cac2ae0193cb: Waiting +74d1dac84ecc: Waiting +47de5dd0b812: Download complete +c172f21841df: Verifying Checksum +c172f21841df: Download complete +99515e7b4d35: Verifying Checksum +99515e7b4d35: Download complete +99ba982a9142: Verifying Checksum +99ba982a9142: Download complete +d6b1b89eccac: Verifying Checksum +d6b1b89eccac: Download complete +2780920e5dbf: Verifying Checksum +2780920e5dbf: Download complete +47de5dd0b812: Pull complete +7c12895b777b: Verifying Checksum +7c12895b777b: Download complete +3214acf345c0: Verifying Checksum +3214acf345c0: Download complete +52630fc75a18: Verifying Checksum +52630fc75a18: Download complete +dd64bf2dd177: Verifying Checksum +dd64bf2dd177: Download complete +c172f21841df: Pull complete +b839dfae01f6: Verifying Checksum +b839dfae01f6: Download complete +ebddc55facdc: Verifying Checksum +ebddc55facdc: Download complete +bdfd7f7e5bf6: Verifying Checksum +bdfd7f7e5bf6: Download complete +40008157d8d2: Verifying Checksum +40008157d8d2: Download complete +2d4d7adf6272: Verifying Checksum +2d4d7adf6272: Download complete +bd8962e29291: Verifying Checksum +bd8962e29291: Download complete +cac2ae0193cb: Verifying Checksum +cac2ae0193cb: Download complete +99515e7b4d35: Pull complete +74d1dac84ecc: Verifying Checksum +74d1dac84ecc: Download complete +99ba982a9142: Pull complete +d6b1b89eccac: Pull complete +2780920e5dbf: Pull complete +7c12895b777b: Pull complete +3214acf345c0: Pull complete +52630fc75a18: Pull complete +dd64bf2dd177: Pull complete +b839dfae01f6: Pull complete +ebddc55facdc: Pull complete +bdfd7f7e5bf6: Pull complete +2d4d7adf6272: Pull complete +40008157d8d2: Pull complete +bd8962e29291: Pull complete +cac2ae0193cb: Pull complete +74d1dac84ecc: Pull complete +Digest: sha256:ba461b8aa9c042156dbd39c38657fe7431bafa063220eab8d5330a523863da9f +Status: Downloaded newer image for cloudflare/cloudflared:2026.6.0 +docker.io/cloudflare/cloudflared:2026.6.0 +1.3.3-stable: Pulling from gtstef/filebrowser +6a0ac1617861: Pulling fs layer +ef8806083e82: Pulling fs layer +b74107c861c7: Pulling fs layer +adc935def003: Pulling fs layer +4f4fb700ef54: Pulling fs layer +18695ccc900a: Pulling fs layer +45d119d5c397: Pulling fs layer +dac52db4fc51: Pulling fs layer +6d598f86b2f2: Pulling fs layer +8aa349c8396c: Pulling fs layer +dac52db4fc51: Waiting +6d598f86b2f2: Waiting +8aa349c8396c: Waiting +4f4fb700ef54: Waiting +18695ccc900a: Waiting +45d119d5c397: Waiting +adc935def003: Waiting +6a0ac1617861: Verifying Checksum +6a0ac1617861: Download complete +adc935def003: Verifying Checksum +adc935def003: Download complete +4f4fb700ef54: Verifying Checksum +4f4fb700ef54: Download complete +b74107c861c7: Verifying Checksum +b74107c861c7: Download complete +45d119d5c397: Verifying Checksum +45d119d5c397: Download complete +18695ccc900a: Verifying Checksum +18695ccc900a: Download complete +dac52db4fc51: Verifying Checksum +dac52db4fc51: Download complete +6a0ac1617861: Pull complete +ef8806083e82: Verifying Checksum +ef8806083e82: Download complete +6d598f86b2f2: Verifying Checksum +6d598f86b2f2: Download complete +8aa349c8396c: Download complete +ef8806083e82: Pull complete +b74107c861c7: Pull complete +adc935def003: Pull complete +4f4fb700ef54: Pull complete +18695ccc900a: Pull complete +45d119d5c397: Pull complete +dac52db4fc51: Pull complete +6d598f86b2f2: Pull complete +8aa349c8396c: Pull complete +Digest: sha256:eb3733681db8757412632c61a99ad656f0d94ed6781bb2ea114b4d70babab78c +Status: Downloaded newer image for gtstef/filebrowser:1.3.3-stable +docker.io/gtstef/filebrowser:1.3.3-stable +1.1.0: Pulling from admin/felhom-samba +897d797d2723: Pulling fs layer +3051591aa250: Pulling fs layer +ce57a3f93416: Pulling fs layer +fb94eeec2fe1: Pulling fs layer +fb94eeec2fe1: Waiting +ce57a3f93416: Verifying Checksum +ce57a3f93416: Download complete +fb94eeec2fe1: Verifying Checksum +fb94eeec2fe1: Download complete +897d797d2723: Verifying Checksum +897d797d2723: Download complete +3051591aa250: Verifying Checksum +3051591aa250: Download complete +897d797d2723: Pull complete +3051591aa250: Pull complete +ce57a3f93416: Pull complete +fb94eeec2fe1: Pull complete +Digest: sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10 +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.1.0 +gitea.dooplex.hu/admin/felhom-samba:1.1.0 +[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'. +[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'. +[golden] baking the first-boot SSH host-key regeneration unit (F3) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'. +[golden] identity-clean + minimize … +[golden] stop + archive … +INFO: including mount point rootfs ('/') in backup +INFO: including mount point mp0 ('/var/lib/felhom') in backup +INFO: archive file size: 623MB +INFO: Finished Backup of VM 9100 (00:00:31) +[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_09_16-17_29_41.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive) +[golden] publishing golden (653609190 bytes, sha256 18328a3c7579628b…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.244.0/golden.tar.zst +[golden] pre-delete existing: HTTP 404 (404/204 expected) +[golden] upload OK (HTTP 201) +GOLDEN_VERSION=0.244.0 +GOLDEN_SHA256=18328a3c7579628b8a7e9639777043db48063c86d37a2e0c221a6ccba6d755a0 +[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.244.0 / 18328a3c7579628b8a7e9639777043db48063c86d37a2e0c221a6ccba6d755a0 +[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge) diff --git a/documentation/audits/evidence-backup-promise-2026-09-16/phaseE-golden-bake.txt b/documentation/audits/evidence-backup-promise-2026-09-16/phaseE-golden-bake.txt index c7a20fcb..4408a2e9 100644 --- a/documentation/audits/evidence-backup-promise-2026-09-16/phaseE-golden-bake.txt +++ b/documentation/audits/evidence-backup-promise-2026-09-16/phaseE-golden-bake.txt @@ -29,3 +29,71 @@ script + token landed in the VM (both non-empty) bake launched as a transient unit at 2026-09-16T15:15:37Z token leak check on the unit (must be 0, and the token is non-empty so the grep cannot match everything): 0 +## 2026-09-16T15:19Z — MY WATCHER WAS KILLED, THE BAKE WAS NOT. Recorded so the two are not confused. +## The system stopped my background watcher shell for low memory (DooPlex: 62 GB total, ~11 GB +## available, the largest consumer a 10.6 GB java service; the bake VM itself was 4.1 GB RSS). +## The VM is daemonized and the bake runs INSIDE it as a transient systemd unit, so it kept going: +## `systemctl is-active golden-bake` = active, 120 log lines, at „baking infra images (4)". +## Production was unaffected — the k3s pods stayed Running and the hub kept serving. +## CONSEQUENCE: the watcher's post-bake steps (copy the log out, markers, token-leak control, +## destroy 9100, poweroff, revert to virgin) did NOT run and are done by hand below. + unit finished: inactive at 2026-09-16T15:21:10Z + log copied off the machine FIRST (R-320): 316 lines + markers: overlay2=1 mountpoints=2 upload=0 FATAL=0 excluding=0 + token-leak control — planted copy must be 1: 1; the COMMITTED log must be 0: 0 + teardown: shredded + qemu processes now: 0 + disk reverted to virgin +## bake finished 2026-09-16T15:21:37Z +## MY SECOND MISTAKE ON THIS BAKE, and it is the more serious one (15:21Z): +## The bake BUILT the golden and then printed „Gitea publish SKIPPED (set GITEA_USER+GITEA_TOKEN or +## REGISTRY_USER+REGISTRY_TOKEN to enable)". My runner exported GITEA_TOKEN only. The script needs +## BOTH (build-golden.sh:417-418 — PUB_USER="${GITEA_USER:-${REGISTRY_USER:-}}"). +## Then I tore the guest down and reverted the disk BEFORE checking the outcome, so the archive +## (local:backup/vzdump-lxc-9100-2026_09_16-17_19_53.tar.zst, 623 MB) went with it. +## Registry check confirms the miss: golden 0.244.0 -> HTTP 404; newest published is still 0.243.0. +## TWO rules of this project were broken by one habit — I read the message the script prints +## ("DONE. golden archive volid: …") as the outcome, when the outcome is the UPLOAD: +## * "an absent log line is not evidence" — I checked markers AFTER destroying, not before; +## * "presence is not success" — a built archive is an attempt, the published package is the result. +## The re-run below sets both variables and REFUSES to tear anything down until the upload marker +## and the registry both say the golden exists. +## 2026-09-16T15:23:07Z RE-BAKE golden 0.244.0 — with GITEA_USER this time, and teardown gated on the OUTCOME + reverted to virgin + ssh up: pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve) + template: debian-13-standard_13.6-1_amd64.tar.zst + bake launched 2026-09-16T15:23:54Z; token leak check on the unit (must be 0): 0 + unit state: inactive at 2026-09-16T15:24:10Z + log: 1 lines + markers: overlay2=0 mountpoints=0 upload=0 FATAL=0 skipped=0 + token-leak control — planted must be 1: 1; committed log must be 0: 0 + REGISTRY CHECK (the outcome, not the attempt): golden 0.244.0 -> http=404 + NOT PUBLISHED — the VM and the archive are LEFT IN PLACE on purpose so the artifact is not lost again. +## re-bake done 2026-09-16T15:24:10Z +## 2026-09-16T15:25:31Z THIRD attempt — same running VM (template + files already there), fixed: + (a) build-golden.sh is made EXECUTABLE (it landed 0644 and died 'Permission denied'); + (b) GITEA_USER is the ADMIN account, not the first credential line (the package namespace is admin). + VM still up: pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve) + script executable: yes; token non-empty: yes + template already on the box: debian-13-standard_13.6-1_amd64.tar.zst + launched 2026-09-16T15:25:35Z; leak check on the unit (must be 0): 0 +## MY THIRD MISTAKE ON THIS BAKE (15:24Z) — two of them in one run, both mine, both recorded: +## (a) When I rewrote the bake script I DROPPED the `chmod 0700 /root/build-golden.sh` line the +## first version had. The file landed 0644 and the run died instantly: +## „/root/bake-run.sh: line 5: /root/build-golden.sh: Permission denied" +## (b) `GITEA_USER` was taken with `grep -m1` from the credentials file and resolved to `kisfenyo`, +## but the package namespace is `admin` (`/api/packages/admin/generic/…`). Even a runnable +## script would have authenticated as the wrong account. +## AND THE UNIT REPORTED SUCCESS: `systemctl show golden-bake -p Result` = `Result=success`, +## `ExecMainStatus=0` — because the UNIT ran fine; the script inside it failed. That is exactly the +## "exit codes that lie" class this project keeps re-learning, and it is why the teardown here is +## gated on the REGISTRY answering 200 for the package, not on any exit code or log sentence. +## Nothing was lost this time: the gate held the VM and the archive in place. + unit state: inactive at 2026-09-16T15:30:47Z + log: 321 lines; markers: overlay2=1 mountpoints=2 upload=1 FATAL=0 skipped=0 +GOLDEN_VERSION=0.244.0 +GOLDEN_SHA256=18328a3c7579628b8a7e9639777043db48063c86d37a2e0c221a6ccba6d755a0 + token-leak control — planted must be 1: 1; committed log must be 0: 0 + REGISTRY CHECK (the outcome): golden 0.244.0 -> http=200 + PUBLISHED; guest destroyed, qemu exited, disk reverted to virgin +## attempt 3 done 2026-09-16T15:31:23Z diff --git a/documentation/audits/evidence-backup-promise-2026-09-16/phaseE-vouch.txt b/documentation/audits/evidence-backup-promise-2026-09-16/phaseE-vouch.txt new file mode 100644 index 00000000..8ddbd986 --- /dev/null +++ b/documentation/audits/evidence-backup-promise-2026-09-16/phaseE-vouch.txt @@ -0,0 +1,7 @@ +## 2026-09-16T15:31:52Z vouch + floor for golden 0.244.0 + BEFORE: golden=0.243.0 agent=0.131.0 min_agent=0.131.0 + vouch POST -> http=200 + AFTER: golden=0.244.0 agent=0.131.0 min_agent=0.131.0 + floor BEFORE: controller=0.242.0 agent=0.129.0 + floor POST -> http=200 + floor AFTER: controller=0.244.0 agent=0.131.0