diff --git a/documentation/audits/DRILL-prove-fixes-0243-2026-09-16.md b/documentation/audits/DRILL-prove-fixes-0243-2026-09-16.md new file mode 100644 index 00000000..477aab92 --- /dev/null +++ b/documentation/audits/DRILL-prove-fixes-0243-2026-09-16.md @@ -0,0 +1,45 @@ +# DRILL — prove the P1 fixes on a fresh box (2026-09-16) + +**Interventions: _pending_** (O1/O2 pre-declared, counted apart). +**Ready for a volunteer: _pending_.** +**The automatic connect e-mail: _pending_.** + +> Baselines at start (re-verified against live Gitea): controller `383a30b3c07b` v0.243.0 (`Unreleased`: the +> „0 B" tile fix), agent `e98b857684f4` v0.131.0, felhom.eu `351296114c4d` hub v0.114.0, catalog `94bc5febaca2`. +> Golden before this run: 0.242.0 (2026-09-14). Customer `tester-1`, domain `enkicsifelhom.hu`, e-mail +> `tester1@felhom.eu`, **no host**, DR tier ticked, ep0 token held with no descriptor, namespace EMPTY +> (`phase3-ep0-before.txt`). + +## Phase 0 — rulings, the golden, the N100 + +**Ruling 1 (2026-09-16, operator).** The signing keys stay on DooPlex, owner-only +(`/mnt/5_hdd/felhom.eu/felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519`, mode 0600); +CC may sign `agent_update` jobs with them until the first PAYING customer — testers excluded. Recorded in +`CONTEXT.md` and `04-control-plane-authorization.md` §3.1. **R-533 closes** (path + mode); +**R-530 narrows** to "a fleet rollout step is still undesigned". + +**N100 signed and updated (R-530).** `felhom-opsign -op agent_update -host demo-felhom-8363b5 -key-id +felhom-op-1 … -agent-version 0.131.0 -sha256 1118b552…c9c`, queued 08:51:34Z, **the box took it at +09:04:43Z** (its own poll, ~13 min), dwelled 60 s and committed; `controller-supervisor: started` +(interval 30s, confirm 2, crash-loop 3/15m) in its journal. Peti's box untouched. The N100 keeps +controller 0.242.0; its floor was not moved by this run. + +**Ruling 2 — hub v0.115.0 (R-529).** `host_stale` / `host_down` / `host_recovered` join the `node_*` +cooldown bypass with the same 5-minute dedupe. Red-proof: with the three host types removed the test fails +at "host_stale 39 min after the previous one was suppressed (sent=1)". Recorded in `08-alarm-ladder.md` §6.2 +beside the 2026-09-15 node ruling. Deployed by GitOps: ArgoCD Synced, `deploy/hub` image 0.115.0. + +**Golden 0.243.0 baked and vouched.** First attempt FAILED and is recorded: the template picker took the +LAST `debian-13` line, which is **arm64**, and the container would not start ("Detected container +architecture: arm64"); no golden was produced, the scratch guest was destroyed and the disk reverted. +Re-run with the amd64 template: `GOLDEN_VERSION=0.243.0`, +`GOLDEN_SHA256=e2d1843c8b648910ddde7cef4fe9f9ba2ee25002cdb58fbc42543a3e8967c10a`. Markers from the saved log +(`phase0-bake-full.log`, 326 lines): `docker OK (overlay2` ×1, `including mount point` ×2 (rootfs + mp0), +`upload OK (HTTP 201)` ×1, `FATAL` 0, `excluding` 0. Token leak checks: control copy 1, committed log 0. +Vouched as a three-field change — golden 0.243.0 + agent 0.131.0 + min agent 0.131.0 — hub logged +`Artifact manifest set: agent=0.131.0 golden=0.243.0 min_agent="0.131.0" wrapper_sha=true`. + +## Phase 1 — the walk + +_(in progress)_ + diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/phase0-bake-full.log b/documentation/audits/evidence-drill-0243-2026-09-16/phase0-bake-full.log new file mode 100644 index 00000000..16e34374 --- /dev/null +++ b/documentation/audits/evidence-drill-0243-2026-09-16/phase0-bake-full.log @@ -0,0 +1,326 @@ +[golden] build-golden.sh v3.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.243.0 +[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) … + Logical volume "vm-9100-disk-0" created. + Logical volume pve/vm-9100-disk-0 changed. +Creating filesystem with 8388608 4k blocks and 2097152 inodes +Filesystem UUID: 0fdf18ca-d88c-42c5-8b0f-ed31623da33a +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, + 4096000, 7962624 + Logical volume "vm-9100-disk-1" created. + Logical volume pve/vm-9100-disk-1 changed. +Creating filesystem with 6291456 4k blocks and 1572864 inodes +Filesystem UUID: d425f3e9-0c5d-40cd-89ce-80f2988e1294 +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, +extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst' +Total bytes read: 553512960 (528MiB, 117MiB/s) +Detected container architecture: amd64 +Creating SSH host key 'ssh_host_rsa_key' - this may take some time ... +done: SHA256:JfhGxgHqSFFzsiIiMBN4G/hL338Pff/Wyft5RQzy6MI root@felhom-golden +Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ... +done: SHA256:nZa+KA0vkryve5RLYv6h9XyQjQGiI8IqZ5Uank6oL4A root@felhom-golden +Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ... +done: SHA256:7gT/lVkeweXCbxPAQQam7E4ZvsSjwloKwb6M3GaB2wE root@felhom-golden +[golden] starting + installing Docker (official repo, trixie channel) … +apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = (unset), + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to the standard locale ("C"). +locale: Cannot set LC_CTYPE to default locale: No such file or directory +locale: Cannot set LC_MESSAGES to default locale: No such file or directory +locale: Cannot set LC_ALL to default locale: No such file or directory +apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct! +perl: warning: Setting locale failed. +perl: warning: Please check that your locale settings: + LANGUAGE = (unset), + LC_ALL = (unset), + LC_CTYPE = (unset), + LC_NUMERIC = (unset), + LC_COLLATE = (unset), + LC_TIME = (unset), + LC_MESSAGES = (unset), + LC_MONETARY = (unset), + LC_ADDRESS = (unset), + LC_IDENTIFICATION = (unset), + LC_MEASUREMENT = (unset), + LC_PAPER = (unset), + LC_TELEPHONE = (unset), + LC_NAME = (unset), + LANG = "en_US.UTF-8" + are supported and installed on your system. +perl: warning: Falling back to the standard locale ("C"). +locale: Cannot set LC_CTYPE to default locale: No such file or directory +locale: Cannot set LC_MESSAGES to default locale: No such file or directory +locale: Cannot set LC_ALL to default locale: No such file or directory +[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation … +[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds … +[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) … +Unable to find image 'hello-world:latest' locally +latest: Pulling from library/hello-world +4f55086f7dd0: Pulling fs layer +4f55086f7dd0: Verifying Checksum +4f55086f7dd0: Download complete +4f55086f7dd0: Pull complete +Digest: sha256:5e23090353324d887c48ad5e5c56d294eab81588df9605b07d1afe895f9cc8f8 +Status: Downloaded newer image for hello-world:latest + docker OK (overlay2; data-root /var/lib/docker) + /var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4 + /mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4 + both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576 +[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.243.0 (no registry cred at deploy) … + +WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'. +Configure a credential helper to remove this warning. See +https://docs.docker.com/go/credential-store/ + +0.243.0: Pulling from admin/felhom-controller +a8ac7f6c67ab: Pulling fs layer +bf30769d36e7: Pulling fs layer +044b66fbe46c: Pulling fs layer +b5c41a28e83f: Pulling fs layer +24733ed700a4: Pulling fs layer +ca058f1603e5: Pulling fs layer +b5c41a28e83f: Waiting +24733ed700a4: Waiting +ca058f1603e5: Waiting +a8ac7f6c67ab: Verifying Checksum +a8ac7f6c67ab: Download complete +044b66fbe46c: Verifying Checksum +044b66fbe46c: Download complete +b5c41a28e83f: Verifying Checksum +b5c41a28e83f: Download complete +24733ed700a4: Verifying Checksum +24733ed700a4: Download complete +ca058f1603e5: Verifying Checksum +ca058f1603e5: Download complete +bf30769d36e7: Verifying Checksum +bf30769d36e7: Download complete +a8ac7f6c67ab: Pull complete +bf30769d36e7: Pull complete +044b66fbe46c: Pull complete +b5c41a28e83f: Pull complete +24733ed700a4: Pull complete +ca058f1603e5: Pull complete +Digest: sha256:b5314c302ff959bbc1dff0e59fefd018d602ced8f986208ea9ac930d6e3f0905 +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.243.0 +gitea.dooplex.hu/admin/felhom-controller:0.243.0 +[golden] asking the controller which infra images it manages … +[golden] baking infra images (4): traefik:v3.6.7 cloudflare/cloudflared:2026.6.0 gtstef/filebrowser:1.3.3-stable gitea.dooplex.hu/admin/felhom-samba:1.1.0 … +v3.6.7: Pulling from library/traefik +589002ba0eae: Pulling fs layer +ef63511ea6cc: Pulling fs layer +0738e5cb835e: Pulling fs layer +3e6813f70c64: Pulling fs layer +3e6813f70c64: Waiting +ef63511ea6cc: Verifying Checksum +ef63511ea6cc: Download complete +3e6813f70c64: Verifying Checksum +3e6813f70c64: Download complete +589002ba0eae: Verifying Checksum +589002ba0eae: Download complete +0738e5cb835e: Verifying Checksum +0738e5cb835e: Download complete +589002ba0eae: Pull complete +ef63511ea6cc: Pull complete +0738e5cb835e: Pull complete +3e6813f70c64: Pull complete +Digest: sha256:a9890c898f379c1905ee5b28342f6b408dc863f08db2dab20e46c267d1ff463a +Status: Downloaded newer image for traefik:v3.6.7 +docker.io/library/traefik:v3.6.7 +2026.6.0: Pulling from cloudflare/cloudflared +47de5dd0b812: Pulling fs layer +c172f21841df: Pulling fs layer +99515e7b4d35: Pulling fs layer +99ba982a9142: Pulling fs layer +d6b1b89eccac: Pulling fs layer +2780920e5dbf: Pulling fs layer +7c12895b777b: Pulling fs layer +3214acf345c0: Pulling fs layer +52630fc75a18: Pulling fs layer +dd64bf2dd177: Pulling fs layer +b839dfae01f6: Pulling fs layer +ebddc55facdc: Pulling fs layer +bdfd7f7e5bf6: Pulling fs layer +2d4d7adf6272: Pulling fs layer +40008157d8d2: Pulling fs layer +bd8962e29291: Pulling fs layer +cac2ae0193cb: Pulling fs layer +74d1dac84ecc: Pulling fs layer +99ba982a9142: Waiting +d6b1b89eccac: Waiting +2780920e5dbf: Waiting +7c12895b777b: Waiting +3214acf345c0: Waiting +52630fc75a18: Waiting +dd64bf2dd177: Waiting +b839dfae01f6: Waiting +ebddc55facdc: Waiting +bdfd7f7e5bf6: Waiting +2d4d7adf6272: Waiting +40008157d8d2: Waiting +bd8962e29291: Waiting +cac2ae0193cb: Waiting +74d1dac84ecc: Waiting +47de5dd0b812: Download complete +c172f21841df: Verifying Checksum +c172f21841df: Download complete +47de5dd0b812: Pull complete +99515e7b4d35: Verifying Checksum +99515e7b4d35: Download complete +99ba982a9142: Verifying Checksum +99ba982a9142: Download complete +d6b1b89eccac: Verifying Checksum +d6b1b89eccac: Download complete +2780920e5dbf: Verifying Checksum +2780920e5dbf: Download complete +7c12895b777b: Verifying Checksum +7c12895b777b: Download complete +3214acf345c0: Verifying Checksum +3214acf345c0: Download complete +c172f21841df: Pull complete +52630fc75a18: Verifying Checksum +52630fc75a18: Download complete +dd64bf2dd177: Download complete +b839dfae01f6: Verifying Checksum +b839dfae01f6: Download complete +ebddc55facdc: Verifying Checksum +ebddc55facdc: Download complete +bdfd7f7e5bf6: Verifying Checksum +bdfd7f7e5bf6: Download complete +40008157d8d2: Verifying Checksum +40008157d8d2: Download complete +bd8962e29291: Verifying Checksum +bd8962e29291: Download complete +2d4d7adf6272: Verifying Checksum +2d4d7adf6272: Download complete +99515e7b4d35: Pull complete +cac2ae0193cb: Verifying Checksum +cac2ae0193cb: Download complete +74d1dac84ecc: Verifying Checksum +74d1dac84ecc: Download complete +99ba982a9142: Pull complete +d6b1b89eccac: Pull complete +2780920e5dbf: Pull complete +7c12895b777b: Pull complete +3214acf345c0: Pull complete +52630fc75a18: Pull complete +dd64bf2dd177: Pull complete +b839dfae01f6: Pull complete +ebddc55facdc: Pull complete +bdfd7f7e5bf6: Pull complete +2d4d7adf6272: Pull complete +40008157d8d2: Pull complete +bd8962e29291: Pull complete +cac2ae0193cb: Pull complete +74d1dac84ecc: Pull complete +Digest: sha256:ba461b8aa9c042156dbd39c38657fe7431bafa063220eab8d5330a523863da9f +Status: Downloaded newer image for cloudflare/cloudflared:2026.6.0 +docker.io/cloudflare/cloudflared:2026.6.0 +1.3.3-stable: Pulling from gtstef/filebrowser +6a0ac1617861: Pulling fs layer +ef8806083e82: Pulling fs layer +b74107c861c7: Pulling fs layer +adc935def003: Pulling fs layer +4f4fb700ef54: Pulling fs layer +18695ccc900a: Pulling fs layer +45d119d5c397: Pulling fs layer +dac52db4fc51: Pulling fs layer +6d598f86b2f2: Pulling fs layer +8aa349c8396c: Pulling fs layer +4f4fb700ef54: Waiting +adc935def003: Waiting +18695ccc900a: Waiting +45d119d5c397: Waiting +dac52db4fc51: Waiting +6d598f86b2f2: Waiting +8aa349c8396c: Waiting +6a0ac1617861: Verifying Checksum +6a0ac1617861: Download complete +adc935def003: Verifying Checksum +adc935def003: Download complete +4f4fb700ef54: Verifying Checksum +4f4fb700ef54: Download complete +ef8806083e82: Verifying Checksum +ef8806083e82: Download complete +b74107c861c7: Verifying Checksum +b74107c861c7: Download complete +45d119d5c397: Verifying Checksum +45d119d5c397: Download complete +dac52db4fc51: Verifying Checksum +dac52db4fc51: Download complete +6d598f86b2f2: Verifying Checksum +6d598f86b2f2: Download complete +18695ccc900a: Download complete +6a0ac1617861: Pull complete +8aa349c8396c: Verifying Checksum +8aa349c8396c: Download complete +ef8806083e82: Pull complete +b74107c861c7: Pull complete +adc935def003: Pull complete +4f4fb700ef54: Pull complete +18695ccc900a: Pull complete +45d119d5c397: Pull complete +dac52db4fc51: Pull complete +6d598f86b2f2: Pull complete +8aa349c8396c: Pull complete +Digest: sha256:eb3733681db8757412632c61a99ad656f0d94ed6781bb2ea114b4d70babab78c +Status: Downloaded newer image for gtstef/filebrowser:1.3.3-stable +docker.io/gtstef/filebrowser:1.3.3-stable +1.1.0: Pulling from admin/felhom-samba +897d797d2723: Pulling fs layer +3051591aa250: Pulling fs layer +ce57a3f93416: Pulling fs layer +fb94eeec2fe1: Pulling fs layer +fb94eeec2fe1: Waiting +ce57a3f93416: Verifying Checksum +ce57a3f93416: Download complete +fb94eeec2fe1: Verifying Checksum +fb94eeec2fe1: Download complete +897d797d2723: Verifying Checksum +897d797d2723: Download complete +3051591aa250: Verifying Checksum +3051591aa250: Download complete +897d797d2723: Pull complete +3051591aa250: Pull complete +ce57a3f93416: Pull complete +fb94eeec2fe1: Pull complete +Digest: sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10 +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.1.0 +gitea.dooplex.hu/admin/felhom-samba:1.1.0 +[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'. +[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'. +[golden] baking the first-boot SSH host-key regeneration unit (F3) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'. +[golden] identity-clean + minimize … +[golden] stop + archive … +INFO: including mount point rootfs ('/') in backup +INFO: including mount point mp0 ('/var/lib/felhom') in backup +INFO: archive file size: 623MB +INFO: Finished Backup of VM 9100 (00:00:44) +[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_09_16-11_01_01.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive) +[golden] publishing golden (653997919 bytes, sha256 e2d1843c8b648910…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.243.0/golden.tar.zst +[golden] pre-delete existing: HTTP 404 (404/204 expected) +[golden] upload OK (HTTP 201) +GOLDEN_VERSION=0.243.0 +GOLDEN_SHA256=e2d1843c8b648910ddde7cef4fe9f9ba2ee25002cdb58fbc42543a3e8967c10a +[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.243.0 / e2d1843c8b648910ddde7cef4fe9f9ba2ee25002cdb58fbc42543a3e8967c10a +[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge) diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/phase0-golden-bake.txt b/documentation/audits/evidence-drill-0243-2026-09-16/phase0-golden-bake.txt new file mode 100644 index 00000000..46a4faec --- /dev/null +++ b/documentation/audits/evidence-drill-0243-2026-09-16/phase0-golden-bake.txt @@ -0,0 +1,101 @@ +## 2026-09-16T08:52:26Z revert to virgin +## 2026-09-16T08:52:26Z cold boot +## 2026-09-16T08:52:59Z ssh up: pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve) +system debian-13-standard_13.6-1_amd64.tar.zst +system debian-13-standard_13.6-1_arm64.tar.zst +## 2026-09-16T08:53:06Z template: debian-13-standard_13.6-1_arm64.tar.zst +calculating checksum...OK, checksum verified +download of 'http://download.proxmox.com/images/system/debian-13-standard_13.6-1_arm64.tar.zst' to '/var/lib/vz/template/cache/debian-13-standard_13.6-1_arm64.tar.zst' finished +## 2026-09-16T08:53:13Z launching bake (transient unit) +## 2026-09-16T08:53:13Z token leak check on the unit: 0 +## 2026-09-16T08:53:55Z bake unit state: inactive +[golden] build-golden.sh v3.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.243.0 +[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) … + Logical volume "vm-9100-disk-0" created. + Logical volume pve/vm-9100-disk-0 changed. +Creating filesystem with 8388608 4k blocks and 2097152 inodes +Filesystem UUID: d27b2e03-4001-489d-a62c-938da32e3be5 +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, + 4096000, 7962624 + Logical volume "vm-9100-disk-1" created. + Logical volume pve/vm-9100-disk-1 changed. +Creating filesystem with 6291456 4k blocks and 1572864 inodes +Filesystem UUID: 1072606a-823b-4274-9dbc-7ff74bd5790b +Superblock backups stored on blocks: + 32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, +extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_arm64.tar.zst' +Total bytes read: 593008640 (566MiB, 97MiB/s) +Detected container architecture: arm64 +Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ... +done: SHA256:rZ9I+1iM0R+3Z6Zh2NfLJdD981ROGZ2M233hGq5GFT4 root@felhom-golden +Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ... +done: SHA256:KjILoijRNE+lzXid6lepHRX9NF005HRiMIvEQmO6R3M root@felhom-golden +Creating SSH host key 'ssh_host_rsa_key' - this may take some time ... +done: SHA256:X8iecif9F1WuFTElfBp7qzWlz9vf/q2vNvzuFyAmabM root@felhom-golden +[golden] starting + installing Docker (official repo, trixie channel) … +sync_wait: 34 An error occurred in another process (expected sequence number 7) +__lxc_start: 2288 Failed to spawn container "9100" +startup for container '9100' failed +## 2026-09-16T08:53:55Z markers: overlay2=0 mountpoints=0 upload=0 fatal=0 excluding=0 +## 2026-09-16T08:53:55Z token grep control (must be 1): 1; committed log (must be 0): 0 +## 2026-09-16T08:54:16Z qemu exited: 0 ; reverting disk to virgin +## 2026-09-16T08:54:16Z bake done +## RE-RUN after the arm64 mistake 2026-09-16T08:55:35Z +## 2026-09-16T08:55:35Z revert to virgin +## 2026-09-16T08:55:35Z cold boot +## 2026-09-16T08:56:07Z ssh up: pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve) +system debian-13-standard_13.6-1_amd64.tar.zst +system debian-13-standard_13.6-1_arm64.tar.zst +## 2026-09-16T08:56:15Z template: debian-13-standard_13.6-1_amd64.tar.zst +calculating checksum...OK, checksum verified +download of 'http://download.proxmox.com/images/system/debian-13-standard_13.6-1_amd64.tar.zst' to '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst' finished +## 2026-09-16T08:56:31Z launching bake (transient unit) +## 2026-09-16T08:56:31Z token leak check on the unit: 0 +## 2026-09-16T09:02:21Z bake unit state: inactive +897d797d2723: Pulling fs layer +3051591aa250: Pulling fs layer +ce57a3f93416: Pulling fs layer +fb94eeec2fe1: Pulling fs layer +fb94eeec2fe1: Waiting +ce57a3f93416: Verifying Checksum +ce57a3f93416: Download complete +fb94eeec2fe1: Verifying Checksum +fb94eeec2fe1: Download complete +897d797d2723: Verifying Checksum +897d797d2723: Download complete +3051591aa250: Verifying Checksum +3051591aa250: Download complete +897d797d2723: Pull complete +3051591aa250: Pull complete +ce57a3f93416: Pull complete +fb94eeec2fe1: Pull complete +Digest: sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10 +Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.1.0 +gitea.dooplex.hu/admin/felhom-samba:1.1.0 +[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'. +[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'. +[golden] baking the first-boot SSH host-key regeneration unit (F3) … +Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'. +[golden] identity-clean + minimize … +[golden] stop + archive … +INFO: including mount point rootfs ('/') in backup +INFO: including mount point mp0 ('/var/lib/felhom') in backup +INFO: archive file size: 623MB +INFO: Finished Backup of VM 9100 (00:00:44) +[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_09_16-11_01_01.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive) +[golden] publishing golden (653997919 bytes, sha256 e2d1843c8b648910…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.243.0/golden.tar.zst +[golden] pre-delete existing: HTTP 404 (404/204 expected) +[golden] upload OK (HTTP 201) +GOLDEN_VERSION=0.243.0 +GOLDEN_SHA256=e2d1843c8b648910ddde7cef4fe9f9ba2ee25002cdb58fbc42543a3e8967c10a +[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.243.0 / e2d1843c8b648910ddde7cef4fe9f9ba2ee25002cdb58fbc42543a3e8967c10a +[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge) +## 2026-09-16T09:02:21Z markers: overlay2=1 mountpoints=2 upload=1 fatal=0 excluding=0 +GOLDEN_VERSION=0.243.0 +GOLDEN_SHA256=e2d1843c8b648910ddde7cef4fe9f9ba2ee25002cdb58fbc42543a3e8967c10a +## 2026-09-16T09:02:21Z token grep control (must be 1): 1; committed log (must be 0): 0 +## 2026-09-16T09:02:47Z qemu exited: 0 ; reverting disk to virgin +## 2026-09-16T09:02:48Z bake done diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/phase0-n100-agent.txt b/documentation/audits/evidence-drill-0243-2026-09-16/phase0-n100-agent.txt new file mode 100644 index 00000000..a90f8f27 --- /dev/null +++ b/documentation/audits/evidence-drill-0243-2026-09-16/phase0-n100-agent.txt @@ -0,0 +1,29 @@ +## 2026-09-16T08:51:33Z N100 before: felhom-agent 0.130.0 +signed: op=agent_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=aaf3180cccab52e02ce8a380fd5e6b8a expires=2026-09-16T09:36:34Z +wrote envelope to /tmp/claude-1000/n100-agent-update.json +uploaded signed op to the hub jobs queue +## 2026-09-16T09:04:46Z N100 after: felhom-agent 0.131.0 +Sep 16 11:04:40 demo-felhom felhom-agent[2564828]: time=2026-09-16T11:04:40.489+02:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=23e6ad40067f3bfa op=agent_update key_id=felhom-op-1 nonce=aaf3180cccab52e02ce8a380fd5e6b8a +Sep 16 11:04:40 demo-felhom felhom-agent[2564828]: time=2026-09-16T11:04:40.489+02:00 level=WARN msg="agent_update: downloading operator-signed binary" version=0.131.0 url=https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/0.131.0/felhom-agent sha256=1118b552f7e775fbde9544c7764ede7e6046e0a7db16ae8494d07a18e3c2ac9c +Sep 16 11:04:40 demo-felhom felhom-agent[2564828]: time=2026-09-16T11:04:40.770+02:00 level=WARN msg="agent_update: handing staged binary to the guarded wrapper" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.131.0 version=0.131.0 +Sep 16 11:04:40 demo-felhom felhom-agent[2564828]: time=2026-09-16T11:04:40.885+02:00 level=WARN msg="agent_update: apply handed off; restart scheduled" version=0.131.0 wrapper="" +Sep 16 11:04:40 demo-felhom felhom-agent[2564828]: time=2026-09-16T11:04:40.885+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=23e6ad40067f3bfa op=agent_update +Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.778+02:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s guests_dir=/var/lib/felhom-agent/guests +## 2026-09-16T09:04:49Z N100 agent: felhom-agent 0.131.0 +Sep 16 11:04:43 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:43.552+02:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s +Sep 16 11:04:43 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:43.552+02:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s +Sep 16 11:04:43 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:43.552+02:00 level=INFO msg="poke: listening for hub sync-pokes (WG-confined, contentless)" addr=10.77.0.2:51822 +Sep 16 11:04:43 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:43.552+02:00 level=INFO msg="fast-tick armed: 30s out-of-band cadence while desired-state is unapplied" interval=30s +Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.741+02:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60 +Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.742+02:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s +Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.773+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.775+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1 +Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.778+02:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s guests_dir=/var/lib/felhom-agent/guests +Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.778+02:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3 +Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.778+02:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.131.0 prev=0.130.0 dwell=1m0s +Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.780+02:00 level=INFO msg="local-api server listening" addr=169.254.253.1:8443 +Sep 16 11:04:45 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:45.396+02:00 level=INFO msg="felhomsshd: config applied" port=8822 action=reload +Sep 16 11:04:46 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:46.087+02:00 level=INFO msg="desired: updated from hub" generation=9 guests=0 +Sep 16 11:04:46 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:46.100+02:00 level=INFO msg="felhomsshd: operator authorized_keys updated" user=felhom-op present=true +## 2026-09-16T09:05:29Z hub floor lines for demo-felhom: +gitea.dooplex.hu/admin/felhom-controller:0.242.0 diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/phase0-vouch.txt b/documentation/audits/evidence-drill-0243-2026-09-16/phase0-vouch.txt new file mode 100644 index 00000000..2eee389b --- /dev/null +++ b/documentation/audits/evidence-drill-0243-2026-09-16/phase0-vouch.txt @@ -0,0 +1,13 @@ +## 2026-09-16T09:04:20Z vouch BEFORE: +Agent +Golden +Min agent +Agent:. The hub HOLDS the floor for any box whose agent is below this — blank = uncoupled release, no gating. +HTTP/1.1 303 See Other +Location: /configuration?flash=artifacts_set +## AFTER: +value="0.131.0" data-sha="1118b552f7e775fbde9544c7764ede7e6046e0a7db16ae8494d07a18e3c2ac9c" selected +value="0.243.0" data-sha="e2d1843c8b648910ddde7cef4fe9f9ba2ee25002cdb58fbc42543a3e8967c10a" selected +golden (v0.243.0 +min_agent" value="0.131.0 +2026/09/16 11:04:39 [INFO] Artifact manifest set: agent=0.131.0 golden=0.243.0 min_agent="0.131.0" wrapper_sha=true diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/phase1-download.txt b/documentation/audits/evidence-drill-0243-2026-09-16/phase1-download.txt new file mode 100644 index 00000000..0822d805 --- /dev/null +++ b/documentation/audits/evidence-drill-0243-2026-09-16/phase1-download.txt @@ -0,0 +1,7 @@ +## 2026-09-16T08:55:41Z Phase 1 step 1 — the volunteer's download, from the public page +page says: +1 705 322 496 +25637007d5a7120ff9faa6b5b7ead3e33c0a361ac2d67e9fd4e0ee77c034c053 +felhom-installer-1.27.1-pve9.2-1.iso +-rw-r--r-- 1 root root 1705322496 Sep 16 10:56 felhom-installer-1.27.1-pve9.2-1.iso +25637007d5a7120ff9faa6b5b7ead3e33c0a361ac2d67e9fd4e0ee77c034c053 felhom-installer-1.27.1-pve9.2-1.iso diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/phase1-hub-baseline.txt b/documentation/audits/evidence-drill-0243-2026-09-16/phase1-hub-baseline.txt new file mode 100644 index 00000000..eda2fb4e --- /dev/null +++ b/documentation/audits/evidence-drill-0243-2026-09-16/phase1-hub-baseline.txt @@ -0,0 +1,11 @@ +## 2026-09-16T09:12:26Z hub baseline for tester-1 (before the drill box registers) +page size: 125233 +No host +dr_tier" checked +enkicsifelhom.hu +no host +pbsdr_storage_id" name="pbsdr_storage_id" +pbsdr_storage_id">PVE storage id +selfbind-link" style="margin-top: 0.5rem;"> +selfbind_tokens) dies.push('self-bind token(s)') +tester1@felhom.eu diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/phase1-install.txt b/documentation/audits/evidence-drill-0243-2026-09-16/phase1-install.txt new file mode 100644 index 00000000..947274ab --- /dev/null +++ b/documentation/audits/evidence-drill-0243-2026-09-16/phase1-install.txt @@ -0,0 +1,7 @@ +## 2026-09-16T09:02:22Z DEVIATION: the graphical entry needs blind Tab-walking over unlabeled buttons; switching to the installer's TEXT-MODE entry (the second boot-menu entry a volunteer is offered). The install path is shared (Install.pm). +update VM 334: -boot order=scsi0;ide2 +boot: order=scsi0;ide2 +ide2: local:iso/felhom-installer-1.27.1-pve9.2-1.iso,media=cdrom,size=1665354K +## 2026-09-16T09:12:16Z INSTALL started (text mode, summary as captured in screens/24) +## 2026-09-16T09:12:50Z box answers on 192.168.0.128 (installed and booted) +## CORRECTION 2026-09-16T09:13:39Z: the '09:12:50Z box answers on 192.168.0.128' line is WRONG — the INSTALLER's live system already holds that address, so the ping proved nothing about the install. Install progress is read from the console instead. diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/phase1-vm-create.txt b/documentation/audits/evidence-drill-0243-2026-09-16/phase1-vm-create.txt new file mode 100644 index 00000000..6f04a26c --- /dev/null +++ b/documentation/audits/evidence-drill-0243-2026-09-16/phase1-vm-create.txt @@ -0,0 +1,23 @@ ++ qm create 334 --name tester1-drill-0243 --memory 8192 --cores 4 --sockets 1 --cpu host --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-single --ostype l26 --agent 1 ++ qm set 334 --scsi0 nvme-scratch:32 --scsi1 nvme-scratch:100 --ide2 local:iso/felhom-installer-1.27.1-pve9.2-1.iso,media=cdrom --vga std +update VM 334: -ide2 local:iso/felhom-installer-1.27.1-pve9.2-1.iso,media=cdrom -scsi0 nvme-scratch:32 -scsi1 nvme-scratch:100 -vga std +Formatting '/mnt/hdd_1/images/334/vm-334-disk-0.raw', fmt=raw size=34359738368 preallocation=off +scsi0: successfully created disk 'nvme-scratch:334/vm-334-disk-0.raw,size=32G' +Formatting '/mnt/hdd_1/images/334/vm-334-disk-1.raw', fmt=raw size=107374182400 preallocation=off +scsi1: successfully created disk 'nvme-scratch:334/vm-334-disk-1.raw,size=100G' ++ qm set 334 --boot 'order=ide2;scsi0' +update VM 334: -boot order=ide2;scsi0 ++ qm config 334 ++ grep -E 'scsi|ide2|boot|memory|cores|name' +boot: order=ide2;scsi0 +cores: 4 +ide2: local:iso/felhom-installer-1.27.1-pve9.2-1.iso,media=cdrom,size=1665354K +memory: 8192 +name: tester1-drill-0243 +scsi0: nvme-scratch:334/vm-334-disk-0.raw,size=32G +scsi1: nvme-scratch:334/vm-334-disk-1.raw,size=100G +scsihw: virtio-scsi-single ++ qm start 334 ++ sleep 20 ++ qm status 334 +status: running diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/phase3-ep0-before.txt b/documentation/audits/evidence-drill-0243-2026-09-16/phase3-ep0-before.txt new file mode 100644 index 00000000..37c507bd --- /dev/null +++ b/documentation/audits/evidence-drill-0243-2026-09-16/phase3-ep0-before.txt @@ -0,0 +1,8 @@ +## 2026-09-16T09:12:21Z ep0 BASELINE before the drill box exists — scope: ns tester-1 only +[] +--- namespace dir: +drwxr-xr-x 4096 /mnt/pbs-datastore/ns/tester-1 +drwxr-xr-x 4096 /mnt/pbs-datastore/ns/tester-1/ct +--- token: +Path: /datastore/felhom-offsite/tester-1 +- Datastore.Backup (*) diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/01-boot-menu.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/01-boot-menu.png new file mode 100644 index 00000000..8f675188 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/01-boot-menu.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/02-installer.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/02-installer.png new file mode 100644 index 00000000..65c0e312 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/02-installer.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/03-installer.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/03-installer.png new file mode 100644 index 00000000..7ce84231 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/03-installer.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/04-after-eula.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/04-after-eula.png new file mode 100644 index 00000000..5effee1a Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/04-after-eula.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/05-after-enter.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/05-after-enter.png new file mode 100644 index 00000000..c6c347b1 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/05-after-enter.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/06-tab-enter.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/06-tab-enter.png new file mode 100644 index 00000000..c5ef634c Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/06-tab-enter.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/07-location.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/07-location.png new file mode 100644 index 00000000..f59f1d58 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/07-location.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/08-boot-menu-text.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/08-boot-menu-text.png new file mode 100644 index 00000000..d8e284fe Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/08-boot-menu-text.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/09-tui-first.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/09-tui-first.png new file mode 100644 index 00000000..79f62c94 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/09-tui-first.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/10-tui-disk.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/10-tui-disk.png new file mode 100644 index 00000000..82d0536a Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/10-tui-disk.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/11-tui-location.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/11-tui-location.png new file mode 100644 index 00000000..54967b04 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/11-tui-location.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/12-tui-location-focus.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/12-tui-location-focus.png new file mode 100644 index 00000000..4bb08f9c Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/12-tui-location-focus.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/13-tui-kbd.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/13-tui-kbd.png new file mode 100644 index 00000000..071140a1 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/13-tui-kbd.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/14-tui-kbd-enter.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/14-tui-kbd-enter.png new file mode 100644 index 00000000..090a0239 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/14-tui-kbd-enter.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/15-tui-kbd-us.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/15-tui-kbd-us.png new file mode 100644 index 00000000..b02a7a45 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/15-tui-kbd-us.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/16-tui-password.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/16-tui-password.png new file mode 100644 index 00000000..634251fc Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/16-tui-password.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/17-tui-focus-next.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/17-tui-focus-next.png new file mode 100644 index 00000000..82f86c9a Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/17-tui-focus-next.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/18-tui-focus-after-down.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/18-tui-focus-after-down.png new file mode 100644 index 00000000..ab9adeda Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/18-tui-focus-after-down.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/19-tui-password.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/19-tui-password.png new file mode 100644 index 00000000..40471404 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/19-tui-password.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/20-tui-password-filled.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/20-tui-password-filled.png new file mode 100644 index 00000000..dcd08640 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/20-tui-password-filled.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/21-tui-network.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/21-tui-network.png new file mode 100644 index 00000000..ddf39ac8 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/21-tui-network.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/22-tui-network-filled.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/22-tui-network-filled.png new file mode 100644 index 00000000..c6937be5 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/22-tui-network-filled.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/23-tui-network-next.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/23-tui-network-next.png new file mode 100644 index 00000000..cc3c3e58 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/23-tui-network-next.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/24-tui-summary.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/24-tui-summary.png new file mode 100644 index 00000000..da01ad27 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/24-tui-summary.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/25-tui-install-focus.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/25-tui-install-focus.png new file mode 100644 index 00000000..e17485fc Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/25-tui-install-focus.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/26-install-progress.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/26-install-progress.png new file mode 100644 index 00000000..722be312 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/26-install-progress.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/27-first-boot.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/27-first-boot.png new file mode 100644 index 00000000..5d61f2a1 Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/27-first-boot.png differ diff --git a/documentation/audits/evidence-drill-0243-2026-09-16/screens/28-install-now.png b/documentation/audits/evidence-drill-0243-2026-09-16/screens/28-install-now.png new file mode 100644 index 00000000..a79ea8ab Binary files /dev/null and b/documentation/audits/evidence-drill-0243-2026-09-16/screens/28-install-now.png differ diff --git a/documentation/backlog/CLOSED-ITEMS.md b/documentation/backlog/CLOSED-ITEMS.md index 65d9af5d..b409ea5c 100644 --- a/documentation/backlog/CLOSED-ITEMS.md +++ b/documentation/backlog/CLOSED-ITEMS.md @@ -26,6 +26,15 @@ --- +## 2026-09-16 — the drill's Phase 0 (hub v0.115.0, golden 0.243.0, the signing ruling) + +Two rows closed. **Full original text: `git show ^ -- documentation/backlog/OPEN-ITEMS.md`.** + +| ID | Title | Shipped | Evidence | +|---|---|---|---| +| **R-529** | [P3-LOW] The agent-plane `host_stale` / `host_down` / `host_recovered` mails still wait out the one-hour quiet rule. | hub v0.115.0 (2026-09-16, operator ruling 2) | `nodeLivenessEvents` + red-proof `TestOperatorCooldown_NodeLivenessBypassesQuietHour`; `08-alarm-ladder.md` §6.2 | +| **R-533** | [P3-LOW] The operator signing keys were placed on DooPlex world-readable (mode 664) and sit outside any documented location. | operator ruling 1, 2026-09-16 | keys at `/mnt/5_hdd/felhom.eu/felhom-op-{operational,rec-recovery}` + `felhom_op_ed25519`, mode 0600 owner `kisfenyo`; `CONTEXT.md`, `04-control-plane-authorization.md` §3.1 | + ## 2026-09-15 — the big night's P1 fixes (agent v0.131.0, controller v0.243.0, hub v0.114.0, catalog templates, ISO 1.27.1 published) Nine rows closed. **Full original text: `git show ^ -- documentation/backlog/OPEN-ITEMS.md`.** Evidence folder: `documentation/audits/evidence-p1fixes-2026-09-15/`. diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 677f0fb0..e3a1cf60 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -723,11 +723,9 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server` | **R-526** | **[P3-LOW] A host delete cannot release only the customer's ep0 PBS token: the endpoint's one removal op destroys every backup group too.** MEASURED 2026-09-15 from source: `tenantsync.Deprovision` „DESTROYS the customer's PBS namespace, all its backup groups, and its token". The task asked for „PBS token elengedése" on host delete; building it needs a new token-only op in the ep0 tenantsync script — a new operation on a protected box. Not built. R-511's adopt path makes the kept token usable instead. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (new ep0 op yes/no), CC (build)** | | **R-527** | **[P3-LOW] The catalog flag `locked_after_deploy` is read by no controller code — every setting is read-only after install whatever the catalog says.** FOUND 2026-09-15: `stacks/metadata.go` parses it; `grep -rn LockedAfterDeploy` finds no reader; `deploy.html` renders „Az alábbi beállítások csak olvashatók" for every field. Recorded as the design in `02-controller-module-map.md`; the flag is a seam never wired. **Fix shape:** remove the flag from the catalog, or wire an editable-after-install allow-list (a bigger change). | **READY — rank P3-LOW; owner: CC** | | **R-528** | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend. | **READY — rank P2-MEDIUM; owner: CC** | -| **R-529** | **[P3-LOW] The agent-plane `host_stale` / `host_down` / `host_recovered` mails still wait out the one-hour quiet rule.** The 2026-09-15 ruling (decision A) named `node_*` only, and the task fenced „a design is not a defect". The same F9 silence can happen on the host plane. **What it needs:** the operator's word whether the ruling extends to `host_*`. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator** | -| **R-530** | **[P2-MEDIUM] A floor does not deliver an agent: agents update only by an operator-signed `agent_update` job per box, and nothing records which boxes still run 0.130.0.** MEASURED 2026-09-15: the hub HOLDS a floor whose declared MinAgent is above the box's agent (`api/handler.go` ResolveManagedFloor); the agent's only update path is `signedjobs` + `selfupdate.Executor`. demo-hp reached 0.131.0 by `felhom-opsign -op agent_update` (key `felhom-op-1`) at 08:44:16Z and its controller floor was then SERVED in 3 s. **demo-felhom (N100) and Peti's box still run 0.130.0** — not touched (Peti fenced; N100 not asked). **What it needs:** the operator signs per box, or rules a fleet rollout step. | **WAITING-ON-OPERATOR — rank P2-MEDIUM; owner: operator (signing)** | +| **R-530** | **[P2-MEDIUM] A floor does not deliver an agent: agents update only by an operator-signed `agent_update` job per box, and nothing records which boxes still run 0.130.0.** MEASURED 2026-09-15: the hub HOLDS a floor whose declared MinAgent is above the box's agent (`api/handler.go` ResolveManagedFloor); the agent's only update path is `signedjobs` + `selfupdate.Executor`. demo-hp reached 0.131.0 by `felhom-opsign -op agent_update` (key `felhom-op-1`) at 08:44:16Z and its controller floor was then SERVED in 3 s. **demo-felhom (N100) and Peti's box still run 0.130.0** — not touched (Peti fenced; N100 not asked). **What it needs:** the operator signs per box, or rules a fleet rollout step. **NARROWED 2026-09-16 (operator ruling 1):** the keys stay on DooPlex owner-only and CC may sign `agent_update` until the first PAYING customer (testers excluded) — recorded in `CONTEXT.md` + `04-control-plane-authorization.md` §3.1. Both demo boxes now run agent 0.131.0 (demo-hp 2026-09-15, demo-felhom 2026-09-16, each by a per-box signed job; Peti's box untouched, still 0.130.0). **What remains:** a fleet rollout step — signing per box does not scale past a handful, and nothing lists which boxes are behind. | **WAITING-ON-OPERATOR — rank P2-MEDIUM; owner: operator (signing)** | | **R-531** | **[P3-LOW] Three supervisor facts measured live and not pinned: restart timing during a deploy was not measured; restarts before the hub first sees the stanza produce no `controller_restarted_by_agent`; deliberate operator kills spend the crash-loop budget.** MEASURED 2026-09-15 on 9201: after 3 test restarts in 13 minutes the 4th kill tripped the 30-minute pause and the dashboard stayed down (the guard as designed, `A4-kill-middeploy-9201.txt`). The hub checker seeds silently on first sight, so the three restarts before the first v0.131.0 report emitted nothing (only the crash-loop did). **What it needs:** a deploy-kill timing on a fresh budget; the operator's view whether a restart after minutes of uptime should count toward the budget. | **READY — rank P3-LOW; owner: CC (measure) · operator (budget rule)** | | **R-532** | **[P3-LOW] Vaultwarden's `/api/config` still says `disableUserRegistration:false` with signups off, so the web vault shows a register form that the server then refuses.** MEASURED 2026-09-15 in the E.1 spike. Cosmetic: the server refuses (400). A household following the invite-first card is not affected; a stranger sees a form that fails. | **READY — rank P3-LOW; owner: CC (catalog/upstream note)** | -| **R-533** | **[P3-LOW] The operator signing keys were placed on DooPlex world-readable (mode 664) and sit outside any documented location.** OBSERVED 2026-09-15: `/mnt/5_hdd/felhom.eu/felhom-op-operational`, `felhom_op_ed25519`, `felhom-rec-recovery` arrived 664; CC set them to 600 (no other change). Their fingerprints match the signers demo-hp pins. **What it needs:** the operator decides where the keys live between sessions (hardware key, or a documented 0600 path) and records it in `operations/nodes.md`. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator** |