drill phase 0: golden 0.243.0 baked+vouched, N100 signed to agent 0.131.0, rulings 1 and 2 recorded; R-529/R-533 closed, R-530 narrowed

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-09-16 11:14:10 +02:00
parent 2dd80a5d28
commit 7eedaac33e
40 changed files with 580 additions and 3 deletions
@@ -0,0 +1,45 @@
+# DRILL — prove the P1 fixes on a fresh box (2026-09-16)
+
+**Interventions: _pending_** (O1/O2 pre-declared, counted apart).
+**Ready for a volunteer: _pending_.**
+**The automatic connect e-mail: _pending_.**
+
+> Baselines at start (re-verified against live Gitea): controller `383a30b3c07b` v0.243.0 (`Unreleased`: the
+> „0 B" tile fix), agent `e98b857684f4` v0.131.0, felhom.eu `351296114c4d` hub v0.114.0, catalog `94bc5febaca2`.
+> Golden before this run: 0.242.0 (2026-09-14). Customer `tester-1`, domain `enkicsifelhom.hu`, e-mail
+> `tester1@felhom.eu`, **no host**, DR tier ticked, ep0 token held with no descriptor, namespace EMPTY
+> (`phase3-ep0-before.txt`).
+
+## Phase 0 — rulings, the golden, the N100
+
+**Ruling 1 (2026-09-16, operator).** The signing keys stay on DooPlex, owner-only
+(`/mnt/5_hdd/felhom.eu/felhom-op-operational`, `felhom-rec-recovery`, `felhom_op_ed25519`, mode 0600);
+CC may sign `agent_update` jobs with them until the first PAYING customer — testers excluded. Recorded in
+`CONTEXT.md` and `04-control-plane-authorization.md` §3.1. **R-533 closes** (path + mode);
+**R-530 narrows** to "a fleet rollout step is still undesigned".
+
+**N100 signed and updated (R-530).** `felhom-opsign -op agent_update -host demo-felhom-8363b5 -key-id
+felhom-op-1 … -agent-version 0.131.0 -sha256 1118b552…c9c`, queued 08:51:34Z, **the box took it at
+09:04:43Z** (its own poll, ~13 min), dwelled 60 s and committed; `controller-supervisor: started`
+(interval 30s, confirm 2, crash-loop 3/15m) in its journal. Peti's box untouched. The N100 keeps
+controller 0.242.0; its floor was not moved by this run.
+
+**Ruling 2 — hub v0.115.0 (R-529).** `host_stale` / `host_down` / `host_recovered` join the `node_*`
+cooldown bypass with the same 5-minute dedupe. Red-proof: with the three host types removed the test fails
+at "host_stale 39 min after the previous one was suppressed (sent=1)". Recorded in `08-alarm-ladder.md` §6.2
+beside the 2026-09-15 node ruling. Deployed by GitOps: ArgoCD Synced, `deploy/hub` image 0.115.0.
+
+**Golden 0.243.0 baked and vouched.** First attempt FAILED and is recorded: the template picker took the
+LAST `debian-13` line, which is **arm64**, and the container would not start ("Detected container
+architecture: arm64"); no golden was produced, the scratch guest was destroyed and the disk reverted.
+Re-run with the amd64 template: `GOLDEN_VERSION=0.243.0`,
+`GOLDEN_SHA256=e2d1843c8b648910ddde7cef4fe9f9ba2ee25002cdb58fbc42543a3e8967c10a`. Markers from the saved log
+(`phase0-bake-full.log`, 326 lines): `docker OK (overlay2` ×1, `including mount point` ×2 (rootfs + mp0),
+`upload OK (HTTP 201)` ×1, `FATAL` 0, `excluding` 0. Token leak checks: control copy 1, committed log 0.
+Vouched as a three-field change — golden 0.243.0 + agent 0.131.0 + min agent 0.131.0 — hub logged
+`Artifact manifest set: agent=0.131.0 golden=0.243.0 min_agent="0.131.0" wrapper_sha=true`.
+
+## Phase 1 — the walk
+
+_(in progress)_
+
@@ -0,0 +1,326 @@
+[golden] build-golden.sh v3.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.243.0
+[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) …
+  Logical volume "vm-9100-disk-0" created.
+  Logical volume pve/vm-9100-disk-0 changed.
+Creating filesystem with 8388608 4k blocks and 2097152 inodes
+Filesystem UUID: 0fdf18ca-d88c-42c5-8b0f-ed31623da33a
+Superblock backups stored on blocks: 
+	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
+	4096000, 7962624
+  Logical volume "vm-9100-disk-1" created.
+  Logical volume pve/vm-9100-disk-1 changed.
+Creating filesystem with 6291456 4k blocks and 1572864 inodes
+Filesystem UUID: d425f3e9-0c5d-40cd-89ce-80f2988e1294
+Superblock backups stored on blocks: 
+	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
+extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst'
+Total bytes read: 553512960 (528MiB, 117MiB/s)
+Detected container architecture: amd64
+Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
+done: SHA256:JfhGxgHqSFFzsiIiMBN4G/hL338Pff/Wyft5RQzy6MI root@felhom-golden
+Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
+done: SHA256:nZa+KA0vkryve5RLYv6h9XyQjQGiI8IqZ5Uank6oL4A root@felhom-golden
+Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
+done: SHA256:7gT/lVkeweXCbxPAQQam7E4ZvsSjwloKwb6M3GaB2wE root@felhom-golden
+[golden] starting + installing Docker (official repo, trixie channel) …
+apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
+perl: warning: Setting locale failed.
+perl: warning: Please check that your locale settings:
+	LANGUAGE = (unset),
+	LC_ALL = (unset),
+	LC_CTYPE = (unset),
+	LC_NUMERIC = (unset),
+	LC_COLLATE = (unset),
+	LC_TIME = (unset),
+	LC_MESSAGES = (unset),
+	LC_MONETARY = (unset),
+	LC_ADDRESS = (unset),
+	LC_IDENTIFICATION = (unset),
+	LC_MEASUREMENT = (unset),
+	LC_PAPER = (unset),
+	LC_TELEPHONE = (unset),
+	LC_NAME = (unset),
+	LANG = "en_US.UTF-8"
+    are supported and installed on your system.
+perl: warning: Falling back to the standard locale ("C").
+locale: Cannot set LC_CTYPE to default locale: No such file or directory
+locale: Cannot set LC_MESSAGES to default locale: No such file or directory
+locale: Cannot set LC_ALL to default locale: No such file or directory
+apt-listchanges: Can't set locale; make sure $LC_* and $LANG are correct!
+perl: warning: Setting locale failed.
+perl: warning: Please check that your locale settings:
+	LANGUAGE = (unset),
+	LC_ALL = (unset),
+	LC_CTYPE = (unset),
+	LC_NUMERIC = (unset),
+	LC_COLLATE = (unset),
+	LC_TIME = (unset),
+	LC_MESSAGES = (unset),
+	LC_MONETARY = (unset),
+	LC_ADDRESS = (unset),
+	LC_IDENTIFICATION = (unset),
+	LC_MEASUREMENT = (unset),
+	LC_PAPER = (unset),
+	LC_TELEPHONE = (unset),
+	LC_NAME = (unset),
+	LANG = "en_US.UTF-8"
+    are supported and installed on your system.
+perl: warning: Falling back to the standard locale ("C").
+locale: Cannot set LC_CTYPE to default locale: No such file or directory
+locale: Cannot set LC_MESSAGES to default locale: No such file or directory
+locale: Cannot set LC_ALL to default locale: No such file or directory
+[golden] baking daemon.json: classic overlay2 driver (containerd-snapshotter OFF) + log rotation …
+[golden] wiring the single data volume (R-165 variant V-c): /var/lib/felhom/{docker,sys_drive} -> binds …
+[golden] verifying Docker works in the build guest (storage driver should be overlay2 on the ext4 data volume) …
+Unable to find image 'hello-world:latest' locally
+latest: Pulling from library/hello-world
+4f55086f7dd0: Pulling fs layer
+4f55086f7dd0: Verifying Checksum
+4f55086f7dd0: Download complete
+4f55086f7dd0: Pull complete
+Digest: sha256:5e23090353324d887c48ad5e5c56d294eab81588df9605b07d1afe895f9cc8f8
+Status: Downloaded newer image for hello-world:latest
+  docker OK (overlay2; data-root /var/lib/docker)
+  /var/lib/docker is a real mount: /dev/mapper/pve-vm--9100--disk--1[/docker] ext4
+  /mnt/sys_drive is a real mount: /dev/mapper/pve-vm--9100--disk--1[/sys_drive] ext4
+  both paths are ONE filesystem: /dev/mapper/pve-vm--9100--disk--1 23317576
+[golden] baking the in-guest controller image gitea.dooplex.hu/admin/felhom-controller:0.243.0 (no registry cred at deploy) …
+
+WARNING! Your credentials are stored unencrypted in '/root/.docker/config.json'.
+Configure a credential helper to remove this warning. See
+https://docs.docker.com/go/credential-store/
+
+0.243.0: Pulling from admin/felhom-controller
+a8ac7f6c67ab: Pulling fs layer
+bf30769d36e7: Pulling fs layer
+044b66fbe46c: Pulling fs layer
+b5c41a28e83f: Pulling fs layer
+24733ed700a4: Pulling fs layer
+ca058f1603e5: Pulling fs layer
+b5c41a28e83f: Waiting
+24733ed700a4: Waiting
+ca058f1603e5: Waiting
+a8ac7f6c67ab: Verifying Checksum
+a8ac7f6c67ab: Download complete
+044b66fbe46c: Verifying Checksum
+044b66fbe46c: Download complete
+b5c41a28e83f: Verifying Checksum
+b5c41a28e83f: Download complete
+24733ed700a4: Verifying Checksum
+24733ed700a4: Download complete
+ca058f1603e5: Verifying Checksum
+ca058f1603e5: Download complete
+bf30769d36e7: Verifying Checksum
+bf30769d36e7: Download complete
+a8ac7f6c67ab: Pull complete
+bf30769d36e7: Pull complete
+044b66fbe46c: Pull complete
+b5c41a28e83f: Pull complete
+24733ed700a4: Pull complete
+ca058f1603e5: Pull complete
+Digest: sha256:b5314c302ff959bbc1dff0e59fefd018d602ced8f986208ea9ac930d6e3f0905
+Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-controller:0.243.0
+gitea.dooplex.hu/admin/felhom-controller:0.243.0
+[golden] asking the controller which infra images it manages …
+[golden] baking infra images (4): traefik:v3.6.7 cloudflare/cloudflared:2026.6.0 gtstef/filebrowser:1.3.3-stable gitea.dooplex.hu/admin/felhom-samba:1.1.0 …
+v3.6.7: Pulling from library/traefik
+589002ba0eae: Pulling fs layer
+ef63511ea6cc: Pulling fs layer
+0738e5cb835e: Pulling fs layer
+3e6813f70c64: Pulling fs layer
+3e6813f70c64: Waiting
+ef63511ea6cc: Verifying Checksum
+ef63511ea6cc: Download complete
+3e6813f70c64: Verifying Checksum
+3e6813f70c64: Download complete
+589002ba0eae: Verifying Checksum
+589002ba0eae: Download complete
+0738e5cb835e: Verifying Checksum
+0738e5cb835e: Download complete
+589002ba0eae: Pull complete
+ef63511ea6cc: Pull complete
+0738e5cb835e: Pull complete
+3e6813f70c64: Pull complete
+Digest: sha256:a9890c898f379c1905ee5b28342f6b408dc863f08db2dab20e46c267d1ff463a
+Status: Downloaded newer image for traefik:v3.6.7
+docker.io/library/traefik:v3.6.7
+2026.6.0: Pulling from cloudflare/cloudflared
+47de5dd0b812: Pulling fs layer
+c172f21841df: Pulling fs layer
+99515e7b4d35: Pulling fs layer
+99ba982a9142: Pulling fs layer
+d6b1b89eccac: Pulling fs layer
+2780920e5dbf: Pulling fs layer
+7c12895b777b: Pulling fs layer
+3214acf345c0: Pulling fs layer
+52630fc75a18: Pulling fs layer
+dd64bf2dd177: Pulling fs layer
+b839dfae01f6: Pulling fs layer
+ebddc55facdc: Pulling fs layer
+bdfd7f7e5bf6: Pulling fs layer
+2d4d7adf6272: Pulling fs layer
+40008157d8d2: Pulling fs layer
+bd8962e29291: Pulling fs layer
+cac2ae0193cb: Pulling fs layer
+74d1dac84ecc: Pulling fs layer
+99ba982a9142: Waiting
+d6b1b89eccac: Waiting
+2780920e5dbf: Waiting
+7c12895b777b: Waiting
+3214acf345c0: Waiting
+52630fc75a18: Waiting
+dd64bf2dd177: Waiting
+b839dfae01f6: Waiting
+ebddc55facdc: Waiting
+bdfd7f7e5bf6: Waiting
+2d4d7adf6272: Waiting
+40008157d8d2: Waiting
+bd8962e29291: Waiting
+cac2ae0193cb: Waiting
+74d1dac84ecc: Waiting
+47de5dd0b812: Download complete
+c172f21841df: Verifying Checksum
+c172f21841df: Download complete
+47de5dd0b812: Pull complete
+99515e7b4d35: Verifying Checksum
+99515e7b4d35: Download complete
+99ba982a9142: Verifying Checksum
+99ba982a9142: Download complete
+d6b1b89eccac: Verifying Checksum
+d6b1b89eccac: Download complete
+2780920e5dbf: Verifying Checksum
+2780920e5dbf: Download complete
+7c12895b777b: Verifying Checksum
+7c12895b777b: Download complete
+3214acf345c0: Verifying Checksum
+3214acf345c0: Download complete
+c172f21841df: Pull complete
+52630fc75a18: Verifying Checksum
+52630fc75a18: Download complete
+dd64bf2dd177: Download complete
+b839dfae01f6: Verifying Checksum
+b839dfae01f6: Download complete
+ebddc55facdc: Verifying Checksum
+ebddc55facdc: Download complete
+bdfd7f7e5bf6: Verifying Checksum
+bdfd7f7e5bf6: Download complete
+40008157d8d2: Verifying Checksum
+40008157d8d2: Download complete
+bd8962e29291: Verifying Checksum
+bd8962e29291: Download complete
+2d4d7adf6272: Verifying Checksum
+2d4d7adf6272: Download complete
+99515e7b4d35: Pull complete
+cac2ae0193cb: Verifying Checksum
+cac2ae0193cb: Download complete
+74d1dac84ecc: Verifying Checksum
+74d1dac84ecc: Download complete
+99ba982a9142: Pull complete
+d6b1b89eccac: Pull complete
+2780920e5dbf: Pull complete
+7c12895b777b: Pull complete
+3214acf345c0: Pull complete
+52630fc75a18: Pull complete
+dd64bf2dd177: Pull complete
+b839dfae01f6: Pull complete
+ebddc55facdc: Pull complete
+bdfd7f7e5bf6: Pull complete
+2d4d7adf6272: Pull complete
+40008157d8d2: Pull complete
+bd8962e29291: Pull complete
+cac2ae0193cb: Pull complete
+74d1dac84ecc: Pull complete
+Digest: sha256:ba461b8aa9c042156dbd39c38657fe7431bafa063220eab8d5330a523863da9f
+Status: Downloaded newer image for cloudflare/cloudflared:2026.6.0
+docker.io/cloudflare/cloudflared:2026.6.0
+1.3.3-stable: Pulling from gtstef/filebrowser
+6a0ac1617861: Pulling fs layer
+ef8806083e82: Pulling fs layer
+b74107c861c7: Pulling fs layer
+adc935def003: Pulling fs layer
+4f4fb700ef54: Pulling fs layer
+18695ccc900a: Pulling fs layer
+45d119d5c397: Pulling fs layer
+dac52db4fc51: Pulling fs layer
+6d598f86b2f2: Pulling fs layer
+8aa349c8396c: Pulling fs layer
+4f4fb700ef54: Waiting
+adc935def003: Waiting
+18695ccc900a: Waiting
+45d119d5c397: Waiting
+dac52db4fc51: Waiting
+6d598f86b2f2: Waiting
+8aa349c8396c: Waiting
+6a0ac1617861: Verifying Checksum
+6a0ac1617861: Download complete
+adc935def003: Verifying Checksum
+adc935def003: Download complete
+4f4fb700ef54: Verifying Checksum
+4f4fb700ef54: Download complete
+ef8806083e82: Verifying Checksum
+ef8806083e82: Download complete
+b74107c861c7: Verifying Checksum
+b74107c861c7: Download complete
+45d119d5c397: Verifying Checksum
+45d119d5c397: Download complete
+dac52db4fc51: Verifying Checksum
+dac52db4fc51: Download complete
+6d598f86b2f2: Verifying Checksum
+6d598f86b2f2: Download complete
+18695ccc900a: Download complete
+6a0ac1617861: Pull complete
+8aa349c8396c: Verifying Checksum
+8aa349c8396c: Download complete
+ef8806083e82: Pull complete
+b74107c861c7: Pull complete
+adc935def003: Pull complete
+4f4fb700ef54: Pull complete
+18695ccc900a: Pull complete
+45d119d5c397: Pull complete
+dac52db4fc51: Pull complete
+6d598f86b2f2: Pull complete
+8aa349c8396c: Pull complete
+Digest: sha256:eb3733681db8757412632c61a99ad656f0d94ed6781bb2ea114b4d70babab78c
+Status: Downloaded newer image for gtstef/filebrowser:1.3.3-stable
+docker.io/gtstef/filebrowser:1.3.3-stable
+1.1.0: Pulling from admin/felhom-samba
+897d797d2723: Pulling fs layer
+3051591aa250: Pulling fs layer
+ce57a3f93416: Pulling fs layer
+fb94eeec2fe1: Pulling fs layer
+fb94eeec2fe1: Waiting
+ce57a3f93416: Verifying Checksum
+ce57a3f93416: Download complete
+fb94eeec2fe1: Verifying Checksum
+fb94eeec2fe1: Download complete
+897d797d2723: Verifying Checksum
+897d797d2723: Download complete
+3051591aa250: Verifying Checksum
+3051591aa250: Download complete
+897d797d2723: Pull complete
+3051591aa250: Pull complete
+ce57a3f93416: Pull complete
+fb94eeec2fe1: Pull complete
+Digest: sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10
+Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.1.0
+gitea.dooplex.hu/admin/felhom-samba:1.1.0
+[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) …
+Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'.
+[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …
+Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'.
+[golden] baking the first-boot SSH host-key regeneration unit (F3) …
+Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'.
+[golden] identity-clean + minimize …
+[golden] stop + archive …
+INFO: including mount point rootfs ('/') in backup
+INFO: including mount point mp0 ('/var/lib/felhom') in backup
+INFO: archive file size: 623MB
+INFO: Finished Backup of VM 9100 (00:00:44)
+[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_09_16-11_01_01.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive)
+[golden] publishing golden (653997919 bytes, sha256 e2d1843c8b648910…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.243.0/golden.tar.zst
+[golden]   pre-delete existing: HTTP 404 (404/204 expected)
+[golden]   upload OK (HTTP 201)
+GOLDEN_VERSION=0.243.0
+GOLDEN_SHA256=e2d1843c8b648910ddde7cef4fe9f9ba2ee25002cdb58fbc42543a3e8967c10a
+[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.243.0 / e2d1843c8b648910ddde7cef4fe9f9ba2ee25002cdb58fbc42543a3e8967c10a
+[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge)
@@ -0,0 +1,101 @@
+## 2026-09-16T08:52:26Z revert to virgin
+## 2026-09-16T08:52:26Z cold boot
+## 2026-09-16T08:52:59Z ssh up: pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
+system          debian-13-standard_13.6-1_amd64.tar.zst
+system          debian-13-standard_13.6-1_arm64.tar.zst
+## 2026-09-16T08:53:06Z template: debian-13-standard_13.6-1_arm64.tar.zst
+calculating checksum...OK, checksum verified
+download of 'http://download.proxmox.com/images/system/debian-13-standard_13.6-1_arm64.tar.zst' to '/var/lib/vz/template/cache/debian-13-standard_13.6-1_arm64.tar.zst' finished
+## 2026-09-16T08:53:13Z launching bake (transient unit)
+## 2026-09-16T08:53:13Z token leak check on the unit: 0
+## 2026-09-16T08:53:55Z bake unit state: inactive
+[golden] build-golden.sh v3.0.0 — baking controller gitea.dooplex.hu/admin/felhom-controller:0.243.0
+[golden] creating build LXC 9100 (nesting=1,keyctl=1, unprivileged; rootfs 32G + ONE data volume 24G @ /var/lib/felhom, backup=1) …
+  Logical volume "vm-9100-disk-0" created.
+  Logical volume pve/vm-9100-disk-0 changed.
+Creating filesystem with 8388608 4k blocks and 2097152 inodes
+Filesystem UUID: d27b2e03-4001-489d-a62c-938da32e3be5
+Superblock backups stored on blocks: 
+	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
+	4096000, 7962624
+  Logical volume "vm-9100-disk-1" created.
+  Logical volume pve/vm-9100-disk-1 changed.
+Creating filesystem with 6291456 4k blocks and 1572864 inodes
+Filesystem UUID: 1072606a-823b-4274-9dbc-7ff74bd5790b
+Superblock backups stored on blocks: 
+	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
+extracting archive '/var/lib/vz/template/cache/debian-13-standard_13.6-1_arm64.tar.zst'
+Total bytes read: 593008640 (566MiB, 97MiB/s)
+Detected container architecture: arm64
+Creating SSH host key 'ssh_host_ed25519_key' - this may take some time ...
+done: SHA256:rZ9I+1iM0R+3Z6Zh2NfLJdD981ROGZ2M233hGq5GFT4 root@felhom-golden
+Creating SSH host key 'ssh_host_ecdsa_key' - this may take some time ...
+done: SHA256:KjILoijRNE+lzXid6lepHRX9NF005HRiMIvEQmO6R3M root@felhom-golden
+Creating SSH host key 'ssh_host_rsa_key' - this may take some time ...
+done: SHA256:X8iecif9F1WuFTElfBp7qzWlz9vf/q2vNvzuFyAmabM root@felhom-golden
+[golden] starting + installing Docker (official repo, trixie channel) …
+sync_wait: 34 An error occurred in another process (expected sequence number 7)
+__lxc_start: 2288 Failed to spawn container "9100"
+startup for container '9100' failed
+## 2026-09-16T08:53:55Z markers: overlay2=0 mountpoints=0 upload=0 fatal=0 excluding=0
+## 2026-09-16T08:53:55Z token grep control (must be 1): 1; committed log (must be 0): 0
+## 2026-09-16T08:54:16Z qemu exited: 0 ; reverting disk to virgin
+## 2026-09-16T08:54:16Z bake done
+## RE-RUN after the arm64 mistake 2026-09-16T08:55:35Z
+## 2026-09-16T08:55:35Z revert to virgin
+## 2026-09-16T08:55:35Z cold boot
+## 2026-09-16T08:56:07Z ssh up: pve-manager/9.2.2/b9984c6d90a4bd80 (running kernel: 7.0.2-6-pve)
+system          debian-13-standard_13.6-1_amd64.tar.zst
+system          debian-13-standard_13.6-1_arm64.tar.zst
+## 2026-09-16T08:56:15Z template: debian-13-standard_13.6-1_amd64.tar.zst
+calculating checksum...OK, checksum verified
+download of 'http://download.proxmox.com/images/system/debian-13-standard_13.6-1_amd64.tar.zst' to '/var/lib/vz/template/cache/debian-13-standard_13.6-1_amd64.tar.zst' finished
+## 2026-09-16T08:56:31Z launching bake (transient unit)
+## 2026-09-16T08:56:31Z token leak check on the unit: 0
+## 2026-09-16T09:02:21Z bake unit state: inactive
+897d797d2723: Pulling fs layer
+3051591aa250: Pulling fs layer
+ce57a3f93416: Pulling fs layer
+fb94eeec2fe1: Pulling fs layer
+fb94eeec2fe1: Waiting
+ce57a3f93416: Verifying Checksum
+ce57a3f93416: Download complete
+fb94eeec2fe1: Verifying Checksum
+fb94eeec2fe1: Download complete
+897d797d2723: Verifying Checksum
+897d797d2723: Download complete
+3051591aa250: Verifying Checksum
+3051591aa250: Download complete
+897d797d2723: Pull complete
+3051591aa250: Pull complete
+ce57a3f93416: Pull complete
+fb94eeec2fe1: Pull complete
+Digest: sha256:1c17c09422bec0366d7cf0e0fcfc1486ba6c90334a0a5d5c851073a9342f8f10
+Status: Downloaded newer image for gitea.dooplex.hu/admin/felhom-samba:1.1.0
+gitea.dooplex.hu/admin/felhom-samba:1.1.0
+[golden] baking the controller-bootstrap unit (deploys the BAKED controller from the config mount) …
+Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.service' → '/etc/systemd/system/felhom-controller-bootstrap.service'.
+[golden] baking the controller-bootstrap PATH unit (starts the service on bootstrap-mount hot-plug — B1) …
+Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-controller-bootstrap.path' → '/etc/systemd/system/felhom-controller-bootstrap.path'.
+[golden] baking the first-boot SSH host-key regeneration unit (F3) …
+Created symlink '/etc/systemd/system/multi-user.target.wants/felhom-regen-hostkeys.service' → '/etc/systemd/system/felhom-regen-hostkeys.service'.
+[golden] identity-clean + minimize …
+[golden] stop + archive …
+INFO: including mount point rootfs ('/') in backup
+INFO: including mount point mp0 ('/var/lib/felhom') in backup
+INFO: archive file size: 623MB
+INFO: Finished Backup of VM 9100 (00:00:44)
+[golden] DONE. golden archive volid: local:backup/vzdump-lxc-9100-2026_09_16-11_01_01.tar.zst (rootfs 32G + ONE data volume 24G @ /var/lib/felhom, all in the archive)
+[golden] publishing golden (653997919 bytes, sha256 e2d1843c8b648910…) → https://gitea.dooplex.hu/api/packages/admin/generic/felhom-golden/0.243.0/golden.tar.zst
+[golden]   pre-delete existing: HTTP 404 (404/204 expected)
+[golden]   upload OK (HTTP 201)
+GOLDEN_VERSION=0.243.0
+GOLDEN_SHA256=e2d1843c8b648910ddde7cef4fe9f9ba2ee25002cdb58fbc42543a3e8967c10a
+[golden] Record in the hub operator UI (Configs → Day-0 artifacts): golden 0.243.0 / e2d1843c8b648910ddde7cef4fe9f9ba2ee25002cdb58fbc42543a3e8967c10a
+[golden] (the build guest 9100 is stopped; destroy it with: pct destroy 9100 --purge)
+## 2026-09-16T09:02:21Z markers: overlay2=1 mountpoints=2 upload=1 fatal=0 excluding=0
+GOLDEN_VERSION=0.243.0
+GOLDEN_SHA256=e2d1843c8b648910ddde7cef4fe9f9ba2ee25002cdb58fbc42543a3e8967c10a
+## 2026-09-16T09:02:21Z token grep control (must be 1): 1; committed log (must be 0): 0
+## 2026-09-16T09:02:47Z qemu exited: 0 ; reverting disk to virgin
+## 2026-09-16T09:02:48Z bake done
@@ -0,0 +1,29 @@
+## 2026-09-16T08:51:33Z N100 before: felhom-agent 0.130.0
+signed: op=agent_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=aaf3180cccab52e02ce8a380fd5e6b8a expires=2026-09-16T09:36:34Z
+wrote envelope to /tmp/claude-1000/n100-agent-update.json
+uploaded signed op to the hub jobs queue
+## 2026-09-16T09:04:46Z N100 after: felhom-agent 0.131.0
+Sep 16 11:04:40 demo-felhom felhom-agent[2564828]: time=2026-09-16T11:04:40.489+02:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=23e6ad40067f3bfa op=agent_update key_id=felhom-op-1 nonce=aaf3180cccab52e02ce8a380fd5e6b8a
+Sep 16 11:04:40 demo-felhom felhom-agent[2564828]: time=2026-09-16T11:04:40.489+02:00 level=WARN msg="agent_update: downloading operator-signed binary" version=0.131.0 url=https://gitea.dooplex.hu/api/packages/admin/generic/felhom-agent/0.131.0/felhom-agent sha256=1118b552f7e775fbde9544c7764ede7e6046e0a7db16ae8494d07a18e3c2ac9c
+Sep 16 11:04:40 demo-felhom felhom-agent[2564828]: time=2026-09-16T11:04:40.770+02:00 level=WARN msg="agent_update: handing staged binary to the guarded wrapper" staged=/var/lib/felhom-agent/selfupdate/felhom-agent-0.131.0 version=0.131.0
+Sep 16 11:04:40 demo-felhom felhom-agent[2564828]: time=2026-09-16T11:04:40.885+02:00 level=WARN msg="agent_update: apply handed off; restart scheduled" version=0.131.0 wrapper=""
+Sep 16 11:04:40 demo-felhom felhom-agent[2564828]: time=2026-09-16T11:04:40.885+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=23e6ad40067f3bfa op=agent_update
+Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.778+02:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s guests_dir=/var/lib/felhom-agent/guests
+## 2026-09-16T09:04:49Z N100 agent: felhom-agent 0.131.0
+Sep 16 11:04:43 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:43.552+02:00 level=INFO msg="pbs: verify loop starting" cadence=6h0m0s
+Sep 16 11:04:43 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:43.552+02:00 level=INFO msg="storage: watchdog starting" interval=5s debounce=15s
+Sep 16 11:04:43 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:43.552+02:00 level=INFO msg="poke: listening for hub sync-pokes (WG-confined, contentless)" addr=10.77.0.2:51822
+Sep 16 11:04:43 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:43.552+02:00 level=INFO msg="fast-tick armed: 30s out-of-band cadence while desired-state is unapplied" interval=30s
+Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.741+02:00 level=INFO msg="selfheal: node watchdog starting" mode=appliance interval_s=60
+Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.742+02:00 level=INFO msg="guestnet: watchdog starting" interval=1m0s min_heal_interval=10m0s max_heals_per_hour=3 settle=3m0s
+Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.773+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
+Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.775+02:00 level=INFO msg="stale-lock: scanning pool guests" pool=felhom listed=1 scanned=1
+Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.778+02:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=15m0s guests_dir=/var/lib/felhom-agent/guests
+Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.778+02:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
+Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.778+02:00 level=WARN msg="selfupdate: new version running — dwelling before commit" version=0.131.0 prev=0.130.0 dwell=1m0s
+Sep 16 11:04:44 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:44.780+02:00 level=INFO msg="local-api server listening" addr=169.254.253.1:8443
+Sep 16 11:04:45 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:45.396+02:00 level=INFO msg="felhomsshd: config applied" port=8822 action=reload
+Sep 16 11:04:46 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:46.087+02:00 level=INFO msg="desired: updated from hub" generation=9 guests=0
+Sep 16 11:04:46 demo-felhom felhom-agent[2416121]: time=2026-09-16T11:04:46.100+02:00 level=INFO msg="felhomsshd: operator authorized_keys updated" user=felhom-op present=true
+## 2026-09-16T09:05:29Z hub floor lines for demo-felhom:
+gitea.dooplex.hu/admin/felhom-controller:0.242.0
@@ -0,0 +1,13 @@
+## 2026-09-16T09:04:20Z vouch BEFORE:
+Agent</label>
+Golden</label>
+Min agent</label>
+Agent:</code>. The hub HOLDS the floor for any box whose agent is below this — blank = uncoupled release, no gating.
+HTTP/1.1 303 See Other
+Location: /configuration?flash=artifacts_set
+## AFTER:
+value="0.131.0" data-sha="1118b552f7e775fbde9544c7764ede7e6046e0a7db16ae8494d07a18e3c2ac9c" selected
+value="0.243.0" data-sha="e2d1843c8b648910ddde7cef4fe9f9ba2ee25002cdb58fbc42543a3e8967c10a" selected
+golden (v0.243.0
+min_agent" value="0.131.0
+2026/09/16 11:04:39 [INFO] Artifact manifest set: agent=0.131.0 golden=0.243.0 min_agent="0.131.0" wrapper_sha=true
@@ -0,0 +1,7 @@
+## 2026-09-16T08:55:41Z Phase 1 step 1 — the volunteer's download, from the public page
+page says:
+1 705 322 496
+25637007d5a7120ff9faa6b5b7ead3e33c0a361ac2d67e9fd4e0ee77c034c053
+felhom-installer-1.27.1-pve9.2-1.iso
+-rw-r--r-- 1 root root 1705322496 Sep 16 10:56 felhom-installer-1.27.1-pve9.2-1.iso
+25637007d5a7120ff9faa6b5b7ead3e33c0a361ac2d67e9fd4e0ee77c034c053  felhom-installer-1.27.1-pve9.2-1.iso
@@ -0,0 +1,11 @@
+## 2026-09-16T09:12:26Z hub baseline for tester-1 (before the drill box registers)
+page size: 125233
+No host
+dr_tier" checked
+enkicsifelhom.hu
+no host
+pbsdr_storage_id" name="pbsdr_storage_id"
+pbsdr_storage_id">PVE storage id
+selfbind-link" style="margin-top: 0.5rem;">
+selfbind_tokens) dies.push('self-bind token(s)')
+tester1@felhom.eu
@@ -0,0 +1,7 @@
+## 2026-09-16T09:02:22Z DEVIATION: the graphical entry needs blind Tab-walking over unlabeled buttons; switching to the installer's TEXT-MODE entry (the second boot-menu entry a volunteer is offered). The install path is shared (Install.pm).
+update VM 334: -boot order=scsi0;ide2
+boot: order=scsi0;ide2
+ide2: local:iso/felhom-installer-1.27.1-pve9.2-1.iso,media=cdrom,size=1665354K
+## 2026-09-16T09:12:16Z INSTALL started (text mode, summary as captured in screens/24)
+## 2026-09-16T09:12:50Z box answers on 192.168.0.128 (installed and booted)
+## CORRECTION 2026-09-16T09:13:39Z: the '09:12:50Z box answers on 192.168.0.128' line is WRONG — the INSTALLER's live system already holds that address, so the ping proved nothing about the install. Install progress is read from the console instead.
@@ -0,0 +1,23 @@
+ qm create 334 --name tester1-drill-0243 --memory 8192 --cores 4 --sockets 1 --cpu host --net0 virtio,bridge=vmbr0 --scsihw virtio-scsi-single --ostype l26 --agent 1
+ qm set 334 --scsi0 nvme-scratch:32 --scsi1 nvme-scratch:100 --ide2 local:iso/felhom-installer-1.27.1-pve9.2-1.iso,media=cdrom --vga std
+update VM 334: -ide2 local:iso/felhom-installer-1.27.1-pve9.2-1.iso,media=cdrom -scsi0 nvme-scratch:32 -scsi1 nvme-scratch:100 -vga std
+Formatting '/mnt/hdd_1/images/334/vm-334-disk-0.raw', fmt=raw size=34359738368 preallocation=off
+scsi0: successfully created disk 'nvme-scratch:334/vm-334-disk-0.raw,size=32G'
+Formatting '/mnt/hdd_1/images/334/vm-334-disk-1.raw', fmt=raw size=107374182400 preallocation=off
+scsi1: successfully created disk 'nvme-scratch:334/vm-334-disk-1.raw,size=100G'
+ qm set 334 --boot 'order=ide2;scsi0'
+update VM 334: -boot order=ide2;scsi0
+ qm config 334
+ grep -E 'scsi|ide2|boot|memory|cores|name'
+boot: order=ide2;scsi0
+cores: 4
+ide2: local:iso/felhom-installer-1.27.1-pve9.2-1.iso,media=cdrom,size=1665354K
+memory: 8192
+name: tester1-drill-0243
+scsi0: nvme-scratch:334/vm-334-disk-0.raw,size=32G
+scsi1: nvme-scratch:334/vm-334-disk-1.raw,size=100G
+scsihw: virtio-scsi-single
+ qm start 334
+ sleep 20
+ qm status 334
+status: running
@@ -0,0 +1,8 @@
+## 2026-09-16T09:12:21Z ep0 BASELINE before the drill box exists — scope: ns tester-1 only
+[]
+--- namespace dir:
+drwxr-xr-x 4096 /mnt/pbs-datastore/ns/tester-1
+drwxr-xr-x 4096 /mnt/pbs-datastore/ns/tester-1/ct
+--- token:
+Path: /datastore/felhom-offsite/tester-1
+- Datastore.Backup (*)
@@ -26,6 +26,15 @@

 ---

+## 2026-09-16 — the drill's Phase 0 (hub v0.115.0, golden 0.243.0, the signing ruling)
+
+Two rows closed. **Full original text: `git show <this commit>^ -- documentation/backlog/OPEN-ITEMS.md`.**
+
+| ID | Title | Shipped | Evidence |
+|---|---|---|---|
+| **R-529** | [P3-LOW] The agent-plane `host_stale` / `host_down` / `host_recovered` mails still wait out the one-hour quiet rule. | hub v0.115.0 (2026-09-16, operator ruling 2) | `nodeLivenessEvents` + red-proof `TestOperatorCooldown_NodeLivenessBypassesQuietHour`; `08-alarm-ladder.md` §6.2 |
+| **R-533** | [P3-LOW] The operator signing keys were placed on DooPlex world-readable (mode 664) and sit outside any documented location. | operator ruling 1, 2026-09-16 | keys at `/mnt/5_hdd/felhom.eu/felhom-op-{operational,rec-recovery}` + `felhom_op_ed25519`, mode 0600 owner `kisfenyo`; `CONTEXT.md`, `04-control-plane-authorization.md` §3.1 |
+
 ## 2026-09-15 — the big night's P1 fixes (agent v0.131.0, controller v0.243.0, hub v0.114.0, catalog templates, ISO 1.27.1 published)

 Nine rows closed. **Full original text: `git show <this commit>^ -- documentation/backlog/OPEN-ITEMS.md`.** Evidence folder: `documentation/audits/evidence-p1fixes-2026-09-15/`.
@@ -723,11 +723,9 @@ class (an image `VOLUME` at an unmounted path) is still live — `immich-server`
 | **R-526** | **[P3-LOW] A host delete cannot release only the customer's ep0 PBS token: the endpoint's one removal op destroys every backup group too.** MEASURED 2026-09-15 from source: `tenantsync.Deprovision` „DESTROYS the customer's PBS namespace, all its backup groups, and its token". The task asked for „PBS token elengedése" on host delete; building it needs a new token-only op in the ep0 tenantsync script — a new operation on a protected box. Not built. R-511's adopt path makes the kept token usable instead. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (new ep0 op yes/no), CC (build)** |
 | **R-527** | **[P3-LOW] The catalog flag `locked_after_deploy` is read by no controller code — every setting is read-only after install whatever the catalog says.** FOUND 2026-09-15: `stacks/metadata.go` parses it; `grep -rn LockedAfterDeploy` finds no reader; `deploy.html` renders „Az alábbi beállítások csak olvashatók" for every field. Recorded as the design in `02-controller-module-map.md`; the flag is a seam never wired. **Fix shape:** remove the flag from the catalog, or wire an editable-after-install allow-list (a bigger change). | **READY — rank P3-LOW; owner: CC** |
 | **R-528** | **[P2-MEDIUM] Docker does not report an OOM kill inside a Felhom LXC guest: `OOMKilled` stays false and no `oom` event fires, so the v0.243.0 OOM line is not proven live.** MEASURED 2026-09-15 on scratch 9202 (Docker 29.8.0): Paperless capped at 128M restarted 11 times with `OOMKilled=false` and zero `docker events --filter event=oom`; a memory hog inside the running container was killed (rc 137) with the same silence (`E2-oom-signal-measure-9202.txt`). BIGNIGHT VM 333 did read `oomkilled=true`, so the shape differs by case. **Fix shape:** the agent reads the guest container cgroups' `memory.events oom_kill` counters (host-side, reliable), or the controller alarms on a restart-count trend. | **READY — rank P2-MEDIUM; owner: CC** |
-| **R-529** | **[P3-LOW] The agent-plane `host_stale` / `host_down` / `host_recovered` mails still wait out the one-hour quiet rule.** The 2026-09-15 ruling (decision A) named `node_*` only, and the task fenced „a design is not a defect". The same F9 silence can happen on the host plane. **What it needs:** the operator's word whether the ruling extends to `host_*`. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator** |
-| **R-530** | **[P2-MEDIUM] A floor does not deliver an agent: agents update only by an operator-signed `agent_update` job per box, and nothing records which boxes still run 0.130.0.** MEASURED 2026-09-15: the hub HOLDS a floor whose declared MinAgent is above the box's agent (`api/handler.go` ResolveManagedFloor); the agent's only update path is `signedjobs` + `selfupdate.Executor`. demo-hp reached 0.131.0 by `felhom-opsign -op agent_update` (key `felhom-op-1`) at 08:44:16Z and its controller floor was then SERVED in 3 s. **demo-felhom (N100) and Peti's box still run 0.130.0** — not touched (Peti fenced; N100 not asked). **What it needs:** the operator signs per box, or rules a fleet rollout step. | **WAITING-ON-OPERATOR — rank P2-MEDIUM; owner: operator (signing)** |
+| **R-530** | **[P2-MEDIUM] A floor does not deliver an agent: agents update only by an operator-signed `agent_update` job per box, and nothing records which boxes still run 0.130.0.** MEASURED 2026-09-15: the hub HOLDS a floor whose declared MinAgent is above the box's agent (`api/handler.go` ResolveManagedFloor); the agent's only update path is `signedjobs` + `selfupdate.Executor`. demo-hp reached 0.131.0 by `felhom-opsign -op agent_update` (key `felhom-op-1`) at 08:44:16Z and its controller floor was then SERVED in 3 s. **demo-felhom (N100) and Peti's box still run 0.130.0** — not touched (Peti fenced; N100 not asked). **What it needs:** the operator signs per box, or rules a fleet rollout step. **NARROWED 2026-09-16 (operator ruling 1):** the keys stay on DooPlex owner-only and CC may sign `agent_update` until the first PAYING customer (testers excluded) — recorded in `CONTEXT.md` + `04-control-plane-authorization.md` §3.1. Both demo boxes now run agent 0.131.0 (demo-hp 2026-09-15, demo-felhom 2026-09-16, each by a per-box signed job; Peti's box untouched, still 0.130.0). **What remains:** a fleet rollout step — signing per box does not scale past a handful, and nothing lists which boxes are behind. | **WAITING-ON-OPERATOR — rank P2-MEDIUM; owner: operator (signing)** |
 | **R-531** | **[P3-LOW] Three supervisor facts measured live and not pinned: restart timing during a deploy was not measured; restarts before the hub first sees the stanza produce no `controller_restarted_by_agent`; deliberate operator kills spend the crash-loop budget.** MEASURED 2026-09-15 on 9201: after 3 test restarts in 13 minutes the 4th kill tripped the 30-minute pause and the dashboard stayed down (the guard as designed, `A4-kill-middeploy-9201.txt`). The hub checker seeds silently on first sight, so the three restarts before the first v0.131.0 report emitted nothing (only the crash-loop did). **What it needs:** a deploy-kill timing on a fresh budget; the operator's view whether a restart after minutes of uptime should count toward the budget. | **READY — rank P3-LOW; owner: CC (measure) · operator (budget rule)** |
 | **R-532** | **[P3-LOW] Vaultwarden's `/api/config` still says `disableUserRegistration:false` with signups off, so the web vault shows a register form that the server then refuses.** MEASURED 2026-09-15 in the E.1 spike. Cosmetic: the server refuses (400). A household following the invite-first card is not affected; a stranger sees a form that fails. | **READY — rank P3-LOW; owner: CC (catalog/upstream note)** |
-| **R-533** | **[P3-LOW] The operator signing keys were placed on DooPlex world-readable (mode 664) and sit outside any documented location.** OBSERVED 2026-09-15: `/mnt/5_hdd/felhom.eu/felhom-op-operational`, `felhom_op_ed25519`, `felhom-rec-recovery` arrived 664; CC set them to 600 (no other change). Their fingerprints match the signers demo-hp pins. **What it needs:** the operator decides where the keys live between sessions (hardware key, or a documented 0600 path) and records it in `operations/nodes.md`. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator** |

 <!-- DUE-CHECKS-BEGIN — machine-readable. Parsed by scripts/due_checks_gate.py.
     One row per dated check. The R-number must have a row above. Dates are UTC.