diff --git a/documentation/architecture/11-os-updates.md b/documentation/architecture/11-os-updates.md index 0a703a76..6efb6cc6 100644 --- a/documentation/architecture/11-os-updates.md +++ b/documentation/architecture/11-os-updates.md @@ -635,9 +635,10 @@ until a later step ends `good`. **Proven** on demo-hp by hand: 8 kernels → 2, 6.4 GB gained on the system disk, the default, the pinned file and both generators unchanged, five refusals by simulation, one orderly reboot back onto the same kernel with a clean stop for -the crash guard (§5.9), a second run that removed nothing. **Not proven:** a box whose running kernel is not its -newest; a boot of the kept old kernel after the clean-up; the step as a program at night; a box that boots with -`proxmox-boot-tool`. Evidence, the rule as a read-only script, and the full "not measured" list: +the crash guard (§5.9), a second run that removed nothing. The kept old kernel then booted the box to healthy through +the one-shot flag (313 s), and the box went back to its newest kernel (324 s); the pinned file, `grub.cfg` and the +step's state file did not change. **Not proven:** a box whose running kernel is not its newest; a self-revert into +the kept kernel chosen by the product; the step as a program at night; a box that boots with `proxmox-boot-tool`. Evidence, the rule as a read-only script, and the full "not measured" list: `audits/kernel-cleanup-spike-2026-11-10/`. ## 6. Risks and edge cases diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/README.md b/documentation/audits/kernel-cleanup-spike-2026-11-10/README.md index f0ddae0b..d7ed0707 100644 --- a/documentation/audits/kernel-cleanup-spike-2026-11-10/README.md +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/README.md @@ -81,13 +81,39 @@ and 42% after. - `felhom-os-apply --kernel-status` needs a plan file in the agent's folder. The run read its inputs directly (the state file, the flag, `grub.cfg`). +## Measured later the same day: the kept old kernel boots (`evidence/fallback-boot/`) + +**Yes. The kept old kernel (7.0.14-26-pve) booted demo-hp to healthy, and the box went back to its newest kernel.** +A second run, 2026-11-10 14:29–14:46 UTC, two orderly restarts. + +| | Old kernel (26) | Back on the newest (28) | +|---|---|---| +| Restart sent (UTC) | 14:30:24 | 14:40:09 | +| SSH answered after | 166 s | 176 s | +| Every health point good after | 313 s | 324 s | + +- **Method.** The product's own boot-side switch, set by hand: `grub-editenv /boot/efi/EFI/felhom/oneshot.env set + felhom_next=7.0.14-26-pve`, then `systemctl reboot`. GRUB cleared the flag and booted 26 once, with the panic + parameters of a trial boot (`softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10`). The second restart + had no flag, so the box booted its default, 28. No wrapper verb ran; nothing was built or changed in the product. +- **Healthy** means the host health rule of `11-os-updates.md` §8.2, read point by point: the agent and the four + Proxmox services active, both guests running, Docker answering, both guests resolving `deb.debian.org`, the + controllers healthy, every container back, the tunnel connected. The limit was 15 minutes. +- **Nothing was left behind.** The pinned default file, `grub.cfg` and the kernel-step state file have the same sha256 + before, on the old kernel and after. The step state stayed `good`. The crash guard saw two clean stops. +- **The hub.** No kernel-step report came from the test (3 lines for this host before and after, all from the + morning's step). No alarm line. Only the "controller started" notes of the two restarts. +- **A different channel.** On the old kernel both public dashboards answered through Cloudflare (302). The hub's row + for the host said "unknown (no running customer guest)" then: it was the agent's first report after the boot, and no + newer report came in the six minutes the box stayed on 26. +- **Not covered by this.** A step where the product itself chooses the old kernel (a self-revert after a bad trial + boot). Here a person set the flag. + ## Not measured - A box whose running kernel is **not** its newest one (after a fall-back or a self-revert). The rule keeps both; no such box was tried. - A real kernel step that has not ended. Control C is a pretended phase. -- **A boot of the kept old kernel (26) after the clean-up.** Its files and its menu entries exist and `grub.cfg` - passes the syntax check; it was not booted. - The N100 and the Tester 1 VM (8 and 3 kernels). Read-only in this run. - The step as a program, at night, next to the agent's own apt runs. A person checked the lock by hand. - A box that boots with `proxmox-boot-tool` (demo-hp does not use it), and a box with out-of-tree kernel modules diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/01-before.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/01-before.txt new file mode 100644 index 00000000..04bc2e81 --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/01-before.txt @@ -0,0 +1,87 @@ +### BEFORE — 2026-11-10T14:29:24Z (read on demo-hp, read only) +uname -r: 7.0.14-28-pve +boot_id: d1167d94-31bf-459a-bf2a-569e8997a569 +up since (UTC): 2026-11-10T12:57:00Z +cmdline: BOOT_IMAGE=/boot/vmlinuz-7.0.14-28-pve root=/dev/mapper/pve-root ro quiet +== one-shot flag (grub-editenv /boot/efi/EFI/felhom/oneshot.env list) +felhom_next= +== pinned default file +6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg +GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43" +== grub.cfg +805cc2ba2da6054d676601590412b31ba36061aa7558e4f1391b6874710e16a3 /boot/grub/grub.cfg +22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43" +== kernel step state +607a2ce7ce3b6ca1c644695587f66ca280227d98670d6e9fb73d1b5f0080b8d0 /var/lib/felhom-kernel/state.json +{'phase': 'good', 'from': '7.0.14-26-pve', 'to': '7.0.14-28-pve', 'step_id': 'ring0-20261110T014802Z', 'result_at': '2026-11-10T01:56:47Z', 'updated_at': '2026-11-10T01:56:47Z'} +== kernels installed +/boot/vmlinuz-7.0.14-26-pve +/boot/vmlinuz-7.0.14-28-pve +== hand pin files +ls: cannot access '/etc/kernel/proxmox-boot-pin': No such file or directory +ls: cannot access '/etc/kernel/next-boot-pin': No such file or directory +== crash guard +{'armed': True, 'tripped': False, 'boot_id': 'd1167d94-31bf-459a-bf2a-569e8997a569', 'last_boot_at': '2026-11-10T12:56:59Z', 'last_boot_unclean': False, 'unclean_boots': [], 'unclean_boots_in_window': 0, 'unclean_boots_24h': 0, 'updated_at': '2026-11-10T14:12:09Z'} +== host services +felhom-agent: active +pve-cluster: active +pvedaemon: active +pveproxy: active +pvestatd: active +failed units: 0 +== tunnel +handshake age s: 87 +== guests +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness + VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID + 341 night1004-tester1 running 8192 200.00 1847 +== running jobs (vzdump, restore, apt, dpkg, backup client, wrapper) + 321567 00:00 /usr/bin/proxmox-backup-client status --output-format=json --repository felhom@pbs!demo-hp@10.77.0.1:felhom-offsite +(end) +== containers in 9201 +adventurelog-frontend Up 2 hours (healthy) +adventurelog-postgres Up 2 hours (healthy) +adventurelog Up 2 hours (healthy) +bentopdf Up 2 hours (healthy) +bookstack-db Up 2 hours (healthy) +bookstack Up 2 hours (healthy) +calibre-web Up 2 hours (healthy) +cloudflared Up 2 hours (healthy) +docmost-postgres Up 2 hours (healthy) +docmost-redis Up 2 hours (healthy) +docmost Up 2 hours (healthy) +felhom-controller Up 2 hours (healthy) +filebrowser Up 2 hours (healthy) +home-assistant Up 2 hours (healthy) +kimai-db Up 2 hours (healthy) +kimai Up 2 hours (healthy) +opengist Up 2 hours (healthy) +paperless-postgres Up 2 hours (healthy) +paperless-redis Up 2 hours (healthy) +paperless-webserver Up 2 hours (healthy) +plex Up 2 hours (healthy) +privatebin Up 2 hours (healthy) +romm-db Up 2 hours (healthy) +romm-redis Up 2 hours (healthy) +romm Up 2 hours (healthy) +traefik Up 2 hours +count: 26 running; not healthy: 1 +== containers in 9202 +felhom-controller Up 2 hours (healthy) +filebrowser Up 2 hours (healthy) +paperless-postgres Up 2 hours (healthy) +paperless-redis Up 2 hours (healthy) +paperless-webserver Up 2 hours (healthy) +traefik Up 2 hours +count: 6 running; not healthy: 1 +== Tester 1 VM 341 +status: running +guest agent: no answer (rc 255) +== disk +/dev/mapper/pve-root 39G 17G 21G 45% / +== agent log, kernel lines since boot +Nov 10 13:57:15 demo-hp sudo[1275]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T125715Z-kernel-kernel-boot.json +(end) diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/02-hub-before.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/02-hub-before.txt new file mode 100644 index 00000000..6c6fd695 --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/02-hub-before.txt @@ -0,0 +1,7 @@ +### before — 2026-11-10T14:29:35Z — hub, read only: GET /hosts -> 200, GET /os/fleet -> 403 +== /hosts rows + demo-felhom-8363b5 Demo Ügyfél 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 5% 23% 38% running (connected) 38% local + demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 11% 38% 42% running (connected) 71% local-lvm + tester-1-d70be4 Tester 1 0.172.1 floor: declared MinAgent 9.2.2 7.0.14-22-pve ONLINE 1/2 0% 38% 34% running (connected) 34% local +== /os/fleet (kernel parts) + not JSON: Expecting value: line 1 column 1 (char 0) diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/03-boot-old-kernel.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/03-boot-old-kernel.txt new file mode 100644 index 00000000..d7e4867a --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/03-boot-old-kernel.txt @@ -0,0 +1,20 @@ +### Boot the kept older kernel ONCE — the product's own boot flag, set by hand (no wrapper verb, no step state) +# command on demo-hp: grub-editenv /boot/efi/EFI/felhom/oneshot.env set felhom_next=7.0.14-26-pve ; then: systemctl reboot +set rc=0 +read back: felhom_next=7.0.14-26-pve +607a2ce7ce3b6ca1c644695587f66ca280227d98670d6e9fb73d1b5f0080b8d0 /var/lib/felhom-kernel/state.json +6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg +805cc2ba2da6054d676601590412b31ba36061aa7558e4f1391b6874710e16a3 /boot/grub/grub.cfg +reboot sent at 2026-11-10T14:30:24Z (T0=1794321024), old boot_id d1167d94-31bf-459a-bf2a-569e8997a569 ++129s ping DOWN ++166s ssh answers, NEW boot_id=e2cc9e3e-60f1-4bde-8949-f01d7ba35ccf, uname -r=7.0.14-26-pve ++171s 9201 n=0 healthy=0 other=0 | 9202 n=0 healthy=0 other=0 | vm341=stopped | services-not-active=0 | 9201=stopped 9202=stopped ++191s 9201 n=26 healthy=0 other=25 | 9202 n=6 healthy=3 other=2 | vm341=running | services-not-active=0 | 9201=running 9202=running ++210s 9201 n=26 healthy=15 other=10 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running ++228s 9201 n=26 healthy=15 other=10 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running ++246s 9201 n=26 healthy=20 other=5 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running ++264s 9201 n=26 healthy=22 other=3 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running ++282s 9201 n=26 healthy=24 other=1 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running ++297s 9201 n=26 healthy=24 other=1 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running ++313s 9201 n=26 healthy=25 other=0 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running +WATCH RESULT=ok kernel=7.0.14-26-pve want=7.0.14-26-pve ping-down=+129s ssh=+166s all-healthy=+313s diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/04-on-old-kernel.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/04-on-old-kernel.txt new file mode 100644 index 00000000..7cd227bd --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/04-on-old-kernel.txt @@ -0,0 +1,98 @@ +### ON-OLD-KERNEL — 2026-11-10T14:35:44Z (read on demo-hp, read only) +uname -r: 7.0.14-26-pve +boot_id: e2cc9e3e-60f1-4bde-8949-f01d7ba35ccf +up since (UTC): 2026-11-10T14:32:55Z +cmdline: BOOT_IMAGE=/boot/vmlinuz-7.0.14-26-pve root=/dev/mapper/pve-root ro quiet softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10 +== one-shot flag (grub-editenv /boot/efi/EFI/felhom/oneshot.env list) +felhom_next= +== pinned default file +6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg +GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43" +== grub.cfg +805cc2ba2da6054d676601590412b31ba36061aa7558e4f1391b6874710e16a3 /boot/grub/grub.cfg +22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43" +== kernel step state +607a2ce7ce3b6ca1c644695587f66ca280227d98670d6e9fb73d1b5f0080b8d0 /var/lib/felhom-kernel/state.json +{'phase': 'good', 'from': '7.0.14-26-pve', 'to': '7.0.14-28-pve', 'step_id': 'ring0-20261110T014802Z', 'result_at': '2026-11-10T01:56:47Z', 'updated_at': '2026-11-10T01:56:47Z'} +== kernels installed +/boot/vmlinuz-7.0.14-26-pve +/boot/vmlinuz-7.0.14-28-pve +== hand pin files +ls: cannot access '/etc/kernel/proxmox-boot-pin': No such file or directory +ls: cannot access '/etc/kernel/next-boot-pin': No such file or directory +== crash guard +{'armed': True, 'tripped': False, 'boot_id': 'e2cc9e3e-60f1-4bde-8949-f01d7ba35ccf', 'last_boot_at': '2026-11-10T14:32:54Z', 'last_boot_unclean': False, 'unclean_boots': [], 'unclean_boots_in_window': 0, 'unclean_boots_24h': 0, 'updated_at': '2026-11-10T14:33:04Z'} +== host services +felhom-agent: active +pve-cluster: active +pvedaemon: active +pveproxy: active +pvestatd: active +failed units: 0 +== tunnel +handshake age s: 32 +== guests +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness + VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID + 341 night1004-tester1 running 8192 200.00 1871 +== running jobs (vzdump, restore, apt, dpkg, backup client, wrapper) +(end) +== containers in 9201 +adventurelog-frontend Up 2 minutes (healthy) +adventurelog-postgres Up 2 minutes (healthy) +adventurelog Up 2 minutes (healthy) +bentopdf Up 2 minutes (healthy) +bookstack-db Up 2 minutes (healthy) +bookstack Up 2 minutes (healthy) +calibre-web Up 2 minutes (healthy) +cloudflared Up 2 minutes (healthy) +docmost-postgres Up 2 minutes (healthy) +docmost-redis Up 2 minutes (healthy) +docmost Up 2 minutes (healthy) +felhom-controller Up 2 minutes (healthy) +filebrowser Up 2 minutes (healthy) +home-assistant Up 2 minutes (healthy) +kimai-db Up 2 minutes (healthy) +kimai Up 2 minutes (healthy) +opengist Up 2 minutes (healthy) +paperless-postgres Up 2 minutes (healthy) +paperless-redis Up 2 minutes (healthy) +paperless-webserver Up About a minute (healthy) +plex Up About a minute (healthy) +privatebin Up 2 minutes (healthy) +romm-db Up About a minute (healthy) +romm-redis Up About a minute (healthy) +romm Up About a minute (healthy) +traefik Up 2 minutes +count: 26 running; not healthy: 1 +== containers in 9202 +felhom-controller Up 2 minutes (healthy) +filebrowser Up 2 minutes (healthy) +paperless-postgres Up 2 minutes (healthy) +paperless-redis Up 2 minutes (healthy) +paperless-webserver Up 2 minutes (healthy) +traefik Up 2 minutes +count: 6 running; not healthy: 1 +== Tester 1 VM 341 +status: running +guest agent: no answer (rc 255) +== disk +/dev/mapper/pve-root 39G 17G 21G 45% / +== agent log, kernel lines since boot +Nov 10 15:33:10 demo-hp sudo[1318]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T143309Z-kernel-kernel-boot.json +(end) +== host health rule, the two points snap.sh does not read (2026-11-10T14:36:21Z) +uname -r: 7.0.14-26-pve +-- 9201 +2a04:4e42:400::644 debian.map.fastlydns.net deb.debian.org +felhom-controller Up 2 minutes (healthy) +cloudflared Up 2 minutes (healthy) +-- 9202 +2a04:4e42:400::644 debian.map.fastlydns.net deb.debian.org +felhom-controller Up 3 minutes (healthy) +== hub row + demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-26-pve ONLINE 0/2 0% 7% 42% unknown (no running customer guest) 71% local-lvm + tester-1-d70be4 Tester 1 0.172.1 floor: declared MinAgent 9.2.2 7.0.14-22-pve ONLINE 0/2 0% 19% 34% unknown (no running customer guest) 34% local diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/04b-hub-on-old-kernel.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/04b-hub-on-old-kernel.txt new file mode 100644 index 00000000..23524491 --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/04b-hub-on-old-kernel.txt @@ -0,0 +1,24 @@ +== hub snapshot on the old kernel 2026-11-10T14:36:01Z +### ON-OLD-KERNEL — 2026-11-10T14:36:01Z — hub, read only: GET /hosts -> 200, GET /os/fleet -> 403 +== /hosts rows + demo-felhom-8363b5 Demo Ügyfél 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 5% 23% 38% running (connected) 38% local + demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-26-pve ONLINE 0/2 0% 7% 42% unknown (no running customer guest) 71% local-lvm + tester-1-d70be4 Tester 1 0.172.1 floor: declared MinAgent 9.2.2 7.0.14-22-pve ONLINE 0/2 0% 19% 34% unknown (no running customer guest) 34% local +== /os/fleet (kernel parts) + not JSON: Expecting value: line 1 column 1 (char 0) +== hub pod kernel lines (channel 2; 3 before the test) +3 +2026/11/10 02:50:50 [INFO] osupdates: demo-hp-bb76ea reported kernel run 20261110T014802Z: ring=0 mode=apply outcome=staged healthy=true upgraded=2 pending=0 not-covered=0 restart-needed=0 reboot-need +2026/11/10 02:54:08 [INFO] osupdates: demo-hp-bb76ea reported kernel run boot-20261110T015407Z: ring=0 mode=kernel-boot outcome=judging healthy=false upgraded=0 pending=0 not-covered=0 restart-needed= +2026/11/10 02:56:47 [INFO] osupdates: demo-hp-bb76ea reported kernel run boot-20261110T015407Z: ring=0 mode=kernel-good outcome=applied healthy=true upgraded=0 pending=0 not-covered=0 restart-needed=0 +== hub row again 2026-11-10T14:36:50Z + demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-26-pve ONLINE 0/2 0% 7% 42% unknown (no running customer guest) 71% local-lvm + tester-1-d70be4 Tester 1 0.172.1 floor: declared MinAgent 9.2.2 7.0.14-22-pve ONLINE 0/2 0% 19% 34% unknown (no running customer guest) 34% local +== hub row again 2026-11-10T14:37:08Z + demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-26-pve ONLINE 0/2 0% 7% 42% unknown (no running customer guest) 71% local-lvm + tester-1-d70be4 Tester 1 0.172.1 floor: declared MinAgent 9.2.2 7.0.14-22-pve ONLINE 0/2 0% 19% 34% unknown (no running customer guest) 34% local +NO MATCH by 2026-11-10T14:39:31Z: demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-26-pve ONLINE 0/2 0% 7% 42% unknown (no running customer guest) 71% local-lvm +== from outside, through Cloudflare (2026-11-10T14:39:45Z); the hub row had not taken a new report yet (cadence up to 15 min) +https://felhom.enkisfelhom.hu/ -> 302 +https://felhom.enkicsifelhom.hu/ -> 302 +Tester 1 host by SSH: 7.0.14-22-pve 9201 running tester-1 ;9 diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/05-boot-back.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/05-boot-back.txt new file mode 100644 index 00000000..7b2a29ae --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/05-boot-back.txt @@ -0,0 +1,23 @@ +### BOOT BACK — demo-hp, one orderly reboot, no flag set +uname -r: 7.0.14-26-pve +boot_id: e2cc9e3e-60f1-4bde-8949-f01d7ba35ccf +flag: felhom_next= +jobs: 0 +locks: 1 +607a2ce7ce3b6ca1c644695587f66ca280227d98670d6e9fb73d1b5f0080b8d0 /var/lib/felhom-kernel/state.json +6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg +805cc2ba2da6054d676601590412b31ba36061aa7558e4f1391b6874710e16a3 /boot/grub/grub.cfg +note: 'locks: 1' above counted the header line of 'pct list' (column 'Lock'); no guest config holds a lock line (checked /etc/pve/lxc/*.conf and qemu-server/*.conf), no job process runs +reboot sent at 2026-11-10T14:40:09Z (T0=1794321609), old boot_id e2cc9e3e-60f1-4bde-8949-f01d7ba35ccf ++139s ping DOWN ++176s ssh answers, NEW boot_id=bed106f7-9e80-4eb1-ace3-14a26da7f049, uname -r=7.0.14-28-pve ++181s 9201 n=0 healthy=0 other=0 | 9202 n=0 healthy=0 other=0 | vm341=stopped | services-not-active=0 | 9201=stopped 9202=stopped ++201s 9201 n=26 healthy=0 other=25 | 9202 n=6 healthy=3 other=2 | vm341=running | services-not-active=0 | 9201=running 9202=running ++220s 9201 n=26 healthy=14 other=11 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running ++239s 9201 n=26 healthy=15 other=10 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running ++256s 9201 n=26 healthy=20 other=5 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running ++275s 9201 n=26 healthy=22 other=3 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running ++293s 9201 n=26 healthy=24 other=1 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running ++309s 9201 n=26 healthy=24 other=1 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running ++324s 9201 n=26 healthy=25 other=0 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running +WATCH RESULT=ok kernel=7.0.14-28-pve want=7.0.14-28-pve ping-down=+139s ssh=+176s all-healthy=+324s diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/06-after.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/06-after.txt new file mode 100644 index 00000000..3941fea3 --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/06-after.txt @@ -0,0 +1,91 @@ +### AFTER — 2026-11-10T14:45:36Z (read on demo-hp, read only) +uname -r: 7.0.14-28-pve +boot_id: bed106f7-9e80-4eb1-ace3-14a26da7f049 +up since (UTC): 2026-11-10T14:42:50Z +cmdline: BOOT_IMAGE=/boot/vmlinuz-7.0.14-28-pve root=/dev/mapper/pve-root ro quiet +== one-shot flag (grub-editenv /boot/efi/EFI/felhom/oneshot.env list) +felhom_next= +== pinned default file +6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg +GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43" +== grub.cfg +805cc2ba2da6054d676601590412b31ba36061aa7558e4f1391b6874710e16a3 /boot/grub/grub.cfg +22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43" +== kernel step state +607a2ce7ce3b6ca1c644695587f66ca280227d98670d6e9fb73d1b5f0080b8d0 /var/lib/felhom-kernel/state.json +{'phase': 'good', 'from': '7.0.14-26-pve', 'to': '7.0.14-28-pve', 'step_id': 'ring0-20261110T014802Z', 'result_at': '2026-11-10T01:56:47Z', 'updated_at': '2026-11-10T01:56:47Z'} +== kernels installed +/boot/vmlinuz-7.0.14-26-pve +/boot/vmlinuz-7.0.14-28-pve +== hand pin files +ls: cannot access '/etc/kernel/proxmox-boot-pin': No such file or directory +ls: cannot access '/etc/kernel/next-boot-pin': No such file or directory +== crash guard +{'armed': True, 'tripped': False, 'boot_id': 'bed106f7-9e80-4eb1-ace3-14a26da7f049', 'last_boot_at': '2026-11-10T14:42:49Z', 'last_boot_unclean': False, 'unclean_boots': [], 'unclean_boots_in_window': 0, 'unclean_boots_24h': 0, 'updated_at': '2026-11-10T14:42:59Z'} +== host services +felhom-agent: active +pve-cluster: active +pvedaemon: active +pveproxy: active +pvestatd: active +failed units: 0 +== tunnel +handshake age s: 30 +== guests +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness + VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID + 341 night1004-tester1 running 8192 200.00 1950 +== running jobs (vzdump, restore, apt, dpkg, backup client, wrapper) +(end) +== containers in 9201 +adventurelog-frontend Up 2 minutes (healthy) +adventurelog-postgres Up 2 minutes (healthy) +adventurelog Up 2 minutes (healthy) +bentopdf Up 2 minutes (healthy) +bookstack-db Up 2 minutes (healthy) +bookstack Up 2 minutes (healthy) +calibre-web Up 2 minutes (healthy) +cloudflared Up 2 minutes (healthy) +docmost-postgres Up 2 minutes (healthy) +docmost-redis Up 2 minutes (healthy) +docmost Up 2 minutes (healthy) +felhom-controller Up 2 minutes (healthy) +filebrowser Up 2 minutes (healthy) +home-assistant Up 2 minutes (healthy) +kimai-db Up 2 minutes (healthy) +kimai Up 2 minutes (healthy) +opengist Up 2 minutes (healthy) +paperless-postgres Up About a minute (healthy) +paperless-redis Up 2 minutes (healthy) +paperless-webserver Up About a minute (healthy) +plex Up About a minute (healthy) +privatebin Up 2 minutes (healthy) +romm-db Up About a minute (healthy) +romm-redis Up About a minute (healthy) +romm Up About a minute (healthy) +traefik Up 2 minutes +count: 26 running; not healthy: 1 +== containers in 9202 +felhom-controller Up 2 minutes (healthy) +filebrowser Up 2 minutes (healthy) +paperless-postgres Up 2 minutes (healthy) +paperless-redis Up 2 minutes (healthy) +paperless-webserver Up 2 minutes (healthy) +traefik Up 2 minutes +count: 6 running; not healthy: 1 +== Tester 1 VM 341 +status: running +guest agent: no answer (rc 255) +== disk +/dev/mapper/pve-root 39G 17G 21G 45% / +== agent log, kernel lines since boot +Nov 10 15:43:05 demo-hp sudo[1323]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T144304Z-kernel-kernel-boot.json +(end) +2a04:4e42:200::644 debian.map.fastlydns.net deb.debian.org +felhom-controller Up 2 minutes (healthy) +cloudflared Up 2 minutes (healthy) +2a04:4e42::644 debian.map.fastlydns.net deb.debian.org +felhom-controller Up 2 minutes (healthy) diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/07-hub-after.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/07-hub-after.txt new file mode 100644 index 00000000..aee94e4f --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/07-hub-after.txt @@ -0,0 +1,24 @@ +### HUB AFTER — 2026-11-10T14:46:02Z +== hub pod kernel lines for demo-hp, last 14 h (3 before the test; the pod log prints CET) +3 +2026/11/10 02:50:50 [INFO] osupdates: demo-hp-bb76ea reported kernel run 20261110T014802Z: ring=0 mode=apply outcome=staged healthy=true upgraded=2 pending=0 no +2026/11/10 02:54:08 [INFO] osupdates: demo-hp-bb76ea reported kernel run boot-20261110T015407Z: ring=0 mode=kernel-boot outcome=judging healthy=false upgraded=0 +2026/11/10 02:56:47 [INFO] osupdates: demo-hp-bb76ea reported kernel run boot-20261110T015407Z: ring=0 mode=kernel-good outcome=applied healthy=true upgraded=0 +== positive control: the same log holds lines for demo-hp since the test began (15:29 CET) +6 +== every hub log line since the test began that names an event, an alarm or a mail +2026/11/10 15:33:40 [INFO] Event from demo-hp: controller_started (info) — Controller elindult (0.351.0) +2026/11/10 15:34:18 [INFO] Notification prefs push for tester-1: household cleared its mail address — deleted +2026/11/10 15:34:18 [INFO] Notification preferences updated for tester-1: address set=false, household=true, events=[] +2026/11/10 15:34:23 [INFO] Event from tester-1: controller_started (info) — Controller elindult (0.351.0) +2026/11/10 15:43:36 [INFO] Event from demo-hp: controller_started (info) — Controller elindult (0.351.0) +2026/11/10 15:44:12 [INFO] Notification prefs push for tester-1: household cleared its mail address — deleted +2026/11/10 15:44:12 [INFO] Notification preferences updated for tester-1: address set=false, household=true, events=[] +2026/11/10 15:44:17 [INFO] Event from tester-1: controller_started (info) — Controller elindult (0.351.0) +== 'household cleared its mail address' lines for tester-1, last 30 h (is it the same at every controller start?) +2026/11/09 13:09:54 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete +2026/11/09 13:14:39 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete +2026/11/10 02:55:11 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete +2026/11/10 13:58:18 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete +2026/11/10 15:34:18 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete +2026/11/10 15:44:12 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 45c60763..3b8d9fc4 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -281,7 +281,7 @@ stopping line that lies. | **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC | | **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC | | **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC | -| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC | +| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC | | **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) **2026-11-10 (afternoon) — the runbook is written:** `documentation/runbooks/disk-full.md`, five parts, each hand step marked PROVEN (date and evidence; all on demo-hp) or NOT PROVEN. Still open: one rehearsal of the whole runbook on demo-hp; the NOT PROVEN steps (copies on a former target, template archives, the `.bak-*` binaries, the guest root disk); what fails when the EFI partition or a temp folder is full. | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC | | **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. **2026-11-10 (afternoon) — the section is written:** `16-storage-physical-identity.md` §14 (45 places, the list „no limit, by ruling”, the budget, the open design points); `PROMPT-TEMPLATE.md` §N.5 keeps it true. The budget from measured numbers: the root needs about 14 GB when the whole-box copies go to a backup drive, and 28–54 GB at keep 3 when they do not. **D75 (for the operator):** the smallest system disk Felhom accepts. CC's pick: 128 GB only with a backup drive, 200 GB without one. The other option, 128 GB always, lets a box with many apps and no backup drive stop its local copies. With no ruling the installer accepts any disk. Open after the ruling: the installer's refusal. **2026-11-10 15:19 — RULED (D75 = one number, 200 GB, with or without a backup drive; `09` §3 decision 295).** This replaces CC's two-step pick. **Open, not ruled:** what the installer does below 200 GB — refuse, or warn. demo-hp (128 GB) stays as a test box. `16` §14.3 carries the ruling. | — | Design first: what the installer does below 200 GB (refuse, or warn) — a later session. Then the installer. | CC | | **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC | diff --git a/documentation/runbooks/disk-full.md b/documentation/runbooks/disk-full.md index f2b1806b..6df7123c 100644 --- a/documentation/runbooks/disk-full.md +++ b/documentation/runbooks/disk-full.md @@ -43,7 +43,7 @@ hub alarms at 90 % and 95 %. | What | Step | Mark | |---|---|---| -| Old kernels | `ssh python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py`. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window | **PROVEN 2026-11-10**: 8 → 2 kernels, 6.4 GB, 50 s, one restart (`audits/kernel-cleanup-spike-2026-11-10/README.md`). **NOT PROVEN:** on a second box; a boot of the kept older kernel | +| Old kernels | `ssh python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py`. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window | **PROVEN 2026-11-10**: 8 → 2 kernels, 6.4 GB, 50 s, one restart (`audits/kernel-cleanup-spike-2026-11-10/README.md`). The kept older kernel then booted the box to healthy and the box went back (`audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **NOT PROVEN:** on a second box | | The host journal | Save it first, then `journalctl --vacuum-size=500M` | **PROVEN 2026-11-07**: 3.8 GB → 468 MB (`audits/demo-check-2026-11-07/README.md`; the record is off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/cleanup.txt`). It grows back (R-1367) | | An installer ISO that a newer one replaced; a test archive of a guest that is gone | `pvesm free `, after `pct list` and `qm list` show that no guest uses it, and after its checksum matches a copy elsewhere | **PROVEN 2026-10-23** (three ISOs, `audits/day-2026-10-23/09-demohp-space.txt`), **2026-11-07** (one archive of a removed guest), **2026-11-09** (one ISO). Test boxes only | | Whole-box copies on a former target (H4) | — | **NOT PROVEN.** Needs the operator's word (R-1087) |