kernel clean-up spike: the kept old kernel boots demo-hp to healthy, and the box goes back (R-1368)

The one-shot flag booted 7.0.14-26-pve: every health point good after 313 s. An orderly restart
brought 7.0.14-28-pve back: 324 s. The pinned default, grub.cfg and the step's state file did not
change. No kernel report and no alarm at the hub. Evidence: evidence/fallback-boot/.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-10 15:54:40 +01:00
1 parent e83a78ce61
commit 5f408caca8
12 files changed
+408 -7

No files matched your search

+4 -3
View File
@@ -635,9 +635,10 @@ until a later step ends `good`.
**Proven** on demo-hp by hand: 8 kernels → 2, 6.4 GB gained on the system disk, the default, the pinned file and both
generators unchanged, five refusals by simulation, one orderly reboot back onto the same kernel with a clean stop for
the crash guard (§5.9), a second run that removed nothing. **Not proven:** a box whose running kernel is not its
newest; a boot of the kept old kernel after the clean-up; the step as a program at night; a box that boots with
`proxmox-boot-tool`. Evidence, the rule as a read-only script, and the full "not measured" list:
the crash guard (§5.9), a second run that removed nothing. The kept old kernel then booted the box to healthy through
the one-shot flag (313 s), and the box went back to its newest kernel (324 s); the pinned file, `grub.cfg` and the
step's state file did not change. **Not proven:** a box whose running kernel is not its newest; a self-revert into
the kept kernel chosen by the product; the step as a program at night; a box that boots with `proxmox-boot-tool`. Evidence, the rule as a read-only script, and the full "not measured" list:
`audits/kernel-cleanup-spike-2026-11-10/`.
## 6. Risks and edge cases
@@ -81,13 +81,39 @@ and 42% after.
- `felhom-os-apply --kernel-status` needs a plan file in the agent's folder. The run read its inputs directly (the
state file, the flag, `grub.cfg`).
## Measured later the same day: the kept old kernel boots (`evidence/fallback-boot/`)
**Yes. The kept old kernel (7.0.14-26-pve) booted demo-hp to healthy, and the box went back to its newest kernel.**
A second run, 2026-11-10 14:29–14:46 UTC, two orderly restarts.
| | Old kernel (26) | Back on the newest (28) |
|---|---|---|
| Restart sent (UTC) | 14:30:24 | 14:40:09 |
| SSH answered after | 166 s | 176 s |
| Every health point good after | 313 s | 324 s |
- **Method.** The product's own boot-side switch, set by hand: `grub-editenv /boot/efi/EFI/felhom/oneshot.env set
felhom_next=7.0.14-26-pve`, then `systemctl reboot`. GRUB cleared the flag and booted 26 once, with the panic
parameters of a trial boot (`softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10`). The second restart
had no flag, so the box booted its default, 28. No wrapper verb ran; nothing was built or changed in the product.
- **Healthy** means the host health rule of `11-os-updates.md` §8.2, read point by point: the agent and the four
Proxmox services active, both guests running, Docker answering, both guests resolving `deb.debian.org`, the
controllers healthy, every container back, the tunnel connected. The limit was 15 minutes.
- **Nothing was left behind.** The pinned default file, `grub.cfg` and the kernel-step state file have the same sha256
before, on the old kernel and after. The step state stayed `good`. The crash guard saw two clean stops.
- **The hub.** No kernel-step report came from the test (3 lines for this host before and after, all from the
morning's step). No alarm line. Only the "controller started" notes of the two restarts.
- **A different channel.** On the old kernel both public dashboards answered through Cloudflare (302). The hub's row
for the host said "unknown (no running customer guest)" then: it was the agent's first report after the boot, and no
newer report came in the six minutes the box stayed on 26.
- **Not covered by this.** A step where the product itself chooses the old kernel (a self-revert after a bad trial
boot). Here a person set the flag.
## Not measured
- A box whose running kernel is **not** its newest one (after a fall-back or a self-revert). The rule keeps both; no
such box was tried.
- A real kernel step that has not ended. Control C is a pretended phase.
- **A boot of the kept old kernel (26) after the clean-up.** Its files and its menu entries exist and `grub.cfg`
passes the syntax check; it was not booted.
- The N100 and the Tester 1 VM (8 and 3 kernels). Read-only in this run.
- The step as a program, at night, next to the agent's own apt runs. A person checked the lock by hand.
- A box that boots with `proxmox-boot-tool` (demo-hp does not use it), and a box with out-of-tree kernel modules
@@ -0,0 +1,87 @@
### BEFORE — 2026-11-10T14:29:24Z (read on demo-hp, read only)
uname -r: 7.0.14-28-pve
boot_id: d1167d94-31bf-459a-bf2a-569e8997a569
up since (UTC): 2026-11-10T12:57:00Z
cmdline: BOOT_IMAGE=/boot/vmlinuz-7.0.14-28-pve root=/dev/mapper/pve-root ro quiet
== one-shot flag (grub-editenv /boot/efi/EFI/felhom/oneshot.env list)
felhom_next=
== pinned default file
6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg
GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
== grub.cfg
805cc2ba2da6054d676601590412b31ba36061aa7558e4f1391b6874710e16a3 /boot/grub/grub.cfg
22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
== kernel step state
607a2ce7ce3b6ca1c644695587f66ca280227d98670d6e9fb73d1b5f0080b8d0 /var/lib/felhom-kernel/state.json
{'phase': 'good', 'from': '7.0.14-26-pve', 'to': '7.0.14-28-pve', 'step_id': 'ring0-20261110T014802Z', 'result_at': '2026-11-10T01:56:47Z', 'updated_at': '2026-11-10T01:56:47Z'}
== kernels installed
/boot/vmlinuz-7.0.14-26-pve
/boot/vmlinuz-7.0.14-28-pve
== hand pin files
ls: cannot access '/etc/kernel/proxmox-boot-pin': No such file or directory
ls: cannot access '/etc/kernel/next-boot-pin': No such file or directory
== crash guard
{'armed': True, 'tripped': False, 'boot_id': 'd1167d94-31bf-459a-bf2a-569e8997a569', 'last_boot_at': '2026-11-10T12:56:59Z', 'last_boot_unclean': False, 'unclean_boots': [], 'unclean_boots_in_window': 0, 'unclean_boots_24h': 0, 'updated_at': '2026-11-10T14:12:09Z'}
== host services
felhom-agent: active
pve-cluster: active
pvedaemon: active
pveproxy: active
pvestatd: active
failed units: 0
== tunnel
handshake age s: 87
== guests
VMID Status Lock Name
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
341 night1004-tester1 running 8192 200.00 1847
== running jobs (vzdump, restore, apt, dpkg, backup client, wrapper)
321567 00:00 /usr/bin/proxmox-backup-client status --output-format=json --repository felhom@pbs!demo-hp@10.77.0.1:felhom-offsite
(end)
== containers in 9201
adventurelog-frontend Up 2 hours (healthy)
adventurelog-postgres Up 2 hours (healthy)
adventurelog Up 2 hours (healthy)
bentopdf Up 2 hours (healthy)
bookstack-db Up 2 hours (healthy)
bookstack Up 2 hours (healthy)
calibre-web Up 2 hours (healthy)
cloudflared Up 2 hours (healthy)
docmost-postgres Up 2 hours (healthy)
docmost-redis Up 2 hours (healthy)
docmost Up 2 hours (healthy)
felhom-controller Up 2 hours (healthy)
filebrowser Up 2 hours (healthy)
home-assistant Up 2 hours (healthy)
kimai-db Up 2 hours (healthy)
kimai Up 2 hours (healthy)
opengist Up 2 hours (healthy)
paperless-postgres Up 2 hours (healthy)
paperless-redis Up 2 hours (healthy)
paperless-webserver Up 2 hours (healthy)
plex Up 2 hours (healthy)
privatebin Up 2 hours (healthy)
romm-db Up 2 hours (healthy)
romm-redis Up 2 hours (healthy)
romm Up 2 hours (healthy)
traefik Up 2 hours
count: 26 running; not healthy: 1
== containers in 9202
felhom-controller Up 2 hours (healthy)
filebrowser Up 2 hours (healthy)
paperless-postgres Up 2 hours (healthy)
paperless-redis Up 2 hours (healthy)
paperless-webserver Up 2 hours (healthy)
traefik Up 2 hours
count: 6 running; not healthy: 1
== Tester 1 VM 341
status: running
guest agent: no answer (rc 255)
== disk
/dev/mapper/pve-root 39G 17G 21G 45% /
== agent log, kernel lines since boot
Nov 10 13:57:15 demo-hp sudo[1275]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T125715Z-kernel-kernel-boot.json
(end)
@@ -0,0 +1,7 @@
### before — 2026-11-10T14:29:35Z — hub, read only: GET /hosts -> 200, GET /os/fleet -> 403
== /hosts rows
demo-felhom-8363b5 Demo Ügyfél 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 5% 23% 38% running (connected) 38% local
demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 11% 38% 42% running (connected) 71% local-lvm
tester-1-d70be4 Tester 1 0.172.1 floor: declared MinAgent 9.2.2 7.0.14-22-pve ONLINE 1/2 0% 38% 34% running (connected) 34% local
== /os/fleet (kernel parts)
not JSON: Expecting value: line 1 column 1 (char 0)
@@ -0,0 +1,20 @@
### Boot the kept older kernel ONCE — the product's own boot flag, set by hand (no wrapper verb, no step state)
# command on demo-hp: grub-editenv /boot/efi/EFI/felhom/oneshot.env set felhom_next=7.0.14-26-pve ; then: systemctl reboot
set rc=0
read back: felhom_next=7.0.14-26-pve
607a2ce7ce3b6ca1c644695587f66ca280227d98670d6e9fb73d1b5f0080b8d0 /var/lib/felhom-kernel/state.json
6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg
805cc2ba2da6054d676601590412b31ba36061aa7558e4f1391b6874710e16a3 /boot/grub/grub.cfg
reboot sent at 2026-11-10T14:30:24Z (T0=1794321024), old boot_id d1167d94-31bf-459a-bf2a-569e8997a569
+129s ping DOWN
+166s ssh answers, NEW boot_id=e2cc9e3e-60f1-4bde-8949-f01d7ba35ccf, uname -r=7.0.14-26-pve
+171s 9201 n=0 healthy=0 other=0 | 9202 n=0 healthy=0 other=0 | vm341=stopped | services-not-active=0 | 9201=stopped 9202=stopped
+191s 9201 n=26 healthy=0 other=25 | 9202 n=6 healthy=3 other=2 | vm341=running | services-not-active=0 | 9201=running 9202=running
+210s 9201 n=26 healthy=15 other=10 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running
+228s 9201 n=26 healthy=15 other=10 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running
+246s 9201 n=26 healthy=20 other=5 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running
+264s 9201 n=26 healthy=22 other=3 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running
+282s 9201 n=26 healthy=24 other=1 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running
+297s 9201 n=26 healthy=24 other=1 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running
+313s 9201 n=26 healthy=25 other=0 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running
WATCH RESULT=ok kernel=7.0.14-26-pve want=7.0.14-26-pve ping-down=+129s ssh=+166s all-healthy=+313s
@@ -0,0 +1,98 @@
### ON-OLD-KERNEL — 2026-11-10T14:35:44Z (read on demo-hp, read only)
uname -r: 7.0.14-26-pve
boot_id: e2cc9e3e-60f1-4bde-8949-f01d7ba35ccf
up since (UTC): 2026-11-10T14:32:55Z
cmdline: BOOT_IMAGE=/boot/vmlinuz-7.0.14-26-pve root=/dev/mapper/pve-root ro quiet softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10
== one-shot flag (grub-editenv /boot/efi/EFI/felhom/oneshot.env list)
felhom_next=
== pinned default file
6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg
GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
== grub.cfg
805cc2ba2da6054d676601590412b31ba36061aa7558e4f1391b6874710e16a3 /boot/grub/grub.cfg
22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
== kernel step state
607a2ce7ce3b6ca1c644695587f66ca280227d98670d6e9fb73d1b5f0080b8d0 /var/lib/felhom-kernel/state.json
{'phase': 'good', 'from': '7.0.14-26-pve', 'to': '7.0.14-28-pve', 'step_id': 'ring0-20261110T014802Z', 'result_at': '2026-11-10T01:56:47Z', 'updated_at': '2026-11-10T01:56:47Z'}
== kernels installed
/boot/vmlinuz-7.0.14-26-pve
/boot/vmlinuz-7.0.14-28-pve
== hand pin files
ls: cannot access '/etc/kernel/proxmox-boot-pin': No such file or directory
ls: cannot access '/etc/kernel/next-boot-pin': No such file or directory
== crash guard
{'armed': True, 'tripped': False, 'boot_id': 'e2cc9e3e-60f1-4bde-8949-f01d7ba35ccf', 'last_boot_at': '2026-11-10T14:32:54Z', 'last_boot_unclean': False, 'unclean_boots': [], 'unclean_boots_in_window': 0, 'unclean_boots_24h': 0, 'updated_at': '2026-11-10T14:33:04Z'}
== host services
felhom-agent: active
pve-cluster: active
pvedaemon: active
pveproxy: active
pvestatd: active
failed units: 0
== tunnel
handshake age s: 32
== guests
VMID Status Lock Name
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
341 night1004-tester1 running 8192 200.00 1871
== running jobs (vzdump, restore, apt, dpkg, backup client, wrapper)
(end)
== containers in 9201
adventurelog-frontend Up 2 minutes (healthy)
adventurelog-postgres Up 2 minutes (healthy)
adventurelog Up 2 minutes (healthy)
bentopdf Up 2 minutes (healthy)
bookstack-db Up 2 minutes (healthy)
bookstack Up 2 minutes (healthy)
calibre-web Up 2 minutes (healthy)
cloudflared Up 2 minutes (healthy)
docmost-postgres Up 2 minutes (healthy)
docmost-redis Up 2 minutes (healthy)
docmost Up 2 minutes (healthy)
felhom-controller Up 2 minutes (healthy)
filebrowser Up 2 minutes (healthy)
home-assistant Up 2 minutes (healthy)
kimai-db Up 2 minutes (healthy)
kimai Up 2 minutes (healthy)
opengist Up 2 minutes (healthy)
paperless-postgres Up 2 minutes (healthy)
paperless-redis Up 2 minutes (healthy)
paperless-webserver Up About a minute (healthy)
plex Up About a minute (healthy)
privatebin Up 2 minutes (healthy)
romm-db Up About a minute (healthy)
romm-redis Up About a minute (healthy)
romm Up About a minute (healthy)
traefik Up 2 minutes
count: 26 running; not healthy: 1
== containers in 9202
felhom-controller Up 2 minutes (healthy)
filebrowser Up 2 minutes (healthy)
paperless-postgres Up 2 minutes (healthy)
paperless-redis Up 2 minutes (healthy)
paperless-webserver Up 2 minutes (healthy)
traefik Up 2 minutes
count: 6 running; not healthy: 1
== Tester 1 VM 341
status: running
guest agent: no answer (rc 255)
== disk
/dev/mapper/pve-root 39G 17G 21G 45% /
== agent log, kernel lines since boot
Nov 10 15:33:10 demo-hp sudo[1318]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T143309Z-kernel-kernel-boot.json
(end)
== host health rule, the two points snap.sh does not read (2026-11-10T14:36:21Z)
uname -r: 7.0.14-26-pve
-- 9201
2a04:4e42:400::644 debian.map.fastlydns.net deb.debian.org
felhom-controller Up 2 minutes (healthy)
cloudflared Up 2 minutes (healthy)
-- 9202
2a04:4e42:400::644 debian.map.fastlydns.net deb.debian.org
felhom-controller Up 3 minutes (healthy)
== hub row
demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-26-pve ONLINE 0/2 0% 7% 42% unknown (no running customer guest) 71% local-lvm
tester-1-d70be4 Tester 1 0.172.1 floor: declared MinAgent 9.2.2 7.0.14-22-pve ONLINE 0/2 0% 19% 34% unknown (no running customer guest) 34% local
@@ -0,0 +1,24 @@
== hub snapshot on the old kernel 2026-11-10T14:36:01Z
### ON-OLD-KERNEL — 2026-11-10T14:36:01Z — hub, read only: GET /hosts -> 200, GET /os/fleet -> 403
== /hosts rows
demo-felhom-8363b5 Demo Ügyfél 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 5% 23% 38% running (connected) 38% local
demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-26-pve ONLINE 0/2 0% 7% 42% unknown (no running customer guest) 71% local-lvm
tester-1-d70be4 Tester 1 0.172.1 floor: declared MinAgent 9.2.2 7.0.14-22-pve ONLINE 0/2 0% 19% 34% unknown (no running customer guest) 34% local
== /os/fleet (kernel parts)
not JSON: Expecting value: line 1 column 1 (char 0)
== hub pod kernel lines (channel 2; 3 before the test)
3
2026/11/10 02:50:50 [INFO] osupdates: demo-hp-bb76ea reported kernel run 20261110T014802Z: ring=0 mode=apply outcome=staged healthy=true upgraded=2 pending=0 not-covered=0 restart-needed=0 reboot-need
2026/11/10 02:54:08 [INFO] osupdates: demo-hp-bb76ea reported kernel run boot-20261110T015407Z: ring=0 mode=kernel-boot outcome=judging healthy=false upgraded=0 pending=0 not-covered=0 restart-needed=
2026/11/10 02:56:47 [INFO] osupdates: demo-hp-bb76ea reported kernel run boot-20261110T015407Z: ring=0 mode=kernel-good outcome=applied healthy=true upgraded=0 pending=0 not-covered=0 restart-needed=0
== hub row again 2026-11-10T14:36:50Z
demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-26-pve ONLINE 0/2 0% 7% 42% unknown (no running customer guest) 71% local-lvm
tester-1-d70be4 Tester 1 0.172.1 floor: declared MinAgent 9.2.2 7.0.14-22-pve ONLINE 0/2 0% 19% 34% unknown (no running customer guest) 34% local
== hub row again 2026-11-10T14:37:08Z
demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-26-pve ONLINE 0/2 0% 7% 42% unknown (no running customer guest) 71% local-lvm
tester-1-d70be4 Tester 1 0.172.1 floor: declared MinAgent 9.2.2 7.0.14-22-pve ONLINE 0/2 0% 19% 34% unknown (no running customer guest) 34% local
NO MATCH by 2026-11-10T14:39:31Z: demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-26-pve ONLINE 0/2 0% 7% 42% unknown (no running customer guest) 71% local-lvm
== from outside, through Cloudflare (2026-11-10T14:39:45Z); the hub row had not taken a new report yet (cadence up to 15 min)
https://felhom.enkisfelhom.hu/ -> 302
https://felhom.enkicsifelhom.hu/ -> 302
Tester 1 host by SSH: 7.0.14-22-pve 9201 running tester-1 ;9
@@ -0,0 +1,23 @@
### BOOT BACK — demo-hp, one orderly reboot, no flag set
uname -r: 7.0.14-26-pve
boot_id: e2cc9e3e-60f1-4bde-8949-f01d7ba35ccf
flag: felhom_next=
jobs: 0
locks: 1
607a2ce7ce3b6ca1c644695587f66ca280227d98670d6e9fb73d1b5f0080b8d0 /var/lib/felhom-kernel/state.json
6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg
805cc2ba2da6054d676601590412b31ba36061aa7558e4f1391b6874710e16a3 /boot/grub/grub.cfg
note: 'locks: 1' above counted the header line of 'pct list' (column 'Lock'); no guest config holds a lock line (checked /etc/pve/lxc/*.conf and qemu-server/*.conf), no job process runs
reboot sent at 2026-11-10T14:40:09Z (T0=1794321609), old boot_id e2cc9e3e-60f1-4bde-8949-f01d7ba35ccf
+139s ping DOWN
+176s ssh answers, NEW boot_id=bed106f7-9e80-4eb1-ace3-14a26da7f049, uname -r=7.0.14-28-pve
+181s 9201 n=0 healthy=0 other=0 | 9202 n=0 healthy=0 other=0 | vm341=stopped | services-not-active=0 | 9201=stopped 9202=stopped
+201s 9201 n=26 healthy=0 other=25 | 9202 n=6 healthy=3 other=2 | vm341=running | services-not-active=0 | 9201=running 9202=running
+220s 9201 n=26 healthy=14 other=11 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running
+239s 9201 n=26 healthy=15 other=10 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running
+256s 9201 n=26 healthy=20 other=5 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running
+275s 9201 n=26 healthy=22 other=3 | 9202 n=6 healthy=4 other=1 | vm341=running | services-not-active=0 | 9201=running 9202=running
+293s 9201 n=26 healthy=24 other=1 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running
+309s 9201 n=26 healthy=24 other=1 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running
+324s 9201 n=26 healthy=25 other=0 | 9202 n=6 healthy=5 other=0 | vm341=running | services-not-active=0 | 9201=running 9202=running
WATCH RESULT=ok kernel=7.0.14-28-pve want=7.0.14-28-pve ping-down=+139s ssh=+176s all-healthy=+324s
@@ -0,0 +1,91 @@
### AFTER — 2026-11-10T14:45:36Z (read on demo-hp, read only)
uname -r: 7.0.14-28-pve
boot_id: bed106f7-9e80-4eb1-ace3-14a26da7f049
up since (UTC): 2026-11-10T14:42:50Z
cmdline: BOOT_IMAGE=/boot/vmlinuz-7.0.14-28-pve root=/dev/mapper/pve-root ro quiet
== one-shot flag (grub-editenv /boot/efi/EFI/felhom/oneshot.env list)
felhom_next=
== pinned default file
6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg
GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
== grub.cfg
805cc2ba2da6054d676601590412b31ba36061aa7558e4f1391b6874710e16a3 /boot/grub/grub.cfg
22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
== kernel step state
607a2ce7ce3b6ca1c644695587f66ca280227d98670d6e9fb73d1b5f0080b8d0 /var/lib/felhom-kernel/state.json
{'phase': 'good', 'from': '7.0.14-26-pve', 'to': '7.0.14-28-pve', 'step_id': 'ring0-20261110T014802Z', 'result_at': '2026-11-10T01:56:47Z', 'updated_at': '2026-11-10T01:56:47Z'}
== kernels installed
/boot/vmlinuz-7.0.14-26-pve
/boot/vmlinuz-7.0.14-28-pve
== hand pin files
ls: cannot access '/etc/kernel/proxmox-boot-pin': No such file or directory
ls: cannot access '/etc/kernel/next-boot-pin': No such file or directory
== crash guard
{'armed': True, 'tripped': False, 'boot_id': 'bed106f7-9e80-4eb1-ace3-14a26da7f049', 'last_boot_at': '2026-11-10T14:42:49Z', 'last_boot_unclean': False, 'unclean_boots': [], 'unclean_boots_in_window': 0, 'unclean_boots_24h': 0, 'updated_at': '2026-11-10T14:42:59Z'}
== host services
felhom-agent: active
pve-cluster: active
pvedaemon: active
pveproxy: active
pvestatd: active
failed units: 0
== tunnel
handshake age s: 30
== guests
VMID Status Lock Name
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
341 night1004-tester1 running 8192 200.00 1950
== running jobs (vzdump, restore, apt, dpkg, backup client, wrapper)
(end)
== containers in 9201
adventurelog-frontend Up 2 minutes (healthy)
adventurelog-postgres Up 2 minutes (healthy)
adventurelog Up 2 minutes (healthy)
bentopdf Up 2 minutes (healthy)
bookstack-db Up 2 minutes (healthy)
bookstack Up 2 minutes (healthy)
calibre-web Up 2 minutes (healthy)
cloudflared Up 2 minutes (healthy)
docmost-postgres Up 2 minutes (healthy)
docmost-redis Up 2 minutes (healthy)
docmost Up 2 minutes (healthy)
felhom-controller Up 2 minutes (healthy)
filebrowser Up 2 minutes (healthy)
home-assistant Up 2 minutes (healthy)
kimai-db Up 2 minutes (healthy)
kimai Up 2 minutes (healthy)
opengist Up 2 minutes (healthy)
paperless-postgres Up About a minute (healthy)
paperless-redis Up 2 minutes (healthy)
paperless-webserver Up About a minute (healthy)
plex Up About a minute (healthy)
privatebin Up 2 minutes (healthy)
romm-db Up About a minute (healthy)
romm-redis Up About a minute (healthy)
romm Up About a minute (healthy)
traefik Up 2 minutes
count: 26 running; not healthy: 1
== containers in 9202
felhom-controller Up 2 minutes (healthy)
filebrowser Up 2 minutes (healthy)
paperless-postgres Up 2 minutes (healthy)
paperless-redis Up 2 minutes (healthy)
paperless-webserver Up 2 minutes (healthy)
traefik Up 2 minutes
count: 6 running; not healthy: 1
== Tester 1 VM 341
status: running
guest agent: no answer (rc 255)
== disk
/dev/mapper/pve-root 39G 17G 21G 45% /
== agent log, kernel lines since boot
Nov 10 15:43:05 demo-hp sudo[1323]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261110T144304Z-kernel-kernel-boot.json
(end)
2a04:4e42:200::644 debian.map.fastlydns.net deb.debian.org
felhom-controller Up 2 minutes (healthy)
cloudflared Up 2 minutes (healthy)
2a04:4e42::644 debian.map.fastlydns.net deb.debian.org
felhom-controller Up 2 minutes (healthy)
@@ -0,0 +1,24 @@
### HUB AFTER — 2026-11-10T14:46:02Z
== hub pod kernel lines for demo-hp, last 14 h (3 before the test; the pod log prints CET)
3
2026/11/10 02:50:50 [INFO] osupdates: demo-hp-bb76ea reported kernel run 20261110T014802Z: ring=0 mode=apply outcome=staged healthy=true upgraded=2 pending=0 no
2026/11/10 02:54:08 [INFO] osupdates: demo-hp-bb76ea reported kernel run boot-20261110T015407Z: ring=0 mode=kernel-boot outcome=judging healthy=false upgraded=0
2026/11/10 02:56:47 [INFO] osupdates: demo-hp-bb76ea reported kernel run boot-20261110T015407Z: ring=0 mode=kernel-good outcome=applied healthy=true upgraded=0
== positive control: the same log holds lines for demo-hp since the test began (15:29 CET)
6
== every hub log line since the test began that names an event, an alarm or a mail
2026/11/10 15:33:40 [INFO] Event from demo-hp: controller_started (info) — Controller elindult (0.351.0)
2026/11/10 15:34:18 [INFO] Notification prefs push for tester-1: household cleared its mail address — deleted
2026/11/10 15:34:18 [INFO] Notification preferences updated for tester-1: address set=false, household=true, events=[]
2026/11/10 15:34:23 [INFO] Event from tester-1: controller_started (info) — Controller elindult (0.351.0)
2026/11/10 15:43:36 [INFO] Event from demo-hp: controller_started (info) — Controller elindult (0.351.0)
2026/11/10 15:44:12 [INFO] Notification prefs push for tester-1: household cleared its mail address — deleted
2026/11/10 15:44:12 [INFO] Notification preferences updated for tester-1: address set=false, household=true, events=[]
2026/11/10 15:44:17 [INFO] Event from tester-1: controller_started (info) — Controller elindult (0.351.0)
== 'household cleared its mail address' lines for tester-1, last 30 h (is it the same at every controller start?)
2026/11/09 13:09:54 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete
2026/11/09 13:14:39 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete
2026/11/10 02:55:11 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete
2026/11/10 13:58:18 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete
2026/11/10 15:34:18 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete
2026/11/10 15:44:12 [INFO] Notification prefs push for tester-1: household cleared its mail address — delete
+1 -1
View File
@@ -281,7 +281,7 @@ stopping line that lies.
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC |
| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC |
| **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC |
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. **2026-11-10 — part (2) FIXED IN SOURCE, NOT RELEASED** (felhom-agent `776025c8`): after an update that ended well the agent removes every staged binary but the running version's; nothing is removed while an update is pending. `TestPrune_*` (6 tests, one runs the wrapper's real commit and rollback verbs), red seen three times. A box cleans its folder at the first good update after the release that carries it. Part (4), the `.bak-*` copies, is not reached (root-owned, made by a person). Parts (1), (3), (5) stay open. **2026-11-10 14:46 UTC — the kept old kernel boots:** on demo-hp the one-shot flag booted 7.0.14-26-pve to a healthy host in 313 s, and an orderly restart brought 7.0.14-28-pve back in 324 s; the pinned default, `grub.cfg` and the step's state file did not change, and the hub got no kernel report and no alarm (`documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). Still open for the kernel part: a second box; the step built as a program. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
| **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) **2026-11-10 (afternoon) — the runbook is written:** `documentation/runbooks/disk-full.md`, five parts, each hand step marked PROVEN (date and evidence; all on demo-hp) or NOT PROVEN. Still open: one rehearsal of the whole runbook on demo-hp; the NOT PROVEN steps (copies on a former target, template archives, the `.bak-*` binaries, the guest root disk); what fails when the EFI partition or a temp folder is full. | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC |
| **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. **2026-11-10 (afternoon) — the section is written:** `16-storage-physical-identity.md` §14 (45 places, the list „no limit, by ruling”, the budget, the open design points); `PROMPT-TEMPLATE.md` §N.5 keeps it true. The budget from measured numbers: the root needs about 14 GB when the whole-box copies go to a backup drive, and 28–54 GB at keep 3 when they do not. **D75 (for the operator):** the smallest system disk Felhom accepts. CC's pick: 128 GB only with a backup drive, 200 GB without one. The other option, 128 GB always, lets a box with many apps and no backup drive stop its local copies. With no ruling the installer accepts any disk. Open after the ruling: the installer's refusal. **2026-11-10 15:19 — RULED (D75 = one number, 200 GB, with or without a backup drive; `09` §3 decision 295).** This replaces CC's two-step pick. **Open, not ruled:** what the installer does below 200 GB — refuse, or warn. demo-hp (128 GB) stays as a test box. `16` §14.3 carries the ruling. | — | Design first: what the installer does below 200 GB (refuse, or warn) — a later session. Then the installer. | CC |
| **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |
+1 -1
View File
@@ -43,7 +43,7 @@ hub alarms at 90 % and 95 %.
| What | Step | Mark |
|---|---|---|
| Old kernels | `ssh <host> python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py`. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window | **PROVEN 2026-11-10**: 8 → 2 kernels, 6.4 GB, 50 s, one restart (`audits/kernel-cleanup-spike-2026-11-10/README.md`). **NOT PROVEN:** on a second box; a boot of the kept older kernel |
| Old kernels | `ssh <host> python3 - < documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py`. It changes nothing. It prints the plan and "the one command" with exact package names; exit 3 = REFUSED, then stop. Run that one command. Restart the host in an agreed window | **PROVEN 2026-11-10**: 8 → 2 kernels, 6.4 GB, 50 s, one restart (`audits/kernel-cleanup-spike-2026-11-10/README.md`). The kept older kernel then booted the box to healthy and the box went back (`audits/kernel-cleanup-spike-2026-11-10/evidence/fallback-boot/`). **NOT PROVEN:** on a second box |
| The host journal | Save it first, then `journalctl --vacuum-size=500M` | **PROVEN 2026-11-07**: 3.8 GB → 468 MB (`audits/demo-check-2026-11-07/README.md`; the record is off-git, `/mnt/5_hdd/felhom.eu/evidence/demo-hp-space-2026-11-07/cleanup.txt`). It grows back (R-1367) |
| An installer ISO that a newer one replaced; a test archive of a guest that is gone | `pvesm free <volid>`, after `pct list` and `qm list` show that no guest uses it, and after its checksum matches a copy elsewhere | **PROVEN 2026-10-23** (three ISOs, `audits/day-2026-10-23/09-demohp-space.txt`), **2026-11-07** (one archive of a removed guest), **2026-11-09** (one ISO). Test boxes only |
| Whole-box copies on a former target (H4) | — | **NOT PROVEN.** Needs the operator's word (R-1087) |