Files
felhom.eu/documentation/audits/kernel-cleanup-spike-2026-11-10
admin 10ea4c2cbe
gates / gates (push) Successful in 2m1s
Part B: the old-kernel clean-up rule, proven by hand on demo-hp (spike; nothing built)
One run on the demo-hp host (Tier 0), decision 286. The rule removed six of
eight kernel image packages and nothing else: 6.4 GB gained on the system
disk, every boot setting unchanged, one orderly reboot back on the same
kernel, all containers healthy 311 s after the command, the crash guard saw
a clean stop, a second run removed nothing. Five refusals shown by
simulation only.

- documentation/audits/kernel-cleanup-spike-2026-11-10/: README (one page),
  evidence 00-15, the rule as a read-only plan script
- 11-os-updates.md §5.11.1: the design of the step (DESIGN, not built)
- R-1368: a dated pointer line

felhom-os-apply, the config bundle, the agent and rule R4 are unchanged.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-10 14:13:41 +01:00
..

Old kernels: the clean-up rule, proven by hand on demo-hp — 2026-11-10

A spike. One person-driven run on the demo-hp host (Tier 0). Nothing was built: felhom-os-apply, the config bundle and the agent are unchanged, and rule R4 is unchanged. The ruling that asked for this proof is 09 §3 decision 286; the design it feeds is 11-os-updates.md §5.11.1; the register row is R-1368. Files: evidence/ (numbered, in the order of the run), tools/kernel-cleanup-rule.py (the rule as a read-only plan).

The answer

Yes. The rule removed six old kernels and nothing else, the box kept every boot setting, and it came back from one orderly reboot on the same kernel. A second run of the rule removed nothing.

Why the old kernels were still there (evidence/01, 02)

  1. The kernel step installs each kernel image by its name (AG/configs/felhom-os-apply, the install line of the kernel step). apt marks a package that is named in an install as "manual". apt's own clean-up (autoremove) removes only packages marked "automatic". Seven of the eight kernels were manual; the simulation of autoremove removed one.
  2. Nothing runs autoremove on a box in any case, and an update plan that removes a package is refused (rule R4).

So the brief's guess is true: the step names its packages, and that keeps them. It is one of two causes.

The rule, as proven

Keep the running kernel · the last kernel that booted well before it (the step's from; if no step record, the next lower one) · the newest installed kernel · every kernel the kernel lane still names: the step's to and default_before, the default in grub.cfg, the pinned default file, the one-shot flag, an operator's boot pin
Remove every other installed kernel image package, by its exact name, with one apt-get purge
Refuse the whole clean-up P1 a kernel step is in a phase that has not ended · P2 the one-shot flag names a kernel · P3 the pinned default and grub.cfg do not name the same installed kernel · K1 a keep-kernel is in the removal set · N1 a name that is not an installed kernel image · S1 apt's simulation fails, or would install anything · S2 the simulation would remove anything that is not in the removal set

On demo-hp: keep 7.0.14-28-pve (running, newest, default) and 7.0.14-26-pve (the last good one); remove 7.0.2-6 and 7.0.14-20, -22, -23, -24, -25 (evidence/03).

Refusals, by simulation only — nothing was removed in these runs (evidence/04, each ended REFUSED, exit 3):

Control Input Refused by
A the running kernel in the removal set K1, S1
B the last good kernel in the removal set K1
C a kernel step in phase judging P1
D the meta-package proxmox-kernel-7.0 N1, S2 (apt would also remove proxmox-ve)
E an old kernel and pve-firmware N1, S2

The brief asked for two controls: the running kernel and the pinned default. On this box they are the same kernel (control A). B to E were added. Control A shows why the rule cannot lean on apt: apt does not refuse to remove the running signed kernel — it plans to put the unsigned image in its place. Only the rule's own check stops it.

The run (evidence/05 to 15)

What Before After Proof
Kernel image packages 8 2 05, 08
System disk /, used 22,906 MB (62%) 16,495 MB (45%) — 6,411 MB gained 05, 08
/lib/modules 7,772 MB 1,945 MB 05, 08
/boot 795 MB 217 MB 05, 08
Default in grub.cfg, pinned default file, both one-shot generators the 28 entry; three checksums the same entry; the same three checksums 05, 08
Kernel step state, one-shot flag good, 26 → 28; empty the same 08, 13
dpkg --audit; simulated autoremove — empty; 0 to remove 08

The removal took 50 s and ended with code 0 (07). Every kernel file that grub.cfg names exists (08).

The reboot (09 to 13; sent 12:54:26 UTC with systemctl reboot):

Moment Seconds after the command
The old boot ended (journal stopped) 129
The new boot started 153
The host answered ping again; the hub showed a new host report 172 at most
Every container healthy in guest 9201 (26) and 9202 (6) 311 (read every 19 s)

After it: kernel 7.0.14-28-pve, a new boot id, system state running, the agent and the Proxmox services active, guests 9201 and 9202 and the Tester 1 VM running, guest 9401 stopped as before. The crash guard saw a clean stop: last_boot_unclean: false, no unclean boot in its list (13). The second run of the rule: NOTHING TO REMOVE (14). For comparison, the reboot of 2026-10-07 took 272 s to all healthy. The hub agrees (its own page, 16 minutes after the boot, 15): ONLINE, tunnel connected, system disk 58% before and 42% after.

Choices made in the run

  • purge, not remove: no /lib/modules/<version> folder and no half-removed package state stays.
  • The removal used the wrapper's own apt settings (no prompt, keep config files).
  • felhom-os-apply --kernel-status needs a plan file in the agent's folder. The run read its inputs directly (the state file, the flag, grub.cfg).

Not measured

  • A box whose running kernel is not its newest one (after a fall-back or a self-revert). The rule keeps both; no such box was tried.
  • A real kernel step that has not ended. Control C is a pretended phase.
  • A boot of the kept old kernel (26) after the clean-up. Its files and its menu entries exist and grub.cfg passes the syntax check; it was not booted.
  • The N100 and the Tester 1 VM (8 and 3 kernels). Read-only in this run.
  • The step as a program, at night, next to the agent's own apt runs. A person checked the lock by hand.
  • A box that boots with proxmox-boot-tool (demo-hp does not use it), and a box with out-of-tree kernel modules (demo-hp has none).
  • A power cut in the middle of the removal.
  • apt's own keep-list still holds a pattern for 7.0.14-23, a kernel that is gone. Harmless here; not followed up.