Files
felhom.eu/documentation/audits/kernel-cleanup-spike-2026-11-10/README.md
T
admin 5f408caca8 kernel clean-up spike: the kept old kernel boots demo-hp to healthy, and the box goes back (R-1368)
The one-shot flag booted 7.0.14-26-pve: every health point good after 313 s. An orderly restart
brought 7.0.14-28-pve back: 324 s. The pinned default, grub.cfg and the step's state file did not
change. No kernel report and no alarm at the hub. Evidence: evidence/fallback-boot/.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-11-10 15:54:40 +01:00

8.0 KiB
Raw Blame History

Old kernels: the clean-up rule, proven by hand on demo-hp — 2026-11-10

A spike. One person-driven run on the demo-hp host (Tier 0). Nothing was built: felhom-os-apply, the config bundle and the agent are unchanged, and rule R4 is unchanged. The ruling that asked for this proof is 09 §3 decision 286; the design it feeds is 11-os-updates.md §5.11.1; the register row is R-1368. Files: evidence/ (numbered, in the order of the run), tools/kernel-cleanup-rule.py (the rule as a read-only plan).

The answer

Yes. The rule removed six old kernels and nothing else, the box kept every boot setting, and it came back from one orderly reboot on the same kernel. A second run of the rule removed nothing.

Why the old kernels were still there (evidence/01, 02)

  1. The kernel step installs each kernel image by its name (AG/configs/felhom-os-apply, the install line of the kernel step). apt marks a package that is named in an install as "manual". apt's own clean-up (autoremove) removes only packages marked "automatic". Seven of the eight kernels were manual; the simulation of autoremove removed one.
  2. Nothing runs autoremove on a box in any case, and an update plan that removes a package is refused (rule R4).

So the brief's guess is true: the step names its packages, and that keeps them. It is one of two causes.

The rule, as proven

Keep the running kernel · the last kernel that booted well before it (the step's from; if no step record, the next lower one) · the newest installed kernel · every kernel the kernel lane still names: the step's to and default_before, the default in grub.cfg, the pinned default file, the one-shot flag, an operator's boot pin
Remove every other installed kernel image package, by its exact name, with one apt-get purge
Refuse the whole clean-up P1 a kernel step is in a phase that has not ended · P2 the one-shot flag names a kernel · P3 the pinned default and grub.cfg do not name the same installed kernel · K1 a keep-kernel is in the removal set · N1 a name that is not an installed kernel image · S1 apt's simulation fails, or would install anything · S2 the simulation would remove anything that is not in the removal set

On demo-hp: keep 7.0.14-28-pve (running, newest, default) and 7.0.14-26-pve (the last good one); remove 7.0.2-6 and 7.0.14-20, -22, -23, -24, -25 (evidence/03).

Refusals, by simulation only — nothing was removed in these runs (evidence/04, each ended REFUSED, exit 3):

Control Input Refused by
A the running kernel in the removal set K1, S1
B the last good kernel in the removal set K1
C a kernel step in phase judging P1
D the meta-package proxmox-kernel-7.0 N1, S2 (apt would also remove proxmox-ve)
E an old kernel and pve-firmware N1, S2

The brief asked for two controls: the running kernel and the pinned default. On this box they are the same kernel (control A). B to E were added. Control A shows why the rule cannot lean on apt: apt does not refuse to remove the running signed kernel — it plans to put the unsigned image in its place. Only the rule's own check stops it.

The run (evidence/05 to 15)

What Before After Proof
Kernel image packages 8 2 05, 08
System disk /, used 22,906 MB (62%) 16,495 MB (45%) — 6,411 MB gained 05, 08
/lib/modules 7,772 MB 1,945 MB 05, 08
/boot 795 MB 217 MB 05, 08
Default in grub.cfg, pinned default file, both one-shot generators the 28 entry; three checksums the same entry; the same three checksums 05, 08
Kernel step state, one-shot flag good, 26 → 28; empty the same 08, 13
dpkg --audit; simulated autoremove — empty; 0 to remove 08

The removal took 50 s and ended with code 0 (07). Every kernel file that grub.cfg names exists (08).

The reboot (09 to 13; sent 12:54:26 UTC with systemctl reboot):

Moment Seconds after the command
The old boot ended (journal stopped) 129
The new boot started 153
The host answered ping again; the hub showed a new host report 172 at most
Every container healthy in guest 9201 (26) and 9202 (6) 311 (read every 19 s)

After it: kernel 7.0.14-28-pve, a new boot id, system state running, the agent and the Proxmox services active, guests 9201 and 9202 and the Tester 1 VM running, guest 9401 stopped as before. The crash guard saw a clean stop: last_boot_unclean: false, no unclean boot in its list (13). The second run of the rule: NOTHING TO REMOVE (14). For comparison, the reboot of 2026-10-07 took 272 s to all healthy. The hub agrees (its own page, 16 minutes after the boot, 15): ONLINE, tunnel connected, system disk 58% before and 42% after.

Choices made in the run

  • purge, not remove: no /lib/modules/<version> folder and no half-removed package state stays.
  • The removal used the wrapper's own apt settings (no prompt, keep config files).
  • felhom-os-apply --kernel-status needs a plan file in the agent's folder. The run read its inputs directly (the state file, the flag, grub.cfg).

Measured later the same day: the kept old kernel boots (evidence/fallback-boot/)

Yes. The kept old kernel (7.0.14-26-pve) booted demo-hp to healthy, and the box went back to its newest kernel. A second run, 2026-11-10 14:29–14:46 UTC, two orderly restarts.

Old kernel (26) Back on the newest (28)
Restart sent (UTC) 14:30:24 14:40:09
SSH answered after 166 s 176 s
Every health point good after 313 s 324 s
  • Method. The product's own boot-side switch, set by hand: grub-editenv /boot/efi/EFI/felhom/oneshot.env set felhom_next=7.0.14-26-pve, then systemctl reboot. GRUB cleared the flag and booted 26 once, with the panic parameters of a trial boot (softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10). The second restart had no flag, so the box booted its default, 28. No wrapper verb ran; nothing was built or changed in the product.
  • Healthy means the host health rule of 11-os-updates.md §8.2, read point by point: the agent and the four Proxmox services active, both guests running, Docker answering, both guests resolving deb.debian.org, the controllers healthy, every container back, the tunnel connected. The limit was 15 minutes.
  • Nothing was left behind. The pinned default file, grub.cfg and the kernel-step state file have the same sha256 before, on the old kernel and after. The step state stayed good. The crash guard saw two clean stops.
  • The hub. No kernel-step report came from the test (3 lines for this host before and after, all from the morning's step). No alarm line. Only the "controller started" notes of the two restarts.
  • A different channel. On the old kernel both public dashboards answered through Cloudflare (302). The hub's row for the host said "unknown (no running customer guest)" then: it was the agent's first report after the boot, and no newer report came in the six minutes the box stayed on 26.
  • Not covered by this. A step where the product itself chooses the old kernel (a self-revert after a bad trial boot). Here a person set the flag.

Not measured

  • A box whose running kernel is not its newest one (after a fall-back or a self-revert). The rule keeps both; no such box was tried.
  • A real kernel step that has not ended. Control C is a pretended phase.
  • The N100 and the Tester 1 VM (8 and 3 kernels). Read-only in this run.
  • The step as a program, at night, next to the agent's own apt runs. A person checked the lock by hand.
  • A box that boots with proxmox-boot-tool (demo-hp does not use it), and a box with out-of-tree kernel modules (demo-hp has none).
  • A power cut in the middle of the removal.
  • apt's own keep-list still holds a pattern for 7.0.14-23, a kernel that is gone. Harmless here; not followed up.