diff --git a/documentation/architecture/11-os-updates.md b/documentation/architecture/11-os-updates.md index b867f953..0a703a76 100644 --- a/documentation/architecture/11-os-updates.md +++ b/documentation/architecture/11-os-updates.md @@ -600,6 +600,46 @@ another kernel), and the kernel step (the newest result, the kernel due, whether 7.0.14-20), so a soft lockup turning into a panic is recorded as unmeasured (no tool was built). A true dead freeze needs a person (accepted, decision 172). Evidence and proofs: `audits/kernel-lane-2026-10-07/`. +### 5.11.1 Removing old kernels `[DESIGN — proven by hand on demo-hp 2026-11-10; not built]` + +**The ruling** (`09` §3 decision 286): the box may remove old kernels by itself, as its own step with its own narrow +rule. **Rule R4 does not change**: an update plan that removes a package is still refused. Why a step is needed: the +kernel step installs each image by name, so apt marks it "manual" and apt's own clean-up never takes it; and nothing +runs that clean-up on a box. A ring-0 box got 7 kernels in 37 days, about 1 GB each (R-1368). + +**The rule.** Keep the running kernel; the last kernel that booted well before it (the step's `from`); the newest +installed kernel; and every kernel the lane still names — the step's `to` and `default_before`, the default in +`grub.cfg`, the pinned default file, the one-shot flag, an operator's boot pin. Remove every other installed kernel +image package, by its exact name, in one `apt-get purge`. + +**It refuses the whole clean-up** — it never removes a part of the set — when: + +| | | +|---|---| +| P1 | a kernel step is in a phase that has not ended (`staged`, `oneshot`, `judging`, `reverting`), or in a phase it does not know | +| P2 | the one-shot flag names a kernel | +| P3 | the pinned default and `grub.cfg`'s default do not name the same installed kernel | +| K1 | a keep-kernel is in the removal set, or the running kernel is not an installed package | +| N1 | a name in the removal set is not an installed kernel image package | +| S1 | apt's simulation of exactly that set fails, or would install anything | +| S2 | the simulation would remove anything that is not in the removal set | + +K1 is the rule's own check and must stay its own: asked to remove the running signed kernel, apt does not refuse — it +plans to put the unsigned image in its place (control A of the spike). + +**When it may run.** Only when the newest kernel step ended `good` and the box runs its newest installed kernel — that +is "booted well on its newest kernel" (decision 286). Never in the night of a kernel step: the step's judging, and a +self-revert into `from`, must be over first. In the night's order it belongs where the OS leg is (§5.5): after the +whole-guest backup. It needs no reboot and stops no guest. A box that fell back or reverted keeps all its kernels +until a later step ends `good`. + +**Proven** on demo-hp by hand: 8 kernels → 2, 6.4 GB gained on the system disk, the default, the pinned file and both +generators unchanged, five refusals by simulation, one orderly reboot back onto the same kernel with a clean stop for +the crash guard (§5.9), a second run that removed nothing. **Not proven:** a box whose running kernel is not its +newest; a boot of the kept old kernel after the clean-up; the step as a program at night; a box that boots with +`proxmox-boot-tool`. Evidence, the rule as a read-only script, and the full "not measured" list: +`audits/kernel-cleanup-spike-2026-11-10/`. + ## 6. Risks and edge cases | # | What can go wrong | What the design does | diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/README.md b/documentation/audits/kernel-cleanup-spike-2026-11-10/README.md new file mode 100644 index 00000000..f0ddae0b --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/README.md @@ -0,0 +1,96 @@ +# Old kernels: the clean-up rule, proven by hand on demo-hp — 2026-11-10 + +A spike. One person-driven run on the demo-hp host (Tier 0). Nothing was built: `felhom-os-apply`, the config bundle +and the agent are unchanged, and rule R4 is unchanged. The ruling that asked for this proof is `09` §3 decision 286; +the design it feeds is `11-os-updates.md` §5.11.1; the register row is R-1368. Files: `evidence/` (numbered, in the +order of the run), `tools/kernel-cleanup-rule.py` (the rule as a read-only plan). + +## The answer + +**Yes. The rule removed six old kernels and nothing else, the box kept every boot setting, and it came back from one +orderly reboot on the same kernel.** A second run of the rule removed nothing. + +## Why the old kernels were still there (`evidence/01`, `02`) + +1. The kernel step installs each kernel image **by its name** (`AG/configs/felhom-os-apply`, the `install` line of the + kernel step). apt marks a package that is named in an install as "manual". apt's own clean-up (`autoremove`) removes + only packages marked "automatic". Seven of the eight kernels were manual; the simulation of `autoremove` removed one. +2. Nothing runs `autoremove` on a box in any case, and an update plan that removes a package is refused (rule R4). + +So the brief's guess is true: the step names its packages, and that keeps them. It is one of two causes. + +## The rule, as proven + +| | | +|---|---| +| **Keep** | the running kernel · the last kernel that booted well before it (the step's `from`; if no step record, the next lower one) · the newest installed kernel · every kernel the kernel lane still names: the step's `to` and `default_before`, the default in `grub.cfg`, the pinned default file, the one-shot flag, an operator's boot pin | +| **Remove** | every other installed kernel image package, by its exact name, with one `apt-get purge` | +| **Refuse the whole clean-up** | `P1` a kernel step is in a phase that has not ended · `P2` the one-shot flag names a kernel · `P3` the pinned default and `grub.cfg` do not name the same installed kernel · `K1` a keep-kernel is in the removal set · `N1` a name that is not an installed kernel image · `S1` apt's simulation fails, or would install anything · `S2` the simulation would remove anything that is not in the removal set | + +On demo-hp: keep `7.0.14-28-pve` (running, newest, default) and `7.0.14-26-pve` (the last good one); remove 7.0.2-6 and +7.0.14-20, -22, -23, -24, -25 (`evidence/03`). + +**Refusals, by simulation only — nothing was removed in these runs** (`evidence/04`, each ended `REFUSED`, exit 3): + +| Control | Input | Refused by | +|---|---|---| +| A | the running kernel in the removal set | `K1`, `S1` | +| B | the last good kernel in the removal set | `K1` | +| C | a kernel step in phase `judging` | `P1` | +| D | the meta-package `proxmox-kernel-7.0` | `N1`, `S2` (apt would also remove `proxmox-ve`) | +| E | an old kernel and `pve-firmware` | `N1`, `S2` | + +The brief asked for two controls: the running kernel and the pinned default. On this box they are the same kernel +(control A). B to E were added. **Control A shows why the rule cannot lean on apt:** apt does not refuse to remove the +running signed kernel — it plans to put the unsigned image in its place. Only the rule's own check stops it. + +## The run (`evidence/05` to `15`) + +| What | Before | After | Proof | +|---|---|---|---| +| Kernel image packages | 8 | 2 | `05`, `08` | +| System disk `/`, used | 22,906 MB (62%) | 16,495 MB (45%) — **6,411 MB gained** | `05`, `08` | +| `/lib/modules` | 7,772 MB | 1,945 MB | `05`, `08` | +| `/boot` | 795 MB | 217 MB | `05`, `08` | +| Default in `grub.cfg`, pinned default file, both one-shot generators | the `28` entry; three checksums | the same entry; the same three checksums | `05`, `08` | +| Kernel step state, one-shot flag | `good`, 26 → 28; empty | the same | `08`, `13` | +| `dpkg --audit`; simulated `autoremove` | — | empty; 0 to remove | `08` | + +The removal took 50 s and ended with code 0 (`07`). Every kernel file that `grub.cfg` names exists (`08`). + +**The reboot** (`09` to `13`; sent 12:54:26 UTC with `systemctl reboot`): + +| Moment | Seconds after the command | +|---|---| +| The old boot ended (journal stopped) | 129 | +| The new boot started | 153 | +| The host answered ping again; the hub showed a new host report | 172 at most | +| Every container healthy in guest 9201 (26) and 9202 (6) | 311 (read every 19 s) | + +After it: kernel `7.0.14-28-pve`, a new boot id, system state `running`, the agent and the Proxmox services active, +guests 9201 and 9202 and the Tester 1 VM running, guest 9401 stopped as before. **The crash guard saw a clean stop:** +`last_boot_unclean: false`, no unclean boot in its list (`13`). The second run of the rule: `NOTHING TO REMOVE` (`14`). +For comparison, the reboot of 2026-10-07 took 272 s to all healthy. +**The hub agrees** (its own page, 16 minutes after the boot, `15`): `ONLINE`, tunnel connected, system disk 58% before +and 42% after. + +## Choices made in the run + +- `purge`, not `remove`: no `/lib/modules/` folder and no half-removed package state stays. +- The removal used the wrapper's own apt settings (no prompt, keep config files). +- `felhom-os-apply --kernel-status` needs a plan file in the agent's folder. The run read its inputs directly (the + state file, the flag, `grub.cfg`). + +## Not measured + +- A box whose running kernel is **not** its newest one (after a fall-back or a self-revert). The rule keeps both; no + such box was tried. +- A real kernel step that has not ended. Control C is a pretended phase. +- **A boot of the kept old kernel (26) after the clean-up.** Its files and its menu entries exist and `grub.cfg` + passes the syntax check; it was not booted. +- The N100 and the Tester 1 VM (8 and 3 kernels). Read-only in this run. +- The step as a program, at night, next to the agent's own apt runs. A person checked the lock by hand. +- A box that boots with `proxmox-boot-tool` (demo-hp does not use it), and a box with out-of-tree kernel modules + (demo-hp has none). +- A power cut in the middle of the removal. +- apt's own keep-list still holds a pattern for 7.0.14-23, a kernel that is gone. Harmless here; not followed up. diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/00-preconditions.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/00-preconditions.txt new file mode 100644 index 00000000..fc0d8682 --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/00-preconditions.txt @@ -0,0 +1,26 @@ +== when +2026-11-10T12:43:38Z +demo-hp + 13:43:38 up 10:49, 1 user, load average: 0.95, 1.16, 1.62 +== running +7.0.14-28-pve +== guests +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness + VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID + 341 night1004-tester1 running 8192 200.00 1861 +== busy? +none of: vzdump pbs-restore proxmox-backup-client apt-get dpkg felhom-os-apply restic +running PVE tasks: 0 [] +== df +Filesystem Size Used Avail Use% Mounted on +/dev/mapper/pve-root 39G 23G 15G 62% / +/dev/sda2 1022M 9.1M 1013M 1% /boot/efi +== agent +active +active +active +active +felhom-agent 0.172.1 diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/01-why-still-there.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/01-why-still-there.txt new file mode 100644 index 00000000..664b09aa --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/01-why-still-there.txt @@ -0,0 +1,331 @@ +== running kernel +7.0.14-28-pve +== kernel packages (dpkg) +un linux-image-7.0.14-20-pve-amd64 +un linux-image-7.0.14-22-pve-amd64 +un linux-image-7.0.14-23-pve-amd64 +un linux-image-7.0.14-24-pve-amd64 +un linux-image-7.0.14-25-pve-amd64 +un linux-image-7.0.14-26-pve-amd64 +un linux-image-7.0.14-28-pve-amd64 +un linux-image-7.0.2-6-pve-amd64 +un linux-image-amd64 +un linux-image-generic +ii proxmox-default-kernel 2.1.0 10 +ii proxmox-kernel-7.0 7.0.14-28 37 +un proxmox-kernel-7.0.14-20-pve +ii proxmox-kernel-7.0.14-20-pve-signed 7.0.14-20 1006525 +un proxmox-kernel-7.0.14-22-pve +ii proxmox-kernel-7.0.14-22-pve-signed 7.0.14-22 1006522 +un proxmox-kernel-7.0.14-23-pve +ii proxmox-kernel-7.0.14-23-pve-signed 7.0.14-23 1006525 +un proxmox-kernel-7.0.14-24-pve +ii proxmox-kernel-7.0.14-24-pve-signed 7.0.14-24 1006528 +un proxmox-kernel-7.0.14-25-pve +ii proxmox-kernel-7.0.14-25-pve-signed 7.0.14-25 1007819 +un proxmox-kernel-7.0.14-26-pve +ii proxmox-kernel-7.0.14-26-pve-signed 7.0.14-26 1007815 +un proxmox-kernel-7.0.14-28-pve +ii proxmox-kernel-7.0.14-28-pve-signed 7.0.14-28 1007817 +un proxmox-kernel-7.0.2-6-pve +ii proxmox-kernel-7.0.2-6-pve-signed 7.0.2-6 1005219 +ii proxmox-kernel-helper 9.1.0+fde2 109 +ii pve-firmware 3.18-3 543876 +un pve-kernel-7.0 +un pve-kernel-helper +== manual marks +proxmox-default-kernel +proxmox-kernel-7.0 +proxmox-kernel-7.0.14-22-pve-signed +proxmox-kernel-7.0.14-23-pve-signed +proxmox-kernel-7.0.14-24-pve-signed +proxmox-kernel-7.0.14-25-pve-signed +proxmox-kernel-7.0.14-26-pve-signed +proxmox-kernel-7.0.14-28-pve-signed +proxmox-kernel-7.0.2-6-pve-signed +proxmox-kernel-helper +== auto marks +proxmox-kernel-7.0.14-20-pve-signed +== holds +== simulated autoremove +Reading package lists... +Building dependency tree... +Reading state information... +The following packages will be REMOVED: + proxmox-kernel-7.0.14-20-pve-signed +0 upgraded, 0 newly installed, 1 to remove and 16 not upgraded. +Remv proxmox-kernel-7.0.14-20-pve-signed [7.0.14-20] +== apt never-remove list +APT::NeverAutoRemove ""; +APT::NeverAutoRemove:: "^firmware-linux.*"; +APT::NeverAutoRemove:: "^linux-firmware$"; +APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$"; +APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$"; +APT::NeverAutoRemove:: "^pve-kernel-7\.0$"; +APT::NeverAutoRemove:: "^proxmox-kernel-7\.0$"; +APT::NeverAutoRemove:: "^proxmox-kernel-7\.0-signed$"; +APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-26-pve$"; +APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve$"; +APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve-signed$"; +APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-28-pve$"; +APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve$"; +APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve-signed$"; +APT::VersionedKernelPackages ""; +APT::VersionedKernelPackages:: "linux-.*"; +APT::VersionedKernelPackages:: "kfreebsd-.*"; +APT::VersionedKernelPackages:: "gnumach-.*"; +APT::VersionedKernelPackages:: ".*-modules"; +APT::VersionedKernelPackages:: ".*-kernel"; +== files that set it +-rw-r--r-- 1 root root 399 Jun 24 2025 01autoremove +total 32 +drwxr-xr-x 2 root root 4096 Aug 21 17:42 . +drwxr-xr-x 5 root root 4096 Oct 4 09:47 .. +-rwxr-xr-x 1 root root 951 May 13 2025 initramfs-tools +-rwxr-xr-x 1 root root 1023 May 5 2026 proxmox-auto-removal +-rwxr-xr-x 1 root root 11627 May 5 2026 zz-proxmox-boot +-rwxr-xr-x 1 root root 646 Mar 16 2023 zz-update-grub +== pinned default +# felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily +GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43" +== grub.cfg default line +17: set default="${next_entry}" +22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43" +110: if [ "${felhom_boot}" = "7.0.14-20-pve" ]; then set default="felhom-oneshot-7.0.14-20-pve"; fi +111: if [ "${felhom_boot}" = "7.0.14-22-pve" ]; then set default="felhom-oneshot-7.0.14-22-pve"; fi +112: if [ "${felhom_boot}" = "7.0.14-23-pve" ]; then set default="felhom-oneshot-7.0.14-23-pve"; fi +113: if [ "${felhom_boot}" = "7.0.14-24-pve" ]; then set default="felhom-oneshot-7.0.14-24-pve"; fi +114: if [ "${felhom_boot}" = "7.0.14-25-pve" ]; then set default="felhom-oneshot-7.0.14-25-pve"; fi +115: if [ "${felhom_boot}" = "7.0.14-26-pve" ]; then set default="felhom-oneshot-7.0.14-26-pve"; fi +116: if [ "${felhom_boot}" = "7.0.14-28-pve" ]; then set default="felhom-oneshot-7.0.14-28-pve"; fi +117: if [ "${felhom_boot}" = "7.0.2-6-pve" ]; then set default="felhom-oneshot-7.0.2-6-pve"; fi +== one-shot flag +felhom_next= +rc=0 +== kernel step state +{ + "authority": "ring0", + "default_before": "7.0.14-26-pve", + "from": "7.0.14-26-pve", + "health_before": { + "guest": { + "containers": { + "adventurelog": { + "health": "healthy", + "id": "50af393c36a37d01a10847456d391d70572c121045e28626f220f83cec1cefcc", + "state": "running" + }, + "adventurelog-frontend": { + "health": "healthy", + "id": "abd6155bdad527c65b82f75bc0dd0f98f9690ec81587c8cd07cea6c9c019cabe", + "state": "running" + }, + "adventurelog-postgres": { + "health": "healthy", + "id": "841257c7abb7b545567aa8fa539cdcd421c94ce4edf97a0dea5e45831ea399f4", + "state": "running" + }, + "bentopdf": { + "health": "healthy", + "id": "9db9aba521daf9f19352a64511f4f84872ade957b643c0db40dd510837143cba", + "state": "running" + }, + "bookstack": { + "health": "healthy", + "id": "c83bbf5562cb68ee23d71ece2362d8be482ac4d7d59ad9bdfa5493f0b53c9687", + "state": "running" + }, + "bookstack-db": { + "health": "healthy", + "id": "ea5212ba45188144fb2b2aca5bd53007e01e15a7306a411e651e5b64aea782d8", + "state": "running" + }, + "calibre-web": { + "health": "healthy", + "id": "ac9e6b2302c502a9207fc9e7cfdc0a6999e7304730957263212a1300ed525d10", + "state": "running" + }, + "cloudflared": { + "health": "healthy", + "id": "549f296e6ef9f6594f5e8789ecb86579f0c6ee45091b38bafdca69b50377d27a", + "state": "running" + }, + "docmost": { + "health": "healthy", + "id": "b7ce8b91c0f44c389f4a9153778c910c226c0a4f1118030d6da84d4860c33da6", + "state": "running" + }, + "docmost-postgres": { + "health": "healthy", + "id": "d16587ff24479fa15fa24ce56c23d47005623483315497460140fc563a3c1e8b", + "state": "running" + }, + "docmost-redis": { + "health": "healthy", + "id": "be025075e63097a7df7fd37c56fd4034942742beff6f2ef34d83df003a36c590", + "state": "running" + }, + "felhom-controller": { + "health": "healthy", + "id": "1af906eb4abc5270c0a291b14ac1638be79c5a3fe9c47dce973a1c42d1678b0d", + "state": "running" + }, + "filebrowser": { + "health": "healthy", + "id": "f6446630708007c3b2f97a8d62043193c6516a7a62daf31037e42989a51c9393", + "state": "running" + }, + "home-assistant": { + "health": "healthy", + "id": "588d1a667f65af137effb7acbc28a220d63f4f5d417039aca9ef9df9ad07a923", + "state": "running" + }, + "kimai": { + "health": "healthy", + "id": "e5443c7db71d859c4ae97c551566e3a278511282b192abe33746baff88d62a34", + "state": "running" + }, + "kimai-db": { + "health": "healthy", + "id": "90fe1d5c54138fe8fde39fca173f5b735f6fbae23d8c6156eeacb06f716836c2", + "state": "running" + }, + "opengist": { + "health": "healthy", + "id": "0991f49cd94bea7a3f6db7c597f716e8a080de6492528db8f31ee737d783e528", + "state": "running" + }, + "paperless-postgres": { + "health": "healthy", + "id": "e5862350b148d28e5eee065a7ca4d12c2e2d64d7ccc4c40708781dd5255e9296", + "state": "running" + }, + "paperless-redis": { + "health": "healthy", + "id": "cfb5b8b9e711a12027c47b8b7a7f3d2fd74999338fec699372883c19157ddbfa", + "state": "running" + }, + "paperless-webserver": { + "health": "healthy", + "id": "74c1e55159dd9b856af7a0d1a7cdb56680fcefa195913537768f3789221c3c29", + "state": "running" + }, + "plex": { + "health": "healthy", + "id": "5091a7dd6259f31e28552895f9f714d1f7cd0938b0e525d6a03133355c50d83c", + "state": "running" + }, + "privatebin": { + "health": "healthy", + "id": "5e6c930258cb8f4e76fa0fd2ab80bad7b2a62c9cd93882edece8e096e39a29f4", + "state": "running" + }, + "romm": { + "health": "healthy", + "id": "8368d5dbb4be09989cd5101a50335ee4e80d7e139ad6a70c30d7fedd933b16d0", + "state": "running" + }, + "romm-db": { + "health": "healthy", + "id": "6bbe9a4d9bb95606052f2bcb206df17b78b3058ee68a4af55450b4e3d1cecc1c", + "state": "running" + }, + "romm-redis": { + "health": "healthy", + "id": "519aa50394d17f06467be060d732f060f3bb23e08a19aa8c6170c7198c3c3c78", + "state": "running" + }, + "traefik": { + "health": "none", + "id": "fc0bfe5b53c6a7711d5d742b8ba9c4b3fbdbf96da7270311ffcf4e23f28817c8", + "state": "running" + } + }, + "controller": "healthy", + "controller_docker_ok": true, + "docker_ok": true, + "network_ok": true + }, + "guest_running": true, + "host_services": { + "felhom-agent": "active", + "pve-cluster": "active", + "pvedaemon": "active", + "pveproxy": "active", + "pvestatd": "active" + } + }, + "judging_since": "2026-11-10T01:54:08Z", + "packages": [ + { + "name": "proxmox-kernel-7.0", + "version": "7.0.14-28" + }, + { + "name": "proxmox-kernel-7.0.14-28-pve-signed", + "version": "7.0.14-28" + } + ], + "phase": "good", + "rebooted_at": "2026-11-10T01:50:56Z", + "result_at": "2026-11-10T01:56:47Z", + "self_revert_used": false, + "staged_at": "2026-11-10T01:50:50Z", + "step_id": "ring0-20261110T014802Z", + "to": "7.0.14-28-pve", + "updated_at": "2026-11-10T01:56:47Z", + "vmid": 9201 +} +== one-shot entries in grub.cfg +felhom-oneshot-7.0.14-20-pve +felhom-oneshot-7.0.14-22-pve +felhom-oneshot-7.0.14-23-pve +felhom-oneshot-7.0.14-24-pve +felhom-oneshot-7.0.14-25-pve +felhom-oneshot-7.0.14-26-pve +felhom-oneshot-7.0.14-28-pve +felhom-oneshot-7.0.2-6-pve +== menu entries (kernels) +gnulinux-7.0.14-20-pve-advanced +gnulinux-7.0.14-22-pve-advanced +gnulinux-7.0.14-23-pve-advanced +gnulinux-7.0.14-24-pve-advanced +gnulinux-7.0.14-25-pve-advanced +gnulinux-7.0.14-26-pve-advanced +gnulinux-7.0.14-28-pve-advanced +gnulinux-7.0.2-6-pve-advanced +== /lib/modules +972 /lib/modules/7.0.14-20-pve +972 /lib/modules/7.0.14-22-pve +972 /lib/modules/7.0.14-23-pve +972 /lib/modules/7.0.14-24-pve +973 /lib/modules/7.0.14-25-pve +973 /lib/modules/7.0.14-26-pve +973 /lib/modules/7.0.14-28-pve +970 /lib/modules/7.0.2-6-pve +7772 /lib/modules +== /boot +786 /boot +-rw-r--r-- 1 root root 73094554 Oct 10 04:41 initrd.img-7.0.14-20-pve +-rw-r--r-- 1 root root 73101983 Oct 10 04:42 initrd.img-7.0.14-22-pve +-rw-r--r-- 1 root root 73131609 Oct 11 04:42 initrd.img-7.0.14-23-pve +-rw-r--r-- 1 root root 73115537 Oct 16 04:49 initrd.img-7.0.14-24-pve +-rw-r--r-- 1 root root 73286572 Oct 21 04:45 initrd.img-7.0.14-25-pve +-rw-r--r-- 1 root root 73492957 Oct 29 05:07 initrd.img-7.0.14-26-pve +-rw-r--r-- 1 root root 73547547 Nov 10 02:50 initrd.img-7.0.14-28-pve +-rw-r--r-- 1 root root 77353337 Oct 4 09:32 initrd.img-7.0.2-6-pve +-rw-r--r-- 1 root root 16284776 Sep 24 12:43 vmlinuz-7.0.14-20-pve +-rw-r--r-- 1 root root 16288872 Oct 6 07:41 vmlinuz-7.0.14-22-pve +-rw-r--r-- 1 root root 16288872 Oct 7 22:25 vmlinuz-7.0.14-23-pve +-rw-r--r-- 1 root root 16288872 Oct 9 13:32 vmlinuz-7.0.14-24-pve +-rw-r--r-- 1 root root 16284776 Oct 17 22:42 vmlinuz-7.0.14-25-pve +-rw-r--r-- 1 root root 16280680 Oct 22 20:45 vmlinuz-7.0.14-26-pve +-rw-r--r-- 1 root root 16280680 Nov 2 12:54 vmlinuz-7.0.14-28-pve +-rw-r--r-- 1 root root 16227432 May 20 10:55 vmlinuz-7.0.2-6-pve +== crash guard +active +kernel.panic = 10 +total 12 +drwxr-xr-x 2 root root 4096 Nov 10 13:09 . +drwxr-xr-x 46 root root 4096 Oct 10 04:42 .. +-rw-r--r-- 1 root root 679 Nov 10 13:09 state.json diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/02-history-and-keep-set.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/02-history-and-keep-set.txt new file mode 100644 index 00000000..2f28c6ca --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/02-history-and-keep-set.txt @@ -0,0 +1,47 @@ +== what installed each kernel (apt history: command line + the kernel packages it installed) +2026-10-04 09:42:20 | Commandline: apt-get install -y -q -o Dpkg::Options::=--force-confold proxmox-kernel-7.0 | Install: proxmox-kernel-7.0.14-20-pve-signed:amd64 +2026-10-10 04:41:43 | Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --no-install-recommends proxmox-kernel-7.0=7. | Install: proxmox-kernel-7.0.14-22-pve-signed:amd64 +2026-10-11 04:42:11 | Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --no-install-recommends proxmox-kernel-7.0=7. | Install: proxmox-kernel-7.0.14-23-pve-signed:amd64 +2026-10-16 04:48:58 | Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --no-install-recommends proxmox-kernel-7.0=7. | Install: proxmox-kernel-7.0.14-24-pve-signed:amd64 +2026-10-21 04:45:13 | Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --no-install-recommends proxmox-kernel-7.0=7. | Install: proxmox-kernel-7.0.14-25-pve-signed:amd64 +2026-10-29 05:06:42 | Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --no-install-recommends proxmox-kernel-7.0=7. | Install: proxmox-kernel-7.0.14-26-pve-signed:amd64 +2026-11-10 02:50:13 | Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --no-install-recommends proxmox-kernel-7.0=7. | Install: proxmox-kernel-7.0.14-28-pve-signed:amd64 +== installer-era record +2026-10-04 09:42:20 install proxmox-kernel-7.0.14-20-pve-signed:amd64 7.0.14-20 +2026-08-21 15:41:58 install proxmox-kernel-7.0.2-6-pve-signed:amd64 7.0.2-6 +== boots (last 12) and their kernel +boot 0 2026-11-10T02:53:57+01:00 7.0.14-28-pve +== proxmox-boot-tool +Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace.. +E: /etc/kernel/proxmox-boot-uuids does not exist. +Manually selected kernels: +None. + +Automatically selected kernels: +7.0.14-26-pve +7.0.14-28-pve +== can each be installed back? (apt-cache policy) +proxmox-kernel-7.0.2-6-pve-signed: Installed: 7.0.2-6 Candidate: 7.0.2-6 *** 7.0.2-6 500 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages +proxmox-kernel-7.0.14-20-pve-signed: Installed: 7.0.14-20 Candidate: 7.0.14-20 *** 7.0.14-20 500 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages +proxmox-kernel-7.0.14-22-pve-signed: Installed: 7.0.14-22 Candidate: 7.0.14-22 *** 7.0.14-22 500 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages +proxmox-kernel-7.0.14-23-pve-signed: Installed: 7.0.14-23 Candidate: 7.0.14-23 *** 7.0.14-23 500 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages +proxmox-kernel-7.0.14-24-pve-signed: Installed: 7.0.14-24 Candidate: 7.0.14-24 *** 7.0.14-24 500 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages +proxmox-kernel-7.0.14-25-pve-signed: Installed: 7.0.14-25 Candidate: 7.0.14-25 *** 7.0.14-25 500 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages +proxmox-kernel-7.0.14-26-pve-signed: Installed: 7.0.14-26 Candidate: 7.0.14-26 *** 7.0.14-26 500 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages +proxmox-kernel-7.0.14-28-pve-signed: Installed: 7.0.14-28 Candidate: 7.0.14-28 *** 7.0.14-28 500 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages +== reverse dependencies of an old image +proxmox-kernel-7.0.14-25-pve-signed +Reverse Depends: + |proxmox-kernel-7.0 +proxmox-kernel-7.0 + Depends: pve-firmware + |Depends: proxmox-kernel-7.0.14-28-pve-signed + Depends: proxmox-kernel-7.0.14-28-pve + proxmox-kernel-7.0.14-28-pve-signed +== dkms / extra modules +no dkms +/etc/kernel/postrm.d/: +initramfs-tools +proxmox-auto-removal +zz-proxmox-boot +zz-update-grub diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/03-rule-plan.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/03-rule-plan.txt new file mode 100644 index 00000000..6b979b5b --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/03-rule-plan.txt @@ -0,0 +1,76 @@ +2026-11-10T12:50:17Z +{ + "host": "demo-hp", + "running": "7.0.14-28-pve", + "newest": "7.0.14-28-pve", + "phase": "good", + "step": { + "step_id": "ring0-20261110T014802Z", + "from": "7.0.14-26-pve", + "to": "7.0.14-28-pve", + "default_before": "7.0.14-26-pve", + "result_at": "2026-11-10T01:56:47Z" + }, + "oneshot_flag": "", + "grub_default": "7.0.14-28-pve", + "pinned_default": "7.0.14-28-pve", + "boot_pin": null, + "installed": { + "7.0.2-6-pve": "proxmox-kernel-7.0.2-6-pve-signed", + "7.0.14-20-pve": "proxmox-kernel-7.0.14-20-pve-signed", + "7.0.14-22-pve": "proxmox-kernel-7.0.14-22-pve-signed", + "7.0.14-23-pve": "proxmox-kernel-7.0.14-23-pve-signed", + "7.0.14-24-pve": "proxmox-kernel-7.0.14-24-pve-signed", + "7.0.14-25-pve": "proxmox-kernel-7.0.14-25-pve-signed", + "7.0.14-26-pve": "proxmox-kernel-7.0.14-26-pve-signed", + "7.0.14-28-pve": "proxmox-kernel-7.0.14-28-pve-signed" + }, + "keep": { + "7.0.14-26-pve": [ + "the last kernel that booted well before the running one (state.json `from`)", + "kernel step `from` (a self-revert boots it)", + "kernel step `default_before`" + ], + "7.0.14-28-pve": [ + "running", + "newest installed", + "kernel step `to`", + "GRUB default in grub.cfg", + "pinned default file" + ] + }, + "computed_removal": [ + "proxmox-kernel-7.0.2-6-pve-signed", + "proxmox-kernel-7.0.14-20-pve-signed", + "proxmox-kernel-7.0.14-22-pve-signed", + "proxmox-kernel-7.0.14-23-pve-signed", + "proxmox-kernel-7.0.14-24-pve-signed", + "proxmox-kernel-7.0.14-25-pve-signed" + ], + "removal": [ + "proxmox-kernel-7.0.2-6-pve-signed", + "proxmox-kernel-7.0.14-20-pve-signed", + "proxmox-kernel-7.0.14-22-pve-signed", + "proxmox-kernel-7.0.14-23-pve-signed", + "proxmox-kernel-7.0.14-24-pve-signed", + "proxmox-kernel-7.0.14-25-pve-signed" + ], + "pretend": [], + "simulation": { + "rc": 0, + "removes": [ + "proxmox-kernel-7.0.14-20-pve-signed", + "proxmox-kernel-7.0.14-22-pve-signed", + "proxmox-kernel-7.0.14-23-pve-signed", + "proxmox-kernel-7.0.14-24-pve-signed", + "proxmox-kernel-7.0.14-25-pve-signed", + "proxmox-kernel-7.0.2-6-pve-signed" + ], + "installs": [] + }, + "refuse": [], + "verdict": "SAFE TO RUN" +} + +VERDICT: SAFE TO RUN + the one command: apt-get -y -q purge proxmox-kernel-7.0.2-6-pve-signed proxmox-kernel-7.0.14-20-pve-signed proxmox-kernel-7.0.14-22-pve-signed proxmox-kernel-7.0.14-23-pve-signed proxmox-kernel-7.0.14-24-pve-signed proxmox-kernel-7.0.14-25-pve-signed diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/04-negative-controls.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/04-negative-controls.txt new file mode 100644 index 00000000..fda5fd6e --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/04-negative-controls.txt @@ -0,0 +1,414 @@ +=== CONTROL A running kernel (also the pinned default here) in the removal set :: python3 - --pretend-add proxmox-kernel-7.0.14-28-pve-signed +{ + "host": "demo-hp", + "running": "7.0.14-28-pve", + "newest": "7.0.14-28-pve", + "phase": "good", + "step": { + "step_id": "ring0-20261110T014802Z", + "from": "7.0.14-26-pve", + "to": "7.0.14-28-pve", + "default_before": "7.0.14-26-pve", + "result_at": "2026-11-10T01:56:47Z" + }, + "oneshot_flag": "", + "grub_default": "7.0.14-28-pve", + "pinned_default": "7.0.14-28-pve", + "boot_pin": null, + "installed": { + "7.0.2-6-pve": "proxmox-kernel-7.0.2-6-pve-signed", + "7.0.14-20-pve": "proxmox-kernel-7.0.14-20-pve-signed", + "7.0.14-22-pve": "proxmox-kernel-7.0.14-22-pve-signed", + "7.0.14-23-pve": "proxmox-kernel-7.0.14-23-pve-signed", + "7.0.14-24-pve": "proxmox-kernel-7.0.14-24-pve-signed", + "7.0.14-25-pve": "proxmox-kernel-7.0.14-25-pve-signed", + "7.0.14-26-pve": "proxmox-kernel-7.0.14-26-pve-signed", + "7.0.14-28-pve": "proxmox-kernel-7.0.14-28-pve-signed" + }, + "keep": { + "7.0.14-26-pve": [ + "the last kernel that booted well before the running one (state.json `from`)", + "kernel step `from` (a self-revert boots it)", + "kernel step `default_before`" + ], + "7.0.14-28-pve": [ + "running", + "newest installed", + "kernel step `to`", + "GRUB default in grub.cfg", + "pinned default file" + ] + }, + "computed_removal": [ + "proxmox-kernel-7.0.2-6-pve-signed", + "proxmox-kernel-7.0.14-20-pve-signed", + "proxmox-kernel-7.0.14-22-pve-signed", + "proxmox-kernel-7.0.14-23-pve-signed", + "proxmox-kernel-7.0.14-24-pve-signed", + "proxmox-kernel-7.0.14-25-pve-signed" + ], + "removal": [ + "proxmox-kernel-7.0.2-6-pve-signed", + "proxmox-kernel-7.0.14-20-pve-signed", + "proxmox-kernel-7.0.14-22-pve-signed", + "proxmox-kernel-7.0.14-23-pve-signed", + "proxmox-kernel-7.0.14-24-pve-signed", + "proxmox-kernel-7.0.14-25-pve-signed", + "proxmox-kernel-7.0.14-28-pve-signed" + ], + "pretend": [ + "--pretend-add", + "proxmox-kernel-7.0.14-28-pve-signed" + ], + "simulation": { + "rc": 0, + "removes": [ + "proxmox-kernel-7.0.14-20-pve-signed", + "proxmox-kernel-7.0.14-22-pve-signed", + "proxmox-kernel-7.0.14-23-pve-signed", + "proxmox-kernel-7.0.14-24-pve-signed", + "proxmox-kernel-7.0.14-25-pve-signed", + "proxmox-kernel-7.0.14-28-pve-signed", + "proxmox-kernel-7.0.2-6-pve-signed" + ], + "installs": [ + "proxmox-kernel-7.0.14-28-pve" + ] + }, + "refuse": [ + "K1 proxmox-kernel-7.0.14-28-pve-signed is a keep-kernel (running; newest installed; kernel step `to`; GRUB default in grub.cfg; pinned default file)", + "S1 the simulation would install or configure proxmox-kernel-7.0.14-28-pve" + ], + "verdict": "REFUSED" +} + +VERDICT: REFUSED + REFUSE K1 proxmox-kernel-7.0.14-28-pve-signed is a keep-kernel (running; newest installed; kernel step `to`; GRUB default in grub.cfg; pinned default file) + REFUSE S1 the simulation would install or configure proxmox-kernel-7.0.14-28-pve +rc=3 + +=== CONTROL B last good kernel 26 in the removal set :: python3 - --pretend-add proxmox-kernel-7.0.14-26-pve-signed +{ + "host": "demo-hp", + "running": "7.0.14-28-pve", + "newest": "7.0.14-28-pve", + "phase": "good", + "step": { + "step_id": "ring0-20261110T014802Z", + "from": "7.0.14-26-pve", + "to": "7.0.14-28-pve", + "default_before": "7.0.14-26-pve", + "result_at": "2026-11-10T01:56:47Z" + }, + "oneshot_flag": "", + "grub_default": "7.0.14-28-pve", + "pinned_default": "7.0.14-28-pve", + "boot_pin": null, + "installed": { + "7.0.2-6-pve": "proxmox-kernel-7.0.2-6-pve-signed", + "7.0.14-20-pve": "proxmox-kernel-7.0.14-20-pve-signed", + "7.0.14-22-pve": "proxmox-kernel-7.0.14-22-pve-signed", + "7.0.14-23-pve": "proxmox-kernel-7.0.14-23-pve-signed", + "7.0.14-24-pve": "proxmox-kernel-7.0.14-24-pve-signed", + "7.0.14-25-pve": "proxmox-kernel-7.0.14-25-pve-signed", + "7.0.14-26-pve": "proxmox-kernel-7.0.14-26-pve-signed", + "7.0.14-28-pve": "proxmox-kernel-7.0.14-28-pve-signed" + }, + "keep": { + "7.0.14-26-pve": [ + "the last kernel that booted well before the running one (state.json `from`)", + "kernel step `from` (a self-revert boots it)", + "kernel step `default_before`" + ], + "7.0.14-28-pve": [ + "running", + "newest installed", + "kernel step `to`", + "GRUB default in grub.cfg", + "pinned default file" + ] + }, + "computed_removal": [ + "proxmox-kernel-7.0.2-6-pve-signed", + "proxmox-kernel-7.0.14-20-pve-signed", + "proxmox-kernel-7.0.14-22-pve-signed", + "proxmox-kernel-7.0.14-23-pve-signed", + "proxmox-kernel-7.0.14-24-pve-signed", + "proxmox-kernel-7.0.14-25-pve-signed" + ], + "removal": [ + "proxmox-kernel-7.0.2-6-pve-signed", + "proxmox-kernel-7.0.14-20-pve-signed", + "proxmox-kernel-7.0.14-22-pve-signed", + "proxmox-kernel-7.0.14-23-pve-signed", + "proxmox-kernel-7.0.14-24-pve-signed", + "proxmox-kernel-7.0.14-25-pve-signed", + "proxmox-kernel-7.0.14-26-pve-signed" + ], + "pretend": [ + "--pretend-add", + "proxmox-kernel-7.0.14-26-pve-signed" + ], + "simulation": { + "rc": 0, + "removes": [ + "proxmox-kernel-7.0.14-20-pve-signed", + "proxmox-kernel-7.0.14-22-pve-signed", + "proxmox-kernel-7.0.14-23-pve-signed", + "proxmox-kernel-7.0.14-24-pve-signed", + "proxmox-kernel-7.0.14-25-pve-signed", + "proxmox-kernel-7.0.14-26-pve-signed", + "proxmox-kernel-7.0.2-6-pve-signed" + ], + "installs": [] + }, + "refuse": [ + "K1 proxmox-kernel-7.0.14-26-pve-signed is a keep-kernel (the last kernel that booted well before the running one (state.json `from`); kernel step `from` (a self-revert boots it); kernel step `default_before`)" + ], + "verdict": "REFUSED" +} + +VERDICT: REFUSED + REFUSE K1 proxmox-kernel-7.0.14-26-pve-signed is a keep-kernel (the last kernel that booted well before the running one (state.json `from`); kernel step `from` (a self-revert boots it); kernel step `default_before`) +rc=3 + +=== CONTROL C a kernel step in phase judging :: python3 - --pretend-phase judging +{ + "host": "demo-hp", + "running": "7.0.14-28-pve", + "newest": "7.0.14-28-pve", + "phase": "judging", + "step": { + "step_id": "ring0-20261110T014802Z", + "from": "7.0.14-26-pve", + "to": "7.0.14-28-pve", + "default_before": "7.0.14-26-pve", + "result_at": "2026-11-10T01:56:47Z" + }, + "oneshot_flag": "", + "grub_default": "7.0.14-28-pve", + "pinned_default": "7.0.14-28-pve", + "boot_pin": null, + "installed": { + "7.0.2-6-pve": "proxmox-kernel-7.0.2-6-pve-signed", + "7.0.14-20-pve": "proxmox-kernel-7.0.14-20-pve-signed", + "7.0.14-22-pve": "proxmox-kernel-7.0.14-22-pve-signed", + "7.0.14-23-pve": "proxmox-kernel-7.0.14-23-pve-signed", + "7.0.14-24-pve": "proxmox-kernel-7.0.14-24-pve-signed", + "7.0.14-25-pve": "proxmox-kernel-7.0.14-25-pve-signed", + "7.0.14-26-pve": "proxmox-kernel-7.0.14-26-pve-signed", + "7.0.14-28-pve": "proxmox-kernel-7.0.14-28-pve-signed" + }, + "keep": { + "7.0.14-26-pve": [ + "the last kernel that booted well before the running one (state.json `from`)", + "kernel step `from` (a self-revert boots it)", + "kernel step `default_before`" + ], + "7.0.14-28-pve": [ + "running", + "newest installed", + "kernel step `to`", + "GRUB default in grub.cfg", + "pinned default file" + ] + }, + "computed_removal": [ + "proxmox-kernel-7.0.2-6-pve-signed", + "proxmox-kernel-7.0.14-20-pve-signed", + "proxmox-kernel-7.0.14-22-pve-signed", + "proxmox-kernel-7.0.14-23-pve-signed", + "proxmox-kernel-7.0.14-24-pve-signed", + "proxmox-kernel-7.0.14-25-pve-signed" + ], + "removal": [ + "proxmox-kernel-7.0.2-6-pve-signed", + "proxmox-kernel-7.0.14-20-pve-signed", + "proxmox-kernel-7.0.14-22-pve-signed", + "proxmox-kernel-7.0.14-23-pve-signed", + "proxmox-kernel-7.0.14-24-pve-signed", + "proxmox-kernel-7.0.14-25-pve-signed" + ], + "pretend": [ + "--pretend-phase", + "judging" + ], + "simulation": { + "rc": 0, + "removes": [ + "proxmox-kernel-7.0.14-20-pve-signed", + "proxmox-kernel-7.0.14-22-pve-signed", + "proxmox-kernel-7.0.14-23-pve-signed", + "proxmox-kernel-7.0.14-24-pve-signed", + "proxmox-kernel-7.0.14-25-pve-signed", + "proxmox-kernel-7.0.2-6-pve-signed" + ], + "installs": [] + }, + "refuse": [ + "P1 a kernel step is in phase 'judging' \u2014 not ended" + ], + "verdict": "REFUSED" +} + +VERDICT: REFUSED + REFUSE P1 a kernel step is in phase 'judging' — not ended +rc=3 + +=== CONTROL D a name that is not a kernel image (the series meta-package) :: python3 - --pretend-set proxmox-kernel-7.0 +{ + "host": "demo-hp", + "running": "7.0.14-28-pve", + "newest": "7.0.14-28-pve", + "phase": "good", + "step": { + "step_id": "ring0-20261110T014802Z", + "from": "7.0.14-26-pve", + "to": "7.0.14-28-pve", + "default_before": "7.0.14-26-pve", + "result_at": "2026-11-10T01:56:47Z" + }, + "oneshot_flag": "", + "grub_default": "7.0.14-28-pve", + "pinned_default": "7.0.14-28-pve", + "boot_pin": null, + "installed": { + "7.0.2-6-pve": "proxmox-kernel-7.0.2-6-pve-signed", + "7.0.14-20-pve": "proxmox-kernel-7.0.14-20-pve-signed", + "7.0.14-22-pve": "proxmox-kernel-7.0.14-22-pve-signed", + "7.0.14-23-pve": "proxmox-kernel-7.0.14-23-pve-signed", + "7.0.14-24-pve": "proxmox-kernel-7.0.14-24-pve-signed", + "7.0.14-25-pve": "proxmox-kernel-7.0.14-25-pve-signed", + "7.0.14-26-pve": "proxmox-kernel-7.0.14-26-pve-signed", + "7.0.14-28-pve": "proxmox-kernel-7.0.14-28-pve-signed" + }, + "keep": { + "7.0.14-26-pve": [ + "the last kernel that booted well before the running one (state.json `from`)", + "kernel step `from` (a self-revert boots it)", + "kernel step `default_before`" + ], + "7.0.14-28-pve": [ + "running", + "newest installed", + "kernel step `to`", + "GRUB default in grub.cfg", + "pinned default file" + ] + }, + "computed_removal": [ + "proxmox-kernel-7.0.2-6-pve-signed", + "proxmox-kernel-7.0.14-20-pve-signed", + "proxmox-kernel-7.0.14-22-pve-signed", + "proxmox-kernel-7.0.14-23-pve-signed", + "proxmox-kernel-7.0.14-24-pve-signed", + "proxmox-kernel-7.0.14-25-pve-signed" + ], + "removal": [ + "proxmox-kernel-7.0" + ], + "pretend": [ + "--pretend-set", + "proxmox-kernel-7.0" + ], + "simulation": { + "rc": 0, + "removes": [ + "proxmox-default-kernel", + "proxmox-kernel-7.0", + "proxmox-ve" + ], + "installs": [] + }, + "refuse": [ + "N1 proxmox-kernel-7.0 is not an installed kernel image package", + "S2 the simulation would also remove proxmox-default-kernel, proxmox-ve \u2014 not in the removal set" + ], + "verdict": "REFUSED" +} + +VERDICT: REFUSED + REFUSE N1 proxmox-kernel-7.0 is not an installed kernel image package + REFUSE S2 the simulation would also remove proxmox-default-kernel, proxmox-ve — not in the removal set +rc=3 + +=== CONTROL E a non-kernel package :: python3 - --pretend-set proxmox-kernel-7.0.14-25-pve-signed,pve-firmware +{ + "host": "demo-hp", + "running": "7.0.14-28-pve", + "newest": "7.0.14-28-pve", + "phase": "good", + "step": { + "step_id": "ring0-20261110T014802Z", + "from": "7.0.14-26-pve", + "to": "7.0.14-28-pve", + "default_before": "7.0.14-26-pve", + "result_at": "2026-11-10T01:56:47Z" + }, + "oneshot_flag": "", + "grub_default": "7.0.14-28-pve", + "pinned_default": "7.0.14-28-pve", + "boot_pin": null, + "installed": { + "7.0.2-6-pve": "proxmox-kernel-7.0.2-6-pve-signed", + "7.0.14-20-pve": "proxmox-kernel-7.0.14-20-pve-signed", + "7.0.14-22-pve": "proxmox-kernel-7.0.14-22-pve-signed", + "7.0.14-23-pve": "proxmox-kernel-7.0.14-23-pve-signed", + "7.0.14-24-pve": "proxmox-kernel-7.0.14-24-pve-signed", + "7.0.14-25-pve": "proxmox-kernel-7.0.14-25-pve-signed", + "7.0.14-26-pve": "proxmox-kernel-7.0.14-26-pve-signed", + "7.0.14-28-pve": "proxmox-kernel-7.0.14-28-pve-signed" + }, + "keep": { + "7.0.14-26-pve": [ + "the last kernel that booted well before the running one (state.json `from`)", + "kernel step `from` (a self-revert boots it)", + "kernel step `default_before`" + ], + "7.0.14-28-pve": [ + "running", + "newest installed", + "kernel step `to`", + "GRUB default in grub.cfg", + "pinned default file" + ] + }, + "computed_removal": [ + "proxmox-kernel-7.0.2-6-pve-signed", + "proxmox-kernel-7.0.14-20-pve-signed", + "proxmox-kernel-7.0.14-22-pve-signed", + "proxmox-kernel-7.0.14-23-pve-signed", + "proxmox-kernel-7.0.14-24-pve-signed", + "proxmox-kernel-7.0.14-25-pve-signed" + ], + "removal": [ + "proxmox-kernel-7.0.14-25-pve-signed", + "pve-firmware" + ], + "pretend": [ + "--pretend-set", + "proxmox-kernel-7.0.14-25-pve-signed,pve-firmware" + ], + "simulation": { + "rc": 0, + "removes": [ + "proxmox-default-kernel", + "proxmox-kernel-7.0", + "proxmox-kernel-7.0.14-25-pve-signed", + "proxmox-ve", + "pve-firmware" + ], + "installs": [] + }, + "refuse": [ + "N1 pve-firmware is not an installed kernel image package", + "S2 the simulation would also remove proxmox-default-kernel, proxmox-kernel-7.0, proxmox-ve \u2014 not in the removal set" + ], + "verdict": "REFUSED" +} + +VERDICT: REFUSED + REFUSE N1 pve-firmware is not an installed kernel image package + REFUSE S2 the simulation would also remove proxmox-default-kernel, proxmox-kernel-7.0, proxmox-ve — not in the removal set +rc=3 + diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/05-before-removal.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/05-before-removal.txt new file mode 100644 index 00000000..73931c0f --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/05-before-removal.txt @@ -0,0 +1,64 @@ +== 2026-11-10T12:50:49Z preconditions just before the removal +7.0.14-28-pve +-- heavy processes (expect none) +none +2256165 bash -c echo "== $(date -u +%FT%TZ) preconditions just before the removal"; uname -r; echo "-- heavy processes (expect none)"; pgrep -a -x "vzdump|apt-get|dpkg|restic|proxmox-backup-client|pbs-restore" || echo none; pgrep -af "[f]elhom-os-apply" || echo "no felhom-os-apply"; echo "-- running PVE tasks"; pvesh get /nodes/$(hostname)/tasks --source active --output-format json | python3 -c "import json,sys; t=json.load(sys.stdin); print(len(t), [x.get(\"type\") for x in t])"; echo "-- guests"; pct list; qm list; echo "-- kernel state"; python3 -c "import json; s=json.load(open(\"/var/lib/felhom-kernel/state.json\")); print(s[\"phase\"], s[\"from\"], s[\"to\"])"; grub-editenv /boot/efi/EFI/felhom/oneshot.env list; echo "-- locks"; ls -la /var/lib/dpkg/lock* /var/lib/apt/lists/lock 2>&1 | head; fuser /var/lib/dpkg/lock-frontend 2>&1 || echo "dpkg lock free"; echo "-- sizes before"; df -B1M / /boot/efi | cat; du -sm /lib/modules /boot 2>/dev/null; ls /lib/modules; echo "-- grub default"; /usr/bin/grep -n "set default=" /boot/grub/grub.cfg | head -3; /usr/bin/grep -c "felhom-oneshot-" /boot/grub/grub.cfg; /usr/bin/grep -o "felhom-oneshot-[0-9.a-z-]*" /boot/grub/grub.cfg | sort -u; cat /etc/default/grub.d/zz-felhom-kernel-default.cfg; sha256sum /etc/default/grub.d/zz-felhom-kernel-default.cfg /etc/grub.d/01_felhom_oneshot /etc/grub.d/42_felhom_oneshot; echo "-- crash guard"; ls -la /var/lib/felhom-crash-guard 2>&1 | head; systemctl list-timers --no-pager 2>/dev/null | /usr/bin/grep -i -E "felhom|pve|apt" | cut -c1-150 +-- running PVE tasks +0 [] +-- guests +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness + VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID + 341 night1004-tester1 running 8192 200.00 1861 +-- kernel state +good 7.0.14-26-pve 7.0.14-28-pve +felhom_next= +-- locks +-rw-r----- 1 root root 0 May 21 13:31 /var/lib/apt/lists/lock +-rw-r----- 1 root root 0 Nov 10 02:50 /var/lib/dpkg/lock +-rw-r----- 1 root root 0 Oct 4 14:25 /var/lib/dpkg/lock-frontend +dpkg lock free +-- sizes before +Filesystem 1M-blocks Used Available Use% Mounted on +/dev/mapper/pve-root 39506 22906 14562 62% / +/dev/sda2 1022 10 1013 1% /boot/efi +7772 /lib/modules +795 /boot +7.0.14-20-pve +7.0.14-22-pve +7.0.14-23-pve +7.0.14-24-pve +7.0.14-25-pve +7.0.14-26-pve +7.0.14-28-pve +7.0.2-6-pve +-- grub default +17: set default="${next_entry}" +22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43" +110: if [ "${felhom_boot}" = "7.0.14-20-pve" ]; then set default="felhom-oneshot-7.0.14-20-pve"; fi +16 +felhom-oneshot-7.0.14-20-pve +felhom-oneshot-7.0.14-22-pve +felhom-oneshot-7.0.14-23-pve +felhom-oneshot-7.0.14-24-pve +felhom-oneshot-7.0.14-25-pve +felhom-oneshot-7.0.14-26-pve +felhom-oneshot-7.0.14-28-pve +felhom-oneshot-7.0.2-6-pve +# felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily +GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43" +6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg +f2f56d59c6356067279baa457e84bba7ffcf46ac341b7d482845e5a5cf9cc27b /etc/grub.d/01_felhom_oneshot +c34907ce68599873889179be333e848e02aa1c594b7370116b95af31b41b7fe6 /etc/grub.d/42_felhom_oneshot +-- crash guard +total 12 +drwxr-xr-x 2 root root 4096 Nov 10 13:09 . +drwxr-xr-x 46 root root 4096 Oct 10 04:42 .. +-rw-r--r-- 1 root root 679 Nov 10 13:09 state.json +Tue 2026-11-10 13:51:28 CET 35s Tue 2026-11-10 13:50:28 CET 24s ago felhom-mgmt-watchdog.timer felhom-mgmt-watchdog.service +Tue 2026-11-10 14:09:17 CET 18min Tue 2026-11-10 13:09:17 CET 41min ago felhom-crash-guard-check.timer felhom-crash-guard-check.service +Tue 2026-11-10 15:51:50 CET 2h 0min Tue 2026-11-10 04:06:22 CET 9h ago apt-daily.timer apt-daily.service +Wed 2026-11-11 04:54:05 CET 15h Tue 2026-11-10 03:05:09 CET 10h ago pve-daily-update.timer pve-daily-update.service +Wed 2026-11-11 06:38:33 CET 16h Tue 2026-11-10 06:55:09 CET 6h ago apt-daily-upgrade.timer apt-daily-upgrade.service diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/06-removal-plan.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/06-removal-plan.txt new file mode 100644 index 00000000..51b9a3ae --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/06-removal-plan.txt @@ -0,0 +1,43 @@ +== 2026-11-10T12:50:59Z crash guard state before +{ + "armed": true, + "boot_id": "b6e0edea-2b56-47df-a36f-5bb3825adc3a", + "config": { + "LIMIT": 3, + "PANIC_SECONDS": 10, + "REARM_HOURS": 24, + "WINDOW_MINUTES": 60 + }, + "kernel_panic": 10, + "last_boot_at": "2026-11-10T01:53:52Z", + "last_boot_unclean": false, + "last_trip": { + "at": "2026-10-13T20:14:04Z", + "reason": "2 unclean boots within 60 minutes \u2014 the next crash leaves the box off (limit 3)" + }, + "rearmed_at": "2026-10-15T15:41:56Z", + "rearmed_by": "timer (24 h of normal running)", + "tripped": false, + "unclean_boots": [], + "unclean_boots_24h": 0, + "unclean_boots_in_window": 0, + "updated_at": "2026-11-10T12:09:17Z", + "version": 1 +} + +== SIMULATION (apt-get -s), the exact command line +Reading package lists... +Building dependency tree... +Reading state information... +The following packages will be REMOVED: + proxmox-kernel-7.0.14-20-pve-signed* proxmox-kernel-7.0.14-22-pve-signed* + proxmox-kernel-7.0.14-23-pve-signed* proxmox-kernel-7.0.14-24-pve-signed* + proxmox-kernel-7.0.14-25-pve-signed* proxmox-kernel-7.0.2-6-pve-signed* +0 upgraded, 0 newly installed, 6 to remove and 16 not upgraded. +Purg proxmox-kernel-7.0.14-20-pve-signed [7.0.14-20] +Purg proxmox-kernel-7.0.14-22-pve-signed [7.0.14-22] +Purg proxmox-kernel-7.0.14-23-pve-signed [7.0.14-23] +Purg proxmox-kernel-7.0.14-24-pve-signed [7.0.14-24] +Purg proxmox-kernel-7.0.14-25-pve-signed [7.0.14-25] +Purg proxmox-kernel-7.0.2-6-pve-signed [7.0.2-6] +sim rc=0 diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/07-removal-output.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/07-removal-output.txt new file mode 100644 index 00000000..6704316e --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/07-removal-output.txt @@ -0,0 +1,284 @@ +== START 2026-11-10T12:51:10Z +Reading package lists... +Building dependency tree... +Reading state information... +The following packages will be REMOVED: + proxmox-kernel-7.0.14-20-pve-signed* proxmox-kernel-7.0.14-22-pve-signed* + proxmox-kernel-7.0.14-23-pve-signed* proxmox-kernel-7.0.14-24-pve-signed* + proxmox-kernel-7.0.14-25-pve-signed* proxmox-kernel-7.0.2-6-pve-signed* +0 upgraded, 0 newly installed, 6 to remove and 16 not upgraded. +After this operation, 6184 MB disk space will be freed. +(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 110097 files and directories currently installed.) +Removing proxmox-kernel-7.0.14-20-pve-signed (7.0.14-20) ... +Examining /etc/kernel/postrm.d. +run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-20-pve /boot/vmlinuz-7.0.14-20-pve +update-initramfs: Deleting /boot/initrd.img-7.0.14-20-pve +run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-20-pve /boot/vmlinuz-7.0.14-20-pve +run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-20-pve /boot/vmlinuz-7.0.14-20-pve +Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace.. +No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync. +run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-20-pve /boot/vmlinuz-7.0.14-20-pve +Generating grub configuration file ... +Found linux image: /boot/vmlinuz-7.0.14-28-pve +Found initrd image: /boot/initrd.img-7.0.14-28-pve +Found linux image: /boot/vmlinuz-7.0.14-26-pve +Found initrd image: /boot/initrd.img-7.0.14-26-pve +Found linux image: /boot/vmlinuz-7.0.14-25-pve +Found initrd image: /boot/initrd.img-7.0.14-25-pve +Found linux image: /boot/vmlinuz-7.0.14-24-pve +Found initrd image: /boot/initrd.img-7.0.14-24-pve +Found linux image: /boot/vmlinuz-7.0.14-23-pve +Found initrd image: /boot/initrd.img-7.0.14-23-pve +Found linux image: /boot/vmlinuz-7.0.14-22-pve +Found initrd image: /boot/initrd.img-7.0.14-22-pve +Found linux image: /boot/vmlinuz-7.0.2-6-pve +Found initrd image: /boot/initrd.img-7.0.2-6-pve +Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi +Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi +Found memtest86+ 64bit image: /boot/memtest86+x64.bin +Found memtest86+ 32bit image: /boot/memtest86+ia32.bin +Adding boot menu entry for UEFI Firmware Settings ... +done +Removing proxmox-kernel-7.0.14-22-pve-signed (7.0.14-22) ... +Examining /etc/kernel/postrm.d. +run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-22-pve /boot/vmlinuz-7.0.14-22-pve +update-initramfs: Deleting /boot/initrd.img-7.0.14-22-pve +run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-22-pve /boot/vmlinuz-7.0.14-22-pve +run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-22-pve /boot/vmlinuz-7.0.14-22-pve +Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace.. +No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync. +run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-22-pve /boot/vmlinuz-7.0.14-22-pve +Generating grub configuration file ... +Found linux image: /boot/vmlinuz-7.0.14-28-pve +Found initrd image: /boot/initrd.img-7.0.14-28-pve +Found linux image: /boot/vmlinuz-7.0.14-26-pve +Found initrd image: /boot/initrd.img-7.0.14-26-pve +Found linux image: /boot/vmlinuz-7.0.14-25-pve +Found initrd image: /boot/initrd.img-7.0.14-25-pve +Found linux image: /boot/vmlinuz-7.0.14-24-pve +Found initrd image: /boot/initrd.img-7.0.14-24-pve +Found linux image: /boot/vmlinuz-7.0.14-23-pve +Found initrd image: /boot/initrd.img-7.0.14-23-pve +Found linux image: /boot/vmlinuz-7.0.2-6-pve +Found initrd image: /boot/initrd.img-7.0.2-6-pve +Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi +Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi +Found memtest86+ 64bit image: /boot/memtest86+x64.bin +Found memtest86+ 32bit image: /boot/memtest86+ia32.bin +Adding boot menu entry for UEFI Firmware Settings ... +done +Removing proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23) ... +Examining /etc/kernel/postrm.d. +run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-23-pve /boot/vmlinuz-7.0.14-23-pve +update-initramfs: Deleting /boot/initrd.img-7.0.14-23-pve +run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-23-pve /boot/vmlinuz-7.0.14-23-pve +run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-23-pve /boot/vmlinuz-7.0.14-23-pve +Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace.. +No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync. +run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-23-pve /boot/vmlinuz-7.0.14-23-pve +Generating grub configuration file ... +Found linux image: /boot/vmlinuz-7.0.14-28-pve +Found initrd image: /boot/initrd.img-7.0.14-28-pve +Found linux image: /boot/vmlinuz-7.0.14-26-pve +Found initrd image: /boot/initrd.img-7.0.14-26-pve +Found linux image: /boot/vmlinuz-7.0.14-25-pve +Found initrd image: /boot/initrd.img-7.0.14-25-pve +Found linux image: /boot/vmlinuz-7.0.14-24-pve +Found initrd image: /boot/initrd.img-7.0.14-24-pve +Found linux image: /boot/vmlinuz-7.0.2-6-pve +Found initrd image: /boot/initrd.img-7.0.2-6-pve +Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi +Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi +Found memtest86+ 64bit image: /boot/memtest86+x64.bin +Found memtest86+ 32bit image: /boot/memtest86+ia32.bin +Adding boot menu entry for UEFI Firmware Settings ... +done +Removing proxmox-kernel-7.0.14-24-pve-signed (7.0.14-24) ... +Examining /etc/kernel/postrm.d. +run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-24-pve /boot/vmlinuz-7.0.14-24-pve +update-initramfs: Deleting /boot/initrd.img-7.0.14-24-pve +run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-24-pve /boot/vmlinuz-7.0.14-24-pve +run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-24-pve /boot/vmlinuz-7.0.14-24-pve +Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace.. +No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync. +run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-24-pve /boot/vmlinuz-7.0.14-24-pve +Generating grub configuration file ... +Found linux image: /boot/vmlinuz-7.0.14-28-pve +Found initrd image: /boot/initrd.img-7.0.14-28-pve +Found linux image: /boot/vmlinuz-7.0.14-26-pve +Found initrd image: /boot/initrd.img-7.0.14-26-pve +Found linux image: /boot/vmlinuz-7.0.14-25-pve +Found initrd image: /boot/initrd.img-7.0.14-25-pve +Found linux image: /boot/vmlinuz-7.0.2-6-pve +Found initrd image: /boot/initrd.img-7.0.2-6-pve +Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi +Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi +Found memtest86+ 64bit image: /boot/memtest86+x64.bin +Found memtest86+ 32bit image: /boot/memtest86+ia32.bin +Adding boot menu entry for UEFI Firmware Settings ... +done +Removing proxmox-kernel-7.0.14-25-pve-signed (7.0.14-25) ... +Examining /etc/kernel/postrm.d. +run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-25-pve /boot/vmlinuz-7.0.14-25-pve +update-initramfs: Deleting /boot/initrd.img-7.0.14-25-pve +run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-25-pve /boot/vmlinuz-7.0.14-25-pve +run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-25-pve /boot/vmlinuz-7.0.14-25-pve +Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace.. +No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync. +run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-25-pve /boot/vmlinuz-7.0.14-25-pve +Generating grub configuration file ... +Found linux image: /boot/vmlinuz-7.0.14-28-pve +Found initrd image: /boot/initrd.img-7.0.14-28-pve +Found linux image: /boot/vmlinuz-7.0.14-26-pve +Found initrd image: /boot/initrd.img-7.0.14-26-pve +Found linux image: /boot/vmlinuz-7.0.2-6-pve +Found initrd image: /boot/initrd.img-7.0.2-6-pve +Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi +Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi +Found memtest86+ 64bit image: /boot/memtest86+x64.bin +Found memtest86+ 32bit image: /boot/memtest86+ia32.bin +Adding boot menu entry for UEFI Firmware Settings ... +done +Removing proxmox-kernel-7.0.2-6-pve-signed (7.0.2-6) ... +Examining /etc/kernel/postrm.d. +run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.2-6-pve /boot/vmlinuz-7.0.2-6-pve +update-initramfs: Deleting /boot/initrd.img-7.0.2-6-pve +run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.2-6-pve /boot/vmlinuz-7.0.2-6-pve +run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.2-6-pve /boot/vmlinuz-7.0.2-6-pve +Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace.. +No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync. +run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.2-6-pve /boot/vmlinuz-7.0.2-6-pve +Generating grub configuration file ... +Found linux image: /boot/vmlinuz-7.0.14-28-pve +Found initrd image: /boot/initrd.img-7.0.14-28-pve +Found linux image: /boot/vmlinuz-7.0.14-26-pve +Found initrd image: /boot/initrd.img-7.0.14-26-pve +Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi +Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi +Found memtest86+ 64bit image: /boot/memtest86+x64.bin +Found memtest86+ 32bit image: /boot/memtest86+ia32.bin +Adding boot menu entry for UEFI Firmware Settings ... +done +(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 60863 files and directories currently installed.) +Purging configuration files for proxmox-kernel-7.0.14-22-pve-signed (7.0.14-22) ... +Examining /etc/kernel/postrm.d. +run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-22-pve /boot/vmlinuz-7.0.14-22-pve +update-initramfs: Deleting /boot/initrd.img-7.0.14-22-pve +run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-22-pve /boot/vmlinuz-7.0.14-22-pve +run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-22-pve /boot/vmlinuz-7.0.14-22-pve +Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace.. +No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync. +run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-22-pve /boot/vmlinuz-7.0.14-22-pve +Generating grub configuration file ... +Found linux image: /boot/vmlinuz-7.0.14-28-pve +Found initrd image: /boot/initrd.img-7.0.14-28-pve +Found linux image: /boot/vmlinuz-7.0.14-26-pve +Found initrd image: /boot/initrd.img-7.0.14-26-pve +Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi +Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi +Found memtest86+ 64bit image: /boot/memtest86+x64.bin +Found memtest86+ 32bit image: /boot/memtest86+ia32.bin +Adding boot menu entry for UEFI Firmware Settings ... +done +Purging configuration files for proxmox-kernel-7.0.14-20-pve-signed (7.0.14-20) ... +Examining /etc/kernel/postrm.d. +run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-20-pve /boot/vmlinuz-7.0.14-20-pve +update-initramfs: Deleting /boot/initrd.img-7.0.14-20-pve +run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-20-pve /boot/vmlinuz-7.0.14-20-pve +run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-20-pve /boot/vmlinuz-7.0.14-20-pve +Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace.. +No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync. +run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-20-pve /boot/vmlinuz-7.0.14-20-pve +Generating grub configuration file ... +Found linux image: /boot/vmlinuz-7.0.14-28-pve +Found initrd image: /boot/initrd.img-7.0.14-28-pve +Found linux image: /boot/vmlinuz-7.0.14-26-pve +Found initrd image: /boot/initrd.img-7.0.14-26-pve +Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi +Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi +Found memtest86+ 64bit image: /boot/memtest86+x64.bin +Found memtest86+ 32bit image: /boot/memtest86+ia32.bin +Adding boot menu entry for UEFI Firmware Settings ... +done +Purging configuration files for proxmox-kernel-7.0.2-6-pve-signed (7.0.2-6) ... +Examining /etc/kernel/postrm.d. +run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.2-6-pve /boot/vmlinuz-7.0.2-6-pve +update-initramfs: Deleting /boot/initrd.img-7.0.2-6-pve +run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.2-6-pve /boot/vmlinuz-7.0.2-6-pve +run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.2-6-pve /boot/vmlinuz-7.0.2-6-pve +Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace.. +No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync. +run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.2-6-pve /boot/vmlinuz-7.0.2-6-pve +Generating grub configuration file ... +Found linux image: /boot/vmlinuz-7.0.14-28-pve +Found initrd image: /boot/initrd.img-7.0.14-28-pve +Found linux image: /boot/vmlinuz-7.0.14-26-pve +Found initrd image: /boot/initrd.img-7.0.14-26-pve +Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi +Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi +Found memtest86+ 64bit image: /boot/memtest86+x64.bin +Found memtest86+ 32bit image: /boot/memtest86+ia32.bin +Adding boot menu entry for UEFI Firmware Settings ... +done +Purging configuration files for proxmox-kernel-7.0.14-25-pve-signed (7.0.14-25) ... +Examining /etc/kernel/postrm.d. +run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-25-pve /boot/vmlinuz-7.0.14-25-pve +update-initramfs: Deleting /boot/initrd.img-7.0.14-25-pve +run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-25-pve /boot/vmlinuz-7.0.14-25-pve +run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-25-pve /boot/vmlinuz-7.0.14-25-pve +Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace.. +No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync. +run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-25-pve /boot/vmlinuz-7.0.14-25-pve +Generating grub configuration file ... +Found linux image: /boot/vmlinuz-7.0.14-28-pve +Found initrd image: /boot/initrd.img-7.0.14-28-pve +Found linux image: /boot/vmlinuz-7.0.14-26-pve +Found initrd image: /boot/initrd.img-7.0.14-26-pve +Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi +Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi +Found memtest86+ 64bit image: /boot/memtest86+x64.bin +Found memtest86+ 32bit image: /boot/memtest86+ia32.bin +Adding boot menu entry for UEFI Firmware Settings ... +done +Purging configuration files for proxmox-kernel-7.0.14-24-pve-signed (7.0.14-24) ... +Examining /etc/kernel/postrm.d. +run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-24-pve /boot/vmlinuz-7.0.14-24-pve +update-initramfs: Deleting /boot/initrd.img-7.0.14-24-pve +run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-24-pve /boot/vmlinuz-7.0.14-24-pve +run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-24-pve /boot/vmlinuz-7.0.14-24-pve +Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace.. +No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync. +run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-24-pve /boot/vmlinuz-7.0.14-24-pve +Generating grub configuration file ... +Found linux image: /boot/vmlinuz-7.0.14-28-pve +Found initrd image: /boot/initrd.img-7.0.14-28-pve +Found linux image: /boot/vmlinuz-7.0.14-26-pve +Found initrd image: /boot/initrd.img-7.0.14-26-pve +Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi +Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi +Found memtest86+ 64bit image: /boot/memtest86+x64.bin +Found memtest86+ 32bit image: /boot/memtest86+ia32.bin +Adding boot menu entry for UEFI Firmware Settings ... +done +Purging configuration files for proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23) ... +Examining /etc/kernel/postrm.d. +run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-23-pve /boot/vmlinuz-7.0.14-23-pve +update-initramfs: Deleting /boot/initrd.img-7.0.14-23-pve +run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-23-pve /boot/vmlinuz-7.0.14-23-pve +run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-23-pve /boot/vmlinuz-7.0.14-23-pve +Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace.. +No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync. +run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-23-pve /boot/vmlinuz-7.0.14-23-pve +Generating grub configuration file ... +Found linux image: /boot/vmlinuz-7.0.14-28-pve +Found initrd image: /boot/initrd.img-7.0.14-28-pve +Found linux image: /boot/vmlinuz-7.0.14-26-pve +Found initrd image: /boot/initrd.img-7.0.14-26-pve +Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi +Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi +Found memtest86+ 64bit image: /boot/memtest86+x64.bin +Found memtest86+ 32bit image: /boot/memtest86+ia32.bin +Adding boot menu entry for UEFI Firmware Settings ... +done +APT_RC=0 +== END 2026-11-10T12:52:00Z diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/08-after-removal.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/08-after-removal.txt new file mode 100644 index 00000000..9f472e98 --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/08-after-removal.txt @@ -0,0 +1,95 @@ +== 2026-11-10T12:52:12Z after the removal, before the reboot +7.0.14-28-pve +-- kernel packages +ii proxmox-default-kernel 2.1.0 +ii proxmox-kernel-7.0 7.0.14-28 +un proxmox-kernel-7.0.14-26-pve +ii proxmox-kernel-7.0.14-26-pve-signed 7.0.14-26 +un proxmox-kernel-7.0.14-28-pve +ii proxmox-kernel-7.0.14-28-pve-signed 7.0.14-28 +ii proxmox-kernel-helper 9.1.0+fde2 +-- dpkg audit (expect empty) +audit rc=0 +-- sizes after +Filesystem 1M-blocks Used Available Use% Mounted on +/dev/mapper/pve-root 39506 16495 20973 45% / +/dev/sda2 1022 10 1013 1% /boot/efi +1945 /lib/modules +217 /boot +7.0.14-26-pve +7.0.14-28-pve +-rw-r--r-- 1 root root 308500 Oct 22 20:45 config-7.0.14-26-pve +-rw-r--r-- 1 root root 308500 Nov 2 12:54 config-7.0.14-28-pve +-rw-r--r-- 1 root root 73492957 Oct 29 05:07 initrd.img-7.0.14-26-pve +-rw-r--r-- 1 root root 73547547 Nov 10 02:50 initrd.img-7.0.14-28-pve +-rw-r--r-- 1 root root 10341401 Oct 22 20:45 System.map-7.0.14-26-pve +-rw-r--r-- 1 root root 10341916 Nov 2 12:54 System.map-7.0.14-28-pve +-rw-r--r-- 1 root root 16280680 Oct 22 20:45 vmlinuz-7.0.14-26-pve +-rw-r--r-- 1 root root 16280680 Nov 2 12:54 vmlinuz-7.0.14-28-pve +-- grub.cfg default + pinned file +17: set default="${next_entry}" +22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43" +110: if [ "${felhom_boot}" = "7.0.14-26-pve" ]; then set default="felhom-oneshot-7.0.14-26-pve"; fi +111: if [ "${felhom_boot}" = "7.0.14-28-pve" ]; then set default="felhom-oneshot-7.0.14-28-pve"; fi +# felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily +GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43" +6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg +f2f56d59c6356067279baa457e84bba7ffcf46ac341b7d482845e5a5cf9cc27b /etc/grub.d/01_felhom_oneshot +c34907ce68599873889179be333e848e02aa1c594b7370116b95af31b41b7fe6 /etc/grub.d/42_felhom_oneshot +-- menu entries +menuentry 'Proxmox VE GNU/Linux' +submenu 'Advanced options for Proxmox VE GNU/Linux' $menuentry_id_option 'gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43' { + menuentry 'Proxmox VE GNU/Linux, with Linux 7.0.14-28-pve' + menuentry 'Proxmox VE GNU/Linux, with Linux 7.0.14-28-pve (recovery mode)' + menuentry 'Proxmox VE GNU/Linux, with Linux 7.0.14-26-pve' + menuentry 'Proxmox VE GNU/Linux, with Linux 7.0.14-26-pve (recovery mode)' + menuentry "Memory test (memtest86+x64.efi)" + menuentry "Memory test (memtest86+x64.efi, serial console)" + menuentry "Memory test (memtest86+ia32.efi)" + menuentry "Memory test (memtest86+ia32.efi, serial console)" + menuentry "Memory test (memtest86+x64.bin)" + menuentry "Memory test (memtest86+x64.bin, serial console)" + menuentry "Memory test (memtest86+ia32.bin)" + menuentry "Memory test (memtest86+ia32.bin, serial console)" + menuentry 'UEFI Firmware Settings' $menuentry_id_option 'uefi-firmware' { +menuentry 'Felhom one-shot: 7.0.14-26-pve' +menuentry 'Felhom one-shot: 7.0.14-28-pve' +-- one-shot entries + 2 felhom-oneshot-7.0.14-26-pve + 2 felhom-oneshot-7.0.14-28-pve +-- every kernel file grub.cfg names exists +ok /boot/initrd.img-7.0.14-26-pve +ok /boot/initrd.img-7.0.14-28-pve +ok /boot/vmlinuz-7.0.14-26-pve +ok /boot/vmlinuz-7.0.14-28-pve +-- grub syntax +grub-script-check rc=0 +-- flag + state +felhom_next= +good 7.0.14-26-pve 7.0.14-28-pve +-- NeverAutoRemove now +APT::NeverAutoRemove:: "^pve-kernel-7\.0$"; +APT::NeverAutoRemove:: "^proxmox-kernel-7\.0$"; +APT::NeverAutoRemove:: "^proxmox-kernel-7\.0-signed$"; +APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-23-pve$"; +APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-23-pve$"; +APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-23-pve-signed$"; +APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-26-pve$"; +APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve$"; +APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve-signed$"; +APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-28-pve$"; +APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve$"; +APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve-signed$"; +-- simulated autoremove after +0 upgraded, 0 newly installed, 0 to remove and 16 not upgraded. +-- services +active +active +active +active +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness + VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID + 341 night1004-tester1 running 8192 200.00 1861 diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/09-reboot-poll.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/09-reboot-poll.txt new file mode 100644 index 00000000..59a3c032 --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/09-reboot-poll.txt @@ -0,0 +1,102 @@ +2026-11-10T12:53:45Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 49% 58% running (connected) 71% local-lvm +2026-11-10T12:53:55Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 49% 58% running (connected) 71% local-lvm +2026-11-10T12:54:13Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 49% 58% running (connected) 71% local-lvm +2026-11-10T12:54:23Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:54:33Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:54:43Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:54:53Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:55:03Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:55:14Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:55:24Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:55:34Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:55:44Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:55:54Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:56:04Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:56:14Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:56:25Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:56:35Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:56:46Z ping=DOWN hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:56:57Z ping=DOWN hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:57:08Z ping=DOWN hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm +2026-11-10T12:57:18Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T12:57:29Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T12:57:39Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T12:57:49Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T12:57:59Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T12:58:09Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T12:58:19Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T12:58:30Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T12:58:40Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T12:58:50Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T12:59:00Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T12:59:10Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T12:59:20Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T12:59:31Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T12:59:41Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T12:59:51Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:00:01Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:00:12Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:00:22Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:00:32Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:00:42Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:00:53Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:01:03Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:01:13Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:01:23Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:01:33Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:01:43Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:01:53Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:02:04Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:02:14Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:02:24Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:02:34Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:02:44Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:02:54Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:03:05Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:03:15Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:03:25Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:03:35Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:03:45Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:03:55Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:04:05Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:04:16Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:04:26Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:04:36Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:04:46Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:04:56Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:05:06Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:05:16Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:05:27Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:05:37Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:05:47Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:05:57Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:06:08Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:06:18Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:06:28Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:06:38Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:06:48Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:06:58Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:07:09Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:07:19Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:07:29Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:07:39Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:07:49Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:07:59Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:08:10Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:08:20Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:08:30Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:08:40Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:08:50Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:09:00Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:09:11Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:09:21Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:09:31Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:09:41Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:09:51Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:10:01Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:10:12Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:10:22Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:10:32Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:10:42Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:10:52Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm +2026-11-10T13:11:02Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/10-before-reboot.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/10-before-reboot.txt new file mode 100644 index 00000000..7b388473 --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/10-before-reboot.txt @@ -0,0 +1,46 @@ +== 2026-11-10T12:54:17Z last check before the one orderly reboot +7.0.14-28-pve +b6e0edea-2b56-47df-a36f-5bb3825adc3a + 13:54:17 up 11:00, 1 user, load average: 1.52, 1.28, 1.43 +no heavy process +running PVE tasks: 0 +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness + VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID + 341 night1004-tester1 running 8192 200.00 1861 +-- guest 9201 containers (names + status only) +adventurelog-frontend Up 10 hours (healthy) +adventurelog-postgres Up 10 hours (healthy) +adventurelog Up 10 hours (healthy) +bentopdf Up 11 hours (healthy) +bookstack-db Up 10 hours (healthy) +bookstack Up 10 hours (healthy) +calibre-web Up 10 hours (healthy) +cloudflared Up 11 hours (healthy) +docmost-postgres Up 10 hours (healthy) +docmost-redis Up 10 hours (healthy) +docmost Up 10 hours (healthy) +felhom-controller Up 11 hours (healthy) +filebrowser Up 11 hours (healthy) +home-assistant Up 10 hours (healthy) +kimai-db Up 10 hours (healthy) +kimai Up 10 hours (healthy) +opengist Up 10 hours (healthy) +paperless-postgres Up 10 hours (healthy) +paperless-redis Up 10 hours (healthy) +paperless-webserver Up 10 hours (healthy) +plex Up 10 hours (healthy) +privatebin Up 10 hours (healthy) +romm-db Up 10 hours (healthy) +romm-redis Up 10 hours (healthy) +romm Up 10 hours (healthy) +traefik Up 11 hours +-- guest 9202 +felhom-controller Up 22 minutes (healthy) +filebrowser Up 23 minutes (healthy) +paperless-postgres Up 11 hours (healthy) +paperless-redis Up 11 hours (healthy) +paperless-webserver Up 11 hours (healthy) +traefik Up 11 hours diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/11-reboot.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/11-reboot.txt new file mode 100644 index 00000000..770b3073 --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/11-reboot.txt @@ -0,0 +1,4 @@ +T0 DooPlex 2026-11-10T12:54:26Z — systemctl reboot sent to demo-hp +host clock 2026-11-10T12:54:26Z +systemctl reboot rc=0 +ssh rc=0 diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/12-containers-poll.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/12-containers-poll.txt new file mode 100644 index 00000000..7178f827 --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/12-containers-poll.txt @@ -0,0 +1,4 @@ +2026-11-10T12:58:41Z g9201 total=26 up=26 healthy=22 starting=3 unhealthy=0 ; g9202 total=6 up=6 healthy=5 starting=0 unhealthy=0 ; +2026-11-10T12:58:59Z g9201 total=26 up=26 healthy=24 starting=1 unhealthy=0 ; g9202 total=6 up=6 healthy=5 starting=0 unhealthy=0 ; +2026-11-10T12:59:18Z g9201 total=26 up=26 healthy=24 starting=1 unhealthy=0 ; g9202 total=6 up=6 healthy=5 starting=0 unhealthy=0 ; +2026-11-10T12:59:37Z g9201 total=26 up=26 healthy=25 starting=0 unhealthy=0 ; g9202 total=6 up=6 healthy=5 starting=0 unhealthy=0 ; diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/13-after-reboot.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/13-after-reboot.txt new file mode 100644 index 00000000..845e9d12 --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/13-after-reboot.txt @@ -0,0 +1,298 @@ +# after the one reboot — read on demo-hp; DooPlex clock 2026-11-10T12:59:51Z +== clock +2026-11-10T12:59:51Z +== kernel +7.0.14-28-pve +== boot_id +d1167d94-31bf-459a-bf2a-569e8997a569 +== up since (host clock, CET) +2026-11-10 13:56:59 + 13:59:51 up 2 min, 1 user, load average: 6.29, 4.13, 1.67 +== system state +running +== services +felhom-agent active +pvedaemon active +pveproxy active +pvestatd active +pve-cluster active +== crash guard state +{ + "armed": true, + "boot_id": "d1167d94-31bf-459a-bf2a-569e8997a569", + "config": { + "LIMIT": 3, + "PANIC_SECONDS": 10, + "REARM_HOURS": 24, + "WINDOW_MINUTES": 60 + }, + "kernel_panic": 10, + "last_boot_at": "2026-11-10T12:56:59Z", + "last_boot_unclean": false, + "last_trip": { + "at": "2026-10-13T20:14:04Z", + "reason": "2 unclean boots within 60 minutes \u2014 the next crash leaves the box off (limit 3)" + }, + "rearmed_at": "2026-10-15T15:41:56Z", + "rearmed_by": "timer (24 h of normal running)", + "tripped": false, + "unclean_boots": [], + "unclean_boots_24h": 0, + "unclean_boots_in_window": 0, + "updated_at": "2026-11-10T12:57:09Z", + "version": 1 +} + +== previous boot: how it ended (journal -b -1, last lines) +2026-11-10T13:56:35+01:00 demo-hp systemd-shutdown[1]: Using hardware watchdog 'Software Watchdog', version 0, device /dev/watchdog0 +2026-11-10T13:56:35+01:00 demo-hp systemd-shutdown[1]: Watchdog running with a hardware timeout of 10min. +2026-11-10T13:56:35+01:00 demo-hp systemd-shutdown[1]: Syncing filesystems and block devices. +2026-11-10T13:56:35+01:00 demo-hp systemd-shutdown[1]: Sending SIGTERM to remaining processes... +2026-11-10T13:56:35+01:00 demo-hp systemd-journald[351]: Received SIGTERM from PID 1 (systemd-shutdow). +2026-11-10T13:56:35+01:00 demo-hp systemd-journald[351]: Journal stopped +== kernel step state +{ + "authority": "ring0", + "default_before": "7.0.14-26-pve", + "from": "7.0.14-26-pve", + "health_before": { + "guest": { + "containers": { + "adventurelog": { + "health": "healthy", + "id": "50af393c36a37d01a10847456d391d70572c121045e28626f220f83cec1cefcc", + "state": "running" + }, + "adventurelog-frontend": { + "health": "healthy", + "id": "abd6155bdad527c65b82f75bc0dd0f98f9690ec81587c8cd07cea6c9c019cabe", + "state": "running" + }, + "adventurelog-postgres": { + "health": "healthy", + "id": "841257c7abb7b545567aa8fa539cdcd421c94ce4edf97a0dea5e45831ea399f4", + "state": "running" + }, + "bentopdf": { + "health": "healthy", + "id": "9db9aba521daf9f19352a64511f4f84872ade957b643c0db40dd510837143cba", + "state": "running" + }, + "bookstack": { + "health": "healthy", + "id": "c83bbf5562cb68ee23d71ece2362d8be482ac4d7d59ad9bdfa5493f0b53c9687", + "state": "running" + }, + "bookstack-db": { + "health": "healthy", + "id": "ea5212ba45188144fb2b2aca5bd53007e01e15a7306a411e651e5b64aea782d8", + "state": "running" + }, + "calibre-web": { + "health": "healthy", + "id": "ac9e6b2302c502a9207fc9e7cfdc0a6999e7304730957263212a1300ed525d10", + "state": "running" + }, + "cloudflared": { + "health": "healthy", + "id": "549f296e6ef9f6594f5e8789ecb86579f0c6ee45091b38bafdca69b50377d27a", + "state": "running" + }, + "docmost": { + "health": "healthy", + "id": "b7ce8b91c0f44c389f4a9153778c910c226c0a4f1118030d6da84d4860c33da6", + "state": "running" + }, + "docmost-postgres": { + "health": "healthy", + "id": "d16587ff24479fa15fa24ce56c23d47005623483315497460140fc563a3c1e8b", + "state": "running" + }, + "docmost-redis": { + "health": "healthy", + "id": "be025075e63097a7df7fd37c56fd4034942742beff6f2ef34d83df003a36c590", + "state": "running" + }, + "felhom-controller": { + "health": "healthy", + "id": "1af906eb4abc5270c0a291b14ac1638be79c5a3fe9c47dce973a1c42d1678b0d", + "state": "running" + }, + "filebrowser": { + "health": "healthy", + "id": "f6446630708007c3b2f97a8d62043193c6516a7a62daf31037e42989a51c9393", + "state": "running" + }, + "home-assistant": { + "health": "healthy", + "id": "588d1a667f65af137effb7acbc28a220d63f4f5d417039aca9ef9df9ad07a923", + "state": "running" + }, + "kimai": { + "health": "healthy", + "id": "e5443c7db71d859c4ae97c551566e3a278511282b192abe33746baff88d62a34", + "state": "running" + }, + "kimai-db": { + "health": "healthy", + "id": "90fe1d5c54138fe8fde39fca173f5b735f6fbae23d8c6156eeacb06f716836c2", + "state": "running" + }, + "opengist": { + "health": "healthy", + "id": "0991f49cd94bea7a3f6db7c597f716e8a080de6492528db8f31ee737d783e528", + "state": "running" + }, + "paperless-postgres": { + "health": "healthy", + "id": "e5862350b148d28e5eee065a7ca4d12c2e2d64d7ccc4c40708781dd5255e9296", + "state": "running" + }, + "paperless-redis": { + "health": "healthy", + "id": "cfb5b8b9e711a12027c47b8b7a7f3d2fd74999338fec699372883c19157ddbfa", + "state": "running" + }, + "paperless-webserver": { + "health": "healthy", + "id": "74c1e55159dd9b856af7a0d1a7cdb56680fcefa195913537768f3789221c3c29", + "state": "running" + }, + "plex": { + "health": "healthy", + "id": "5091a7dd6259f31e28552895f9f714d1f7cd0938b0e525d6a03133355c50d83c", + "state": "running" + }, + "privatebin": { + "health": "healthy", + "id": "5e6c930258cb8f4e76fa0fd2ab80bad7b2a62c9cd93882edece8e096e39a29f4", + "state": "running" + }, + "romm": { + "health": "healthy", + "id": "8368d5dbb4be09989cd5101a50335ee4e80d7e139ad6a70c30d7fedd933b16d0", + "state": "running" + }, + "romm-db": { + "health": "healthy", + "id": "6bbe9a4d9bb95606052f2bcb206df17b78b3058ee68a4af55450b4e3d1cecc1c", + "state": "running" + }, + "romm-redis": { + "health": "healthy", + "id": "519aa50394d17f06467be060d732f060f3bb23e08a19aa8c6170c7198c3c3c78", + "state": "running" + }, + "traefik": { + "health": "none", + "id": "fc0bfe5b53c6a7711d5d742b8ba9c4b3fbdbf96da7270311ffcf4e23f28817c8", + "state": "running" + } + }, + "controller": "healthy", + "controller_docker_ok": true, + "docker_ok": true, + "network_ok": true + }, + "guest_running": true, + "host_services": { + "felhom-agent": "active", + "pve-cluster": "active", + "pvedaemon": "active", + "pveproxy": "active", + "pvestatd": "active" + } + }, + "judging_since": "2026-11-10T01:54:08Z", + "packages": [ + { + "name": "proxmox-kernel-7.0", + "version": "7.0.14-28" + }, + { + "name": "proxmox-kernel-7.0.14-28-pve-signed", + "version": "7.0.14-28" + } + ], + "phase": "good", + "rebooted_at": "2026-11-10T01:50:56Z", + "result_at": "2026-11-10T01:56:47Z", + "self_revert_used": false, + "staged_at": "2026-11-10T01:50:50Z", + "step_id": "ring0-20261110T014802Z", + "to": "7.0.14-28-pve", + "updated_at": "2026-11-10T01:56:47Z", + "vmid": 9201 +} + +== one-shot flag +felhom_next= +== grub default +17: set default="${next_entry}" +22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43" +# felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily +GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43" +== kernel packages (ii) +ii proxmox-default-kernel 2.1.0 +ii proxmox-kernel-7.0 7.0.14-28 +ii proxmox-kernel-7.0.14-26-pve-signed 7.0.14-26 +ii proxmox-kernel-7.0.14-28-pve-signed 7.0.14-28 +ii proxmox-kernel-helper 9.1.0+fde2 +== dpkg audit +rc=0 +== sizes +Filesystem 1M-blocks Used Available Use% Mounted on +/dev/mapper/pve-root 39506 16492 20976 45% / +/dev/sda2 1022 10 1013 1% /boot/efi +1945 /lib/modules +217 /boot +== guests +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness + VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID + 341 night1004-tester1 running 8192 200.00 1847 +== 9201 containers (name status) +romm Up About a minute (healthy) +romm-redis Up About a minute (healthy) +romm-db Up About a minute (healthy) +plex Up About a minute (healthy) +paperless-webserver Up About a minute (healthy) +paperless-redis Up 2 minutes (healthy) +paperless-postgres Up 2 minutes (healthy) +calibre-web Up 2 minutes (healthy) +felhom-controller Up 2 minutes (healthy) +privatebin Up 2 minutes (healthy) +opengist Up 2 minutes (healthy) +kimai Up 2 minutes (healthy) +kimai-db Up 2 minutes (healthy) +home-assistant Up 2 minutes (healthy) +docmost Up 2 minutes (healthy) +docmost-redis Up 2 minutes (healthy) +docmost-postgres Up 2 minutes (healthy) +bookstack Up 2 minutes (healthy) +bookstack-db Up 2 minutes (healthy) +adventurelog Up 2 minutes (healthy) +adventurelog-postgres Up 2 minutes (healthy) +adventurelog-frontend Up 2 minutes (healthy) +bentopdf Up 2 minutes (healthy) +traefik Up 2 minutes +cloudflared Up 2 minutes (healthy) +filebrowser Up 2 minutes (healthy) +== 9202 containers (name status) +felhom-controller Up 2 minutes (healthy) +filebrowser Up 2 minutes (healthy) +paperless-webserver Up 2 minutes (healthy) +paperless-postgres Up 2 minutes (healthy) +paperless-redis Up 2 minutes (healthy) +traefik Up 2 minutes +== agent log since boot (last 8, no payloads) +2026-11-10T13:58:58+01:00 demo-hp sudo[17988]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +2026-11-10T13:59:17+01:00 demo-hp sudo[19197]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +2026-11-10T13:59:17+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:59:17.882+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/f +2026-11-10T13:59:27+01:00 demo-hp sudo[19822]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +2026-11-10T13:59:28+01:00 demo-hp sudo[19859]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +2026-11-10T13:59:37+01:00 demo-hp sudo[20371]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data +2026-11-10T13:59:37+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:59:37.578+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/f +2026-11-10T13:59:41+01:00 demo-hp sudo[20807]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/14-rule-second-run.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/14-rule-second-run.txt new file mode 100644 index 00000000..ec90fa72 --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/14-rule-second-run.txt @@ -0,0 +1,47 @@ +{ + "host": "demo-hp", + "running": "7.0.14-28-pve", + "newest": "7.0.14-28-pve", + "phase": "good", + "step": { + "step_id": "ring0-20261110T014802Z", + "from": "7.0.14-26-pve", + "to": "7.0.14-28-pve", + "default_before": "7.0.14-26-pve", + "result_at": "2026-11-10T01:56:47Z" + }, + "oneshot_flag": "", + "grub_default": "7.0.14-28-pve", + "pinned_default": "7.0.14-28-pve", + "boot_pin": null, + "installed": { + "7.0.14-26-pve": "proxmox-kernel-7.0.14-26-pve-signed", + "7.0.14-28-pve": "proxmox-kernel-7.0.14-28-pve-signed" + }, + "keep": { + "7.0.14-26-pve": [ + "the last kernel that booted well before the running one (state.json `from`)", + "kernel step `from` (a self-revert boots it)", + "kernel step `default_before`" + ], + "7.0.14-28-pve": [ + "running", + "newest installed", + "kernel step `to`", + "GRUB default in grub.cfg", + "pinned default file" + ] + }, + "computed_removal": [], + "removal": [], + "pretend": [], + "simulation": { + "rc": null, + "removes": [], + "installs": [] + }, + "refuse": [], + "verdict": "NOTHING TO REMOVE" +} + +VERDICT: NOTHING TO REMOVE diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/15-hub-after.txt b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/15-hub-after.txt new file mode 100644 index 00000000..dc3b469e --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/evidence/15-hub-after.txt @@ -0,0 +1,7 @@ +# The hub's own row for demo-hp, read with CC's token (GET /hosts, read-only) at 2026-11-10T13:13:15Z — 16 minutes after the boot. +# A different channel from the box: this is what the hub holds from the agent's host report. +columns: Host | Customer | Agent | Proxmox / kernel | Status | Guests | CPU | Mem | Disk | Cloudflared | Worst Storage +row: demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 37% 42% running (connected) 71% local-lvm + +# before the run (same page, read 2026-11-10 before the removal): +row: demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 49% 58% running (connected) 71% local-lvm diff --git a/documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py b/documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py new file mode 100644 index 00000000..d320fc3a --- /dev/null +++ b/documentation/audits/kernel-cleanup-spike-2026-11-10/tools/kernel-cleanup-rule.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python3 +"""kernel-cleanup-rule.py — the old-kernel clean-up RULE, as a read-only plan. EVIDENCE, NOT PRODUCT (spike 2026-11-10). + +It runs ON the Proxmox host as root (`ssh demo-hp python3 - [options] < kernel-cleanup-rule.py`). It changes nothing: +it reads dpkg, `uname -r`, the kernel step's state file, the one-shot flag and grub.cfg, and it asks apt for a +SIMULATION (`apt-get -s`). It prints the keep set, the removal set and every reason to refuse. Exit code: 0 = the plan +is safe to run (or there is nothing to remove), 3 = REFUSED. The removal itself is one `apt-get purge` of the exact +names this prints; a person ran it by hand in the spike. + +The rule (the brief's, `11-os-updates.md` §5.11.1): + KEEP the running kernel; the last kernel that booted well before it (the step's `from`, else the next lower + installed one); the newest installed kernel; and every kernel the kernel lane still names: the step's `to`, + the GRUB default in grub.cfg, the pinned default file, the one-shot flag, an operator's proxmox-boot pin. + REMOVE every other installed kernel image package, by its exact name. + REFUSE the whole clean-up when + P1 a kernel step is in a phase that has not ended (staged, oneshot, judging, reverting) or in an unknown phase; + P2 the one-shot flag names a kernel (a one-shot boot is armed); + P3 the pinned default and grub.cfg's default do not name the same kernel, or that kernel is not installed; + K1 a keep-kernel is in the removal set; + N1 a name in the removal set is not an installed kernel image package; + S1 the simulation fails, or would install or upgrade anything; + S2 the simulation would remove anything that is not in the removal set (a meta-package, proxmox-ve, anything). + +Test inputs (negative controls, SIMULATION ONLY — nothing is ever removed by this file): + --pretend-set a,b,c use this removal set in place of the computed one + --pretend-add a,b add these names to the computed removal set + --pretend-phase NAME judge as if the kernel step were in this phase +""" +import json, os, re, subprocess, sys + +STATE = "/var/lib/felhom-kernel/state.json" +DEFAULT_CFG = "/etc/default/grub.d/zz-felhom-kernel-default.cfg" +GRUB_CFG = "/boot/grub/grub.cfg" +ONESHOT_ENV = "/boot/efi/EFI/felhom/oneshot.env" +PIN_FILE = "/etc/kernel/proxmox-boot-pin" +IMAGE_RE = re.compile(r"^proxmox-kernel-([0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve)(-signed)?$") # = the wrapper's KERNEL_IMAGE_RE +KVER_RE = re.compile(r"[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve") +ENDED = ("good", "self_reverted", "revert_failed", "fell_back", "cancelled", "none") # the wrapper's phases, minus KERNEL_ACTIVE + + +def sh(cmd): + p = subprocess.run(cmd, capture_output=True, text=True) + return p.returncode, p.stdout, p.stderr + + +def key(kver): + return tuple(int(x) for x in re.findall(r"\d+", kver)) + + +def read(path): + try: + return open(path, encoding="utf-8", errors="replace").read() + except OSError: + return None + + +def opt(name): + return sys.argv[sys.argv.index(name) + 1] if name in sys.argv else None + + +def main(): + refuse = [] + # --- what is installed (dpkg status "ii" only) --- + _, out, _ = sh(["dpkg-query", "-W", "-f", "${Package}\t${db:Status-Abbrev}\n", "proxmox-kernel-*"]) + images = {} # kver -> package name + for line in out.splitlines(): + name, _, st = line.partition("\t") + m = IMAGE_RE.match(name) + if m and st.strip() == "ii": + images[m.group(1)] = name + running = os.uname().release + newest = max(images, key=key) if images else None + # --- the kernel lane's own records --- + raw = read(STATE) + st = json.loads(raw) if raw else {} + phase = opt("--pretend-phase") or st.get("phase", "none") + if phase not in ENDED: + refuse.append(f"P1 a kernel step is in phase {phase!r} — not ended") + flag = "" + if os.path.exists(ONESHOT_ENV): + _, o, _ = sh(["grub-editenv", ONESHOT_ENV, "list"]) + flag = dict(l.partition("=")[::2] for l in o.splitlines()).get("felhom_next", "") + if flag: + refuse.append(f"P2 the one-shot flag names {flag!r}") + cfg = read(GRUB_CFG) or "" + m = re.search(r'^\s*set default="[^"]*?(%s)[^"]*"' % KVER_RE.pattern, cfg, re.M) + grub_default = m.group(1) if m else None + m = KVER_RE.search(read(DEFAULT_CFG) or "") + pinned = m.group(0) if m else None + if pinned is None or pinned != grub_default or pinned not in images: + refuse.append(f"P3 pinned default {pinned!r}, grub.cfg default {grub_default!r} — not the same installed kernel") + m = KVER_RE.search(read(PIN_FILE) or "") + boot_pin = m.group(0) if m else None + # --- the keep set --- + keep = {} + def k(kver, why): + if kver: + keep.setdefault(kver, []).append(why) + k(running, "running") + last_good = st.get("from") if st.get("to") == running and st.get("from") in images else None + if not last_good: + lower = [v for v in images if key(v) < key(running)] + last_good = max(lower, key=key) if lower else None + k(last_good, "next lower installed kernel (no step record names the last good one)") + else: + k(last_good, "the last kernel that booted well before the running one (state.json `from`)") + k(newest, "newest installed") + k(st.get("from"), "kernel step `from` (a self-revert boots it)") + k(st.get("to"), "kernel step `to`") + k(st.get("default_before"), "kernel step `default_before`") + k(grub_default, "GRUB default in grub.cfg") + k(pinned, "pinned default file") + k(flag and (KVER_RE.search(flag) or [None])[0], "one-shot flag") + k(boot_pin, "operator's proxmox-boot pin") + if running not in images: + refuse.append(f"K1 the running kernel {running} is not an installed image package — the box is not as expected") + # --- the removal set: exact package names --- + computed = sorted((images[v] for v in images if v not in keep), key=lambda n: key(IMAGE_RE.match(n).group(1))) + removal = list(computed) + if opt("--pretend-set") is not None: + removal = [x for x in opt("--pretend-set").split(",") if x] + if opt("--pretend-add"): + removal += [x for x in opt("--pretend-add").split(",") if x] + by_name = {n: v for v, n in images.items()} + for n in removal: + if n not in by_name: + refuse.append(f"N1 {n} is not an installed kernel image package") + elif by_name[n] in keep: + refuse.append(f"K1 {n} is a keep-kernel ({'; '.join(keep[by_name[n]])})") + # --- apt's simulation of exactly that set --- + sim = {"rc": None, "removes": [], "installs": []} + if removal: + rc, o, e = sh(["apt-get", "-s", "-q", "-o", "Debug::NoLocking=1", "purge"] + removal) + sim["rc"] = rc + sim["removes"] = sorted({l.split()[1] for l in o.splitlines() if l.startswith(("Remv ", "Purg "))}) + sim["installs"] = sorted({l.split()[1] for l in o.splitlines() if l.startswith(("Inst ", "Conf "))}) + if rc != 0: + refuse.append("S1 the simulation failed: " + ((e or o).strip().splitlines() or ["?"])[-1][:200]) + if sim["installs"]: + refuse.append(f"S1 the simulation would install or configure {', '.join(sim['installs'][:6])}") + extra = [p for p in sim["removes"] if p not in removal] + if extra: + refuse.append(f"S2 the simulation would also remove {', '.join(extra[:8])} — not in the removal set") + missing = [p for p in removal if p not in sim["removes"] and p in by_name] + if missing and rc == 0: + refuse.append(f"S1 the simulation does not remove {', '.join(missing[:6])}") + verdict = "REFUSED" if refuse else ("NOTHING TO REMOVE" if not removal else "SAFE TO RUN") + res = {"host": os.uname().nodename, "running": running, "newest": newest, "phase": phase, "step": {x: st.get(x) for x in + ("step_id", "from", "to", "default_before", "result_at")}, "oneshot_flag": flag, "grub_default": grub_default, + "pinned_default": pinned, "boot_pin": boot_pin, "installed": {v: images[v] for v in sorted(images, key=key)}, + "keep": {v: keep[v] for v in sorted(keep, key=key)}, "computed_removal": computed, "removal": removal, + "pretend": [a for a in sys.argv[1:]], "simulation": sim, "refuse": refuse, "verdict": verdict} + print(json.dumps(res, indent=1)) + print(f"\nVERDICT: {verdict}") + for r in refuse: + print(" REFUSE " + r) + if not refuse and removal: + print(" the one command: apt-get -y -q purge " + " ".join(removal)) + return 3 if refuse else 0 + + +sys.exit(main()) diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index 440109c2..7bdbf717 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -281,7 +281,7 @@ stopping line that lies. | **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC | | **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC | | **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC | -| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC | +| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC | | **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC | | **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. | — | Docs: a short „disk space” section in an architecture document: each part of the system disk with its limit and its cleaner, pointing at the audit table. Then the installer can refuse a disk that cannot hold the sum. | CC | | **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |