Part B: the old-kernel clean-up rule, proven by hand on demo-hp (spike; nothing built)
gates / gates (push) Successful in 2m1s

One run on the demo-hp host (Tier 0), decision 286. The rule removed six of
eight kernel image packages and nothing else: 6.4 GB gained on the system
disk, every boot setting unchanged, one orderly reboot back on the same
kernel, all containers healthy 311 s after the command, the crash guard saw
a clean stop, a second run removed nothing. Five refusals shown by
simulation only.

- documentation/audits/kernel-cleanup-spike-2026-11-10/: README (one page),
  evidence 00-15, the rule as a read-only plan script
- 11-os-updates.md §5.11.1: the design of the step (DESIGN, not built)
- R-1368: a dated pointer line

felhom-os-apply, the config bundle, the agent and rule R4 are unchanged.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-10 14:13:41 +01:00
1 parent 807175c27d
commit 10ea4c2cbe
20 files changed
+2187 -1

No files matched your search

@@ -600,6 +600,46 @@ another kernel), and the kernel step (the newest result, the kernel due, whether
7.0.14-20), so a soft lockup turning into a panic is recorded as unmeasured (no tool was built). A true dead freeze
needs a person (accepted, decision 172). Evidence and proofs: `audits/kernel-lane-2026-10-07/`.
### 5.11.1 Removing old kernels `[DESIGN — proven by hand on demo-hp 2026-11-10; not built]`
**The ruling** (`09` §3 decision 286): the box may remove old kernels by itself, as its own step with its own narrow
rule. **Rule R4 does not change**: an update plan that removes a package is still refused. Why a step is needed: the
kernel step installs each image by name, so apt marks it "manual" and apt's own clean-up never takes it; and nothing
runs that clean-up on a box. A ring-0 box got 7 kernels in 37 days, about 1 GB each (R-1368).
**The rule.** Keep the running kernel; the last kernel that booted well before it (the step's `from`); the newest
installed kernel; and every kernel the lane still names — the step's `to` and `default_before`, the default in
`grub.cfg`, the pinned default file, the one-shot flag, an operator's boot pin. Remove every other installed kernel
image package, by its exact name, in one `apt-get purge`.
**It refuses the whole clean-up** — it never removes a part of the set — when:
| | |
|---|---|
| P1 | a kernel step is in a phase that has not ended (`staged`, `oneshot`, `judging`, `reverting`), or in a phase it does not know |
| P2 | the one-shot flag names a kernel |
| P3 | the pinned default and `grub.cfg`'s default do not name the same installed kernel |
| K1 | a keep-kernel is in the removal set, or the running kernel is not an installed package |
| N1 | a name in the removal set is not an installed kernel image package |
| S1 | apt's simulation of exactly that set fails, or would install anything |
| S2 | the simulation would remove anything that is not in the removal set |
K1 is the rule's own check and must stay its own: asked to remove the running signed kernel, apt does not refuse — it
plans to put the unsigned image in its place (control A of the spike).
**When it may run.** Only when the newest kernel step ended `good` and the box runs its newest installed kernel — that
is "booted well on its newest kernel" (decision 286). Never in the night of a kernel step: the step's judging, and a
self-revert into `from`, must be over first. In the night's order it belongs where the OS leg is (§5.5): after the
whole-guest backup. It needs no reboot and stops no guest. A box that fell back or reverted keeps all its kernels
until a later step ends `good`.
**Proven** on demo-hp by hand: 8 kernels → 2, 6.4 GB gained on the system disk, the default, the pinned file and both
generators unchanged, five refusals by simulation, one orderly reboot back onto the same kernel with a clean stop for
the crash guard (§5.9), a second run that removed nothing. **Not proven:** a box whose running kernel is not its
newest; a boot of the kept old kernel after the clean-up; the step as a program at night; a box that boots with
`proxmox-boot-tool`. Evidence, the rule as a read-only script, and the full "not measured" list:
`audits/kernel-cleanup-spike-2026-11-10/`.
## 6. Risks and edge cases
| # | What can go wrong | What the design does |
@@ -0,0 +1,96 @@
# Old kernels: the clean-up rule, proven by hand on demo-hp — 2026-11-10
A spike. One person-driven run on the demo-hp host (Tier 0). Nothing was built: `felhom-os-apply`, the config bundle
and the agent are unchanged, and rule R4 is unchanged. The ruling that asked for this proof is `09` §3 decision 286;
the design it feeds is `11-os-updates.md` §5.11.1; the register row is R-1368. Files: `evidence/` (numbered, in the
order of the run), `tools/kernel-cleanup-rule.py` (the rule as a read-only plan).
## The answer
**Yes. The rule removed six old kernels and nothing else, the box kept every boot setting, and it came back from one
orderly reboot on the same kernel.** A second run of the rule removed nothing.
## Why the old kernels were still there (`evidence/01`, `02`)
1. The kernel step installs each kernel image **by its name** (`AG/configs/felhom-os-apply`, the `install` line of the
kernel step). apt marks a package that is named in an install as "manual". apt's own clean-up (`autoremove`) removes
only packages marked "automatic". Seven of the eight kernels were manual; the simulation of `autoremove` removed one.
2. Nothing runs `autoremove` on a box in any case, and an update plan that removes a package is refused (rule R4).
So the brief's guess is true: the step names its packages, and that keeps them. It is one of two causes.
## The rule, as proven
| | |
|---|---|
| **Keep** | the running kernel · the last kernel that booted well before it (the step's `from`; if no step record, the next lower one) · the newest installed kernel · every kernel the kernel lane still names: the step's `to` and `default_before`, the default in `grub.cfg`, the pinned default file, the one-shot flag, an operator's boot pin |
| **Remove** | every other installed kernel image package, by its exact name, with one `apt-get purge` |
| **Refuse the whole clean-up** | `P1` a kernel step is in a phase that has not ended · `P2` the one-shot flag names a kernel · `P3` the pinned default and `grub.cfg` do not name the same installed kernel · `K1` a keep-kernel is in the removal set · `N1` a name that is not an installed kernel image · `S1` apt's simulation fails, or would install anything · `S2` the simulation would remove anything that is not in the removal set |
On demo-hp: keep `7.0.14-28-pve` (running, newest, default) and `7.0.14-26-pve` (the last good one); remove 7.0.2-6 and
7.0.14-20, -22, -23, -24, -25 (`evidence/03`).
**Refusals, by simulation only — nothing was removed in these runs** (`evidence/04`, each ended `REFUSED`, exit 3):
| Control | Input | Refused by |
|---|---|---|
| A | the running kernel in the removal set | `K1`, `S1` |
| B | the last good kernel in the removal set | `K1` |
| C | a kernel step in phase `judging` | `P1` |
| D | the meta-package `proxmox-kernel-7.0` | `N1`, `S2` (apt would also remove `proxmox-ve`) |
| E | an old kernel and `pve-firmware` | `N1`, `S2` |
The brief asked for two controls: the running kernel and the pinned default. On this box they are the same kernel
(control A). B to E were added. **Control A shows why the rule cannot lean on apt:** apt does not refuse to remove the
running signed kernel — it plans to put the unsigned image in its place. Only the rule's own check stops it.
## The run (`evidence/05` to `15`)
| What | Before | After | Proof |
|---|---|---|---|
| Kernel image packages | 8 | 2 | `05`, `08` |
| System disk `/`, used | 22,906 MB (62%) | 16,495 MB (45%) — **6,411 MB gained** | `05`, `08` |
| `/lib/modules` | 7,772 MB | 1,945 MB | `05`, `08` |
| `/boot` | 795 MB | 217 MB | `05`, `08` |
| Default in `grub.cfg`, pinned default file, both one-shot generators | the `28` entry; three checksums | the same entry; the same three checksums | `05`, `08` |
| Kernel step state, one-shot flag | `good`, 26 → 28; empty | the same | `08`, `13` |
| `dpkg --audit`; simulated `autoremove` | — | empty; 0 to remove | `08` |
The removal took 50 s and ended with code 0 (`07`). Every kernel file that `grub.cfg` names exists (`08`).
**The reboot** (`09` to `13`; sent 12:54:26 UTC with `systemctl reboot`):
| Moment | Seconds after the command |
|---|---|
| The old boot ended (journal stopped) | 129 |
| The new boot started | 153 |
| The host answered ping again; the hub showed a new host report | 172 at most |
| Every container healthy in guest 9201 (26) and 9202 (6) | 311 (read every 19 s) |
After it: kernel `7.0.14-28-pve`, a new boot id, system state `running`, the agent and the Proxmox services active,
guests 9201 and 9202 and the Tester 1 VM running, guest 9401 stopped as before. **The crash guard saw a clean stop:**
`last_boot_unclean: false`, no unclean boot in its list (`13`). The second run of the rule: `NOTHING TO REMOVE` (`14`).
For comparison, the reboot of 2026-10-07 took 272 s to all healthy.
**The hub agrees** (its own page, 16 minutes after the boot, `15`): `ONLINE`, tunnel connected, system disk 58% before
and 42% after.
## Choices made in the run
- `purge`, not `remove`: no `/lib/modules/<version>` folder and no half-removed package state stays.
- The removal used the wrapper's own apt settings (no prompt, keep config files).
- `felhom-os-apply --kernel-status` needs a plan file in the agent's folder. The run read its inputs directly (the
state file, the flag, `grub.cfg`).
## Not measured
- A box whose running kernel is **not** its newest one (after a fall-back or a self-revert). The rule keeps both; no
such box was tried.
- A real kernel step that has not ended. Control C is a pretended phase.
- **A boot of the kept old kernel (26) after the clean-up.** Its files and its menu entries exist and `grub.cfg`
passes the syntax check; it was not booted.
- The N100 and the Tester 1 VM (8 and 3 kernels). Read-only in this run.
- The step as a program, at night, next to the agent's own apt runs. A person checked the lock by hand.
- A box that boots with `proxmox-boot-tool` (demo-hp does not use it), and a box with out-of-tree kernel modules
(demo-hp has none).
- A power cut in the middle of the removal.
- apt's own keep-list still holds a pattern for 7.0.14-23, a kernel that is gone. Harmless here; not followed up.
@@ -0,0 +1,26 @@
== when
2026-11-10T12:43:38Z
demo-hp
13:43:38 up 10:49, 1 user, load average: 0.95, 1.16, 1.62
== running
7.0.14-28-pve
== guests
VMID Status Lock Name
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
341 night1004-tester1 running 8192 200.00 1861
== busy?
none of: vzdump pbs-restore proxmox-backup-client apt-get dpkg felhom-os-apply restic
running PVE tasks: 0 []
== df
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/pve-root 39G 23G 15G 62% /
/dev/sda2 1022M 9.1M 1013M 1% /boot/efi
== agent
active
active
active
active
felhom-agent 0.172.1
@@ -0,0 +1,331 @@
== running kernel
7.0.14-28-pve
== kernel packages (dpkg)
un linux-image-7.0.14-20-pve-amd64
un linux-image-7.0.14-22-pve-amd64
un linux-image-7.0.14-23-pve-amd64
un linux-image-7.0.14-24-pve-amd64
un linux-image-7.0.14-25-pve-amd64
un linux-image-7.0.14-26-pve-amd64
un linux-image-7.0.14-28-pve-amd64
un linux-image-7.0.2-6-pve-amd64
un linux-image-amd64
un linux-image-generic
ii proxmox-default-kernel 2.1.0 10
ii proxmox-kernel-7.0 7.0.14-28 37
un proxmox-kernel-7.0.14-20-pve
ii proxmox-kernel-7.0.14-20-pve-signed 7.0.14-20 1006525
un proxmox-kernel-7.0.14-22-pve
ii proxmox-kernel-7.0.14-22-pve-signed 7.0.14-22 1006522
un proxmox-kernel-7.0.14-23-pve
ii proxmox-kernel-7.0.14-23-pve-signed 7.0.14-23 1006525
un proxmox-kernel-7.0.14-24-pve
ii proxmox-kernel-7.0.14-24-pve-signed 7.0.14-24 1006528
un proxmox-kernel-7.0.14-25-pve
ii proxmox-kernel-7.0.14-25-pve-signed 7.0.14-25 1007819
un proxmox-kernel-7.0.14-26-pve
ii proxmox-kernel-7.0.14-26-pve-signed 7.0.14-26 1007815
un proxmox-kernel-7.0.14-28-pve
ii proxmox-kernel-7.0.14-28-pve-signed 7.0.14-28 1007817
un proxmox-kernel-7.0.2-6-pve
ii proxmox-kernel-7.0.2-6-pve-signed 7.0.2-6 1005219
ii proxmox-kernel-helper 9.1.0+fde2 109
ii pve-firmware 3.18-3 543876
un pve-kernel-7.0
un pve-kernel-helper
== manual marks
proxmox-default-kernel
proxmox-kernel-7.0
proxmox-kernel-7.0.14-22-pve-signed
proxmox-kernel-7.0.14-23-pve-signed
proxmox-kernel-7.0.14-24-pve-signed
proxmox-kernel-7.0.14-25-pve-signed
proxmox-kernel-7.0.14-26-pve-signed
proxmox-kernel-7.0.14-28-pve-signed
proxmox-kernel-7.0.2-6-pve-signed
proxmox-kernel-helper
== auto marks
proxmox-kernel-7.0.14-20-pve-signed
== holds
== simulated autoremove
Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be REMOVED:
proxmox-kernel-7.0.14-20-pve-signed
0 upgraded, 0 newly installed, 1 to remove and 16 not upgraded.
Remv proxmox-kernel-7.0.14-20-pve-signed [7.0.14-20]
== apt never-remove list
APT::NeverAutoRemove "";
APT::NeverAutoRemove:: "^firmware-linux.*";
APT::NeverAutoRemove:: "^linux-firmware$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*$";
APT::NeverAutoRemove:: "^linux-image-[a-z0-9]*-[a-z0-9]*$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-26-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-28-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve-signed$";
APT::VersionedKernelPackages "";
APT::VersionedKernelPackages:: "linux-.*";
APT::VersionedKernelPackages:: "kfreebsd-.*";
APT::VersionedKernelPackages:: "gnumach-.*";
APT::VersionedKernelPackages:: ".*-modules";
APT::VersionedKernelPackages:: ".*-kernel";
== files that set it
-rw-r--r-- 1 root root 399 Jun 24 2025 01autoremove
total 32
drwxr-xr-x 2 root root 4096 Aug 21 17:42 .
drwxr-xr-x 5 root root 4096 Oct 4 09:47 ..
-rwxr-xr-x 1 root root 951 May 13 2025 initramfs-tools
-rwxr-xr-x 1 root root 1023 May 5 2026 proxmox-auto-removal
-rwxr-xr-x 1 root root 11627 May 5 2026 zz-proxmox-boot
-rwxr-xr-x 1 root root 646 Mar 16 2023 zz-update-grub
== pinned default
# felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily
GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
== grub.cfg default line
17: set default="${next_entry}"
22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
110: if [ "${felhom_boot}" = "7.0.14-20-pve" ]; then set default="felhom-oneshot-7.0.14-20-pve"; fi
111: if [ "${felhom_boot}" = "7.0.14-22-pve" ]; then set default="felhom-oneshot-7.0.14-22-pve"; fi
112: if [ "${felhom_boot}" = "7.0.14-23-pve" ]; then set default="felhom-oneshot-7.0.14-23-pve"; fi
113: if [ "${felhom_boot}" = "7.0.14-24-pve" ]; then set default="felhom-oneshot-7.0.14-24-pve"; fi
114: if [ "${felhom_boot}" = "7.0.14-25-pve" ]; then set default="felhom-oneshot-7.0.14-25-pve"; fi
115: if [ "${felhom_boot}" = "7.0.14-26-pve" ]; then set default="felhom-oneshot-7.0.14-26-pve"; fi
116: if [ "${felhom_boot}" = "7.0.14-28-pve" ]; then set default="felhom-oneshot-7.0.14-28-pve"; fi
117: if [ "${felhom_boot}" = "7.0.2-6-pve" ]; then set default="felhom-oneshot-7.0.2-6-pve"; fi
== one-shot flag
felhom_next=
rc=0
== kernel step state
{
"authority": "ring0",
"default_before": "7.0.14-26-pve",
"from": "7.0.14-26-pve",
"health_before": {
"guest": {
"containers": {
"adventurelog": {
"health": "healthy",
"id": "50af393c36a37d01a10847456d391d70572c121045e28626f220f83cec1cefcc",
"state": "running"
},
"adventurelog-frontend": {
"health": "healthy",
"id": "abd6155bdad527c65b82f75bc0dd0f98f9690ec81587c8cd07cea6c9c019cabe",
"state": "running"
},
"adventurelog-postgres": {
"health": "healthy",
"id": "841257c7abb7b545567aa8fa539cdcd421c94ce4edf97a0dea5e45831ea399f4",
"state": "running"
},
"bentopdf": {
"health": "healthy",
"id": "9db9aba521daf9f19352a64511f4f84872ade957b643c0db40dd510837143cba",
"state": "running"
},
"bookstack": {
"health": "healthy",
"id": "c83bbf5562cb68ee23d71ece2362d8be482ac4d7d59ad9bdfa5493f0b53c9687",
"state": "running"
},
"bookstack-db": {
"health": "healthy",
"id": "ea5212ba45188144fb2b2aca5bd53007e01e15a7306a411e651e5b64aea782d8",
"state": "running"
},
"calibre-web": {
"health": "healthy",
"id": "ac9e6b2302c502a9207fc9e7cfdc0a6999e7304730957263212a1300ed525d10",
"state": "running"
},
"cloudflared": {
"health": "healthy",
"id": "549f296e6ef9f6594f5e8789ecb86579f0c6ee45091b38bafdca69b50377d27a",
"state": "running"
},
"docmost": {
"health": "healthy",
"id": "b7ce8b91c0f44c389f4a9153778c910c226c0a4f1118030d6da84d4860c33da6",
"state": "running"
},
"docmost-postgres": {
"health": "healthy",
"id": "d16587ff24479fa15fa24ce56c23d47005623483315497460140fc563a3c1e8b",
"state": "running"
},
"docmost-redis": {
"health": "healthy",
"id": "be025075e63097a7df7fd37c56fd4034942742beff6f2ef34d83df003a36c590",
"state": "running"
},
"felhom-controller": {
"health": "healthy",
"id": "1af906eb4abc5270c0a291b14ac1638be79c5a3fe9c47dce973a1c42d1678b0d",
"state": "running"
},
"filebrowser": {
"health": "healthy",
"id": "f6446630708007c3b2f97a8d62043193c6516a7a62daf31037e42989a51c9393",
"state": "running"
},
"home-assistant": {
"health": "healthy",
"id": "588d1a667f65af137effb7acbc28a220d63f4f5d417039aca9ef9df9ad07a923",
"state": "running"
},
"kimai": {
"health": "healthy",
"id": "e5443c7db71d859c4ae97c551566e3a278511282b192abe33746baff88d62a34",
"state": "running"
},
"kimai-db": {
"health": "healthy",
"id": "90fe1d5c54138fe8fde39fca173f5b735f6fbae23d8c6156eeacb06f716836c2",
"state": "running"
},
"opengist": {
"health": "healthy",
"id": "0991f49cd94bea7a3f6db7c597f716e8a080de6492528db8f31ee737d783e528",
"state": "running"
},
"paperless-postgres": {
"health": "healthy",
"id": "e5862350b148d28e5eee065a7ca4d12c2e2d64d7ccc4c40708781dd5255e9296",
"state": "running"
},
"paperless-redis": {
"health": "healthy",
"id": "cfb5b8b9e711a12027c47b8b7a7f3d2fd74999338fec699372883c19157ddbfa",
"state": "running"
},
"paperless-webserver": {
"health": "healthy",
"id": "74c1e55159dd9b856af7a0d1a7cdb56680fcefa195913537768f3789221c3c29",
"state": "running"
},
"plex": {
"health": "healthy",
"id": "5091a7dd6259f31e28552895f9f714d1f7cd0938b0e525d6a03133355c50d83c",
"state": "running"
},
"privatebin": {
"health": "healthy",
"id": "5e6c930258cb8f4e76fa0fd2ab80bad7b2a62c9cd93882edece8e096e39a29f4",
"state": "running"
},
"romm": {
"health": "healthy",
"id": "8368d5dbb4be09989cd5101a50335ee4e80d7e139ad6a70c30d7fedd933b16d0",
"state": "running"
},
"romm-db": {
"health": "healthy",
"id": "6bbe9a4d9bb95606052f2bcb206df17b78b3058ee68a4af55450b4e3d1cecc1c",
"state": "running"
},
"romm-redis": {
"health": "healthy",
"id": "519aa50394d17f06467be060d732f060f3bb23e08a19aa8c6170c7198c3c3c78",
"state": "running"
},
"traefik": {
"health": "none",
"id": "fc0bfe5b53c6a7711d5d742b8ba9c4b3fbdbf96da7270311ffcf4e23f28817c8",
"state": "running"
}
},
"controller": "healthy",
"controller_docker_ok": true,
"docker_ok": true,
"network_ok": true
},
"guest_running": true,
"host_services": {
"felhom-agent": "active",
"pve-cluster": "active",
"pvedaemon": "active",
"pveproxy": "active",
"pvestatd": "active"
}
},
"judging_since": "2026-11-10T01:54:08Z",
"packages": [
{
"name": "proxmox-kernel-7.0",
"version": "7.0.14-28"
},
{
"name": "proxmox-kernel-7.0.14-28-pve-signed",
"version": "7.0.14-28"
}
],
"phase": "good",
"rebooted_at": "2026-11-10T01:50:56Z",
"result_at": "2026-11-10T01:56:47Z",
"self_revert_used": false,
"staged_at": "2026-11-10T01:50:50Z",
"step_id": "ring0-20261110T014802Z",
"to": "7.0.14-28-pve",
"updated_at": "2026-11-10T01:56:47Z",
"vmid": 9201
}
== one-shot entries in grub.cfg
felhom-oneshot-7.0.14-20-pve
felhom-oneshot-7.0.14-22-pve
felhom-oneshot-7.0.14-23-pve
felhom-oneshot-7.0.14-24-pve
felhom-oneshot-7.0.14-25-pve
felhom-oneshot-7.0.14-26-pve
felhom-oneshot-7.0.14-28-pve
felhom-oneshot-7.0.2-6-pve
== menu entries (kernels)
gnulinux-7.0.14-20-pve-advanced
gnulinux-7.0.14-22-pve-advanced
gnulinux-7.0.14-23-pve-advanced
gnulinux-7.0.14-24-pve-advanced
gnulinux-7.0.14-25-pve-advanced
gnulinux-7.0.14-26-pve-advanced
gnulinux-7.0.14-28-pve-advanced
gnulinux-7.0.2-6-pve-advanced
== /lib/modules
972 /lib/modules/7.0.14-20-pve
972 /lib/modules/7.0.14-22-pve
972 /lib/modules/7.0.14-23-pve
972 /lib/modules/7.0.14-24-pve
973 /lib/modules/7.0.14-25-pve
973 /lib/modules/7.0.14-26-pve
973 /lib/modules/7.0.14-28-pve
970 /lib/modules/7.0.2-6-pve
7772 /lib/modules
== /boot
786 /boot
-rw-r--r-- 1 root root 73094554 Oct 10 04:41 initrd.img-7.0.14-20-pve
-rw-r--r-- 1 root root 73101983 Oct 10 04:42 initrd.img-7.0.14-22-pve
-rw-r--r-- 1 root root 73131609 Oct 11 04:42 initrd.img-7.0.14-23-pve
-rw-r--r-- 1 root root 73115537 Oct 16 04:49 initrd.img-7.0.14-24-pve
-rw-r--r-- 1 root root 73286572 Oct 21 04:45 initrd.img-7.0.14-25-pve
-rw-r--r-- 1 root root 73492957 Oct 29 05:07 initrd.img-7.0.14-26-pve
-rw-r--r-- 1 root root 73547547 Nov 10 02:50 initrd.img-7.0.14-28-pve
-rw-r--r-- 1 root root 77353337 Oct 4 09:32 initrd.img-7.0.2-6-pve
-rw-r--r-- 1 root root 16284776 Sep 24 12:43 vmlinuz-7.0.14-20-pve
-rw-r--r-- 1 root root 16288872 Oct 6 07:41 vmlinuz-7.0.14-22-pve
-rw-r--r-- 1 root root 16288872 Oct 7 22:25 vmlinuz-7.0.14-23-pve
-rw-r--r-- 1 root root 16288872 Oct 9 13:32 vmlinuz-7.0.14-24-pve
-rw-r--r-- 1 root root 16284776 Oct 17 22:42 vmlinuz-7.0.14-25-pve
-rw-r--r-- 1 root root 16280680 Oct 22 20:45 vmlinuz-7.0.14-26-pve
-rw-r--r-- 1 root root 16280680 Nov 2 12:54 vmlinuz-7.0.14-28-pve
-rw-r--r-- 1 root root 16227432 May 20 10:55 vmlinuz-7.0.2-6-pve
== crash guard
active
kernel.panic = 10
total 12
drwxr-xr-x 2 root root 4096 Nov 10 13:09 .
drwxr-xr-x 46 root root 4096 Oct 10 04:42 ..
-rw-r--r-- 1 root root 679 Nov 10 13:09 state.json
@@ -0,0 +1,47 @@
== what installed each kernel (apt history: command line + the kernel packages it installed)
2026-10-04 09:42:20 | Commandline: apt-get install -y -q -o Dpkg::Options::=--force-confold proxmox-kernel-7.0 | Install: proxmox-kernel-7.0.14-20-pve-signed:amd64
2026-10-10 04:41:43 | Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --no-install-recommends proxmox-kernel-7.0=7. | Install: proxmox-kernel-7.0.14-22-pve-signed:amd64
2026-10-11 04:42:11 | Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --no-install-recommends proxmox-kernel-7.0=7. | Install: proxmox-kernel-7.0.14-23-pve-signed:amd64
2026-10-16 04:48:58 | Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --no-install-recommends proxmox-kernel-7.0=7. | Install: proxmox-kernel-7.0.14-24-pve-signed:amd64
2026-10-21 04:45:13 | Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --no-install-recommends proxmox-kernel-7.0=7. | Install: proxmox-kernel-7.0.14-25-pve-signed:amd64
2026-10-29 05:06:42 | Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --no-install-recommends proxmox-kernel-7.0=7. | Install: proxmox-kernel-7.0.14-26-pve-signed:amd64
2026-11-10 02:50:13 | Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef install --no-install-recommends proxmox-kernel-7.0=7. | Install: proxmox-kernel-7.0.14-28-pve-signed:amd64
== installer-era record
2026-10-04 09:42:20 install proxmox-kernel-7.0.14-20-pve-signed:amd64 <none> 7.0.14-20
2026-08-21 15:41:58 install proxmox-kernel-7.0.2-6-pve-signed:amd64 <none> 7.0.2-6
== boots (last 12) and their kernel
boot 0 2026-11-10T02:53:57+01:00 7.0.14-28-pve
== proxmox-boot-tool
Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
E: /etc/kernel/proxmox-boot-uuids does not exist.
Manually selected kernels:
None.
Automatically selected kernels:
7.0.14-26-pve
7.0.14-28-pve
== can each be installed back? (apt-cache policy)
proxmox-kernel-7.0.2-6-pve-signed: Installed: 7.0.2-6 Candidate: 7.0.2-6 *** 7.0.2-6 500 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages
proxmox-kernel-7.0.14-20-pve-signed: Installed: 7.0.14-20 Candidate: 7.0.14-20 *** 7.0.14-20 500 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages
proxmox-kernel-7.0.14-22-pve-signed: Installed: 7.0.14-22 Candidate: 7.0.14-22 *** 7.0.14-22 500 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages
proxmox-kernel-7.0.14-23-pve-signed: Installed: 7.0.14-23 Candidate: 7.0.14-23 *** 7.0.14-23 500 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages
proxmox-kernel-7.0.14-24-pve-signed: Installed: 7.0.14-24 Candidate: 7.0.14-24 *** 7.0.14-24 500 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages
proxmox-kernel-7.0.14-25-pve-signed: Installed: 7.0.14-25 Candidate: 7.0.14-25 *** 7.0.14-25 500 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages
proxmox-kernel-7.0.14-26-pve-signed: Installed: 7.0.14-26 Candidate: 7.0.14-26 *** 7.0.14-26 500 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages
proxmox-kernel-7.0.14-28-pve-signed: Installed: 7.0.14-28 Candidate: 7.0.14-28 *** 7.0.14-28 500 500 http://download.proxmox.com/debian/pve trixie/pve-no-subscription amd64 Packages
== reverse dependencies of an old image
proxmox-kernel-7.0.14-25-pve-signed
Reverse Depends:
|proxmox-kernel-7.0
proxmox-kernel-7.0
Depends: pve-firmware
|Depends: proxmox-kernel-7.0.14-28-pve-signed
Depends: proxmox-kernel-7.0.14-28-pve
proxmox-kernel-7.0.14-28-pve-signed
== dkms / extra modules
no dkms
/etc/kernel/postrm.d/:
initramfs-tools
proxmox-auto-removal
zz-proxmox-boot
zz-update-grub
@@ -0,0 +1,76 @@
2026-11-10T12:50:17Z
{
"host": "demo-hp",
"running": "7.0.14-28-pve",
"newest": "7.0.14-28-pve",
"phase": "good",
"step": {
"step_id": "ring0-20261110T014802Z",
"from": "7.0.14-26-pve",
"to": "7.0.14-28-pve",
"default_before": "7.0.14-26-pve",
"result_at": "2026-11-10T01:56:47Z"
},
"oneshot_flag": "",
"grub_default": "7.0.14-28-pve",
"pinned_default": "7.0.14-28-pve",
"boot_pin": null,
"installed": {
"7.0.2-6-pve": "proxmox-kernel-7.0.2-6-pve-signed",
"7.0.14-20-pve": "proxmox-kernel-7.0.14-20-pve-signed",
"7.0.14-22-pve": "proxmox-kernel-7.0.14-22-pve-signed",
"7.0.14-23-pve": "proxmox-kernel-7.0.14-23-pve-signed",
"7.0.14-24-pve": "proxmox-kernel-7.0.14-24-pve-signed",
"7.0.14-25-pve": "proxmox-kernel-7.0.14-25-pve-signed",
"7.0.14-26-pve": "proxmox-kernel-7.0.14-26-pve-signed",
"7.0.14-28-pve": "proxmox-kernel-7.0.14-28-pve-signed"
},
"keep": {
"7.0.14-26-pve": [
"the last kernel that booted well before the running one (state.json `from`)",
"kernel step `from` (a self-revert boots it)",
"kernel step `default_before`"
],
"7.0.14-28-pve": [
"running",
"newest installed",
"kernel step `to`",
"GRUB default in grub.cfg",
"pinned default file"
]
},
"computed_removal": [
"proxmox-kernel-7.0.2-6-pve-signed",
"proxmox-kernel-7.0.14-20-pve-signed",
"proxmox-kernel-7.0.14-22-pve-signed",
"proxmox-kernel-7.0.14-23-pve-signed",
"proxmox-kernel-7.0.14-24-pve-signed",
"proxmox-kernel-7.0.14-25-pve-signed"
],
"removal": [
"proxmox-kernel-7.0.2-6-pve-signed",
"proxmox-kernel-7.0.14-20-pve-signed",
"proxmox-kernel-7.0.14-22-pve-signed",
"proxmox-kernel-7.0.14-23-pve-signed",
"proxmox-kernel-7.0.14-24-pve-signed",
"proxmox-kernel-7.0.14-25-pve-signed"
],
"pretend": [],
"simulation": {
"rc": 0,
"removes": [
"proxmox-kernel-7.0.14-20-pve-signed",
"proxmox-kernel-7.0.14-22-pve-signed",
"proxmox-kernel-7.0.14-23-pve-signed",
"proxmox-kernel-7.0.14-24-pve-signed",
"proxmox-kernel-7.0.14-25-pve-signed",
"proxmox-kernel-7.0.2-6-pve-signed"
],
"installs": []
},
"refuse": [],
"verdict": "SAFE TO RUN"
}
VERDICT: SAFE TO RUN
the one command: apt-get -y -q purge proxmox-kernel-7.0.2-6-pve-signed proxmox-kernel-7.0.14-20-pve-signed proxmox-kernel-7.0.14-22-pve-signed proxmox-kernel-7.0.14-23-pve-signed proxmox-kernel-7.0.14-24-pve-signed proxmox-kernel-7.0.14-25-pve-signed
@@ -0,0 +1,414 @@
=== CONTROL A running kernel (also the pinned default here) in the removal set :: python3 - --pretend-add proxmox-kernel-7.0.14-28-pve-signed
{
"host": "demo-hp",
"running": "7.0.14-28-pve",
"newest": "7.0.14-28-pve",
"phase": "good",
"step": {
"step_id": "ring0-20261110T014802Z",
"from": "7.0.14-26-pve",
"to": "7.0.14-28-pve",
"default_before": "7.0.14-26-pve",
"result_at": "2026-11-10T01:56:47Z"
},
"oneshot_flag": "",
"grub_default": "7.0.14-28-pve",
"pinned_default": "7.0.14-28-pve",
"boot_pin": null,
"installed": {
"7.0.2-6-pve": "proxmox-kernel-7.0.2-6-pve-signed",
"7.0.14-20-pve": "proxmox-kernel-7.0.14-20-pve-signed",
"7.0.14-22-pve": "proxmox-kernel-7.0.14-22-pve-signed",
"7.0.14-23-pve": "proxmox-kernel-7.0.14-23-pve-signed",
"7.0.14-24-pve": "proxmox-kernel-7.0.14-24-pve-signed",
"7.0.14-25-pve": "proxmox-kernel-7.0.14-25-pve-signed",
"7.0.14-26-pve": "proxmox-kernel-7.0.14-26-pve-signed",
"7.0.14-28-pve": "proxmox-kernel-7.0.14-28-pve-signed"
},
"keep": {
"7.0.14-26-pve": [
"the last kernel that booted well before the running one (state.json `from`)",
"kernel step `from` (a self-revert boots it)",
"kernel step `default_before`"
],
"7.0.14-28-pve": [
"running",
"newest installed",
"kernel step `to`",
"GRUB default in grub.cfg",
"pinned default file"
]
},
"computed_removal": [
"proxmox-kernel-7.0.2-6-pve-signed",
"proxmox-kernel-7.0.14-20-pve-signed",
"proxmox-kernel-7.0.14-22-pve-signed",
"proxmox-kernel-7.0.14-23-pve-signed",
"proxmox-kernel-7.0.14-24-pve-signed",
"proxmox-kernel-7.0.14-25-pve-signed"
],
"removal": [
"proxmox-kernel-7.0.2-6-pve-signed",
"proxmox-kernel-7.0.14-20-pve-signed",
"proxmox-kernel-7.0.14-22-pve-signed",
"proxmox-kernel-7.0.14-23-pve-signed",
"proxmox-kernel-7.0.14-24-pve-signed",
"proxmox-kernel-7.0.14-25-pve-signed",
"proxmox-kernel-7.0.14-28-pve-signed"
],
"pretend": [
"--pretend-add",
"proxmox-kernel-7.0.14-28-pve-signed"
],
"simulation": {
"rc": 0,
"removes": [
"proxmox-kernel-7.0.14-20-pve-signed",
"proxmox-kernel-7.0.14-22-pve-signed",
"proxmox-kernel-7.0.14-23-pve-signed",
"proxmox-kernel-7.0.14-24-pve-signed",
"proxmox-kernel-7.0.14-25-pve-signed",
"proxmox-kernel-7.0.14-28-pve-signed",
"proxmox-kernel-7.0.2-6-pve-signed"
],
"installs": [
"proxmox-kernel-7.0.14-28-pve"
]
},
"refuse": [
"K1 proxmox-kernel-7.0.14-28-pve-signed is a keep-kernel (running; newest installed; kernel step `to`; GRUB default in grub.cfg; pinned default file)",
"S1 the simulation would install or configure proxmox-kernel-7.0.14-28-pve"
],
"verdict": "REFUSED"
}
VERDICT: REFUSED
REFUSE K1 proxmox-kernel-7.0.14-28-pve-signed is a keep-kernel (running; newest installed; kernel step `to`; GRUB default in grub.cfg; pinned default file)
REFUSE S1 the simulation would install or configure proxmox-kernel-7.0.14-28-pve
rc=3
=== CONTROL B last good kernel 26 in the removal set :: python3 - --pretend-add proxmox-kernel-7.0.14-26-pve-signed
{
"host": "demo-hp",
"running": "7.0.14-28-pve",
"newest": "7.0.14-28-pve",
"phase": "good",
"step": {
"step_id": "ring0-20261110T014802Z",
"from": "7.0.14-26-pve",
"to": "7.0.14-28-pve",
"default_before": "7.0.14-26-pve",
"result_at": "2026-11-10T01:56:47Z"
},
"oneshot_flag": "",
"grub_default": "7.0.14-28-pve",
"pinned_default": "7.0.14-28-pve",
"boot_pin": null,
"installed": {
"7.0.2-6-pve": "proxmox-kernel-7.0.2-6-pve-signed",
"7.0.14-20-pve": "proxmox-kernel-7.0.14-20-pve-signed",
"7.0.14-22-pve": "proxmox-kernel-7.0.14-22-pve-signed",
"7.0.14-23-pve": "proxmox-kernel-7.0.14-23-pve-signed",
"7.0.14-24-pve": "proxmox-kernel-7.0.14-24-pve-signed",
"7.0.14-25-pve": "proxmox-kernel-7.0.14-25-pve-signed",
"7.0.14-26-pve": "proxmox-kernel-7.0.14-26-pve-signed",
"7.0.14-28-pve": "proxmox-kernel-7.0.14-28-pve-signed"
},
"keep": {
"7.0.14-26-pve": [
"the last kernel that booted well before the running one (state.json `from`)",
"kernel step `from` (a self-revert boots it)",
"kernel step `default_before`"
],
"7.0.14-28-pve": [
"running",
"newest installed",
"kernel step `to`",
"GRUB default in grub.cfg",
"pinned default file"
]
},
"computed_removal": [
"proxmox-kernel-7.0.2-6-pve-signed",
"proxmox-kernel-7.0.14-20-pve-signed",
"proxmox-kernel-7.0.14-22-pve-signed",
"proxmox-kernel-7.0.14-23-pve-signed",
"proxmox-kernel-7.0.14-24-pve-signed",
"proxmox-kernel-7.0.14-25-pve-signed"
],
"removal": [
"proxmox-kernel-7.0.2-6-pve-signed",
"proxmox-kernel-7.0.14-20-pve-signed",
"proxmox-kernel-7.0.14-22-pve-signed",
"proxmox-kernel-7.0.14-23-pve-signed",
"proxmox-kernel-7.0.14-24-pve-signed",
"proxmox-kernel-7.0.14-25-pve-signed",
"proxmox-kernel-7.0.14-26-pve-signed"
],
"pretend": [
"--pretend-add",
"proxmox-kernel-7.0.14-26-pve-signed"
],
"simulation": {
"rc": 0,
"removes": [
"proxmox-kernel-7.0.14-20-pve-signed",
"proxmox-kernel-7.0.14-22-pve-signed",
"proxmox-kernel-7.0.14-23-pve-signed",
"proxmox-kernel-7.0.14-24-pve-signed",
"proxmox-kernel-7.0.14-25-pve-signed",
"proxmox-kernel-7.0.14-26-pve-signed",
"proxmox-kernel-7.0.2-6-pve-signed"
],
"installs": []
},
"refuse": [
"K1 proxmox-kernel-7.0.14-26-pve-signed is a keep-kernel (the last kernel that booted well before the running one (state.json `from`); kernel step `from` (a self-revert boots it); kernel step `default_before`)"
],
"verdict": "REFUSED"
}
VERDICT: REFUSED
REFUSE K1 proxmox-kernel-7.0.14-26-pve-signed is a keep-kernel (the last kernel that booted well before the running one (state.json `from`); kernel step `from` (a self-revert boots it); kernel step `default_before`)
rc=3
=== CONTROL C a kernel step in phase judging :: python3 - --pretend-phase judging
{
"host": "demo-hp",
"running": "7.0.14-28-pve",
"newest": "7.0.14-28-pve",
"phase": "judging",
"step": {
"step_id": "ring0-20261110T014802Z",
"from": "7.0.14-26-pve",
"to": "7.0.14-28-pve",
"default_before": "7.0.14-26-pve",
"result_at": "2026-11-10T01:56:47Z"
},
"oneshot_flag": "",
"grub_default": "7.0.14-28-pve",
"pinned_default": "7.0.14-28-pve",
"boot_pin": null,
"installed": {
"7.0.2-6-pve": "proxmox-kernel-7.0.2-6-pve-signed",
"7.0.14-20-pve": "proxmox-kernel-7.0.14-20-pve-signed",
"7.0.14-22-pve": "proxmox-kernel-7.0.14-22-pve-signed",
"7.0.14-23-pve": "proxmox-kernel-7.0.14-23-pve-signed",
"7.0.14-24-pve": "proxmox-kernel-7.0.14-24-pve-signed",
"7.0.14-25-pve": "proxmox-kernel-7.0.14-25-pve-signed",
"7.0.14-26-pve": "proxmox-kernel-7.0.14-26-pve-signed",
"7.0.14-28-pve": "proxmox-kernel-7.0.14-28-pve-signed"
},
"keep": {
"7.0.14-26-pve": [
"the last kernel that booted well before the running one (state.json `from`)",
"kernel step `from` (a self-revert boots it)",
"kernel step `default_before`"
],
"7.0.14-28-pve": [
"running",
"newest installed",
"kernel step `to`",
"GRUB default in grub.cfg",
"pinned default file"
]
},
"computed_removal": [
"proxmox-kernel-7.0.2-6-pve-signed",
"proxmox-kernel-7.0.14-20-pve-signed",
"proxmox-kernel-7.0.14-22-pve-signed",
"proxmox-kernel-7.0.14-23-pve-signed",
"proxmox-kernel-7.0.14-24-pve-signed",
"proxmox-kernel-7.0.14-25-pve-signed"
],
"removal": [
"proxmox-kernel-7.0.2-6-pve-signed",
"proxmox-kernel-7.0.14-20-pve-signed",
"proxmox-kernel-7.0.14-22-pve-signed",
"proxmox-kernel-7.0.14-23-pve-signed",
"proxmox-kernel-7.0.14-24-pve-signed",
"proxmox-kernel-7.0.14-25-pve-signed"
],
"pretend": [
"--pretend-phase",
"judging"
],
"simulation": {
"rc": 0,
"removes": [
"proxmox-kernel-7.0.14-20-pve-signed",
"proxmox-kernel-7.0.14-22-pve-signed",
"proxmox-kernel-7.0.14-23-pve-signed",
"proxmox-kernel-7.0.14-24-pve-signed",
"proxmox-kernel-7.0.14-25-pve-signed",
"proxmox-kernel-7.0.2-6-pve-signed"
],
"installs": []
},
"refuse": [
"P1 a kernel step is in phase 'judging' \u2014 not ended"
],
"verdict": "REFUSED"
}
VERDICT: REFUSED
REFUSE P1 a kernel step is in phase 'judging' — not ended
rc=3
=== CONTROL D a name that is not a kernel image (the series meta-package) :: python3 - --pretend-set proxmox-kernel-7.0
{
"host": "demo-hp",
"running": "7.0.14-28-pve",
"newest": "7.0.14-28-pve",
"phase": "good",
"step": {
"step_id": "ring0-20261110T014802Z",
"from": "7.0.14-26-pve",
"to": "7.0.14-28-pve",
"default_before": "7.0.14-26-pve",
"result_at": "2026-11-10T01:56:47Z"
},
"oneshot_flag": "",
"grub_default": "7.0.14-28-pve",
"pinned_default": "7.0.14-28-pve",
"boot_pin": null,
"installed": {
"7.0.2-6-pve": "proxmox-kernel-7.0.2-6-pve-signed",
"7.0.14-20-pve": "proxmox-kernel-7.0.14-20-pve-signed",
"7.0.14-22-pve": "proxmox-kernel-7.0.14-22-pve-signed",
"7.0.14-23-pve": "proxmox-kernel-7.0.14-23-pve-signed",
"7.0.14-24-pve": "proxmox-kernel-7.0.14-24-pve-signed",
"7.0.14-25-pve": "proxmox-kernel-7.0.14-25-pve-signed",
"7.0.14-26-pve": "proxmox-kernel-7.0.14-26-pve-signed",
"7.0.14-28-pve": "proxmox-kernel-7.0.14-28-pve-signed"
},
"keep": {
"7.0.14-26-pve": [
"the last kernel that booted well before the running one (state.json `from`)",
"kernel step `from` (a self-revert boots it)",
"kernel step `default_before`"
],
"7.0.14-28-pve": [
"running",
"newest installed",
"kernel step `to`",
"GRUB default in grub.cfg",
"pinned default file"
]
},
"computed_removal": [
"proxmox-kernel-7.0.2-6-pve-signed",
"proxmox-kernel-7.0.14-20-pve-signed",
"proxmox-kernel-7.0.14-22-pve-signed",
"proxmox-kernel-7.0.14-23-pve-signed",
"proxmox-kernel-7.0.14-24-pve-signed",
"proxmox-kernel-7.0.14-25-pve-signed"
],
"removal": [
"proxmox-kernel-7.0"
],
"pretend": [
"--pretend-set",
"proxmox-kernel-7.0"
],
"simulation": {
"rc": 0,
"removes": [
"proxmox-default-kernel",
"proxmox-kernel-7.0",
"proxmox-ve"
],
"installs": []
},
"refuse": [
"N1 proxmox-kernel-7.0 is not an installed kernel image package",
"S2 the simulation would also remove proxmox-default-kernel, proxmox-ve \u2014 not in the removal set"
],
"verdict": "REFUSED"
}
VERDICT: REFUSED
REFUSE N1 proxmox-kernel-7.0 is not an installed kernel image package
REFUSE S2 the simulation would also remove proxmox-default-kernel, proxmox-ve — not in the removal set
rc=3
=== CONTROL E a non-kernel package :: python3 - --pretend-set proxmox-kernel-7.0.14-25-pve-signed,pve-firmware
{
"host": "demo-hp",
"running": "7.0.14-28-pve",
"newest": "7.0.14-28-pve",
"phase": "good",
"step": {
"step_id": "ring0-20261110T014802Z",
"from": "7.0.14-26-pve",
"to": "7.0.14-28-pve",
"default_before": "7.0.14-26-pve",
"result_at": "2026-11-10T01:56:47Z"
},
"oneshot_flag": "",
"grub_default": "7.0.14-28-pve",
"pinned_default": "7.0.14-28-pve",
"boot_pin": null,
"installed": {
"7.0.2-6-pve": "proxmox-kernel-7.0.2-6-pve-signed",
"7.0.14-20-pve": "proxmox-kernel-7.0.14-20-pve-signed",
"7.0.14-22-pve": "proxmox-kernel-7.0.14-22-pve-signed",
"7.0.14-23-pve": "proxmox-kernel-7.0.14-23-pve-signed",
"7.0.14-24-pve": "proxmox-kernel-7.0.14-24-pve-signed",
"7.0.14-25-pve": "proxmox-kernel-7.0.14-25-pve-signed",
"7.0.14-26-pve": "proxmox-kernel-7.0.14-26-pve-signed",
"7.0.14-28-pve": "proxmox-kernel-7.0.14-28-pve-signed"
},
"keep": {
"7.0.14-26-pve": [
"the last kernel that booted well before the running one (state.json `from`)",
"kernel step `from` (a self-revert boots it)",
"kernel step `default_before`"
],
"7.0.14-28-pve": [
"running",
"newest installed",
"kernel step `to`",
"GRUB default in grub.cfg",
"pinned default file"
]
},
"computed_removal": [
"proxmox-kernel-7.0.2-6-pve-signed",
"proxmox-kernel-7.0.14-20-pve-signed",
"proxmox-kernel-7.0.14-22-pve-signed",
"proxmox-kernel-7.0.14-23-pve-signed",
"proxmox-kernel-7.0.14-24-pve-signed",
"proxmox-kernel-7.0.14-25-pve-signed"
],
"removal": [
"proxmox-kernel-7.0.14-25-pve-signed",
"pve-firmware"
],
"pretend": [
"--pretend-set",
"proxmox-kernel-7.0.14-25-pve-signed,pve-firmware"
],
"simulation": {
"rc": 0,
"removes": [
"proxmox-default-kernel",
"proxmox-kernel-7.0",
"proxmox-kernel-7.0.14-25-pve-signed",
"proxmox-ve",
"pve-firmware"
],
"installs": []
},
"refuse": [
"N1 pve-firmware is not an installed kernel image package",
"S2 the simulation would also remove proxmox-default-kernel, proxmox-kernel-7.0, proxmox-ve \u2014 not in the removal set"
],
"verdict": "REFUSED"
}
VERDICT: REFUSED
REFUSE N1 pve-firmware is not an installed kernel image package
REFUSE S2 the simulation would also remove proxmox-default-kernel, proxmox-kernel-7.0, proxmox-ve — not in the removal set
rc=3
@@ -0,0 +1,64 @@
== 2026-11-10T12:50:49Z preconditions just before the removal
7.0.14-28-pve
-- heavy processes (expect none)
none
2256165 bash -c echo "== $(date -u +%FT%TZ) preconditions just before the removal"; uname -r; echo "-- heavy processes (expect none)"; pgrep -a -x "vzdump|apt-get|dpkg|restic|proxmox-backup-client|pbs-restore" || echo none; pgrep -af "[f]elhom-os-apply" || echo "no felhom-os-apply"; echo "-- running PVE tasks"; pvesh get /nodes/$(hostname)/tasks --source active --output-format json | python3 -c "import json,sys; t=json.load(sys.stdin); print(len(t), [x.get(\"type\") for x in t])"; echo "-- guests"; pct list; qm list; echo "-- kernel state"; python3 -c "import json; s=json.load(open(\"/var/lib/felhom-kernel/state.json\")); print(s[\"phase\"], s[\"from\"], s[\"to\"])"; grub-editenv /boot/efi/EFI/felhom/oneshot.env list; echo "-- locks"; ls -la /var/lib/dpkg/lock* /var/lib/apt/lists/lock 2>&1 | head; fuser /var/lib/dpkg/lock-frontend 2>&1 || echo "dpkg lock free"; echo "-- sizes before"; df -B1M / /boot/efi | cat; du -sm /lib/modules /boot 2>/dev/null; ls /lib/modules; echo "-- grub default"; /usr/bin/grep -n "set default=" /boot/grub/grub.cfg | head -3; /usr/bin/grep -c "felhom-oneshot-" /boot/grub/grub.cfg; /usr/bin/grep -o "felhom-oneshot-[0-9.a-z-]*" /boot/grub/grub.cfg | sort -u; cat /etc/default/grub.d/zz-felhom-kernel-default.cfg; sha256sum /etc/default/grub.d/zz-felhom-kernel-default.cfg /etc/grub.d/01_felhom_oneshot /etc/grub.d/42_felhom_oneshot; echo "-- crash guard"; ls -la /var/lib/felhom-crash-guard 2>&1 | head; systemctl list-timers --no-pager 2>/dev/null | /usr/bin/grep -i -E "felhom|pve|apt" | cut -c1-150
-- running PVE tasks
0 []
-- guests
VMID Status Lock Name
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
341 night1004-tester1 running 8192 200.00 1861
-- kernel state
good 7.0.14-26-pve 7.0.14-28-pve
felhom_next=
-- locks
-rw-r----- 1 root root 0 May 21 13:31 /var/lib/apt/lists/lock
-rw-r----- 1 root root 0 Nov 10 02:50 /var/lib/dpkg/lock
-rw-r----- 1 root root 0 Oct 4 14:25 /var/lib/dpkg/lock-frontend
dpkg lock free
-- sizes before
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/pve-root 39506 22906 14562 62% /
/dev/sda2 1022 10 1013 1% /boot/efi
7772 /lib/modules
795 /boot
7.0.14-20-pve
7.0.14-22-pve
7.0.14-23-pve
7.0.14-24-pve
7.0.14-25-pve
7.0.14-26-pve
7.0.14-28-pve
7.0.2-6-pve
-- grub default
17: set default="${next_entry}"
22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
110: if [ "${felhom_boot}" = "7.0.14-20-pve" ]; then set default="felhom-oneshot-7.0.14-20-pve"; fi
16
felhom-oneshot-7.0.14-20-pve
felhom-oneshot-7.0.14-22-pve
felhom-oneshot-7.0.14-23-pve
felhom-oneshot-7.0.14-24-pve
felhom-oneshot-7.0.14-25-pve
felhom-oneshot-7.0.14-26-pve
felhom-oneshot-7.0.14-28-pve
felhom-oneshot-7.0.2-6-pve
# felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily
GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg
f2f56d59c6356067279baa457e84bba7ffcf46ac341b7d482845e5a5cf9cc27b /etc/grub.d/01_felhom_oneshot
c34907ce68599873889179be333e848e02aa1c594b7370116b95af31b41b7fe6 /etc/grub.d/42_felhom_oneshot
-- crash guard
total 12
drwxr-xr-x 2 root root 4096 Nov 10 13:09 .
drwxr-xr-x 46 root root 4096 Oct 10 04:42 ..
-rw-r--r-- 1 root root 679 Nov 10 13:09 state.json
Tue 2026-11-10 13:51:28 CET 35s Tue 2026-11-10 13:50:28 CET 24s ago felhom-mgmt-watchdog.timer felhom-mgmt-watchdog.service
Tue 2026-11-10 14:09:17 CET 18min Tue 2026-11-10 13:09:17 CET 41min ago felhom-crash-guard-check.timer felhom-crash-guard-check.service
Tue 2026-11-10 15:51:50 CET 2h 0min Tue 2026-11-10 04:06:22 CET 9h ago apt-daily.timer apt-daily.service
Wed 2026-11-11 04:54:05 CET 15h Tue 2026-11-10 03:05:09 CET 10h ago pve-daily-update.timer pve-daily-update.service
Wed 2026-11-11 06:38:33 CET 16h Tue 2026-11-10 06:55:09 CET 6h ago apt-daily-upgrade.timer apt-daily-upgrade.service
@@ -0,0 +1,43 @@
== 2026-11-10T12:50:59Z crash guard state before
{
"armed": true,
"boot_id": "b6e0edea-2b56-47df-a36f-5bb3825adc3a",
"config": {
"LIMIT": 3,
"PANIC_SECONDS": 10,
"REARM_HOURS": 24,
"WINDOW_MINUTES": 60
},
"kernel_panic": 10,
"last_boot_at": "2026-11-10T01:53:52Z",
"last_boot_unclean": false,
"last_trip": {
"at": "2026-10-13T20:14:04Z",
"reason": "2 unclean boots within 60 minutes \u2014 the next crash leaves the box off (limit 3)"
},
"rearmed_at": "2026-10-15T15:41:56Z",
"rearmed_by": "timer (24 h of normal running)",
"tripped": false,
"unclean_boots": [],
"unclean_boots_24h": 0,
"unclean_boots_in_window": 0,
"updated_at": "2026-11-10T12:09:17Z",
"version": 1
}
== SIMULATION (apt-get -s), the exact command line
Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be REMOVED:
proxmox-kernel-7.0.14-20-pve-signed* proxmox-kernel-7.0.14-22-pve-signed*
proxmox-kernel-7.0.14-23-pve-signed* proxmox-kernel-7.0.14-24-pve-signed*
proxmox-kernel-7.0.14-25-pve-signed* proxmox-kernel-7.0.2-6-pve-signed*
0 upgraded, 0 newly installed, 6 to remove and 16 not upgraded.
Purg proxmox-kernel-7.0.14-20-pve-signed [7.0.14-20]
Purg proxmox-kernel-7.0.14-22-pve-signed [7.0.14-22]
Purg proxmox-kernel-7.0.14-23-pve-signed [7.0.14-23]
Purg proxmox-kernel-7.0.14-24-pve-signed [7.0.14-24]
Purg proxmox-kernel-7.0.14-25-pve-signed [7.0.14-25]
Purg proxmox-kernel-7.0.2-6-pve-signed [7.0.2-6]
sim rc=0
@@ -0,0 +1,284 @@
== START 2026-11-10T12:51:10Z
Reading package lists...
Building dependency tree...
Reading state information...
The following packages will be REMOVED:
proxmox-kernel-7.0.14-20-pve-signed* proxmox-kernel-7.0.14-22-pve-signed*
proxmox-kernel-7.0.14-23-pve-signed* proxmox-kernel-7.0.14-24-pve-signed*
proxmox-kernel-7.0.14-25-pve-signed* proxmox-kernel-7.0.2-6-pve-signed*
0 upgraded, 0 newly installed, 6 to remove and 16 not upgraded.
After this operation, 6184 MB disk space will be freed.
(Reading database ... ␍(Reading database ... 5%␍(Reading database ... 10%␍(Reading database ... 15%␍(Reading database ... 20%␍(Reading database ... 25%␍(Reading database ... 30%␍(Reading database ... 35%␍(Reading database ... 40%␍(Reading database ... 45%␍(Reading database ... 50%␍(Reading database ... 55%␍(Reading database ... 60%␍(Reading database ... 65%␍(Reading database ... 70%␍(Reading database ... 75%␍(Reading database ... 80%␍(Reading database ... 85%␍(Reading database ... 90%␍(Reading database ... 95%␍(Reading database ... 100%␍(Reading database ... 110097 files and directories currently installed.)
Removing proxmox-kernel-7.0.14-20-pve-signed (7.0.14-20) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-20-pve /boot/vmlinuz-7.0.14-20-pve
update-initramfs: Deleting /boot/initrd.img-7.0.14-20-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-20-pve /boot/vmlinuz-7.0.14-20-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-20-pve /boot/vmlinuz-7.0.14-20-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-20-pve /boot/vmlinuz-7.0.14-20-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-7.0.14-28-pve
Found initrd image: /boot/initrd.img-7.0.14-28-pve
Found linux image: /boot/vmlinuz-7.0.14-26-pve
Found initrd image: /boot/initrd.img-7.0.14-26-pve
Found linux image: /boot/vmlinuz-7.0.14-25-pve
Found initrd image: /boot/initrd.img-7.0.14-25-pve
Found linux image: /boot/vmlinuz-7.0.14-24-pve
Found initrd image: /boot/initrd.img-7.0.14-24-pve
Found linux image: /boot/vmlinuz-7.0.14-23-pve
Found initrd image: /boot/initrd.img-7.0.14-23-pve
Found linux image: /boot/vmlinuz-7.0.14-22-pve
Found initrd image: /boot/initrd.img-7.0.14-22-pve
Found linux image: /boot/vmlinuz-7.0.2-6-pve
Found initrd image: /boot/initrd.img-7.0.2-6-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Adding boot menu entry for UEFI Firmware Settings ...
done
Removing proxmox-kernel-7.0.14-22-pve-signed (7.0.14-22) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-22-pve /boot/vmlinuz-7.0.14-22-pve
update-initramfs: Deleting /boot/initrd.img-7.0.14-22-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-22-pve /boot/vmlinuz-7.0.14-22-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-22-pve /boot/vmlinuz-7.0.14-22-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-22-pve /boot/vmlinuz-7.0.14-22-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-7.0.14-28-pve
Found initrd image: /boot/initrd.img-7.0.14-28-pve
Found linux image: /boot/vmlinuz-7.0.14-26-pve
Found initrd image: /boot/initrd.img-7.0.14-26-pve
Found linux image: /boot/vmlinuz-7.0.14-25-pve
Found initrd image: /boot/initrd.img-7.0.14-25-pve
Found linux image: /boot/vmlinuz-7.0.14-24-pve
Found initrd image: /boot/initrd.img-7.0.14-24-pve
Found linux image: /boot/vmlinuz-7.0.14-23-pve
Found initrd image: /boot/initrd.img-7.0.14-23-pve
Found linux image: /boot/vmlinuz-7.0.2-6-pve
Found initrd image: /boot/initrd.img-7.0.2-6-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Adding boot menu entry for UEFI Firmware Settings ...
done
Removing proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-23-pve /boot/vmlinuz-7.0.14-23-pve
update-initramfs: Deleting /boot/initrd.img-7.0.14-23-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-23-pve /boot/vmlinuz-7.0.14-23-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-23-pve /boot/vmlinuz-7.0.14-23-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-23-pve /boot/vmlinuz-7.0.14-23-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-7.0.14-28-pve
Found initrd image: /boot/initrd.img-7.0.14-28-pve
Found linux image: /boot/vmlinuz-7.0.14-26-pve
Found initrd image: /boot/initrd.img-7.0.14-26-pve
Found linux image: /boot/vmlinuz-7.0.14-25-pve
Found initrd image: /boot/initrd.img-7.0.14-25-pve
Found linux image: /boot/vmlinuz-7.0.14-24-pve
Found initrd image: /boot/initrd.img-7.0.14-24-pve
Found linux image: /boot/vmlinuz-7.0.2-6-pve
Found initrd image: /boot/initrd.img-7.0.2-6-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Adding boot menu entry for UEFI Firmware Settings ...
done
Removing proxmox-kernel-7.0.14-24-pve-signed (7.0.14-24) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-24-pve /boot/vmlinuz-7.0.14-24-pve
update-initramfs: Deleting /boot/initrd.img-7.0.14-24-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-24-pve /boot/vmlinuz-7.0.14-24-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-24-pve /boot/vmlinuz-7.0.14-24-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-24-pve /boot/vmlinuz-7.0.14-24-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-7.0.14-28-pve
Found initrd image: /boot/initrd.img-7.0.14-28-pve
Found linux image: /boot/vmlinuz-7.0.14-26-pve
Found initrd image: /boot/initrd.img-7.0.14-26-pve
Found linux image: /boot/vmlinuz-7.0.14-25-pve
Found initrd image: /boot/initrd.img-7.0.14-25-pve
Found linux image: /boot/vmlinuz-7.0.2-6-pve
Found initrd image: /boot/initrd.img-7.0.2-6-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Adding boot menu entry for UEFI Firmware Settings ...
done
Removing proxmox-kernel-7.0.14-25-pve-signed (7.0.14-25) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-25-pve /boot/vmlinuz-7.0.14-25-pve
update-initramfs: Deleting /boot/initrd.img-7.0.14-25-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-25-pve /boot/vmlinuz-7.0.14-25-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-25-pve /boot/vmlinuz-7.0.14-25-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-25-pve /boot/vmlinuz-7.0.14-25-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-7.0.14-28-pve
Found initrd image: /boot/initrd.img-7.0.14-28-pve
Found linux image: /boot/vmlinuz-7.0.14-26-pve
Found initrd image: /boot/initrd.img-7.0.14-26-pve
Found linux image: /boot/vmlinuz-7.0.2-6-pve
Found initrd image: /boot/initrd.img-7.0.2-6-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Adding boot menu entry for UEFI Firmware Settings ...
done
Removing proxmox-kernel-7.0.2-6-pve-signed (7.0.2-6) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.2-6-pve /boot/vmlinuz-7.0.2-6-pve
update-initramfs: Deleting /boot/initrd.img-7.0.2-6-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.2-6-pve /boot/vmlinuz-7.0.2-6-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.2-6-pve /boot/vmlinuz-7.0.2-6-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.2-6-pve /boot/vmlinuz-7.0.2-6-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-7.0.14-28-pve
Found initrd image: /boot/initrd.img-7.0.14-28-pve
Found linux image: /boot/vmlinuz-7.0.14-26-pve
Found initrd image: /boot/initrd.img-7.0.14-26-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Adding boot menu entry for UEFI Firmware Settings ...
done
(Reading database ... ␍(Reading database ... 5%␍(Reading database ... 10%␍(Reading database ... 15%␍(Reading database ... 20%␍(Reading database ... 25%␍(Reading database ... 30%␍(Reading database ... 35%␍(Reading database ... 40%␍(Reading database ... 45%␍(Reading database ... 50%␍(Reading database ... 55%␍(Reading database ... 60%␍(Reading database ... 65%␍(Reading database ... 70%␍(Reading database ... 75%␍(Reading database ... 80%␍(Reading database ... 85%␍(Reading database ... 90%␍(Reading database ... 95%␍(Reading database ... 100%␍(Reading database ... 60863 files and directories currently installed.)
Purging configuration files for proxmox-kernel-7.0.14-22-pve-signed (7.0.14-22) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-22-pve /boot/vmlinuz-7.0.14-22-pve
update-initramfs: Deleting /boot/initrd.img-7.0.14-22-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-22-pve /boot/vmlinuz-7.0.14-22-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-22-pve /boot/vmlinuz-7.0.14-22-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-22-pve /boot/vmlinuz-7.0.14-22-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-7.0.14-28-pve
Found initrd image: /boot/initrd.img-7.0.14-28-pve
Found linux image: /boot/vmlinuz-7.0.14-26-pve
Found initrd image: /boot/initrd.img-7.0.14-26-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Adding boot menu entry for UEFI Firmware Settings ...
done
Purging configuration files for proxmox-kernel-7.0.14-20-pve-signed (7.0.14-20) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-20-pve /boot/vmlinuz-7.0.14-20-pve
update-initramfs: Deleting /boot/initrd.img-7.0.14-20-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-20-pve /boot/vmlinuz-7.0.14-20-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-20-pve /boot/vmlinuz-7.0.14-20-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-20-pve /boot/vmlinuz-7.0.14-20-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-7.0.14-28-pve
Found initrd image: /boot/initrd.img-7.0.14-28-pve
Found linux image: /boot/vmlinuz-7.0.14-26-pve
Found initrd image: /boot/initrd.img-7.0.14-26-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Adding boot menu entry for UEFI Firmware Settings ...
done
Purging configuration files for proxmox-kernel-7.0.2-6-pve-signed (7.0.2-6) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.2-6-pve /boot/vmlinuz-7.0.2-6-pve
update-initramfs: Deleting /boot/initrd.img-7.0.2-6-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.2-6-pve /boot/vmlinuz-7.0.2-6-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.2-6-pve /boot/vmlinuz-7.0.2-6-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.2-6-pve /boot/vmlinuz-7.0.2-6-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-7.0.14-28-pve
Found initrd image: /boot/initrd.img-7.0.14-28-pve
Found linux image: /boot/vmlinuz-7.0.14-26-pve
Found initrd image: /boot/initrd.img-7.0.14-26-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Adding boot menu entry for UEFI Firmware Settings ...
done
Purging configuration files for proxmox-kernel-7.0.14-25-pve-signed (7.0.14-25) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-25-pve /boot/vmlinuz-7.0.14-25-pve
update-initramfs: Deleting /boot/initrd.img-7.0.14-25-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-25-pve /boot/vmlinuz-7.0.14-25-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-25-pve /boot/vmlinuz-7.0.14-25-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-25-pve /boot/vmlinuz-7.0.14-25-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-7.0.14-28-pve
Found initrd image: /boot/initrd.img-7.0.14-28-pve
Found linux image: /boot/vmlinuz-7.0.14-26-pve
Found initrd image: /boot/initrd.img-7.0.14-26-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Adding boot menu entry for UEFI Firmware Settings ...
done
Purging configuration files for proxmox-kernel-7.0.14-24-pve-signed (7.0.14-24) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-24-pve /boot/vmlinuz-7.0.14-24-pve
update-initramfs: Deleting /boot/initrd.img-7.0.14-24-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-24-pve /boot/vmlinuz-7.0.14-24-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-24-pve /boot/vmlinuz-7.0.14-24-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-24-pve /boot/vmlinuz-7.0.14-24-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-7.0.14-28-pve
Found initrd image: /boot/initrd.img-7.0.14-28-pve
Found linux image: /boot/vmlinuz-7.0.14-26-pve
Found initrd image: /boot/initrd.img-7.0.14-26-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Adding boot menu entry for UEFI Firmware Settings ...
done
Purging configuration files for proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23) ...
Examining /etc/kernel/postrm.d.
run-parts: executing /etc/kernel/postrm.d/initramfs-tools 7.0.14-23-pve /boot/vmlinuz-7.0.14-23-pve
update-initramfs: Deleting /boot/initrd.img-7.0.14-23-pve
run-parts: executing /etc/kernel/postrm.d/proxmox-auto-removal 7.0.14-23-pve /boot/vmlinuz-7.0.14-23-pve
run-parts: executing /etc/kernel/postrm.d/zz-proxmox-boot 7.0.14-23-pve /boot/vmlinuz-7.0.14-23-pve
Re-executing '/etc/kernel/postrm.d/zz-proxmox-boot' in new private mount namespace..
No /etc/kernel/proxmox-boot-uuids found, skipping ESP sync.
run-parts: executing /etc/kernel/postrm.d/zz-update-grub 7.0.14-23-pve /boot/vmlinuz-7.0.14-23-pve
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-7.0.14-28-pve
Found initrd image: /boot/initrd.img-7.0.14-28-pve
Found linux image: /boot/vmlinuz-7.0.14-26-pve
Found initrd image: /boot/initrd.img-7.0.14-26-pve
Found memtest86+ 64bit EFI image: /boot/memtest86+x64.efi
Found memtest86+ 32bit EFI image: /boot/memtest86+ia32.efi
Found memtest86+ 64bit image: /boot/memtest86+x64.bin
Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
Adding boot menu entry for UEFI Firmware Settings ...
done
APT_RC=0
== END 2026-11-10T12:52:00Z
@@ -0,0 +1,95 @@
== 2026-11-10T12:52:12Z after the removal, before the reboot
7.0.14-28-pve
-- kernel packages
ii proxmox-default-kernel 2.1.0
ii proxmox-kernel-7.0 7.0.14-28
un proxmox-kernel-7.0.14-26-pve
ii proxmox-kernel-7.0.14-26-pve-signed 7.0.14-26
un proxmox-kernel-7.0.14-28-pve
ii proxmox-kernel-7.0.14-28-pve-signed 7.0.14-28
ii proxmox-kernel-helper 9.1.0+fde2
-- dpkg audit (expect empty)
audit rc=0
-- sizes after
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/pve-root 39506 16495 20973 45% /
/dev/sda2 1022 10 1013 1% /boot/efi
1945 /lib/modules
217 /boot
7.0.14-26-pve
7.0.14-28-pve
-rw-r--r-- 1 root root 308500 Oct 22 20:45 config-7.0.14-26-pve
-rw-r--r-- 1 root root 308500 Nov 2 12:54 config-7.0.14-28-pve
-rw-r--r-- 1 root root 73492957 Oct 29 05:07 initrd.img-7.0.14-26-pve
-rw-r--r-- 1 root root 73547547 Nov 10 02:50 initrd.img-7.0.14-28-pve
-rw-r--r-- 1 root root 10341401 Oct 22 20:45 System.map-7.0.14-26-pve
-rw-r--r-- 1 root root 10341916 Nov 2 12:54 System.map-7.0.14-28-pve
-rw-r--r-- 1 root root 16280680 Oct 22 20:45 vmlinuz-7.0.14-26-pve
-rw-r--r-- 1 root root 16280680 Nov 2 12:54 vmlinuz-7.0.14-28-pve
-- grub.cfg default + pinned file
17: set default="${next_entry}"
22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
110: if [ "${felhom_boot}" = "7.0.14-26-pve" ]; then set default="felhom-oneshot-7.0.14-26-pve"; fi
111: if [ "${felhom_boot}" = "7.0.14-28-pve" ]; then set default="felhom-oneshot-7.0.14-28-pve"; fi
# felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily
GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
6b35d9aeddc1ddd0820eefb9a3859c772eea77b66c2971459515cb1f45bbf392 /etc/default/grub.d/zz-felhom-kernel-default.cfg
f2f56d59c6356067279baa457e84bba7ffcf46ac341b7d482845e5a5cf9cc27b /etc/grub.d/01_felhom_oneshot
c34907ce68599873889179be333e848e02aa1c594b7370116b95af31b41b7fe6 /etc/grub.d/42_felhom_oneshot
-- menu entries
menuentry 'Proxmox VE GNU/Linux'
submenu 'Advanced options for Proxmox VE GNU/Linux' $menuentry_id_option 'gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43' {
menuentry 'Proxmox VE GNU/Linux, with Linux 7.0.14-28-pve'
menuentry 'Proxmox VE GNU/Linux, with Linux 7.0.14-28-pve (recovery mode)'
menuentry 'Proxmox VE GNU/Linux, with Linux 7.0.14-26-pve'
menuentry 'Proxmox VE GNU/Linux, with Linux 7.0.14-26-pve (recovery mode)'
menuentry "Memory test (memtest86+x64.efi)"
menuentry "Memory test (memtest86+x64.efi, serial console)"
menuentry "Memory test (memtest86+ia32.efi)"
menuentry "Memory test (memtest86+ia32.efi, serial console)"
menuentry "Memory test (memtest86+x64.bin)"
menuentry "Memory test (memtest86+x64.bin, serial console)"
menuentry "Memory test (memtest86+ia32.bin)"
menuentry "Memory test (memtest86+ia32.bin, serial console)"
menuentry 'UEFI Firmware Settings' $menuentry_id_option 'uefi-firmware' {
menuentry 'Felhom one-shot: 7.0.14-26-pve'
menuentry 'Felhom one-shot: 7.0.14-28-pve'
-- one-shot entries
2 felhom-oneshot-7.0.14-26-pve
2 felhom-oneshot-7.0.14-28-pve
-- every kernel file grub.cfg names exists
ok /boot/initrd.img-7.0.14-26-pve
ok /boot/initrd.img-7.0.14-28-pve
ok /boot/vmlinuz-7.0.14-26-pve
ok /boot/vmlinuz-7.0.14-28-pve
-- grub syntax
grub-script-check rc=0
-- flag + state
felhom_next=
good 7.0.14-26-pve 7.0.14-28-pve
-- NeverAutoRemove now
APT::NeverAutoRemove:: "^pve-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-23-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-23-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-23-pve-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-26-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-26-pve-signed$";
APT::NeverAutoRemove:: "^pve-kernel-7\.0\.14-28-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve$";
APT::NeverAutoRemove:: "^proxmox-kernel-7\.0\.14-28-pve-signed$";
-- simulated autoremove after
0 upgraded, 0 newly installed, 0 to remove and 16 not upgraded.
-- services
active
active
active
active
VMID Status Lock Name
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
341 night1004-tester1 running 8192 200.00 1861
@@ -0,0 +1,102 @@
2026-11-10T12:53:45Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 49% 58% running (connected) 71% local-lvm
2026-11-10T12:53:55Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 49% 58% running (connected) 71% local-lvm
2026-11-10T12:54:13Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 49% 58% running (connected) 71% local-lvm
2026-11-10T12:54:23Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:54:33Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:54:43Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:54:53Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:55:03Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:55:14Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:55:24Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:55:34Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:55:44Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:55:54Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:56:04Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:56:14Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:56:25Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:56:35Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:56:46Z ping=DOWN hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:56:57Z ping=DOWN hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:57:08Z ping=DOWN hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 12% 49% 42% running (connected) 71% local-lvm
2026-11-10T12:57:18Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T12:57:29Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T12:57:39Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T12:57:49Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T12:57:59Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T12:58:09Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T12:58:19Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T12:58:30Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T12:58:40Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T12:58:50Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T12:59:00Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T12:59:10Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T12:59:20Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T12:59:31Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T12:59:41Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T12:59:51Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:00:01Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:00:12Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:00:22Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:00:32Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:00:42Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:00:53Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:01:03Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:01:13Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:01:23Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:01:33Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:01:43Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:01:53Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:02:04Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:02:14Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:02:24Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:02:34Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:02:44Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:02:54Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:03:05Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:03:15Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:03:25Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:03:35Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:03:45Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:03:55Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:04:05Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:04:16Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:04:26Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:04:36Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:04:46Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:04:56Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:05:06Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:05:16Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:05:27Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:05:37Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:05:47Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:05:57Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:06:08Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:06:18Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:06:28Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:06:38Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:06:48Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:06:58Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:07:09Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:07:19Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:07:29Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:07:39Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:07:49Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:07:59Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:08:10Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:08:20Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:08:30Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:08:40Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:08:50Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:09:00Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:09:11Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:09:21Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:09:31Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:09:41Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:09:51Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:10:01Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:10:12Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:10:22Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:10:32Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:10:42Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:10:52Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
2026-11-10T13:11:02Z ping=up hub=200 | demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 0/2 0% 6% 42% unknown (no running customer guest) 71% local-lvm
@@ -0,0 +1,46 @@
== 2026-11-10T12:54:17Z last check before the one orderly reboot
7.0.14-28-pve
b6e0edea-2b56-47df-a36f-5bb3825adc3a
13:54:17 up 11:00, 1 user, load average: 1.52, 1.28, 1.43
no heavy process
running PVE tasks: 0
VMID Status Lock Name
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
341 night1004-tester1 running 8192 200.00 1861
-- guest 9201 containers (names + status only)
adventurelog-frontend Up 10 hours (healthy)
adventurelog-postgres Up 10 hours (healthy)
adventurelog Up 10 hours (healthy)
bentopdf Up 11 hours (healthy)
bookstack-db Up 10 hours (healthy)
bookstack Up 10 hours (healthy)
calibre-web Up 10 hours (healthy)
cloudflared Up 11 hours (healthy)
docmost-postgres Up 10 hours (healthy)
docmost-redis Up 10 hours (healthy)
docmost Up 10 hours (healthy)
felhom-controller Up 11 hours (healthy)
filebrowser Up 11 hours (healthy)
home-assistant Up 10 hours (healthy)
kimai-db Up 10 hours (healthy)
kimai Up 10 hours (healthy)
opengist Up 10 hours (healthy)
paperless-postgres Up 10 hours (healthy)
paperless-redis Up 10 hours (healthy)
paperless-webserver Up 10 hours (healthy)
plex Up 10 hours (healthy)
privatebin Up 10 hours (healthy)
romm-db Up 10 hours (healthy)
romm-redis Up 10 hours (healthy)
romm Up 10 hours (healthy)
traefik Up 11 hours
-- guest 9202
felhom-controller Up 22 minutes (healthy)
filebrowser Up 23 minutes (healthy)
paperless-postgres Up 11 hours (healthy)
paperless-redis Up 11 hours (healthy)
paperless-webserver Up 11 hours (healthy)
traefik Up 11 hours
@@ -0,0 +1,4 @@
T0 DooPlex 2026-11-10T12:54:26Z — systemctl reboot sent to demo-hp
host clock 2026-11-10T12:54:26Z
systemctl reboot rc=0
ssh rc=0
@@ -0,0 +1,4 @@
2026-11-10T12:58:41Z g9201 total=26 up=26 healthy=22 starting=3 unhealthy=0 ; g9202 total=6 up=6 healthy=5 starting=0 unhealthy=0 ;
2026-11-10T12:58:59Z g9201 total=26 up=26 healthy=24 starting=1 unhealthy=0 ; g9202 total=6 up=6 healthy=5 starting=0 unhealthy=0 ;
2026-11-10T12:59:18Z g9201 total=26 up=26 healthy=24 starting=1 unhealthy=0 ; g9202 total=6 up=6 healthy=5 starting=0 unhealthy=0 ;
2026-11-10T12:59:37Z g9201 total=26 up=26 healthy=25 starting=0 unhealthy=0 ; g9202 total=6 up=6 healthy=5 starting=0 unhealthy=0 ;
@@ -0,0 +1,298 @@
# after the one reboot — read on demo-hp; DooPlex clock 2026-11-10T12:59:51Z
== clock
2026-11-10T12:59:51Z
== kernel
7.0.14-28-pve
== boot_id
d1167d94-31bf-459a-bf2a-569e8997a569
== up since (host clock, CET)
2026-11-10 13:56:59
13:59:51 up 2 min, 1 user, load average: 6.29, 4.13, 1.67
== system state
running
== services
felhom-agent active
pvedaemon active
pveproxy active
pvestatd active
pve-cluster active
== crash guard state
{
"armed": true,
"boot_id": "d1167d94-31bf-459a-bf2a-569e8997a569",
"config": {
"LIMIT": 3,
"PANIC_SECONDS": 10,
"REARM_HOURS": 24,
"WINDOW_MINUTES": 60
},
"kernel_panic": 10,
"last_boot_at": "2026-11-10T12:56:59Z",
"last_boot_unclean": false,
"last_trip": {
"at": "2026-10-13T20:14:04Z",
"reason": "2 unclean boots within 60 minutes \u2014 the next crash leaves the box off (limit 3)"
},
"rearmed_at": "2026-10-15T15:41:56Z",
"rearmed_by": "timer (24 h of normal running)",
"tripped": false,
"unclean_boots": [],
"unclean_boots_24h": 0,
"unclean_boots_in_window": 0,
"updated_at": "2026-11-10T12:57:09Z",
"version": 1
}
== previous boot: how it ended (journal -b -1, last lines)
2026-11-10T13:56:35+01:00 demo-hp systemd-shutdown[1]: Using hardware watchdog 'Software Watchdog', version 0, device /dev/watchdog0
2026-11-10T13:56:35+01:00 demo-hp systemd-shutdown[1]: Watchdog running with a hardware timeout of 10min.
2026-11-10T13:56:35+01:00 demo-hp systemd-shutdown[1]: Syncing filesystems and block devices.
2026-11-10T13:56:35+01:00 demo-hp systemd-shutdown[1]: Sending SIGTERM to remaining processes...
2026-11-10T13:56:35+01:00 demo-hp systemd-journald[351]: Received SIGTERM from PID 1 (systemd-shutdow).
2026-11-10T13:56:35+01:00 demo-hp systemd-journald[351]: Journal stopped
== kernel step state
{
"authority": "ring0",
"default_before": "7.0.14-26-pve",
"from": "7.0.14-26-pve",
"health_before": {
"guest": {
"containers": {
"adventurelog": {
"health": "healthy",
"id": "50af393c36a37d01a10847456d391d70572c121045e28626f220f83cec1cefcc",
"state": "running"
},
"adventurelog-frontend": {
"health": "healthy",
"id": "abd6155bdad527c65b82f75bc0dd0f98f9690ec81587c8cd07cea6c9c019cabe",
"state": "running"
},
"adventurelog-postgres": {
"health": "healthy",
"id": "841257c7abb7b545567aa8fa539cdcd421c94ce4edf97a0dea5e45831ea399f4",
"state": "running"
},
"bentopdf": {
"health": "healthy",
"id": "9db9aba521daf9f19352a64511f4f84872ade957b643c0db40dd510837143cba",
"state": "running"
},
"bookstack": {
"health": "healthy",
"id": "c83bbf5562cb68ee23d71ece2362d8be482ac4d7d59ad9bdfa5493f0b53c9687",
"state": "running"
},
"bookstack-db": {
"health": "healthy",
"id": "ea5212ba45188144fb2b2aca5bd53007e01e15a7306a411e651e5b64aea782d8",
"state": "running"
},
"calibre-web": {
"health": "healthy",
"id": "ac9e6b2302c502a9207fc9e7cfdc0a6999e7304730957263212a1300ed525d10",
"state": "running"
},
"cloudflared": {
"health": "healthy",
"id": "549f296e6ef9f6594f5e8789ecb86579f0c6ee45091b38bafdca69b50377d27a",
"state": "running"
},
"docmost": {
"health": "healthy",
"id": "b7ce8b91c0f44c389f4a9153778c910c226c0a4f1118030d6da84d4860c33da6",
"state": "running"
},
"docmost-postgres": {
"health": "healthy",
"id": "d16587ff24479fa15fa24ce56c23d47005623483315497460140fc563a3c1e8b",
"state": "running"
},
"docmost-redis": {
"health": "healthy",
"id": "be025075e63097a7df7fd37c56fd4034942742beff6f2ef34d83df003a36c590",
"state": "running"
},
"felhom-controller": {
"health": "healthy",
"id": "1af906eb4abc5270c0a291b14ac1638be79c5a3fe9c47dce973a1c42d1678b0d",
"state": "running"
},
"filebrowser": {
"health": "healthy",
"id": "f6446630708007c3b2f97a8d62043193c6516a7a62daf31037e42989a51c9393",
"state": "running"
},
"home-assistant": {
"health": "healthy",
"id": "588d1a667f65af137effb7acbc28a220d63f4f5d417039aca9ef9df9ad07a923",
"state": "running"
},
"kimai": {
"health": "healthy",
"id": "e5443c7db71d859c4ae97c551566e3a278511282b192abe33746baff88d62a34",
"state": "running"
},
"kimai-db": {
"health": "healthy",
"id": "90fe1d5c54138fe8fde39fca173f5b735f6fbae23d8c6156eeacb06f716836c2",
"state": "running"
},
"opengist": {
"health": "healthy",
"id": "0991f49cd94bea7a3f6db7c597f716e8a080de6492528db8f31ee737d783e528",
"state": "running"
},
"paperless-postgres": {
"health": "healthy",
"id": "e5862350b148d28e5eee065a7ca4d12c2e2d64d7ccc4c40708781dd5255e9296",
"state": "running"
},
"paperless-redis": {
"health": "healthy",
"id": "cfb5b8b9e711a12027c47b8b7a7f3d2fd74999338fec699372883c19157ddbfa",
"state": "running"
},
"paperless-webserver": {
"health": "healthy",
"id": "74c1e55159dd9b856af7a0d1a7cdb56680fcefa195913537768f3789221c3c29",
"state": "running"
},
"plex": {
"health": "healthy",
"id": "5091a7dd6259f31e28552895f9f714d1f7cd0938b0e525d6a03133355c50d83c",
"state": "running"
},
"privatebin": {
"health": "healthy",
"id": "5e6c930258cb8f4e76fa0fd2ab80bad7b2a62c9cd93882edece8e096e39a29f4",
"state": "running"
},
"romm": {
"health": "healthy",
"id": "8368d5dbb4be09989cd5101a50335ee4e80d7e139ad6a70c30d7fedd933b16d0",
"state": "running"
},
"romm-db": {
"health": "healthy",
"id": "6bbe9a4d9bb95606052f2bcb206df17b78b3058ee68a4af55450b4e3d1cecc1c",
"state": "running"
},
"romm-redis": {
"health": "healthy",
"id": "519aa50394d17f06467be060d732f060f3bb23e08a19aa8c6170c7198c3c3c78",
"state": "running"
},
"traefik": {
"health": "none",
"id": "fc0bfe5b53c6a7711d5d742b8ba9c4b3fbdbf96da7270311ffcf4e23f28817c8",
"state": "running"
}
},
"controller": "healthy",
"controller_docker_ok": true,
"docker_ok": true,
"network_ok": true
},
"guest_running": true,
"host_services": {
"felhom-agent": "active",
"pve-cluster": "active",
"pvedaemon": "active",
"pveproxy": "active",
"pvestatd": "active"
}
},
"judging_since": "2026-11-10T01:54:08Z",
"packages": [
{
"name": "proxmox-kernel-7.0",
"version": "7.0.14-28"
},
{
"name": "proxmox-kernel-7.0.14-28-pve-signed",
"version": "7.0.14-28"
}
],
"phase": "good",
"rebooted_at": "2026-11-10T01:50:56Z",
"result_at": "2026-11-10T01:56:47Z",
"self_revert_used": false,
"staged_at": "2026-11-10T01:50:50Z",
"step_id": "ring0-20261110T014802Z",
"to": "7.0.14-28-pve",
"updated_at": "2026-11-10T01:56:47Z",
"vmid": 9201
}
== one-shot flag
felhom_next=
== grub default
17: set default="${next_entry}"
22: set default="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
# felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily
GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
== kernel packages (ii)
ii proxmox-default-kernel 2.1.0
ii proxmox-kernel-7.0 7.0.14-28
ii proxmox-kernel-7.0.14-26-pve-signed 7.0.14-26
ii proxmox-kernel-7.0.14-28-pve-signed 7.0.14-28
ii proxmox-kernel-helper 9.1.0+fde2
== dpkg audit
rc=0
== sizes
Filesystem 1M-blocks Used Available Use% Mounted on
/dev/mapper/pve-root 39506 16492 20976 45% /
/dev/sda2 1022 10 1013 1% /boot/efi
1945 /lib/modules
217 /boot
== guests
VMID Status Lock Name
9201 running demo-hp
9202 running demo-hp-scratch
9401 stopped upgrade-harness
VMID NAME STATUS MEM(MB) BOOTDISK(GB) PID
341 night1004-tester1 running 8192 200.00 1847
== 9201 containers (name status)
romm Up About a minute (healthy)
romm-redis Up About a minute (healthy)
romm-db Up About a minute (healthy)
plex Up About a minute (healthy)
paperless-webserver Up About a minute (healthy)
paperless-redis Up 2 minutes (healthy)
paperless-postgres Up 2 minutes (healthy)
calibre-web Up 2 minutes (healthy)
felhom-controller Up 2 minutes (healthy)
privatebin Up 2 minutes (healthy)
opengist Up 2 minutes (healthy)
kimai Up 2 minutes (healthy)
kimai-db Up 2 minutes (healthy)
home-assistant Up 2 minutes (healthy)
docmost Up 2 minutes (healthy)
docmost-redis Up 2 minutes (healthy)
docmost-postgres Up 2 minutes (healthy)
bookstack Up 2 minutes (healthy)
bookstack-db Up 2 minutes (healthy)
adventurelog Up 2 minutes (healthy)
adventurelog-postgres Up 2 minutes (healthy)
adventurelog-frontend Up 2 minutes (healthy)
bentopdf Up 2 minutes (healthy)
traefik Up 2 minutes
cloudflared Up 2 minutes (healthy)
filebrowser Up 2 minutes (healthy)
== 9202 containers (name status)
felhom-controller Up 2 minutes (healthy)
filebrowser Up 2 minutes (healthy)
paperless-webserver Up 2 minutes (healthy)
paperless-postgres Up 2 minutes (healthy)
paperless-redis Up 2 minutes (healthy)
traefik Up 2 minutes
== agent log since boot (last 8, no payloads)
2026-11-10T13:58:58+01:00 demo-hp sudo[17988]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
2026-11-10T13:59:17+01:00 demo-hp sudo[19197]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
2026-11-10T13:59:17+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:59:17.882+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/f
2026-11-10T13:59:27+01:00 demo-hp sudo[19822]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
2026-11-10T13:59:28+01:00 demo-hp sudo[19859]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
2026-11-10T13:59:37+01:00 demo-hp sudo[20371]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
2026-11-10T13:59:37+01:00 demo-hp felhom-agent[1262]: time=2026-11-10T13:59:37.578+01:00 level=INFO msg="reconcile: enrolled drive bound under shared parent (live, no reboot)" vmid=9201 where=/mnt/hdd_1 guest_path=/mnt/f
2026-11-10T13:59:41+01:00 demo-hp sudo[20807]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
@@ -0,0 +1,47 @@
{
"host": "demo-hp",
"running": "7.0.14-28-pve",
"newest": "7.0.14-28-pve",
"phase": "good",
"step": {
"step_id": "ring0-20261110T014802Z",
"from": "7.0.14-26-pve",
"to": "7.0.14-28-pve",
"default_before": "7.0.14-26-pve",
"result_at": "2026-11-10T01:56:47Z"
},
"oneshot_flag": "",
"grub_default": "7.0.14-28-pve",
"pinned_default": "7.0.14-28-pve",
"boot_pin": null,
"installed": {
"7.0.14-26-pve": "proxmox-kernel-7.0.14-26-pve-signed",
"7.0.14-28-pve": "proxmox-kernel-7.0.14-28-pve-signed"
},
"keep": {
"7.0.14-26-pve": [
"the last kernel that booted well before the running one (state.json `from`)",
"kernel step `from` (a self-revert boots it)",
"kernel step `default_before`"
],
"7.0.14-28-pve": [
"running",
"newest installed",
"kernel step `to`",
"GRUB default in grub.cfg",
"pinned default file"
]
},
"computed_removal": [],
"removal": [],
"pretend": [],
"simulation": {
"rc": null,
"removes": [],
"installs": []
},
"refuse": [],
"verdict": "NOTHING TO REMOVE"
}
VERDICT: NOTHING TO REMOVE
@@ -0,0 +1,7 @@
# The hub's own row for demo-hp, read with CC's token (GET /hosts, read-only) at 2026-11-10T13:13:15Z — 16 minutes after the boot.
# A different channel from the box: this is what the hub holds from the agent's host report.
columns: Host | Customer | Agent | Proxmox / kernel | Status | Guests | CPU | Mem | Disk | Cloudflared | Worst Storage
row: demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 37% 42% running (connected) 71% local-lvm
# before the run (same page, read 2026-11-10 before the removal):
row: demo-hp-bb76ea Demo HP 0.172.1 floor: declared MinAgent 9.2.26 7.0.14-28-pve ONLINE 1/2 0% 49% 58% running (connected) 71% local-lvm
@@ -0,0 +1,162 @@
#!/usr/bin/env python3
"""kernel-cleanup-rule.py — the old-kernel clean-up RULE, as a read-only plan. EVIDENCE, NOT PRODUCT (spike 2026-11-10).
It runs ON the Proxmox host as root (`ssh demo-hp python3 - [options] < kernel-cleanup-rule.py`). It changes nothing:
it reads dpkg, `uname -r`, the kernel step's state file, the one-shot flag and grub.cfg, and it asks apt for a
SIMULATION (`apt-get -s`). It prints the keep set, the removal set and every reason to refuse. Exit code: 0 = the plan
is safe to run (or there is nothing to remove), 3 = REFUSED. The removal itself is one `apt-get purge` of the exact
names this prints; a person ran it by hand in the spike.
The rule (the brief's, `11-os-updates.md` §5.11.1):
KEEP the running kernel; the last kernel that booted well before it (the step's `from`, else the next lower
installed one); the newest installed kernel; and every kernel the kernel lane still names: the step's `to`,
the GRUB default in grub.cfg, the pinned default file, the one-shot flag, an operator's proxmox-boot pin.
REMOVE every other installed kernel image package, by its exact name.
REFUSE the whole clean-up when
P1 a kernel step is in a phase that has not ended (staged, oneshot, judging, reverting) or in an unknown phase;
P2 the one-shot flag names a kernel (a one-shot boot is armed);
P3 the pinned default and grub.cfg's default do not name the same kernel, or that kernel is not installed;
K1 a keep-kernel is in the removal set;
N1 a name in the removal set is not an installed kernel image package;
S1 the simulation fails, or would install or upgrade anything;
S2 the simulation would remove anything that is not in the removal set (a meta-package, proxmox-ve, anything).
Test inputs (negative controls, SIMULATION ONLY — nothing is ever removed by this file):
--pretend-set a,b,c use this removal set in place of the computed one
--pretend-add a,b add these names to the computed removal set
--pretend-phase NAME judge as if the kernel step were in this phase
"""
import json, os, re, subprocess, sys
STATE = "/var/lib/felhom-kernel/state.json"
DEFAULT_CFG = "/etc/default/grub.d/zz-felhom-kernel-default.cfg"
GRUB_CFG = "/boot/grub/grub.cfg"
ONESHOT_ENV = "/boot/efi/EFI/felhom/oneshot.env"
PIN_FILE = "/etc/kernel/proxmox-boot-pin"
IMAGE_RE = re.compile(r"^proxmox-kernel-([0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve)(-signed)?$") # = the wrapper's KERNEL_IMAGE_RE
KVER_RE = re.compile(r"[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve")
ENDED = ("good", "self_reverted", "revert_failed", "fell_back", "cancelled", "none") # the wrapper's phases, minus KERNEL_ACTIVE
def sh(cmd):
p = subprocess.run(cmd, capture_output=True, text=True)
return p.returncode, p.stdout, p.stderr
def key(kver):
return tuple(int(x) for x in re.findall(r"\d+", kver))
def read(path):
try:
return open(path, encoding="utf-8", errors="replace").read()
except OSError:
return None
def opt(name):
return sys.argv[sys.argv.index(name) + 1] if name in sys.argv else None
def main():
refuse = []
# --- what is installed (dpkg status "ii" only) ---
_, out, _ = sh(["dpkg-query", "-W", "-f", "${Package}\t${db:Status-Abbrev}\n", "proxmox-kernel-*"])
images = {} # kver -> package name
for line in out.splitlines():
name, _, st = line.partition("\t")
m = IMAGE_RE.match(name)
if m and st.strip() == "ii":
images[m.group(1)] = name
running = os.uname().release
newest = max(images, key=key) if images else None
# --- the kernel lane's own records ---
raw = read(STATE)
st = json.loads(raw) if raw else {}
phase = opt("--pretend-phase") or st.get("phase", "none")
if phase not in ENDED:
refuse.append(f"P1 a kernel step is in phase {phase!r} — not ended")
flag = ""
if os.path.exists(ONESHOT_ENV):
_, o, _ = sh(["grub-editenv", ONESHOT_ENV, "list"])
flag = dict(l.partition("=")[::2] for l in o.splitlines()).get("felhom_next", "")
if flag:
refuse.append(f"P2 the one-shot flag names {flag!r}")
cfg = read(GRUB_CFG) or ""
m = re.search(r'^\s*set default="[^"]*?(%s)[^"]*"' % KVER_RE.pattern, cfg, re.M)
grub_default = m.group(1) if m else None
m = KVER_RE.search(read(DEFAULT_CFG) or "")
pinned = m.group(0) if m else None
if pinned is None or pinned != grub_default or pinned not in images:
refuse.append(f"P3 pinned default {pinned!r}, grub.cfg default {grub_default!r} — not the same installed kernel")
m = KVER_RE.search(read(PIN_FILE) or "")
boot_pin = m.group(0) if m else None
# --- the keep set ---
keep = {}
def k(kver, why):
if kver:
keep.setdefault(kver, []).append(why)
k(running, "running")
last_good = st.get("from") if st.get("to") == running and st.get("from") in images else None
if not last_good:
lower = [v for v in images if key(v) < key(running)]
last_good = max(lower, key=key) if lower else None
k(last_good, "next lower installed kernel (no step record names the last good one)")
else:
k(last_good, "the last kernel that booted well before the running one (state.json `from`)")
k(newest, "newest installed")
k(st.get("from"), "kernel step `from` (a self-revert boots it)")
k(st.get("to"), "kernel step `to`")
k(st.get("default_before"), "kernel step `default_before`")
k(grub_default, "GRUB default in grub.cfg")
k(pinned, "pinned default file")
k(flag and (KVER_RE.search(flag) or [None])[0], "one-shot flag")
k(boot_pin, "operator's proxmox-boot pin")
if running not in images:
refuse.append(f"K1 the running kernel {running} is not an installed image package — the box is not as expected")
# --- the removal set: exact package names ---
computed = sorted((images[v] for v in images if v not in keep), key=lambda n: key(IMAGE_RE.match(n).group(1)))
removal = list(computed)
if opt("--pretend-set") is not None:
removal = [x for x in opt("--pretend-set").split(",") if x]
if opt("--pretend-add"):
removal += [x for x in opt("--pretend-add").split(",") if x]
by_name = {n: v for v, n in images.items()}
for n in removal:
if n not in by_name:
refuse.append(f"N1 {n} is not an installed kernel image package")
elif by_name[n] in keep:
refuse.append(f"K1 {n} is a keep-kernel ({'; '.join(keep[by_name[n]])})")
# --- apt's simulation of exactly that set ---
sim = {"rc": None, "removes": [], "installs": []}
if removal:
rc, o, e = sh(["apt-get", "-s", "-q", "-o", "Debug::NoLocking=1", "purge"] + removal)
sim["rc"] = rc
sim["removes"] = sorted({l.split()[1] for l in o.splitlines() if l.startswith(("Remv ", "Purg "))})
sim["installs"] = sorted({l.split()[1] for l in o.splitlines() if l.startswith(("Inst ", "Conf "))})
if rc != 0:
refuse.append("S1 the simulation failed: " + ((e or o).strip().splitlines() or ["?"])[-1][:200])
if sim["installs"]:
refuse.append(f"S1 the simulation would install or configure {', '.join(sim['installs'][:6])}")
extra = [p for p in sim["removes"] if p not in removal]
if extra:
refuse.append(f"S2 the simulation would also remove {', '.join(extra[:8])} — not in the removal set")
missing = [p for p in removal if p not in sim["removes"] and p in by_name]
if missing and rc == 0:
refuse.append(f"S1 the simulation does not remove {', '.join(missing[:6])}")
verdict = "REFUSED" if refuse else ("NOTHING TO REMOVE" if not removal else "SAFE TO RUN")
res = {"host": os.uname().nodename, "running": running, "newest": newest, "phase": phase, "step": {x: st.get(x) for x in
("step_id", "from", "to", "default_before", "result_at")}, "oneshot_flag": flag, "grub_default": grub_default,
"pinned_default": pinned, "boot_pin": boot_pin, "installed": {v: images[v] for v in sorted(images, key=key)},
"keep": {v: keep[v] for v in sorted(keep, key=key)}, "computed_removal": computed, "removal": removal,
"pretend": [a for a in sys.argv[1:]], "simulation": sim, "refuse": refuse, "verdict": verdict}
print(json.dumps(res, indent=1))
print(f"\nVERDICT: {verdict}")
for r in refuse:
print(" REFUSE " + r)
if not refuse and removal:
print(" the one command: apt-get -y -q purge " + " ".join(removal))
return 3 if refuse else 0
sys.exit(main())
+1 -1
View File
@@ -281,7 +281,7 @@ stopping line that lies.
| **R-1301** | Box system & updates | P3 | **The N100's Proxmox host keeps no time: `chrony` fails at every boot** („Fatal error : Could not send notification to $NOTIFY_SOCKET", seen 2026-10-16 and 2026-10-21 boots); measured 2026-10-26 20:06 UTC: the host clock 14 s behind DooPlex and demo-hp, `System clock synchronized: no`. demo-hp has the same package (4.8-4~bpo13+2) and `DAEMON_OPTS="-F 1"` and runs it. A drifting host clock feeds backup times (R-1206's class) and TLS. Found while timing the cloudflared update (R-1300). `audits/cloudflared-2026.10.3-2026-10-26/obs-n100-clock.txt` | OPEN — filed 2026-10-26 | — | find why the notify socket fails on this host only (compare the unit and `systemd-analyze` with demo-hp), fix it, and check whether a fresh install could hit it too | CC |
| **R-1260** | Box system & updates | P3 | **Spring 2027 (2027-03-28): two clock-change effects in the controller's night (review round 2, HUNT-4 H2/H3).** The update leg's deadline is W + 5 h of REAL time while the full-system backup's gate counts wall-clock minutes, so a step can start after the gate stopped waiting (`felhom-controller/controller/internal/stacks/unattended.go:225`; `internal/quiesce/quiesce.go:922-933`); with W in 02:00–02:59 the DB dump and the second-drive copy start at the same instant (`internal/scheduler/scheduler.go:509`; VERIFIED: Go maps 2027-03-28 02:30 Europe/Budapest to 03:30 CEST). | OPEN | — | before 2027-03-28: one wall-clock basis for both, and a test at the spring date | CC |
| **R-1367** | Box system & updates | P3 | **The host journal fills a small system disk: 61% of its lines are `pam_unix(sudo:session)` open/close pairs from the agent's sudo calls.** MEASURED 2026-11-09, one hour, same on demo-hp and the N100: ~14,000 journal lines, ~8,500 of them sudo session lines; the top commands are `smartctl -a` (1,074 per hour — one every ~3 s), `lxc-info -n` (662), `mkdir -p` (540), `lvs` (537), `pct exec` (~520). The journal grows by journald's default cap (10% of the filesystem, max 4 GB): the N100 sits at 4 GB, demo-hp was 3.8 GB on 2026-11-07 and reached 0.7 GB again two days after a vacuum to 500 MB. On a 39 GB root that space is what the whole-guest local copy needs (R-548). | OPEN — filed 2026-11-09 **2026-11-10 (space list, `documentation/audits/space-list-2026-11-10/TABLE.md`):** place H7: the N100 (4.28 GB, 74 files) and Tester 1 (4.29 GB, 81 files) sit at the journal's own cap; Tester 1 writes about 150 MB a day. Lines written by `sudo` are 92 / 92 / 90 % of all lines in one hour — this counts three lines for each call (the command and the session pair), while this row's 61 % counts the session pair only: the same fact, counted two ways. No rank change is proposed: the journal stops at its own cap (about 4.3 GB), so it costs space and log history, not safety. | — | Agent, through the signed config bundle: `Defaults:felhom-agent !pam_session` (and `!syslog` only if the command log has no reader — check the audit trail's needs first) in the sudoers file; a journald cap drop-in (e.g. `SystemMaxUse=500M`) on the host; and read why `smartctl -a` runs every ~3 s (the SMART poll's interval or a loop). Red-proof each in the agent's tests; measure the line rate after on demo-hp. Evidence: `/mnt/5_hdd/felhom.eu/evidence/demo-hp-images-2026-11-09/journal-noise.txt` (off-git) | CC |
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
| **R-1368** | Box system & updates | P3 | **Five places on the host's system disk only grow: nothing deletes them and no rule says how many to keep.** MEASURED 2026-11-10 on demo-hp, demo-felhom and Tester 1 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` places H5, H6, H9, H12, H13). (1) **Old kernels** (`/lib/modules`, `/boot`): 8 / 8 / 3 kernels = 8.15 / 8.15 / 3.05 GB. The ring-0 boxes got 7 kernels in 37 days and no kernel was ever removed: the update wrapper refuses a plan that removes a package (`11-os-updates.md` rule R4), and nothing runs `apt autoremove` (searched: architecture folder, both registers, runbooks, three source trees — 0 hits). The N100's system disk grew 4.6 GiB in 26 days, about a kernel a week. (2) **The agent's self-update folder** `/var/lib/felhom-agent/selfupdate`: 49 / 48 / 33 binaries = 740 / 724 / 509 MB, one for each agent version the box ever got; no document names the folder. (3) **Guest template archives** `vzdump-lxc-9100-*` in `/var/lib/vz/dump`: 1 / 4 / 1 = 0.65 / 2.56 / 0.65 GB. (4) **Old agent binary copies** `/usr/local/bin/felhom-agent.bak-*`: 1 / 11 / 0 files, up to 155 MB. (5) **Logs of failed whole-guest backup runs** in the dump folder (small; 9 on demo-hp). **Members (answers marked N):** H5-3, H5-4, H6-3, H9-3, H9-4, H9-8, H12-3, H12-4, H13-3, H13-4. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D67 = yes, as a separate step; `09` §3 decision 286):** the box may remove old kernels by itself; rule R4 stays as it is for update plans; the clean-up is its own step with its own narrow rule (old kernel packages only, only after a good boot on the newest kernel). **2026-11-10:** the endings list leans on this row (`documentation/audits/endings-list-2026-11-10/TABLE.md` EA01-4, EA23-4, EA24-4, EA26-4). **2026-11-10 — the kernel part's fix shape is PROVEN by hand on demo-hp** (`documentation/audits/kernel-cleanup-spike-2026-11-10/`; design `11-os-updates.md` §5.11.1, not built): the rule removed 6 of 8 kernels and nothing else, 6.4 GB gained, boot settings unchanged, one orderly reboot back on the same kernel, a second run removed nothing. Not proven: a box whose running kernel is not its newest; a boot of the kept old kernel. | the kernel part: the hand proof on demo-hp (`documentation/audits/kernel-cleanup-spike-2026-11-10/`), then an agent release | Agent: after a good update keep the running kernel, the one before it and the newest, and remove the rest — ring 0 first; keep the newest N self-update binaries and `.bak` copies; keep one template archive. Each with a test that a kept item is never removed. | CC |
| **R-1369** | Box system & updates | P3 | **When a disk is full there is no written way out: no runbook and no self-repair, for the host system disk, the thin pool, the guest disks or a data drive.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md`, question 7). One missing runbook, seen from 21 places; the search of the runbooks folder for a disk-full procedure gave 0 hits (`evidence/searches.txt` S6). The alarms exist (hub: 90 / 95 % host disk, 85 / 90 % thin pool; controller: 85 / 95 % on the data volume and each drive) but they end in a mail, not in a step. For three small places nobody wrote what fails when they are full (EFI partition, host and guest temp folders). **Members (answers marked N):** H1-7, H2-7, H5-7, H9-7, H10-5, H10-7, H12-7, H13-7, H15-5, G1-7, G2-7, G7-7, G12-7, G13-7, G14-7, G17-5, D1-7, D2-7, D3-7, D6-7, D7-7, D8-7, D9-7, D10-7. | OPEN — filed 2026-11-10 (space list) | — | Docs: one runbook „a disk is full” with one section per disk (host system disk, thin pool, guest root, data volume, data drive): what to look at first, what is safe to delete (the places of R-1368 and R-1367), what is never deleted by hand (household data, the kept-data rulings). Rehearse it once on demo-hp. | CC |
| **R-1370** | Box system & updates | P3 | **The host's system disk has no space budget: no document says how much the host's own files may take, and the installer's sizing rule covers the guest only.** Found 2026-11-10 (the space list, `documentation/audits/space-list-2026-11-10/TABLE.md` place H1, answer H1-3; search S5 — 0 hits). Today: demo-hp 58 % of 41 GB, N100 38 % of 101 GB, Tester 1 34 % of 63 GB — and the parts that grow are the journal (R-1367), old kernels and agent binaries (R-1368), old backup copies (R-1087) and the local whole-guest copy (R-548). Second half, known but with no row until now: on a small disk the guest's own guards read the guest's 70 GiB while the thin pool behind it holds 54 GiB (demo-hp: 102 GiB promised on a 53.9 GiB pool, 70.67 % used), so only the hub's thin-pool alarm stands before a full pool. **Member:** H1-3. | OPEN — filed 2026-11-10 (space list) **2026-11-10 12:22 — RULED (D69 = yes; `09` §3 decision 288):** the section goes inside `16-storage-physical-identity.md`, no new document. | — | Docs: a short „disk space” section in an architecture document: each part of the system disk with its limit and its cleaner, pointing at the audit table. Then the installer can refuse a disk that cannot hold the sum. | CC |
| **R-984** | Box system & updates | P4 | **Host and agent hardening, low risk each: the backup-target route has no role check; the unsigned guest spec has no size bounds; signed ops have no maximum validity; the guarded mkfs does not refuse LVs, zvols or RAID devices; the local API is reachable from the LAN by a static route; a sudoers inspect pattern is wider than needed; install passes secrets on the command line; guest settings files are world-readable; the host offers rpcbind and DNS on the LAN.** Fix each in its own small change. Security review 2026-10-12 (`audits/security-review-2026-10-12/`); detail held off-git for the operator (`/mnt/5_hdd/felhom.eu/evidence/security-review-2026-10-12-PRIVATE/`). | OPEN | — | Batch into hardening work | CC |