night build 2026-11-14: the N100's first night kernel removal read (D82: 1 of 2), close-out — report, status, register
gates / gates (push) Successful in 2m8s

Reads only on the N100 and demo-hp; no box changed, nothing released.

Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
admin committed 2026-11-15 06:41:21 +01:00
1 parent 5da5b05192
commit fffbb40ce0
17 files changed
+830 -4

No files matched your search

@@ -0,0 +1,56 @@
# Part E — the morning read after the first night a box removed kernels by itself (2026-11-15, 06:33 CET)
Reads only. Nothing was written on a box: no service action, no package action, no guest started or stopped, no file
made. The three read scripts are in this folder as they were run (`partE-read.sh`, `partE-read2.sh`,
`partE-read3.sh`), each as `ssh <box> 'bash -s' < script`. Tester 1 was not read on the box: the hub's page did not
show the line that would have asked for it.
## The N100 (`demo-felhom-8363b5`) — a real removal, healthy
| Fact | Before | After | Where |
|---|---|---|---|
| Kernel packages installed | 8 (the wrapper's own count at 05:58:01) | 2: 7.0.14-26 and 7.0.14-28 | `felhom-pve-3.txt`, `felhom-pve-2.txt` (dpkg state) |
| Removed | — | 6: 7.0.2-6, 7.0.14-20, -22, -23, -24, -25; purged, 0 left-over paths | `felhom-pve-2.txt` (dpkg log, apt history: requested by the agent's user) |
| Time | 05:58:01 | 05:58:35 (34.6 s) | the same |
| Space gained | — | 6,810,329,088 bytes (6.81 GB, 6.3 GiB) by the wrapper's figure; 65.10 GB free after | `felhom-pve-3.txt`, `felhom-pve.txt` |
| Running kernel | 7.0.14-28 | 7.0.14-28 (no reboot; up 1 week) | `felhom-pve.txt` |
| Default kernel | 7.0.14-28 | 7.0.14-28, „proved from grub.cfg” by the wrapper and read again from `grub.cfg` | both |
| The pinned file and the kernel state file | written 2026-11-08 05:36 | the same times | `felhom-pve.txt` |
| Health | — | 0 failed units; guest 9201 running, 9 containers, 0 unhealthy, 0 exited; dpkg audit empty | `felhom-pve.txt` |
| Time service | — | active, leap normal; the box is +0.064 s from DooPlex (round trip 0.139 s) | `felhom-pve.txt` |
| The hub's „Kernel step” cell | — | „cleaned 36 min ago”, the hover text names the 6 packages and the 2 kept | `hub-system-page-0634.txt` line 97 |
The clean-up ran at the end of the night's update run, after the Proxmox step applied 12 packages. One line in the
journal reads „dd invoked oom-killer” at 05:57:20: that is the update run's own memory-kill check (it reports
`OOM-CHECK result=pass`), not a fault.
## demo-hp (`demo-hp-bb76ea`) — a quiet night
The night run at 05:02–05:04 found 2 kernels installed, 2 to keep, 0 to remove („nothing to remove”). The box's six
old kernels were purged on 2026-11-10 at 13:51, by day (apt history). Health: 0 failed units; guests 9201 (26
containers) and 9202 (6 containers) healthy; clock +0.071 s from DooPlex. The hub's cell still shows its newest kernel
step („applied 5 days ago”), which is correct for a clean-up that removed nothing.
## Tester 1 (`tester-1-d70be4`) — the hub's line
„Crash guard: armed”; root files 0.174.0; kernel.panic 10 s; health „fine”. The line „crash guard not installed” is
not there, so no read on the box and no row.
## The D82 count (R-1368)
Two real night removals on each ring-0 box. **The N100: 1 of 2. demo-hp: 0 of 2.** Both boxes now hold 2 kernels and
keep 2, so the next real removal needs new kernels to arrive first (the hub shows 7.0.14-29 as due on both).
## Two instrument defects in the first read, said plainly
1. The first script's kernel count printed 0 on both boxes. Its pattern looked for the package name at the end of a
line that ends with the state. The second read lists the dpkg states directly.
2. The first and second scripts' journal filters printed nothing, and the positive control (a line count) read 0:
`--since "yesterday 20:00"` gave no lines over this ssh form, with the error hidden. The third read uses a full
date and shows 139,573 and 140,523 lines.
## One answer to a question of the night's review
The folder that holds the profile-refresh record is `drwx------ root`, so the agent's user cannot read the record
file (read on both boxes). The night's build takes the result from the root step's answer and treats the file as a
bonus, so this does not break it; the file read stays dead until the folder's mode changes.
@@ -0,0 +1,37 @@
#### demo-hp 2026-11-15 06:33:42 CET
## kernel packages by dpkg state
ii |proxmox-kernel-7.0|7.0.14-28
un |proxmox-kernel-7.0.14-26-pve|
ii |proxmox-kernel-7.0.14-26-pve-signed|7.0.14-26
un |proxmox-kernel-7.0.14-28-pve|
ii |proxmox-kernel-7.0.14-28-pve-signed|7.0.14-28
ii |proxmox-kernel-helper|9.1.0+fde2
## dpkg log: kernel package removals and purges, last 3 days
## apt history: last 3 entries that name a kernel
Start-Date: 2026-11-10 13:51:11
Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef purge proxmox-kernel-7.0.2-6-pve-signed proxmox-kernel-7.0.14-20-pve-signed proxmox-kernel-7.0.14-22-pve-signed proxmox-kernel-7.0.14-23-pve-signed proxmox-kernel-7.0.14-24-pve-signed proxmox-kernel-7.0.14-25-pve-signed
Purge: proxmox-kernel-7.0.14-20-pve-signed:amd64 (7.0.14-20), proxmox-kernel-7.0.14-22-pve-signed:amd64 (7.0.14-22), proxmox-kernel-7.0.14-23-pve-signed:amd64 (7.0.14-23), proxmox-kernel-7.0.14-24-pve-signed:amd64 (7.0.14-24), proxmox-kernel-7.0.2-6-pve-signed:amd64 (7.0.2-6), proxmox-kernel-7.0.14-25-pve-signed:amd64 (7.0.14-25)
End-Date: 2026-11-10 13:52:00
Start-Date: 2026-11-14 05:00:45
## agent journal since yesterday 20:00: every line with osupdate / kernel / clean
## agent journal line count since yesterday 20:00 (a positive control that the journal is readable)
0
## the whole journal since yesterday 20:00: lines with felhom-os-apply or KERNEL
## wrapper state folder (names and times only)
total 24
drwx------ 4 root root 4096 2026-11-14 12:33 .
drwxr-xr-x 46 root root 4096 2026-10-10 04:42 ..
drwxr-xr-x 2 root root 4096 2026-11-14 12:18 agent-update
drwxr-xr-x 5 root root 4096 2026-11-14 12:33 bundle-prev
-rw-r--r-- 1 root root 96 2026-11-14 12:33 nonces.json
-rw-r--r-- 1 root root 137 2026-11-14 12:33 profile-refresh.json
total 16
drwxr-xr-x 2 root root 4096 2026-11-10 02:56 .
drwxr-xr-x 46 root root 4096 2026-10-10 04:42 ..
-rw-r--r-- 1 root root 5701 2026-11-10 02:56 state.json
## disk
21968302080 17318748160 45%
21968302080 17318748160 45%
#### end
@@ -0,0 +1,50 @@
#### demo-hp 2026-11-15 06:34:01 CET
## control: agent journal lines since 2026-11-14 20:00
140523
Nov 14 20:00:01 demo-hp sudo[3471829]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
Nov 14 20:00:01 demo-hp sudo[3471829]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
## agent journal: osupdate / kernel / clean-up lines
2026-11-15T05:02:48+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:02:48.557+01:00 level=INFO msg="osupdate: START" run=20261115T040248Z layer=guest vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T040248Z
2026-11-15T05:03:02+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:02.991+01:00 level=INFO msg="osupdate: a pass is running — its start sends any kept report"
2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.208+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T040248Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.208+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.208+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.208+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.209+01:00 level=INFO msg="osupdate: DONE" run=20261115T040248Z layer=guest vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=19.5
2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.303+01:00 level=INFO msg="osupdate: START" run=20261115T040248Z layer=host vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T040248Z
2026-11-15T05:03:25+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:25.491+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T040248Z layer=host lane=fast mode=apply select=pending-fast packages=0"
2026-11-15T05:03:25+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:25.491+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-15T05:03:25+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:25.491+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-15T05:03:25+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:25.491+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-15T05:03:26+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:26.644+01:00 level=INFO msg="osupdate: DONE" run=20261115T040248Z layer=host vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=30 not_covered=30 restart_needed=1 reboot_needed=false wrapper_seconds=17.1
2026-11-15T05:03:29+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:29.145+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on"
2026-11-15T05:03:29+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:29.145+01:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"result\": \"already on\"}"
2026-11-15T05:03:29+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:29.145+01:00 level=INFO msg="osupdate: START" run=20261115T040248Z layer=docker vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T040248Z
2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.415+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T040248Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0"
2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.415+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.415+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.415+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.415+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: OOM-CHECK result=pass oom_killed=True oom_event=True exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0.352.1 — the engine reported the memory kill: OOMKilled=true and the oom event"
2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.416+01:00 level=INFO msg="osupdate: DONE" run=20261115T040248Z layer=docker vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=30.2
2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.423+01:00 level=INFO msg="osupdate: pve step holds the /etc/pve write gate — the agent's own writes wait until it ends" run=20261115T040248Z layer=pve vmid=9201 trigger=night
2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.423+01:00 level=INFO msg="osupdate: START" run=20261115T040248Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T040248Z
2026-11-15T05:04:46+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:46.366+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T040248Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0"
2026-11-15T05:04:46+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:46.366+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-15T05:04:46+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:46.366+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0"
2026-11-15T05:04:46+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:46.366+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=25.4 upgraded=12 restart-needed=kvm reboot-needed=no"
2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.480+01:00 level=INFO msg="osupdate: DONE" run=20261115T040248Z layer=pve vmid=9201 ring=0 trigger=night outcome=applied healthy=true reason="" upgraded=12 pending=2 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=46.8
2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.509+01:00 level=INFO msg="osupdate: pve step released the /etc/pve write gate" run=20261115T040248Z layer=pve vmid=9201 trigger=night
2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.509+01:00 level=INFO msg="osupdate: kernel step skipped — the hub names no kernel step for this box" run=20261115T040248Z layer=kernel vmid=9201 trigger=night ring=0
2026-11-15T05:04:47+01:00 demo-hp sudo[1134694]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T040248Z-kernel-kernel-cleanup.json
2026-11-15T05:04:47+01:00 demo-hp felhom-os-apply[1134710]: os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=2 keep=2 removal=0 not_run=[] refusals=[]
2026-11-15T05:04:47+01:00 demo-hp felhom-os-apply[1134711]: os-apply: KERNEL CLEANUP nothing to remove
2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.855+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.855+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=2 keep=2 removal=0 not_run=[] refusals=[]"
2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.855+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP nothing to remove"
2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.855+01:00 level=INFO msg="osupdate: kernel clean-up — nothing to remove" run=20261115T040248Z layer=kernel mode=kernel-cleanup vmid=9201 trigger=night kept=7.0.14-26-pve,7.0.14-28-pve took=0s
## whole journal 05:30-06:30: wrapper and sudo lines
2026-11-15T05:33:13+01:00 demo-hp sudo[1229107]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts043312-host-facts.json
2026-11-15T05:48:13+01:00 demo-hp sudo[1279117]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts044813-host-facts.json
2026-11-15T06:03:13+01:00 demo-hp sudo[1329378]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts050313-host-facts.json
2026-11-15T06:18:13+01:00 demo-hp sudo[1379432]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts051813-host-facts.json
#### end
@@ -0,0 +1,33 @@
#### demo-hp 2026-11-15T05:33:09Z local 2026-11-15 06:33:09 CET epoch 1794720789.075455874
agent: felhom-agent 0.174.0; uptime: up 4 days, 14 hours, 50 minutes; running kernel: 7.0.14-28-pve
## kernels installed (ii)
count: 0
/boot: 7.0.14-26-pve 7.0.14-28-pve
/lib/modules: 7.0.14-26-pve 7.0.14-28-pve
## default kernel and the pinned file
grub.cfg default: gnulinux-7.0.14-28-pve-advanced
6b35d9aeddc1ddd0 2026-11-10 02:56:43 /etc/default/grub.d/zz-felhom-kernel-default.cfg
607a2ce7ce3b6ca1 2026-11-10 02:56:47 /var/lib/felhom-kernel/state.json
f2f56d59c6356067 2026-10-07 18:01:30 /etc/grub.d/01_felhom_oneshot
c34907ce68599873 2026-10-07 18:01:30 /etc/grub.d/42_felhom_oneshot
pinned file content: # felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-a
kernel state: { "authority": "ring0", "default_before": "7.0.14-26-pve", "from": "7.0.14-26-pve", "health_before": { "guest": { "containers": { "adventurelog": { "health": "healthy", "id": "50af393c36a37d01a10847456d391d70572c121045e28626f220f83cec1cefcc", "state": "running" }, "adventurelog-frontend": { "health": "healthy", "id": "ab
## disk
/ free 21.97 GB, used 45%
## the night's lines: the agent's OS leg and the root wrapper (since yesterday 22:00 local)
## the wrapper's own journal lines about the clean-up
## health
failed units: 0
guest 9201: running; containers 26 running, 0 unhealthy, 0 exited
guest 9202: running; containers 6 running, 0 unhealthy, 0 exited
guest 9401: stopped
dpkg audit: []
## time service
chrony: active; Reference ID : C1292F54 (ntp1.iszt.hu);System time : 0.000095636 seconds slow of NTP time;Leap status : Normal;
## the profile-refresh record's folder (a question of the night's review: can the agent read it?)
drwx------ 4 root root 4096 Nov 14 12:33 /var/lib/felhom-os-apply
-rw-r--r-- 1 root root 137 Nov 14 12:33 /var/lib/felhom-os-apply/profile-refresh.json
can the agent user read it: no
#### end demo-hp epoch 1794720799.360448067
#### DooPlex: ssh rc=0, started epoch 1794720788.467514838, ended epoch 1794720799.365384080
#### clock: box minus DooPlex (middle of the round trip) = +0.071 s; round trip 0.151 s
@@ -0,0 +1,65 @@
#### demo-felhom 2026-11-15 06:33:41 CET
## kernel packages by dpkg state
ii |proxmox-kernel-7.0|7.0.14-28
un |proxmox-kernel-7.0.14-26-pve|
ii |proxmox-kernel-7.0.14-26-pve-signed|7.0.14-26
un |proxmox-kernel-7.0.14-28-pve|
ii |proxmox-kernel-7.0.14-28-pve-signed|7.0.14-28
ii |proxmox-kernel-helper|9.1.0+fde2
## dpkg log: kernel package removals and purges, last 3 days
2026-11-15 05:58:01 remove proxmox-kernel-7.0.14-20-pve-signed:amd64 7.0.14-20 <none>
2026-11-15 05:58:04 status config-files proxmox-kernel-7.0.14-20-pve-signed:amd64 7.0.14-20
2026-11-15 05:58:04 remove proxmox-kernel-7.0.14-22-pve-signed:amd64 7.0.14-22 <none>
2026-11-15 05:58:07 status config-files proxmox-kernel-7.0.14-22-pve-signed:amd64 7.0.14-22
2026-11-15 05:58:07 remove proxmox-kernel-7.0.14-23-pve-signed:amd64 7.0.14-23 <none>
2026-11-15 05:58:10 status config-files proxmox-kernel-7.0.14-23-pve-signed:amd64 7.0.14-23
2026-11-15 05:58:10 remove proxmox-kernel-7.0.14-24-pve-signed:amd64 7.0.14-24 <none>
2026-11-15 05:58:13 status config-files proxmox-kernel-7.0.14-24-pve-signed:amd64 7.0.14-24
2026-11-15 05:58:13 remove proxmox-kernel-7.0.14-25-pve-signed:amd64 7.0.14-25 <none>
2026-11-15 05:58:16 status config-files proxmox-kernel-7.0.14-25-pve-signed:amd64 7.0.14-25
2026-11-15 05:58:16 remove proxmox-kernel-7.0.2-6-pve-signed:amd64 7.0.2-6 <none>
2026-11-15 05:58:19 status config-files proxmox-kernel-7.0.2-6-pve-signed:amd64 7.0.2-6
2026-11-15 05:58:19 purge proxmox-kernel-7.0.14-22-pve-signed:amd64 7.0.14-22 <none>
2026-11-15 05:58:19 status config-files proxmox-kernel-7.0.14-22-pve-signed:amd64 7.0.14-22
2026-11-15 05:58:22 status not-installed proxmox-kernel-7.0.14-22-pve-signed:amd64 <none>
2026-11-15 05:58:22 purge proxmox-kernel-7.0.14-20-pve-signed:amd64 7.0.14-20 <none>
2026-11-15 05:58:22 status config-files proxmox-kernel-7.0.14-20-pve-signed:amd64 7.0.14-20
2026-11-15 05:58:24 status not-installed proxmox-kernel-7.0.14-20-pve-signed:amd64 <none>
2026-11-15 05:58:24 purge proxmox-kernel-7.0.2-6-pve-signed:amd64 7.0.2-6 <none>
2026-11-15 05:58:24 status config-files proxmox-kernel-7.0.2-6-pve-signed:amd64 7.0.2-6
2026-11-15 05:58:27 status not-installed proxmox-kernel-7.0.2-6-pve-signed:amd64 <none>
2026-11-15 05:58:27 purge proxmox-kernel-7.0.14-25-pve-signed:amd64 7.0.14-25 <none>
2026-11-15 05:58:27 status config-files proxmox-kernel-7.0.14-25-pve-signed:amd64 7.0.14-25
2026-11-15 05:58:30 status not-installed proxmox-kernel-7.0.14-25-pve-signed:amd64 <none>
2026-11-15 05:58:30 purge proxmox-kernel-7.0.14-24-pve-signed:amd64 7.0.14-24 <none>
2026-11-15 05:58:30 status config-files proxmox-kernel-7.0.14-24-pve-signed:amd64 7.0.14-24
2026-11-15 05:58:32 status not-installed proxmox-kernel-7.0.14-24-pve-signed:amd64 <none>
2026-11-15 05:58:32 purge proxmox-kernel-7.0.14-23-pve-signed:amd64 7.0.14-23 <none>
2026-11-15 05:58:32 status config-files proxmox-kernel-7.0.14-23-pve-signed:amd64 7.0.14-23
2026-11-15 05:58:35 status not-installed proxmox-kernel-7.0.14-23-pve-signed:amd64 <none>
## apt history: last 3 entries that name a kernel
Start-Date: 2026-11-15 05:58:01
Commandline: apt-get -y -q purge proxmox-kernel-7.0.14-20-pve-signed proxmox-kernel-7.0.14-22-pve-signed proxmox-kernel-7.0.14-23-pve-signed proxmox-kernel-7.0.14-24-pve-signed proxmox-kernel-7.0.14-25-pve-signed proxmox-kernel-7.0.2-6-pve-signed
Requested-By: felhom-agent (999)
Purge: proxmox-kernel-7.0.14-20-pve-signed:amd64 (7.0.14-20), proxmox-kernel-7.0.14-22-pve-signed:amd64 (7.0.14-22), proxmox-kernel-7.0.14-23-pve-signed:amd64 (7.0.14-23), proxmox-kernel-7.0.14-24-pve-signed:amd64 (7.0.14-24), proxmox-kernel-7.0.2-6-pve-signed:amd64 (7.0.2-6), proxmox-kernel-7.0.14-25-pve-signed:amd64 (7.0.14-25)
End-Date: 2026-11-15 05:58:35
## agent journal since yesterday 20:00: every line with osupdate / kernel / clean
## agent journal line count since yesterday 20:00 (a positive control that the journal is readable)
0
## the whole journal since yesterday 20:00: lines with felhom-os-apply or KERNEL
## wrapper state folder (names and times only)
total 24
drwx------ 4 root root 4096 2026-11-14 12:51 .
drwxr-xr-x 48 root root 4096 2026-10-08 04:39 ..
drwxr-xr-x 2 root root 4096 2026-11-14 12:35 agent-update
drwxr-xr-x 5 root root 4096 2026-11-14 12:51 bundle-prev
-rw-r--r-- 1 root root 96 2026-11-14 12:51 nonces.json
-rw-r--r-- 1 root root 196 2026-11-14 12:51 profile-refresh.json
total 12
drwxr-xr-x 2 root root 4096 2026-11-08 05:36 .
drwxr-xr-x 48 root root 4096 2026-10-08 04:39 ..
-rw-r--r-- 1 root root 2615 2026-11-08 05:36 state.json
## disk
65095643136 30595346432 32%
65095643136 30595346432 32%
#### end
@@ -0,0 +1,76 @@
#### demo-felhom 2026-11-15 06:33:58 CET
## control: agent journal lines since 2026-11-14 20:00
139573
Nov 14 20:00:00 demo-felhom sudo[2436686]: pam_unix(sudo:session): session closed for user root
Nov 14 20:00:00 demo-felhom sudo[2436691]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
## agent journal: osupdate / kernel / clean-up lines
2026-11-15T05:56:34+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:34.495+01:00 level=INFO msg="osupdate: START" run=20261115T045634Z layer=guest vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T045634Z
2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.839+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T045634Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.839+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.840+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.840+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.840+01:00 level=INFO msg="osupdate: DONE" run=20261115T045634Z layer=guest vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=14.3
2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.901+01:00 level=INFO msg="osupdate: START" run=20261115T045634Z layer=host vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T045634Z
2026-11-15T05:57:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:00.040+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T045634Z layer=host lane=fast mode=apply select=pending-fast packages=0"
2026-11-15T05:57:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:00.040+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-15T05:57:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:00.040+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-15T05:57:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:00.040+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-15T05:57:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:00.867+01:00 level=INFO msg="osupdate: DONE" run=20261115T045634Z layer=host vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=29 not_covered=29 restart_needed=0 reboot_needed=false wrapper_seconds=11.1
2026-11-15T05:57:02+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:02.546+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on"
2026-11-15T05:57:02+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:02.546+01:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"result\": \"already on\"}"
2026-11-15T05:57:02+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:02.546+01:00 level=INFO msg="osupdate: START" run=20261115T045634Z layer=docker vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T045634Z
2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.217+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T045634Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0"
2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.217+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.217+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.217+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.217+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: OOM-CHECK result=pass oom_killed=True oom_event=True exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0.352.1 — the engine reported the memory kill: OOMKilled=true and the oom event"
2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.218+01:00 level=INFO msg="osupdate: DONE" run=20261115T045634Z layer=docker vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=22.6
2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.225+01:00 level=INFO msg="osupdate: pve step holds the /etc/pve write gate — the agent's own writes wait until it ends" run=20261115T045634Z layer=pve vmid=9201 trigger=night
2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.225+01:00 level=INFO msg="osupdate: START" run=20261115T045634Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T045634Z
2026-11-15T05:57:59+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:59.155+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T045634Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0"
2026-11-15T05:57:59+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:59.155+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-15T05:57:59+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:59.155+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0"
2026-11-15T05:57:59+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:59.155+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=20.1 upgraded=12 restart-needed=- reboot-needed=no"
2026-11-15T05:57:59+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:59.998+01:00 level=INFO msg="osupdate: DONE" run=20261115T045634Z layer=pve vmid=9201 ring=0 trigger=night outcome=applied healthy=true reason="" upgraded=12 pending=1 not_covered=0 restart_needed=0 reboot_needed=false wrapper_seconds=33.9
2026-11-15T05:58:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:00.007+01:00 level=INFO msg="osupdate: pve step released the /etc/pve write gate" run=20261115T045634Z layer=pve vmid=9201 trigger=night
2026-11-15T05:58:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:00.007+01:00 level=INFO msg="osupdate: kernel step skipped — the hub names no kernel step for this box" run=20261115T045634Z layer=kernel vmid=9201 trigger=night ring=0
2026-11-15T05:58:00+01:00 demo-felhom sudo[3123080]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-kernel-kernel-cleanup.json
2026-11-15T05:58:01+01:00 demo-felhom felhom-os-apply[3123143]: os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=8 keep=2 removal=6 not_run=[] refusals=[]
2026-11-15T05:58:35+01:00 demo-felhom felhom-os-apply[3132623]: os-apply: KERNEL CLEANUP DONE removed=6 (proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed) kept=7.0.14-26-pve, 7.0.14-28-pve gained=6810329088 B seconds=34.6 leftovers=0; default 7.0.14-28-pve (proved from grub.cfg)
2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=8 keep=2 removal=6 not_run=[] refusals=[]"
2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP DONE removed=6 (proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed) kept=7.0.14-26-pve, 7.0.14-28-pve gained=6810329088 B seconds=34.6 leftovers=0; default 7.0.14-28-pve (proved from grub.cfg)"
2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: kernel clean-up — old kernels removed" run=20261115T045634Z layer=kernel mode=kernel-cleanup vmid=9201 trigger=night removed=6 names=proxmox-kernel-7.0.14-20-pve-signed,proxmox-kernel-7.0.14-22-pve-signed,proxmox-kernel-7.0.14-23-pve-signed,proxmox-kernel-7.0.14-24-pve-signed,proxmox-kernel-7.0.14-25-pve-signed,proxmox-kernel-7.0.2-6-pve-signed kept=7.0.14-26-pve,7.0.14-28-pve bytes_gained=0x118111de4540 leftovers=0 took=36s
2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: DONE" run=20261115T045634Z layer=kernel mode=kernel-cleanup vmid=9201 trigger=night outcome=cleaned healthy=true reason="removed 6 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed; kept 7.0.14-26-pve, 7.0.14-28-pve; 6.3 GiB gained; 0 left-over path(s)" upgraded=0 pending=0 not_covered
## whole journal 05:30-06:30: wrapper and sudo lines
2026-11-15T05:56:39+01:00 demo-felhom felhom-os-apply[3119297]: os-apply: REPAIR configured=0 journal=0 fixed=0
2026-11-15T05:56:43+01:00 demo-felhom felhom-os-apply[3119391]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0
2026-11-15T05:56:43+01:00 demo-felhom felhom-os-apply[3119392]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
2026-11-15T05:56:48+01:00 demo-felhom sudo[3119650]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-host-apply.json
2026-11-15T05:56:49+01:00 demo-felhom felhom-os-apply[3119656]: os-apply: START release=ring0-20261115T045634Z layer=host lane=fast mode=apply select=pending-fast packages=0
2026-11-15T05:56:52+01:00 demo-felhom felhom-os-apply[3119745]: os-apply: REPAIR configured=0 journal=0 fixed=0
2026-11-15T05:56:54+01:00 demo-felhom felhom-os-apply[3119801]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0
2026-11-15T05:56:54+01:00 demo-felhom felhom-os-apply[3119802]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
2026-11-15T05:57:00+01:00 demo-felhom sudo[3120431]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-guest-live-restore-on.json
2026-11-15T05:57:02+01:00 demo-felhom felhom-os-apply[3120495]: os-apply: LIVE-RESTORE already on
2026-11-15T05:57:02+01:00 demo-felhom sudo[3120496]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-docker-apply.json
2026-11-15T05:57:04+01:00 demo-felhom felhom-os-apply[3120626]: os-apply: START release=ring0-20261115T045634Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0
2026-11-15T05:57:08+01:00 demo-felhom felhom-os-apply[3120807]: os-apply: REPAIR configured=0 journal=0 fixed=0
2026-11-15T05:57:11+01:00 demo-felhom felhom-os-apply[3120839]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0
2026-11-15T05:57:11+01:00 demo-felhom felhom-os-apply[3120840]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
2026-11-15T05:57:20+01:00 demo-felhom kernel: dd invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
2026-11-15T05:57:25+01:00 demo-felhom felhom-os-apply[3121371]: os-apply: OOM-CHECK result=pass oom_killed=True oom_event=True exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0.352.1 — the engine reported the memory kill: OOMKilled=true and the oom event
2026-11-15T05:57:25+01:00 demo-felhom sudo[3121372]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-pve-apply.json
2026-11-15T05:57:26+01:00 demo-felhom felhom-os-apply[3121379]: os-apply: START release=ring0-20261115T045634Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0
2026-11-15T05:57:29+01:00 demo-felhom felhom-os-apply[3121477]: os-apply: REPAIR configured=0 journal=0 fixed=0
2026-11-15T05:57:31+01:00 demo-felhom felhom-os-apply[3121582]: os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0
2026-11-15T05:57:53+01:00 demo-felhom felhom-os-apply[3122820]: os-apply: DONE rc=0 seconds=20.1 upgraded=12 restart-needed=- reboot-needed=no
2026-11-15T05:58:00+01:00 demo-felhom sudo[3123080]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-kernel-kernel-cleanup.json
2026-11-15T05:58:00+01:00 demo-felhom felhom-os-apply[3123087]: os-apply: REPAIR configured=0 journal=0 fixed=0
2026-11-15T05:58:01+01:00 demo-felhom felhom-os-apply[3123143]: os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=8 keep=2 removal=6 not_run=[] refusals=[]
2026-11-15T05:58:35+01:00 demo-felhom felhom-os-apply[3132623]: os-apply: KERNEL CLEANUP DONE removed=6 (proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed) kept=7.0.14-26-pve, 7.0.14-28-pve gained=6810329088 B seconds=34.6 leftovers=0; default 7.0.14-28-pve (proved from grub.cfg)
2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=8 keep=2 removal=6 not_run=[] refusals=[]"
2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP DONE removed=6 (proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed) kept=7.0.14-26-pve, 7.0.14-28-pve gained=6810329088 B seconds=34.6 leftovers=0; default 7.0.14-28-pve
2026-11-15T06:06:42+01:00 demo-felhom sudo[3142189]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts050641-host-facts.json
2026-11-15T06:21:42+01:00 demo-felhom sudo[3159228]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts052142-host-facts.json
#### end
@@ -0,0 +1,31 @@
#### demo-felhom 2026-11-15T05:33:04Z local 2026-11-15 06:33:04 CET epoch 1794720784.224381391
agent: felhom-agent 0.174.0; uptime: up 1 week, 57 minutes; running kernel: 7.0.14-28-pve
## kernels installed (ii)
count: 0
/boot: 7.0.14-26-pve 7.0.14-28-pve
/lib/modules: 7.0.14-26-pve 7.0.14-28-pve
## default kernel and the pinned file
grub.cfg default: gnulinux-7.0.14-28-pve-advanced
513c1822cc58fc81 2026-11-08 05:36:39 /etc/default/grub.d/zz-felhom-kernel-default.cfg
4301585f3b987114 2026-11-08 05:36:42 /var/lib/felhom-kernel/state.json
f2f56d59c6356067 2026-10-07 17:57:10 /etc/grub.d/01_felhom_oneshot
c34907ce68599873 2026-10-07 17:57:10 /etc/grub.d/42_felhom_oneshot
pinned file content: # felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily GRUB_DEFAULT="gnulinux-advanced-1af1fcc6-639c-416b-a7e5-c4470d41a502>gnulinux-7.0.14-28-pve-a
kernel state: { "authority": "ring0", "default_before": "7.0.14-26-pve", "from": "7.0.14-26-pve", "health_before": { "guest": { "containers": { "cloudflared": { "health": "healthy", "id": "372fb8861b8465781dd368079d5e49753e4fe9435f062efa363bf5dd0f60e493", "state": "running" }, "felhom-controller": { "health": "healthy", "id": "13db198
## disk
/ free 65.10 GB, used 32%
## the night's lines: the agent's OS leg and the root wrapper (since yesterday 22:00 local)
## the wrapper's own journal lines about the clean-up
## health
failed units: 0
guest 9201: running; containers 9 running, 0 unhealthy, 0 exited
dpkg audit: []
## time service
chrony: active; Reference ID : C2255A28 (h90-40.pool194-37.dyn.tolna.net);System time : 0.000012041 seconds slow of NTP time;Leap status : Normal;
## the profile-refresh record's folder (a question of the night's review: can the agent read it?)
drwx------ 4 root root 4096 Nov 14 12:51 /var/lib/felhom-os-apply
-rw-r--r-- 1 root root 196 Nov 14 12:51 /var/lib/felhom-os-apply/profile-refresh.json
can the agent user read it: no
#### end demo-felhom epoch 1794720788.294832561
#### DooPlex: ssh rc=0, started epoch 1794720783.844141907, ended epoch 1794720788.299159339
#### clock: box minus DooPlex (middle of the round trip) = +0.064 s; round trip 0.139 s
@@ -0,0 +1,248 @@
System — Felhom Hub
Felhom Hub
Dashboard
Customers
Apps
Hosts
System
Offsite
Configuration
System — versions and OS updates
Needs attention
One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3).
[hover: worth a look] look
Tester4-46836f Tester4
[hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] kernel step: — · the host needs a restart since 2026-11-14 · [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] root files behind the vouched agent's · [hover: 1 minor release behind — sign an agent_update for this box] agent behind: 0.173.0 → 0.174.0 (since 2026-11-14) · [hover: the box could not read these values (agent older than v0.142.0, or the guest is down) — never a guess] could not read: default kernel
Waiting for you
Update sets that only you approve: the kernel, the Docker engine and the Proxmox packages. Guest and host Debian updates approve themselves after 24 h and one night on the [hover: Ring 0 = the demo boxes. They install every new fix first; the other boxes (ring 1) install only what is approved.] ring-0 boxes.
Docker engine 29.9.3-1
Ran on every ring-0 box — demo-felhom-8363b5: 2 healthy night(s), demo-hp-bb76ea: 2 healthy night(s). The memory-kill check passed.
first seen 2 days ago (2026-11-13 04:49 UTC). Approving installs nothing by itself: a ring-1 box takes it only through a signed operator job.
Approve Docker set
Proxmox packages 9.2.27 — still being tested: demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set
Boxes
Click a box to see every value it reports, and its ring and update switches. "unknown": the box could not read the value — never a guess.
Box Ring Health Controller · agent OS updates Kernel [hover: The newest OS update run] Last night
Tester4-46836f Tester4
[hover: Ring 1: a normal box — it installs only approved releases] 1
[hover: worth a look] look
0.352.1 [hover: 1 minor release behind — sign an agent_update for this box] agent 0.173.0 → 0.174.0 (since 2026-11-14)
[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind
7.0.2-6-pve
[hover: 19 h ago · applied · 93 s — last successful leg: 19 h ago] ok · 19 h ago
Ring and updates
Ring ring 1
→ demo
OS updates updates ON
switch off
Tunnel [hover: no cloudflared container in the guest (agent state: absent)] none set up
Controller 0.352.1
Agent [hover: 1 minor release behind — sign an agent_update for this box] 0.173.0 → 0.174.0 (since 2026-11-14)
Last OS leg [hover: last successful leg: 19 h ago] 19 h ago · applied · 93 s
Docker engine
Docker 29.9.2
containerd 2.4.2-3~debian.13~trixie
live-restore on
Docker release —
Host
Proxmox 9.2.2
Kernel (running) 7.0.2-6-pve
Kernel (next boot) 7.0.2-6-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] —
Debian 13.7
Felhom release os-host-20261111-034654
Pending 100
Not covered 0
Held none
Reboot needed since 2026-11-14
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0
Guest
Guest Debian 13.7
Felhom release os-guest-20261111-034654
Pending 6
Restart needed 7
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 19 h ago · 78.4 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] not yet
demo-felhom-8363b5 Demo Ügyfél
[hover: Ring 0: a demo box — it installs every new fix first] 0
[hover: nothing amber or red] fine
0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0
[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind
7.0.14-28-pve
[hover: 37 min ago · nothing · 23 s — last successful leg: 37 min ago] ok · 37 min ago
Ring and updates
Ring ring 0
→ normal
OS updates updates ON
switch off
Tunnel running
Controller 0.352.1
Agent [hover: current (vouched 0.174.0)] 0.174.0
Last OS leg [hover: last successful leg: 37 min ago] 37 min ago · nothing · 23 s
Docker engine
Docker 29.9.3
containerd 2.4.2-4~debian.13~trixie
live-restore on
Docker release ring0-20261115T045634Z
Host
Proxmox 9.2.27
Kernel (running) 7.0.14-28-pve
Kernel (next boot) 7.0.14-28-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: removed 6 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed; kept 7.0.14-26-pve, 7.0.14-28-pve; 6.3 GiB gained; 0 left-over path(s)] cleaned 36 min ago · due 7.0.14-29-pve — not told yet (mails 09–20 h) · phase good
Debian 13.7
Felhom release ring0-20261115T045634Z
Pending 29
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0
Guest
Guest Debian 13.7
Felhom release ring0-20261115T045634Z
Pending 0
Restart needed 1
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 272.7 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 92.7 Mbit/s · 4 days ago
demo-hp-bb76ea Demo HP
[hover: Ring 0: a demo box — it installs every new fix first] 0
[hover: nothing amber or red] fine
0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0
[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind
7.0.14-28-pve
[hover: 1 h ago · nothing · 30 s — last successful leg: 1 h ago] ok · 1 h ago
Ring and updates
Ring ring 0
→ normal
OS updates updates ON
switch off
Tunnel running
Controller 0.352.1
Agent [hover: current (vouched 0.174.0)] 0.174.0
Last OS leg [hover: last successful leg: 1 h ago] 1 h ago · nothing · 30 s
Docker engine
Docker 29.9.3
containerd 2.4.2-4~debian.13~trixie
live-restore on
Docker release ring0-20261115T040248Z
Host
Proxmox 9.2.27
Kernel (running) 7.0.14-28-pve
Kernel (next boot) 7.0.14-28-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 2m39s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 5 days ago · due 7.0.14-29-pve — not told yet (mails 09–20 h) · phase good
Debian 13.7
Felhom release ring0-20261115T040248Z
Pending 30
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0
Guest
Guest Debian 13.7
Felhom release ring0-20261115T040248Z
Pending 0
Restart needed 1
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 70.5 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 94.3 Mbit/s · 4 days ago
tester-1-d70be4 Tester 1
[hover: Ring 1: a normal box — it installs only approved releases] 1
[hover: nothing amber or red] fine
0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0
[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind
7.0.14-22-pve
[hover: 39 min ago · nothing · 16 s — last successful leg: 39 min ago] ok · 39 min ago
Ring and updates
Ring ring 1
→ demo
OS updates updates ON
switch off
Tunnel running
Controller 0.352.1
Agent [hover: current (vouched 0.174.0)] 0.174.0
Last OS leg [hover: last successful leg: 39 min ago] 39 min ago · nothing · 16 s
Docker engine
Docker 29.8.2
containerd 2.3.6-1~debian.13~trixie
live-restore on
Docker release —
Host
Proxmox 9.2.2
Kernel (running) 7.0.14-22-pve
Kernel (next boot) 7.0.14-22-pve
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-22-pve
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: ring 1: no kernel staged by a signed os_kernel_step healthy 1m17s after the agent started; the new kernel is the default] 7.0.14-22-pve applied 38 days ago · phase good
Debian 13.7
Felhom release os-host-20261111-034654
Pending 101
Not covered 0
Held none
Reboot needed no
kernel.panic 10 s
Oops no
Crash restarts 24 h 0
Crash guard armed
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0
Guest
Guest Debian 13.7
Felhom release os-guest-20261111-034654
Pending 6
Restart needed 0
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 90.4 GiB
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 93.0 Mbit/s · 4 days ago
Details
Release ids, cancelled approvals, what ring 0 runs, the version floors, and every box's crash guard and root files.
Approved releases
The newest approved set of each layer. A [hover: Approved while a TEST wait override was active. It is cancelled when the hub starts without the override (`11` §5.3.1).] TEST approval is amber.
guest : os-guest-20261111-034654
272 packages · 2026-11-11 03:46 UTC · by auto
host : os-host-20261115-045718
607 packages · 2026-11-15 04:57 UTC · by auto
docker : os-docker-20261026-190351
6 packages · 2026-10-26 19:03 UTC · by operator
pve : os-pve-20261114-075712
121 packages · 2026-11-14 07:57 UTC · by operator
kernel : os-kernel-20261114-075716
2 packages · 2026-11-14 07:57 UTC · by operator
Approve now (guest + host)
An urgent fix only — normally the hub approves after 24 h and one night.
What ring 0 runs now
guest :
272 packages, first seen 2026-11-10 01:49 UTC
approved as os-guest-20261111-034654
host :
607 packages, first seen 2026-11-14 04:52 UTC
approved as os-host-20261115-045718
docker :
6 packages, first seen 2026-11-13 04:49 UTC
ready — see "Waiting for you"
pve :
121 packages, first seen 2026-11-15 04:58 UTC
demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set
kernel :
2 packages, first seen 2026-11-10 01:57 UTC
approved as os-kernel-20261114-075716
Version floors
Global controller floor: 0.352.1 · vouched agent: 0.174.0
[hover: A floor is the lowest controller version a box must run; the hub moves a box below it up.] What is a floor? · [hover: The agent version the operator checked and signed off for the fleet. Agents update only by a per-box signed job.] What is a vouched agent?
No per-customer floors: every box follows the global floor.
Crash guard and root files
Box ¦ kernel.panic ¦ Oops ¦ Crash restarts 24 h ¦ Crash guard ¦ [hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files ¦
Tester4-46836f ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0 ¦
demo-felhom-8363b5 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦
demo-hp-bb76ea ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦
tester-1-d70be4 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦
Felhom Hub 0.181.0
@@ -0,0 +1,38 @@
# Part E — READS ONLY on a test box, the morning after the first night a box may remove kernels by itself.
# Nothing here writes: no systemctl action, no apt, no pct start/stop, no file made on the box.
set -u
echo "#### $(hostname) $(date -u +%FT%TZ) local $(date +%F\ %T\ %Z) epoch $(date +%s.%N)"
echo "agent: $(felhom-agent --version 2>&1 | head -1); uptime: $(uptime -p); running kernel: $(uname -r)"
echo "## kernels installed (ii)"
dpkg-query -W -f '${Package} ${db:Status-Abbrev}\n' 'proxmox-kernel-*' 2>/dev/null | awk '$2=="ii" && /pve(-signed)?$/ {print " " $1}'
echo "count: $(dpkg-query -W -f '${Package} ${db:Status-Abbrev}\n' 'proxmox-kernel-*' 2>/dev/null | awk '$2=="ii" && /pve(-signed)?$/' | wc -l)"
echo "/boot: $(ls /boot/vmlinuz-* 2>/dev/null | sed 's#/boot/vmlinuz-##' | tr '\n' ' ')"
echo "/lib/modules: $(ls /lib/modules | tr '\n' ' ')"
echo "## default kernel and the pinned file"
echo "grub.cfg default: $(grep -m1 'set default="gnu' /boot/grub/grub.cfg | grep -o 'gnulinux-[0-9][^"]*-pve-advanced' | head -1)"
for f in /etc/default/grub.d/zz-felhom-kernel-default.cfg /var/lib/felhom-kernel/state.json /etc/grub.d/01_felhom_oneshot /etc/grub.d/42_felhom_oneshot; do
[ -e "$f" ] && echo " $(sha256sum "$f" | cut -c1-16) $(stat -c '%y' "$f" | cut -c1-19) $f" || echo " (absent) $f"; done
echo "pinned file content: $(tr '\n' ' ' < /etc/default/grub.d/zz-felhom-kernel-default.cfg 2>/dev/null | cut -c1-200)"
echo "kernel state: $(tr -d '\n' < /var/lib/felhom-kernel/state.json 2>/dev/null | cut -c1-400)"
echo "## disk"
df -B1 --output=avail,pcent / | tail -1 | awk '{printf " / free %.2f GB, used %s\n", $1/1e9, $2}'
echo "## the night's lines: the agent's OS leg and the root wrapper (since yesterday 22:00 local)"
journalctl -u felhom-agent --since "yesterday 22:00" --no-pager -o short-iso 2>/dev/null | grep -E "kernel clean-up|osupdate: (pass|leg|night|kernel|host step|the night)|osupdate: wrapper.*(KERNEL CLEANUP|HOST|KERNEL)|whole-guest backup (done|finished|FAILED)|os leg" | cut -c1-420 | tail -40
echo "## the wrapper's own journal lines about the clean-up"
journalctl -t felhom-os-apply --since "yesterday 22:00" --no-pager -o short-iso 2>/dev/null | grep -E "KERNEL CLEANUP|KERNEL |PROFILE REFRESH|REFUSED|FAILED" | cut -c1-420 | tail -20
echo "## health"
echo "failed units: $(systemctl --failed --no-legend 2>/dev/null | wc -l) $(systemctl --failed --no-legend 2>/dev/null | awk '{print $1}' | tr '\n' ' ')"
for id in $(pct list 2>/dev/null | awk 'NR>1 {print $1}'); do
st=$(pct status $id 2>/dev/null | awk '{print $2}')
if [ "$st" = running ]; then
echo " guest $id: running; containers $(pct exec $id -- docker ps -q 2>/dev/null | wc -l) running, $(pct exec $id -- docker ps --filter health=unhealthy -q 2>/dev/null | wc -l) unhealthy, $(pct exec $id -- docker ps -a --filter status=exited -q 2>/dev/null | wc -l) exited"
else echo " guest $id: $st"; fi
done
echo "dpkg audit: [$(dpkg --audit 2>/dev/null | head -1)]"
echo "## time service"
echo "chrony: $(systemctl is-active chrony 2>&1); $(chronyc tracking 2>/dev/null | grep -E 'System time|Leap status|Reference ID' | tr '\n' ';' | tr -s ' ')"
echo "## the profile-refresh record's folder (a question of the night's review: can the agent read it?)"
ls -ld /var/lib/felhom-os-apply 2>&1 | cut -c1-120
ls -l /var/lib/felhom-os-apply/profile-refresh.json 2>&1 | cut -c1-140
echo "can the agent user read it: $(sudo -n -u felhom-agent test -r /var/lib/felhom-os-apply/profile-refresh.json 2>/dev/null && echo yes || echo no)"
echo "#### end $(hostname) epoch $(date +%s.%N)"
@@ -0,0 +1,23 @@
# Part E, second read — READS ONLY. The first script's kernel count and journal filters matched nothing
# (the count's pattern was wrong; the journal words were guessed). This one reads wider.
set -u
echo "#### $(hostname) $(date +%F\ %T\ %Z)"
echo "## kernel packages by dpkg state"
dpkg-query -W -f '${db:Status-Abbrev}|${Package}|${Version}\n' 'proxmox-kernel-*' 2>/dev/null | sed 's/^/ /'
echo "## dpkg log: kernel package removals and purges, last 3 days"
grep -hE " (remove|purge) proxmox-kernel| status (not-installed|config-files) proxmox-kernel" /var/log/dpkg.log 2>/dev/null | grep -E "^2026-11-1[345]" | cut -c1-160 | tail -40
echo "## apt history: last 3 entries that name a kernel"
grep -B3 -A3 -E "^(Remove|Purge):.*proxmox-kernel" /var/log/apt/history.log 2>/dev/null | cut -c1-600 | tail -30
echo "## agent journal since yesterday 20:00: every line with osupdate / kernel / clean"
journalctl -u felhom-agent --since "yesterday 20:00" --no-pager -o short-iso 2>/dev/null | grep -iE "osupdate|kernel|clean-?up|cleanup" | cut -c1-500 | tail -60
echo "## agent journal line count since yesterday 20:00 (a positive control that the journal is readable)"
journalctl -u felhom-agent --since "yesterday 20:00" --no-pager 2>/dev/null | wc -l
echo "## the whole journal since yesterday 20:00: lines with felhom-os-apply or KERNEL"
journalctl --since "yesterday 20:00" --no-pager -o short-iso 2>/dev/null | grep -E "felhom-os-apply|KERNEL" | cut -c1-500 | tail -40
echo "## wrapper state folder (names and times only)"
ls -la --time-style=long-iso /var/lib/felhom-os-apply/ 2>&1 | cut -c1-160
ls -la --time-style=long-iso /var/lib/felhom-kernel/ 2>&1 | cut -c1-160
echo "## disk"
df -B1 --output=avail,used,pcent / | tail -1
df -B1 --output=avail,used,pcent /boot 2>/dev/null | tail -1
echo "#### end"
@@ -0,0 +1,12 @@
# Part E, third read — READS ONLY. The journal filters of reads 1 and 2 gave 0 lines, positive control included:
# the --since form was wrong. This one uses a full date and shows journalctl's own error.
set -u
echo "#### $(hostname) $(date +%F\ %T\ %Z)"
echo "## control: agent journal lines since 2026-11-14 20:00"
journalctl -u felhom-agent --since "2026-11-14 20:00:00" --no-pager 2>&1 | wc -l
journalctl -u felhom-agent --since "2026-11-14 20:00:00" --no-pager 2>&1 | head -2 | cut -c1-200
echo "## agent journal: osupdate / kernel / clean-up lines"
journalctl -u felhom-agent --since "2026-11-14 20:00:00" --no-pager -o short-iso 2>&1 | grep -iE "osupdate|kernel|clean-?up|cleanup" | cut -c1-600 | tail -50
echo "## whole journal 05:30-06:30: wrapper and sudo lines"
journalctl --since "2026-11-15 05:30:00" --until "2026-11-15 06:30:00" --no-pager -o short-iso 2>&1 | grep -E "felhom-os-apply|KERNEL|apt-get" | cut -c1-500 | tail -30
echo "#### end"