night build 2026-11-14: the N100's first night kernel removal read (D82: 1 of 2), close-out — report, status, register
gates / gates (push) Successful in 2m8s
gates / gates (push) Successful in 2m8s
Reads only on the N100 and demo-hp; no box changed, nothing released. Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
1 parent
5da5b05192
commit
fffbb40ce0
17 files changed
+830
-4
No files matched your search
@@ -0,0 +1,56 @@
|
||||
# Part E — the morning read after the first night a box removed kernels by itself (2026-11-15, 06:33 CET)
|
||||
|
||||
Reads only. Nothing was written on a box: no service action, no package action, no guest started or stopped, no file
|
||||
made. The three read scripts are in this folder as they were run (`partE-read.sh`, `partE-read2.sh`,
|
||||
`partE-read3.sh`), each as `ssh <box> 'bash -s' < script`. Tester 1 was not read on the box: the hub's page did not
|
||||
show the line that would have asked for it.
|
||||
|
||||
## The N100 (`demo-felhom-8363b5`) — a real removal, healthy
|
||||
|
||||
| Fact | Before | After | Where |
|
||||
|---|---|---|---|
|
||||
| Kernel packages installed | 8 (the wrapper's own count at 05:58:01) | 2: 7.0.14-26 and 7.0.14-28 | `felhom-pve-3.txt`, `felhom-pve-2.txt` (dpkg state) |
|
||||
| Removed | — | 6: 7.0.2-6, 7.0.14-20, -22, -23, -24, -25; purged, 0 left-over paths | `felhom-pve-2.txt` (dpkg log, apt history: requested by the agent's user) |
|
||||
| Time | 05:58:01 | 05:58:35 (34.6 s) | the same |
|
||||
| Space gained | — | 6,810,329,088 bytes (6.81 GB, 6.3 GiB) by the wrapper's figure; 65.10 GB free after | `felhom-pve-3.txt`, `felhom-pve.txt` |
|
||||
| Running kernel | 7.0.14-28 | 7.0.14-28 (no reboot; up 1 week) | `felhom-pve.txt` |
|
||||
| Default kernel | 7.0.14-28 | 7.0.14-28, „proved from grub.cfg” by the wrapper and read again from `grub.cfg` | both |
|
||||
| The pinned file and the kernel state file | written 2026-11-08 05:36 | the same times | `felhom-pve.txt` |
|
||||
| Health | — | 0 failed units; guest 9201 running, 9 containers, 0 unhealthy, 0 exited; dpkg audit empty | `felhom-pve.txt` |
|
||||
| Time service | — | active, leap normal; the box is +0.064 s from DooPlex (round trip 0.139 s) | `felhom-pve.txt` |
|
||||
| The hub's „Kernel step” cell | — | „cleaned 36 min ago”, the hover text names the 6 packages and the 2 kept | `hub-system-page-0634.txt` line 97 |
|
||||
|
||||
The clean-up ran at the end of the night's update run, after the Proxmox step applied 12 packages. One line in the
|
||||
journal reads „dd invoked oom-killer” at 05:57:20: that is the update run's own memory-kill check (it reports
|
||||
`OOM-CHECK result=pass`), not a fault.
|
||||
|
||||
## demo-hp (`demo-hp-bb76ea`) — a quiet night
|
||||
|
||||
The night run at 05:02–05:04 found 2 kernels installed, 2 to keep, 0 to remove („nothing to remove”). The box's six
|
||||
old kernels were purged on 2026-11-10 at 13:51, by day (apt history). Health: 0 failed units; guests 9201 (26
|
||||
containers) and 9202 (6 containers) healthy; clock +0.071 s from DooPlex. The hub's cell still shows its newest kernel
|
||||
step („applied 5 days ago”), which is correct for a clean-up that removed nothing.
|
||||
|
||||
## Tester 1 (`tester-1-d70be4`) — the hub's line
|
||||
|
||||
„Crash guard: armed”; root files 0.174.0; kernel.panic 10 s; health „fine”. The line „crash guard not installed” is
|
||||
not there, so no read on the box and no row.
|
||||
|
||||
## The D82 count (R-1368)
|
||||
|
||||
Two real night removals on each ring-0 box. **The N100: 1 of 2. demo-hp: 0 of 2.** Both boxes now hold 2 kernels and
|
||||
keep 2, so the next real removal needs new kernels to arrive first (the hub shows 7.0.14-29 as due on both).
|
||||
|
||||
## Two instrument defects in the first read, said plainly
|
||||
|
||||
1. The first script's kernel count printed 0 on both boxes. Its pattern looked for the package name at the end of a
|
||||
line that ends with the state. The second read lists the dpkg states directly.
|
||||
2. The first and second scripts' journal filters printed nothing, and the positive control (a line count) read 0:
|
||||
`--since "yesterday 20:00"` gave no lines over this ssh form, with the error hidden. The third read uses a full
|
||||
date and shows 139,573 and 140,523 lines.
|
||||
|
||||
## One answer to a question of the night's review
|
||||
|
||||
The folder that holds the profile-refresh record is `drwx------ root`, so the agent's user cannot read the record
|
||||
file (read on both boxes). The night's build takes the result from the root step's answer and treats the file as a
|
||||
bonus, so this does not break it; the file read stays dead until the folder's mode changes.
|
||||
@@ -0,0 +1,37 @@
|
||||
#### demo-hp 2026-11-15 06:33:42 CET
|
||||
## kernel packages by dpkg state
|
||||
ii |proxmox-kernel-7.0|7.0.14-28
|
||||
un |proxmox-kernel-7.0.14-26-pve|
|
||||
ii |proxmox-kernel-7.0.14-26-pve-signed|7.0.14-26
|
||||
un |proxmox-kernel-7.0.14-28-pve|
|
||||
ii |proxmox-kernel-7.0.14-28-pve-signed|7.0.14-28
|
||||
ii |proxmox-kernel-helper|9.1.0+fde2
|
||||
## dpkg log: kernel package removals and purges, last 3 days
|
||||
## apt history: last 3 entries that name a kernel
|
||||
|
||||
Start-Date: 2026-11-10 13:51:11
|
||||
Commandline: apt-get -y -q -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef purge proxmox-kernel-7.0.2-6-pve-signed proxmox-kernel-7.0.14-20-pve-signed proxmox-kernel-7.0.14-22-pve-signed proxmox-kernel-7.0.14-23-pve-signed proxmox-kernel-7.0.14-24-pve-signed proxmox-kernel-7.0.14-25-pve-signed
|
||||
Purge: proxmox-kernel-7.0.14-20-pve-signed:amd64 (7.0.14-20), proxmox-kernel-7.0.14-22-pve-signed:amd64 (7.0.14-22), proxmox-kernel-7.0.14-23-pve-signed:amd64 (7.0.14-23), proxmox-kernel-7.0.14-24-pve-signed:amd64 (7.0.14-24), proxmox-kernel-7.0.2-6-pve-signed:amd64 (7.0.2-6), proxmox-kernel-7.0.14-25-pve-signed:amd64 (7.0.14-25)
|
||||
End-Date: 2026-11-10 13:52:00
|
||||
|
||||
Start-Date: 2026-11-14 05:00:45
|
||||
## agent journal since yesterday 20:00: every line with osupdate / kernel / clean
|
||||
## agent journal line count since yesterday 20:00 (a positive control that the journal is readable)
|
||||
0
|
||||
## the whole journal since yesterday 20:00: lines with felhom-os-apply or KERNEL
|
||||
## wrapper state folder (names and times only)
|
||||
total 24
|
||||
drwx------ 4 root root 4096 2026-11-14 12:33 .
|
||||
drwxr-xr-x 46 root root 4096 2026-10-10 04:42 ..
|
||||
drwxr-xr-x 2 root root 4096 2026-11-14 12:18 agent-update
|
||||
drwxr-xr-x 5 root root 4096 2026-11-14 12:33 bundle-prev
|
||||
-rw-r--r-- 1 root root 96 2026-11-14 12:33 nonces.json
|
||||
-rw-r--r-- 1 root root 137 2026-11-14 12:33 profile-refresh.json
|
||||
total 16
|
||||
drwxr-xr-x 2 root root 4096 2026-11-10 02:56 .
|
||||
drwxr-xr-x 46 root root 4096 2026-10-10 04:42 ..
|
||||
-rw-r--r-- 1 root root 5701 2026-11-10 02:56 state.json
|
||||
## disk
|
||||
21968302080 17318748160 45%
|
||||
21968302080 17318748160 45%
|
||||
#### end
|
||||
@@ -0,0 +1,50 @@
|
||||
#### demo-hp 2026-11-15 06:34:01 CET
|
||||
## control: agent journal lines since 2026-11-14 20:00
|
||||
140523
|
||||
Nov 14 20:00:01 demo-hp sudo[3471829]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||
Nov 14 20:00:01 demo-hp sudo[3471829]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=999)
|
||||
## agent journal: osupdate / kernel / clean-up lines
|
||||
2026-11-15T05:02:48+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:02:48.557+01:00 level=INFO msg="osupdate: START" run=20261115T040248Z layer=guest vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T040248Z
|
||||
2026-11-15T05:03:02+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:02.991+01:00 level=INFO msg="osupdate: a pass is running — its start sends any kept report"
|
||||
2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.208+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T040248Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
|
||||
2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.208+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
|
||||
2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.208+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
|
||||
2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.208+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
|
||||
2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.209+01:00 level=INFO msg="osupdate: DONE" run=20261115T040248Z layer=guest vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=19.5
|
||||
2026-11-15T05:03:08+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:08.303+01:00 level=INFO msg="osupdate: START" run=20261115T040248Z layer=host vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T040248Z
|
||||
2026-11-15T05:03:25+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:25.491+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T040248Z layer=host lane=fast mode=apply select=pending-fast packages=0"
|
||||
2026-11-15T05:03:25+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:25.491+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
|
||||
2026-11-15T05:03:25+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:25.491+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
|
||||
2026-11-15T05:03:25+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:25.491+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
|
||||
2026-11-15T05:03:26+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:26.644+01:00 level=INFO msg="osupdate: DONE" run=20261115T040248Z layer=host vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=30 not_covered=30 restart_needed=1 reboot_needed=false wrapper_seconds=17.1
|
||||
2026-11-15T05:03:29+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:29.145+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on"
|
||||
2026-11-15T05:03:29+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:29.145+01:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"result\": \"already on\"}"
|
||||
2026-11-15T05:03:29+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:29.145+01:00 level=INFO msg="osupdate: START" run=20261115T040248Z layer=docker vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T040248Z
|
||||
2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.415+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T040248Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0"
|
||||
2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.415+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
|
||||
2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.415+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
|
||||
2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.415+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
|
||||
2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.415+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: OOM-CHECK result=pass oom_killed=True oom_event=True exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0.352.1 — the engine reported the memory kill: OOMKilled=true and the oom event"
|
||||
2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.416+01:00 level=INFO msg="osupdate: DONE" run=20261115T040248Z layer=docker vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=30.2
|
||||
2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.423+01:00 level=INFO msg="osupdate: pve step holds the /etc/pve write gate — the agent's own writes wait until it ends" run=20261115T040248Z layer=pve vmid=9201 trigger=night
|
||||
2026-11-15T05:03:59+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:03:59.423+01:00 level=INFO msg="osupdate: START" run=20261115T040248Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T040248Z
|
||||
2026-11-15T05:04:46+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:46.366+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T040248Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0"
|
||||
2026-11-15T05:04:46+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:46.366+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
|
||||
2026-11-15T05:04:46+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:46.366+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0"
|
||||
2026-11-15T05:04:46+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:46.366+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=25.4 upgraded=12 restart-needed=kvm reboot-needed=no"
|
||||
2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.480+01:00 level=INFO msg="osupdate: DONE" run=20261115T040248Z layer=pve vmid=9201 ring=0 trigger=night outcome=applied healthy=true reason="" upgraded=12 pending=2 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=46.8
|
||||
2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.509+01:00 level=INFO msg="osupdate: pve step released the /etc/pve write gate" run=20261115T040248Z layer=pve vmid=9201 trigger=night
|
||||
2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.509+01:00 level=INFO msg="osupdate: kernel step skipped — the hub names no kernel step for this box" run=20261115T040248Z layer=kernel vmid=9201 trigger=night ring=0
|
||||
2026-11-15T05:04:47+01:00 demo-hp sudo[1134694]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T040248Z-kernel-kernel-cleanup.json
|
||||
2026-11-15T05:04:47+01:00 demo-hp felhom-os-apply[1134710]: os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=2 keep=2 removal=0 not_run=[] refusals=[]
|
||||
2026-11-15T05:04:47+01:00 demo-hp felhom-os-apply[1134711]: os-apply: KERNEL CLEANUP nothing to remove
|
||||
2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.855+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
|
||||
2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.855+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=2 keep=2 removal=0 not_run=[] refusals=[]"
|
||||
2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.855+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP nothing to remove"
|
||||
2026-11-15T05:04:47+01:00 demo-hp felhom-agent[1929140]: time=2026-11-15T05:04:47.855+01:00 level=INFO msg="osupdate: kernel clean-up — nothing to remove" run=20261115T040248Z layer=kernel mode=kernel-cleanup vmid=9201 trigger=night kept=7.0.14-26-pve,7.0.14-28-pve took=0s
|
||||
## whole journal 05:30-06:30: wrapper and sudo lines
|
||||
2026-11-15T05:33:13+01:00 demo-hp sudo[1229107]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts043312-host-facts.json
|
||||
2026-11-15T05:48:13+01:00 demo-hp sudo[1279117]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts044813-host-facts.json
|
||||
2026-11-15T06:03:13+01:00 demo-hp sudo[1329378]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts050313-host-facts.json
|
||||
2026-11-15T06:18:13+01:00 demo-hp sudo[1379432]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts051813-host-facts.json
|
||||
#### end
|
||||
@@ -0,0 +1,33 @@
|
||||
#### demo-hp 2026-11-15T05:33:09Z local 2026-11-15 06:33:09 CET epoch 1794720789.075455874
|
||||
agent: felhom-agent 0.174.0; uptime: up 4 days, 14 hours, 50 minutes; running kernel: 7.0.14-28-pve
|
||||
## kernels installed (ii)
|
||||
count: 0
|
||||
/boot: 7.0.14-26-pve 7.0.14-28-pve
|
||||
/lib/modules: 7.0.14-26-pve 7.0.14-28-pve
|
||||
## default kernel and the pinned file
|
||||
grub.cfg default: gnulinux-7.0.14-28-pve-advanced
|
||||
6b35d9aeddc1ddd0 2026-11-10 02:56:43 /etc/default/grub.d/zz-felhom-kernel-default.cfg
|
||||
607a2ce7ce3b6ca1 2026-11-10 02:56:47 /var/lib/felhom-kernel/state.json
|
||||
f2f56d59c6356067 2026-10-07 18:01:30 /etc/grub.d/01_felhom_oneshot
|
||||
c34907ce68599873 2026-10-07 18:01:30 /etc/grub.d/42_felhom_oneshot
|
||||
pinned file content: # felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-28-pve-a
|
||||
kernel state: { "authority": "ring0", "default_before": "7.0.14-26-pve", "from": "7.0.14-26-pve", "health_before": { "guest": { "containers": { "adventurelog": { "health": "healthy", "id": "50af393c36a37d01a10847456d391d70572c121045e28626f220f83cec1cefcc", "state": "running" }, "adventurelog-frontend": { "health": "healthy", "id": "ab
|
||||
## disk
|
||||
/ free 21.97 GB, used 45%
|
||||
## the night's lines: the agent's OS leg and the root wrapper (since yesterday 22:00 local)
|
||||
## the wrapper's own journal lines about the clean-up
|
||||
## health
|
||||
failed units: 0
|
||||
guest 9201: running; containers 26 running, 0 unhealthy, 0 exited
|
||||
guest 9202: running; containers 6 running, 0 unhealthy, 0 exited
|
||||
guest 9401: stopped
|
||||
dpkg audit: []
|
||||
## time service
|
||||
chrony: active; Reference ID : C1292F54 (ntp1.iszt.hu);System time : 0.000095636 seconds slow of NTP time;Leap status : Normal;
|
||||
## the profile-refresh record's folder (a question of the night's review: can the agent read it?)
|
||||
drwx------ 4 root root 4096 Nov 14 12:33 /var/lib/felhom-os-apply
|
||||
-rw-r--r-- 1 root root 137 Nov 14 12:33 /var/lib/felhom-os-apply/profile-refresh.json
|
||||
can the agent user read it: no
|
||||
#### end demo-hp epoch 1794720799.360448067
|
||||
#### DooPlex: ssh rc=0, started epoch 1794720788.467514838, ended epoch 1794720799.365384080
|
||||
#### clock: box minus DooPlex (middle of the round trip) = +0.071 s; round trip 0.151 s
|
||||
@@ -0,0 +1,65 @@
|
||||
#### demo-felhom 2026-11-15 06:33:41 CET
|
||||
## kernel packages by dpkg state
|
||||
ii |proxmox-kernel-7.0|7.0.14-28
|
||||
un |proxmox-kernel-7.0.14-26-pve|
|
||||
ii |proxmox-kernel-7.0.14-26-pve-signed|7.0.14-26
|
||||
un |proxmox-kernel-7.0.14-28-pve|
|
||||
ii |proxmox-kernel-7.0.14-28-pve-signed|7.0.14-28
|
||||
ii |proxmox-kernel-helper|9.1.0+fde2
|
||||
## dpkg log: kernel package removals and purges, last 3 days
|
||||
2026-11-15 05:58:01 remove proxmox-kernel-7.0.14-20-pve-signed:amd64 7.0.14-20 <none>
|
||||
2026-11-15 05:58:04 status config-files proxmox-kernel-7.0.14-20-pve-signed:amd64 7.0.14-20
|
||||
2026-11-15 05:58:04 remove proxmox-kernel-7.0.14-22-pve-signed:amd64 7.0.14-22 <none>
|
||||
2026-11-15 05:58:07 status config-files proxmox-kernel-7.0.14-22-pve-signed:amd64 7.0.14-22
|
||||
2026-11-15 05:58:07 remove proxmox-kernel-7.0.14-23-pve-signed:amd64 7.0.14-23 <none>
|
||||
2026-11-15 05:58:10 status config-files proxmox-kernel-7.0.14-23-pve-signed:amd64 7.0.14-23
|
||||
2026-11-15 05:58:10 remove proxmox-kernel-7.0.14-24-pve-signed:amd64 7.0.14-24 <none>
|
||||
2026-11-15 05:58:13 status config-files proxmox-kernel-7.0.14-24-pve-signed:amd64 7.0.14-24
|
||||
2026-11-15 05:58:13 remove proxmox-kernel-7.0.14-25-pve-signed:amd64 7.0.14-25 <none>
|
||||
2026-11-15 05:58:16 status config-files proxmox-kernel-7.0.14-25-pve-signed:amd64 7.0.14-25
|
||||
2026-11-15 05:58:16 remove proxmox-kernel-7.0.2-6-pve-signed:amd64 7.0.2-6 <none>
|
||||
2026-11-15 05:58:19 status config-files proxmox-kernel-7.0.2-6-pve-signed:amd64 7.0.2-6
|
||||
2026-11-15 05:58:19 purge proxmox-kernel-7.0.14-22-pve-signed:amd64 7.0.14-22 <none>
|
||||
2026-11-15 05:58:19 status config-files proxmox-kernel-7.0.14-22-pve-signed:amd64 7.0.14-22
|
||||
2026-11-15 05:58:22 status not-installed proxmox-kernel-7.0.14-22-pve-signed:amd64 <none>
|
||||
2026-11-15 05:58:22 purge proxmox-kernel-7.0.14-20-pve-signed:amd64 7.0.14-20 <none>
|
||||
2026-11-15 05:58:22 status config-files proxmox-kernel-7.0.14-20-pve-signed:amd64 7.0.14-20
|
||||
2026-11-15 05:58:24 status not-installed proxmox-kernel-7.0.14-20-pve-signed:amd64 <none>
|
||||
2026-11-15 05:58:24 purge proxmox-kernel-7.0.2-6-pve-signed:amd64 7.0.2-6 <none>
|
||||
2026-11-15 05:58:24 status config-files proxmox-kernel-7.0.2-6-pve-signed:amd64 7.0.2-6
|
||||
2026-11-15 05:58:27 status not-installed proxmox-kernel-7.0.2-6-pve-signed:amd64 <none>
|
||||
2026-11-15 05:58:27 purge proxmox-kernel-7.0.14-25-pve-signed:amd64 7.0.14-25 <none>
|
||||
2026-11-15 05:58:27 status config-files proxmox-kernel-7.0.14-25-pve-signed:amd64 7.0.14-25
|
||||
2026-11-15 05:58:30 status not-installed proxmox-kernel-7.0.14-25-pve-signed:amd64 <none>
|
||||
2026-11-15 05:58:30 purge proxmox-kernel-7.0.14-24-pve-signed:amd64 7.0.14-24 <none>
|
||||
2026-11-15 05:58:30 status config-files proxmox-kernel-7.0.14-24-pve-signed:amd64 7.0.14-24
|
||||
2026-11-15 05:58:32 status not-installed proxmox-kernel-7.0.14-24-pve-signed:amd64 <none>
|
||||
2026-11-15 05:58:32 purge proxmox-kernel-7.0.14-23-pve-signed:amd64 7.0.14-23 <none>
|
||||
2026-11-15 05:58:32 status config-files proxmox-kernel-7.0.14-23-pve-signed:amd64 7.0.14-23
|
||||
2026-11-15 05:58:35 status not-installed proxmox-kernel-7.0.14-23-pve-signed:amd64 <none>
|
||||
## apt history: last 3 entries that name a kernel
|
||||
Start-Date: 2026-11-15 05:58:01
|
||||
Commandline: apt-get -y -q purge proxmox-kernel-7.0.14-20-pve-signed proxmox-kernel-7.0.14-22-pve-signed proxmox-kernel-7.0.14-23-pve-signed proxmox-kernel-7.0.14-24-pve-signed proxmox-kernel-7.0.14-25-pve-signed proxmox-kernel-7.0.2-6-pve-signed
|
||||
Requested-By: felhom-agent (999)
|
||||
Purge: proxmox-kernel-7.0.14-20-pve-signed:amd64 (7.0.14-20), proxmox-kernel-7.0.14-22-pve-signed:amd64 (7.0.14-22), proxmox-kernel-7.0.14-23-pve-signed:amd64 (7.0.14-23), proxmox-kernel-7.0.14-24-pve-signed:amd64 (7.0.14-24), proxmox-kernel-7.0.2-6-pve-signed:amd64 (7.0.2-6), proxmox-kernel-7.0.14-25-pve-signed:amd64 (7.0.14-25)
|
||||
End-Date: 2026-11-15 05:58:35
|
||||
## agent journal since yesterday 20:00: every line with osupdate / kernel / clean
|
||||
## agent journal line count since yesterday 20:00 (a positive control that the journal is readable)
|
||||
0
|
||||
## the whole journal since yesterday 20:00: lines with felhom-os-apply or KERNEL
|
||||
## wrapper state folder (names and times only)
|
||||
total 24
|
||||
drwx------ 4 root root 4096 2026-11-14 12:51 .
|
||||
drwxr-xr-x 48 root root 4096 2026-10-08 04:39 ..
|
||||
drwxr-xr-x 2 root root 4096 2026-11-14 12:35 agent-update
|
||||
drwxr-xr-x 5 root root 4096 2026-11-14 12:51 bundle-prev
|
||||
-rw-r--r-- 1 root root 96 2026-11-14 12:51 nonces.json
|
||||
-rw-r--r-- 1 root root 196 2026-11-14 12:51 profile-refresh.json
|
||||
total 12
|
||||
drwxr-xr-x 2 root root 4096 2026-11-08 05:36 .
|
||||
drwxr-xr-x 48 root root 4096 2026-10-08 04:39 ..
|
||||
-rw-r--r-- 1 root root 2615 2026-11-08 05:36 state.json
|
||||
## disk
|
||||
65095643136 30595346432 32%
|
||||
65095643136 30595346432 32%
|
||||
#### end
|
||||
@@ -0,0 +1,76 @@
|
||||
#### demo-felhom 2026-11-15 06:33:58 CET
|
||||
## control: agent journal lines since 2026-11-14 20:00
|
||||
139573
|
||||
Nov 14 20:00:00 demo-felhom sudo[2436686]: pam_unix(sudo:session): session closed for user root
|
||||
Nov 14 20:00:00 demo-felhom sudo[2436691]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
|
||||
## agent journal: osupdate / kernel / clean-up lines
|
||||
2026-11-15T05:56:34+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:34.495+01:00 level=INFO msg="osupdate: START" run=20261115T045634Z layer=guest vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T045634Z
|
||||
2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.839+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T045634Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
|
||||
2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.839+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
|
||||
2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.840+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
|
||||
2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.840+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
|
||||
2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.840+01:00 level=INFO msg="osupdate: DONE" run=20261115T045634Z layer=guest vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=14.3
|
||||
2026-11-15T05:56:48+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:56:48.901+01:00 level=INFO msg="osupdate: START" run=20261115T045634Z layer=host vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T045634Z
|
||||
2026-11-15T05:57:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:00.040+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T045634Z layer=host lane=fast mode=apply select=pending-fast packages=0"
|
||||
2026-11-15T05:57:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:00.040+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
|
||||
2026-11-15T05:57:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:00.040+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
|
||||
2026-11-15T05:57:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:00.040+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
|
||||
2026-11-15T05:57:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:00.867+01:00 level=INFO msg="osupdate: DONE" run=20261115T045634Z layer=host vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=29 not_covered=29 restart_needed=0 reboot_needed=false wrapper_seconds=11.1
|
||||
2026-11-15T05:57:02+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:02.546+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on"
|
||||
2026-11-15T05:57:02+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:02.546+01:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"result\": \"already on\"}"
|
||||
2026-11-15T05:57:02+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:02.546+01:00 level=INFO msg="osupdate: START" run=20261115T045634Z layer=docker vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T045634Z
|
||||
2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.217+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T045634Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0"
|
||||
2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.217+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
|
||||
2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.217+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
|
||||
2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.217+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
|
||||
2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.217+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: OOM-CHECK result=pass oom_killed=True oom_event=True exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0.352.1 — the engine reported the memory kill: OOMKilled=true and the oom event"
|
||||
2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.218+01:00 level=INFO msg="osupdate: DONE" run=20261115T045634Z layer=docker vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=22.6
|
||||
2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.225+01:00 level=INFO msg="osupdate: pve step holds the /etc/pve write gate — the agent's own writes wait until it ends" run=20261115T045634Z layer=pve vmid=9201 trigger=night
|
||||
2026-11-15T05:57:25+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:25.225+01:00 level=INFO msg="osupdate: START" run=20261115T045634Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261115T045634Z
|
||||
2026-11-15T05:57:59+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:59.155+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261115T045634Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0"
|
||||
2026-11-15T05:57:59+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:59.155+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
|
||||
2026-11-15T05:57:59+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:59.155+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0"
|
||||
2026-11-15T05:57:59+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:59.155+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=20.1 upgraded=12 restart-needed=- reboot-needed=no"
|
||||
2026-11-15T05:57:59+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:57:59.998+01:00 level=INFO msg="osupdate: DONE" run=20261115T045634Z layer=pve vmid=9201 ring=0 trigger=night outcome=applied healthy=true reason="" upgraded=12 pending=1 not_covered=0 restart_needed=0 reboot_needed=false wrapper_seconds=33.9
|
||||
2026-11-15T05:58:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:00.007+01:00 level=INFO msg="osupdate: pve step released the /etc/pve write gate" run=20261115T045634Z layer=pve vmid=9201 trigger=night
|
||||
2026-11-15T05:58:00+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:00.007+01:00 level=INFO msg="osupdate: kernel step skipped — the hub names no kernel step for this box" run=20261115T045634Z layer=kernel vmid=9201 trigger=night ring=0
|
||||
2026-11-15T05:58:00+01:00 demo-felhom sudo[3123080]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-kernel-kernel-cleanup.json
|
||||
2026-11-15T05:58:01+01:00 demo-felhom felhom-os-apply[3123143]: os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=8 keep=2 removal=6 not_run=[] refusals=[]
|
||||
2026-11-15T05:58:35+01:00 demo-felhom felhom-os-apply[3132623]: os-apply: KERNEL CLEANUP DONE removed=6 (proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed) kept=7.0.14-26-pve, 7.0.14-28-pve gained=6810329088 B seconds=34.6 leftovers=0; default 7.0.14-28-pve (proved from grub.cfg)
|
||||
2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 journal=0 fixed=0"
|
||||
2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=8 keep=2 removal=6 not_run=[] refusals=[]"
|
||||
2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP DONE removed=6 (proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed) kept=7.0.14-26-pve, 7.0.14-28-pve gained=6810329088 B seconds=34.6 leftovers=0; default 7.0.14-28-pve (proved from grub.cfg)"
|
||||
2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: kernel clean-up — old kernels removed" run=20261115T045634Z layer=kernel mode=kernel-cleanup vmid=9201 trigger=night removed=6 names=proxmox-kernel-7.0.14-20-pve-signed,proxmox-kernel-7.0.14-22-pve-signed,proxmox-kernel-7.0.14-23-pve-signed,proxmox-kernel-7.0.14-24-pve-signed,proxmox-kernel-7.0.14-25-pve-signed,proxmox-kernel-7.0.2-6-pve-signed kept=7.0.14-26-pve,7.0.14-28-pve bytes_gained=0x118111de4540 leftovers=0 took=36s
|
||||
2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: DONE" run=20261115T045634Z layer=kernel mode=kernel-cleanup vmid=9201 trigger=night outcome=cleaned healthy=true reason="removed 6 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed; kept 7.0.14-26-pve, 7.0.14-28-pve; 6.3 GiB gained; 0 left-over path(s)" upgraded=0 pending=0 not_covered
|
||||
## whole journal 05:30-06:30: wrapper and sudo lines
|
||||
2026-11-15T05:56:39+01:00 demo-felhom felhom-os-apply[3119297]: os-apply: REPAIR configured=0 journal=0 fixed=0
|
||||
2026-11-15T05:56:43+01:00 demo-felhom felhom-os-apply[3119391]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0
|
||||
2026-11-15T05:56:43+01:00 demo-felhom felhom-os-apply[3119392]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
|
||||
2026-11-15T05:56:48+01:00 demo-felhom sudo[3119650]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-host-apply.json
|
||||
2026-11-15T05:56:49+01:00 demo-felhom felhom-os-apply[3119656]: os-apply: START release=ring0-20261115T045634Z layer=host lane=fast mode=apply select=pending-fast packages=0
|
||||
2026-11-15T05:56:52+01:00 demo-felhom felhom-os-apply[3119745]: os-apply: REPAIR configured=0 journal=0 fixed=0
|
||||
2026-11-15T05:56:54+01:00 demo-felhom felhom-os-apply[3119801]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0
|
||||
2026-11-15T05:56:54+01:00 demo-felhom felhom-os-apply[3119802]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
|
||||
2026-11-15T05:57:00+01:00 demo-felhom sudo[3120431]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-guest-live-restore-on.json
|
||||
2026-11-15T05:57:02+01:00 demo-felhom felhom-os-apply[3120495]: os-apply: LIVE-RESTORE already on
|
||||
2026-11-15T05:57:02+01:00 demo-felhom sudo[3120496]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-docker-apply.json
|
||||
2026-11-15T05:57:04+01:00 demo-felhom felhom-os-apply[3120626]: os-apply: START release=ring0-20261115T045634Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0
|
||||
2026-11-15T05:57:08+01:00 demo-felhom felhom-os-apply[3120807]: os-apply: REPAIR configured=0 journal=0 fixed=0
|
||||
2026-11-15T05:57:11+01:00 demo-felhom felhom-os-apply[3120839]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0
|
||||
2026-11-15T05:57:11+01:00 demo-felhom felhom-os-apply[3120840]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
|
||||
2026-11-15T05:57:20+01:00 demo-felhom kernel: dd invoked oom-killer: gfp_mask=0xcc0(GFP_KERNEL), order=0, oom_score_adj=0
|
||||
2026-11-15T05:57:25+01:00 demo-felhom felhom-os-apply[3121371]: os-apply: OOM-CHECK result=pass oom_killed=True oom_event=True exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0.352.1 — the engine reported the memory kill: OOMKilled=true and the oom event
|
||||
2026-11-15T05:57:25+01:00 demo-felhom sudo[3121372]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-pve-apply.json
|
||||
2026-11-15T05:57:26+01:00 demo-felhom felhom-os-apply[3121379]: os-apply: START release=ring0-20261115T045634Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0
|
||||
2026-11-15T05:57:29+01:00 demo-felhom felhom-os-apply[3121477]: os-apply: REPAIR configured=0 journal=0 fixed=0
|
||||
2026-11-15T05:57:31+01:00 demo-felhom felhom-os-apply[3121582]: os-apply: PLAN upgrade=12 already=0 not-installed=0 from-snapshot=0
|
||||
2026-11-15T05:57:53+01:00 demo-felhom felhom-os-apply[3122820]: os-apply: DONE rc=0 seconds=20.1 upgraded=12 restart-needed=- reboot-needed=no
|
||||
2026-11-15T05:58:00+01:00 demo-felhom sudo[3123080]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261115T045634Z-kernel-kernel-cleanup.json
|
||||
2026-11-15T05:58:00+01:00 demo-felhom felhom-os-apply[3123087]: os-apply: REPAIR configured=0 journal=0 fixed=0
|
||||
2026-11-15T05:58:01+01:00 demo-felhom felhom-os-apply[3123143]: os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=8 keep=2 removal=6 not_run=[] refusals=[]
|
||||
2026-11-15T05:58:35+01:00 demo-felhom felhom-os-apply[3132623]: os-apply: KERNEL CLEANUP DONE removed=6 (proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed) kept=7.0.14-26-pve, 7.0.14-28-pve gained=6810329088 B seconds=34.6 leftovers=0; default 7.0.14-28-pve (proved from grub.cfg)
|
||||
2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP running=7.0.14-28-pve newest=7.0.14-28-pve phase=good installed=8 keep=2 removal=6 not_run=[] refusals=[]"
|
||||
2026-11-15T05:58:35+01:00 demo-felhom felhom-agent[1931271]: time=2026-11-15T05:58:35.724+01:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL CLEANUP DONE removed=6 (proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed) kept=7.0.14-26-pve, 7.0.14-28-pve gained=6810329088 B seconds=34.6 leftovers=0; default 7.0.14-28-pve
|
||||
2026-11-15T06:06:42+01:00 demo-felhom sudo[3142189]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts050641-host-facts.json
|
||||
2026-11-15T06:21:42+01:00 demo-felhom sudo[3159228]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-facts052142-host-facts.json
|
||||
#### end
|
||||
@@ -0,0 +1,31 @@
|
||||
#### demo-felhom 2026-11-15T05:33:04Z local 2026-11-15 06:33:04 CET epoch 1794720784.224381391
|
||||
agent: felhom-agent 0.174.0; uptime: up 1 week, 57 minutes; running kernel: 7.0.14-28-pve
|
||||
## kernels installed (ii)
|
||||
count: 0
|
||||
/boot: 7.0.14-26-pve 7.0.14-28-pve
|
||||
/lib/modules: 7.0.14-26-pve 7.0.14-28-pve
|
||||
## default kernel and the pinned file
|
||||
grub.cfg default: gnulinux-7.0.14-28-pve-advanced
|
||||
513c1822cc58fc81 2026-11-08 05:36:39 /etc/default/grub.d/zz-felhom-kernel-default.cfg
|
||||
4301585f3b987114 2026-11-08 05:36:42 /var/lib/felhom-kernel/state.json
|
||||
f2f56d59c6356067 2026-10-07 17:57:10 /etc/grub.d/01_felhom_oneshot
|
||||
c34907ce68599873 2026-10-07 17:57:10 /etc/grub.d/42_felhom_oneshot
|
||||
pinned file content: # felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily GRUB_DEFAULT="gnulinux-advanced-1af1fcc6-639c-416b-a7e5-c4470d41a502>gnulinux-7.0.14-28-pve-a
|
||||
kernel state: { "authority": "ring0", "default_before": "7.0.14-26-pve", "from": "7.0.14-26-pve", "health_before": { "guest": { "containers": { "cloudflared": { "health": "healthy", "id": "372fb8861b8465781dd368079d5e49753e4fe9435f062efa363bf5dd0f60e493", "state": "running" }, "felhom-controller": { "health": "healthy", "id": "13db198
|
||||
## disk
|
||||
/ free 65.10 GB, used 32%
|
||||
## the night's lines: the agent's OS leg and the root wrapper (since yesterday 22:00 local)
|
||||
## the wrapper's own journal lines about the clean-up
|
||||
## health
|
||||
failed units: 0
|
||||
guest 9201: running; containers 9 running, 0 unhealthy, 0 exited
|
||||
dpkg audit: []
|
||||
## time service
|
||||
chrony: active; Reference ID : C2255A28 (h90-40.pool194-37.dyn.tolna.net);System time : 0.000012041 seconds slow of NTP time;Leap status : Normal;
|
||||
## the profile-refresh record's folder (a question of the night's review: can the agent read it?)
|
||||
drwx------ 4 root root 4096 Nov 14 12:51 /var/lib/felhom-os-apply
|
||||
-rw-r--r-- 1 root root 196 Nov 14 12:51 /var/lib/felhom-os-apply/profile-refresh.json
|
||||
can the agent user read it: no
|
||||
#### end demo-felhom epoch 1794720788.294832561
|
||||
#### DooPlex: ssh rc=0, started epoch 1794720783.844141907, ended epoch 1794720788.299159339
|
||||
#### clock: box minus DooPlex (middle of the round trip) = +0.064 s; round trip 0.139 s
|
||||
@@ -0,0 +1,248 @@
|
||||
System — Felhom Hub
|
||||
Felhom Hub
|
||||
Dashboard
|
||||
Customers
|
||||
Apps
|
||||
Hosts
|
||||
System
|
||||
Offsite
|
||||
Configuration
|
||||
System — versions and OS updates
|
||||
Needs attention
|
||||
One line per box that is amber or red, and why. Amber: worth a look. Red: an operator alarm fires (`08` §6.3).
|
||||
[hover: worth a look] look
|
||||
Tester4-46836f Tester4
|
||||
[hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] kernel step: — · the host needs a restart since 2026-11-14 · [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] root files behind the vouched agent's · [hover: 1 minor release behind — sign an agent_update for this box] agent behind: 0.173.0 → 0.174.0 (since 2026-11-14) · [hover: the box could not read these values (agent older than v0.142.0, or the guest is down) — never a guess] could not read: default kernel
|
||||
Waiting for you
|
||||
Update sets that only you approve: the kernel, the Docker engine and the Proxmox packages. Guest and host Debian updates approve themselves after 24 h and one night on the [hover: Ring 0 = the demo boxes. They install every new fix first; the other boxes (ring 1) install only what is approved.] ring-0 boxes.
|
||||
Docker engine 29.9.3-1
|
||||
Ran on every ring-0 box — demo-felhom-8363b5: 2 healthy night(s), demo-hp-bb76ea: 2 healthy night(s). The memory-kill check passed.
|
||||
first seen 2 days ago (2026-11-13 04:49 UTC). Approving installs nothing by itself: a ring-1 box takes it only through a signed operator job.
|
||||
Approve Docker set
|
||||
Proxmox packages 9.2.27 — still being tested: demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set
|
||||
Boxes
|
||||
Click a box to see every value it reports, and its ring and update switches. "unknown": the box could not read the value — never a guess.
|
||||
Box Ring Health Controller · agent OS updates Kernel [hover: The newest OS update run] Last night
|
||||
Tester4-46836f Tester4
|
||||
[hover: Ring 1: a normal box — it installs only approved releases] 1
|
||||
[hover: worth a look] look
|
||||
0.352.1 [hover: 1 minor release behind — sign an agent_update for this box] agent 0.173.0 → 0.174.0 (since 2026-11-14)
|
||||
[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind
|
||||
7.0.2-6-pve
|
||||
[hover: 19 h ago · applied · 93 s — last successful leg: 19 h ago] ok · 19 h ago
|
||||
Ring and updates
|
||||
Ring ring 1
|
||||
→ demo
|
||||
OS updates updates ON
|
||||
switch off
|
||||
Tunnel [hover: no cloudflared container in the guest (agent state: absent)] none set up
|
||||
Controller 0.352.1
|
||||
Agent [hover: 1 minor release behind — sign an agent_update for this box] 0.173.0 → 0.174.0 (since 2026-11-14)
|
||||
Last OS leg [hover: last successful leg: 19 h ago] 19 h ago · applied · 93 s
|
||||
Docker engine
|
||||
Docker 29.9.2
|
||||
containerd 2.4.2-3~debian.13~trixie
|
||||
live-restore on
|
||||
Docker release —
|
||||
Host
|
||||
Proxmox 9.2.2
|
||||
Kernel (running) 7.0.2-6-pve
|
||||
Kernel (next boot) 7.0.2-6-pve
|
||||
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) [hover: the box could not read it (agent older than v0.142.0, or the guest is down)] unknown
|
||||
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: cannot do a one-shot boot: the box does not boot UEFI; /boot/efi is not a mounted vfat ESP (not mounted)] —
|
||||
Debian 13.7
|
||||
Felhom release os-host-20261111-034654
|
||||
Pending 100
|
||||
Not covered 0
|
||||
Held none
|
||||
Reboot needed since 2026-11-14
|
||||
kernel.panic 10 s
|
||||
Oops no
|
||||
Crash restarts 24 h 0
|
||||
Crash guard armed
|
||||
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0
|
||||
Guest
|
||||
Guest Debian 13.7
|
||||
Felhom release os-guest-20261111-034654
|
||||
Pending 6
|
||||
Restart needed 7
|
||||
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 19 h ago · 78.4 GiB
|
||||
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] not yet
|
||||
demo-felhom-8363b5 Demo Ügyfél
|
||||
[hover: Ring 0: a demo box — it installs every new fix first] 0
|
||||
[hover: nothing amber or red] fine
|
||||
0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0
|
||||
[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind
|
||||
7.0.14-28-pve
|
||||
[hover: 37 min ago · nothing · 23 s — last successful leg: 37 min ago] ok · 37 min ago
|
||||
Ring and updates
|
||||
Ring ring 0
|
||||
→ normal
|
||||
OS updates updates ON
|
||||
switch off
|
||||
Tunnel running
|
||||
Controller 0.352.1
|
||||
Agent [hover: current (vouched 0.174.0)] 0.174.0
|
||||
Last OS leg [hover: last successful leg: 37 min ago] 37 min ago · nothing · 23 s
|
||||
Docker engine
|
||||
Docker 29.9.3
|
||||
containerd 2.4.2-4~debian.13~trixie
|
||||
live-restore on
|
||||
Docker release ring0-20261115T045634Z
|
||||
Host
|
||||
Proxmox 9.2.27
|
||||
Kernel (running) 7.0.14-28-pve
|
||||
Kernel (next boot) 7.0.14-28-pve
|
||||
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve
|
||||
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: removed 6 old kernel package(s): proxmox-kernel-7.0.14-20-pve-signed, proxmox-kernel-7.0.14-22-pve-signed, proxmox-kernel-7.0.14-23-pve-signed, proxmox-kernel-7.0.14-24-pve-signed, proxmox-kernel-7.0.14-25-pve-signed, proxmox-kernel-7.0.2-6-pve-signed; kept 7.0.14-26-pve, 7.0.14-28-pve; 6.3 GiB gained; 0 left-over path(s)] cleaned 36 min ago · due 7.0.14-29-pve — not told yet (mails 09–20 h) · phase good
|
||||
Debian 13.7
|
||||
Felhom release ring0-20261115T045634Z
|
||||
Pending 29
|
||||
Not covered 0
|
||||
Held none
|
||||
Reboot needed no
|
||||
kernel.panic 10 s
|
||||
Oops no
|
||||
Crash restarts 24 h 0
|
||||
Crash guard armed
|
||||
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0
|
||||
Guest
|
||||
Guest Debian 13.7
|
||||
Felhom release ring0-20261115T045634Z
|
||||
Pending 0
|
||||
Restart needed 1
|
||||
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 272.7 GiB
|
||||
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 92.7 Mbit/s · 4 days ago
|
||||
demo-hp-bb76ea Demo HP
|
||||
[hover: Ring 0: a demo box — it installs every new fix first] 0
|
||||
[hover: nothing amber or red] fine
|
||||
0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0
|
||||
[hover: not on the newest approved release yet — a box takes it at its next night run] on · guest + host behind
|
||||
7.0.14-28-pve
|
||||
[hover: 1 h ago · nothing · 30 s — last successful leg: 1 h ago] ok · 1 h ago
|
||||
Ring and updates
|
||||
Ring ring 0
|
||||
→ normal
|
||||
OS updates updates ON
|
||||
switch off
|
||||
Tunnel running
|
||||
Controller 0.352.1
|
||||
Agent [hover: current (vouched 0.174.0)] 0.174.0
|
||||
Last OS leg [hover: last successful leg: 1 h ago] 1 h ago · nothing · 30 s
|
||||
Docker engine
|
||||
Docker 29.9.3
|
||||
containerd 2.4.2-4~debian.13~trixie
|
||||
live-restore on
|
||||
Docker release ring0-20261115T040248Z
|
||||
Host
|
||||
Proxmox 9.2.27
|
||||
Kernel (running) 7.0.14-28-pve
|
||||
Kernel (next boot) 7.0.14-28-pve
|
||||
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-28-pve
|
||||
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: healthy 2m39s after the agent started; the new kernel is the default] 7.0.14-28-pve applied 5 days ago · due 7.0.14-29-pve — not told yet (mails 09–20 h) · phase good
|
||||
Debian 13.7
|
||||
Felhom release ring0-20261115T040248Z
|
||||
Pending 30
|
||||
Not covered 0
|
||||
Held none
|
||||
Reboot needed no
|
||||
kernel.panic 10 s
|
||||
Oops no
|
||||
Crash restarts 24 h 0
|
||||
Crash guard armed
|
||||
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0
|
||||
Guest
|
||||
Guest Debian 13.7
|
||||
Felhom release ring0-20261115T040248Z
|
||||
Pending 0
|
||||
Restart needed 1
|
||||
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 70.5 GiB
|
||||
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 94.3 Mbit/s · 4 days ago
|
||||
tester-1-d70be4 Tester 1
|
||||
[hover: Ring 1: a normal box — it installs only approved releases] 1
|
||||
[hover: nothing amber or red] fine
|
||||
0.352.1 [hover: current (vouched 0.174.0)] agent 0.174.0
|
||||
[hover: not on the newest approved release yet — a box takes it at its next night run] on · host behind
|
||||
7.0.14-22-pve
|
||||
[hover: 39 min ago · nothing · 16 s — last successful leg: 39 min ago] ok · 39 min ago
|
||||
Ring and updates
|
||||
Ring ring 1
|
||||
→ demo
|
||||
OS updates updates ON
|
||||
switch off
|
||||
Tunnel running
|
||||
Controller 0.352.1
|
||||
Agent [hover: current (vouched 0.174.0)] 0.174.0
|
||||
Last OS leg [hover: last successful leg: 39 min ago] 39 min ago · nothing · 16 s
|
||||
Docker engine
|
||||
Docker 29.8.2
|
||||
containerd 2.3.6-1~debian.13~trixie
|
||||
live-restore on
|
||||
Docker release —
|
||||
Host
|
||||
Proxmox 9.2.2
|
||||
Kernel (running) 7.0.14-22-pve
|
||||
Kernel (next boot) 7.0.14-22-pve
|
||||
[hover: The kernel GRUB boots normally (R-836). Amber: a one-shot flag names another kernel for the next boot only.] Kernel (default) 7.0.14-22-pve
|
||||
[hover: The kernel lane (R-836): the newest step, the kernel the box is due, and whether its household was told for tonight (no mail, no step).] Kernel step [hover: ring 1: no kernel staged by a signed os_kernel_step healthy 1m17s after the agent started; the new kernel is the default] 7.0.14-22-pve applied 38 days ago · phase good
|
||||
Debian 13.7
|
||||
Felhom release os-host-20261111-034654
|
||||
Pending 101
|
||||
Not covered 0
|
||||
Held none
|
||||
Reboot needed no
|
||||
kernel.panic 10 s
|
||||
Oops no
|
||||
Crash restarts 24 h 0
|
||||
Crash guard armed
|
||||
[hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files 0.174.0
|
||||
Guest
|
||||
Guest Debian 13.7
|
||||
Felhom release os-guest-20261111-034654
|
||||
Pending 6
|
||||
Restart needed 0
|
||||
[hover: The newest pct fstrim of the guest's disks, and what it freed (R-444). The boxes trim weekly.] Last disk trim [hover: schedule: weekly, due Wednesday from 10:00 host-local time; starts only 10:00-20:59; never beside a backup or restore-test] 3 days ago · 90.4 GiB
|
||||
[hover: The host's weekly 10-second iperf3 upload to ep0 inside the tunnel, best of three (R-1247). qBittorrent's default limit is half of it.] Upload line [hover: schedule: weekly, due Tuesday from 10:00 host-local time; starts 10:00-15:59 after 10 quiet minutes; never beside a backup, restore-test or update] 93.0 Mbit/s · 4 days ago
|
||||
Details
|
||||
Release ids, cancelled approvals, what ring 0 runs, the version floors, and every box's crash guard and root files.
|
||||
Approved releases
|
||||
The newest approved set of each layer. A [hover: Approved while a TEST wait override was active. It is cancelled when the hub starts without the override (`11` §5.3.1).] TEST approval is amber.
|
||||
guest : os-guest-20261111-034654
|
||||
272 packages · 2026-11-11 03:46 UTC · by auto
|
||||
host : os-host-20261115-045718
|
||||
607 packages · 2026-11-15 04:57 UTC · by auto
|
||||
docker : os-docker-20261026-190351
|
||||
6 packages · 2026-10-26 19:03 UTC · by operator
|
||||
pve : os-pve-20261114-075712
|
||||
121 packages · 2026-11-14 07:57 UTC · by operator
|
||||
kernel : os-kernel-20261114-075716
|
||||
2 packages · 2026-11-14 07:57 UTC · by operator
|
||||
Approve now (guest + host)
|
||||
An urgent fix only — normally the hub approves after 24 h and one night.
|
||||
What ring 0 runs now
|
||||
guest :
|
||||
272 packages, first seen 2026-11-10 01:49 UTC
|
||||
approved as os-guest-20261111-034654
|
||||
host :
|
||||
607 packages, first seen 2026-11-14 04:52 UTC
|
||||
approved as os-host-20261115-045718
|
||||
docker :
|
||||
6 packages, first seen 2026-11-13 04:49 UTC
|
||||
ready — see "Waiting for you"
|
||||
pve :
|
||||
121 packages, first seen 2026-11-15 04:58 UTC
|
||||
demo-felhom-8363b5 has 0 of 2 healthy night Proxmox step(s) with this set
|
||||
kernel :
|
||||
2 packages, first seen 2026-11-10 01:57 UTC
|
||||
approved as os-kernel-20261114-075716
|
||||
Version floors
|
||||
Global controller floor: 0.352.1 · vouched agent: 0.174.0
|
||||
[hover: A floor is the lowest controller version a box must run; the hub moves a box below it up.] What is a floor? · [hover: The agent version the operator checked and signed off for the fleet. Agents update only by a per-box signed job.] What is a vouched agent?
|
||||
No per-customer floors: every box follows the global floor.
|
||||
Crash guard and root files
|
||||
Box ¦ kernel.panic ¦ Oops ¦ Crash restarts 24 h ¦ Crash guard ¦ [hover: The root-owned config bundle: sudoers, wrappers, units (R-840)] Root files ¦
|
||||
Tester4-46836f ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ [hover: behind the vouched agent 0.174.0's bundle — send it with a signed agent_config_update (behind since 2026-11-14 15:03 UTC)] 0.173.0 ¦
|
||||
demo-felhom-8363b5 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦
|
||||
demo-hp-bb76ea ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦
|
||||
tester-1-d70be4 ¦ 10 s ¦ no ¦ 0 ¦ armed ¦ 0.174.0 ¦
|
||||
Felhom Hub 0.181.0
|
||||
@@ -0,0 +1,38 @@
|
||||
# Part E — READS ONLY on a test box, the morning after the first night a box may remove kernels by itself.
|
||||
# Nothing here writes: no systemctl action, no apt, no pct start/stop, no file made on the box.
|
||||
set -u
|
||||
echo "#### $(hostname) $(date -u +%FT%TZ) local $(date +%F\ %T\ %Z) epoch $(date +%s.%N)"
|
||||
echo "agent: $(felhom-agent --version 2>&1 | head -1); uptime: $(uptime -p); running kernel: $(uname -r)"
|
||||
echo "## kernels installed (ii)"
|
||||
dpkg-query -W -f '${Package} ${db:Status-Abbrev}\n' 'proxmox-kernel-*' 2>/dev/null | awk '$2=="ii" && /pve(-signed)?$/ {print " " $1}'
|
||||
echo "count: $(dpkg-query -W -f '${Package} ${db:Status-Abbrev}\n' 'proxmox-kernel-*' 2>/dev/null | awk '$2=="ii" && /pve(-signed)?$/' | wc -l)"
|
||||
echo "/boot: $(ls /boot/vmlinuz-* 2>/dev/null | sed 's#/boot/vmlinuz-##' | tr '\n' ' ')"
|
||||
echo "/lib/modules: $(ls /lib/modules | tr '\n' ' ')"
|
||||
echo "## default kernel and the pinned file"
|
||||
echo "grub.cfg default: $(grep -m1 'set default="gnu' /boot/grub/grub.cfg | grep -o 'gnulinux-[0-9][^"]*-pve-advanced' | head -1)"
|
||||
for f in /etc/default/grub.d/zz-felhom-kernel-default.cfg /var/lib/felhom-kernel/state.json /etc/grub.d/01_felhom_oneshot /etc/grub.d/42_felhom_oneshot; do
|
||||
[ -e "$f" ] && echo " $(sha256sum "$f" | cut -c1-16) $(stat -c '%y' "$f" | cut -c1-19) $f" || echo " (absent) $f"; done
|
||||
echo "pinned file content: $(tr '\n' ' ' < /etc/default/grub.d/zz-felhom-kernel-default.cfg 2>/dev/null | cut -c1-200)"
|
||||
echo "kernel state: $(tr -d '\n' < /var/lib/felhom-kernel/state.json 2>/dev/null | cut -c1-400)"
|
||||
echo "## disk"
|
||||
df -B1 --output=avail,pcent / | tail -1 | awk '{printf " / free %.2f GB, used %s\n", $1/1e9, $2}'
|
||||
echo "## the night's lines: the agent's OS leg and the root wrapper (since yesterday 22:00 local)"
|
||||
journalctl -u felhom-agent --since "yesterday 22:00" --no-pager -o short-iso 2>/dev/null | grep -E "kernel clean-up|osupdate: (pass|leg|night|kernel|host step|the night)|osupdate: wrapper.*(KERNEL CLEANUP|HOST|KERNEL)|whole-guest backup (done|finished|FAILED)|os leg" | cut -c1-420 | tail -40
|
||||
echo "## the wrapper's own journal lines about the clean-up"
|
||||
journalctl -t felhom-os-apply --since "yesterday 22:00" --no-pager -o short-iso 2>/dev/null | grep -E "KERNEL CLEANUP|KERNEL |PROFILE REFRESH|REFUSED|FAILED" | cut -c1-420 | tail -20
|
||||
echo "## health"
|
||||
echo "failed units: $(systemctl --failed --no-legend 2>/dev/null | wc -l) $(systemctl --failed --no-legend 2>/dev/null | awk '{print $1}' | tr '\n' ' ')"
|
||||
for id in $(pct list 2>/dev/null | awk 'NR>1 {print $1}'); do
|
||||
st=$(pct status $id 2>/dev/null | awk '{print $2}')
|
||||
if [ "$st" = running ]; then
|
||||
echo " guest $id: running; containers $(pct exec $id -- docker ps -q 2>/dev/null | wc -l) running, $(pct exec $id -- docker ps --filter health=unhealthy -q 2>/dev/null | wc -l) unhealthy, $(pct exec $id -- docker ps -a --filter status=exited -q 2>/dev/null | wc -l) exited"
|
||||
else echo " guest $id: $st"; fi
|
||||
done
|
||||
echo "dpkg audit: [$(dpkg --audit 2>/dev/null | head -1)]"
|
||||
echo "## time service"
|
||||
echo "chrony: $(systemctl is-active chrony 2>&1); $(chronyc tracking 2>/dev/null | grep -E 'System time|Leap status|Reference ID' | tr '\n' ';' | tr -s ' ')"
|
||||
echo "## the profile-refresh record's folder (a question of the night's review: can the agent read it?)"
|
||||
ls -ld /var/lib/felhom-os-apply 2>&1 | cut -c1-120
|
||||
ls -l /var/lib/felhom-os-apply/profile-refresh.json 2>&1 | cut -c1-140
|
||||
echo "can the agent user read it: $(sudo -n -u felhom-agent test -r /var/lib/felhom-os-apply/profile-refresh.json 2>/dev/null && echo yes || echo no)"
|
||||
echo "#### end $(hostname) epoch $(date +%s.%N)"
|
||||
@@ -0,0 +1,23 @@
|
||||
# Part E, second read — READS ONLY. The first script's kernel count and journal filters matched nothing
|
||||
# (the count's pattern was wrong; the journal words were guessed). This one reads wider.
|
||||
set -u
|
||||
echo "#### $(hostname) $(date +%F\ %T\ %Z)"
|
||||
echo "## kernel packages by dpkg state"
|
||||
dpkg-query -W -f '${db:Status-Abbrev}|${Package}|${Version}\n' 'proxmox-kernel-*' 2>/dev/null | sed 's/^/ /'
|
||||
echo "## dpkg log: kernel package removals and purges, last 3 days"
|
||||
grep -hE " (remove|purge) proxmox-kernel| status (not-installed|config-files) proxmox-kernel" /var/log/dpkg.log 2>/dev/null | grep -E "^2026-11-1[345]" | cut -c1-160 | tail -40
|
||||
echo "## apt history: last 3 entries that name a kernel"
|
||||
grep -B3 -A3 -E "^(Remove|Purge):.*proxmox-kernel" /var/log/apt/history.log 2>/dev/null | cut -c1-600 | tail -30
|
||||
echo "## agent journal since yesterday 20:00: every line with osupdate / kernel / clean"
|
||||
journalctl -u felhom-agent --since "yesterday 20:00" --no-pager -o short-iso 2>/dev/null | grep -iE "osupdate|kernel|clean-?up|cleanup" | cut -c1-500 | tail -60
|
||||
echo "## agent journal line count since yesterday 20:00 (a positive control that the journal is readable)"
|
||||
journalctl -u felhom-agent --since "yesterday 20:00" --no-pager 2>/dev/null | wc -l
|
||||
echo "## the whole journal since yesterday 20:00: lines with felhom-os-apply or KERNEL"
|
||||
journalctl --since "yesterday 20:00" --no-pager -o short-iso 2>/dev/null | grep -E "felhom-os-apply|KERNEL" | cut -c1-500 | tail -40
|
||||
echo "## wrapper state folder (names and times only)"
|
||||
ls -la --time-style=long-iso /var/lib/felhom-os-apply/ 2>&1 | cut -c1-160
|
||||
ls -la --time-style=long-iso /var/lib/felhom-kernel/ 2>&1 | cut -c1-160
|
||||
echo "## disk"
|
||||
df -B1 --output=avail,used,pcent / | tail -1
|
||||
df -B1 --output=avail,used,pcent /boot 2>/dev/null | tail -1
|
||||
echo "#### end"
|
||||
@@ -0,0 +1,12 @@
|
||||
# Part E, third read — READS ONLY. The journal filters of reads 1 and 2 gave 0 lines, positive control included:
|
||||
# the --since form was wrong. This one uses a full date and shows journalctl's own error.
|
||||
set -u
|
||||
echo "#### $(hostname) $(date +%F\ %T\ %Z)"
|
||||
echo "## control: agent journal lines since 2026-11-14 20:00"
|
||||
journalctl -u felhom-agent --since "2026-11-14 20:00:00" --no-pager 2>&1 | wc -l
|
||||
journalctl -u felhom-agent --since "2026-11-14 20:00:00" --no-pager 2>&1 | head -2 | cut -c1-200
|
||||
echo "## agent journal: osupdate / kernel / clean-up lines"
|
||||
journalctl -u felhom-agent --since "2026-11-14 20:00:00" --no-pager -o short-iso 2>&1 | grep -iE "osupdate|kernel|clean-?up|cleanup" | cut -c1-600 | tail -50
|
||||
echo "## whole journal 05:30-06:30: wrapper and sudo lines"
|
||||
journalctl --since "2026-11-15 05:30:00" --until "2026-11-15 06:30:00" --no-pager -o short-iso 2>&1 | grep -E "felhom-os-apply|KERNEL|apt-get" | cut -c1-500 | tail -30
|
||||
echo "#### end"
|
||||
Reference in new issue
Block a user