kernel night 9→10 read back: demo-hp 7.0.14-22 (~3.5 min down), demo-felhom 7.0.14-23 (~65 s), no false alarm, Approve kernel set not shown (boxes on different kernels)
gates / gates (push) Successful in 5m53s
gates / gates (push) Successful in 5m53s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
@@ -0,0 +1,26 @@
|
||||
# Kernel night 9→10 October 2026 — read-back (2026-10-10, read only)
|
||||
|
||||
| | demo-hp | demo-felhom |
|
||||
|---|---|---|
|
||||
| Told (hub `os_kernel_notice`; household mail) | 7.0.14-22-pve, mail 3 of 3, 2026-10-09 08:39Z | 7.0.14-23-pve, mail 1, 2026-10-09 07:00Z (mail in the inbox 07:00:29Z) |
|
||||
| Staged (agent + hub `os_kernel_step`) | 02:42:15Z | 02:44:26Z |
|
||||
| Restart started (agent WARN) | 02:42:20Z | 02:44:30Z |
|
||||
| Judged healthy, default now (agent; hub event) | 7.0.14-22-pve, 1 m 59 s after the agent started (02:47:10Z) | 7.0.14-23-pve, 39 s after (02:46:03Z) |
|
||||
| Running (`uname -r`, 05:28Z) | 7.0.14-22-pve | 7.0.14-23-pve |
|
||||
| Default after (grub.cfg `set default`, `/etc/default/grub.d/zz-felhom-kernel-default.cfg`) | 7.0.14-22-pve | 7.0.14-23-pve |
|
||||
| Apps down for the restart | ~3 min 28 s: 02:42:20Z → last app 02:45:48Z (Docker StartedAt); metrics gap 02:42:17–02:45:37Z, all 21 back 02:46:35Z | ~65 s: 02:44:30Z → opengist 02:45:35Z (Docker StartedAt); metrics: one sample missing (02:44:41) |
|
||||
| Other stops that night | local backup tier 02:31Z, 6 app containers for under ~1 min (metrics 21→15→21); 00:30Z 3 containers ~1 min; 02:15Z 1 container ~1 min | none seen (5 containers every sample) |
|
||||
| False alarm | none (hub: no error/warning/backup-missed event since 2026-10-09 10:00Z; inbox: no demo-box alarm; the positive control: Tester 2's expected 05:00 mails arrived) | none (same two channels; last night's „backup missed" did not return) |
|
||||
|
||||
**„Approve kernel set": NOT shown.** The System page's kernel candidate reads „the ring-0 boxes booted different kernels
|
||||
(7.0.14-23-pve, 7.0.14-22-pve)" — no `/os/approve-kernel` form on the page (template `system.html:77`). demo-hp is due
|
||||
7.0.14-23-pve, „not told yet (mails 09–20 h)".
|
||||
|
||||
Noted, not a fault: demo-hp `dd invoked oom-killer` at 02:39:28Z is the planned post-Docker-update memory probe
|
||||
(`os-apply: OOM-CHECK result=pass`). A stale `saved_entry=…7.0.14-20-pve` in demo-hp's grubenv is not read (the
|
||||
grub.cfg default is a fixed entry, not `saved`).
|
||||
|
||||
Channels: host journals (`*-night.txt`, trimmed of the 20-second reconcile line and kernel noise), `uname`/grub
|
||||
(`*-night.txt` head), Docker StartedAt (`*-apps.txt`), the controller's metrics.db (`*-metrics.txt`), the hub DB (copy
|
||||
with -wal/-shm, read, shredded — `hub-events.txt`), the hub System page (kernel lines only — `hub-system-page-kernel.txt`),
|
||||
the Gmail inbox.
|
||||
@@ -0,0 +1,9 @@
|
||||
## container StartedAt in guest 9201 (UTC)
|
||||
2026-10-10T02:45:35 cloudflared
|
||||
2026-10-10T02:45:35 filebrowser
|
||||
2026-10-10T02:45:35 opengist
|
||||
2026-10-10T02:45:35 traefik
|
||||
2026-10-10T02:45:38 felhom-controller
|
||||
## metrics.db running-container count per sample 02:25-02:55Z
|
||||
db=/var/lib/docker/volumes/felhom-controller-data/_data/data/metrics.db
|
||||
tables ['system_metrics', 'container_metrics']
|
||||
@@ -0,0 +1,32 @@
|
||||
sample ts: (1791610242,)
|
||||
2026-10-09T18:00:41+00:00 5
|
||||
samples: 690
|
||||
2026-10-10T02:25:41+00:00 5
|
||||
2026-10-10T02:26:41+00:00 5
|
||||
2026-10-10T02:27:41+00:00 5
|
||||
2026-10-10T02:28:41+00:00 5
|
||||
2026-10-10T02:29:41+00:00 5
|
||||
2026-10-10T02:30:41+00:00 5
|
||||
2026-10-10T02:31:41+00:00 5
|
||||
2026-10-10T02:32:41+00:00 5
|
||||
2026-10-10T02:33:41+00:00 5
|
||||
2026-10-10T02:34:41+00:00 5
|
||||
2026-10-10T02:35:41+00:00 5
|
||||
2026-10-10T02:36:41+00:00 5
|
||||
2026-10-10T02:37:41+00:00 5
|
||||
2026-10-10T02:38:41+00:00 5
|
||||
2026-10-10T02:39:41+00:00 5
|
||||
2026-10-10T02:40:41+00:00 5
|
||||
2026-10-10T02:41:41+00:00 5
|
||||
2026-10-10T02:42:41+00:00 5
|
||||
2026-10-10T02:43:41+00:00 5
|
||||
2026-10-10T02:45:42+00:00 5
|
||||
2026-10-10T02:46:42+00:00 5
|
||||
2026-10-10T02:47:42+00:00 5
|
||||
2026-10-10T02:48:42+00:00 5
|
||||
2026-10-10T02:49:42+00:00 5
|
||||
2026-10-10T02:50:42+00:00 5
|
||||
2026-10-10T02:51:42+00:00 5
|
||||
2026-10-10T02:52:42+00:00 5
|
||||
2026-10-10T02:53:42+00:00 5
|
||||
2026-10-10T02:54:42+00:00 5
|
||||
@@ -0,0 +1,68 @@
|
||||
## demo-felhom now 2026-10-10T07:28:31+02:00
|
||||
uname: 7.0.14-23-pve
|
||||
-2 bdac5bc7bea14fe993b766f2529159af Thu 2026-10-08 04:39:52 CEST Fri 2026-10-09 04:45:20 CEST
|
||||
-1 e1faa7a04fc344f19cbc333f93724a67 Fri 2026-10-09 04:45:39 CEST Sat 2026-10-10 04:44:56 CEST
|
||||
0 700dcc3cb7f44fb1a2d5791ad802517b Sat 2026-10-10 04:45:16 CEST Sat 2026-10-10 07:28:31 CEST
|
||||
## proxmox-boot-tool kernel list
|
||||
Manually selected kernels:
|
||||
None.
|
||||
|
||||
Automatically selected kernels:
|
||||
7.0.14-22-pve
|
||||
7.0.14-23-pve
|
||||
## grub default
|
||||
GRUB_DEFAULT="gnulinux-advanced-1af1fcc6-639c-416b-a7e5-c4470d41a502>gnulinux-7.0.2-6-pve-advanced-1af1fcc6-639c-416b-a7e5-c4470d41a502"
|
||||
## night journal (agent + os-apply + boot), 2026-10-09 18:00 → now
|
||||
2026-10-10T04:40:44+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:40:44.703+02:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=felhom-backup last_archive_bytes=945537270 need_bytes=2255663411 avail_bytes=926714380288
|
||||
2026-10-10T04:40:45+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:40:45.738+02:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=felhom-backup job=backup-9201-1791600044332044446
|
||||
2026-10-10T04:41:15+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:41:15.826+02:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201-2026_10_10-04_40_44.tar.zst size_bytes=947755779 uncov
|
||||
2026-10-10T04:42:45+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:42:45.831+02:00 level=INFO msg="osupdate: START" run=20261010T024245Z layer=guest vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261010T024245Z
|
||||
2026-10-10T04:42:46+02:00 demo-felhom felhom-os-apply[1287756]: os-apply: START release=ring0-20261010T024245Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0
|
||||
2026-10-10T04:42:54+02:00 demo-felhom felhom-os-apply[1287890]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0
|
||||
2026-10-10T04:42:54+02:00 demo-felhom felhom-os-apply[1287891]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
|
||||
2026-10-10T04:42:59+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:42:59.231+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T024245Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
|
||||
2026-10-10T04:42:59+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:42:59.231+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
|
||||
2026-10-10T04:42:59+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:42:59.231+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
|
||||
2026-10-10T04:42:59+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:42:59.231+02:00 level=INFO msg="osupdate: DONE" run=20261010T024245Z layer=guest vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_cover
|
||||
2026-10-10T04:42:59+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:42:59.308+02:00 level=INFO msg="osupdate: START" run=20261010T024245Z layer=host vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261010T024245Z
|
||||
2026-10-10T04:43:00+02:00 demo-felhom felhom-os-apply[1288092]: os-apply: START release=ring0-20261010T024245Z layer=host lane=fast mode=apply select=pending-fast packages=0
|
||||
2026-10-10T04:43:05+02:00 demo-felhom felhom-os-apply[1288189]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0
|
||||
2026-10-10T04:43:05+02:00 demo-felhom felhom-os-apply[1288190]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
|
||||
2026-10-10T04:43:10+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:10.111+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T024245Z layer=host lane=fast mode=apply select=pending-fast packages=0"
|
||||
2026-10-10T04:43:10+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:10.111+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
|
||||
2026-10-10T04:43:10+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:10.111+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
|
||||
2026-10-10T04:43:10+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:10.918+02:00 level=INFO msg="osupdate: DONE" run=20261010T024245Z layer=host vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=9 not_covere
|
||||
2026-10-10T04:43:12+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:12.579+02:00 level=INFO msg="osupdate: START" run=20261010T024245Z layer=docker vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261010T024245Z
|
||||
2026-10-10T04:43:14+02:00 demo-felhom felhom-os-apply[1288848]: os-apply: START release=ring0-20261010T024245Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0
|
||||
2026-10-10T04:43:21+02:00 demo-felhom felhom-os-apply[1288993]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0
|
||||
2026-10-10T04:43:21+02:00 demo-felhom felhom-os-apply[1288994]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
|
||||
2026-10-10T04:43:34+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:34.982+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T024245Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 autho
|
||||
2026-10-10T04:43:34+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:34.982+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
|
||||
2026-10-10T04:43:34+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:34.982+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
|
||||
2026-10-10T04:43:34+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:34.983+02:00 level=INFO msg="osupdate: DONE" run=20261010T024245Z layer=docker vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_cove
|
||||
2026-10-10T04:43:34+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:34.996+02:00 level=INFO msg="osupdate: START" run=20261010T024245Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261010T024245Z
|
||||
2026-10-10T04:43:35+02:00 demo-felhom felhom-os-apply[1289378]: os-apply: START release=ring0-20261010T024245Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0
|
||||
2026-10-10T04:43:40+02:00 demo-felhom felhom-os-apply[1289622]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0
|
||||
2026-10-10T04:43:40+02:00 demo-felhom felhom-os-apply[1289623]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
|
||||
2026-10-10T04:43:46+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:46.321+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T024245Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=r
|
||||
2026-10-10T04:43:46+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:46.321+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
|
||||
2026-10-10T04:43:46+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:46.321+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
|
||||
2026-10-10T04:43:47+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:47.113+02:00 level=INFO msg="osupdate: DONE" run=20261010T024245Z layer=pve vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered
|
||||
2026-10-10T04:43:47+02:00 demo-felhom sudo[1290072]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-kstatus024347-kernel-kernel-status.json
|
||||
2026-10-10T04:43:47+02:00 demo-felhom sudo[1290079]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261010T024245Z-kernel-apply.json
|
||||
2026-10-10T04:43:48+02:00 demo-felhom felhom-os-apply[1290111]: os-apply: START release=ring0-20261010T024245Z layer=kernel lane=slow mode=apply select=listed authority=ring0 running=7.0.14-22-pve
|
||||
2026-10-10T04:44:26+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:44:26.394+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T024245Z layer=kernel lane=slow mode=apply select=listed authority=ring0 running=7.0.1
|
||||
2026-10-10T04:44:26+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:44:26.394+02:00 level=INFO msg="osupdate: DONE" run=20261010T024245Z layer=kernel vmid=9201 trigger=night ring=0 outcome=staged healthy=true reason="" upgraded=2 pending=0 not_cover
|
||||
2026-10-10T04:44:26+02:00 demo-felhom sudo[1297425]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261010T024245Z-kernel-kernel-reboot.json
|
||||
2026-10-10T04:44:30+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:44:30.393+02:00 level=WARN msg="osupdate: kernel step — the box restarts now for its one-shot boot" run=20261010T024245Z layer=kernel vmid=9201 trigger=night ring=0 to=7.0.14-23-p
|
||||
2026-10-10T04:45:24+02:00 demo-felhom sudo[1217]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024524Z-kernel-kernel-boot.json
|
||||
2026-10-10T04:45:24+02:00 demo-felhom felhom-agent[1205]: time=2026-10-10T04:45:24.313+02:00 level=INFO msg="osupdate: kernel step — judging the one-shot boot" run=boot-20261010T024524Z layer=kernel vmid=0 from=7.0.14-22-pve to=7.0.14-23-pve wait=20m0s
|
||||
2026-10-10T04:45:24+02:00 demo-felhom felhom-agent[1205]: time=2026-10-10T04:45:24.442+02:00 level=INFO msg="osupdate: kernel step — the box reached the hub on the new kernel" run=boot-20261010T024524Z layer=kernel vmid=0 after=0s
|
||||
2026-10-10T04:45:24+02:00 demo-felhom sudo[1268]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024524Z-kernel-health.json
|
||||
2026-10-10T04:45:25+02:00 demo-felhom felhom-os-apply[1388]: os-apply: REFUSED: R10 vmid 9201 is not running
|
||||
2026-10-10T04:45:25+02:00 demo-felhom felhom-agent[1205]: time=2026-10-10T04:45:25.418+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REFUSED: R10 vmid 9201 is not running"
|
||||
2026-10-10T04:45:55+02:00 demo-felhom sudo[4115]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024524Z-kernel-health.json
|
||||
2026-10-10T04:46:00+02:00 demo-felhom sudo[4227]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024524Z-kernel-kernel-good.json
|
||||
2026-10-10T04:46:03+02:00 demo-felhom felhom-agent[1205]: time=2026-10-10T04:46:03.067+02:00 level=INFO msg="osupdate: kernel step — applied" run=boot-20261010T024524Z layer=kernel vmid=0 to=7.0.14-23-pve reason="healthy 39s after the agent started; the new
|
||||
2026-10-10T04:46:03+02:00 demo-felhom felhom-agent[1205]: time=2026-10-10T04:46:03.067+02:00 level=INFO msg="osupdate: DONE" run=boot-20261010T024524Z layer=kernel vmid=0 outcome=applied healthy=true reason="healthy 39s after the agent started; the new kernel
|
||||
2026-10-10T05:15:29+02:00 demo-felhom felhom-agent[1205]: time=2026-10-10T05:15:29.007+02:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=restore-test
|
||||
@@ -0,0 +1,25 @@
|
||||
## container StartedAt in guest 9201 (UTC)
|
||||
2026-10-10T02:45:26 adventurelog
|
||||
2026-10-10T02:45:26 adventurelog-frontend
|
||||
2026-10-10T02:45:26 adventurelog-postgres
|
||||
2026-10-10T02:45:26 bentopdf
|
||||
2026-10-10T02:45:26 bookstack
|
||||
2026-10-10T02:45:26 bookstack-db
|
||||
2026-10-10T02:45:26 cloudflared
|
||||
2026-10-10T02:45:26 docmost
|
||||
2026-10-10T02:45:26 docmost-postgres
|
||||
2026-10-10T02:45:26 docmost-redis
|
||||
2026-10-10T02:45:26 kimai
|
||||
2026-10-10T02:45:26 kimai-db
|
||||
2026-10-10T02:45:26 opengist
|
||||
2026-10-10T02:45:26 privatebin
|
||||
2026-10-10T02:45:26 traefik
|
||||
2026-10-10T02:45:30 felhom-controller
|
||||
2026-10-10T02:45:34 filebrowser
|
||||
2026-10-10T02:45:35 calibre-web
|
||||
2026-10-10T02:45:37 paperless-redis
|
||||
2026-10-10T02:45:38 paperless-postgres
|
||||
2026-10-10T02:45:48 paperless-webserver
|
||||
## metrics.db running-container count per sample 02:25-02:55Z
|
||||
db=/var/lib/docker/volumes/felhom-controller-data/_data/data/metrics.db
|
||||
tables ['system_metrics', 'container_metrics']
|
||||
@@ -0,0 +1,19 @@
|
||||
sample ts: (1791610234,)
|
||||
2026-10-09T18:00:45+00:00 21
|
||||
2026-10-10T00:29:45+00:00 21 (last sample at that count)
|
||||
2026-10-10T00:30:47+00:00 18
|
||||
2026-10-10T00:30:47+00:00 18 (last sample at that count)
|
||||
2026-10-10T00:31:45+00:00 21
|
||||
2026-10-10T02:14:45+00:00 21 (last sample at that count)
|
||||
2026-10-10T02:15:45+00:00 20
|
||||
2026-10-10T02:15:45+00:00 20 (last sample at that count)
|
||||
2026-10-10T02:16:45+00:00 21
|
||||
2026-10-10T02:30:45+00:00 21 (last sample at that count)
|
||||
2026-10-10T02:31:45+00:00 15
|
||||
2026-10-10T02:31:45+00:00 15 (last sample at that count)
|
||||
2026-10-10T02:32:45+00:00 21
|
||||
2026-10-10T02:42:17+00:00 21 (last sample at that count)
|
||||
2026-10-10T02:45:37+00:00 18
|
||||
2026-10-10T02:45:37+00:00 18 (last sample at that count)
|
||||
2026-10-10T02:46:35+00:00 21
|
||||
samples: 688
|
||||
@@ -0,0 +1,78 @@
|
||||
## demo-hp now 2026-10-10T07:28:26+02:00
|
||||
uname: 7.0.14-22-pve
|
||||
-2 8ea2cdffd1664af69937156f4ee659a5 Wed 2026-10-07 13:05:28 CEST Wed 2026-10-07 15:15:22 CEST
|
||||
-1 486c0ec4372b46339aa443894b8e6573 Wed 2026-10-07 15:15:55 CEST Sat 2026-10-10 04:44:26 CEST
|
||||
0 578a5f624fe3450d878357b5bd4a1304 Sat 2026-10-10 04:45:00 CEST Sat 2026-10-10 07:28:26 CEST
|
||||
## proxmox-boot-tool kernel list
|
||||
Manually selected kernels:
|
||||
None.
|
||||
|
||||
Automatically selected kernels:
|
||||
7.0.14-20-pve
|
||||
7.0.14-22-pve
|
||||
## grub default
|
||||
GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-20-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
|
||||
saved_entry=gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-20-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43
|
||||
## night journal (agent + os-apply + boot), 2026-10-09 18:00 → now
|
||||
2026-10-10T04:31:07+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:31:07.201+02:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=local last_archive_bytes=4887664196 need_bytes=7183322069 avail_bytes=17693638656
|
||||
2026-10-10T04:31:08+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:31:08.248+02:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=local job=backup-9201-1791599466684810737
|
||||
2026-10-10T04:36:04+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:36:04.038+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_10_10-04_31_07.tar.zst size_bytes=4891163862 uncovered_volumes=2
|
||||
2026-10-10T04:36:49+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:36:49.517+02:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=backup:local
|
||||
2026-10-10T04:37:34+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:37:34.104+02:00 level=INFO msg="osupdate: START" run=20261010T023734Z layer=guest vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261010T023734Z
|
||||
2026-10-10T04:37:35+02:00 demo-hp felhom-os-apply[2182226]: os-apply: START release=ring0-20261010T023734Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0
|
||||
2026-10-10T04:37:45+02:00 demo-hp felhom-os-apply[2182765]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0
|
||||
2026-10-10T04:37:45+02:00 demo-hp felhom-os-apply[2182766]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
|
||||
2026-10-10T04:37:53+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:37:53.741+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T023734Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
|
||||
2026-10-10T04:37:53+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:37:53.741+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
|
||||
2026-10-10T04:37:53+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:37:53.741+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
|
||||
2026-10-10T04:37:53+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:37:53.743+02:00 level=INFO msg="osupdate: DONE" run=20261010T023734Z layer=guest vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=4 not_covered=
|
||||
2026-10-10T04:37:53+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:37:53.829+02:00 level=INFO msg="osupdate: START" run=20261010T023734Z layer=host vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261010T023734Z
|
||||
2026-10-10T04:37:54+02:00 demo-hp felhom-os-apply[2183664]: os-apply: START release=ring0-20261010T023734Z layer=host lane=fast mode=apply select=pending-fast packages=0
|
||||
2026-10-10T04:38:03+02:00 demo-hp felhom-os-apply[2184082]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0
|
||||
2026-10-10T04:38:03+02:00 demo-hp felhom-os-apply[2184083]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
|
||||
2026-10-10T04:38:11+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:38:11.316+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T023734Z layer=host lane=fast mode=apply select=pending-fast packages=0"
|
||||
2026-10-10T04:38:11+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:38:11.316+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
|
||||
2026-10-10T04:38:11+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:38:11.316+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
|
||||
2026-10-10T04:38:12+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:38:12.399+02:00 level=INFO msg="osupdate: DONE" run=20261010T023734Z layer=host vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=80 not_covered=
|
||||
2026-10-10T04:38:14+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:38:14.535+02:00 level=INFO msg="osupdate: START" run=20261010T023734Z layer=docker vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261010T023734Z
|
||||
2026-10-10T04:38:16+02:00 demo-hp felhom-os-apply[2185445]: os-apply: START release=ring0-20261010T023734Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0
|
||||
2026-10-10T04:38:28+02:00 demo-hp felhom-os-apply[2186066]: os-apply: PLAN upgrade=4 already=0 not-installed=0 from-snapshot=0
|
||||
2026-10-10T04:39:14+02:00 demo-hp felhom-os-apply[2189325]: os-apply: SOCKET-USERS restarted=felhom-controller,traefik rc=0 (R-858: they held the old docker socket)
|
||||
2026-10-10T04:39:16+02:00 demo-hp felhom-os-apply[2190082]: os-apply: DONE rc=0 seconds=17.1 upgraded=4 restart-needed=containerd-shim reboot-needed=no
|
||||
2026-10-10T04:39:34+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:39:34.769+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T023734Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authorit
|
||||
2026-10-10T04:39:34+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:39:34.769+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=4 already=0 not-installed=0 from-snapshot=0"
|
||||
2026-10-10T04:39:34+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:39:34.769+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: SOCKET-USERS restarted=felhom-controller,traefik rc=0 (R-858: they held the old docker socket)"
|
||||
2026-10-10T04:39:34+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:39:34.769+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=17.1 upgraded=4 restart-needed=containerd-shim reboot-needed=no"
|
||||
2026-10-10T04:39:34+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:39:34.770+02:00 level=INFO msg="osupdate: DONE" run=20261010T023734Z layer=docker vmid=9201 ring=0 trigger=night outcome=applied healthy=true reason="" upgraded=4 pending=0 not_covered
|
||||
2026-10-10T04:39:34+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:39:34.824+02:00 level=INFO msg="osupdate: START" run=20261010T023734Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261010T023734Z
|
||||
2026-10-10T04:39:35+02:00 demo-hp felhom-os-apply[2191847]: os-apply: START release=ring0-20261010T023734Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0
|
||||
2026-10-10T04:39:44+02:00 demo-hp felhom-os-apply[2192251]: os-apply: PLAN upgrade=69 already=0 not-installed=0 from-snapshot=0
|
||||
2026-10-10T04:41:16+02:00 demo-hp felhom-os-apply[2204558]: os-apply: DONE rc=0 seconds=87.9 upgraded=69 restart-needed=kvm,rrdcached,watchdog-mux reboot-needed=no
|
||||
2026-10-10T04:41:24+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:41:24.760+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T023734Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring
|
||||
2026-10-10T04:41:24+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:41:24.760+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=69 already=0 not-installed=0 from-snapshot=0"
|
||||
2026-10-10T04:41:24+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:41:24.760+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=87.9 upgraded=69 restart-needed=kvm,rrdcached,watchdog-mux reboot-needed=no"
|
||||
2026-10-10T04:41:25+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:41:25.965+02:00 level=INFO msg="osupdate: DONE" run=20261010T023734Z layer=pve vmid=9201 ring=0 trigger=night outcome=applied healthy=true reason="" upgraded=69 pending=1 not_covered=0
|
||||
2026-10-10T04:41:26+02:00 demo-hp sudo[2205330]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-kstatus024126-kernel-kernel-status.json
|
||||
2026-10-10T04:41:26+02:00 demo-hp sudo[2205337]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261010T023734Z-kernel-apply.json
|
||||
2026-10-10T04:41:27+02:00 demo-hp felhom-os-apply[2205347]: os-apply: START release=ring0-20261010T023734Z layer=kernel lane=slow mode=apply select=listed authority=ring0 running=7.0.14-20-pve
|
||||
2026-10-10T04:42:15+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:42:15.457+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T023734Z layer=kernel lane=slow mode=apply select=listed authority=ring0 running=7.0.14-2
|
||||
2026-10-10T04:42:15+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:42:15.457+02:00 level=INFO msg="osupdate: DONE" run=20261010T023734Z layer=kernel vmid=9201 trigger=night ring=0 outcome=staged healthy=true reason="" upgraded=2 pending=0 not_covered=
|
||||
2026-10-10T04:42:15+02:00 demo-hp sudo[2214121]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261010T023734Z-kernel-kernel-reboot.json
|
||||
2026-10-10T04:42:20+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:42:20.762+02:00 level=WARN msg="osupdate: kernel step — the box restarts now for its one-shot boot" run=20261010T023734Z layer=kernel vmid=9201 trigger=night ring=0 to=7.0.14-22-pve
|
||||
2026-10-10T04:45:10+02:00 demo-hp sudo[1274]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024510Z-kernel-kernel-boot.json
|
||||
2026-10-10T04:45:11+02:00 demo-hp felhom-agent[1262]: time=2026-10-10T04:45:11.006+02:00 level=INFO msg="osupdate: kernel step — judging the one-shot boot" run=boot-20261010T024510Z layer=kernel vmid=0 from=7.0.14-20-pve to=7.0.14-22-pve wait=20m0s
|
||||
2026-10-10T04:45:11+02:00 demo-hp felhom-agent[1262]: time=2026-10-10T04:45:11.120+02:00 level=INFO msg="osupdate: kernel step — the box reached the hub on the new kernel" run=boot-20261010T024510Z layer=kernel vmid=0 after=0s
|
||||
2026-10-10T04:45:11+02:00 demo-hp sudo[1327]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024510Z-kernel-health.json
|
||||
2026-10-10T04:45:12+02:00 demo-hp felhom-os-apply[1466]: os-apply: REFUSED: R10 vmid 9201 is not running
|
||||
2026-10-10T04:45:12+02:00 demo-hp felhom-agent[1262]: time=2026-10-10T04:45:12.168+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REFUSED: R10 vmid 9201 is not running"
|
||||
2026-10-10T04:45:42+02:00 demo-hp sudo[9046]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024510Z-kernel-health.json
|
||||
2026-10-10T04:46:22+02:00 demo-hp sudo[13443]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024510Z-kernel-health.json
|
||||
2026-10-10T04:47:00+02:00 demo-hp sudo[15419]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024510Z-kernel-health.json
|
||||
2026-10-10T04:47:06+02:00 demo-hp sudo[15811]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024510Z-kernel-kernel-good.json
|
||||
2026-10-10T04:47:10+02:00 demo-hp felhom-agent[1262]: time=2026-10-10T04:47:10.027+02:00 level=INFO msg="osupdate: kernel step — applied" run=boot-20261010T024510Z layer=kernel vmid=0 to=7.0.14-22-pve reason="healthy 1m59s after the agent started; the new ke
|
||||
2026-10-10T04:47:10+02:00 demo-hp felhom-agent[1262]: time=2026-10-10T04:47:10.027+02:00 level=INFO msg="osupdate: DONE" run=boot-20261010T024510Z layer=kernel vmid=0 outcome=applied healthy=true reason="healthy 1m59s after the agent started; the new kernel is
|
||||
2026-10-10T04:38:03+02:00 demo-hp felhom-os-apply[2184083]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)
|
||||
2026-10-10T04:38:16+02:00 demo-hp felhom-os-apply[2185445]: os-apply: START release=ring0-20261010T023734Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0
|
||||
2026-10-10T04:39:14+02:00 demo-hp felhom-os-apply[2189325]: os-apply: SOCKET-USERS restarted=felhom-controller,traefik rc=0 (R-858: they held the old docker socket)
|
||||
2026-10-10T04:39:16+02:00 demo-hp felhom-os-apply[2190082]: os-apply: DONE rc=0 seconds=17.1 upgraded=4 restart-needed=containerd-shim reboot-needed=no
|
||||
2026-10-10T04:39:34+02:00 demo-hp felhom-os-apply[2191829]: os-apply: OOM-CHECK result=pass oom_killed=True oom_event=True exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0.304.0 — the engine reported the memory kill: OOMKilled=true and the oom event
|
||||
@@ -0,0 +1,17 @@
|
||||
('2026-10-10 02:39:34', 'demo-hp', 'os_update_applied', 'info', "System security fixes installed on the box's app engine (Docker 29.9.0) (4 package(s)).")
|
||||
('2026-10-10 02:41:25', 'demo-hp', 'os_update_applied', 'info', "System security fixes installed on the box's Proxmox system (69 package(s)).")
|
||||
('2026-10-10 02:42:15', 'demo-hp', 'os_kernel_step', 'info', 'Kernel 7.0.14-22-pve staged on demo-hp-bb76ea (trigger night): installed, never the default; the box boots it ONCE at its night reboot.')
|
||||
('2026-10-10 02:44:27', 'demo-felhom', 'os_kernel_step', 'info', 'Kernel 7.0.14-23-pve staged on demo-felhom-8363b5 (trigger night): installed, never the default; the box boots it ONCE at its night reboot.')
|
||||
('2026-10-10 02:46:04', 'demo-felhom', 'os_update_applied', 'info', "System security fixes installed on the box's kernel (the box restarted at night).")
|
||||
('2026-10-10 02:46:04', 'demo-felhom', 'os_kernel_step', 'info', 'Kernel 7.0.14-23-pve booted healthily on demo-felhom-8363b5 and is the default now (was 7.0.14-22-pve). healthy 39s after the agent started; the new kernel is the default')
|
||||
('2026-10-10 02:47:10', 'demo-hp', 'os_update_applied', 'info', "System security fixes installed on the box's kernel (the box restarted at night).")
|
||||
('2026-10-10 02:47:10', 'demo-hp', 'os_kernel_step', 'info', 'Kernel 7.0.14-22-pve booted healthily on demo-hp-bb76ea and is the default now (was 7.0.14-20-pve). healthy 1m59s after the agent started; the new kernel is the default')
|
||||
--- os_kernel_notice since 2026-10-05, demo boxes
|
||||
('2026-10-07 16:12:11', 'demo-felhom', 'os_kernel_notice', 'Kernel 7.0.14-20-pve on demo-felhom-8363b5: the household was told the box restarts tonight (mail 1 of at most 3).')
|
||||
('2026-10-07 16:38:11', 'demo-hp', 'os_kernel_notice', 'Kernel 7.0.14-22-pve on demo-hp-bb76ea: the household was told the box restarts tonight (mail 1 of at most 3).')
|
||||
('2026-10-08 07:00:40', 'demo-felhom', 'os_kernel_notice', 'Kernel 7.0.14-22-pve on demo-felhom-8363b5: the household was told the box restarts tonight (mail 1 of at most 3).')
|
||||
('2026-10-08 12:38:40', 'demo-hp', 'os_kernel_notice', 'Kernel 7.0.14-22-pve on demo-hp-bb76ea: the household was told the box restarts tonight (mail 2 of at most 3).')
|
||||
('2026-10-09 07:00:24', 'demo-felhom', 'os_kernel_notice', 'Kernel 7.0.14-23-pve on demo-felhom-8363b5: the household was told the box restarts tonight (mail 1 of at most 3).')
|
||||
('2026-10-09 08:39:24', 'demo-hp', 'os_kernel_notice', 'Kernel 7.0.14-22-pve on demo-hp-bb76ea: the household was told the box restarts tonight (mail 3 of at most 3).')
|
||||
--- notification log (kernel) since 2026-10-08
|
||||
['id', 'customer_id', 'event_type', 'severity', 'message', 'status', 'error_message', 'created_at', 'channel']
|
||||
@@ -0,0 +1,6 @@
|
||||
kernel :
|
||||
the ring-0 boxes booted different kernels (7.0.14-23-pve, 7.0.14-22-pve)
|
||||
Proxmox Kernel (running) Kernel (next boot) Kernel (default) Kernel step Debian Felhom release Pending Not covered Held Reboot needed kernel.panic Oops Crash restarts 24 h Crash guard Root files Agent
|
||||
7.0.14-23-pve 7.0.14-23-pve 7.0.14-23-pve 7.0.14-23-pve applied 2 h ago · phase good 13.7
|
||||
7.0.14-22-pve 7.0.14-22-pve 7.0.14-22-pve 7.0.14-22-pve applied 2 h ago · due 7.0.14-23-pve — not told yet (mails 09–20 h) · phase good 13.7
|
||||
7.0.14-22-pve 7.0.14-22-pve 7.0.14-22-pve 7.0.14-22-pve applied 2 days ago · phase good 13.7
|
||||
Reference in New Issue
Block a user