diff --git a/documentation/audits/kernel-night-2026-10-09/readback/RESULT.md b/documentation/audits/kernel-night-2026-10-09/readback/RESULT.md new file mode 100644 index 00000000..cf995d8b --- /dev/null +++ b/documentation/audits/kernel-night-2026-10-09/readback/RESULT.md @@ -0,0 +1,26 @@ +# Kernel night 9→10 October 2026 — read-back (2026-10-10, read only) + +| | demo-hp | demo-felhom | +|---|---|---| +| Told (hub `os_kernel_notice`; household mail) | 7.0.14-22-pve, mail 3 of 3, 2026-10-09 08:39Z | 7.0.14-23-pve, mail 1, 2026-10-09 07:00Z (mail in the inbox 07:00:29Z) | +| Staged (agent + hub `os_kernel_step`) | 02:42:15Z | 02:44:26Z | +| Restart started (agent WARN) | 02:42:20Z | 02:44:30Z | +| Judged healthy, default now (agent; hub event) | 7.0.14-22-pve, 1 m 59 s after the agent started (02:47:10Z) | 7.0.14-23-pve, 39 s after (02:46:03Z) | +| Running (`uname -r`, 05:28Z) | 7.0.14-22-pve | 7.0.14-23-pve | +| Default after (grub.cfg `set default`, `/etc/default/grub.d/zz-felhom-kernel-default.cfg`) | 7.0.14-22-pve | 7.0.14-23-pve | +| Apps down for the restart | ~3 min 28 s: 02:42:20Z → last app 02:45:48Z (Docker StartedAt); metrics gap 02:42:17–02:45:37Z, all 21 back 02:46:35Z | ~65 s: 02:44:30Z → opengist 02:45:35Z (Docker StartedAt); metrics: one sample missing (02:44:41) | +| Other stops that night | local backup tier 02:31Z, 6 app containers for under ~1 min (metrics 21→15→21); 00:30Z 3 containers ~1 min; 02:15Z 1 container ~1 min | none seen (5 containers every sample) | +| False alarm | none (hub: no error/warning/backup-missed event since 2026-10-09 10:00Z; inbox: no demo-box alarm; the positive control: Tester 2's expected 05:00 mails arrived) | none (same two channels; last night's „backup missed" did not return) | + +**„Approve kernel set": NOT shown.** The System page's kernel candidate reads „the ring-0 boxes booted different kernels +(7.0.14-23-pve, 7.0.14-22-pve)" — no `/os/approve-kernel` form on the page (template `system.html:77`). demo-hp is due +7.0.14-23-pve, „not told yet (mails 09–20 h)". + +Noted, not a fault: demo-hp `dd invoked oom-killer` at 02:39:28Z is the planned post-Docker-update memory probe +(`os-apply: OOM-CHECK result=pass`). A stale `saved_entry=…7.0.14-20-pve` in demo-hp's grubenv is not read (the +grub.cfg default is a fixed entry, not `saved`). + +Channels: host journals (`*-night.txt`, trimmed of the 20-second reconcile line and kernel noise), `uname`/grub +(`*-night.txt` head), Docker StartedAt (`*-apps.txt`), the controller's metrics.db (`*-metrics.txt`), the hub DB (copy +with -wal/-shm, read, shredded — `hub-events.txt`), the hub System page (kernel lines only — `hub-system-page-kernel.txt`), +the Gmail inbox. diff --git a/documentation/audits/kernel-night-2026-10-09/readback/demo-felhom-apps.txt b/documentation/audits/kernel-night-2026-10-09/readback/demo-felhom-apps.txt new file mode 100644 index 00000000..b3413303 --- /dev/null +++ b/documentation/audits/kernel-night-2026-10-09/readback/demo-felhom-apps.txt @@ -0,0 +1,9 @@ +## container StartedAt in guest 9201 (UTC) +2026-10-10T02:45:35 cloudflared +2026-10-10T02:45:35 filebrowser +2026-10-10T02:45:35 opengist +2026-10-10T02:45:35 traefik +2026-10-10T02:45:38 felhom-controller +## metrics.db running-container count per sample 02:25-02:55Z +db=/var/lib/docker/volumes/felhom-controller-data/_data/data/metrics.db +tables ['system_metrics', 'container_metrics'] diff --git a/documentation/audits/kernel-night-2026-10-09/readback/demo-felhom-metrics.txt b/documentation/audits/kernel-night-2026-10-09/readback/demo-felhom-metrics.txt new file mode 100644 index 00000000..6152023b --- /dev/null +++ b/documentation/audits/kernel-night-2026-10-09/readback/demo-felhom-metrics.txt @@ -0,0 +1,32 @@ +sample ts: (1791610242,) +2026-10-09T18:00:41+00:00 5 +samples: 690 +2026-10-10T02:25:41+00:00 5 +2026-10-10T02:26:41+00:00 5 +2026-10-10T02:27:41+00:00 5 +2026-10-10T02:28:41+00:00 5 +2026-10-10T02:29:41+00:00 5 +2026-10-10T02:30:41+00:00 5 +2026-10-10T02:31:41+00:00 5 +2026-10-10T02:32:41+00:00 5 +2026-10-10T02:33:41+00:00 5 +2026-10-10T02:34:41+00:00 5 +2026-10-10T02:35:41+00:00 5 +2026-10-10T02:36:41+00:00 5 +2026-10-10T02:37:41+00:00 5 +2026-10-10T02:38:41+00:00 5 +2026-10-10T02:39:41+00:00 5 +2026-10-10T02:40:41+00:00 5 +2026-10-10T02:41:41+00:00 5 +2026-10-10T02:42:41+00:00 5 +2026-10-10T02:43:41+00:00 5 +2026-10-10T02:45:42+00:00 5 +2026-10-10T02:46:42+00:00 5 +2026-10-10T02:47:42+00:00 5 +2026-10-10T02:48:42+00:00 5 +2026-10-10T02:49:42+00:00 5 +2026-10-10T02:50:42+00:00 5 +2026-10-10T02:51:42+00:00 5 +2026-10-10T02:52:42+00:00 5 +2026-10-10T02:53:42+00:00 5 +2026-10-10T02:54:42+00:00 5 diff --git a/documentation/audits/kernel-night-2026-10-09/readback/demo-felhom-night.txt b/documentation/audits/kernel-night-2026-10-09/readback/demo-felhom-night.txt new file mode 100644 index 00000000..93b7a2fe --- /dev/null +++ b/documentation/audits/kernel-night-2026-10-09/readback/demo-felhom-night.txt @@ -0,0 +1,68 @@ +## demo-felhom now 2026-10-10T07:28:31+02:00 +uname: 7.0.14-23-pve + -2 bdac5bc7bea14fe993b766f2529159af Thu 2026-10-08 04:39:52 CEST Fri 2026-10-09 04:45:20 CEST + -1 e1faa7a04fc344f19cbc333f93724a67 Fri 2026-10-09 04:45:39 CEST Sat 2026-10-10 04:44:56 CEST + 0 700dcc3cb7f44fb1a2d5791ad802517b Sat 2026-10-10 04:45:16 CEST Sat 2026-10-10 07:28:31 CEST +## proxmox-boot-tool kernel list +Manually selected kernels: +None. + +Automatically selected kernels: +7.0.14-22-pve +7.0.14-23-pve +## grub default +GRUB_DEFAULT="gnulinux-advanced-1af1fcc6-639c-416b-a7e5-c4470d41a502>gnulinux-7.0.2-6-pve-advanced-1af1fcc6-639c-416b-a7e5-c4470d41a502" +## night journal (agent + os-apply + boot), 2026-10-09 18:00 → now +2026-10-10T04:40:44+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:40:44.703+02:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=felhom-backup last_archive_bytes=945537270 need_bytes=2255663411 avail_bytes=926714380288 +2026-10-10T04:40:45+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:40:45.738+02:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=felhom-backup job=backup-9201-1791600044332044446 +2026-10-10T04:41:15+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:41:15.826+02:00 level=INFO msg="backup: completed" vmid=9201 target=felhom-backup archive=felhom-backup:backup/vzdump-lxc-9201-2026_10_10-04_40_44.tar.zst size_bytes=947755779 uncov +2026-10-10T04:42:45+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:42:45.831+02:00 level=INFO msg="osupdate: START" run=20261010T024245Z layer=guest vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261010T024245Z +2026-10-10T04:42:46+02:00 demo-felhom felhom-os-apply[1287756]: os-apply: START release=ring0-20261010T024245Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0 +2026-10-10T04:42:54+02:00 demo-felhom felhom-os-apply[1287890]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0 +2026-10-10T04:42:54+02:00 demo-felhom felhom-os-apply[1287891]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do) +2026-10-10T04:42:59+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:42:59.231+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T024245Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0" +2026-10-10T04:42:59+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:42:59.231+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-10-10T04:42:59+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:42:59.231+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-10-10T04:42:59+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:42:59.231+02:00 level=INFO msg="osupdate: DONE" run=20261010T024245Z layer=guest vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_cover +2026-10-10T04:42:59+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:42:59.308+02:00 level=INFO msg="osupdate: START" run=20261010T024245Z layer=host vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261010T024245Z +2026-10-10T04:43:00+02:00 demo-felhom felhom-os-apply[1288092]: os-apply: START release=ring0-20261010T024245Z layer=host lane=fast mode=apply select=pending-fast packages=0 +2026-10-10T04:43:05+02:00 demo-felhom felhom-os-apply[1288189]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0 +2026-10-10T04:43:05+02:00 demo-felhom felhom-os-apply[1288190]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do) +2026-10-10T04:43:10+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:10.111+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T024245Z layer=host lane=fast mode=apply select=pending-fast packages=0" +2026-10-10T04:43:10+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:10.111+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-10-10T04:43:10+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:10.111+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-10-10T04:43:10+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:10.918+02:00 level=INFO msg="osupdate: DONE" run=20261010T024245Z layer=host vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=9 not_covere +2026-10-10T04:43:12+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:12.579+02:00 level=INFO msg="osupdate: START" run=20261010T024245Z layer=docker vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261010T024245Z +2026-10-10T04:43:14+02:00 demo-felhom felhom-os-apply[1288848]: os-apply: START release=ring0-20261010T024245Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0 +2026-10-10T04:43:21+02:00 demo-felhom felhom-os-apply[1288993]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0 +2026-10-10T04:43:21+02:00 demo-felhom felhom-os-apply[1288994]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do) +2026-10-10T04:43:34+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:34.982+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T024245Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 autho +2026-10-10T04:43:34+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:34.982+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-10-10T04:43:34+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:34.982+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-10-10T04:43:34+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:34.983+02:00 level=INFO msg="osupdate: DONE" run=20261010T024245Z layer=docker vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_cove +2026-10-10T04:43:34+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:34.996+02:00 level=INFO msg="osupdate: START" run=20261010T024245Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261010T024245Z +2026-10-10T04:43:35+02:00 demo-felhom felhom-os-apply[1289378]: os-apply: START release=ring0-20261010T024245Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0 +2026-10-10T04:43:40+02:00 demo-felhom felhom-os-apply[1289622]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0 +2026-10-10T04:43:40+02:00 demo-felhom felhom-os-apply[1289623]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do) +2026-10-10T04:43:46+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:46.321+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T024245Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=r +2026-10-10T04:43:46+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:46.321+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-10-10T04:43:46+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:46.321+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-10-10T04:43:47+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:43:47.113+02:00 level=INFO msg="osupdate: DONE" run=20261010T024245Z layer=pve vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered +2026-10-10T04:43:47+02:00 demo-felhom sudo[1290072]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-kstatus024347-kernel-kernel-status.json +2026-10-10T04:43:47+02:00 demo-felhom sudo[1290079]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261010T024245Z-kernel-apply.json +2026-10-10T04:43:48+02:00 demo-felhom felhom-os-apply[1290111]: os-apply: START release=ring0-20261010T024245Z layer=kernel lane=slow mode=apply select=listed authority=ring0 running=7.0.14-22-pve +2026-10-10T04:44:26+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:44:26.394+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T024245Z layer=kernel lane=slow mode=apply select=listed authority=ring0 running=7.0.1 +2026-10-10T04:44:26+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:44:26.394+02:00 level=INFO msg="osupdate: DONE" run=20261010T024245Z layer=kernel vmid=9201 trigger=night ring=0 outcome=staged healthy=true reason="" upgraded=2 pending=0 not_cover +2026-10-10T04:44:26+02:00 demo-felhom sudo[1297425]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261010T024245Z-kernel-kernel-reboot.json +2026-10-10T04:44:30+02:00 demo-felhom felhom-agent[141055]: time=2026-10-10T04:44:30.393+02:00 level=WARN msg="osupdate: kernel step — the box restarts now for its one-shot boot" run=20261010T024245Z layer=kernel vmid=9201 trigger=night ring=0 to=7.0.14-23-p +2026-10-10T04:45:24+02:00 demo-felhom sudo[1217]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024524Z-kernel-kernel-boot.json +2026-10-10T04:45:24+02:00 demo-felhom felhom-agent[1205]: time=2026-10-10T04:45:24.313+02:00 level=INFO msg="osupdate: kernel step — judging the one-shot boot" run=boot-20261010T024524Z layer=kernel vmid=0 from=7.0.14-22-pve to=7.0.14-23-pve wait=20m0s +2026-10-10T04:45:24+02:00 demo-felhom felhom-agent[1205]: time=2026-10-10T04:45:24.442+02:00 level=INFO msg="osupdate: kernel step — the box reached the hub on the new kernel" run=boot-20261010T024524Z layer=kernel vmid=0 after=0s +2026-10-10T04:45:24+02:00 demo-felhom sudo[1268]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024524Z-kernel-health.json +2026-10-10T04:45:25+02:00 demo-felhom felhom-os-apply[1388]: os-apply: REFUSED: R10 vmid 9201 is not running +2026-10-10T04:45:25+02:00 demo-felhom felhom-agent[1205]: time=2026-10-10T04:45:25.418+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REFUSED: R10 vmid 9201 is not running" +2026-10-10T04:45:55+02:00 demo-felhom sudo[4115]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024524Z-kernel-health.json +2026-10-10T04:46:00+02:00 demo-felhom sudo[4227]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024524Z-kernel-kernel-good.json +2026-10-10T04:46:03+02:00 demo-felhom felhom-agent[1205]: time=2026-10-10T04:46:03.067+02:00 level=INFO msg="osupdate: kernel step — applied" run=boot-20261010T024524Z layer=kernel vmid=0 to=7.0.14-23-pve reason="healthy 39s after the agent started; the new +2026-10-10T04:46:03+02:00 demo-felhom felhom-agent[1205]: time=2026-10-10T04:46:03.067+02:00 level=INFO msg="osupdate: DONE" run=boot-20261010T024524Z layer=kernel vmid=0 outcome=applied healthy=true reason="healthy 39s after the agent started; the new kernel +2026-10-10T05:15:29+02:00 demo-felhom felhom-agent[1205]: time=2026-10-10T05:15:29.007+02:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=restore-test diff --git a/documentation/audits/kernel-night-2026-10-09/readback/demo-hp-apps.txt b/documentation/audits/kernel-night-2026-10-09/readback/demo-hp-apps.txt new file mode 100644 index 00000000..f4ddaf58 --- /dev/null +++ b/documentation/audits/kernel-night-2026-10-09/readback/demo-hp-apps.txt @@ -0,0 +1,25 @@ +## container StartedAt in guest 9201 (UTC) +2026-10-10T02:45:26 adventurelog +2026-10-10T02:45:26 adventurelog-frontend +2026-10-10T02:45:26 adventurelog-postgres +2026-10-10T02:45:26 bentopdf +2026-10-10T02:45:26 bookstack +2026-10-10T02:45:26 bookstack-db +2026-10-10T02:45:26 cloudflared +2026-10-10T02:45:26 docmost +2026-10-10T02:45:26 docmost-postgres +2026-10-10T02:45:26 docmost-redis +2026-10-10T02:45:26 kimai +2026-10-10T02:45:26 kimai-db +2026-10-10T02:45:26 opengist +2026-10-10T02:45:26 privatebin +2026-10-10T02:45:26 traefik +2026-10-10T02:45:30 felhom-controller +2026-10-10T02:45:34 filebrowser +2026-10-10T02:45:35 calibre-web +2026-10-10T02:45:37 paperless-redis +2026-10-10T02:45:38 paperless-postgres +2026-10-10T02:45:48 paperless-webserver +## metrics.db running-container count per sample 02:25-02:55Z +db=/var/lib/docker/volumes/felhom-controller-data/_data/data/metrics.db +tables ['system_metrics', 'container_metrics'] diff --git a/documentation/audits/kernel-night-2026-10-09/readback/demo-hp-metrics.txt b/documentation/audits/kernel-night-2026-10-09/readback/demo-hp-metrics.txt new file mode 100644 index 00000000..8a7fa2b7 --- /dev/null +++ b/documentation/audits/kernel-night-2026-10-09/readback/demo-hp-metrics.txt @@ -0,0 +1,19 @@ +sample ts: (1791610234,) +2026-10-09T18:00:45+00:00 21 + 2026-10-10T00:29:45+00:00 21 (last sample at that count) +2026-10-10T00:30:47+00:00 18 + 2026-10-10T00:30:47+00:00 18 (last sample at that count) +2026-10-10T00:31:45+00:00 21 + 2026-10-10T02:14:45+00:00 21 (last sample at that count) +2026-10-10T02:15:45+00:00 20 + 2026-10-10T02:15:45+00:00 20 (last sample at that count) +2026-10-10T02:16:45+00:00 21 + 2026-10-10T02:30:45+00:00 21 (last sample at that count) +2026-10-10T02:31:45+00:00 15 + 2026-10-10T02:31:45+00:00 15 (last sample at that count) +2026-10-10T02:32:45+00:00 21 + 2026-10-10T02:42:17+00:00 21 (last sample at that count) +2026-10-10T02:45:37+00:00 18 + 2026-10-10T02:45:37+00:00 18 (last sample at that count) +2026-10-10T02:46:35+00:00 21 +samples: 688 diff --git a/documentation/audits/kernel-night-2026-10-09/readback/demo-hp-night.txt b/documentation/audits/kernel-night-2026-10-09/readback/demo-hp-night.txt new file mode 100644 index 00000000..1dea523e --- /dev/null +++ b/documentation/audits/kernel-night-2026-10-09/readback/demo-hp-night.txt @@ -0,0 +1,78 @@ +## demo-hp now 2026-10-10T07:28:26+02:00 +uname: 7.0.14-22-pve + -2 8ea2cdffd1664af69937156f4ee659a5 Wed 2026-10-07 13:05:28 CEST Wed 2026-10-07 15:15:22 CEST + -1 486c0ec4372b46339aa443894b8e6573 Wed 2026-10-07 15:15:55 CEST Sat 2026-10-10 04:44:26 CEST + 0 578a5f624fe3450d878357b5bd4a1304 Sat 2026-10-10 04:45:00 CEST Sat 2026-10-10 07:28:26 CEST +## proxmox-boot-tool kernel list +Manually selected kernels: +None. + +Automatically selected kernels: +7.0.14-20-pve +7.0.14-22-pve +## grub default +GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-20-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43" +saved_entry=gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-20-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43 +## night journal (agent + os-apply + boot), 2026-10-09 18:00 → now +2026-10-10T04:31:07+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:31:07.201+02:00 level=INFO msg="backup: space preflight passed" vmid=9201 target=local last_archive_bytes=4887664196 need_bytes=7183322069 avail_bytes=17693638656 +2026-10-10T04:31:08+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:31:08.248+02:00 level=INFO msg="local-api: backup reached snapshotted (app may resume)" vmid=9201 target=local job=backup-9201-1791599466684810737 +2026-10-10T04:36:04+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:36:04.038+02:00 level=INFO msg="backup: completed" vmid=9201 target=local archive=local:backup/vzdump-lxc-9201-2026_10_10-04_31_07.tar.zst size_bytes=4891163862 uncovered_volumes=2 +2026-10-10T04:36:49+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:36:49.517+02:00 level=INFO msg="janitor: stale-lock sweep deferred — a heavy operation is in flight" busy=backup:local +2026-10-10T04:37:34+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:37:34.104+02:00 level=INFO msg="osupdate: START" run=20261010T023734Z layer=guest vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261010T023734Z +2026-10-10T04:37:35+02:00 demo-hp felhom-os-apply[2182226]: os-apply: START release=ring0-20261010T023734Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0 +2026-10-10T04:37:45+02:00 demo-hp felhom-os-apply[2182765]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0 +2026-10-10T04:37:45+02:00 demo-hp felhom-os-apply[2182766]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do) +2026-10-10T04:37:53+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:37:53.741+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T023734Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0" +2026-10-10T04:37:53+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:37:53.741+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-10-10T04:37:53+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:37:53.741+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-10-10T04:37:53+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:37:53.743+02:00 level=INFO msg="osupdate: DONE" run=20261010T023734Z layer=guest vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=4 not_covered= +2026-10-10T04:37:53+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:37:53.829+02:00 level=INFO msg="osupdate: START" run=20261010T023734Z layer=host vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261010T023734Z +2026-10-10T04:37:54+02:00 demo-hp felhom-os-apply[2183664]: os-apply: START release=ring0-20261010T023734Z layer=host lane=fast mode=apply select=pending-fast packages=0 +2026-10-10T04:38:03+02:00 demo-hp felhom-os-apply[2184082]: os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0 +2026-10-10T04:38:03+02:00 demo-hp felhom-os-apply[2184083]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do) +2026-10-10T04:38:11+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:38:11.316+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T023734Z layer=host lane=fast mode=apply select=pending-fast packages=0" +2026-10-10T04:38:11+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:38:11.316+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0" +2026-10-10T04:38:11+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:38:11.316+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)" +2026-10-10T04:38:12+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:38:12.399+02:00 level=INFO msg="osupdate: DONE" run=20261010T023734Z layer=host vmid=9201 ring=0 trigger=night outcome=nothing healthy=true reason="" upgraded=0 pending=80 not_covered= +2026-10-10T04:38:14+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:38:14.535+02:00 level=INFO msg="osupdate: START" run=20261010T023734Z layer=docker vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261010T023734Z +2026-10-10T04:38:16+02:00 demo-hp felhom-os-apply[2185445]: os-apply: START release=ring0-20261010T023734Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0 +2026-10-10T04:38:28+02:00 demo-hp felhom-os-apply[2186066]: os-apply: PLAN upgrade=4 already=0 not-installed=0 from-snapshot=0 +2026-10-10T04:39:14+02:00 demo-hp felhom-os-apply[2189325]: os-apply: SOCKET-USERS restarted=felhom-controller,traefik rc=0 (R-858: they held the old docker socket) +2026-10-10T04:39:16+02:00 demo-hp felhom-os-apply[2190082]: os-apply: DONE rc=0 seconds=17.1 upgraded=4 restart-needed=containerd-shim reboot-needed=no +2026-10-10T04:39:34+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:39:34.769+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T023734Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authorit +2026-10-10T04:39:34+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:39:34.769+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=4 already=0 not-installed=0 from-snapshot=0" +2026-10-10T04:39:34+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:39:34.769+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: SOCKET-USERS restarted=felhom-controller,traefik rc=0 (R-858: they held the old docker socket)" +2026-10-10T04:39:34+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:39:34.769+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=17.1 upgraded=4 restart-needed=containerd-shim reboot-needed=no" +2026-10-10T04:39:34+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:39:34.770+02:00 level=INFO msg="osupdate: DONE" run=20261010T023734Z layer=docker vmid=9201 ring=0 trigger=night outcome=applied healthy=true reason="" upgraded=4 pending=0 not_covered +2026-10-10T04:39:34+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:39:34.824+02:00 level=INFO msg="osupdate: START" run=20261010T023734Z layer=pve vmid=9201 ring=0 trigger=night enabled=true release=ring0-20261010T023734Z +2026-10-10T04:39:35+02:00 demo-hp felhom-os-apply[2191847]: os-apply: START release=ring0-20261010T023734Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring0 +2026-10-10T04:39:44+02:00 demo-hp felhom-os-apply[2192251]: os-apply: PLAN upgrade=69 already=0 not-installed=0 from-snapshot=0 +2026-10-10T04:41:16+02:00 demo-hp felhom-os-apply[2204558]: os-apply: DONE rc=0 seconds=87.9 upgraded=69 restart-needed=kvm,rrdcached,watchdog-mux reboot-needed=no +2026-10-10T04:41:24+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:41:24.760+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T023734Z layer=pve:9201 lane=slow mode=apply select=pending-pve packages=0 authority=ring +2026-10-10T04:41:24+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:41:24.760+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=69 already=0 not-installed=0 from-snapshot=0" +2026-10-10T04:41:24+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:41:24.760+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=87.9 upgraded=69 restart-needed=kvm,rrdcached,watchdog-mux reboot-needed=no" +2026-10-10T04:41:25+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:41:25.965+02:00 level=INFO msg="osupdate: DONE" run=20261010T023734Z layer=pve vmid=9201 ring=0 trigger=night outcome=applied healthy=true reason="" upgraded=69 pending=1 not_covered=0 +2026-10-10T04:41:26+02:00 demo-hp sudo[2205330]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-kstatus024126-kernel-kernel-status.json +2026-10-10T04:41:26+02:00 demo-hp sudo[2205337]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261010T023734Z-kernel-apply.json +2026-10-10T04:41:27+02:00 demo-hp felhom-os-apply[2205347]: os-apply: START release=ring0-20261010T023734Z layer=kernel lane=slow mode=apply select=listed authority=ring0 running=7.0.14-20-pve +2026-10-10T04:42:15+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:42:15.457+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261010T023734Z layer=kernel lane=slow mode=apply select=listed authority=ring0 running=7.0.14-2 +2026-10-10T04:42:15+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:42:15.457+02:00 level=INFO msg="osupdate: DONE" run=20261010T023734Z layer=kernel vmid=9201 trigger=night ring=0 outcome=staged healthy=true reason="" upgraded=2 pending=0 not_covered= +2026-10-10T04:42:15+02:00 demo-hp sudo[2214121]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-20261010T023734Z-kernel-kernel-reboot.json +2026-10-10T04:42:20+02:00 demo-hp felhom-agent[2730121]: time=2026-10-10T04:42:20.762+02:00 level=WARN msg="osupdate: kernel step — the box restarts now for its one-shot boot" run=20261010T023734Z layer=kernel vmid=9201 trigger=night ring=0 to=7.0.14-22-pve +2026-10-10T04:45:10+02:00 demo-hp sudo[1274]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024510Z-kernel-kernel-boot.json +2026-10-10T04:45:11+02:00 demo-hp felhom-agent[1262]: time=2026-10-10T04:45:11.006+02:00 level=INFO msg="osupdate: kernel step — judging the one-shot boot" run=boot-20261010T024510Z layer=kernel vmid=0 from=7.0.14-20-pve to=7.0.14-22-pve wait=20m0s +2026-10-10T04:45:11+02:00 demo-hp felhom-agent[1262]: time=2026-10-10T04:45:11.120+02:00 level=INFO msg="osupdate: kernel step — the box reached the hub on the new kernel" run=boot-20261010T024510Z layer=kernel vmid=0 after=0s +2026-10-10T04:45:11+02:00 demo-hp sudo[1327]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024510Z-kernel-health.json +2026-10-10T04:45:12+02:00 demo-hp felhom-os-apply[1466]: os-apply: REFUSED: R10 vmid 9201 is not running +2026-10-10T04:45:12+02:00 demo-hp felhom-agent[1262]: time=2026-10-10T04:45:12.168+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REFUSED: R10 vmid 9201 is not running" +2026-10-10T04:45:42+02:00 demo-hp sudo[9046]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024510Z-kernel-health.json +2026-10-10T04:46:22+02:00 demo-hp sudo[13443]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024510Z-kernel-health.json +2026-10-10T04:47:00+02:00 demo-hp sudo[15419]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024510Z-kernel-health.json +2026-10-10T04:47:06+02:00 demo-hp sudo[15811]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261010T024510Z-kernel-kernel-good.json +2026-10-10T04:47:10+02:00 demo-hp felhom-agent[1262]: time=2026-10-10T04:47:10.027+02:00 level=INFO msg="osupdate: kernel step — applied" run=boot-20261010T024510Z layer=kernel vmid=0 to=7.0.14-22-pve reason="healthy 1m59s after the agent started; the new ke +2026-10-10T04:47:10+02:00 demo-hp felhom-agent[1262]: time=2026-10-10T04:47:10.027+02:00 level=INFO msg="osupdate: DONE" run=boot-20261010T024510Z layer=kernel vmid=0 outcome=applied healthy=true reason="healthy 1m59s after the agent started; the new kernel is +2026-10-10T04:38:03+02:00 demo-hp felhom-os-apply[2184083]: os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do) +2026-10-10T04:38:16+02:00 demo-hp felhom-os-apply[2185445]: os-apply: START release=ring0-20261010T023734Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0 +2026-10-10T04:39:14+02:00 demo-hp felhom-os-apply[2189325]: os-apply: SOCKET-USERS restarted=felhom-controller,traefik rc=0 (R-858: they held the old docker socket) +2026-10-10T04:39:16+02:00 demo-hp felhom-os-apply[2190082]: os-apply: DONE rc=0 seconds=17.1 upgraded=4 restart-needed=containerd-shim reboot-needed=no +2026-10-10T04:39:34+02:00 demo-hp felhom-os-apply[2191829]: os-apply: OOM-CHECK result=pass oom_killed=True oom_event=True exit=137 image=gitea.dooplex.hu/admin/felhom-controller:0.304.0 — the engine reported the memory kill: OOMKilled=true and the oom event diff --git a/documentation/audits/kernel-night-2026-10-09/readback/hub-events.txt b/documentation/audits/kernel-night-2026-10-09/readback/hub-events.txt new file mode 100644 index 00000000..a448bf65 --- /dev/null +++ b/documentation/audits/kernel-night-2026-10-09/readback/hub-events.txt @@ -0,0 +1,17 @@ +('2026-10-10 02:39:34', 'demo-hp', 'os_update_applied', 'info', "System security fixes installed on the box's app engine (Docker 29.9.0) (4 package(s)).") +('2026-10-10 02:41:25', 'demo-hp', 'os_update_applied', 'info', "System security fixes installed on the box's Proxmox system (69 package(s)).") +('2026-10-10 02:42:15', 'demo-hp', 'os_kernel_step', 'info', 'Kernel 7.0.14-22-pve staged on demo-hp-bb76ea (trigger night): installed, never the default; the box boots it ONCE at its night reboot.') +('2026-10-10 02:44:27', 'demo-felhom', 'os_kernel_step', 'info', 'Kernel 7.0.14-23-pve staged on demo-felhom-8363b5 (trigger night): installed, never the default; the box boots it ONCE at its night reboot.') +('2026-10-10 02:46:04', 'demo-felhom', 'os_update_applied', 'info', "System security fixes installed on the box's kernel (the box restarted at night).") +('2026-10-10 02:46:04', 'demo-felhom', 'os_kernel_step', 'info', 'Kernel 7.0.14-23-pve booted healthily on demo-felhom-8363b5 and is the default now (was 7.0.14-22-pve). healthy 39s after the agent started; the new kernel is the default') +('2026-10-10 02:47:10', 'demo-hp', 'os_update_applied', 'info', "System security fixes installed on the box's kernel (the box restarted at night).") +('2026-10-10 02:47:10', 'demo-hp', 'os_kernel_step', 'info', 'Kernel 7.0.14-22-pve booted healthily on demo-hp-bb76ea and is the default now (was 7.0.14-20-pve). healthy 1m59s after the agent started; the new kernel is the default') +--- os_kernel_notice since 2026-10-05, demo boxes +('2026-10-07 16:12:11', 'demo-felhom', 'os_kernel_notice', 'Kernel 7.0.14-20-pve on demo-felhom-8363b5: the household was told the box restarts tonight (mail 1 of at most 3).') +('2026-10-07 16:38:11', 'demo-hp', 'os_kernel_notice', 'Kernel 7.0.14-22-pve on demo-hp-bb76ea: the household was told the box restarts tonight (mail 1 of at most 3).') +('2026-10-08 07:00:40', 'demo-felhom', 'os_kernel_notice', 'Kernel 7.0.14-22-pve on demo-felhom-8363b5: the household was told the box restarts tonight (mail 1 of at most 3).') +('2026-10-08 12:38:40', 'demo-hp', 'os_kernel_notice', 'Kernel 7.0.14-22-pve on demo-hp-bb76ea: the household was told the box restarts tonight (mail 2 of at most 3).') +('2026-10-09 07:00:24', 'demo-felhom', 'os_kernel_notice', 'Kernel 7.0.14-23-pve on demo-felhom-8363b5: the household was told the box restarts tonight (mail 1 of at most 3).') +('2026-10-09 08:39:24', 'demo-hp', 'os_kernel_notice', 'Kernel 7.0.14-22-pve on demo-hp-bb76ea: the household was told the box restarts tonight (mail 3 of at most 3).') +--- notification log (kernel) since 2026-10-08 +['id', 'customer_id', 'event_type', 'severity', 'message', 'status', 'error_message', 'created_at', 'channel'] diff --git a/documentation/audits/kernel-night-2026-10-09/readback/hub-system-page-kernel.txt b/documentation/audits/kernel-night-2026-10-09/readback/hub-system-page-kernel.txt new file mode 100644 index 00000000..16cd0673 --- /dev/null +++ b/documentation/audits/kernel-night-2026-10-09/readback/hub-system-page-kernel.txt @@ -0,0 +1,6 @@ +kernel : +the ring-0 boxes booted different kernels (7.0.14-23-pve, 7.0.14-22-pve) +Proxmox Kernel (running) Kernel (next boot) Kernel (default) Kernel step Debian Felhom release Pending Not covered Held Reboot needed kernel.panic Oops Crash restarts 24 h Crash guard Root files Agent +7.0.14-23-pve 7.0.14-23-pve 7.0.14-23-pve 7.0.14-23-pve applied 2 h ago · phase good 13.7 +7.0.14-22-pve 7.0.14-22-pve 7.0.14-22-pve 7.0.14-22-pve applied 2 h ago · due 7.0.14-23-pve — not told yet (mails 09–20 h) · phase good 13.7 +7.0.14-22-pve 7.0.14-22-pve 7.0.14-22-pve 7.0.14-22-pve applied 2 days ago · phase good 13.7