kernel lane Part E1 on the Tester 1 box: forced panic falls back, unhealthy boot self-reverts, healthy boot becomes the default (evidence)

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 17:20:19 +02:00
parent 2ccdf22081
commit 2844d0b0e6
111 changed files with 241 additions and 0 deletions
@@ -0,0 +1,25 @@
+# Part E 1 — the kernel lane by hand on the Tester 1 box (VM 341 on demo-hp), 2026-10-07
+
+Agent v0.152.0 + bundle 0.152.0 (delivered by signed jobs: binary, step bundle `0.152.0-step1`, bundle). Hub v0.143.0.
+Every stage went through the PRODUCT's ring-1 path: a signed `os_kernel_step` (felhom-opsign → hub jobs queue → the
+agent's `KernelStepExecutor` → the wrapper, which re-verifies the signature). Each REBOOT was started by hand with the
+wrapper's own `kernel-reboot` mode (the night leg does the same call on a told night). The after-boot judging was the
+agent's own, untouched. Order run: (b), (c), (a) — a crash and a revert must start from the old kernel.
+
+| Test | Kernel | What happened (times local) | Verdict |
+|---|---|---|---|
+| (b) forced panic | 7.0.2-6 → 7.0.14-20 | staged 16:17:16; the one-shot entry broken BY HAND (`rdinit=` and `init=` both missing, only in grub.cfg); reboot 16:17:47. Screens: GRUB selected `Felhom one-shot: 7.0.14-20-pve` (`shots/s0115.png`), the kernel died, the next GRUB menu selected the default (`s0140.png`). Back on 7.0.2-6 156 s after the reboot; boot gap 41 s. `kernel-boot` → `fell_back`; hub logged it and mailed the operator (`os_kernel_step`). Flag empty. Crash guard: `unclean_boots_in_window` 0 (the panic came before userspace — the planned reboot's clean-stop marker still stood, exactly as `test_a_panic_before_userspace_is_not_even_counted` says). | **PASS** — no person |
+| (c) booted but unhealthy | 7.0.2-6 → 7.0.14-20 | reset BY HAND first (meta-package back to 7.0.2-6, the step record moved aside — `b-panic/5-reset-by-hand.txt`). Staged 16:35:52; guest held BY HAND (`pct set 9201 --onboot 0`); reboot. Up on 7.0.14-20 after 131 s; 16:38:43 `judging`, the hub reached at once; the guest never ran → at 16:58:54 (20 min 11 s) `health_failed` to the hub, then ONE `kernel-revert`; back on 7.0.2-6 at 16:59:22; `kernel-boot` → `self_reverted` (reason: `vmid 9201 is not running`). Guard 0 unclean. Then `onboot 1` + `pct start` by hand. | **PASS** — no person |
+| (a) healthy | 7.0.2-6 → 7.0.14-22 | the step record moved aside BY HAND (the 20 h one-step-per-night rule). Staged 17:15:18 (2 packages: the meta and the new signed image, 28.3 s); reboot; up on 7.0.14-22 after 131 s; `judging` 17:17:53; healthy 1 m 17 s after the agent started → `kernel-good`: the default proved from grub.cfg = 7.0.14-22 (`zz-felhom-kernel-default.cfg`), outcome `applied`. Every container healthy. | **PASS** |
+
+**State left:** Tester 1 runs 7.0.14-22-pve as its GRUB default (booted healthily), flag empty, step `good`, guest running,
+`onboot 1`. Kernels installed: 7.0.2-6, 7.0.14-20, 7.0.14-22.
+
+**Option C (E 2): UNMEASURED.** The Proxmox kernel ships no lockup test module: `CONFIG_TEST_LOCKUP is not set` in
+`/boot/config-7.0.14-20-pve` on demo-hp, `modinfo test_lockup` → not found on both demo boxes. No tool was built (the
+brief). The options are on the one-shot entry (`grub.cfg` line in `a-healthy`/`c-unhealthy` setups); whether a soft
+lockup becomes a panic there is not shown.
+
+**Small things seen:** the self-revert's reason is the wrapper's raw refusal JSON (`no health reading: {"code": "R10",
+…}`) — readable, not pretty; left as is. The System page's running/next-boot cells read "unknown" for a few minutes after
+each boot until the next facts read (10 min cadence) — by design.
@@ -0,0 +1 @@
+by hand: step record moved aside (the 20 h one-step-per-night rule would refuse a third test today)
@@ -0,0 +1,4 @@
+== os_kernel_step (stage 7.0.14-22-pve) → tester-1, 2026-10-07T15:00:22Z
+signed: op=os_kernel_step host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=8b3da15f7909c0540b7783921705a4f7 expires=2026-10-07T15:45:22Z
+wrote envelope to /dev/null
+uploaded signed op to the hub jobs queue
@@ -0,0 +1,4 @@
+status: running
+felhom_next=7.0.14-22-pve
+   set default="gnulinux-advanced-e7c978ee-70aa-454b-a5bb-53dc1d19f16e>gnulinux-7.0.2-6-pve-advanced-e7c978ee-70aa-454b-a5bb-53dc1d19f16e"
+10e8ac4f-88af-4cf2-acb8-91d4ddd75953
@@ -0,0 +1,3 @@
+2026-10-07T15:15:32Z
+os-apply: KERNEL REBOOT — one-shot boot of 7.0.14-22-pve (the default stays 7.0.2-6-pve)
+Connection to 192.168.0.154 closed by remote host.
@@ -0,0 +1,16 @@
+7.0.14-22-pve
+ -1 10e8ac4f88af4cf2acb891d4ddd75953 Wed 2026-10-07 16:59:22 CEST Wed 2026-10-07 17:17:29 CEST
+  0 0950e94d7c5741f68d633702ed2ccff6 Wed 2026-10-07 17:17:46 CEST Wed 2026-10-07 17:19:30 CEST
+   set default="gnulinux-advanced-e7c978ee-70aa-454b-a5bb-53dc1d19f16e>gnulinux-7.0.14-22-pve-advanced-e7c978ee-70aa-454b-a5bb-53dc1d19f16e"
+# felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily
+GRUB_DEFAULT="gnulinux-advanced-e7c978ee-70aa-454b-a5bb-53dc1d19f16e>gnulinux-7.0.14-22-pve-advanced-e7c978ee-70aa-454b-a5bb-53dc1d19f16e"
+felhom_next=
+  "phase": "good",
+status: running
+filebrowser Up About a minute (healthy)
+felhom-controller Up About a minute (healthy)
+privatebin Up About a minute (healthy)
+bookstack Up About a minute (healthy)
+bookstack-db Up About a minute (healthy)
+cloudflared Up About a minute (healthy)
+traefik Up About a minute
@@ -0,0 +1 @@
+10e8ac4f-88af-4cf2-acb8-91d4ddd75953
@@ -0,0 +1,29 @@
+0s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+4s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+9s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+13s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+18s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+23s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+27s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+32s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+37s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+41s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+46s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+51s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+55s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+60s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+64s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+69s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+74s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+78s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+83s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+88s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+92s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+97s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+101s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+106s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ]
+110s ssh=[down]
+115s ssh=[down]
+123s ssh=[down]
+131s ssh=[0950e94d-7c57-41f6-8d63-3702ed2ccff6 7.0.14-22-pve ]
+UP after 131s: 0950e94d-7c57-41f6-8d63-3702ed2ccff6 7.0.14-22-pve 
@@ -0,0 +1,2 @@
+OSAPPLY-REPORT {"kernel": {"default": "7.0.2-6-pve", "flag": null, "from": null, "phase": "none", "reason": null, "rebooted_at": null, "result_at": null, "running": "7.0.2-6-pve", "self_revert_used": false, "setup_problems": [], "staged_at": null, "step_id": null, "to": null, "vmid": null}, "kernel_mode": "kernel-status", "layer": "kernel", "mode": "kernel-status", "pass_seconds": 0.0, "refused": null, "release_id": "by-hand", "vmid": 9201}
+rc=0
@@ -0,0 +1,4 @@
+== os_kernel_step (stage 7.0.14-20-pve) → tester-1, 2026-10-07T14:02:28Z
+signed: op=os_kernel_step host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=dcf74e2325224b4b02b2407415e55377 expires=2026-10-07T14:47:28Z
+wrote envelope to /dev/null
+uploaded signed op to the hub jobs queue
@@ -0,0 +1,23 @@
+  linux /boot/vmlinuz-7.0.14-20-pve root=/dev/mapper/pve-root ro  quiet softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10 rdinit=/felhom-test-no-rdinit init=/felhom-test-no-init
+1
+syntax-ok
+felhom_next=7.0.14-20-pve
+{
+  "authority": "signed",
+  "default_before": "7.0.2-6-pve",
+  "from": "7.0.2-6-pve",
+  "packages": [
+    {
+      "name": "proxmox-kernel-7.0",
+      "version": "7.0.14-20"
+    }
+  ],
+  "phase": "staged",
+  "self_revert_used": false,
+  "staged_at": "2026-10-07T14:17:16Z",
+  "step_id": "e1-b-panic",
+  "to": "7.0.14-20-pve",
+  "updated_at": "2026-10-07T14:17:16Z",
+  "vmid": 9201
+}
+7db243cb-7eec-4a96-8582-785d21454458
@@ -0,0 +1,3 @@
+2026-10-07T14:17:47Z
+os-apply: KERNEL REBOOT — one-shot boot of 7.0.14-20-pve (the default stays 7.0.2-6-pve)
+Connection to 192.168.0.154 closed by remote host.
@@ -0,0 +1,16 @@
+7.0.2-6-pve
+felhom_next=
+ -2 73bd8914acb84b1a80aada7daaa1d479 Wed 2026-10-07 13:06:00 CEST Wed 2026-10-07 15:15:11 CEST
+ -1 7db243cb7eec4a968582785d21454458 Wed 2026-10-07 15:16:27 CEST Wed 2026-10-07 16:19:44 CEST
+  0 766262ddb9f34047bf4cb42b61264d54 Wed 2026-10-07 16:20:25 CEST Wed 2026-10-07 16:20:57 CEST
+felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261007T142032Z-kernel-kernel-boot.json
+os-apply: KERNEL AFTER-BOOT oneshot -> fell_back running=7.0.2-6-pve (7.0.2-6-pve -> 7.0.14-20-pve)
+time=2026-10-07T16:20:32.619+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL AFTER-BOOT oneshot -> fell_back running=7.0.2-6-pve (7.0.2-6-pve -> 7.0.14-20-pve)"
+time=2026-10-07T16:20:32.619+02:00 level=WARN msg="osupdate: kernel step FELL BACK — the new kernel did not come up; the box runs the old one" run=boot-20261007T142032Z layer=kernel vmid=0 from=7.0.2-6-pve to=7.0.14-20-pve
+time=2026-10-07T16:20:32.619+02:00 level=INFO msg="osupdate: DONE" run=boot-20261007T142032Z layer=kernel vmid=0 outcome=fell_back healthy=false reason="the box came back on the old kernel by itself (the new one did not boot, or crashed)" upgraded=0 pending=0 not_covered=0 restart_needed=0 reboot_needed=false wrapper_seconds=0
+  "phase": "fell_back",
+  "reason": "the box came back on the old kernel by itself (the new one did not boot, or crashed)",
+1
+  "armed": true,
+  "last_boot_unclean": false,
+  "unclean_boots_in_window": 0,
@@ -0,0 +1,3 @@
+Unpacking proxmox-kernel-7.0 (7.0.2-6) over (7.0.14-20) ...
+Setting up proxmox-kernel-7.0 (7.0.2-6) ...
+proxmox-kernel-7.0 7.0.2-6
@@ -0,0 +1 @@
+7db243cb-7eec-4a96-8582-785d21454458
@@ -0,0 +1,32 @@
+0s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+5s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+9s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+14s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+18s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+23s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+28s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+32s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+37s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+42s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+46s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+51s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+55s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+60s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+65s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+69s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+74s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+78s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+83s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+88s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+92s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+97s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+102s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+106s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ]
+110s ssh=[down]
+115s ssh=[down]
+123s ssh=[down]
+131s ssh=[down]
+140s ssh=[down]
+148s ssh=[down]
+156s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ]
+UP after 156s: 766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve 
@@ -0,0 +1,4 @@
+== os_kernel_step (stage 7.0.14-20-pve again) → tester-1, 2026-10-07T14:21:41Z
+signed: op=os_kernel_step host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=0e5e34cd628b17af29523b11e0109052 expires=2026-10-07T15:06:41Z
+wrote envelope to /dev/null
+uploaded signed op to the hub jobs queue
@@ -0,0 +1,4 @@
+0
+onboot: 0
+felhom_next=7.0.14-20-pve
+766262dd-b9f3-4047-bf4c-b42b61264d54
@@ -0,0 +1,3 @@
+2026-10-07T14:36:23Z
+os-apply: KERNEL REBOOT — one-shot boot of 7.0.14-20-pve (the default stays 7.0.2-6-pve)
+Connection to 192.168.0.154 closed by remote host.
@@ -0,0 +1,21 @@
+7.0.2-6-pve
+ -2 766262ddb9f34047bf4cb42b61264d54 Wed 2026-10-07 16:20:25 CEST Wed 2026-10-07 16:38:19 CEST
+ -1 0da447bee6084d62a1d8de2de0e3fa37 Wed 2026-10-07 16:38:36 CEST Wed 2026-10-07 16:59:06 CEST
+  0 10e8ac4f88af4cf2acb891d4ddd75953 Wed 2026-10-07 16:59:22 CEST Wed 2026-10-07 16:59:59 CEST
+2026-10-07T16:38:43+02:00 felhom felhom-agent[1187]: time=2026-10-07T16:38:43.485+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL AFTER-BOOT oneshot -> judging running=7.0.14-20-pve (7.0.2-6-pve -> 7.0.14-20-pve)"
+2026-10-07T16:38:43+02:00 felhom felhom-agent[1187]: time=2026-10-07T16:38:43.486+02:00 level=INFO msg="osupdate: kernel step — judging the one-shot boot" run=boot-20261007T143843Z layer=kernel vmid=0 from=7.0.2-6-pve to=7.0.14-20-pve wait=20m0s
+2026-10-07T16:38:43+02:00 felhom felhom-agent[1187]: time=2026-10-07T16:38:43.621+02:00 level=INFO msg="osupdate: kernel step — the box reached the hub on the new kernel" run=boot-20261007T143843Z layer=kernel vmid=0 after=0s
+2026-10-07T16:58:54+02:00 felhom felhom-agent[1187]: time=2026-10-07T16:58:54.560+02:00 level=ERROR msg="osupdate: kernel step — the one-shot boot is NOT healthy; restarting ONCE into the old kernel" run=boot-20261007T143843Z layer=kernel vmid=0 reason="no health reading: {\"code\": \"R10\", \"reason\": \"vmid 9201 is not running\"}" waited=20m0s from=7.0.2-6-pve to=7.0.14-20-pve
+2026-10-07T16:58:54+02:00 felhom felhom-os-apply[12234]: os-apply: KERNEL REBOOT — self-revert to 7.0.2-6-pve: no health reading: {"code": "R10", "reason": "vmid 9201 is not running"}
+2026-10-07T16:58:54+02:00 felhom felhom-agent[1187]: time=2026-10-07T16:58:54.690+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL REBOOT — self-revert to 7.0.2-6-pve: no health reading: {\"code\": \"R10\", \"reason\": \"vmid 9201 is not running\"}"
+---
+os-apply: KERNEL AFTER-BOOT reverting -> self_reverted running=7.0.2-6-pve (7.0.2-6-pve -> 7.0.14-20-pve)
+time=2026-10-07T16:59:30.807+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL AFTER-BOOT reverting -> self_reverted running=7.0.2-6-pve (7.0.2-6-pve -> 7.0.14-20-pve)"
+time=2026-10-07T16:59:30.808+02:00 level=WARN msg="osupdate: kernel step SELF-REVERTED — back on the old kernel" run=boot-20261007T145930Z layer=kernel vmid=0 from=7.0.2-6-pve to=7.0.14-20-pve reason="no health reading: {\"code\": \"R10\", \"reason\": \"vmid 9201 is not running\"}"
+  "phase": "self_reverted",
+  "reason": "no health reading: {\"code\": \"R10\", \"reason\": \"vmid 9201 is not running\"}",
+  "self_revert_used": true,
+  "armed": true,
+  "unclean_boots_in_window": 0,
+felhom_next=
+status: running
@@ -0,0 +1 @@
+766262dd-b9f3-4047-bf4c-b42b61264d54
--- a/Show More
+++ b/Show More
				`@@ -0,0 +1 @@`
				`by hand: step record moved aside (the 20 h one-step-per-night rule would refuse a third test today)`