diff --git a/documentation/audits/kernel-lane-2026-10-07/E/RESULT.md b/documentation/audits/kernel-lane-2026-10-07/E/RESULT.md new file mode 100644 index 00000000..2837b850 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/RESULT.md @@ -0,0 +1,25 @@ +# Part E 1 — the kernel lane by hand on the Tester 1 box (VM 341 on demo-hp), 2026-10-07 + +Agent v0.152.0 + bundle 0.152.0 (delivered by signed jobs: binary, step bundle `0.152.0-step1`, bundle). Hub v0.143.0. +Every stage went through the PRODUCT's ring-1 path: a signed `os_kernel_step` (felhom-opsign → hub jobs queue → the +agent's `KernelStepExecutor` → the wrapper, which re-verifies the signature). Each REBOOT was started by hand with the +wrapper's own `kernel-reboot` mode (the night leg does the same call on a told night). The after-boot judging was the +agent's own, untouched. Order run: (b), (c), (a) — a crash and a revert must start from the old kernel. + +| Test | Kernel | What happened (times local) | Verdict | +|---|---|---|---| +| (b) forced panic | 7.0.2-6 → 7.0.14-20 | staged 16:17:16; the one-shot entry broken BY HAND (`rdinit=` and `init=` both missing, only in grub.cfg); reboot 16:17:47. Screens: GRUB selected `Felhom one-shot: 7.0.14-20-pve` (`shots/s0115.png`), the kernel died, the next GRUB menu selected the default (`s0140.png`). Back on 7.0.2-6 156 s after the reboot; boot gap 41 s. `kernel-boot` → `fell_back`; hub logged it and mailed the operator (`os_kernel_step`). Flag empty. Crash guard: `unclean_boots_in_window` 0 (the panic came before userspace — the planned reboot's clean-stop marker still stood, exactly as `test_a_panic_before_userspace_is_not_even_counted` says). | **PASS** — no person | +| (c) booted but unhealthy | 7.0.2-6 → 7.0.14-20 | reset BY HAND first (meta-package back to 7.0.2-6, the step record moved aside — `b-panic/5-reset-by-hand.txt`). Staged 16:35:52; guest held BY HAND (`pct set 9201 --onboot 0`); reboot. Up on 7.0.14-20 after 131 s; 16:38:43 `judging`, the hub reached at once; the guest never ran → at 16:58:54 (20 min 11 s) `health_failed` to the hub, then ONE `kernel-revert`; back on 7.0.2-6 at 16:59:22; `kernel-boot` → `self_reverted` (reason: `vmid 9201 is not running`). Guard 0 unclean. Then `onboot 1` + `pct start` by hand. | **PASS** — no person | +| (a) healthy | 7.0.2-6 → 7.0.14-22 | the step record moved aside BY HAND (the 20 h one-step-per-night rule). Staged 17:15:18 (2 packages: the meta and the new signed image, 28.3 s); reboot; up on 7.0.14-22 after 131 s; `judging` 17:17:53; healthy 1 m 17 s after the agent started → `kernel-good`: the default proved from grub.cfg = 7.0.14-22 (`zz-felhom-kernel-default.cfg`), outcome `applied`. Every container healthy. | **PASS** | + +**State left:** Tester 1 runs 7.0.14-22-pve as its GRUB default (booted healthily), flag empty, step `good`, guest running, +`onboot 1`. Kernels installed: 7.0.2-6, 7.0.14-20, 7.0.14-22. + +**Option C (E 2): UNMEASURED.** The Proxmox kernel ships no lockup test module: `CONFIG_TEST_LOCKUP is not set` in +`/boot/config-7.0.14-20-pve` on demo-hp, `modinfo test_lockup` → not found on both demo boxes. No tool was built (the +brief). The options are on the one-shot entry (`grub.cfg` line in `a-healthy`/`c-unhealthy` setups); whether a soft +lockup becomes a panic there is not shown. + +**Small things seen:** the self-revert's reason is the wrapper's raw refusal JSON (`no health reading: {"code": "R10", +…}`) — readable, not pretty; left as is. The System page's running/next-boot cells read "unknown" for a few minutes after +each boot until the next facts read (10 min cadence) — by design. diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/0-reset-by-hand.txt b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/0-reset-by-hand.txt new file mode 100644 index 00000000..3b1b0a21 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/0-reset-by-hand.txt @@ -0,0 +1 @@ +by hand: step record moved aside (the 20 h one-step-per-night rule would refuse a third test today) diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/1-signed.txt b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/1-signed.txt new file mode 100644 index 00000000..59480aec --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/1-signed.txt @@ -0,0 +1,4 @@ +== os_kernel_step (stage 7.0.14-22-pve) → tester-1, 2026-10-07T15:00:22Z +signed: op=os_kernel_step host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=8b3da15f7909c0540b7783921705a4f7 expires=2026-10-07T15:45:22Z +wrote envelope to /dev/null +uploaded signed op to the hub jobs queue diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/2-before.txt b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/2-before.txt new file mode 100644 index 00000000..6511ae99 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/2-before.txt @@ -0,0 +1,4 @@ +status: running +felhom_next=7.0.14-22-pve + set default="gnulinux-advanced-e7c978ee-70aa-454b-a5bb-53dc1d19f16e>gnulinux-7.0.2-6-pve-advanced-e7c978ee-70aa-454b-a5bb-53dc1d19f16e" +10e8ac4f-88af-4cf2-acb8-91d4ddd75953 diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/3-reboot-sent.txt b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/3-reboot-sent.txt new file mode 100644 index 00000000..fe865376 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/3-reboot-sent.txt @@ -0,0 +1,3 @@ +2026-10-07T15:15:32Z +os-apply: KERNEL REBOOT — one-shot boot of 7.0.14-22-pve (the default stays 7.0.2-6-pve) +Connection to 192.168.0.154 closed by remote host. diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/4-after.txt b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/4-after.txt new file mode 100644 index 00000000..60e1ffed --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/4-after.txt @@ -0,0 +1,16 @@ +7.0.14-22-pve + -1 10e8ac4f88af4cf2acb891d4ddd75953 Wed 2026-10-07 16:59:22 CEST Wed 2026-10-07 17:17:29 CEST + 0 0950e94d7c5741f68d633702ed2ccff6 Wed 2026-10-07 17:17:46 CEST Wed 2026-10-07 17:19:30 CEST + set default="gnulinux-advanced-e7c978ee-70aa-454b-a5bb-53dc1d19f16e>gnulinux-7.0.14-22-pve-advanced-e7c978ee-70aa-454b-a5bb-53dc1d19f16e" +# felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily +GRUB_DEFAULT="gnulinux-advanced-e7c978ee-70aa-454b-a5bb-53dc1d19f16e>gnulinux-7.0.14-22-pve-advanced-e7c978ee-70aa-454b-a5bb-53dc1d19f16e" +felhom_next= + "phase": "good", +status: running +filebrowser Up About a minute (healthy) +felhom-controller Up About a minute (healthy) +privatebin Up About a minute (healthy) +bookstack Up About a minute (healthy) +bookstack-db Up About a minute (healthy) +cloudflared Up About a minute (healthy) +traefik Up About a minute diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/bootid.before b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/bootid.before new file mode 100644 index 00000000..fbec48e7 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/bootid.before @@ -0,0 +1 @@ +10e8ac4f-88af-4cf2-acb8-91d4ddd75953 diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0000.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0000.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0000.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0004.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0004.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0004.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0009.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0009.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0009.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0013.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0013.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0013.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0018.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0018.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0018.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0023.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0023.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0023.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0027.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0027.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0027.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0032.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0032.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0032.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0037.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0037.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0037.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0041.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0041.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0041.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0046.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0046.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0046.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0051.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0051.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0051.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0055.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0055.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0055.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0060.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0060.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0060.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0064.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0064.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0064.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0069.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0069.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0069.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0074.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0074.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0074.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0078.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0078.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0078.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0083.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0083.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0083.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0088.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0088.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0088.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0092.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0092.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0092.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0097.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0097.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0097.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0101.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0101.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0101.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0106.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0106.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0106.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0110.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0110.png new file mode 100644 index 00000000..1e28fc5f Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0110.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0115.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0115.png new file mode 100644 index 00000000..366fc3ce Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0115.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0123.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0123.png new file mode 100644 index 00000000..42148ddc Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0123.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0131.png b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0131.png new file mode 100644 index 00000000..55601d51 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/shots/s0131.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/watch.log b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/watch.log new file mode 100644 index 00000000..91bb6b4b --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/a-healthy/watch.log @@ -0,0 +1,29 @@ ++0s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++4s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++9s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++13s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++18s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++23s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++27s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++32s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++37s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++41s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++46s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++51s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++55s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++60s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++64s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++69s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++74s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++78s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++83s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++88s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++92s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++97s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++101s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++106s ssh=[10e8ac4f-88af-4cf2-acb8-91d4ddd75953 7.0.2-6-pve ] ++110s ssh=[down] ++115s ssh=[down] ++123s ssh=[down] ++131s ssh=[0950e94d-7c57-41f6-8d63-3702ed2ccff6 7.0.14-22-pve ] +UP after 131s: 0950e94d-7c57-41f6-8d63-3702ed2ccff6 7.0.14-22-pve diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/0-status.txt b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/0-status.txt new file mode 100644 index 00000000..8213551d --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/0-status.txt @@ -0,0 +1,2 @@ +OSAPPLY-REPORT {"kernel": {"default": "7.0.2-6-pve", "flag": null, "from": null, "phase": "none", "reason": null, "rebooted_at": null, "result_at": null, "running": "7.0.2-6-pve", "self_revert_used": false, "setup_problems": [], "staged_at": null, "step_id": null, "to": null, "vmid": null}, "kernel_mode": "kernel-status", "layer": "kernel", "mode": "kernel-status", "pass_seconds": 0.0, "refused": null, "release_id": "by-hand", "vmid": 9201} +rc=0 diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/1-signed.txt b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/1-signed.txt new file mode 100644 index 00000000..99bd1371 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/1-signed.txt @@ -0,0 +1,4 @@ +== os_kernel_step (stage 7.0.14-20-pve) → tester-1, 2026-10-07T14:02:28Z +signed: op=os_kernel_step host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=dcf74e2325224b4b02b2407415e55377 expires=2026-10-07T14:47:28Z +wrote envelope to /dev/null +uploaded signed op to the hub jobs queue diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/2-setup.txt b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/2-setup.txt new file mode 100644 index 00000000..ec466a68 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/2-setup.txt @@ -0,0 +1,23 @@ + linux /boot/vmlinuz-7.0.14-20-pve root=/dev/mapper/pve-root ro quiet softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10 rdinit=/felhom-test-no-rdinit init=/felhom-test-no-init +1 +syntax-ok +felhom_next=7.0.14-20-pve +{ + "authority": "signed", + "default_before": "7.0.2-6-pve", + "from": "7.0.2-6-pve", + "packages": [ + { + "name": "proxmox-kernel-7.0", + "version": "7.0.14-20" + } + ], + "phase": "staged", + "self_revert_used": false, + "staged_at": "2026-10-07T14:17:16Z", + "step_id": "e1-b-panic", + "to": "7.0.14-20-pve", + "updated_at": "2026-10-07T14:17:16Z", + "vmid": 9201 +} +7db243cb-7eec-4a96-8582-785d21454458 diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/3-reboot-sent.txt b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/3-reboot-sent.txt new file mode 100644 index 00000000..a9f8d0d4 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/3-reboot-sent.txt @@ -0,0 +1,3 @@ +2026-10-07T14:17:47Z +os-apply: KERNEL REBOOT — one-shot boot of 7.0.14-20-pve (the default stays 7.0.2-6-pve) +Connection to 192.168.0.154 closed by remote host. diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/4-after.txt b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/4-after.txt new file mode 100644 index 00000000..0bb0c238 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/4-after.txt @@ -0,0 +1,16 @@ +7.0.2-6-pve +felhom_next= + -2 73bd8914acb84b1a80aada7daaa1d479 Wed 2026-10-07 13:06:00 CEST Wed 2026-10-07 15:15:11 CEST + -1 7db243cb7eec4a968582785d21454458 Wed 2026-10-07 15:16:27 CEST Wed 2026-10-07 16:19:44 CEST + 0 766262ddb9f34047bf4cb42b61264d54 Wed 2026-10-07 16:20:25 CEST Wed 2026-10-07 16:20:57 CEST +felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-boot-20261007T142032Z-kernel-kernel-boot.json +os-apply: KERNEL AFTER-BOOT oneshot -> fell_back running=7.0.2-6-pve (7.0.2-6-pve -> 7.0.14-20-pve) +time=2026-10-07T16:20:32.619+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL AFTER-BOOT oneshot -> fell_back running=7.0.2-6-pve (7.0.2-6-pve -> 7.0.14-20-pve)" +time=2026-10-07T16:20:32.619+02:00 level=WARN msg="osupdate: kernel step FELL BACK — the new kernel did not come up; the box runs the old one" run=boot-20261007T142032Z layer=kernel vmid=0 from=7.0.2-6-pve to=7.0.14-20-pve +time=2026-10-07T16:20:32.619+02:00 level=INFO msg="osupdate: DONE" run=boot-20261007T142032Z layer=kernel vmid=0 outcome=fell_back healthy=false reason="the box came back on the old kernel by itself (the new one did not boot, or crashed)" upgraded=0 pending=0 not_covered=0 restart_needed=0 reboot_needed=false wrapper_seconds=0 + "phase": "fell_back", + "reason": "the box came back on the old kernel by itself (the new one did not boot, or crashed)", +1 + "armed": true, + "last_boot_unclean": false, + "unclean_boots_in_window": 0, diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/5-reset-by-hand.txt b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/5-reset-by-hand.txt new file mode 100644 index 00000000..caaef576 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/5-reset-by-hand.txt @@ -0,0 +1,3 @@ +Unpacking proxmox-kernel-7.0 (7.0.2-6) over (7.0.14-20) ... +Setting up proxmox-kernel-7.0 (7.0.2-6) ... +proxmox-kernel-7.0 7.0.2-6 diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/bootid.before b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/bootid.before new file mode 100644 index 00000000..d10a326b --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/bootid.before @@ -0,0 +1 @@ +7db243cb-7eec-4a96-8582-785d21454458 diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0000.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0000.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0000.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0005.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0005.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0005.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0009.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0009.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0009.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0014.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0014.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0014.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0018.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0018.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0018.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0023.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0023.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0023.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0028.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0028.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0028.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0032.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0032.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0032.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0037.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0037.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0037.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0042.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0042.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0042.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0046.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0046.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0046.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0051.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0051.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0051.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0055.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0055.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0055.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0060.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0060.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0060.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0065.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0065.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0065.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0069.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0069.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0069.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0074.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0074.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0074.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0078.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0078.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0078.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0083.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0083.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0083.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0088.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0088.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0088.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0092.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0092.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0092.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0097.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0097.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0097.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0102.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0102.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0102.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0106.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0106.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0106.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0110.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0110.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0110.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0115.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0115.png new file mode 100644 index 00000000..ad3f1ee1 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0115.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0123.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0123.png new file mode 100644 index 00000000..45e606e2 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0123.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0131.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0131.png new file mode 100644 index 00000000..45e606e2 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0131.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0140.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0140.png new file mode 100644 index 00000000..4b26d214 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0140.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0148.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0148.png new file mode 100644 index 00000000..328ca7c7 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0148.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0156.png b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0156.png new file mode 100644 index 00000000..363cf27a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/shots/s0156.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/b-panic/watch.log b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/watch.log new file mode 100644 index 00000000..de6f4c41 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/b-panic/watch.log @@ -0,0 +1,32 @@ ++0s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++5s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++9s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++14s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++18s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++23s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++28s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++32s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++37s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++42s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++46s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++51s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++55s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++60s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++65s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++69s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++74s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++78s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++83s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++88s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++92s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++97s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++102s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++106s ssh=[7db243cb-7eec-4a96-8582-785d21454458 7.0.2-6-pve ] ++110s ssh=[down] ++115s ssh=[down] ++123s ssh=[down] ++131s ssh=[down] ++140s ssh=[down] ++148s ssh=[down] ++156s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] +UP after 156s: 766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/1-signed.txt b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/1-signed.txt new file mode 100644 index 00000000..a1258d83 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/1-signed.txt @@ -0,0 +1,4 @@ +== os_kernel_step (stage 7.0.14-20-pve again) → tester-1, 2026-10-07T14:21:41Z +signed: op=os_kernel_step host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=0e5e34cd628b17af29523b11e0109052 expires=2026-10-07T15:06:41Z +wrote envelope to /dev/null +uploaded signed op to the hub jobs queue diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/2-setup.txt b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/2-setup.txt new file mode 100644 index 00000000..b847f3ad --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/2-setup.txt @@ -0,0 +1,4 @@ +0 +onboot: 0 +felhom_next=7.0.14-20-pve +766262dd-b9f3-4047-bf4c-b42b61264d54 diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/3-reboot-sent.txt b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/3-reboot-sent.txt new file mode 100644 index 00000000..ea6248e0 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/3-reboot-sent.txt @@ -0,0 +1,3 @@ +2026-10-07T14:36:23Z +os-apply: KERNEL REBOOT — one-shot boot of 7.0.14-20-pve (the default stays 7.0.2-6-pve) +Connection to 192.168.0.154 closed by remote host. diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/4-after.txt b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/4-after.txt new file mode 100644 index 00000000..801b7b73 --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/4-after.txt @@ -0,0 +1,21 @@ +7.0.2-6-pve + -2 766262ddb9f34047bf4cb42b61264d54 Wed 2026-10-07 16:20:25 CEST Wed 2026-10-07 16:38:19 CEST + -1 0da447bee6084d62a1d8de2de0e3fa37 Wed 2026-10-07 16:38:36 CEST Wed 2026-10-07 16:59:06 CEST + 0 10e8ac4f88af4cf2acb891d4ddd75953 Wed 2026-10-07 16:59:22 CEST Wed 2026-10-07 16:59:59 CEST +2026-10-07T16:38:43+02:00 felhom felhom-agent[1187]: time=2026-10-07T16:38:43.485+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL AFTER-BOOT oneshot -> judging running=7.0.14-20-pve (7.0.2-6-pve -> 7.0.14-20-pve)" +2026-10-07T16:38:43+02:00 felhom felhom-agent[1187]: time=2026-10-07T16:38:43.486+02:00 level=INFO msg="osupdate: kernel step — judging the one-shot boot" run=boot-20261007T143843Z layer=kernel vmid=0 from=7.0.2-6-pve to=7.0.14-20-pve wait=20m0s +2026-10-07T16:38:43+02:00 felhom felhom-agent[1187]: time=2026-10-07T16:38:43.621+02:00 level=INFO msg="osupdate: kernel step — the box reached the hub on the new kernel" run=boot-20261007T143843Z layer=kernel vmid=0 after=0s +2026-10-07T16:58:54+02:00 felhom felhom-agent[1187]: time=2026-10-07T16:58:54.560+02:00 level=ERROR msg="osupdate: kernel step — the one-shot boot is NOT healthy; restarting ONCE into the old kernel" run=boot-20261007T143843Z layer=kernel vmid=0 reason="no health reading: {\"code\": \"R10\", \"reason\": \"vmid 9201 is not running\"}" waited=20m0s from=7.0.2-6-pve to=7.0.14-20-pve +2026-10-07T16:58:54+02:00 felhom felhom-os-apply[12234]: os-apply: KERNEL REBOOT — self-revert to 7.0.2-6-pve: no health reading: {"code": "R10", "reason": "vmid 9201 is not running"} +2026-10-07T16:58:54+02:00 felhom felhom-agent[1187]: time=2026-10-07T16:58:54.690+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL REBOOT — self-revert to 7.0.2-6-pve: no health reading: {\"code\": \"R10\", \"reason\": \"vmid 9201 is not running\"}" +--- +os-apply: KERNEL AFTER-BOOT reverting -> self_reverted running=7.0.2-6-pve (7.0.2-6-pve -> 7.0.14-20-pve) +time=2026-10-07T16:59:30.807+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: KERNEL AFTER-BOOT reverting -> self_reverted running=7.0.2-6-pve (7.0.2-6-pve -> 7.0.14-20-pve)" +time=2026-10-07T16:59:30.808+02:00 level=WARN msg="osupdate: kernel step SELF-REVERTED — back on the old kernel" run=boot-20261007T145930Z layer=kernel vmid=0 from=7.0.2-6-pve to=7.0.14-20-pve reason="no health reading: {\"code\": \"R10\", \"reason\": \"vmid 9201 is not running\"}" + "phase": "self_reverted", + "reason": "no health reading: {\"code\": \"R10\", \"reason\": \"vmid 9201 is not running\"}", + "self_revert_used": true, + "armed": true, + "unclean_boots_in_window": 0, +felhom_next= +status: running diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/bootid.before b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/bootid.before new file mode 100644 index 00000000..477ea9af --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/bootid.before @@ -0,0 +1 @@ +766262dd-b9f3-4047-bf4c-b42b61264d54 diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0000.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0000.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0000.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0004.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0004.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0004.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0009.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0009.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0009.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0013.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0013.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0013.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0018.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0018.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0018.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0023.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0023.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0023.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0027.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0027.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0027.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0032.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0032.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0032.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0036.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0036.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0036.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0041.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0041.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0041.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0046.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0046.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0046.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0050.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0050.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0050.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0055.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0055.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0055.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0059.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0059.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0059.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0064.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0064.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0064.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0069.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0069.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0069.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0073.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0073.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0073.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0078.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0078.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0078.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0083.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0083.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0083.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0087.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0087.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0087.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0092.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0092.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0092.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0096.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0096.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0096.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0101.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0101.png new file mode 100644 index 00000000..ce28fa03 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0101.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0106.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0106.png new file mode 100644 index 00000000..c8d05b2a Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0106.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0110.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0110.png new file mode 100644 index 00000000..7ad7338e Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0110.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0114.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0114.png new file mode 100644 index 00000000..8f11cc67 Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0114.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0123.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0123.png new file mode 100644 index 00000000..9d4061da Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0123.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0131.png b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0131.png new file mode 100644 index 00000000..6bfa9b4f Binary files /dev/null and b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/shots/s0131.png differ diff --git a/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/watch.log b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/watch.log new file mode 100644 index 00000000..24a6a3ef --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/E/c-unhealthy/watch.log @@ -0,0 +1,29 @@ ++0s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++4s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++9s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++13s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++18s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++23s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++27s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++32s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++36s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++41s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++46s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++50s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++55s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++59s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++64s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++69s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++73s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++78s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++83s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++87s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++92s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++96s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++101s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++106s ssh=[766262dd-b9f3-4047-bf4c-b42b61264d54 7.0.2-6-pve ] ++110s ssh=[down] ++114s ssh=[down] ++123s ssh=[down] ++131s ssh=[0da447be-e608-4d62-a1d8-de2de0e3fa37 7.0.14-20-pve ] +UP after 131s: 0da447be-e608-4d62-a1d8-de2de0e3fa37 7.0.14-20-pve diff --git a/documentation/audits/kernel-lane-2026-10-07/delivery/demo-boxes.txt b/documentation/audits/kernel-lane-2026-10-07/delivery/demo-boxes.txt new file mode 100644 index 00000000..fc5933eb --- /dev/null +++ b/documentation/audits/kernel-lane-2026-10-07/delivery/demo-boxes.txt @@ -0,0 +1,8 @@ +== agent_update 0.152.0 → demo-hp-bb76ea, 2026-10-07T15:19:42Z +signed: op=agent_update host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=a6b28e9753b5f265c5726228f78b725c expires=2026-10-07T16:04:42Z +wrote envelope to /dev/null +uploaded signed op to the hub jobs queue +== agent_update 0.152.0 → demo-felhom-8363b5, 2026-10-07T15:19:42Z +signed: op=agent_update host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=1064b6dde1dd050dd5f6f7b788990412 expires=2026-10-07T16:04:42Z +wrote envelope to /dev/null +uploaded signed op to the hub jobs queue diff --git a/documentation/audits/kernel-lane-2026-10-07/delivery/tester1.txt b/documentation/audits/kernel-lane-2026-10-07/delivery/tester1.txt index 4a7e8679..60cba5ae 100644 --- a/documentation/audits/kernel-lane-2026-10-07/delivery/tester1.txt +++ b/documentation/audits/kernel-lane-2026-10-07/delivery/tester1.txt @@ -6,3 +6,7 @@ uploaded signed op to the hub jobs queue signed: op=agent_config_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=ee80cce24f0a4ccccb6f07c401317d99 expires=2026-10-07T14:20:04Z wrote envelope to /dev/null uploaded signed op to the hub jobs queue +== agent_config_update 0.152.0 → tester-1, 2026-10-07T13:47:14Z +signed: op=agent_config_update host=tester-1-d70be4 guest="" key_id=felhom-op-1 nonce=94357bebffdcfc99cac92524558cf6f7 expires=2026-10-07T14:32:14Z +wrote envelope to /dev/null +uploaded signed op to the hub jobs queue