Kernel spike done (R-836): ESP one-shot and BootNext pass on all three boxes; watchdog fails; design with two operator questions

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 13:08:03 +02:00
parent 28da46a764
commit c79e475511
39 changed files with 201 additions and 1 deletions
@@ -0,0 +1,75 @@
+# The kernel lane — what the spike measured, and a design (R-836, R-812 option B; `09` §3 decisions 164, 171)
+
+Spike 2026-10-07, operator present. Boxes: the Tester 1 box (VM 341 on the HP box, OVMF, Secure Boot off), demo-felhom
+(Intel N100, Secure Boot off), demo-hp (AMD, Secure Boot ON). Kernels: 7.0.2-6 and 7.0.14-20. 24 reboots, 2 operator
+power cycles. Evidence per boot in this folder (`<box>/<step>/before.txt`, `after.txt`, `result.txt`; screenshots for
+the VM). Tools: `tools/`.
+
+## 1. The problem, measured
+
+Installing a kernel makes it the GRUB default at once (R-836, seen again today on Tester 1). A new kernel that fails
+before userspace then stays the default on every power cycle. `--next-boot` and `grub-reboot` are not one-shots on these
+hosts: `/boot` is ext4 on LVM, and GRUB cannot clear `next_entry` there (R-836, 2026-10-04).
+
+## 2. What was measured today
+
+| Candidate | Tester 1 (VM) | demo-felhom | demo-hp (Secure Boot on) |
+|---|---|---|---|
+| **2 — a one-shot flag in a GRUB env block on the ESP (vfat)** | one-shot → new kernel; next boot → old; a panic → back to old by itself | same | same |
+| **1 — UEFI `BootNext` to a copy of the signed loader whose `grub.cfg` names the target** | same | same | same |
+| **3 — a hardware watchdog armed by systemd during the reboot** | FAIL: frozen until `qm reset` | FAIL: frozen until the operator's power cycle | FAIL: frozen until the operator's power cycle |
+
+- A **panic** (no init, `panic=10`) is recovered by 1 and 2 with no person: the crash boot shows in the screenshots
+  (Tester 1) and as a longer gap between boots (51–117 s against 15–34 s).
+- A **freeze** (`panic=0`) is recovered by nothing: the machine reset during the reboot clears the watchdog timer, on
+  the emulated i6300ESB, Intel's TCO and AMD's SP5100 alike. Proxmox also blacklists hardware watchdog drivers by default.
+- My first hang simulation (`init=` only) did NOT hang: initramfs-tools falls back to `/sbin/init`. `rdinit=` and `init=`
+  both missing forces the panic. Worth knowing for every future test of this kind.
+
+## 3. Options for the lane
+
+**A. Candidate 2 — the ESP flag.** A GRUB snippet reads `felhom_next` from an env file on the ESP, boots it once and
+clears it. Writes nothing to firmware. Works with Secure Boot (it is plain `grub.cfg`). Costs: one `/etc/grub.d` file the
+config bundle owns; the flag is written by the root wrapper. Can go wrong: an ESP that is not vfat or a GRUB without
+`fat`/`save_env` — both checked once per box before the lane runs.
+
+**B. Candidate 1 — `BootNext`.** The firmware clears it. Costs: a second loader copy on the ESP that must follow every
+shim/GRUB update, and an NVRAM write per kernel step — some boards wear or reorder NVRAM (demo-hp already holds 41 boot
+entries). Can go wrong: a firmware that ignores `BootNext`.
+
+**C. Turn freezes into panics.** Add `softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10` to the one-shot
+entry, so a lockup the kernel can detect becomes a panic that A or B recovers. NOT measured (no clean way to simulate a
+lockup was tried). A true dead freeze stays a person's power cycle.
+
+## 4. The pick — a proposal
+
+**A, with C on the one-shot entry.** The new kernel boots once from the ESP flag. If the box comes back healthy (the host
+health rule of `11` §8.2, the guest running), a userspace "boot good" step makes it the default. Otherwise the next
+reboot returns to the old kernel by itself. A freeze still needs a person — the design says so to the household rather
+than promising more.
+
+## 5. The night slot
+
+The kernel step ends the night: after the host step (`11` §8.2) and only on a night whose whole-guest backup succeeded.
+The reboot then costs ~1–3 min of the household's apps (measured today: 43–181 s back, the guest starts by itself).
+Ring 0 (the demo boxes) first; ring 1 by signed job after the operator's approval, as the Docker and Proxmox sets.
+
+## 6. First slice and its proof
+
+Build A in the wrapper (`kernel` layer, the ESP snippet in the bundle, the "boot good" step), red tests first (a plan
+without the ESP flag refused; a kernel name outside the approved set refused; the snippet clears the flag). Live proof on
+the Tester 1 box: one step to a new kernel; a forced panic falls back with no person (screenshots + boot gap).
+
+## 7. Questions for the operator
+
+1. **May the box restart at night for a kernel update, and do we tell the household?** That is a promise to users (the
+   design does not decide it). If you do nothing: kernels stay manual and no box restarts by itself.
+2. **A frozen new kernel needs a person to switch the box off and on.** Accept that for the first customers (with the
+   household told what to do), or block the kernel lane until C is measured? If you do nothing: the lane is not built.
+
+## 8. State left on the boxes
+
+Every test entry, the ESP flag, the `BootNext` loader and its firmware entry, and the watchdog config are removed
+(`<box>/G9-cleanup.txt`). Tester 1 and demo-felhom: default 7.0.2-6 (running), 7.0.14-20 installed and booted healthily
+in the spike. demo-hp: default 7.0.14-20 (running). VM 341's test watchdog device is removed (applies at its next start).
+Also seen: after a reboot, demo-felhom answered over Tailscale only after more than 6 minutes (the LAN at once).
@@ -0,0 +1,7 @@
+done
+felhom lines in grub.cfg: 0
+GRUB_DEFAULT="gnulinux-advanced-1af1fcc6-639c-416b-a7e5-c4470d41a502>gnulinux-7.0.2-6-pve-advanced-1af1fcc6-639c-416b-a7e5-c4470d41a502"
+7.0.2-6-pve
+BootOrder: 0001,0003,0002,0000
+BOOT
+proxmox
@@ -0,0 +1,3 @@
+lrwxrwxrwx 1 root root 9 Oct  7 12:55 /dev/felhom-hwwd -> watchdog1
+/sys/class/watchdog/watchdog0 Software Watchdog timeout=10 state=active
+/sys/class/watchdog/watchdog1 SP5100 TCO timer timeout=30 state=active
@@ -0,0 +1,7 @@
+done
+felhom lines in grub.cfg: 0
+GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-20-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43"
+7.0.14-20-pve
+BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009
+BOOT
+proxmox
@@ -0,0 +1,7 @@
+7.0.14-20-pve
+BOOT_IMAGE=/boot/vmlinuz-7.0.14-20-pve root=/dev/mapper/pve-root ro quiet
+env: 
+BootCurrent: 0003
+BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B
+ -1 37b2e108fa0b44f985cfa026b908807e Mon 2026-10-05 09:56:46 CEST Wed 2026-10-07 12:47:43 CEST
+  0 cd3ae3a17ea142119259132a38995458 Wed 2026-10-07 12:48:16 CEST Wed 2026-10-07 12:48:29 CEST
@@ -0,0 +1 @@
+r1-baseline: PASS kernel=7.0.14-20-pve back=181s gap-between-boots=33s
@@ -0,0 +1,7 @@
+7.0.2-6-pve
+BOOT_IMAGE=/boot/vmlinuz-7.0.2-6-pve root=/dev/mapper/pve-root ro quiet panic=10
+env: felhom_next=
+BootCurrent: 0003
+BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B
+ -1 cd3ae3a17ea142119259132a38995458 Wed 2026-10-07 12:48:16 CEST Wed 2026-10-07 12:48:41 CEST
+  0 05f158b202134ce6a285a295d5a40f31 Wed 2026-10-07 12:49:15 CEST Wed 2026-10-07 12:49:26 CEST
@@ -0,0 +1,4 @@
+7.0.14-20-pve
+felhom_next=felhom-spike-good
+BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B
+2026-10-07T10:48:31Z
@@ -0,0 +1 @@
+cd3ae3a1-7ea1-4211-9259-132a38995458
@@ -0,0 +1 @@
+r2-c2-good: PASS kernel=7.0.2-6-pve back=53s gap-between-boots=34s
@@ -0,0 +1,7 @@
+7.0.14-20-pve
+BOOT_IMAGE=/boot/vmlinuz-7.0.14-20-pve root=/dev/mapper/pve-root ro quiet
+env: felhom_next=
+BootCurrent: 0003
+BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B
+ -1 05f158b202134ce6a285a295d5a40f31 Wed 2026-10-07 12:49:15 CEST Wed 2026-10-07 12:49:39 CEST
+  0 321e492ea4e64ea9943db9a544e17700 Wed 2026-10-07 12:50:13 CEST Wed 2026-10-07 12:50:24 CEST
@@ -0,0 +1,4 @@
+7.0.2-6-pve
+felhom_next=
+BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B
+2026-10-07T10:49:27Z
@@ -0,0 +1 @@
+05f158b2-0213-4ce6-a285-a295d5a40f31
@@ -0,0 +1 @@
+r3-c2-plain: PASS kernel=7.0.14-20-pve back=55s gap-between-boots=34s
@@ -0,0 +1,7 @@
+7.0.14-20-pve
+BOOT_IMAGE=/boot/vmlinuz-7.0.14-20-pve root=/dev/mapper/pve-root ro quiet
+env: felhom_next=
+BootCurrent: 0003
+BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B
+ -1 321e492ea4e64ea9943db9a544e17700 Wed 2026-10-07 12:50:13 CEST Wed 2026-10-07 12:50:33 CEST
+  0 cf9353298ce94a438c68cce1cdf5de26 Wed 2026-10-07 12:51:47 CEST Wed 2026-10-07 12:51:59 CEST
@@ -0,0 +1,4 @@
+7.0.14-20-pve
+felhom_next=felhom-spike-panic
+BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B
+2026-10-07T10:50:25Z
@@ -0,0 +1 @@
+321e492e-a4e6-4ea9-943d-b9a544e17700
@@ -0,0 +1 @@
+r4-c2-panic: PASS kernel=7.0.14-20-pve back=91s gap-between-boots=74s
@@ -0,0 +1,7 @@
+7.0.2-6-pve
+BOOT_IMAGE=/boot/vmlinuz-7.0.2-6-pve root=/dev/mapper/pve-root ro quiet panic=10
+env: felhom_next=
+BootCurrent: 0000
+BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,0000,000C,000A
+ -1 cf9353298ce94a438c68cce1cdf5de26 Wed 2026-10-07 12:51:47 CEST Wed 2026-10-07 12:52:11 CEST
+  0 18039e256a30463485e70b98c856d620 Wed 2026-10-07 12:53:06 CEST Wed 2026-10-07 12:53:20 CEST
@@ -0,0 +1,5 @@
+7.0.14-20-pve
+felhom_next=
+BootNext: 0000
+BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B
+2026-10-07T10:52:01Z
@@ -0,0 +1 @@
+cf935329-8ce9-4a43-8c68-cce1cdf5de26
@@ -0,0 +1 @@
+r5-c1-good: PASS kernel=7.0.2-6-pve back=75s gap-between-boots=55s
@@ -0,0 +1,7 @@
+7.0.14-20-pve
+BOOT_IMAGE=/boot/vmlinuz-7.0.14-20-pve root=/dev/mapper/pve-root ro quiet
+env: felhom_next=
+BootCurrent: 0003
+BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,0004,000A
+ -1 18039e256a30463485e70b98c856d620 Wed 2026-10-07 12:53:06 CEST Wed 2026-10-07 12:53:29 CEST
+  0 982d4e0ad5ef47a8a7d0b5f110e9ad13 Wed 2026-10-07 12:55:26 CEST Wed 2026-10-07 12:55:37 CEST
@@ -0,0 +1,5 @@
+7.0.2-6-pve
+felhom_next=
+BootNext: 0000
+BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,0000,000C,000A
+2026-10-07T10:53:21Z
@@ -0,0 +1 @@
+18039e25-6a30-4634-85e7-0b98c856d620
@@ -0,0 +1 @@
+r6-c1-panic: PASS kernel=7.0.14-20-pve back=134s gap-between-boots=117s
@@ -0,0 +1,10 @@
+7.0.14-20-pve
+env: felhom_next=
+ -1 982d4e0ad5ef47a8a7d0b5f110e9ad13 Wed 2026-10-07 12:55:26 CEST Wed 2026-10-07 12:57:53 CEST
+  0 8ea2cdffd1664af69937156f4ee659a5 Wed 2026-10-07 13:05:28 CEST Wed 2026-10-07 13:06:04 CEST
+/sys/class/watchdog/watchdog0 Software Watchdog state=active
+       341 night1004-tester1    running    8192             200.00 1816      
+VMID       Status     Lock         Name                
+9201       running                 demo-hp             
+9202       running                 demo-hp-scratch     
+9401       stopped                 upgrade-harness     
@@ -0,0 +1,4 @@
+7.0.14-20-pve
+felhom_next=felhom-spike-freeze
+BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,0004,000A
+2026-10-07T10:56:00Z
@@ -0,0 +1 @@
+982d4e0a-d5ef-47a8-a7d0-b5f110e9ad13
@@ -0,0 +1 @@
+r7-c3-freeze: FAIL — sp5100_tco armed by systemd (RebootWatchdogSec=90s) did NOT reset the frozen kernel; dead (no ping) 6 min after the reboot until the operator's power cycle; then the default 7.0.14-20
@@ -0,0 +1,7 @@
+done
+felhom lines in grub.cfg: 0
+GRUB_DEFAULT="gnulinux-advanced-e7c978ee-70aa-454b-a5bb-53dc1d19f16e>gnulinux-7.0.2-6-pve-advanced-e7c978ee-70aa-454b-a5bb-53dc1d19f16e"
+7.0.2-6-pve
+BootOrder: 0004,0003,0000,0001
+BOOT
+proxmox
@@ -0,0 +1,10 @@
+#!/bin/bash
+# End of the spike: remove the test entries, the ESP one-shot and the BootNext loader + firmware entry. KEEP the explicit
+# GRUB_DEFAULT the setup wrote (the kernel this box runs and booted healthily); the original file stays as *.felhom-spike-orig.
+set -uo pipefail
+rm -f /etc/grub.d/01_felhom_oneshot /etc/grub.d/41_felhom_spike /boot/efi/EFI/proxmox/felhom-oneshot.env
+bash /root/spike-bootnext-undo.sh >/dev/null 2>&1
+update-grub 2>&1 | tail -1
+echo "felhom lines in grub.cfg: $(grep -c felhom /boot/grub/grub.cfg)"
+grep -m1 '^GRUB_DEFAULT' /etc/default/grub; uname -r
+efibootmgr | grep -E 'BootOrder|felhom|BootNext' | cut -c1-80; ls /boot/efi/EFI
				`@@ -0,0 +1 @@`
				`r1-baseline: PASS kernel=7.0.14-20-pve back=181s gap-between-boots=33s`
				`@@ -0,0 +1 @@`
				`r2-c2-good: PASS kernel=7.0.2-6-pve back=53s gap-between-boots=34s`
				`@@ -0,0 +1 @@`
				`r3-c2-plain: PASS kernel=7.0.14-20-pve back=55s gap-between-boots=34s`
				`@@ -0,0 +1 @@`
				`r4-c2-panic: PASS kernel=7.0.14-20-pve back=91s gap-between-boots=74s`
				`@@ -0,0 +1 @@`
				`r5-c1-good: PASS kernel=7.0.2-6-pve back=75s gap-between-boots=55s`
				`@@ -0,0 +1 @@`
				`r6-c1-panic: PASS kernel=7.0.14-20-pve back=134s gap-between-boots=117s`
				`@@ -0,0 +1 @@`
				`r7-c3-freeze: FAIL — sp5100_tco armed by systemd (RebootWatchdogSec=90s) did NOT reset the frozen kernel; dead (no ping) 6 min after the reboot until the operator's power cycle; then the default 7.0.14-20`