From c79e4755110876fc7bf1102bb035f79f0c869cd5 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Wed, 7 Oct 2026 13:08:03 +0200 Subject: [PATCH] Kernel spike done (R-836): ESP one-shot and BootNext pass on all three boxes; watchdog fails; design with two operator questions Co-Authored-By: Claude Opus 5.5 (1M context) Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS --- .../DESIGN-kernel-lane.md | 75 +++++++++++++++++++ .../demo-felhom/G9-cleanup.txt | 7 ++ .../demo-hp/G2-watchdog-setup.txt | 3 + .../demo-hp/G9-cleanup.txt | 7 ++ .../demo-hp/r1-baseline/after.txt | 7 ++ .../demo-hp/r1-baseline/result.txt | 1 + .../demo-hp/r2-c2-good/after.txt | 7 ++ .../demo-hp/r2-c2-good/before.txt | 4 + .../demo-hp/r2-c2-good/bootid.before | 1 + .../demo-hp/r2-c2-good/pre.txt | 0 .../demo-hp/r2-c2-good/result.txt | 1 + .../demo-hp/r3-c2-plain/after.txt | 7 ++ .../demo-hp/r3-c2-plain/before.txt | 4 + .../demo-hp/r3-c2-plain/bootid.before | 1 + .../demo-hp/r3-c2-plain/pre.txt | 0 .../demo-hp/r3-c2-plain/result.txt | 1 + .../demo-hp/r4-c2-panic/after.txt | 7 ++ .../demo-hp/r4-c2-panic/before.txt | 4 + .../demo-hp/r4-c2-panic/bootid.before | 1 + .../demo-hp/r4-c2-panic/pre.txt | 0 .../demo-hp/r4-c2-panic/result.txt | 1 + .../demo-hp/r5-c1-good/after.txt | 7 ++ .../demo-hp/r5-c1-good/before.txt | 5 ++ .../demo-hp/r5-c1-good/bootid.before | 1 + .../demo-hp/r5-c1-good/pre.txt | 0 .../demo-hp/r5-c1-good/result.txt | 1 + .../demo-hp/r6-c1-panic/after.txt | 7 ++ .../demo-hp/r6-c1-panic/before.txt | 5 ++ .../demo-hp/r6-c1-panic/bootid.before | 1 + .../demo-hp/r6-c1-panic/pre.txt | 0 .../demo-hp/r6-c1-panic/result.txt | 1 + .../demo-hp/r7-c3-freeze/after-powercycle.txt | 10 +++ .../demo-hp/r7-c3-freeze/before.txt | 4 + .../demo-hp/r7-c3-freeze/bootid.before | 1 + .../demo-hp/r7-c3-freeze/pre.txt | 0 .../demo-hp/r7-c3-freeze/result.txt | 1 + .../tester1/G9-cleanup.txt | 7 ++ .../tools/spike-cleanup.sh | 10 +++ documentation/backlog/OPEN-ITEMS.md | 2 +- 39 files changed, 201 insertions(+), 1 deletion(-) create mode 100644 documentation/audits/kernel-spike-2026-10-07/DESIGN-kernel-lane.md create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-felhom/G9-cleanup.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/G2-watchdog-setup.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/G9-cleanup.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r1-baseline/after.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r1-baseline/result.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/after.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/before.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/bootid.before create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/pre.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/result.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/after.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/before.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/bootid.before create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/pre.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/result.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/after.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/before.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/bootid.before create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/pre.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/result.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/after.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/before.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/bootid.before create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/pre.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/result.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/after.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/before.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/bootid.before create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/pre.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/result.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/after-powercycle.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/before.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/bootid.before create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/pre.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/result.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/tester1/G9-cleanup.txt create mode 100644 documentation/audits/kernel-spike-2026-10-07/tools/spike-cleanup.sh diff --git a/documentation/audits/kernel-spike-2026-10-07/DESIGN-kernel-lane.md b/documentation/audits/kernel-spike-2026-10-07/DESIGN-kernel-lane.md new file mode 100644 index 00000000..03b83311 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/DESIGN-kernel-lane.md @@ -0,0 +1,75 @@ +# The kernel lane — what the spike measured, and a design (R-836, R-812 option B; `09` §3 decisions 164, 171) + +Spike 2026-10-07, operator present. Boxes: the Tester 1 box (VM 341 on the HP box, OVMF, Secure Boot off), demo-felhom +(Intel N100, Secure Boot off), demo-hp (AMD, Secure Boot ON). Kernels: 7.0.2-6 and 7.0.14-20. 24 reboots, 2 operator +power cycles. Evidence per boot in this folder (`//before.txt`, `after.txt`, `result.txt`; screenshots for +the VM). Tools: `tools/`. + +## 1. The problem, measured + +Installing a kernel makes it the GRUB default at once (R-836, seen again today on Tester 1). A new kernel that fails +before userspace then stays the default on every power cycle. `--next-boot` and `grub-reboot` are not one-shots on these +hosts: `/boot` is ext4 on LVM, and GRUB cannot clear `next_entry` there (R-836, 2026-10-04). + +## 2. What was measured today + +| Candidate | Tester 1 (VM) | demo-felhom | demo-hp (Secure Boot on) | +|---|---|---|---| +| **2 — a one-shot flag in a GRUB env block on the ESP (vfat)** | one-shot → new kernel; next boot → old; a panic → back to old by itself | same | same | +| **1 — UEFI `BootNext` to a copy of the signed loader whose `grub.cfg` names the target** | same | same | same | +| **3 — a hardware watchdog armed by systemd during the reboot** | FAIL: frozen until `qm reset` | FAIL: frozen until the operator's power cycle | FAIL: frozen until the operator's power cycle | + +- A **panic** (no init, `panic=10`) is recovered by 1 and 2 with no person: the crash boot shows in the screenshots + (Tester 1) and as a longer gap between boots (51–117 s against 15–34 s). +- A **freeze** (`panic=0`) is recovered by nothing: the machine reset during the reboot clears the watchdog timer, on + the emulated i6300ESB, Intel's TCO and AMD's SP5100 alike. Proxmox also blacklists hardware watchdog drivers by default. +- My first hang simulation (`init=` only) did NOT hang: initramfs-tools falls back to `/sbin/init`. `rdinit=` and `init=` + both missing forces the panic. Worth knowing for every future test of this kind. + +## 3. Options for the lane + +**A. Candidate 2 — the ESP flag.** A GRUB snippet reads `felhom_next` from an env file on the ESP, boots it once and +clears it. Writes nothing to firmware. Works with Secure Boot (it is plain `grub.cfg`). Costs: one `/etc/grub.d` file the +config bundle owns; the flag is written by the root wrapper. Can go wrong: an ESP that is not vfat or a GRUB without +`fat`/`save_env` — both checked once per box before the lane runs. + +**B. Candidate 1 — `BootNext`.** The firmware clears it. Costs: a second loader copy on the ESP that must follow every +shim/GRUB update, and an NVRAM write per kernel step — some boards wear or reorder NVRAM (demo-hp already holds 41 boot +entries). Can go wrong: a firmware that ignores `BootNext`. + +**C. Turn freezes into panics.** Add `softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10` to the one-shot +entry, so a lockup the kernel can detect becomes a panic that A or B recovers. NOT measured (no clean way to simulate a +lockup was tried). A true dead freeze stays a person's power cycle. + +## 4. The pick — a proposal + +**A, with C on the one-shot entry.** The new kernel boots once from the ESP flag. If the box comes back healthy (the host +health rule of `11` §8.2, the guest running), a userspace "boot good" step makes it the default. Otherwise the next +reboot returns to the old kernel by itself. A freeze still needs a person — the design says so to the household rather +than promising more. + +## 5. The night slot + +The kernel step ends the night: after the host step (`11` §8.2) and only on a night whose whole-guest backup succeeded. +The reboot then costs ~1–3 min of the household's apps (measured today: 43–181 s back, the guest starts by itself). +Ring 0 (the demo boxes) first; ring 1 by signed job after the operator's approval, as the Docker and Proxmox sets. + +## 6. First slice and its proof + +Build A in the wrapper (`kernel` layer, the ESP snippet in the bundle, the "boot good" step), red tests first (a plan +without the ESP flag refused; a kernel name outside the approved set refused; the snippet clears the flag). Live proof on +the Tester 1 box: one step to a new kernel; a forced panic falls back with no person (screenshots + boot gap). + +## 7. Questions for the operator + +1. **May the box restart at night for a kernel update, and do we tell the household?** That is a promise to users (the + design does not decide it). If you do nothing: kernels stay manual and no box restarts by itself. +2. **A frozen new kernel needs a person to switch the box off and on.** Accept that for the first customers (with the + household told what to do), or block the kernel lane until C is measured? If you do nothing: the lane is not built. + +## 8. State left on the boxes + +Every test entry, the ESP flag, the `BootNext` loader and its firmware entry, and the watchdog config are removed +(`/G9-cleanup.txt`). Tester 1 and demo-felhom: default 7.0.2-6 (running), 7.0.14-20 installed and booted healthily +in the spike. demo-hp: default 7.0.14-20 (running). VM 341's test watchdog device is removed (applies at its next start). +Also seen: after a reboot, demo-felhom answered over Tailscale only after more than 6 minutes (the LAN at once). diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-felhom/G9-cleanup.txt b/documentation/audits/kernel-spike-2026-10-07/demo-felhom/G9-cleanup.txt new file mode 100644 index 00000000..76f4a58e --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-felhom/G9-cleanup.txt @@ -0,0 +1,7 @@ +done +felhom lines in grub.cfg: 0 +GRUB_DEFAULT="gnulinux-advanced-1af1fcc6-639c-416b-a7e5-c4470d41a502>gnulinux-7.0.2-6-pve-advanced-1af1fcc6-639c-416b-a7e5-c4470d41a502" +7.0.2-6-pve +BootOrder: 0001,0003,0002,0000 +BOOT +proxmox diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/G2-watchdog-setup.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/G2-watchdog-setup.txt new file mode 100644 index 00000000..4c024bfa --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/G2-watchdog-setup.txt @@ -0,0 +1,3 @@ +lrwxrwxrwx 1 root root 9 Oct 7 12:55 /dev/felhom-hwwd -> watchdog1 +/sys/class/watchdog/watchdog0 Software Watchdog timeout=10 state=active +/sys/class/watchdog/watchdog1 SP5100 TCO timer timeout=30 state=active diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/G9-cleanup.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/G9-cleanup.txt new file mode 100644 index 00000000..d4cca10f --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/G9-cleanup.txt @@ -0,0 +1,7 @@ +done +felhom lines in grub.cfg: 0 +GRUB_DEFAULT="gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-20-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43" +7.0.14-20-pve +BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009 +BOOT +proxmox diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r1-baseline/after.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r1-baseline/after.txt new file mode 100644 index 00000000..27918db4 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r1-baseline/after.txt @@ -0,0 +1,7 @@ +7.0.14-20-pve +BOOT_IMAGE=/boot/vmlinuz-7.0.14-20-pve root=/dev/mapper/pve-root ro quiet +env: +BootCurrent: 0003 +BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B + -1 37b2e108fa0b44f985cfa026b908807e Mon 2026-10-05 09:56:46 CEST Wed 2026-10-07 12:47:43 CEST + 0 cd3ae3a17ea142119259132a38995458 Wed 2026-10-07 12:48:16 CEST Wed 2026-10-07 12:48:29 CEST diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r1-baseline/result.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r1-baseline/result.txt new file mode 100644 index 00000000..b8b83a40 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r1-baseline/result.txt @@ -0,0 +1 @@ +r1-baseline: PASS kernel=7.0.14-20-pve back=181s gap-between-boots=33s diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/after.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/after.txt new file mode 100644 index 00000000..9b262320 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/after.txt @@ -0,0 +1,7 @@ +7.0.2-6-pve +BOOT_IMAGE=/boot/vmlinuz-7.0.2-6-pve root=/dev/mapper/pve-root ro quiet panic=10 +env: felhom_next= +BootCurrent: 0003 +BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B + -1 cd3ae3a17ea142119259132a38995458 Wed 2026-10-07 12:48:16 CEST Wed 2026-10-07 12:48:41 CEST + 0 05f158b202134ce6a285a295d5a40f31 Wed 2026-10-07 12:49:15 CEST Wed 2026-10-07 12:49:26 CEST diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/before.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/before.txt new file mode 100644 index 00000000..77110ae6 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/before.txt @@ -0,0 +1,4 @@ +7.0.14-20-pve +felhom_next=felhom-spike-good +BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B +2026-10-07T10:48:31Z diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/bootid.before b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/bootid.before new file mode 100644 index 00000000..1c906949 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/bootid.before @@ -0,0 +1 @@ +cd3ae3a1-7ea1-4211-9259-132a38995458 diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/pre.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/pre.txt new file mode 100644 index 00000000..e69de29b diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/result.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/result.txt new file mode 100644 index 00000000..d554549f --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r2-c2-good/result.txt @@ -0,0 +1 @@ +r2-c2-good: PASS kernel=7.0.2-6-pve back=53s gap-between-boots=34s diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/after.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/after.txt new file mode 100644 index 00000000..54006770 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/after.txt @@ -0,0 +1,7 @@ +7.0.14-20-pve +BOOT_IMAGE=/boot/vmlinuz-7.0.14-20-pve root=/dev/mapper/pve-root ro quiet +env: felhom_next= +BootCurrent: 0003 +BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B + -1 05f158b202134ce6a285a295d5a40f31 Wed 2026-10-07 12:49:15 CEST Wed 2026-10-07 12:49:39 CEST + 0 321e492ea4e64ea9943db9a544e17700 Wed 2026-10-07 12:50:13 CEST Wed 2026-10-07 12:50:24 CEST diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/before.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/before.txt new file mode 100644 index 00000000..a7681172 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/before.txt @@ -0,0 +1,4 @@ +7.0.2-6-pve +felhom_next= +BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B +2026-10-07T10:49:27Z diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/bootid.before b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/bootid.before new file mode 100644 index 00000000..3a452f20 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/bootid.before @@ -0,0 +1 @@ +05f158b2-0213-4ce6-a285-a295d5a40f31 diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/pre.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/pre.txt new file mode 100644 index 00000000..e69de29b diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/result.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/result.txt new file mode 100644 index 00000000..62805710 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r3-c2-plain/result.txt @@ -0,0 +1 @@ +r3-c2-plain: PASS kernel=7.0.14-20-pve back=55s gap-between-boots=34s diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/after.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/after.txt new file mode 100644 index 00000000..f6993b34 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/after.txt @@ -0,0 +1,7 @@ +7.0.14-20-pve +BOOT_IMAGE=/boot/vmlinuz-7.0.14-20-pve root=/dev/mapper/pve-root ro quiet +env: felhom_next= +BootCurrent: 0003 +BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B + -1 321e492ea4e64ea9943db9a544e17700 Wed 2026-10-07 12:50:13 CEST Wed 2026-10-07 12:50:33 CEST + 0 cf9353298ce94a438c68cce1cdf5de26 Wed 2026-10-07 12:51:47 CEST Wed 2026-10-07 12:51:59 CEST diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/before.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/before.txt new file mode 100644 index 00000000..26b4867c --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/before.txt @@ -0,0 +1,4 @@ +7.0.14-20-pve +felhom_next=felhom-spike-panic +BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B +2026-10-07T10:50:25Z diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/bootid.before b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/bootid.before new file mode 100644 index 00000000..d7678d42 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/bootid.before @@ -0,0 +1 @@ +321e492e-a4e6-4ea9-943d-b9a544e17700 diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/pre.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/pre.txt new file mode 100644 index 00000000..e69de29b diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/result.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/result.txt new file mode 100644 index 00000000..faf54a26 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r4-c2-panic/result.txt @@ -0,0 +1 @@ +r4-c2-panic: PASS kernel=7.0.14-20-pve back=91s gap-between-boots=74s diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/after.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/after.txt new file mode 100644 index 00000000..bd88d9ed --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/after.txt @@ -0,0 +1,7 @@ +7.0.2-6-pve +BOOT_IMAGE=/boot/vmlinuz-7.0.2-6-pve root=/dev/mapper/pve-root ro quiet panic=10 +env: felhom_next= +BootCurrent: 0000 +BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,0000,000C,000A + -1 cf9353298ce94a438c68cce1cdf5de26 Wed 2026-10-07 12:51:47 CEST Wed 2026-10-07 12:52:11 CEST + 0 18039e256a30463485e70b98c856d620 Wed 2026-10-07 12:53:06 CEST Wed 2026-10-07 12:53:20 CEST diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/before.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/before.txt new file mode 100644 index 00000000..decbc243 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/before.txt @@ -0,0 +1,5 @@ +7.0.14-20-pve +felhom_next= +BootNext: 0000 +BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B +2026-10-07T10:52:01Z diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/bootid.before b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/bootid.before new file mode 100644 index 00000000..c0c7090e --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/bootid.before @@ -0,0 +1 @@ +cf935329-8ce9-4a43-8c68-cce1cdf5de26 diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/pre.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/pre.txt new file mode 100644 index 00000000..e69de29b diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/result.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/result.txt new file mode 100644 index 00000000..824bf4b5 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r5-c1-good/result.txt @@ -0,0 +1 @@ +r5-c1-good: PASS kernel=7.0.2-6-pve back=75s gap-between-boots=55s diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/after.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/after.txt new file mode 100644 index 00000000..5c037cb2 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/after.txt @@ -0,0 +1,7 @@ +7.0.14-20-pve +BOOT_IMAGE=/boot/vmlinuz-7.0.14-20-pve root=/dev/mapper/pve-root ro quiet +env: felhom_next= +BootCurrent: 0003 +BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,0004,000A + -1 18039e256a30463485e70b98c856d620 Wed 2026-10-07 12:53:06 CEST Wed 2026-10-07 12:53:29 CEST + 0 982d4e0ad5ef47a8a7d0b5f110e9ad13 Wed 2026-10-07 12:55:26 CEST Wed 2026-10-07 12:55:37 CEST diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/before.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/before.txt new file mode 100644 index 00000000..530d18a7 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/before.txt @@ -0,0 +1,5 @@ +7.0.2-6-pve +felhom_next= +BootNext: 0000 +BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,0000,000C,000A +2026-10-07T10:53:21Z diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/bootid.before b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/bootid.before new file mode 100644 index 00000000..fc4a97e9 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/bootid.before @@ -0,0 +1 @@ +18039e25-6a30-4634-85e7-0b98c856d620 diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/pre.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/pre.txt new file mode 100644 index 00000000..e69de29b diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/result.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/result.txt new file mode 100644 index 00000000..40f10c5d --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r6-c1-panic/result.txt @@ -0,0 +1 @@ +r6-c1-panic: PASS kernel=7.0.14-20-pve back=134s gap-between-boots=117s diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/after-powercycle.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/after-powercycle.txt new file mode 100644 index 00000000..70aa8dd0 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/after-powercycle.txt @@ -0,0 +1,10 @@ +7.0.14-20-pve +env: felhom_next= + -1 982d4e0ad5ef47a8a7d0b5f110e9ad13 Wed 2026-10-07 12:55:26 CEST Wed 2026-10-07 12:57:53 CEST + 0 8ea2cdffd1664af69937156f4ee659a5 Wed 2026-10-07 13:05:28 CEST Wed 2026-10-07 13:06:04 CEST +/sys/class/watchdog/watchdog0 Software Watchdog state=active + 341 night1004-tester1 running 8192 200.00 1816 +VMID Status Lock Name +9201 running demo-hp +9202 running demo-hp-scratch +9401 stopped upgrade-harness diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/before.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/before.txt new file mode 100644 index 00000000..70da3bd9 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/before.txt @@ -0,0 +1,4 @@ +7.0.14-20-pve +felhom_next=felhom-spike-freeze +BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,0004,000A +2026-10-07T10:56:00Z diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/bootid.before b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/bootid.before new file mode 100644 index 00000000..365ed712 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/bootid.before @@ -0,0 +1 @@ +982d4e0a-d5ef-47a8-a7d0-b5f110e9ad13 diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/pre.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/pre.txt new file mode 100644 index 00000000..e69de29b diff --git a/documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/result.txt b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/result.txt new file mode 100644 index 00000000..286236ec --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/demo-hp/r7-c3-freeze/result.txt @@ -0,0 +1 @@ +r7-c3-freeze: FAIL — sp5100_tco armed by systemd (RebootWatchdogSec=90s) did NOT reset the frozen kernel; dead (no ping) 6 min after the reboot until the operator's power cycle; then the default 7.0.14-20 diff --git a/documentation/audits/kernel-spike-2026-10-07/tester1/G9-cleanup.txt b/documentation/audits/kernel-spike-2026-10-07/tester1/G9-cleanup.txt new file mode 100644 index 00000000..83938c02 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/tester1/G9-cleanup.txt @@ -0,0 +1,7 @@ +done +felhom lines in grub.cfg: 0 +GRUB_DEFAULT="gnulinux-advanced-e7c978ee-70aa-454b-a5bb-53dc1d19f16e>gnulinux-7.0.2-6-pve-advanced-e7c978ee-70aa-454b-a5bb-53dc1d19f16e" +7.0.2-6-pve +BootOrder: 0004,0003,0000,0001 +BOOT +proxmox diff --git a/documentation/audits/kernel-spike-2026-10-07/tools/spike-cleanup.sh b/documentation/audits/kernel-spike-2026-10-07/tools/spike-cleanup.sh new file mode 100644 index 00000000..b2ca8e01 --- /dev/null +++ b/documentation/audits/kernel-spike-2026-10-07/tools/spike-cleanup.sh @@ -0,0 +1,10 @@ +#!/bin/bash +# End of the spike: remove the test entries, the ESP one-shot and the BootNext loader + firmware entry. KEEP the explicit +# GRUB_DEFAULT the setup wrote (the kernel this box runs and booted healthily); the original file stays as *.felhom-spike-orig. +set -uo pipefail +rm -f /etc/grub.d/01_felhom_oneshot /etc/grub.d/41_felhom_spike /boot/efi/EFI/proxmox/felhom-oneshot.env +bash /root/spike-bootnext-undo.sh >/dev/null 2>&1 +update-grub 2>&1 | tail -1 +echo "felhom lines in grub.cfg: $(grep -c felhom /boot/grub/grub.cfg)" +grep -m1 '^GRUB_DEFAULT' /etc/default/grub; uname -r +efibootmgr | grep -E 'BootOrder|felhom|BootNext' | cut -c1-80; ls /boot/efi/EFI diff --git a/documentation/backlog/OPEN-ITEMS.md b/documentation/backlog/OPEN-ITEMS.md index f4dac662..30ed9760 100644 --- a/documentation/backlog/OPEN-ITEMS.md +++ b/documentation/backlog/OPEN-ITEMS.md @@ -217,7 +217,7 @@ stopping line that lies. | ID | Category | Sev | What | State | Blocked on | Next action | Owner | |---|---|---|---|---|---|---|---| | **R-812** | Box system & updates | P2 | **[P2] A box never receives operating-system security updates — not the Proxmox host, not the guest's Debian, not its Docker engine.** SEARCHED 2026-10-03 (read-only): `felhom-controller`, `felhom-agent`, `app-catalog-felhom.eu` and `felhom.eu` hold no `apt-get upgrade`, `apt full-upgrade`, `unattended-upgrades`, `pveupgrade` or `needrestart` that runs on a box. The installer aligns the host's Proxmox repositories to no-subscription *"so the box can pull security updates"* and then says plainly *"No upgrades are run"* (`scripts/felhom-host-install.sh:2133-2136`). The guest's Docker engine is installed when the golden is BAKED (`felhom-agent/configs/build-golden.sh:124-125`), so a fresh install gets that week's engine and an installed box keeps it forever. The only `apt full-upgrade` in the project is a by-hand step for the off-site endpoint ep0 (`documentation/runbooks/offsite-endpoint.md:41`), not a box. App images ARE updated (the update arc); the layer under them is not. The intention, with its scope, is **R-808** in `ROADMAP.md`. | **NARROWED 2026-10-04 (evening) — the guest's DOCKER engine slow lane is BUILT and proven live (agent v0.142.0, hub v0.132.0; `11` §5.8): live-restore on everywhere, ring 0 steps under a root-owned mark, ring 1 and undo only by a signed job the wrapper re-verifies; the operator approves each engine set on the System page. LEFT: the kernel lane (R-836); existing boxes (R-840).** Earlier: NARROWED 2026-10-04 (afternoon) — the HOST's Debian fast lane is BUILT and proven live too (agent v0.141.1, hub v0.131.1; `11` §8.2, `audits/os-host-lane-2026-10-04/`): appliances only, never kernel/boot/firmware, after a healthy guest step; fleet view and four alarms (§8.3). LEFT: the Docker and kernel slow lanes (R-836); existing boxes (R-840).** Earlier: NARROWED AGAIN 2026-10-04 (day) — the GUEST's Debian fast lane is BUILT and proven live (agent v0.140.0, hub v0.130.0, installer 1.29.0; `11` §8.1, `audits/os-guest-lane-2026-10-04/`). LEFT: the host, Docker and kernel lanes; the undo (R-842); existing boxes (R-840).** Earlier: NARROWED 2026-10-04 — the SPIKE is done (`11` §7.1, corrections C1–C12, `audits/os-updates-spike-2026-10-04/`); no product code yet. LEFT: the build steps of `11` §8, each with the operator's go; the §5.3 snapshot question is in STATUS; preconditions R-835, R-836, R-837.** **2026-10-06 night: design written** (`audits/night-burndown-2026-10-06/design-R-812.md`) — what is left is `11` §8 step 6: the host's Proxmox-origin packages and its kernel are never updated (read tonight: demo-hp 77 and demo-felhom 78 Proxmox packages pending on the 2026-10-05 lists, `pve-manager` 9.2.2 → 9.2.21; demo-felhom still runs its install-time kernel 7.0.2-6). Pick: a `pve` slow lane for the Proxmox packages without the kernel, approved per set like the Docker engine; the kernel stays an operator-run step until R-836's fallback is measured. Proposed split: close R-812 when that lane ships; R-836 carries the kernel. Two operator questions in the design. **2026-10-07 07:58: `09` §3 decisions 163–164 — option A (the Proxmox package lane, per-set approval) YES; option B (the kernel lane) YES before the first paying customer, starting with a reboot spike carried by R-836.** | — | — | CC + operator | -| **R-836** | Box system & updates | P2 | **A new host kernel that hangs before userspace stays the GRUB default: `--next-boot` is not a one-shot on these hosts.** MEASURED 2026-10-04 on demo-hp (operator's word, 2 reboots): both demo hosts boot UEFI + GRUB without proxmox-boot-tool ESPs; installing a kernel makes it the default at once; `kernel pin --next-boot` writes an ordinary `GRUB_DEFAULT`, and `proxmox-boot-cleanup.service` clears it only after a boot reaches userspace. With the old kernel pinned FIRST, the fallback after a good boot worked (new 60 s, old 76 s). READ FROM THE CODE, not measured: a hang leaves the new kernel default on every power cycle. Only `softdog` runs (useless before userspace); demo-hp's `sp5100_tco` ships unloaded, untested. Fix direction for the slow lane: GRUB's own one-shot (`GRUB_DEFAULT=saved` + `grub-reboot`) with the old kernel saved — to be measured, including Secure Boot (ON on demo-hp). `audits/os-updates-spike-2026-10-04/partH/` | **NARROWED 2026-10-04 (os-host-lane Part E, operator's word before each of 2 reboots) — GRUB's own one-shot is NOT a one-shot here either.** `GRUB_DEFAULT=saved` (old kernel saved) + `grub-reboot `: boot 1 → new kernel, **Secure Boot ON and fine**; but GRUB could not clear `next_entry` (`grub-reboot` itself warns: *environment block on lvm device … will remain the default until manually cleared*; `/boot` is ext4 on LVM `pve-root`), so boot 2 (no command) → **the new kernel again**. `kernel.panic = 0`: a panic leaves the host stopped (R-851). `sp5100_tco` LOADS and answers (`SP5100 TCO timer`, 60 s, inactive, nowayout 0; read from sysfs, never opened, unloaded) — a hardware watchdog exists on demo-hp, but nothing arms it before userspace. demo-hp left on 7.0.14-20 with that as the saved default. **LEFT (fix direction):** a GRUB env block GRUB can write (on the ESP, vfat) or a userspace "boot good" step that rewrites the default, plus arming `sp5100_tco`; to be measured before the kernel slow lane. `audits/os-host-lane-2026-10-04/partE/` **READY — owner: CC + operator (reboots).** **2026-10-07 07:58: raised P3 → P2 (`09` §3 decision 164 — the kernel lane before the first paying customer); the spike runs with the operator present.** | — | — | CC | +| **R-836** | Box system & updates | P2 | **A new host kernel that hangs before userspace stays the GRUB default: `--next-boot` is not a one-shot on these hosts.** MEASURED 2026-10-04 on demo-hp (operator's word, 2 reboots): both demo hosts boot UEFI + GRUB without proxmox-boot-tool ESPs; installing a kernel makes it the default at once; `kernel pin --next-boot` writes an ordinary `GRUB_DEFAULT`, and `proxmox-boot-cleanup.service` clears it only after a boot reaches userspace. With the old kernel pinned FIRST, the fallback after a good boot worked (new 60 s, old 76 s). READ FROM THE CODE, not measured: a hang leaves the new kernel default on every power cycle. Only `softdog` runs (useless before userspace); demo-hp's `sp5100_tco` ships unloaded, untested. Fix direction for the slow lane: GRUB's own one-shot (`GRUB_DEFAULT=saved` + `grub-reboot`) with the old kernel saved — to be measured, including Secure Boot (ON on demo-hp). `audits/os-updates-spike-2026-10-04/partH/` | **NARROWED 2026-10-04 (os-host-lane Part E, operator's word before each of 2 reboots) — GRUB's own one-shot is NOT a one-shot here either.** `GRUB_DEFAULT=saved` (old kernel saved) + `grub-reboot `: boot 1 → new kernel, **Secure Boot ON and fine**; but GRUB could not clear `next_entry` (`grub-reboot` itself warns: *environment block on lvm device … will remain the default until manually cleared*; `/boot` is ext4 on LVM `pve-root`), so boot 2 (no command) → **the new kernel again**. `kernel.panic = 0`: a panic leaves the host stopped (R-851). `sp5100_tco` LOADS and answers (`SP5100 TCO timer`, 60 s, inactive, nowayout 0; read from sysfs, never opened, unloaded) — a hardware watchdog exists on demo-hp, but nothing arms it before userspace. demo-hp left on 7.0.14-20 with that as the saved default. **LEFT (fix direction):** a GRUB env block GRUB can write (on the ESP, vfat) or a userspace "boot good" step that rewrites the default, plus arming `sp5100_tco`; to be measured before the kernel slow lane. `audits/os-host-lane-2026-10-04/partE/` **READY — owner: CC + operator (reboots).** **2026-10-07 07:58: raised P3 → P2 (`09` §3 decision 164 — the kernel lane before the first paying customer); the spike runs with the operator present.** **2026-10-07 SPIKE DONE (operator present; 24 reboots, 2 power cycles; `audits/kernel-spike-2026-10-07/DESIGN-kernel-lane.md`).** On Tester 1 (VM), demo-felhom and demo-hp (Secure Boot on): **a one-shot flag in a GRUB env block on the ESP (vfat) WORKS** — new kernel once, then the old one; a panicking new kernel (`panic=10`) falls back with no person. **UEFI `BootNext` WORKS too.** **A hardware watchdog armed by systemd during the reboot FAILS on all three** (i6300ESB, Intel TCO, AMD SP5100): the reset clears the timer, so a FROZEN new kernel needs a power cycle. Pick: the ESP flag + lockup-to-panic kernel options (unmeasured). Two operator questions in the design (night restarts and the household; a freeze needs a person). Boxes left clean on a healthily booted default. | — | — | CC | | **R-862** | Box system & updates | P3 | **Tester 2 cannot take the config bundle until one by-hand bootstrap is done: its `felhom-os-apply` (agent 0.142.0) predates the bundle mode, and no signed job can write a root file on it.** FOUND 2026-10-04 (R-840 build, Part C): the route reaches every box installed from installer 1.31.0 on, and the demo boxes (bootstrapped by CC); Tester 2 has every root file of agent 0.142.0 (installer 1.30.0) and lacks only the R-858 wrapper fix, which matters only for a Docker step it gets solely from a signed job. CC has no route to Tester 2 (its door admits only the operator's WireGuard peer; `felhom-op` cannot become root). The steps: `runbooks/config-bundle.md` "Tester 2". Then CC signs the bundle and reads it back. | **WAITING-ON-OPERATOR** — the operator said (2026-10-04 ~19:05) he will try through his tunnel **2026-10-07 07:58 (`09` §3 decision 170): the operator believes Tester 2 never set up the off-site backup (no escrow); the laptop may stay off for days; the tester plans to add an HDD.** **2026-10-07 (Part H, read only):** in Tester 2's last 10 notifications (2026-10-05 09:16 → 2026-10-07 06:58) the household got 0 mails, the operator 10 (host_down ×9, expected_dbdump_missed ×1); control: the household channel shows on the other three customers. So the household is not mailed daily while the box is off; no row (`audits/day-2026-10-07/H/`). | the operator's WireGuard tunnel | the operator runs the three bootstrap commands; CC sends the bundle | operator | | **R-35** | Box system & updates | P3 | **Config-apply should not end the customer's session.** The offsite config push bumped `config_version` 10→11 at 16:54:58 and the controller self-restarted (container `StartedAt` 16:54:59Z, back up 16:55:02); in-memory sessions died with it and **customer zero was force-logged-out mid-flow**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size S, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **2026-10-05 (burn-down night): NEEDS A DESIGN.** Hot-apply needs a per-setting ruling; persisting sessions puts login tokens on disk (and into the whole-guest archive, `07` §5). Next: the operator's pick between them. | — | Direction: **hot-apply the offbox target** (no restart for a config the running process can adopt), or **persist sessions** across restart. The restart itself is by design — the collateral is not. Evidence `controller-log-full.txt` | CC | | **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC |