felhom.eu/documentation/audits/night-burndown-2026-10-06/design-R-812.md

R-812 — what is left of OS updates: the Proxmox packages and the kernel — design proposal (burn-down night 2026-10-06, no code)

Baselines read: felhom-agent 74b5eae, felhom.eu 8e2dc204. Architecture: 11-os-updates.md (§1, §3, §5.2, §5.5, §5.6, §5.9, §7.1, §8). Related rows: R-836 (the kernel's fallback), R-808 (the intention, ROADMAP.md).

1. The problem

11 §8 steps 1–5 are BUILT: the guest's Debian, the host's Debian, the fleet view, the Docker engine, the root files (R-840, closed). Step 6 is not: the host's Proxmox packages and its kernel are never updated on any box. The host fast lane takes only Debian-origin packages, so everything from the Proxmox repository waits for ever — including packages with ordinary names (zfsutils-linux, shim-signed, corosync, ceph-common, amd64-microcode, 11 C3).

Measured tonight (read only; apt list --upgradable, pveversion, dpkg -l, ls /boot; no apt update — the package lists are from 2026-10-05, the last daily refresh):

• demo-hp: pve-manager 9.2.2 → 9.2.21 pending; 77 packages pending, all from the Proxmox repository (the Debian lane has taken the rest): qemu-server 9.1.15 → 9.2.10, pve-container 6.1.10 → 6.1.14, proxmox-backup-client 4.2.0 → 4.2.7, zfsutils-linux 2.4.2 → 2.4.4, shim-signed 1.48 → 1.51, pve-firmware, corosync, ceph-*, frr, amd64-microcode, libpve-*. Kernel: runs 7.0.14-20 (installed BY HAND in the 2026-10-04 spike), 7.0.2-6 kept.
• demo-felhom: pve-manager 9.2.2; 78 pending; runs kernel 7.0.2-6, the install-time kernel — 7.0.14-20 is in the repository and not installed.
• So a box keeps its install-time Proxmox and kernel. Proxmox security fixes (for example in pve-manager's web API, in qemu-server, in the Secure Boot loader shim-signed) never arrive. The spike counted 79 Proxmox packages "not covered" on the host (11 §7.1).

2. What the code does today (read in source)

• The fast-lane origin rule: felhom-agent/configs/felhom-os-apply:63 FAST_ORIGINS = ("Debian", "Debian-Security"); a plan package of any other origin is refused R2 (:491-492, and in the simulation :1164 origin_ok).
• The host's name rule on top: :91-92 HOST_SLOW_RE (linux-*, proxmox-kernel*, pve-kernel*, pve-firmware, firmware-*, grub*, shim*, systemd-boot*, *-microcode, efibootmgr) → refusal R14 (:493-494, :1234-1235); the pending selection skips them (:1136).
• Layers and lanes: :437-442 — guest/host are lane fast only; docker is the only slow layer. There is no host slow layer.
• Tests pinning this: configs/test_felhom_os_apply.py:417 test_R2_non_debian_origin_in_the_plan, :422 test_R2_non_debian_origin_in_the_simulation, :584/:590 test_R14_kernel_package_*, :639 test_pending_fast_skips_proxmox_docker_and_kernel (a pending pve-manager 9.2.21 from „Proxmox Debian Repository" is NOT taken).
• What a Proxmox step restarts (spike partH/H5-proxmox-simulate.txt, read from the maintainer scripts, not run): pve-manager → pvedaemon, pveproxy, pvescheduler, pvestatd, spiceproxy; pve-cluster → pmxcfs (/etc/pve gone for seconds — the agent's API calls fail meanwhile); qemu-server, pve-firewall, pve-ha-*, corosync, chrony, zfs-zed. No guest is stopped by these scripts (read, not measured). One new package (proxmox-firewall-data).
• The kernel (11 §5.6, R-836, measured on demo-hp with the operator's word): installing a kernel makes it the GRUB default at once; --next-boot and grub-reboot are NOT one-shots here (/boot is ext4 on LVM; GRUB cannot write its env block); a kernel that hangs before userspace stays the default on every power cycle. sp5100_tco exists on demo-hp, blacklisted, never armed. demo-felhom (Intel N100) has no watchdog measured.

3. Options

A. A host slow lane for the Proxmox USERSPACE packages, no kernel, no reboot.

• What: a new layer pve (lane slow) in the wrapper: origin „Proxmox Debian Repository" only, still never a name in HOST_SLOW_RE (kernel, boot, firmware, microcode stay out), no removals, no new package except a named allow-list (the proxmox-firewall-data kind). Approval like the Docker engine set (11 §5.8): the operator approves one SET per box generation on the System page after ring 0 ran it; ring 1 only by signed job. Runs in the night leg after the host Debian step, under the same heavy-op gate. Health = the host rule (HostHealthVerdict) + pveversion reports the new version.
• Cost: wrapper layer + refusals + tests; hub candidate/approval/System page (copies the Docker set's code); ~1.5 days.
• Can go wrong: pmxcfs restart while the agent writes /etc/pve (the gate already excludes backups/restore-tests; the agent's own reconcile must wait too); a Proxmox point release that needs a newer kernel (proxmox-ve depends on proxmox-default-kernel — the plan must not pull the kernel in: the simulation refusal R14 already catches it); undo is only „install the previous version" (Proxmox keeps 30–66 versions, 11 C2 — good).
• Brings the 77–78 pending packages, minus the boot-chain ones, to every box. Leaves shim-signed, pve-firmware, microcode and the kernel out.

B. The kernel lane, with a fallback that works on these hosts — measured first.

• What: install the kernel (plus shim-signed, pve-firmware, microcode, which also only act at boot), keep the old one as the permanent default, boot the new one ONCE, and make it the default only from userspace after a healthy boot. The one-shot needs one of: (1) UEFI BootNext with a second boot entry whose GRUB config defaults to the new kernel — the firmware clears BootNext itself on use, so a hang falls back on the next power cycle; (2) a GRUB env block on the ESP (vfat, writable by GRUB); (3) arming a hardware watchdog early (sp5100_tco on demo-hp) so a hang power-cycles into the old default. None is measured.
• Cost: a spike with ≥4 reboots per box on both demo hosts (Secure Boot ON on demo-hp, OFF on demo-felhom; AMD vs Intel), then the lane: ~3–5 days. Every customer box restarts all its apps once per kernel (≈1–2 min at night).
• Can go wrong: firmware that ignores BootNext (seen on consumer boards); Secure Boot refusing a second entry; a hang with no watchdog still needs a person — it is a dead box at a household. Telling households that the box may restart at night is a promise to users (11 §5.7) — the operator's.

C. The kernel only by an operator-present maintenance step.

• What: no automatic kernel lane. The System page shows „kernel behind / reboot needed"; the operator runs the step per box by a signed job at a time they choose, ready to power-cycle (or ask the household to).
• Cost: small (a signed job and a runbook), but a person per box per kernel; does not scale past a handful of boxes.
• Can go wrong: kernels are never installed because nobody schedules them — the state today with a button.

4. The pick — PROPOSAL for the operator, not a decision

A now, C for the kernel until B is measured. A closes the biggest gap (78 Proxmox packages, the web API and qemu/lxc tooling) with no reboot and the same approval shape the operator already uses for Docker. The kernel stays a separate, operator-scheduled act (C) until R-836's fallback is measured on both demo hosts; then B replaces C.

R-812 should split. Close R-812 (P2) when A ships to ring 0 and ring 1, with R-836 (kernel fallback, P3) carrying the kernel. The kernel's urgency then becomes its own question: a P2 row „kernel security fixes reach a box" only if the operator wants it before the first paying customer (question 2).

5. First slice and its proof

• Build: wrapper layer pve lane slow — origin „Proxmox Debian Repository" only; R14 still applies; R2 for any other origin; refusal for a removal or an unlisted new package; appliance only (R12's root-owned install record).
• Red test first (must FAIL on today's code): a host plan with pve-manager 9.2.21 origin „Proxmox Debian Repository", layer pve, lane slow → today refused R12 („layer is not guest, host or docker"); after: installed. Companion tests that must stay refused: the same plan carrying proxmox-kernel-7.0 (R14) or shim-signed (R14); a Debian package in a pve plan (R2); test_pending_fast_skips_proxmox_docker_and_kernel unchanged.
• Live proof on demo-felhom (ring 0, Tier 0, its Proxmox is the install-time one): one approved set of the pending Proxmox packages, no kernel. Positive observable: pveversion reads 9.2.21; the host health rule passes; the guest kept running (container StartedAt unchanged). Control from a different channel: the hub's System page row for the box, and an app on the guest answering 200 every 5 s through the step. No reboot needed, so no operator word for a reboot; the operator's word is the approval click, as for Docker. Evidence off the box at the end.

6. Open questions for the operator

1. May Proxmox's own packages (no kernel) update by night, after you approve each set on the System page, as the Docker engine does? Pick: yes (option A). If you do nothing: every box keeps its install-time Proxmox; 78 packages behind on the demo boxes today, and growing.
2. Must kernel security fixes reach customer boxes before the first paying customer? If yes, B is a ~1-week arc with reboots on both demo boxes (your word before each). Pick: no — the kernel by your hand (C) until B is measured. If you do nothing: kernels stay at the install-time version; a kernel hole stays open until you run the step by hand.