felhom.eu/documentation/audits/os-updates-spike-2026-10-04

OS-updates spike — 2026-10-04 (answers architecture/11-os-updates.md §7 Q1–Q10)

Venues: demo-hp host + its 9201 + scratch 9202 (changes: 9202 Part G, demo-hp host Part H); demo-felhom host + 9201 (read only). Apt indexes for every read-only step went to a throwaway directory (scripts/os-survey.sh), so no box's lists or sources changed. Hosts are CEST, guests UTC.

Folder	What
partE/	read-only surveys of five systems (*-host.txt, *-guest.txt), host package-set diff
partF/	apt-cache madison (host + guest), Debian DSA re-announcement analysis, snapshot.debian.org probe
partG/	9202: Debian update, undo, killed update, Docker step ×3 — SUMMARY.md
partH/	demo-hp host: package lists before/after, Debian-lane run, one-package undo, kernel install + 2 reboots, Proxmox simulation
partI/	sample approved list (sample-approved-list.tsv) and its read-only simulation on demo-felhom
scripts/	every helper used, as run

The answers

Q1 — exact Debian version a week later. The live Debian archives keep two versions of a package at most: the point-release version in trixie (main) and the newest in trixie-security — never the intermediate ones (partF/madison-demo-hp-host.txt: openssl installed u1, main u2, security u3; u1 is gone). DSA history, 2026-07-04..10-04 (partF/dsa-supersede-analysis.txt): 167 trixie advisories; a second advisory for the same package within 2 days: 3 (1.8 %), all chromium/webkit2gtk — none of them on a box. Restricted to the 517 source packages installed on a box: 20 advisories over 16 packages, 0 re-announced within 2, 7 or 14 days. snapshot.debian.org is reachable from a box: a dated apt-get update takes 2.3–3.0 s (10.1 MB), and the exact gone version openssl 3.5.6-1~deb13u1 downloads in 2.0 s (partF/snapshot-debian-org.md; https needs ca-certificates, present on boxes, absent in the bare debian:trixie image).

Q2 — Proxmox and Docker keep old versions. Yes, many: pve-manager 66, qemu-server 58, proxmox-kernel-7.0 33, pve-container 32, docker-ce 46, containerd.io 18 versions listed. Debian: two at most (Q1). One host undo measured: rsync back to its pre-update 3.4.1+ds1-5+deb13u2 → E: Version … was not found; libpng16-16t64 back to 1.6.48-1+deb13u5 worked because that version is the point-release one in main (partH/H3-undo-one-package.txt).

Q3 — Docker engine update. Without live-restore (today's baked setting): all 6 containers restart, the app's front door is silent 26.5–30 s, everything healthy +42–45 s. With live-restore: 0 restarts, no gap, also across a containerd change. systemctl reload turns it ON; it does NOT turn it off; and a restart that turns it off stops every running container and starts none of them (partG/SUMMARY.md).

Q4 — kernel. Both demo hosts: UEFI, GRUB (no proxmox-boot-tool ESPs), root on LVM ext4; demo-hp has Secure Boot ON, demo-felhom OFF (setup mode). Installing a kernel makes it the GRUB default at once (GRUB_DEFAULT=0, newest first). proxmox-boot-tool kernel pin <ver> --next-boot on GRUB writes a normal GRUB_DEFAULT + update-grub (GRUB's own one-shot next_entry stays empty); proxmox-boot-cleanup.service removes it after a boot reaches userspace. Measured on demo-hp with the operator's word: old kernel pinned permanently first, new pinned for next boot → reboot 1 came up on 7.0.14-20-pve (60 s, Secure Boot on, all healthy); reboot 2 came back on 7.0.2-6-pve (76 s). So the fallback works after a SUCCESSFUL boot; read from the code (not measured), a kernel that hangs before userspace stays the default. Watchdog: only softdog is loaded (by watchdog-mux); demo-hp has an AMD FCH (SMBus 00:14.0) and the sp5100_tco module ships with the kernel, not loaded. A softdog cannot rescue a kernel that never boots.

Q5 — interrupted apt. Killed after 15 unpacks: 5 packages iU, 4 triggers pending. It does NOT recover by itself — the next apt-get install refuses (Unmet dependencies. Try 'apt --fix-broken install'). dpkg --configure -a (1.4 s) + apt-get -f install (3.9 s) repaired it; the rest applied in 14.1 s. Containers stayed up throughout.

Q6 — how far behind. Hosts: 188 pending each (80/79 Proxmox, 66 Debian point release, 27 both, 15 security, +1 tailscale on demo-felhom); guests: 59 (31 point, 15 both, 7 security, 6 Docker) on both 9201s, 54 on 9202. Installed: host 746/747 packages, guest 278. Each guest runs a DIFFERENT Docker (29.7.1, 29.7.2, 29.8.0; golden 0.290.0 bakes 29.8.2). Catching up: guest Debian 24.0 s / 38.1 MB; host Debian 59.7 s / 76 MB; the kernel 47 s / 131 MB.

Q7 — what restarts. Guest Debian run: postfix, journald, networkd restarted by their scripts; dockerd, containerd, sshd, dbus, logind, cron, dhclient keep the OLD libc until restarted. Host Debian run: dnsmasq, postfix, journald restarted; systemd (PID 1), lxc-start, pveproxy, pvedaemon, pvestatd, pvescheduler, watchdog-mux, sshd, zed, chronyd keep the old libc. Proxmox packages (simulated, from their own maintainer scripts): pve-manager → pvedaemon, pveproxy, pvestatd, pvescheduler, spiceproxy; pve-cluster → pmxcfs; corosync; chrony; qemu-server → qmeventd; pve-firewall; pve-ha-manager → lrm, crm; zfs-zed.

Q8 — the night window. Guest (UTC, demo-hp, W = 02:30): db-dump 02:30, tier-2 03:30, off-site ~04:15, whole-guest gate [04:30, 08:30), controller self-update 04:30, offsite-integrity 06:00. Host (CEST): apt-daily and apt-daily-upgrade timers are enabled but install nothing (no unattended-upgrades, no APT::Periodic); pve-daily-update refreshes the lists daily; restore-tests run at any hour (demo-felhom 10:38 daily, demo-hp 16:43 and 22:46) — they are cadence-driven, not windowed; agent updates arrive by signed job at any hour.

Q9 — cloudflared. Not on the host: a container in the guest, cloudflare/cloudflared:2026.6.0, pinned in the controller (internal/infra/infra.go:26) since 2026-06-11 (v0.41.0) and baked into the golden. Upstream is at 2026.9.3 (2026-09-24). It moves only when someone edits the pin.

Q10 — Proxmox enterprise repository (proxmox.com pricing page, 2026-10-04, net, per CPU socket per year): Community €120, Basic €370, Standard €550, Premium €1,100. Every tier includes the Enterprise Repository; Community has no support tickets. Record only — a money decision.

Teardown

• 9202: Debian fully updated, Docker 29.8.2 / containerd 2.3.6 (= golden 0.290.0), daemon.json byte-identical to the baked one, live-restore false, all 6 containers healthy. Its backup in /mnt/hdd_1/dump-9202-spike deleted.
• demo-hp host: 108 Debian packages updated + proxmox-kernel-7.0.14-20-pve-signed installed (110 package changes, partH/H-final-host-packages-after.tsv); running 7.0.2-6-pve, next boot 7.0.14-20-pve (no pins left). Proxmox packages NOT updated (78 pending).
• Helper files removed from both hosts and all guests; no throwaway image or container left.