# OS-updates spike — 2026-10-04 (answers `architecture/11-os-updates.md` §7 Q1–Q10)

Venues: demo-hp host + its 9201 + scratch 9202 (changes: 9202 Part G, demo-hp host Part H); demo-felhom host + 9201
(read only). Apt indexes for every read-only step went to a throwaway directory (`scripts/os-survey.sh`), so no
box's lists or sources changed. Hosts are CEST, guests UTC.

| Folder | What |
|---|---|
| `partE/` | read-only surveys of five systems (`*-host.txt`, `*-guest.txt`), host package-set diff |
| `partF/` | `apt-cache madison` (host + guest), Debian DSA re-announcement analysis, snapshot.debian.org probe |
| `partG/` | 9202: Debian update, undo, killed update, Docker step ×3 — `SUMMARY.md` |
| `partH/` | demo-hp host: package lists before/after, Debian-lane run, one-package undo, kernel install + 2 reboots, Proxmox simulation |
| `partI/` | sample approved list (`sample-approved-list.tsv`) and its read-only simulation on demo-felhom |
| `scripts/` | every helper used, as run |

## The answers

**Q1 — exact Debian version a week later.** The live Debian archives keep **two** versions of a package at most: the
point-release version in `trixie` (main) and the newest in `trixie-security` — never the intermediate ones
(`partF/madison-demo-hp-host.txt`: openssl installed `u1`, main `u2`, security `u3`; `u1` is gone). DSA history,
2026-07-04..10-04 (`partF/dsa-supersede-analysis.txt`): 167 trixie advisories; a second advisory for the same package
within 2 days: **3 (1.8 %), all chromium/webkit2gtk — none of them on a box**. Restricted to the 517 source packages
installed on a box: 20 advisories over 16 packages, **0 re-announced within 2, 7 or 14 days**. `snapshot.debian.org`
is reachable from a box: a dated `apt-get update` takes **2.3–3.0 s** (10.1 MB), and the exact gone version
`openssl 3.5.6-1~deb13u1` downloads in **2.0 s** (`partF/snapshot-debian-org.md`; https needs `ca-certificates`,
present on boxes, absent in the bare `debian:trixie` image).

**Q2 — Proxmox and Docker keep old versions.** Yes, many: `pve-manager` 66, `qemu-server` 58, `proxmox-kernel-7.0`
33, `pve-container` 32, `docker-ce` 46, `containerd.io` 18 versions listed. Debian: two at most (Q1). One host undo
measured: `rsync` back to its pre-update `3.4.1+ds1-5+deb13u2` → `E: Version … was not found`; `libpng16-16t64` back
to `1.6.48-1+deb13u5` worked because that version is the point-release one in main (`partH/H3-undo-one-package.txt`).

**Q3 — Docker engine update.** Without `live-restore` (today's baked setting): all 6 containers restart, the app's
front door is silent **26.5–30 s**, everything healthy **+42–45 s**. With `live-restore`: **0 restarts, no gap**,
also across a containerd change. `systemctl reload` turns it ON; it does NOT turn it off; and a restart that turns it
off **stops every running container and starts none of them** (`partG/SUMMARY.md`).

**Q4 — kernel.** Both demo hosts: UEFI, GRUB (no proxmox-boot-tool ESPs), root on LVM ext4; demo-hp has Secure Boot ON,
demo-felhom OFF (setup mode). Installing a kernel makes it the GRUB default at once (`GRUB_DEFAULT=0`, newest first).
`proxmox-boot-tool kernel pin <ver> --next-boot` on GRUB writes a normal `GRUB_DEFAULT` + `update-grub` (GRUB's own
one-shot `next_entry` stays empty); `proxmox-boot-cleanup.service` removes it **after a boot reaches userspace**.
Measured on demo-hp with the operator's word: old kernel pinned permanently first, new pinned for next boot → reboot
1 came up on `7.0.14-20-pve` (60 s, Secure Boot on, all healthy); reboot 2 came back on `7.0.2-6-pve` (76 s). **So the
fallback works after a SUCCESSFUL boot; read from the code (not measured), a kernel that hangs before userspace stays
the default.** Watchdog: only `softdog` is loaded (by `watchdog-mux`); demo-hp has an AMD FCH (SMBus 00:14.0) and the
`sp5100_tco` module ships with the kernel, not loaded. A softdog cannot rescue a kernel that never boots.

**Q5 — interrupted apt.** Killed after 15 unpacks: 5 packages `iU`, 4 triggers pending. It does NOT recover by itself —
the next `apt-get install` refuses (`Unmet dependencies. Try 'apt --fix-broken install'`). `dpkg --configure -a`
(1.4 s) + `apt-get -f install` (3.9 s) repaired it; the rest applied in 14.1 s. Containers stayed up throughout.

**Q6 — how far behind.** Hosts: 188 pending each (80/79 Proxmox, 66 Debian point release, 27 both, 15 security, +1
tailscale on demo-felhom); guests: 59 (31 point, 15 both, 7 security, 6 Docker) on both 9201s, 54 on 9202. Installed:
host 746/747 packages, guest 278. Each guest runs a DIFFERENT Docker (29.7.1, 29.7.2, 29.8.0; golden 0.290.0 bakes
29.8.2). Catching up: guest Debian 24.0 s / 38.1 MB; host Debian 59.7 s / 76 MB; the kernel 47 s / 131 MB.

**Q7 — what restarts.** Guest Debian run: postfix, journald, networkd restarted by their scripts; dockerd, containerd,
sshd, dbus, logind, cron, dhclient keep the OLD libc until restarted. Host Debian run: dnsmasq, postfix, journald
restarted; **systemd (PID 1), lxc-start, pveproxy, pvedaemon, pvestatd, pvescheduler, watchdog-mux, sshd, zed, chronyd**
keep the old libc. Proxmox packages (simulated, from their own maintainer scripts): pve-manager → pvedaemon, pveproxy,
pvestatd, pvescheduler, spiceproxy; pve-cluster → pmxcfs; corosync; chrony; qemu-server → qmeventd; pve-firewall;
pve-ha-manager → lrm, crm; zfs-zed.

**Q8 — the night window.** Guest (UTC, demo-hp, W = 02:30): db-dump 02:30, tier-2 03:30, off-site ~04:15, whole-guest
gate [04:30, 08:30), controller self-update 04:30, offsite-integrity 06:00. Host (CEST): `apt-daily` and
`apt-daily-upgrade` timers are enabled but install nothing (no `unattended-upgrades`, no `APT::Periodic`);
`pve-daily-update` refreshes the lists daily; **restore-tests run at any hour** (demo-felhom 10:38 daily, demo-hp
16:43 and 22:46) — they are cadence-driven, not windowed; agent updates arrive by signed job at any hour.

**Q9 — cloudflared.** Not on the host: a container in the guest, `cloudflare/cloudflared:2026.6.0`, pinned in the
controller (`internal/infra/infra.go:26`) since 2026-06-11 (v0.41.0) and baked into the golden. Upstream is at
`2026.9.3` (2026-09-24). It moves only when someone edits the pin.

**Q10 — Proxmox enterprise repository** (proxmox.com pricing page, 2026-10-04, net, per CPU socket per year):
Community €120, Basic €370, Standard €550, Premium €1,100. Every tier includes the Enterprise Repository; Community
has no support tickets. Record only — a money decision.

## Teardown
- 9202: Debian fully updated, Docker 29.8.2 / containerd 2.3.6 (= golden 0.290.0), `daemon.json` byte-identical to the
  baked one, live-restore false, all 6 containers healthy. Its backup in `/mnt/hdd_1/dump-9202-spike` deleted.
- demo-hp host: 108 Debian packages updated + `proxmox-kernel-7.0.14-20-pve-signed` installed (110 package changes,
  `partH/H-final-host-packages-after.tsv`); running `7.0.2-6-pve`, **next boot `7.0.14-20-pve`** (no pins left).
  Proxmox packages NOT updated (78 pending).
- Helper files removed from both hosts and all guests; no throwaway image or container left.