admin
1b74ddc0c9
gates / gates (push) Successful in 33s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
85 lines
7.0 KiB
Markdown
85 lines
7.0 KiB
Markdown
# OS-updates spike — 2026-10-04 (answers `architecture/11-os-updates.md` §7 Q1–Q10)
|
||
|
||
Venues: demo-hp host + its 9201 + scratch 9202 (changes: 9202 Part G, demo-hp host Part H); demo-felhom host + 9201
|
||
(read only). Apt indexes for every read-only step went to a throwaway directory (`scripts/os-survey.sh`), so no
|
||
box's lists or sources changed. Hosts are CEST, guests UTC.
|
||
|
||
| Folder | What |
|
||
|---|---|
|
||
| `partE/` | read-only surveys of five systems (`*-host.txt`, `*-guest.txt`), host package-set diff |
|
||
| `partF/` | `apt-cache madison` (host + guest), Debian DSA re-announcement analysis, snapshot.debian.org probe |
|
||
| `partG/` | 9202: Debian update, undo, killed update, Docker step ×3 — `SUMMARY.md` |
|
||
| `partH/` | demo-hp host: package lists before/after, Debian-lane run, one-package undo, kernel install + 2 reboots, Proxmox simulation |
|
||
| `partI/` | sample approved list (`sample-approved-list.tsv`) and its read-only simulation on demo-felhom |
|
||
| `scripts/` | every helper used, as run |
|
||
|
||
## The answers
|
||
|
||
**Q1 — exact Debian version a week later.** The live Debian archives keep **two** versions of a package at most: the
|
||
point-release version in `trixie` (main) and the newest in `trixie-security` — never the intermediate ones
|
||
(`partF/madison-demo-hp-host.txt`: openssl installed `u1`, main `u2`, security `u3`; `u1` is gone). DSA history,
|
||
2026-07-04..10-04 (`partF/dsa-supersede-analysis.txt`): 167 trixie advisories; a second advisory for the same package
|
||
within 2 days: **3 (1.8 %), all chromium/webkit2gtk — none of them on a box**. Restricted to the 517 source packages
|
||
installed on a box: 20 advisories over 16 packages, **0 re-announced within 2, 7 or 14 days**. `snapshot.debian.org`
|
||
is reachable from a box: a dated `apt-get update` takes **2.3–3.0 s** (10.1 MB), and the exact gone version
|
||
`openssl 3.5.6-1~deb13u1` downloads in **2.0 s** (`partF/snapshot-debian-org.md`; https needs `ca-certificates`,
|
||
present on boxes, absent in the bare `debian:trixie` image).
|
||
|
||
**Q2 — Proxmox and Docker keep old versions.** Yes, many: `pve-manager` 66, `qemu-server` 58, `proxmox-kernel-7.0`
|
||
33, `pve-container` 32, `docker-ce` 46, `containerd.io` 18 versions listed. Debian: two at most (Q1). One host undo
|
||
measured: `rsync` back to its pre-update `3.4.1+ds1-5+deb13u2` → `E: Version … was not found`; `libpng16-16t64` back
|
||
to `1.6.48-1+deb13u5` worked because that version is the point-release one in main (`partH/H3-undo-one-package.txt`).
|
||
|
||
**Q3 — Docker engine update.** Without `live-restore` (today's baked setting): all 6 containers restart, the app's
|
||
front door is silent **26.5–30 s**, everything healthy **+42–45 s**. With `live-restore`: **0 restarts, no gap**,
|
||
also across a containerd change. `systemctl reload` turns it ON; it does NOT turn it off; and a restart that turns it
|
||
off **stops every running container and starts none of them** (`partG/SUMMARY.md`).
|
||
|
||
**Q4 — kernel.** Both demo hosts: UEFI, GRUB (no proxmox-boot-tool ESPs), root on LVM ext4; demo-hp has Secure Boot ON,
|
||
demo-felhom OFF (setup mode). Installing a kernel makes it the GRUB default at once (`GRUB_DEFAULT=0`, newest first).
|
||
`proxmox-boot-tool kernel pin <ver> --next-boot` on GRUB writes a normal `GRUB_DEFAULT` + `update-grub` (GRUB's own
|
||
one-shot `next_entry` stays empty); `proxmox-boot-cleanup.service` removes it **after a boot reaches userspace**.
|
||
Measured on demo-hp with the operator's word: old kernel pinned permanently first, new pinned for next boot → reboot
|
||
1 came up on `7.0.14-20-pve` (60 s, Secure Boot on, all healthy); reboot 2 came back on `7.0.2-6-pve` (76 s). **So the
|
||
fallback works after a SUCCESSFUL boot; read from the code (not measured), a kernel that hangs before userspace stays
|
||
the default.** Watchdog: only `softdog` is loaded (by `watchdog-mux`); demo-hp has an AMD FCH (SMBus 00:14.0) and the
|
||
`sp5100_tco` module ships with the kernel, not loaded. A softdog cannot rescue a kernel that never boots.
|
||
|
||
**Q5 — interrupted apt.** Killed after 15 unpacks: 5 packages `iU`, 4 triggers pending. It does NOT recover by itself —
|
||
the next `apt-get install` refuses (`Unmet dependencies. Try 'apt --fix-broken install'`). `dpkg --configure -a`
|
||
(1.4 s) + `apt-get -f install` (3.9 s) repaired it; the rest applied in 14.1 s. Containers stayed up throughout.
|
||
|
||
**Q6 — how far behind.** Hosts: 188 pending each (80/79 Proxmox, 66 Debian point release, 27 both, 15 security, +1
|
||
tailscale on demo-felhom); guests: 59 (31 point, 15 both, 7 security, 6 Docker) on both 9201s, 54 on 9202. Installed:
|
||
host 746/747 packages, guest 278. Each guest runs a DIFFERENT Docker (29.7.1, 29.7.2, 29.8.0; golden 0.290.0 bakes
|
||
29.8.2). Catching up: guest Debian 24.0 s / 38.1 MB; host Debian 59.7 s / 76 MB; the kernel 47 s / 131 MB.
|
||
|
||
**Q7 — what restarts.** Guest Debian run: postfix, journald, networkd restarted by their scripts; dockerd, containerd,
|
||
sshd, dbus, logind, cron, dhclient keep the OLD libc until restarted. Host Debian run: dnsmasq, postfix, journald
|
||
restarted; **systemd (PID 1), lxc-start, pveproxy, pvedaemon, pvestatd, pvescheduler, watchdog-mux, sshd, zed, chronyd**
|
||
keep the old libc. Proxmox packages (simulated, from their own maintainer scripts): pve-manager → pvedaemon, pveproxy,
|
||
pvestatd, pvescheduler, spiceproxy; pve-cluster → pmxcfs; corosync; chrony; qemu-server → qmeventd; pve-firewall;
|
||
pve-ha-manager → lrm, crm; zfs-zed.
|
||
|
||
**Q8 — the night window.** Guest (UTC, demo-hp, W = 02:30): db-dump 02:30, tier-2 03:30, off-site ~04:15, whole-guest
|
||
gate [04:30, 08:30), controller self-update 04:30, offsite-integrity 06:00. Host (CEST): `apt-daily` and
|
||
`apt-daily-upgrade` timers are enabled but install nothing (no `unattended-upgrades`, no `APT::Periodic`);
|
||
`pve-daily-update` refreshes the lists daily; **restore-tests run at any hour** (demo-felhom 10:38 daily, demo-hp
|
||
16:43 and 22:46) — they are cadence-driven, not windowed; agent updates arrive by signed job at any hour.
|
||
|
||
**Q9 — cloudflared.** Not on the host: a container in the guest, `cloudflare/cloudflared:2026.6.0`, pinned in the
|
||
controller (`internal/infra/infra.go:26`) since 2026-06-11 (v0.41.0) and baked into the golden. Upstream is at
|
||
`2026.9.3` (2026-09-24). It moves only when someone edits the pin.
|
||
|
||
**Q10 — Proxmox enterprise repository** (proxmox.com pricing page, 2026-10-04, net, per CPU socket per year):
|
||
Community €120, Basic €370, Standard €550, Premium €1,100. Every tier includes the Enterprise Repository; Community
|
||
has no support tickets. Record only — a money decision.
|
||
|
||
## Teardown
|
||
- 9202: Debian fully updated, Docker 29.8.2 / containerd 2.3.6 (= golden 0.290.0), `daemon.json` byte-identical to the
|
||
baked one, live-restore false, all 6 containers healthy. Its backup in `/mnt/hdd_1/dump-9202-spike` deleted.
|
||
- demo-hp host: 108 Debian packages updated + `proxmox-kernel-7.0.14-20-pve-signed` installed (110 package changes,
|
||
`partH/H-final-host-packages-after.tsv`); running `7.0.2-6-pve`, **next boot `7.0.14-20-pve`** (no pins left).
|
||
Proxmox packages NOT updated (78 pending).
|
||
- Helper files removed from both hosts and all guests; no throwaway image or container left.
|