felhom.eu/documentation/architecture/11-os-updates.md

11 — Operating-system updates: the host, the guest and the Docker engine

Status	NOT RATIFIED — a PROPOSAL with one operator ruling. Ratification is Viktor's review, not an editor's.
Written	2026-10-04, by the reviewer (project Claude), before any spike.
Verified against	felhom.eu d07a1a9 · felhom-controller 99a1497 (v0.290.0) · felhom-agent d766666 (v0.138.0) · hub v0.128.0
Freshness	CURRENT as of 2026-10-04. The spike TASK-backup-close-and-os-updates-spike-2026-10-04 adds measurements here as [FACT] and corrects every claim it disproves. Mark this file STALE when it falls behind what the product does.

How to read this document. Each statement has a label:

• [FACT]: an observed property, with a file:line, a register row or an audit path.
• [RULED]: an operator decision, with its date.
• [PROPOSAL]: the reviewer's design. It is not decided and not built.
• OPEN: a question that nobody has answered yet. The spike measures it. Nobody guesses it.

Why this file exists. No architecture document covered operating-system updates. 00 §G marks the capability MISSING (added 2026-10-03). The finding is R-812. The roadmap intention is R-808. The register carries the work. This file carries the reasoning. The source is the truth.

---

0. In plain language

A box runs three layers that we install and never update: the Proxmox host, the small Debian system inside the guest, and the Docker engine. App images are updated (see 09). The layer under them is not. A box lives in a home for years, so this is a security gap.

The proposal has two lanes. The fast lane applies Debian's security fixes automatically, but only the exact versions that already ran well on the demo boxes for 1–2 days. The slow lane covers the kernel, Proxmox and Docker. These cause restarts or big changes, so a person approves them one version at a time, and they run only at night. The tested versions are recorded automatically from what the demo boxes installed. Nobody keeps a hand-written list.

---

1. Scope

In scope.

• The Proxmox host on an appliance install: the Debian 13 base, the Proxmox packages, and the kernel.
• The guest (the customer LXC): its Debian 13 packages.
• The Docker engine inside the guest (docker-ce, docker-ce-cli, containerd.io).
• How the household and the operator are told, and how a failed update is undone.

Out of scope.

• App images. 09 covers them (the ladder, the monthly same-tag re-test).
• The controller and agent binaries. Their own self-update covers them (03, the self-update section; 09 R-608).
• DooPlex and ep0. The operator updates them by hand (runbooks/offsite-endpoint.md).
• A BYO host (the owner brought their own Proxmox). [FACT] The installer leaves its repositories alone: "apt repo alignment skipped (byo — the owner manages repos)" (scripts/felhom-host-install.sh, align_apt_repos). [PROPOSAL] On a BYO box we update the guest and Docker only, never the host.
• The Proxmox MAJOR upgrade (PVE 9 → 10). It is a later step of its own, drilled first (R-808 item 5).

---

2. What a box runs, as measured

Layer	What it is	Source of packages	Evidence
Host	Proxmox VE 9.2 on Debian 13, LVM-thin	Debian mirrors + pve-no-subscription (enterprise repo switched off on appliance installs)	[FACT] 01 §2; felhom-host-install.sh align_apt_repos (~L2130–2136)
Host kernel	The only kernel on the box. The guest has none.	pve-no-subscription (proxmox-kernel-*)	[FACT] LXC shares the host kernel (01 §2: "one LXC/kernel/Docker daemon")
Guest	Unprivileged LXC, nesting=1,keyctl=1, from debian-13-standard_13.1-2	Debian mirrors	[FACT] felhom-agent/configs/build-golden.sh:67,101-103
Docker engine	docker-ce, docker-ce-cli, containerd.io, installed at golden BAKE time	download.docker.com/linux/debian trixie stable	[FACT] build-golden.sh:108-125
Docker settings	containerd-snapshotter: false, json-file log caps. No live-restore.	baked daemon.json	[FACT] build-golden.sh:138-144
Controller	A container in the guest. A Docker engine restart restarts it too.	Felhom registry	[FACT] 03 §1

[FACT] Nothing updates any of these layers today (R-812, searched 2026-10-03). The installer says "No upgrades are run — repo alignment only" (felhom-host-install.sh ~L2135). A fresh install gets the Docker engine that was current when its golden was baked, and keeps it.

[FACT] The agent may not run apt today, with one exception. Its sudoers allowlist (felhom-agent/configs/felhom-agent.sudoers) holds one apt line: apt-get install -y -q dnsmasq. apt as root runs package scripts as root, so a broad apt grant is a full root grant. §5.4 proposes how to avoid that.

---

3. Operator rulings

2026-10-03. R-808 is on the roadmap at P2: "every box receives operating-system security patches on a schedule, and a failed update is undone."

[RULED] 2026-10-04: the fast lane follows an approved list, with a 1–2 day wait. Every update runs on the demo boxes first. The other boxes install only the exact versions that the demo boxes ran without trouble. The exact wait is set when the feature is built. Rejected: Debian's own unattended-upgrades with no wait. It is simpler and common, but a bad update would reach every customer at the same time.

[RULED] 2026-10-04: the off-site backup topic is closed first. The same brief carries both topics, and the backup part runs first.

The two-lane split (§5.2) is the reviewer's proposal. The operator's ruling above assumes it, but he has not ruled on it as such.

---

4. The constraints that shape the design

1. Unattended. Nobody is at the box. Nobody answers a question that apt asks.
2. No screen. If the box does not boot, the household sees only that nothing works.
3. One kernel for everything. A host kernel update needs a host reboot. A reboot stops every app.
4. The controller lives in Docker. A Docker engine update restarts the controller in the middle of its own work. So the controller cannot drive a Docker update. The agent, on the host, must.
5. The host has no whole-system backup. The guest has three tiers (07). The host has none. A host update can only be undone by installing the previous version again.
6. Every update must already have run on a box we own. This is the lesson of the update arc (09 §3 decision 13: "the test decides").
7. Few packages. Each extra package is one more thing to update and break. The guest is the Debian standard template plus Docker. Keep it that way.

---

5. The proposed shape [PROPOSAL]

5.1 Two rings

• Ring 0: demo-felhom (N100) and demo-hp. They are disposable (runbooks/target-selection.md), and they have different hardware. They take every update first.
• Ring 1: every other box. It takes only what ring 0 approved.

5.2 Two lanes

	Fast lane	Slow lane
What	Debian packages on host and guest, from trixie-security and the stable point releases, except the slow-lane list	Kernel (proxmox-kernel-*), Proxmox (pve-*, proxmox-*, lxc-pve, qemu-server, …), Docker (docker-ce*, containerd.io)
Restart	A service restart at most. No reboot.	Kernel: a host reboot. Docker: every container restarts. Proxmox: its services restart.
Approval	Automatic. A version is approved when ring 0 ran it and stayed healthy for the wait (1–2 days).	A person approves one version at a time, like a controller floor.
Urgent fix	The operator can approve a version on the same day once ring 0 has run it.	The same.
When	The night window (§5.5)	The night window. A kernel reboot only on a night the operator scheduled.

Which packages count as slow lane is a list the spike checks (OPEN Q7). A Debian package that restarts something big (for example systemd, libc6, openssh-server) may belong in the slow lane too.

5.3 The approved list (the "tested versions" record)

Nobody writes the list by hand. It fills itself:

1. A ring-0 box updates. Afterwards it reports to the hub the exact package=version it installed, per layer (host, guest), with each package's origin (Debian-Security, Debian, Proxmox, Docker).
2. The box then reports health for the wait period: the agent, the controller, every app's health, and the guest's network.
3. When the wait passes with ring 0 healthy, the hub marks that set approved. It is one record: an OS release, with an id and a date. The hub stores it. The register does not.
4. A ring-1 box asks the hub for the newest approved OS release. For each package it has installed, if the approved version is newer, it installs that exact version. It never installs a version newer than the approved one.
5. Each box reports which OS release it runs, how many updates are waiting, whether it needs a reboot, and which packages it has that no approved list covers (see §6, edge case 9).

OPEN Q1 is the weak point. The design works only if a box can still download the approved version a week later. Debian's main and security archives keep only the newest version of each package. If Debian publishes a newer fix between approval and install, the approved version is gone. Options: the box waits for the next approval; or the box uses snapshot.debian.org (Debian's own dated archive, still signed by Debian); or Felhom runs a package cache. The spike measures how often this happens.

5.4 Who runs it, and with what permission

• The agent runs every OS update, for the host and for the guest (pct exec). The controller does not, because of constraint 4.
• The agent gets no general apt permission. Like felhom-selfupdate-guarded (03, the self-update section), a small root-owned wrapper does the work. It accepts only an approved list and refuses everything else:
  • no package removal;
  • no downgrade, except the undo of §5.6, which an operator job signs;
  • no package that the box does not already have, unless the approved list records it as a dependency that the same update pulled in on ring 0 (a kernel update installs a NEW package name each time, for example proxmox-kernel-6.x.y-z-pve-signed, pulled by proxmox-default-kernel);
  • no package source other than the ones the installer set up;
  • non-interactive, and it always keeps the existing config file (--force-confold) and reports the conflict.
• Package signatures stay the publishers' (Debian, Proxmox, Docker). The hub sends only names and versions. A broken-into hub can choose an older version or no version. It cannot make a box install a package that the publisher did not sign.

5.5 When

Inside the household's night window, after the backups:

W        DB dump
W+60m    Tier 2
W+105m   off-site → app updates (until W+5h at most)
[W+2h, W+6h)  whole-guest backup (agent)
after it      OS updates — guest first, then host

The OS leg starts after the whole-guest backup has finished, so the guest's newest full copy is minutes old. This is the opposite order to app updates, which run before the whole-guest backup (07 §6.1, 09 decision 11). The reason: the whole-guest backup is the guest's undo.

OPEN Q8: what else runs then. The controller's self-update (default 04:30, and after any hub report when a floor is above it, R-608), the agent's self-update, the restore-tests, and PBS jobs. The OS leg must never overlap a backup, a restore-test or a self-update.

5.6 How a failed update is undone

"Rollback" is not used (09 §4). The shapes:

Layer	Undo	Limit
Guest packages	Restore the guest snapshot taken just before the update	It also undoes app data written after the snapshot. Use it only inside the health window, before apps have written much. After that, install the previous version (needs OPEN Q1).
Docker engine	Install the previous version (Docker's repository keeps old versions — OPEN Q2)	Every container restarts again
Host packages	Install the previous version	Only if the source still has it (OPEN Q1/Q2)
Host kernel	Boot the previous kernel. Proxmox can boot a new kernel once (proxmox-boot-tool kernel pin <ver> --next-boot). If that boot fails, the next boot uses the old kernel again. Make the new kernel permanent only after a healthy boot.	If the new kernel hangs, someone must switch the box off and on. The spike checks whether a hardware watchdog can do that (OPEN Q4).

5.7 Telling people

• Operator: a hub event for each update and each failure; a fleet view showing each box's OS release, how far behind it is, and whether it needs a reboot. A box that is more than N days behind the newest approved release raises an alarm (08).
• Household: one line on the timeline in both languages, informal voice: what was updated and whether the box restarted. Telling households in advance that the box may restart at night is a promise to users. That is the operator's decision when the slow lane is built.

---

6. Risks and edge cases

#	What can go wrong	What the design does
1	A new kernel does not boot	Boot it once (--next-boot); the old kernel stays the default. A hard hang needs a power cycle (OPEN Q4).
2	Power cut during an update: dpkg is half done	The wrapper runs dpkg --configure -a and apt-get -f install first, every time, and reports what it repaired (spike measures, Q5).
3	An update asks a question (changed config file, service restart prompt)	Non-interactive, keep the old config, report the conflict.
4	A Docker update stops every app, and the controller	Stop the apps cleanly first, like before a backup. Measure whether live-restore keeps containers running (Q3). The agent drives it.
5	The approved version is no longer downloadable	OPEN Q1. Until it is answered, the box waits for the next approval and reports it.
6	An urgent security hole	The operator approves the same day once ring 0 has run it.
7	A box was off for months	It catches up through the newest approved release. It does not install every release in between. Packages are not like app data; one apt step is enough. Kernel and Docker still go one approved step at a time.
8	The system disk is full	Check free space before downloading; clean the package cache after; refuse and report.
9	A customer box has a package that ring 0 does not have (other hardware: firmware, CPU microcode, NIC drivers)	It is never approved, so it is never updated. The box reports it as "not covered", and the hub raises it. Fix: add matching hardware to ring 0, or approve it by hand.
10	The package source is down, or its signing key changes (Docker has done this)	The update fails cleanly and the box reports it. A key change is a slow-lane act for a person.
11	A BYO host	Only the guest and Docker are updated (§1).
12	Two boxes on ring 0 is a small sample	Accepted for now. When there are customers, the first tester boxes can become a second ring.
13	A broken-into hub sends a harmful list	The wrapper refuses removals, downgrades and new packages, and the publishers' signatures still apply (§5.4).
14	cloudflared on the host	Not covered by this file yet. How it is installed and updated is OPEN Q9. It faces the internet, so it matters.
15	The no-subscription Proxmox repository gets less testing than the enterprise one	Ring 0 is our test. The enterprise repository costs a yearly fee per box. That is a money decision for the operator, later (Q10).

---

7. Open questions the spike must answer

Q	Question	How to answer
Q1	Can a box install an exact Debian version one week after approval? How often is it already superseded?	apt-cache madison on host and guest for security packages. Debian's security announcement history. Whether snapshot.debian.org is reachable and fast enough.
Q2	Do the Proxmox and Docker sources keep older versions?	apt-cache madison for pve-manager, proxmox-kernel-*, docker-ce, containerd.io.
Q3	What does a Docker engine update do to running containers, with and without live-restore?	On scratch guest 9202: step from one pinned version to the next; time the downtime.
Q4	Does proxmox-boot-tool kernel pin --next-boot work on both demo hosts' boot setups (UEFI or legacy, GRUB or systemd-boot)? Does either box have a hardware watchdog?	On demo-hp, with the operator's go before each reboot.
Q5	What does an interrupted apt run leave, and does the repair recover it?	Kill a run on 9202, after a snapshot.
Q6	How far behind are the demo boxes today, per layer and per source? How long does catching up take, and what restarts?	apt-get -s upgrade (read only) first; then a real run on 9202 and on one demo host.
Q7	Which Debian packages restart something big?	needrestart in list mode after a run.
Q8	What else runs in the night window, and where does the OS leg fit?	Read the timers on host and guest.
Q9	How is cloudflared installed and updated on the host?	Read the installer and the agent.
Q10	What does the Proxmox enterprise repository cost per box per year, and what does it add?	The publisher's price page. Record only. The operator decides later.

---

8. Build order [PROPOSAL]

Each step returns to the operator for go or no-go.

1. Spike (measure Q1–Q10; no product code).
2. Guest Debian, fast lane. Lowest risk: a snapshot undo exists.
3. Host Debian, fast lane (no kernel, no Proxmox packages).
4. Fleet view and alarms (§5.7).
5. Slow lane: Docker engine.
6. Slow lane: host kernel and Proxmox packages, with the reboot.
7. Later: the Proxmox major upgrade (PVE 9 → 10), drilled on ring 0 first.

---

9. Where the rest lives

• The finding: R-812 (backlog/OPEN-ITEMS.md). The intention: R-808 (backlog/ROADMAP.md).
• Related: R-604 (a per-customer floor hides a box from global raises; the same risk applies to an OS-release floor), R-530 (agents update only by a signed job per box).
• App updates: 09. Backups and the night chain: 07 §6.1. The agent's permissions: 03 §3.