From 55e9522d0b9348b9027d33545e43d03a39430bd3 Mon Sep 17 00:00:00 2001 From: kisfenyo Date: Thu, 8 Oct 2026 19:32:58 +0200 Subject: [PATCH] os updates --- documentation/architecture/11-os-updates.md | 275 ++++++++++++++++++++ 1 file changed, 275 insertions(+) create mode 100644 documentation/architecture/11-os-updates.md diff --git a/documentation/architecture/11-os-updates.md b/documentation/architecture/11-os-updates.md new file mode 100644 index 00000000..adfa95d3 --- /dev/null +++ b/documentation/architecture/11-os-updates.md @@ -0,0 +1,275 @@ +# 11 — Operating-system updates: the host, the guest and the Docker engine + +> | | | +> |---|---| +> | **Status** | **NOT RATIFIED — a PROPOSAL with one operator ruling.** Ratification is Viktor's review, not an editor's. | +> | **Written** | 2026-10-04, by the reviewer (project Claude), before any spike. | +> | **Verified against** | felhom.eu `d07a1a9` · felhom-controller `99a1497` (v0.290.0) · felhom-agent `d766666` (v0.138.0) · hub v0.128.0 | +> | **Freshness** | **CURRENT** as of 2026-10-04. The spike `TASK-backup-close-and-os-updates-spike-2026-10-04` adds measurements here as `[FACT]` and corrects every claim it disproves. Mark this file STALE when it falls behind what the product does. | +> +> **How to read this document.** Each statement has a label: +> +> - **[FACT]**: an observed property, with a `file:line`, a register row or an audit path. +> - **[RULED]**: an operator decision, with its date. +> - **[PROPOSAL]**: the reviewer's design. It is not decided and not built. +> - **OPEN**: a question that nobody has answered yet. The spike measures it. Nobody guesses it. +> +> **Why this file exists.** No architecture document covered operating-system updates. `00` §G marks the +> capability MISSING (added 2026-10-03). The finding is **R-812**. The roadmap intention is **R-808**. +> **The register carries the work. This file carries the reasoning. The source is the truth.** + +--- + +## 0. In plain language + +A box runs three layers that we install and never update: the Proxmox host, the small Debian system +inside the guest, and the Docker engine. App images are updated (see `09`). The layer under them is not. +A box lives in a home for years, so this is a security gap. + +The proposal has two lanes. **The fast lane** applies Debian's security fixes automatically, but only +the exact versions that already ran well on the demo boxes for 1–2 days. **The slow lane** covers the +kernel, Proxmox and Docker. These cause restarts or big changes, so a person approves them one version at +a time, and they run only at night. The tested versions are recorded automatically from what the demo +boxes installed. Nobody keeps a hand-written list. + +--- + +## 1. Scope + +**In scope.** +- The Proxmox host on an **appliance** install: the Debian 13 base, the Proxmox packages, and the kernel. +- The guest (the customer LXC): its Debian 13 packages. +- The Docker engine inside the guest (`docker-ce`, `docker-ce-cli`, `containerd.io`). +- How the household and the operator are told, and how a failed update is undone. + +**Out of scope.** +- **App images.** `09` covers them (the ladder, the monthly same-tag re-test). +- **The controller and agent binaries.** Their own self-update covers them (`03`, the self-update section; `09` R-608). +- **DooPlex and ep0.** The operator updates them by hand (`runbooks/offsite-endpoint.md`). +- **A BYO host** (the owner brought their own Proxmox). `[FACT]` The installer leaves its repositories + alone: *"apt repo alignment skipped (byo — the owner manages repos)"* (`scripts/felhom-host-install.sh`, + `align_apt_repos`). `[PROPOSAL]` On a BYO box we update the guest and Docker only, never the host. +- **The Proxmox MAJOR upgrade** (PVE 9 › 10). It is a later step of its own, drilled first (R-808 item 5). + +--- + +## 2. What a box runs, as measured + +| Layer | What it is | Source of packages | Evidence | +|---|---|---|---| +| Host | Proxmox VE 9.2 on Debian 13, LVM-thin | Debian mirrors + `pve-no-subscription` (enterprise repo switched off on appliance installs) | `[FACT]` `01` §2; `felhom-host-install.sh` `align_apt_repos` (~L2130–2136) | +| Host kernel | The only kernel on the box. The guest has none. | `pve-no-subscription` (`proxmox-kernel-*`) | `[FACT]` LXC shares the host kernel (`01` §2: "one LXC/kernel/Docker daemon") | +| Guest | Unprivileged LXC, `nesting=1,keyctl=1`, from `debian-13-standard_13.1-2` | Debian mirrors | `[FACT]` `felhom-agent/configs/build-golden.sh:67,101-103` | +| Docker engine | `docker-ce`, `docker-ce-cli`, `containerd.io`, installed at golden BAKE time | `download.docker.com/linux/debian trixie stable` | `[FACT]` `build-golden.sh:108-125` | +| Docker settings | `containerd-snapshotter: false`, json-file log caps. **No `live-restore`.** | baked `daemon.json` | `[FACT]` `build-golden.sh:138-144` | +| Controller | A container in the guest. A Docker engine restart restarts it too. | Felhom registry | `[FACT]` `03` §1 | + +**[FACT] Nothing updates any of these layers today** (R-812, searched 2026-10-03). The installer +says *"No upgrades are run — repo alignment only"* (`felhom-host-install.sh` ~L2135). A fresh install +gets the Docker engine that was current when its golden was baked, and keeps it. + +**[FACT] The agent may not run `apt` today, with one exception.** Its sudoers allowlist +(`felhom-agent/configs/felhom-agent.sudoers`) holds one apt line: `apt-get install -y -q dnsmasq`. +`apt` as root runs package scripts as root, so a broad `apt` grant is a full root grant. §5.4 +proposes how to avoid that. + +--- + +## 3. Operator rulings + +**2026-10-03.** R-808 is on the roadmap at P2: *"every box receives operating-system security +patches on a schedule, and a failed update is undone."* + +**[RULED] 2026-10-04: the fast lane follows an approved list, with a 1–2 day wait.** Every update +runs on the demo boxes first. The other boxes install only the exact versions that the demo boxes ran +without trouble. The exact wait is set when the feature is built. **Rejected:** Debian's own +`unattended-upgrades` with no wait. It is simpler and common, but a bad update would reach every +customer at the same time. + +**[RULED] 2026-10-04: the off-site backup topic is closed first.** The same brief carries both +topics, and the backup part runs first. + +The two-lane split (§5.2) is the reviewer's proposal. The operator's ruling above assumes it, but he +has not ruled on it as such. + +--- + +## 4. The constraints that shape the design + +1. **Unattended.** Nobody is at the box. Nobody answers a question that `apt` asks. +2. **No screen.** If the box does not boot, the household sees only that nothing works. +3. **One kernel for everything.** A host kernel update needs a host reboot. A reboot stops every app. +4. **The controller lives in Docker.** A Docker engine update restarts the controller in the middle of + its own work. So the controller cannot drive a Docker update. The agent, on the host, must. +5. **The host has no whole-system backup.** The guest has three tiers (`07`). The host has none. + A host update can only be undone by installing the previous version again. +6. **Every update must already have run on a box we own.** This is the lesson of the update arc + (`09` §3 decision 13: "the test decides"). +7. **Few packages.** Each extra package is one more thing to update and break. The guest is the + Debian standard template plus Docker. Keep it that way. + +--- + +## 5. The proposed shape `[PROPOSAL]` + +### 5.1 Two rings + +- **Ring 0:** demo-felhom (N100) and demo-hp. They are disposable (`runbooks/target-selection.md`), and + they have different hardware. They take every update first. +- **Ring 1:** every other box. It takes only what ring 0 approved. + +### 5.2 Two lanes + +| | Fast lane | Slow lane | +|---|---|---| +| What | Debian packages on host and guest, from `trixie-security` and the stable point releases, **except** the slow-lane list | Kernel (`proxmox-kernel-*`), Proxmox (`pve-*`, `proxmox-*`, `lxc-pve`, `qemu-server`, …), Docker (`docker-ce*`, `containerd.io`) | +| Restart | A service restart at most. No reboot. | Kernel: a host reboot. Docker: every container restarts. Proxmox: its services restart. | +| Approval | Automatic. A version is approved when ring 0 ran it and stayed healthy for the wait (1–2 days). | A person approves one version at a time, like a controller floor. | +| Urgent fix | The operator can approve a version on the same day once ring 0 has run it. | The same. | +| When | The night window (§5.5) | The night window. A kernel reboot only on a night the operator scheduled. | + +Which packages count as slow lane is a list the spike checks (OPEN Q7). A Debian package that restarts +something big (for example `systemd`, `libc6`, `openssh-server`) may belong in the slow lane too. + +### 5.3 The approved list (the "tested versions" record) + +Nobody writes the list by hand. It fills itself: + +1. A ring-0 box updates. Afterwards it reports to the hub the exact `package=version` it installed, + per layer (host, guest), with each package's origin (`Debian-Security`, `Debian`, `Proxmox`, + `Docker`). +2. The box then reports health for the wait period: the agent, the controller, every app's health, + and the guest's network. +3. When the wait passes with ring 0 healthy, the hub marks that set **approved**. It is one record: + an **OS release**, with an id and a date. The hub stores it. The register does not. +4. A ring-1 box asks the hub for the newest approved OS release. For each package it has installed, + if the approved version is newer, it installs that **exact** version. It never installs a version + newer than the approved one. +5. Each box reports which OS release it runs, how many updates are waiting, whether it needs a reboot, + and which packages it has that **no approved list covers** (see §6, edge case 9). + +**OPEN Q1 is the weak point.** The design works only if a box can still download the approved version +a week later. Debian's main and security archives keep only the newest version of each package. If +Debian publishes a newer fix between approval and install, the approved version is gone. Options: +the box waits for the next approval; or the box uses `snapshot.debian.org` (Debian's own dated archive, +still signed by Debian); or Felhom runs a package cache. The spike measures how often this happens. + +### 5.4 Who runs it, and with what permission + +- **The agent runs every OS update**, for the host and for the guest (`pct exec`). The controller does + not, because of constraint 4. +- **The agent gets no general `apt` permission.** Like `felhom-selfupdate-guarded` (`03`, the self-update section), + a small root-owned wrapper does the work. It accepts only an approved list and refuses everything else: + - no package removal; + - no downgrade, except the undo of §5.6, which an operator job signs; + - no package that the box does not already have, unless the approved list records it as a dependency + that the same update pulled in on ring 0 (a kernel update installs a NEW package name each time, + for example `proxmox-kernel-6.x.y-z-pve-signed`, pulled by `proxmox-default-kernel`); + - no package source other than the ones the installer set up; + - non-interactive, and it always keeps the existing config file (`--force-confold`) and reports the + conflict. +- **Package signatures stay the publishers'** (Debian, Proxmox, Docker). The hub sends only names and + versions. A broken-into hub can choose an older version or no version. It cannot make a box install a + package that the publisher did not sign. + +### 5.5 When + +Inside the household's night window, after the backups: + +``` +W DB dump +W+60m Tier 2 +W+105m off-site › app updates (until W+5h at most) +[W+2h, W+6h) whole-guest backup (agent) +after it OS updates — guest first, then host +``` + +The OS leg starts **after the whole-guest backup has finished**, so the guest's newest full copy is +minutes old. This is the opposite order to app updates, which run before the whole-guest backup +(`07` §6.1, `09` decision 11). The reason: the whole-guest backup is the guest's undo. + +**OPEN Q8:** what else runs then. The controller's self-update (default 04:30, and after any hub report +when a floor is above it, R-608), the agent's self-update, the restore-tests, and PBS jobs. The OS leg +must never overlap a backup, a restore-test or a self-update. + +### 5.6 How a failed update is undone + +"Rollback" is not used (`09` §4). The shapes: + +| Layer | Undo | Limit | +|---|---|---| +| Guest packages | Restore the guest snapshot taken just before the update | It also undoes app data written after the snapshot. Use it only inside the health window, before apps have written much. After that, install the previous version (needs OPEN Q1). | +| Docker engine | Install the previous version (Docker's repository keeps old versions — OPEN Q2) | Every container restarts again | +| Host packages | Install the previous version | Only if the source still has it (OPEN Q1/Q2) | +| Host kernel | Boot the previous kernel. Proxmox can boot a new kernel **once** (`proxmox-boot-tool kernel pin --next-boot`). If that boot fails, the next boot uses the old kernel again. Make the new kernel permanent only after a healthy boot. | If the new kernel hangs, someone must switch the box off and on. The spike checks whether a hardware watchdog can do that (OPEN Q4). | + +### 5.7 Telling people + +- **Operator:** a hub event for each update and each failure; a fleet view showing each box's OS release, + how far behind it is, and whether it needs a reboot. A box that is more than N days behind the newest + approved release raises an alarm (`08`). +- **Household:** one line on the timeline in both languages, informal voice: what was updated and + whether the box restarted. Telling households in advance that the box may restart at night is a + **promise to users**. That is the operator's decision when the slow lane is built. + +--- + +## 6. Risks and edge cases + +| # | What can go wrong | What the design does | +|---|---|---| +| 1 | A new kernel does not boot | Boot it once (`--next-boot`); the old kernel stays the default. A hard hang needs a power cycle (OPEN Q4). | +| 2 | Power cut during an update: `dpkg` is half done | The wrapper runs `dpkg --configure -a` and `apt-get -f install` first, every time, and reports what it repaired (spike measures, Q5). | +| 3 | An update asks a question (changed config file, service restart prompt) | Non-interactive, keep the old config, report the conflict. | +| 4 | A Docker update stops every app, and the controller | Stop the apps cleanly first, like before a backup. Measure whether `live-restore` keeps containers running (Q3). The agent drives it. | +| 5 | The approved version is no longer downloadable | OPEN Q1. Until it is answered, the box waits for the next approval and reports it. | +| 6 | An urgent security hole | The operator approves the same day once ring 0 has run it. | +| 7 | A box was off for months | It catches up through the newest approved release. It does not install every release in between. Packages are not like app data; one `apt` step is enough. Kernel and Docker still go one approved step at a time. | +| 8 | The system disk is full | Check free space before downloading; clean the package cache after; refuse and report. | +| 9 | A customer box has a package that ring 0 does not have (other hardware: firmware, CPU microcode, NIC drivers) | It is never approved, so it is never updated. The box reports it as "not covered", and the hub raises it. Fix: add matching hardware to ring 0, or approve it by hand. | +| 10 | The package source is down, or its signing key changes (Docker has done this) | The update fails cleanly and the box reports it. A key change is a slow-lane act for a person. | +| 11 | A BYO host | Only the guest and Docker are updated (§1). | +| 12 | Two boxes on ring 0 is a small sample | Accepted for now. When there are customers, the first tester boxes can become a second ring. | +| 13 | A broken-into hub sends a harmful list | The wrapper refuses removals, downgrades and new packages, and the publishers' signatures still apply (§5.4). | +| 14 | `cloudflared` on the host | Not covered by this file yet. How it is installed and updated is OPEN Q9. It faces the internet, so it matters. | +| 15 | The no-subscription Proxmox repository gets less testing than the enterprise one | Ring 0 is our test. The enterprise repository costs a yearly fee per box. That is a **money** decision for the operator, later (Q10). | + +--- + +## 7. Open questions the spike must answer + +| Q | Question | How to answer | +|---|---|---| +| Q1 | Can a box install an exact Debian version one week after approval? How often is it already superseded? | `apt-cache madison` on host and guest for security packages. Debian's security announcement history. Whether `snapshot.debian.org` is reachable and fast enough. | +| Q2 | Do the Proxmox and Docker sources keep older versions? | `apt-cache madison` for `pve-manager`, `proxmox-kernel-*`, `docker-ce`, `containerd.io`. | +| Q3 | What does a Docker engine update do to running containers, with and without `live-restore`? | On scratch guest 9202: step from one pinned version to the next; time the downtime. | +| Q4 | Does `proxmox-boot-tool kernel pin --next-boot` work on both demo hosts' boot setups (UEFI or legacy, GRUB or systemd-boot)? Does either box have a hardware watchdog? | On demo-hp, with the operator's go before each reboot. | +| Q5 | What does an interrupted `apt` run leave, and does the repair recover it? | Kill a run on 9202, after a snapshot. | +| Q6 | How far behind are the demo boxes today, per layer and per source? How long does catching up take, and what restarts? | `apt-get -s upgrade` (read only) first; then a real run on 9202 and on one demo host. | +| Q7 | Which Debian packages restart something big? | `needrestart` in list mode after a run. | +| Q8 | What else runs in the night window, and where does the OS leg fit? | Read the timers on host and guest. | +| Q9 | How is `cloudflared` installed and updated on the host? | Read the installer and the agent. | +| Q10 | What does the Proxmox enterprise repository cost per box per year, and what does it add? | The publisher's price page. Record only. The operator decides later. | + +--- + +## 8. Build order `[PROPOSAL]` + +Each step returns to the operator for go or no-go. + +1. **Spike** (measure Q1–Q10; no product code). +2. **Guest Debian, fast lane.** Lowest risk: a snapshot undo exists. +3. **Host Debian, fast lane** (no kernel, no Proxmox packages). +4. **Fleet view and alarms** (§5.7). +5. **Slow lane: Docker engine.** +6. **Slow lane: host kernel and Proxmox packages, with the reboot.** +7. **Later:** the Proxmox major upgrade (PVE 9 › 10), drilled on ring 0 first. + +--- + +## 9. Where the rest lives + +- The finding: **R-812** (`backlog/OPEN-ITEMS.md`). The intention: **R-808** (`backlog/ROADMAP.md`). +- Related: **R-604** (a per-customer floor hides a box from global raises; the same risk applies to an + OS-release floor), **R-530** (agents update only by a signed job per box). +- App updates: `09`. Backups and the night chain: `07` §6.1. The agent's permissions: `03` §3. \ No newline at end of file