# 11 — Operating-system updates: the host, the guest and the Docker engine > **How to read this document.** Where a statement is marked, it is marked like this — the same wording as > `07-backup-architecture.md:11-17`, carried here on 2026-10-05 (R-376, the three documents written after the > 2026-08-22 pass): > > - **[DESIGN]** — a decision taken. Not derived from code; the code may not implement it yet. > - **[FACT]** — an observed property, carrying a `file:line`, a live command output or a citation. > > **An unmarked statement means "not yet classified", never "observed"** (R-376). > | | | > |---|---| > | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1); steps 2–6 are BUILT by §8's own list (corrected 2026-10-24 — the ratification is still the operator's).** Ratification is Viktor's review, not an editor's. | > | **Written** | 2026-10-04, by the reviewer (project Claude), before any spike. | > | **Verified against** | felhom.eu `d07a1a9` · felhom-controller `99a1497` (v0.290.0) · felhom-agent `d766666` (v0.138.0) · hub v0.128.0 | > | **Freshness** | **CURRENT** as of 2026-10-04. The spike `TASK-backup-close-and-os-updates-spike-2026-10-04` adds measurements here as `[FACT]` and corrects every claim it disproves. Mark this file STALE when it falls behind what the product does. | > > **How to read this document.** Each statement has a label: > > - **[FACT]**: an observed property, with a `file:line`, a register row or an audit path. > - **[RULED]**: an operator decision, with its date. > - **[PROPOSAL]**: the reviewer's design. It is not decided and not built. > - **OPEN**: a question that nobody has answered yet. The spike measures it. Nobody guesses it. > > **Why this file exists.** No architecture document covered operating-system updates. `00` §G marks the > capability MISSING (added 2026-10-03). The finding is **R-812**. The roadmap intention is **R-808**. > **The register carries the work. This file carries the reasoning. The source is the truth.** > > **Spike corrections, 2026-10-04** (`audits/os-updates-spike-2026-10-04/`). Each is made in place: the reviewer's > text is struck (~~like this~~) and the measured text follows, labelled `[FACT]`, with its id. > > | Id | Where | What the spike changed | > |---|---|---| > | C1 | §2 | The agent may run `apt-get install` for TWO packages, not one (dnsmasq and wireguard-tools). | > | C2 | §5.3 | Debian keeps two versions, not one; for box packages no fix was replaced within 14 days in 3 months; `snapshot.debian.org` works from a box in seconds. | > | C3 | §5.2 | The lanes must follow the package's ORIGIN, not its name: 40 Proxmox-repository packages have ordinary names (ZFS, the Secure Boot shim, Ceph, corosync, chrony, CPU microcode). | > | C4 | §5.6 | `--next-boot` is NOT a one-shot on these GRUB hosts. The fallback works only after a boot that reaches userspace. Installing a kernel alone makes it the default. Only a software watchdog runs. | > | C5 | §5.6, §6 row 4 | Docker's `live-restore` keeps every container running across an engine update (measured). Turning it OFF again stops every container and starts none. | > | C6 | §5.6 | A guest snapshot works on LVM-thin (customer guests) but not on `dir` storage; the snapshot rollback itself is unmeasured; a backup-restore undo took 73 s. | > | C7 | §6 row 2 | A killed `apt` run does not recover by itself; the repair took ~5 s. | > | C8 | §1, §6 row 14 | `cloudflared` is not a host package: it is a container in the guest, pinned by the controller since June. It belongs with the controller's infrastructure pins, not this file's lanes. | > | C9 | §5.3 | The approved list must record what ring 0 RUNS healthy, not only what it installed that night. | > | C10 | §5.5 | Restore-tests and agent updates are not windowed; the host's `apt` timers install nothing today. | > | C11 | §5.2 | A Debian (fast-lane) update leaves PID 1, `lxc-start`, the Proxmox daemons and dockerd on the old library: its full effect needs a restart the fast lane does not do. | > | C12 | §6 row 9 | The two demo hosts differ by 7 packages, including the CPU microcode (AMD vs Intel) and Secure Boot (on vs off). | --- ## 0. In plain language A box runs three layers that we install and never update: the Proxmox host, the small Debian system inside the guest, and the Docker engine. App images are updated (see `09`). The layer under them is not. A box lives in a home for years, so this is a security gap. The proposal has two lanes. **The fast lane** applies Debian's security fixes automatically, but only the exact versions that already ran well on the demo boxes for 1–2 days. **The slow lane** covers the kernel, Proxmox and Docker. These cause restarts or big changes, so a person approves them one version at a time, and they run only at night. The tested versions are recorded automatically from what the demo boxes installed. Nobody keeps a hand-written list. --- ## 1. Scope **In scope.** - The Proxmox host on an **appliance** install: the Debian 13 base, the Proxmox packages, and the kernel. - The guest (the customer LXC): its Debian 13 packages. - The Docker engine inside the guest (`docker-ce`, `docker-ce-cli`, `containerd.io`). - How the household and the operator are told, and how a failed update is undone. **Out of scope.** - **App images.** `09` covers them (the ladder, the monthly same-tag re-test). - **The controller and agent binaries.** Their own self-update covers them (`03`, the self-update section; `09` R-608). - **DooPlex and ep0.** The operator updates them by hand (`runbooks/offsite-endpoint.md`). - **A BYO host** (the owner brought their own Proxmox). `[FACT]` The installer leaves its repositories alone: *"apt repo alignment skipped (byo — the owner manages repos)"* (`scripts/felhom-host-install.sh`, `align_apt_repos`). `[PROPOSAL]` On a BYO box we update the guest and Docker only, never the host. **[DESIGN, operator ruling D43 = A, 2026-10-24 — committed agent `c8d3c17`, unreleased]** The host's root-key-only sshd drop-in (D4, R-982) is an APPLIANCE-only file of the config bundle: written only when the root-owned install record says `appliance`; on a `byo` record nothing is written and a copy the bundle wrote earlier (this path, our first line) is removed; a box whose record proves nothing is left as it is (R-1257). **[FACT] 2026-10-25 (BYO test host, agent 0.164.0 → 0.165.0):** the removal came one bundle LATE — the bundle that brings the new wrapper is planned by the OLD one (R-1285). **[DESIGN — R-1285, agent `abcc1e6`, unreleased]:** right after every bundle install the agent asks the wrapper now on disk for a policy pass (`mode: "bundle-policy"`), which removes our appliance-only files from a BYO record (no signature: it only removes a file we wrote; a failing `sshd -t` puts it back). **[DESIGN — R-1285 follow-up, agent `b1bc63d`, unreleased]:** in the normal delivery the bundle job runs in the OLD process (measured on demo-hp 2026-10-26: the bundle 09:03:07–09:03:09, the restart into 0.166.0 at 09:03:10), so the new agent also runs the pass once right after its self-update commits (`selfupdate.Manager.SetOnCommitted`). Only a box coming from 0.164.0 can show it: 0.165.0's wrapper already applies D43 at plan time. - **The Proxmox MAJOR upgrade** (PVE 9 → 10). It is a later step of its own, drilled first (R-808 item 5). - **`cloudflared` and the other infrastructure images** (traefik, filebrowser). **[FACT] (C8)** They are containers in the guest, pinned in the controller (`internal/infra/infra.go:26`: `cloudflare/cloudflared:2026.6.0`, since 2026-06-11 — *`2026.9.3` from 2026-10-24; `2026.10.3` since controller v0.345.1, 2026-10-26 — Cloudflare's eMudhra notice, R-1300*) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release — the app-image question (`09`), not an OS package. The gap is R-838. --- ## 2. What a box runs, as measured | Layer | What it is | Source of packages | Evidence | |---|---|---|---| | Host | Proxmox VE 9.2 on Debian 13, LVM-thin | Debian mirrors + `pve-no-subscription` (enterprise repo switched off on appliance installs) | `[FACT]` `01` §2; `felhom-host-install.sh` `align_apt_repos` (~L2130–2136) | | Host kernel | The only kernel on the box. The guest has none. | `pve-no-subscription` (`proxmox-kernel-*`) | `[FACT]` LXC shares the host kernel (`01` §2: "one LXC/kernel/Docker daemon") | | Guest | Unprivileged LXC, `nesting=1,keyctl=1`, from `debian-13-standard_13.1-2` | Debian mirrors | `[FACT]` `felhom-agent/configs/build-golden.sh:67,101-103` | | Docker engine | `docker-ce`, `docker-ce-cli`, `containerd.io`, installed at golden BAKE time | `download.docker.com/linux/debian trixie stable` | `[FACT]` `build-golden.sh:108-125` | | Docker settings | `containerd-snapshotter: false`, json-file log caps (10 MB × 3 files), **`live-restore: true`** since the 3.1.0 bake (line 429 below; this cell said „No `live-restore`” until 2026-11-10). | baked `daemon.json` | `[FACT]` `build-golden.sh:193-203` | | Controller | A container in the guest. A Docker engine restart restarts it too. | Felhom registry | `[FACT]` `03` §1 | **[FACT] Nothing updates any of these layers today** (R-812, searched 2026-10-03). The installer says *"No upgrades are run — repo alignment only"* (`felhom-host-install.sh` ~L2135). A fresh install gets the Docker engine that was current when its golden was baked, and keeps it. ~~**[FACT] The agent may not run `apt` today, with one exception.** Its sudoers allowlist (`felhom-agent/configs/felhom-agent.sudoers`) holds one apt line: `apt-get install -y -q dnsmasq`.~~ **[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:68` (`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:133`) and `:228` (`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:659`). Nothing else. *(Line numbers corrected 2026-10-24.)* `apt` as root runs package scripts as root, so a broad `apt` grant is a full root grant. §5.4 proposes how to avoid that. --- ## 3. Operator rulings **2026-10-03.** R-808 is on the roadmap at P2: *"every box receives operating-system security patches on a schedule, and a failed update is undone."* **[RULED] 2026-10-04: the fast lane follows an approved list, with a 1–2 day wait.** Every update runs on the demo boxes first. The other boxes install only the exact versions that the demo boxes ran without trouble. The exact wait is set when the feature is built. **Rejected:** Debian's own `unattended-upgrades` with no wait. It is simpler and common, but a bad update would reach every customer at the same time. **[RULED] 2026-10-04: the off-site backup topic is closed first.** The same brief carries both topics, and the backup part runs first. The two-lane split (§5.2) is the reviewer's proposal. The operator's ruling above assumes it, but he has not ruled on it as such. --- ## 4. The constraints that shape the design 1. **Unattended.** Nobody is at the box. Nobody answers a question that `apt` asks. 2. **No screen.** If the box does not boot, the household sees only that nothing works. 3. **One kernel for everything.** A host kernel update needs a host reboot. A reboot stops every app. 4. **The controller lives in Docker.** A Docker engine update restarts the controller in the middle of its own work. So the controller cannot drive a Docker update. The agent, on the host, must. 5. **The host has no whole-system backup.** The guest has three tiers (`07`). The host has none. A host update can only be undone by installing the previous version again. 6. **Every update must already have run on a box we own.** This is the lesson of the update arc (`09` §3 decision 13: "the test decides"). 7. **Few packages.** Each extra package is one more thing to update and break. The guest is the Debian standard template plus Docker. Keep it that way. --- ## 5. The proposed shape `[PROPOSAL]` ### 5.1 Two rings - **Ring 0:** demo-felhom (N100) and demo-hp, on different hardware. They take every update first. demo-hp is disposable; the N100 is the lived-in box (`09` §3 decision 254) — it takes ring 0 through the product like any box, which is the point: a lived-in box meets an update before a customer does (`runbooks/target-selection.md`). - **Ring 1:** every other box. It takes only what ring 0 approved. ### 5.2 Two lanes | | Fast lane | Slow lane | |---|---|---| | What | Debian packages on host and guest, from `trixie-security` and the stable point releases, **except** the slow-lane list | Kernel (`proxmox-kernel-*`), Proxmox (`pve-*`, `proxmox-*`, `lxc-pve`, `qemu-server`, …), Docker (`docker-ce*`, `containerd.io`) | | Restart | A service restart at most. No reboot. | Kernel: a host reboot. Docker: every container restarts. Proxmox: its services restart. | | Approval | Automatic. A version is approved when ring 0 ran it and stayed healthy for the wait (1–2 days). | A person approves one version at a time, like a controller floor. | | Urgent fix | The operator can approve a version on the same day once ring 0 has run it. | The same. | | When | The night window (§5.5) | The night window. A kernel reboot only on a night the operator scheduled. | ~~Which packages count as slow lane is a list the spike checks (OPEN Q7). A Debian package that restarts something big (for example `systemd`, `libc6`, `openssh-server`) may belong in the slow lane too.~~ **[FACT] (C3) The lane must be decided by ORIGIN, not by name.** On demo-hp, 40 pending packages come from the Proxmox repository under ordinary names: `zfsutils-linux`, `zfs-zed`, `libzfs7linux`…, **`shim-signed` and friends (the Secure Boot loader)**, `ceph-common`/`librados2`…, `corosync`, `chrony`, `frr`, `amd64-microcode` (`partH/H2-proxmox-origin-debian-names.txt`). A name rule (`pve-*`, `proxmox-*`) would have put them in the fast lane. The fast lane is: origin `Debian` or `Debian-Security`, and nothing else. On demo-hp that selection was 108 packages and pulled in **zero** Proxmox packages. **[FACT] (C11) What a fast-lane run restarts, and what it does not.** Guest (9202, 49 packages incl. libc6): the packages' own scripts restarted postfix, journald, networkd; **dockerd, containerd, sshd, dbus, logind, cron** kept the old libc; no container stopped (13 samples). Host (demo-hp, 108 packages): dnsmasq, postfix, journald restarted; **systemd (PID 1), `lxc-start`, pveproxy, pvedaemon, pvestatd, pvescheduler, watchdog-mux, sshd, zed, chronyd** kept the old libc; both guests and the agent stayed up. So the fast lane is safe to run unattended, but a libc fix is only fully in force after a reboot (host) or a Docker restart (guest) — which are slow-lane acts. `[PROPOSAL]` the box reports "restart needed" (processes on deleted libraries) and the slow lane's next reboot picks it up. ### 5.3 The approved list (the "tested versions" record) Nobody writes the list by hand. It fills itself: 1. A ring-0 box updates. Afterwards it reports to the hub the exact `package=version` it installed, per layer (host, guest), with each package's origin (`Debian-Security`, `Debian`, `Proxmox`, `Docker`). 2. The box then reports health for the wait period: the agent, the controller, every app's health, and the guest's network. 3. When the wait passes with ring 0 healthy, the hub marks that set **approved**. It is one record: an **OS release**, with an id and a date. The hub stores it. The register does not. 4. A ring-1 box asks the hub for the newest approved OS release. For each package it has installed, if the approved version is newer, it installs that **exact** version. It never installs a version newer than the approved one. 5. Each box reports which OS release it runs, how many updates are waiting, whether it needs a reboot, and which packages it has that **no approved list covers** (see §6, edge case 9). ~~**OPEN Q1 is the weak point.** The design works only if a box can still download the approved version a week later. Debian's main and security archives keep only the newest version of each package. If Debian publishes a newer fix between approval and install, the approved version is gone. Options: the box waits for the next approval; or the box uses `snapshot.debian.org` (Debian's own dated archive, still signed by Debian); or Felhom runs a package cache. The spike measures how often this happens.~~ **[FACT] (C2) Q1 measured.** The live Debian archives keep **two** versions: the point-release one in `trixie` (main) and the newest in `trixie-security`; intermediate versions are gone (openssl: installed `u1`, main `u2`, security `u3`). Over 2026-07-04..10-04, 167 trixie security advisories; for the 517 source packages installed on a box, **no package got a second advisory within 2, 7 or 14 days** (the 3 within 2 days were chromium and webkit2gtk, not on a box). `snapshot.debian.org` answers a box: a dated index in **2.3–3.0 s**, a gone exact version (`openssl 3.5.6-1~deb13u1`) downloaded in **2.0 s**, Debian-signed. Proxmox and Docker keep many old versions (pve-manager 66, docker-ce 46). So with a 1–2 day wait the approved version is almost always still live; the rare miss is fetched from the snapshot taken at approval time. `[PROPOSAL]` each OS release records its approval timestamp; a box installs from its own sources, and only for a Debian package that is no longer there, from `snapshot.debian.org/archive//`. This is the operator decision in STATUS. **[FACT] (C9) Approve what ring 0 RUNS, not what it installed.** In the simulation on demo-felhom (`partI/demo-felhom-simulation.txt`), every one of the 108 host and 49 guest approved versions was installable and downloadable — but `curl`, `libcurl*` and `libssh2` were "not covered" in the guest, only because scratch guest 9202 ALREADY ran the newer version and so installed nothing. `[PROPOSAL]` step 1 reports the full installed `package=version` set after the run, and approval covers every version ring 0 runs healthy. ### 5.3.1 Test approvals end with the test — BUILT 2026-10-04 (hub v0.133.0, R-859) `[FACT]` An approval made while a TEST override (`OS_APPROVE_AFTER`, `OS_APPROVE_NIGHTS`, `OS_DOCKER_APPROVE_NIGHTS`) is active carries a `test` mark (amber on the System page). At every hub start WITHOUT an override, every test approval that no real approval has superseded is cancelled: never served again, ring-1 boxes bumped, one operator event `os_release_cancelled` each; what boxes installed stays; the ruled wait approves the same set again as a real release. A one-time backfill marked the AUTOMATIC approvals made under 24 h after first seen; the 2026-10-04 guest and host test approvals (which Tester 2 installed on its first night) were cancelled at 18:20 UTC. The operator's Docker button approval of that day stays in force (decided by CC unattended — operator may reverse). Runbook: `runbooks/os-updates-test-waits.md`. Evidence: `audits/r840-config-bundle-2026-10-04/partD/`. ### 5.4 Who runs it, and with what permission - **The agent runs every OS update**, for the host and for the guest (`pct exec`). The controller does not, because of constraint 4. - **The agent gets no general `apt` permission.** Like `felhom-selfupdate-guarded` (`03`, the self-update section), a small root-owned wrapper does the work. It accepts only an approved list and refuses everything else: - no package removal; - no downgrade, except the undo of §5.6, which an operator job signs; - no package that the box does not already have, unless the approved list records it as a dependency that the same update pulled in on ring 0 (a kernel update installs a NEW package name each time, for example `proxmox-kernel-6.x.y-z-pve-signed`, pulled by `proxmox-default-kernel`); - no package source other than the ones the installer set up; - non-interactive, and it always keeps the existing config file (`--force-confold`) and reports the conflict. - **Package signatures stay the publishers'** (Debian, Proxmox, Docker). The hub sends only names and versions. A broken-into hub can choose an older version or no version. It cannot make a box install a package that the publisher did not sign. ### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]` — *BUILT and installed since (§8.1); the agent's sudoers holds only the `--plan` line, no `--repair-only` (`felhom-agent/configs/felhom-agent.sudoers:337`); noted 2026-10-24* `felhom-os-apply` — root-owned (`0755 root:root`), installed by the installer beside `felhom-selfupdate-guarded`; the agent's sudoers gets exactly `/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json` and `… --repair-only`. For the guest it runs on the host and enters the guest with `pct exec --` itself, so the agent needs no `pct exec … apt` line. Built from what Parts G and H measured. **Input — one JSON plan file** (written by the agent, from the hub's approved OS release): ```json { "release_id": "os-2026-10-04-1", "approved_at": "2026-10-04T08:00:00Z", "layer": "host", // "host" | "guest" "vmid": 9201, // guest only "snapshot": "20261004T080000Z", // the snapshot.debian.org timestamp of approval (C2) "packages": [ {"name": "libc6", "version": "2.41-12+deb13u4", "origin": "Debian"} ], "allow_new": ["proxmox-kernel-7.0.14-20-pve-signed"], // slow lane only, signed operator job "lane": "fast" // "fast" | "slow" } ``` **Order of work:** (1) refuse checks below; (2) **repair first** — `dpkg --configure -a` then `apt-get -f install`, logging what it repaired (C7); (3) `apt-get -s install` of exactly `name=version` for every package that is installed AND older; (4) refuse if the simulation would remove, downgrade, add an unlisted package, or touch a package whose candidate origin is not the plan's; (5) download — from the box's own sources, or for a Debian version no longer there, from `snapshot.debian.org/archive//` with a temporary sources list it deletes after (C2); (6) install with `DEBIAN_FRONTEND=noninteractive APT_LISTCHANGES_FRONTEND=none -o Dpkg::Options::=--force-confold -o Dpkg::Options::=--force-confdef`; (7) `apt-get clean`; (8) report. **Refusals** (each exits non-zero with one line `os-apply: REFUSED: ` and changes nothing): | # | Refuses when | |---|---| | R1 | the plan file is not under `/var/lib/felhom-agent/os/`, not owned by the agent, or not valid JSON | | R2 | `lane` is `fast` and any package's origin is not `Debian` / `Debian-Security` (C3) | | R3 | `lane` is `slow` and the plan is not carried by a verified signed operator job (R-530's mechanism) | | R4 | the simulation removes any package | | R5 | the simulation downgrades any package (the operator undo is a separate signed op, §5.6) | | R6 | the simulation installs a package that is neither installed nor in `allow_new` | | R7 | a listed version is not downloadable from the sources the installer set up or the named snapshot | | R8 | free space on `/` (or the guest's rootfs) is below 3× the download size, minimum 500 MB (edge case 8). The download size is the `--print-uris` total WITHOUT `-s` (R-865, agent v0.144.0 — with `-s` it read 0 B) | | R9 | another apt/dpkg holds the lock, or the per-guest lane lock is held (a backup, a restore-test, C10) | | R10 | `layer` is `guest` and the vmid is not the box's own customer guest | | R11 | the plan names a package twice, or a version that is not a Debian version string | **Log lines** (to the journal, tag `felhom-os-apply`, and echoed for the agent to forward to the hub): ``` os-apply: START release= layer= lane= packages= os-apply: REPAIR configured= fixed= (always printed; 0 0 when nothing was half-done) os-apply: PLAN upgrade= already= not-installed= from-snapshot= download= os-apply: REFUSED: os-apply: CONFFILE kept (new version saved as .dpkg-dist) os-apply: DONE rc=0 seconds= upgraded= restarted= restart-needed= reboot-needed= os-apply: FAILED rc= step= — dpkg state: ``` `restart-needed` lists processes still mapping deleted libraries (C11); `reboot-needed` is yes when that list holds PID 1 or `lxc-start`, or a kernel was installed. ### 5.4.2 The config bundle: a box's root-owned files by a signed job — BUILT 2026-10-04 (agent v0.143.0, hub v0.133.0, installer 1.31.0, R-840) `[FACT]` Decision 96. Before it, the signed `agent_update` replaced only the binary; wrappers, units and sudoers lines reached an installed box by reinstall or by hand. - **One source of truth.** `BUNDLE_FILES` in `felhom-os-apply` is the ONE table of root-owned paths (22 then, **32** counted 2026-10-24 — plus the vzdump-partial-clean, thin-check, internal-place and priv-apply wrappers, the guest hook, the shared-parent unit, the GRUB one-shot pair and the sshd root-key-only drop-in; sudoers ×2, the five wrappers, the crash guard and its units + config, the agent and rollback units, the start-limit drop-in, the mgmt watchdog and its tmpfiles/units, the OOB belt's four files). `scripts/build-config-bundle.py` builds the bundle from it reproducibly; `release-agent.sh` publishes it beside the binary; the hub vouches its sha with the agent (exact-name lookup); the installer (1.31.0) installs it through the same code (`--install-bundle`, root only, refused through sudo). A test fails on a root path the installer names that the bundle lacks. - **The route.** Signed `agent_config_update` {agent_version, bundle_sha256}. The agent is a courier; the root wrapper re-verifies the signature against the root-owned signers file, the host binding (`os-trust.json`), the window and its own nonce, then the sha, every path (R16) and every content check before the first write: `visudo -cf`, `sh/bash -n`, Python compile, unit sections, no `RuntimeDirectory=` (G1), `User=felhom-agent`, `nft -c`, and that the route survives (the sudoers keeps the `--plan` line; the new wrapper keeps the bundle mode). Policies: replace / if-absent (`crash-guard.conf`, an operator setting) / oob (only on a box with the belt). Atomic per file, sudoers last, previous copies kept (last 3). Self-check: `visudo -c`, `sudo -l -U felhom-agent` lists the route, the new wrapper's `--self-check`, the self-update wrapper's usage, the crash guard's status equals `kernel.panic`; any failure puts every previous copy back. A newly installed crash guard is started (`enable --now`): `kernel.panic` for this boot, no reboot. Record `/etc/felhom/config-bundle.json`; the agent reports it (`system.config_bundle`, "none" when absent); the facts mode adds drift (files changed by hand). - **The trust root is not changed by a bundle** (R17, tested). A missing signers file is created only with the installer's pinned key, only after a job that key signed (pinned equal to the installer by a test). Signer rotation is a later, separate act (`04` §3). - **Bootstrap (corrects the brief).** The self-update wrapper cannot install a bundle (fixed sh, binary-only), and no signed job can write a root file on a box whose `felhom-os-apply` predates 0.143.0. Such a box needs ONE by-hand step (`scripts/felhom-bundle-bootstrap.sh`: only the new `felhom-os-apply`). Done on both demo boxes; Tester 2 needs the operator (`runbooks/config-bundle.md`). - **Visibility.** System page "Root files" column; alarm `os_config_bundle_behind` after 7 days (`OS_ALARM_BUNDLE_BEHIND_AFTER`; decided by CC unattended — operator may reverse). - **„Behind” is judged on the FILES, not on the bundle's label (hub v0.181.0, R-1392, `09` §3 decision 304).** The bundle carries its `agent_version`, so its sha256 moves with every agent release — also when `configs/` did not change and the release sends no bundle job (measured: 0.172.1 → 0.173.0, 32 files equal, two sha256s). When the box's bundle sha256 differs from the vouched one, the hub fetches both bundles from the package registry, accepts each only if its bytes hash to the sha256 already trusted (the vouched one; the one the box reports), and compares everything but the label. Equal → not behind: no amber, no alarm, the „behind since” mark is cleared. Anything it cannot read, verify or parse → the old verdict (behind). `hub/internal/bundlefiles`; pinned by `TestSame*`, `TestBundleAlarm_R1392_*`, `TestBundleCell_R1392`. **Not covered:** a box whose bundle version was pruned from the registry reads „behind” (true enough: it is older than the kept versions). - **Measured live** (`audits/r840-config-bundle-2026-10-04/partB/`): both demo boxes had every file equal to the release except `felhom-os-apply`; after the bootstrap the signed bundle wrote 0 of 22 (21 same, 1 setting kept), self-check ok, capability probe 71/71. demo-hp: a wrong-sha job refused (nothing changed); a bundle with one deliberate change wrote exactly that file; the 0.143.0 bundle undid it; a replayed job was rejected. The installer path on demo-felhom: 0 written, services active. 22 of 22 wrapper rules red-proved. - **Not a boundary yet — R-861.** The agent's sudoers already lets the agent user reach root without the operator key (a hookscript, a boot unit, the escrow self-test run as root, the self-update of its own binary). The trust-root rule is defence in depth until R-861 is closed. ### 5.5 When Inside the household's night window, after the backups: ``` W DB dump W+60m Tier 2 W+105m off-site → app updates (until W+5h at most) [W+2h, W+6h) whole-guest backup (agent) after it OS updates — guest first, then host ``` The OS leg starts **after the whole-guest backup has finished**, so the guest's newest full copy is minutes old. This is the opposite order to app updates, which run before the whole-guest backup (`07` §6.1, `09` decision 11). The reason: the whole-guest backup is the guest's undo. **[FACT] (C10) Q8 measured** (guest UTC; demo-hp W = 02:30): db-dump 02:30, tier-2 03:30, off-site ~04:15, whole-guest gate [04:30, 08:30), controller self-update 04:30, offsite-integrity 06:00. Host: `apt-daily` and `apt-daily-upgrade` run daily but install nothing (no `unattended-upgrades`, no `APT::Periodic`); `pve-daily-update` refreshes the lists daily. **Restore-tests are NOT windowed** — they run on a cadence at any hour (demo-felhom 10:38 daily, demo-hp 16:43 and 22:46); agent updates arrive by signed job at any hour. `[PROPOSAL]` the OS leg takes the same per-guest lane lock the restore-test and the whole-guest backup take, rather than a clock slot. ~~**OPEN Q8:** what else runs then. The controller's self-update (default 04:30, and after any hub report when a floor is above it, R-608), the agent's self-update, the restore-tests, and PBS jobs. The OS leg must never overlap a backup, a restore-test or a self-update.~~ ### 5.6 How a failed update is undone "Rollback" is not used (`09` §4). The shapes: | Layer | Undo | Limit | |---|---|---| | Guest packages | Restore the guest snapshot taken just before the update | It also undoes app data written after the snapshot. Use it only inside the health window, before apps have written much. After that, install the previous version (needs OPEN Q1). **[FACT] (C6)** Customer guests are on LVM-thin and can snapshot; scratch 9202 (`dir` storage) cannot (`snapshot feature is not available`), so the measured undo was a whole-guest backup + restore: **73 s down**, all apps healthy, libc back. The snapshot rollback itself is unmeasured (R-837). | | Docker engine | Install the previous version (Docker's repository keeps old versions — OPEN Q2) | Every container restarts again. **[FACT] (C5)** Docker keeps 46 `docker-ce` versions. A step (either way) without `live-restore`: 6 of 6 containers restart, the app silent **26.5–30 s**, healthy at +42–45 s. With `live-restore` on: **0 restarts, no gap**, also across a containerd step. But a restart that turns `live-restore` OFF stops every container and starts NONE (`unless-stopped` ignored) — on 9202 they stayed down until restarted by hand; `systemctl reload` turns it on but not off. | | Host packages | Install the previous version | Only if the source still has it (OPEN Q1/Q2). **[FACT]** `rsync` back to its pre-update version: `not found`; `libpng16-16t64` back to the point-release version: worked. A Debian undo needs the snapshot archive (C2). **[FACT, 2026-10-04]** Proved by hand: `runbooks/os-updates-host-undo.md` (demo-hp, `tzdata` back one version from `snapshot.debian.org`, held, released). Use the NEW version's `first_seen` as the timestamp when there is no previous host release; a package that pins its siblings (`eject` → `libmount1 =`) goes back only with them. | | Host kernel | ~~Boot the previous kernel. Proxmox can boot a new kernel **once** (`proxmox-boot-tool kernel pin --next-boot`). If that boot fails, the next boot uses the old kernel again. Make the new kernel permanent only after a healthy boot.~~ **[FACT] (C4)** Both demo hosts boot UEFI + GRUB (no proxmox-boot-tool ESPs). **Installing a kernel makes it the GRUB default at once.** `--next-boot` on GRUB writes an ordinary `GRUB_DEFAULT` + `update-grub`; `proxmox-boot-cleanup.service` clears it only once a boot reaches userspace. Measured on demo-hp (old kernel pinned permanently FIRST, new pinned for next boot): boot 1 → `7.0.14-20-pve`, healthy, 60 s; boot 2 → back on `7.0.2-6-pve`, healthy, 76 s. **So the fallback works after a boot that succeeds; read from the code, a kernel that hangs before userspace stays the default on every power cycle.** ~~`[PROPOSAL]` use GRUB's own one-shot (`GRUB_DEFAULT=saved` + `grub-reboot`) with the old kernel as the saved default — unmeasured (R-836).~~ **[FACT, 2026-10-04, demo-hp, operator's word before each reboot] GRUB's one-shot is NOT a one-shot here either.** With `GRUB_DEFAULT=saved` (old 7.0.2-6 saved) and `grub-reboot` 7.0.14-20: boot 1 → 7.0.14-20, **Secure Boot ON, booted fine** (signed kernel, shim → GRUB); but `/boot` is ext4 on LVM, GRUB cannot write its environment block there (`grub-reboot` warns so itself), `next_entry` was never cleared, and boot 2 with no command → **7.0.14-20 again**. A kernel lane needs a writable env block (the ESP) or a userspace "boot good" step — R-836. demo-hp left on 7.0.14-20, saved default 7.0.14-20, both kernels installed (`audits/os-host-lane-2026-10-04/partE/`). | ~~If the new kernel hangs, someone must switch the box off and on. The spike checks whether a hardware watchdog can do that (OPEN Q4).~~ **[FACT]** Only `softdog` runs (loaded by `watchdog-mux`); it cannot rescue a kernel that never boots. demo-hp has an AMD FCH whose `sp5100_tco` driver ships but is not loaded; untested. A hang still needs a person. **[FACT, 2026-10-04]** `sp5100_tco` is blacklisted by the Proxmox kernel package; loaded by hand it answers (`SP5100 TCO timer`, 60 s, inactive, nowayout 0 — read from sysfs, never opened, so never armed; unloaded). `kernel.panic = 0`: a panic leaves the host stopped (R-851). | ### 5.7 Telling people - **Operator:** a hub event for each update and each failure; a fleet view showing each box's OS release, how far behind it is, and whether it needs a reboot. A box that is more than N days behind the newest approved release raises an alarm (`08`). - **[FACT, added 2026-10-24 — hub v0.169.0] The System page's „Upload line" cell** shows the best of the last SUCCESSFUL weekly line test (R-1247), amber when it is older than 15 days or was never measured after an attempt — judged on `last_ok_at`, never on the attempt time (`hub/internal/web/system.go` `lineStaleAfter`, `lineTestCell`). - **[FACT, hub v0.132.0] The System page** (`/system`, R-852, decision 89): one row per box — ring and switch with buttons, the tunnel, host Proxmox / running and next-boot kernel / Debian / release / pending / not covered / held / reboot needed / `kernel.panic` / oops / crash restarts / the guard, guest Debian / release / pending / restart needed, Docker engine / containerd / live-restore / release, the last leg — and above it the releases, what ring 0 runs, "Approve now" and "Approve Docker set". Colours are the alarm thresholds (decision 94). Hosts shows Proxmox / kernel too. **Layout from 2026-10-10 (hub, unreleased at writing):** the page opens with „Needs attention" (one line per amber or red box, the reasons in plain words, from the same colours) and „Waiting for you" (one card per kernel / Docker / Proxmox set the button may approve now, with the version and the ring-0 evidence). The boxes follow in one narrow table whose rows open to every value above; release ids, cancelled approvals, what ring 0 runs, the floors, the crash guard and root files, and „Approve now" (which asks first) sit in a closed „Details". Same data and buttons. - **Household:** one line on the timeline in both languages, informal voice: what was updated and whether the box restarted. Telling households in advance that the box may restart at night is a **promise to users**. That is the operator's decision when the slow lane is built. ### 5.8 The Docker engine slow lane — BUILT 2026-10-04 (agent v0.142.0, hub v0.132.0) `[FACT]` **As built** (evidence `audits/os-docker-crash-2026-10-04/partB/`; decisions 87, 93): - **live-restore ON**: the golden bakes it (`build-golden.sh` 3.1.0, fail-closed assertion); an installed box gets it once by the wrapper's `live-restore-on` (merge into daemon.json + `systemctl reload docker`). Measured: the same container ids after (9202 6/6 by hand — R10 refuses a scratch guest by design —, demo-hp 24/24, demo-felhom 5/5). - **The step** runs in the night leg after a healthy guest and host step, **ring 0 only**, `select pending-docker`, allowed by the wrapper only with the root-owned `ring0_slow_lane` mark. **Ring 1 and every undo** only through a signed `os_docker_step` the wrapper re-verifies itself (decision 93). Measured: 29.7.x → 29.8.2 on both demo boxes, every id kept; a signed undo to 29.7.2 on demo-hp (41 s, ids kept) and back; demo-felhom as ring 1 by a signed job; a replayed job refused by the agent (nonce). - **Approval**: the hub never approves a Docker set automatically; the System page's button works after every ring-0 box ran the set in 2 healthy night Docker steps (`OS_DOCKER_APPROVE_NIGHTS` TEST override, logged). An approval nudges no box. Undo: `runbooks/os-updates-docker-undo.md`. - **The engine must report a memory kill (decision 157, agent v0.150.0 + hub v0.140.0, 2026-10-06).** After a Docker step the wrapper runs `oom_check`: a throwaway container from the image the running controller uses (`--pull never`, `--network none`, no volume, label `felhom.oomcheck=1`, 64 MB cap) asks for one 200 MB block; pass = `OOMKilled=true` AND the `oom` event; the container is removed whatever happens; the result rides the report as `oom_check`. The approval button waits until every ring-0 box reported a PASSING check with the set, and refuses while any report of the set carries a failed or errored one. **Measured by hand on demo-hp's guest (29.8.2):** `OOMKilled=true`, exit 137, the event `create attach start oom die` — but only when the events window ends a second AFTER the run (a window closed in the same second missed it); the wrapper waits 2 s and reads to epoch + 1 (`audits/readback-2026-10-07/F/`). **The design as written before the build:** Built from C5 (measured) and R-835. **The one decision it needs is in STATUS: `live-restore` on, fleet-wide.** - **What moves:** `docker-ce`, `docker-ce-cli`, `containerd.io`, `docker-buildx-plugin`, `docker-compose-plugin`, `docker-ce-rootless-extras` in the customer guest — today the six "not covered" packages on every box. One approved **engine set** at a time, like a controller floor: the operator approves it (slow lane, §5.2), after ring 0 has run it for at least 2 nights healthy. Never two steps in one night. - **Precondition: `live-restore` ON.** Without it an engine step restarts every container — 26.5–30 s of silence, healthy at +42–45 s (C5). With it: 0 restarts, no gap, also across a containerd step. Turning it ON is safe (`systemctl reload docker` applies it without a restart, C5); turning it OFF later by a plain restart stops every container and starts none (R-835) — so it is turned on once, by the golden and by a one-time fleet step, and never turned off by the lane. - **Who and when:** the agent, through the same wrapper (`lane: slow`, refusal R3: only inside a verified signed operator job, R-530's mechanism), in the guest, after the guest and host fast-lane steps, under the same heavy-op gate, on a night the operator scheduled. Debian origin rule replaced by "origin `Docker CE`, exactly these names". - **Health:** the guest rule (§8.1) plus `docker version` reports the approved engine, and every container running at the start is running with the SAME container id (proof that `live-restore` held). A changed id is `health_failed` even if the app is healthy — it means the households' apps restarted when they should not have. - **Undo:** install the previous engine set (Docker's repository keeps 46 versions, C2) — by an operator job, with `live-restore` still on, so the undo is also restart-free. - **Not covered here:** the golden's own engine (baked weekly; a new golden carries the approved set), and BYO hosts (the guest is ours on both, so the lane applies there too). ### 5.9 A crashed host restarts, with a limit — BUILT 2026-10-04 (agent v0.142.0, installer 1.30.0) `[FACT]` Decision 88 (R-851): *"Yes, but maybe not indefinitely."* Evidence `audits/os-docker-crash-2026-10-04/partC/`. - **Measured first (demo-hp, the operator's word before each crash):** `kernel.panic = 10` + `echo c > /proc/sysrq-trigger` → the box restarted by itself in 54 s, on the same kernel (the saved default), the agent up 18 s after the boot. `kernel.panic` set by `sysctl -w` is gone after the restart (back to 0) — it must be set at every boot. - **The crash signal** (decision 90): `efi_pstore` is on, yet a real panic saved NOTHING; the journal and `last` show only "no shutdown". The guard uses a **clean-stop marker** (the unit's ExecStop at every orderly shutdown); a boot without it followed a crash, a power cut or a hard reset — counted alike. - **The guard** (`felhom-crash-guard`, early boot unit + hourly re-arm timer): armed → `kernel.panic = 10`; the 2nd unclean boot within 60 minutes TRIPS it (`kernel.panic = 0`), so **the 3rd crash within the hour leaves the box off** (decision 92 — the operator's words); re-arms after 24 h of normal running or `felhom-crash-guard rearm`. A crash before the unit runs (very early boot) leaves the box off: the safe side. `panic_on_oops` stays 0; an oops is reported (decision 91). - **Telling people:** each unclean boot = an operator mail (`host_crash_restart`) + the household's line; the trip = an alarm (`host_crash_guard_tripped`); re-arm and oops announced once (`08` §6.3). The System page shows the guard. - **Measured live:** crash 1 → back in 54 s; crash 2 → back in 53 s, guard tripped; crash 3 → **stayed off** until the operator switched it on; the hub mailed the trip (15 min after the boot — R-853); re-armed by hand. --- ### 5.10 The Proxmox package lane — BUILT 2026-10-07 (agent v0.151.0, hub v0.142.0), proven on demo-felhom (R-812 option A, `09` §3 decision 163) `[FACT]` **What it updates:** the host's Proxmox USERSPACE packages (pve-manager, qemu-server, pve-container, lxc-pve, libpve-*, proxmox-widget-toolkit, the ceph client libraries …) — the 77–78 packages behind on the demo boxes on 2026-10-07. **What it never touches:** a kernel, boot, firmware or microcode name (`HOST_SLOW_RE`: `proxmox-kernel*`, `proxmox-default-kernel`, `pve-kernel*`, `pve-firmware`, `shim*`, `grub*`, `systemd-boot`, `*-microcode`, `efibootmgr`, and — read 2026-10-24 in `felhom-os-apply` — `linux-(image|headers|kbuild|modules|base)*`, `firmware-*`, `proxmox-secure-boot-support`) — that is R-836's lane (decision 164). No reboot. - **Wrapper** (`felhom-agent/configs/felhom-os-apply`): layer `pve`, lane `slow` only. Origin `Proxmox Debian Repository` only (R2; a Debian package in a pve plan or simulation is refused); R14 on every name in the plan and the simulation; no removal (R4); no undo (R5 — put a version back by hand); a NEW package only from `PVE_NEW_ALLOW` (`proxmox-firewall-data`, the one new userspace package a full upgrade adds on demo-felhom); an appliance only (R12); authority R3 — a signed `os_pve_step` (op bound to the exact package list, host, time window, nonce) or the root-owned ring-0 mark (`ring0_slow_lane`). Ring 0 selects `pending-pve`: every installed Proxmox-origin package with a pending upgrade. The report carries `pve_manager` (pveversion after the step). Tests: `PVELane` (17). - **Agent** (`internal/osupdate`): ring 0 runs it in the night leg AFTER a healthy host step (an appliance; a Docker step's outcome does not gate it); ring 1 never in the night leg — only a signed `os_pve_step` (`PVEStepExecutor`, under the heavy-op gate). Health (`PVEHealthVerdict`) = the host rule (§8.2) + every running container keeps its id (the household's apps were not restarted) + pveversion reads the pve-manager the step installed. - **pmxcfs** (`internal/pvegate`): pve-cluster's postinst restarts pmxcfs (the filesystem behind /etc/pve). While a pve step runs, the agent's own /etc/pve writes WAIT — every non-GET Proxmox API call (`Client.doBody`) and every root CLI that writes /etc/pve (`ExecRunner.RunStdin`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile). The step first waits for writes in flight; after 2 minutes it gives up and does not run (`failed`). Backups and restore-tests are already out (the heavy-op gate). - **Hub** (`osupdates`): layer `pve` — never auto-approved. The candidate is the Proxmox userspace set every ring-0 box agrees on (kernel names left out). The operator's **„Approve Proxmox set"** button on the System page appears only when every ring-0 box ran the set in `DockerNightsEffective` (2) healthy night steps since it was first seen and none was unhealthy — „healthy" is the box's own verdict above (no memory-kill check; that is the Docker engine's, R-528). An approval nudges no box: ring 1 takes the set only by a signed `os_pve_step` (signed per box, as `os_docker_step`). - **Proven live 2026-10-07** on demo-felhom (ring 0, a signed `os_pve_step`): 65 packages in 70 s, pve-manager 9.2.2 → 9.2.21, healthy, every container kept its id, the hub logged the report (`audits/day-2026-10-07/B/RESULT.md`). - **Not yet:** the undo runbook (by hand: `apt-get install =` — Proxmox keeps 30–66 old versions, C2). ### 5.11 The kernel lane — BUILT 2026-10-07 (agent v0.152.0, hub v0.143.0; R-836, `09` §3 decisions 164, 171, 172) `[FACT]` **The ruling (decision 172):** design option A (a one-shot flag on the ESP) with option C (lockup → panic) on the one-shot entry. A box may restart at night for a kernel update; **the household is mailed the day before** with what to do if the box is not back by morning; each kernel set is approved by the operator after ring 0 ran it; **a frozen new kernel that needs a person's power cycle is accepted** for the first customers. Why A: measured in the spike on the Tester 1 VM, demo-felhom and demo-hp (Secure Boot on) — the ESP flag and UEFI `BootNext` both boot a kernel once and fall back after a panic; a hardware watchdog armed during the reboot never fired on any of the three (`audits/kernel-spike-2026-10-07/DESIGN-kernel-lane.md`). A writes nothing to firmware. **The boot side** (config bundle, two GRUB generators): `/etc/grub.d/01_felhom_oneshot` reads `felhom_next` from a GRUB env block on the ESP (`EFI/felhom/oneshot.env`), clears and saves it BEFORE the menu, and — only if it names an installed kernel — boots that kernel's one-shot entry. `/etc/grub.d/42_felhom_oneshot` gives each installed kernel an entry `felhom-oneshot-`: the normal entry plus `softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10` (option C; sorted after `10_linux`, so never entry 0 and never the default). A panic on the one-shot restarts the box in 10 s into the default — the old kernel. Without a vfat ESP both print nothing. **The default** is the kernel the box RUNS, pinned in `/etc/default/grub.d/zz-felhom-kernel-default.cfg` before the install and proved from grub.cfg after it (an install would otherwise make the newest kernel the default — R-836's defect). Only a healthy one-shot boot moves it. **The root wrapper** (`felhom-os-apply`, layer `kernel`, lane slow, an appliance, authority = a signed `os_kernel_step` or the root-owned ring-0 mark): `apply` STAGES (installs the set — the series meta-package, at most one new kernel image, the boot helper and firmware — pins the default, writes the flag; never reboots); `kernel-reboot` reboots a staged step; `kernel-boot` says what became of it after a boot; `kernel-good` moves the default; `kernel-revert` reboots ONCE into the old kernel; `kernel-cancel`; `kernel-status`. Refusals R20 (the box cannot do a one-shot: not UEFI, no vfat ESP, a separate /boot, GRUB without fat/loadenv, the generators missing, a hand pin), R21 (the crash guard tripped or saw an unclean boot within its window), R22 (the step's phase does not allow it; never two steps within 20 h), R23 (not exactly one newer kernel, or not the one signed / told). State `/var/lib/felhom-kernel/state.json`. **The night slot** (agent): the kernel step ENDS the night leg — after the host step (and the Proxmox step when one ran) ended healthy, after the whole-guest backup (the leg runs only after it), under the same heavy-op gate, trigger `night` only, at most one per night. Ring 0 stages the pending kernel and reboots; ring 1 reboots only a kernel a signed `os_kernel_step` staged by day (the job never reboots). The hub hears `staged` before the reboot. **"Boot good" and the self-revert** (agent, at every start): on the new kernel the agent judges the boot for **20 minutes** — the host health rule (§8.2: the Proxmox daemons and the agent active, the customer guest running and its own rule passing, the tunnel running) AND the box reached the hub (the `judging` report itself). Healthy → the new kernel becomes the default (`applied`). Not healthy by the deadline → `health_failed`, then ONE self-revert into the old kernel (`self_reverted` after it boots; a revert that comes back on the new kernel is `revert_failed` and never retried). The wait, measured 2026-10-07 by a plain reboot (`audits/kernel-lane-2026-10-07/B/`): every container healthy 68 s after the reboot on demo-felhom (the hub reached at 63 s; Tailscale back at 53 s — the spike's > 6 min did not recur) and 272 s on demo-hp (the hub at 189 s). A box that never comes back raises the hub's existing `host_stale` (45 min, `alerting.stale_threshold`) and `host_down` (90 min) to the operator. **The crash guard** (§5.9): the step's reboot and the self-revert are orderly (the clean-stop marker), so a step adds at most ONE unclean boot (a panic before userspace adds none: the planned reboot's marker is still there); with R21 the step starts only with none in the window — the box cannot reach the 3rd unclean boot that leaves it off. Pinned by `KernelStepCannotLeaveTheBoxOff`. **The household mail** (hub): a box is DUE when ring 0 has a pending kernel (its host report's `proxmox-kernel-X.Y` upgrade) or a ring-1 box runs a kernel a signed job staged, and no step for that kernel has ended (`applied`, `fell_back`, `health_failed`, `self_reverted`, `revert_failed`, or a refusal R20/R23/R3/R12 — the operator decides; never retried by itself). Its household gets ONE mail (`mail.kernel.*`, its language, informal; to the registered address) between 09:00 and 20:00 Budapest time, at most once per 20 h and three times per kernel. The box's `os_update.kernel` block says `tonight` only within 24 h of a mail the mail service ACCEPTED — **no mail, no step**. Operator events `os_kernel_step` and `os_kernel_notice`; the household's timeline gets the usual "security fixes installed" line after `applied`. **Approval** (hub): "Approve kernel set" on the System page offers the **newest kernel version that every ring-0 box has booted healthily at least once after a NIGHT stage** — a box may have moved on since (operator ruling 2026-10-10, `09` §3 decision 195; hub v0.145.0). Per box and per kernel only the newest ended step counts: a kernel whose latest step fell back, self-reverted or failed is out of that box's set. *(Before 2026-10-10: every ring-0 box's newest ended step had to be `applied` for the SAME kernel.)* The approved set is the series meta-package and the signed image. An approval nudges no box: ring 1 takes it only through a signed `os_kernel_step` (stage), then its own told night. **The System page** shows per box: the running kernel, the next boot, the default (amber while a one-shot flag names another kernel), and the kernel step (the newest result, the kernel due, whether the household was told for tonight). **Not measured:** option C itself — the Proxmox kernel ships no lockup test module (`CONFIG_TEST_LOCKUP` is not set on 7.0.14-20), so a soft lockup turning into a panic is recorded as unmeasured (no tool was built). A true dead freeze needs a person (accepted, decision 172). Evidence and proofs: `audits/kernel-lane-2026-10-07/`. ### 5.11.1 Removing old kernels — BUILT 2026-11-14 (agent v0.174.0; R-1368, `09` §3 decision 286) `[FACT — built]` **The ruling** (`09` §3 decision 286): the box may remove old kernels by itself, as its own step with its own narrow rule. **Rule R4 does not change**: an update plan that removes a package is still refused. Why a step is needed: the kernel step installs each image by name, so apt marks it "manual" and apt's own clean-up never takes it; and nothing runs that clean-up on a box. A ring-0 box got 7 kernels in 37 days, about 1 GB each (R-1368). **The rule.** Keep the running kernel; the last kernel that booted well before it (the step's `from`); the newest installed kernel; and every kernel the lane still names — the step's `to` and `default_before`, the default in `grub.cfg`, the pinned default file, the one-shot flag, an operator's boot pin. Remove every other installed kernel image package, by its exact name, in one `apt-get purge`. **It refuses the whole clean-up** — it never removes a part of the set — when: | | | |---|---| | P1 | a kernel step is in a phase that has not ended (`staged`, `oneshot`, `judging`, `reverting`), or in a phase it does not know | | P2 | the one-shot flag names a kernel | | P3 | the pinned default and `grub.cfg`'s default do not name the same installed kernel | | K1 | a keep-kernel is in the removal set, or the running kernel is not an installed package | | N1 | a name in the removal set is not an installed kernel image package | | S1 | apt's simulation of exactly that set fails, or would install anything | | S2 | the simulation would remove anything that is not in the removal set | K1 is the rule's own check and must stay its own: asked to remove the running signed kernel, apt does not refuse — it plans to put the unsigned image in its place (control A of the spike). **When it may run.** Only when the newest kernel step ended `good` and the box runs its newest installed kernel — that is "booted well on its newest kernel" (decision 286). Never in the night of a kernel step: the step's judging, and a self-revert into `from`, must be over first. In the night's order it belongs where the OS leg is (§5.5): after the whole-guest backup. It needs no reboot and stops no guest. A box that fell back or reverted keeps all its kernels until a later step ends `good`. **As built** (`felhom-agent` `configs/felhom-os-apply`, layer `kernel`, mode `kernel-cleanup`; the agent's `internal/osupdate/kernel.go`): - **The plan names nothing.** The wrapper computes the set from the box; a plan that carries packages, a select, a signed job or an expected kernel is refused (C1). `dry_run: true` is the same code path: it reports the keep set, the removal set and what it would do, and changes nothing (`felhom-agent --selftest=kernel-cleanup-dry-run`). - **Who may:** an appliance (R12, the root-owned install record) with the **root-owned ring-0 mark** (`/etc/felhom/os-trust.json`), read by the wrapper itself — never the agent's config, never the hub's ring. A box without the mark answers „not run". Ring 1 needs an operator ruling first. - **„Not run"** (exit 0, nothing changed, a log line): no ring-0 mark; no kernel step yet (no state file); the newest step ended in another phase than `good` (`fell_back`, `self_reverted`, `revert_failed`, `cancelled`); the box does not run its newest installed kernel; a step was staged, booted or ended within the last 20 hours. - **Refused** (exit 2, nothing changed): the table above, each with its own code; the wrapper repairs dpkg first (§5.4.1), so a purge that a power cut interrupted is made whole and the rule is computed again from the box. - **K1 reads the box a second time.** The removal set is built from the keep set; K1 then reads the running kernel, the step's `from`/`to`/`default_before`, both defaults, the flag and the boot pin again and refuses a name that any of them still needs — also before apt's simulation. - **After the purge** the wrapper proves that the default in `grub.cfg`, the pinned file, both generators and the step's state file did not change, and that every kept kernel is still installed. Anything else is `failed`, said loudly; nothing is repaired in code. A left-over `/lib/modules/` or `/boot` file is reported, never deleted. - **When (the agent):** the night leg's own step, after a healthy host step (and Proxmox step, when one ran), the last call of the night, never in a night with a kernel step (due, staged or reported), never in a debug pass, never on a BYO host. Not retried the same night. - **The sudo rule did not change:** the wrapper is reached by the one `--plan` line. - **What the hub shows (hub v0.181.0, no hub change in this release):** a clean-up that removed kernels arrives as a kernel-layer report with outcome `cleaned`; the System page's „Kernel step" cell then reads „cleaned