admin/felhom.eu
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

off-site closed (R-833, R-834 closed; R-95 dated check; R-726 options) + OS-update spike: 11 corrected C1-C12, wrapper draft, answers; R-835..R-839
gates / gates (push) Successful in 33s
Details

Browse Source 
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
This commit is contained in:
authentik Default Admin
2026-10-04 09:54:31 +02:00
parent 85e3e05e10
commit 1b74ddc0c9
80 changed files with 7659 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/architecture/00-capability-map.md
+1 -1
View File
@@ -232,7 +232,7 @@ likewise silent. Evidence: `audits/DRILL-r361-2026-08-22/evidence/06-part3-decis
| **The hub reports LOSS OF VISIBILITY into either off-site store (not just how full it is)** | hub **v0.106.0** (R-339) | **IMPLEMENTED — deliberately NOT proven-live** | Both box checkers count consecutive failed fetch windows and emit `pbsdr_box_unreachable` / `offsite_box_unreachable` (severity `warning`) past a default 3 windows (≈30–45 min), each with a paired `*_recovered` all-clear routed via `recoveredPairedDownTypes` — required because the recoveries are severity `info`, which `severityNotifies` drops. Scopes stay customer-less (`pbsdr-box` / `pool-box`) → operator channel only. Fill logic untouched: a degraded read still drives no band transition. Evidence: `internal/monitor/box_reachability_test.go` + the cross-package wiring test in `internal/notify/`, which asserts an actual operator mail rather than a map entry. **Filed BECAUSE of a measured gap**, not a hypothesis: the 2026-08-18 ep0 outage ran 9 h 37 m with the hub silent | **The gap that remains is R-340**, and it is not small: the ep0 read is the `usage` op, which rides the LOCAL API daemon — the daemon that incident explicitly cleared — so this check would have shown GREEN for that entire outage. It closes "ep0 is unreachable as a host"; it does not close what actually happened. **No live or constructed outage has exercised the emit path**, and one cannot be manufactured against ep0 (Tier 2, protected) |
| Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | 07-13 closing bundle | |
| Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | 07-13 | |
| Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | — | **MISSING** | `felhom-host-install.sh:2133-2136` ("No upgrades are run") | Nothing runs them after install → finding R-812, intention R-808 (added 2026-10-03). Design: `architecture/11-os-updates.md` (NOT RATIFIED, 2026-10-04) |
| Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | — | **MISSING** | `felhom-host-install.sh:2133-2136` ("No upgrades are run") | Nothing runs them after install → finding R-812, intention R-808 (added 2026-10-03). Design: `architecture/11-os-updates.md` (NOT RATIFIED, 2026-10-04). **Spike done 2026-10-04** (`audits/os-updates-spike-2026-10-04/`): measured, nothing built — still MISSING |
| **An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps** | controller **v0.259.0** + hub **v0.119.0** + ISO 1.29.0 + the whole catalog | **PROVEN-LIVE on 0.258.0 with one blocker; THE BLOCKER IS FIXED AND PROVEN, THE WALK IS NOT REPEATED** | `audits/DRILL-first-hour-en-0258-2026-09-20.md` — a fresh install 2026-09-20, one intervention (R-494), stop rule not reached. Then `audits/i18n-closing-2026-09-21/live/` — the three blockers fixed and each proven on a live box or in the operator's inbox: the claim page answers English through the real cookie path; the Backup page's tier names follow the language; and the setup mail carries **four plain-ASCII English words** where the drill's carried `képző-szkítia-ásatás`, one day apart in the same inbox. | **R-596, R-597 and R-598 are CLOSED.** What this row still does NOT claim: **the fixed journey has not been walked end to end by a stranger on a fresh install.** Three fixes proven at the endpoint are not an hour proven by a person, and this project's own rule is that fixes are not a journey (see the recovery-journey row). **Also not walked:** the recovery code (needs ep0), backup/restore/remove/power-cut (proven 2026-09-14), and the two Backup-page *warnings* themselves — guest 9201 is healthy and a healthy box renders none, so they are covered by handler render tests, not live. **Verdict: nothing known now stands between an English-speaking tester and their box — and that is a different sentence from "the walk passed".** |
| **A deletion of a customer's off-site history is NOTICED within a day** | hub **v0.111.0** (R-431) | **IMPLEMENTED — not yet PROVEN-LIVE** | 09-01 | `hub/internal/monitor/offsite.go` — third signal beside FILL and STALENESS. **On the hub deliberately:** a detector on the box is one the deletion can silence. Alarms when the reported count falls by more than HALF and by at least 5, guarded by `StatsKnown` (R-331), the declared `State` (R-204) and run success (R-100). **Threshold reasoned, not invented:** over 12 898 reports every decrease lands on ZERO and predates `stats_known`; in the 380-report `stats_known` window there are none. **ACCEPTANCE: 9 009 real points replayed → ZERO alarms** (`offsite_r431_test.go`, fixture committed). **What PROVEN-LIVE would need and this does NOT have:** a real drop observed on a live box producing a real mail — the live firing done at ship time was driven through the hub's own path with synthetic counts, which is an end-to-end delivery proof, not a proof that a genuine deletion is caught. |
documentation/architecture/07-backup-architecture.md
+10
View File
@@ -143,6 +143,16 @@ work, by design and by contract**. They are not customer-facing and are not goin
(`--selftest=bring-up -mode dr`), the host-loss plan builder, and the escrow-consume ceremony. Each
needs root on the host or a CLI flag; none is reachable from any customer surface.
**[DESIGN + FACT, 2026-10-04, R-834] A whole-guest restore BESIDE a live original never comes up as a second box.**
A restore that does not replace the box's own guest on a replaced host ends with `onboot 0` and no host-path bind
(`mp8` = the household's drives, `mp9` = the original's bootstrap); the DR route on a replaced host keeps its binds,
because there they are right. Per route: the restore-test sets `onboot=0` at create and gives mp8/mp9 throwaway
volumes (measured live on demo-hp, every poll); the DR bring-up REFUSES when the archive's source guest, or any guest
binding the drives, is still on the host (agent v0.139.0, proven live); the hand route is
`scripts/felhom-restore-beside.sh` (proven live); provisioning restores the golden, which has no binds. Pinned by
`TestRunBringUp_DRRefusesBesideALiveOriginal`, `TestRestoreTest_NoHostPathBindBesideTheOriginal` and
`scripts/test_felhom_restore_beside.py`. `audits/backup-close-2026-10-04/partA/`.
**[FACT] What is proven in Lane 2** (INV Parts G.1, G.3): whole-guest `pct restore` from both tiers
is proven with exact mount parity; the unattended restore-test is proven and currently running on
both boxes; a corrupted snapshot is proven to fail cleanly. **The DR bring-up path has never been
documentation/architecture/11-os-updates.md
+162 -17
View File
@@ -2,7 +2,7 @@
> | | |
> |---|---|
> | **Status** | **NOT RATIFIED — a PROPOSAL with one operator ruling.** Ratification is Viktor's review, not an editor's. |
> | **Status** | **NOT RATIFIED — a PROPOSAL with one operator ruling, corrected by the 2026-10-04 spike (§7.1, corrections C1–C12 below).** Ratification is Viktor's review, not an editor's. |
> | **Written** | 2026-10-04, by the reviewer (project Claude), before any spike. |
> | **Verified against** | felhom.eu `d07a1a9` · felhom-controller `99a1497` (v0.290.0) · felhom-agent `d766666` (v0.138.0) · hub v0.128.0 |
> | **Freshness** | **CURRENT** as of 2026-10-04. The spike `TASK-backup-close-and-os-updates-spike-2026-10-04` adds measurements here as `[FACT]` and corrects every claim it disproves. Mark this file STALE when it falls behind what the product does. |
@@ -17,6 +17,24 @@
> **Why this file exists.** No architecture document covered operating-system updates. `00` §G marks the
> capability MISSING (added 2026-10-03). The finding is **R-812**. The roadmap intention is **R-808**.
> **The register carries the work. This file carries the reasoning. The source is the truth.**
>
> **Spike corrections, 2026-10-04** (`audits/os-updates-spike-2026-10-04/`). Each is made in place: the reviewer's
> text is struck (~~like this~~) and the measured text follows, labelled `[FACT]`, with its id.
>
> | Id | Where | What the spike changed |
> |---|---|---|
> | C1 | §2 | The agent may run `apt-get install` for TWO packages, not one (dnsmasq and wireguard-tools). |
> | C2 | §5.3 | Debian keeps two versions, not one; for box packages no fix was replaced within 14 days in 3 months; `snapshot.debian.org` works from a box in seconds. |
> | C3 | §5.2 | The lanes must follow the package's ORIGIN, not its name: 40 Proxmox-repository packages have ordinary names (ZFS, the Secure Boot shim, Ceph, corosync, chrony, CPU microcode). |
> | C4 | §5.6 | `--next-boot` is NOT a one-shot on these GRUB hosts. The fallback works only after a boot that reaches userspace. Installing a kernel alone makes it the default. Only a software watchdog runs. |
> | C5 | §5.6, §6 row 4 | Docker's `live-restore` keeps every container running across an engine update (measured). Turning it OFF again stops every container and starts none. |
> | C6 | §5.6 | A guest snapshot works on LVM-thin (customer guests) but not on `dir` storage; the snapshot rollback itself is unmeasured; a backup-restore undo took 73 s. |
> | C7 | §6 row 2 | A killed `apt` run does not recover by itself; the repair took ~5 s. |
> | C8 | §1, §6 row 14 | `cloudflared` is not a host package: it is a container in the guest, pinned by the controller since June. It belongs with the controller's infrastructure pins, not this file's lanes. |
> | C9 | §5.3 | The approved list must record what ring 0 RUNS healthy, not only what it installed that night. |
> | C10 | §5.5 | Restore-tests and agent updates are not windowed; the host's `apt` timers install nothing today. |
> | C11 | §5.2 | A Debian (fast-lane) update leaves PID 1, `lxc-start`, the Proxmox daemons and dockerd on the old library: its full effect needs a restart the fast lane does not do. |
> | C12 | §6 row 9 | The two demo hosts differ by 7 packages, including the CPU microcode (AMD vs Intel) and Secure Boot (on vs off). |
---
@@ -50,6 +68,10 @@ boxes installed. Nobody keeps a hand-written list.
alone: *"apt repo alignment skipped (byo — the owner manages repos)"* (`scripts/felhom-host-install.sh`,
`align_apt_repos`). `[PROPOSAL]` On a BYO box we update the guest and Docker only, never the host.
- **The Proxmox MAJOR upgrade** (PVE 9 → 10). It is a later step of its own, drilled first (R-808 item 5).
- **`cloudflared` and the other infrastructure images** (traefik, filebrowser). **[FACT] (C8)** They are containers in
the guest, pinned in the controller (`internal/infra/infra.go:26`: `cloudflare/cloudflared:2026.6.0`, since
2026-06-11) and baked into the golden; upstream was `2026.9.3` on 2026-10-04. They move only by a controller release
— the app-image question (`09`), not an OS package. The gap is R-838.
---
@@ -68,8 +90,11 @@ boxes installed. Nobody keeps a hand-written list.
says *"No upgrades are run — repo alignment only"* (`felhom-host-install.sh` ~L2135). A fresh install
gets the Docker engine that was current when its golden was baked, and keeps it.
**[FACT] The agent may not run `apt` today, with one exception.** Its sudoers allowlist
(`felhom-agent/configs/felhom-agent.sudoers`) holds one apt line: `apt-get install -y -q dnsmasq`.
~~**[FACT] The agent may not run `apt` today, with one exception.** Its sudoers allowlist
(`felhom-agent/configs/felhom-agent.sudoers`) holds one apt line: `apt-get install -y -q dnsmasq`.~~
**[FACT] (C1) The agent may run `apt-get install` for two named packages:** `felhom-agent.sudoers:58`
(`apt-get install -y -q dnsmasq`, used by `internal/lanresolver/lanresolver.go:107`) and `:194`
(`apt-get install -y -q wireguard-tools`, used by `internal/wgtunnel/manager.go:628`). Nothing else.
`apt` as root runs package scripts as root, so a broad `apt` grant is a full root grant. §5.4
proposes how to avoid that.
@@ -128,8 +153,23 @@ has not ruled on it as such.
| Urgent fix | The operator can approve a version on the same day once ring 0 has run it. | The same. |
| When | The night window (§5.5) | The night window. A kernel reboot only on a night the operator scheduled. |
Which packages count as slow lane is a list the spike checks (OPEN Q7). A Debian package that restarts
something big (for example `systemd`, `libc6`, `openssh-server`) may belong in the slow lane too.
~~Which packages count as slow lane is a list the spike checks (OPEN Q7). A Debian package that restarts
something big (for example `systemd`, `libc6`, `openssh-server`) may belong in the slow lane too.~~
**[FACT] (C3) The lane must be decided by ORIGIN, not by name.** On demo-hp, 40 pending packages come from the
Proxmox repository under ordinary names: `zfsutils-linux`, `zfs-zed`, `libzfs7linux`…, **`shim-signed` and friends (the
Secure Boot loader)**, `ceph-common`/`librados2`…, `corosync`, `chrony`, `frr`, `amd64-microcode`
(`partH/H2-proxmox-origin-debian-names.txt`). A name rule (`pve-*`, `proxmox-*`) would have put them in the fast lane.
The fast lane is: origin `Debian` or `Debian-Security`, and nothing else. On demo-hp that selection was 108 packages
and pulled in **zero** Proxmox packages.
**[FACT] (C11) What a fast-lane run restarts, and what it does not.** Guest (9202, 49 packages incl. libc6): the
packages' own scripts restarted postfix, journald, networkd; **dockerd, containerd, sshd, dbus, logind, cron** kept the
old libc; no container stopped (13 samples). Host (demo-hp, 108 packages): dnsmasq, postfix, journald restarted;
**systemd (PID 1), `lxc-start`, pveproxy, pvedaemon, pvestatd, pvescheduler, watchdog-mux, sshd, zed, chronyd** kept the
old libc; both guests and the agent stayed up. So the fast lane is safe to run unattended, but a libc fix is only
fully in force after a reboot (host) or a Docker restart (guest) — which are slow-lane acts. `[PROPOSAL]` the box
reports "restart needed" (processes on deleted libraries) and the slow lane's next reboot picks it up.
### 5.3 The approved list (the "tested versions" record)
@@ -148,11 +188,28 @@ Nobody writes the list by hand. It fills itself:
5. Each box reports which OS release it runs, how many updates are waiting, whether it needs a reboot,
and which packages it has that **no approved list covers** (see §6, edge case 9).
**OPEN Q1 is the weak point.** The design works only if a box can still download the approved version
~~**OPEN Q1 is the weak point.** The design works only if a box can still download the approved version
a week later. Debian's main and security archives keep only the newest version of each package. If
Debian publishes a newer fix between approval and install, the approved version is gone. Options:
the box waits for the next approval; or the box uses `snapshot.debian.org` (Debian's own dated archive,
still signed by Debian); or Felhom runs a package cache. The spike measures how often this happens.
still signed by Debian); or Felhom runs a package cache. The spike measures how often this happens.~~
**[FACT] (C2) Q1 measured.** The live Debian archives keep **two** versions: the point-release one in `trixie`
(main) and the newest in `trixie-security`; intermediate versions are gone (openssl: installed `u1`, main `u2`,
security `u3`). Over 2026-07-04..10-04, 167 trixie security advisories; for the 517 source packages installed on a box,
**no package got a second advisory within 2, 7 or 14 days** (the 3 within 2 days were chromium and webkit2gtk, not on a
box). `snapshot.debian.org` answers a box: a dated index in **2.3–3.0 s**, a gone exact version
(`openssl 3.5.6-1~deb13u1`) downloaded in **2.0 s**, Debian-signed. Proxmox and Docker keep many old versions
(pve-manager 66, docker-ce 46). So with a 1–2 day wait the approved version is almost always still live; the rare
miss is fetched from the snapshot taken at approval time. `[PROPOSAL]` each OS release records its approval
timestamp; a box installs from its own sources, and only for a Debian package that is no longer there, from
`snapshot.debian.org/archive/<debian|debian-security>/<timestamp>`. This is the operator decision in STATUS.
**[FACT] (C9) Approve what ring 0 RUNS, not what it installed.** In the simulation on demo-felhom
(`partI/demo-felhom-simulation.txt`), every one of the 108 host and 49 guest approved versions was installable and
downloadable — but `curl`, `libcurl*` and `libssh2` were "not covered" in the guest, only because scratch guest 9202
ALREADY ran the newer version and so installed nothing. `[PROPOSAL]` step 1 reports the full installed
`package=version` set after the run, and approval covers every version ring 0 runs healthy.
### 5.4 Who runs it, and with what permission
@@ -172,6 +229,66 @@ still signed by Debian); or Felhom runs a package cache. The spike measures how
versions. A broken-into hub can choose an older version or no version. It cannot make a box install a
package that the publisher did not sign.
### 5.4.1 The root wrapper's interface — DRAFT (2026-10-04, design only, nothing installed) `[PROPOSAL]`
`felhom-os-apply` — root-owned (`0755 root:root`), installed by the installer beside `felhom-selfupdate-guarded`;
the agent's sudoers gets exactly `/usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json` and
`… --repair-only`. For the guest it runs on the host and enters the guest with `pct exec <vmid> --` itself, so the
agent needs no `pct exec … apt` line. Built from what Parts G and H measured.
**Input — one JSON plan file** (written by the agent, from the hub's approved OS release):
```json
{
"release_id": "os-2026-10-04-1", "approved_at": "2026-10-04T08:00:00Z",
"layer": "host", // "host" | "guest"
"vmid": 9201, // guest only
"snapshot": "20261004T080000Z", // the snapshot.debian.org timestamp of approval (C2)
"packages": [ {"name": "libc6", "version": "2.41-12+deb13u4", "origin": "Debian"} ],
"allow_new": ["proxmox-kernel-7.0.14-20-pve-signed"], // slow lane only, signed operator job
"lane": "fast" // "fast" | "slow"
}
```
**Order of work:** (1) refuse checks below; (2) **repair first** — `dpkg --configure -a` then `apt-get -f install`,
logging what it repaired (C7); (3) `apt-get -s install` of exactly `name=version` for every package that is
installed AND older; (4) refuse if the simulation would remove, downgrade, add an unlisted package, or touch a package
whose candidate origin is not the plan's; (5) download — from the box's own sources, or for a Debian version no
longer there, from `snapshot.debian.org/archive/<archive>/<snapshot>` with a temporary sources list it deletes after
(C2); (6) install with `DEBIAN_FRONTEND=noninteractive APT_LISTCHANGES_FRONTEND=none -o Dpkg::Options::=--force-confold
-o Dpkg::Options::=--force-confdef`; (7) `apt-get clean`; (8) report.
**Refusals** (each exits non-zero with one line `os-apply: REFUSED: <reason>` and changes nothing):
| # | Refuses when |
|---|---|
| R1 | the plan file is not under `/var/lib/felhom-agent/os/`, not owned by the agent, or not valid JSON |
| R2 | `lane` is `fast` and any package's origin is not `Debian` / `Debian-Security` (C3) |
| R3 | `lane` is `slow` and the plan is not carried by a verified signed operator job (R-530's mechanism) |
| R4 | the simulation removes any package |
| R5 | the simulation downgrades any package (the operator undo is a separate signed op, §5.6) |
| R6 | the simulation installs a package that is neither installed nor in `allow_new` |
| R7 | a listed version is not downloadable from the sources the installer set up or the named snapshot |
| R8 | free space on `/` (or the guest's rootfs) is below 3× the download size, minimum 500 MB (edge case 8) |
| R9 | another apt/dpkg holds the lock, or the per-guest lane lock is held (a backup, a restore-test, C10) |
| R10 | `layer` is `guest` and the vmid is not the box's own customer guest |
| R11 | the plan names a package twice, or a version that is not a Debian version string |
**Log lines** (to the journal, tag `felhom-os-apply`, and echoed for the agent to forward to the hub):
```
os-apply: START release=<id> layer=<host|guest:vmid> lane=<fast|slow> packages=<n>
os-apply: REPAIR configured=<n> fixed=<n> (always printed; 0 0 when nothing was half-done)
os-apply: PLAN upgrade=<n> already=<n> not-installed=<n> from-snapshot=<n> download=<bytes>
os-apply: REFUSED: <R-number> <reason>
os-apply: CONFFILE kept <path> (new version saved as <path>.dpkg-dist)
os-apply: DONE rc=0 seconds=<s> upgraded=<n> restarted=<unit,…> restart-needed=<process,…> reboot-needed=<yes|no>
os-apply: FAILED rc=<n> step=<download|install> — dpkg state: <dpkg --audit first line>
```
`restart-needed` lists processes still mapping deleted libraries (C11); `reboot-needed` is yes when that list holds
PID 1 or `lxc-start`, or a kernel was installed.
### 5.5 When
Inside the household's night window, after the backups:
@@ -188,9 +305,16 @@ The OS leg starts **after the whole-guest backup has finished**, so the guest's
minutes old. This is the opposite order to app updates, which run before the whole-guest backup
(`07` §6.1, `09` decision 11). The reason: the whole-guest backup is the guest's undo.
**OPEN Q8:** what else runs then. The controller's self-update (default 04:30, and after any hub report
**[FACT] (C10) Q8 measured** (guest UTC; demo-hp W = 02:30): db-dump 02:30, tier-2 03:30, off-site ~04:15,
whole-guest gate [04:30, 08:30), controller self-update 04:30, offsite-integrity 06:00. Host: `apt-daily` and
`apt-daily-upgrade` run daily but install nothing (no `unattended-upgrades`, no `APT::Periodic`); `pve-daily-update`
refreshes the lists daily. **Restore-tests are NOT windowed** — they run on a cadence at any hour (demo-felhom 10:38
daily, demo-hp 16:43 and 22:46); agent updates arrive by signed job at any hour. `[PROPOSAL]` the OS leg takes the same
per-guest lane lock the restore-test and the whole-guest backup take, rather than a clock slot.
~~**OPEN Q8:** what else runs then. The controller's self-update (default 04:30, and after any hub report
when a floor is above it, R-608), the agent's self-update, the restore-tests, and PBS jobs. The OS leg
must never overlap a backup, a restore-test or a self-update.
must never overlap a backup, a restore-test or a self-update.~~
### 5.6 How a failed update is undone
@@ -198,10 +322,10 @@ must never overlap a backup, a restore-test or a self-update.
| Layer | Undo | Limit |
|---|---|---|
| Guest packages | Restore the guest snapshot taken just before the update | It also undoes app data written after the snapshot. Use it only inside the health window, before apps have written much. After that, install the previous version (needs OPEN Q1). |
| Docker engine | Install the previous version (Docker's repository keeps old versions — OPEN Q2) | Every container restarts again |
| Host packages | Install the previous version | Only if the source still has it (OPEN Q1/Q2) |
| Host kernel | Boot the previous kernel. Proxmox can boot a new kernel **once** (`proxmox-boot-tool kernel pin <ver> --next-boot`). If that boot fails, the next boot uses the old kernel again. Make the new kernel permanent only after a healthy boot. | If the new kernel hangs, someone must switch the box off and on. The spike checks whether a hardware watchdog can do that (OPEN Q4). |
| Guest packages | Restore the guest snapshot taken just before the update | It also undoes app data written after the snapshot. Use it only inside the health window, before apps have written much. After that, install the previous version (needs OPEN Q1). **[FACT] (C6)** Customer guests are on LVM-thin and can snapshot; scratch 9202 (`dir` storage) cannot (`snapshot feature is not available`), so the measured undo was a whole-guest backup + restore: **73 s down**, all apps healthy, libc back. The snapshot rollback itself is unmeasured (R-837). |
| Docker engine | Install the previous version (Docker's repository keeps old versions — OPEN Q2) | Every container restarts again. **[FACT] (C5)** Docker keeps 46 `docker-ce` versions. A step (either way) without `live-restore`: 6 of 6 containers restart, the app silent **26.5–30 s**, healthy at +42–45 s. With `live-restore` on: **0 restarts, no gap**, also across a containerd step. But a restart that turns `live-restore` OFF stops every container and starts NONE (`unless-stopped` ignored) — on 9202 they stayed down until restarted by hand; `systemctl reload` turns it on but not off. |
| Host packages | Install the previous version | Only if the source still has it (OPEN Q1/Q2). **[FACT]** `rsync` back to its pre-update version: `not found`; `libpng16-16t64` back to the point-release version: worked. A Debian undo needs the snapshot archive (C2). |
| Host kernel | ~~Boot the previous kernel. Proxmox can boot a new kernel **once** (`proxmox-boot-tool kernel pin <ver> --next-boot`). If that boot fails, the next boot uses the old kernel again. Make the new kernel permanent only after a healthy boot.~~ **[FACT] (C4)** Both demo hosts boot UEFI + GRUB (no proxmox-boot-tool ESPs). **Installing a kernel makes it the GRUB default at once.** `--next-boot` on GRUB writes an ordinary `GRUB_DEFAULT` + `update-grub`; `proxmox-boot-cleanup.service` clears it only once a boot reaches userspace. Measured on demo-hp (old kernel pinned permanently FIRST, new pinned for next boot): boot 1 → `7.0.14-20-pve`, healthy, 60 s; boot 2 → back on `7.0.2-6-pve`, healthy, 76 s. **So the fallback works after a boot that succeeds; read from the code, a kernel that hangs before userspace stays the default on every power cycle.** `[PROPOSAL]` use GRUB's own one-shot (`GRUB_DEFAULT=saved` + `grub-reboot`) with the old kernel as the saved default — unmeasured (R-836). | ~~If the new kernel hangs, someone must switch the box off and on. The spike checks whether a hardware watchdog can do that (OPEN Q4).~~ **[FACT]** Only `softdog` runs (loaded by `watchdog-mux`); it cannot rescue a kernel that never boots. demo-hp has an AMD FCH whose `sp5100_tco` driver ships but is not loaded; untested. A hang still needs a person. |
### 5.7 Telling people
@@ -219,19 +343,19 @@ must never overlap a backup, a restore-test or a self-update.
| # | What can go wrong | What the design does |
|---|---|---|
| 1 | A new kernel does not boot | Boot it once (`--next-boot`); the old kernel stays the default. A hard hang needs a power cycle (OPEN Q4). |
| 2 | Power cut during an update: `dpkg` is half done | The wrapper runs `dpkg --configure -a` and `apt-get -f install` first, every time, and reports what it repaired (spike measures, Q5). |
| 2 | Power cut during an update: `dpkg` is half done | The wrapper runs `dpkg --configure -a` and `apt-get -f install` first, every time, and reports what it repaired (spike measures, Q5). **[FACT] (C7)** Killed after 15 unpacks: 5 packages `iU`, 4 triggers pending; the next ordinary `apt-get install` REFUSES (`Unmet dependencies`) — nothing repairs it by itself. The two commands repaired it in 1.4 s + 3.9 s; apps stayed up. |
| 3 | An update asks a question (changed config file, service restart prompt) | Non-interactive, keep the old config, report the conflict. |
| 4 | A Docker update stops every app, and the controller | Stop the apps cleanly first, like before a backup. Measure whether `live-restore` keeps containers running (Q3). The agent drives it. |
| 4 | A Docker update stops every app, and the controller | Stop the apps cleanly first, like before a backup. Measure whether `live-restore` keeps containers running (Q3). The agent drives it. **[FACT] (C5)** `live-restore` keeps them running (0 restarts). The controller keeps running too; its `docker` calls fail for the seconds dockerd is down (logged errors, no app event). Turning `live-restore` on is a golden + fleet change — and turning it off later must not be a plain restart (R-835). |
| 5 | The approved version is no longer downloadable | OPEN Q1. Until it is answered, the box waits for the next approval and reports it. |
| 6 | An urgent security hole | The operator approves the same day once ring 0 has run it. |
| 7 | A box was off for months | It catches up through the newest approved release. It does not install every release in between. Packages are not like app data; one `apt` step is enough. Kernel and Docker still go one approved step at a time. |
| 8 | The system disk is full | Check free space before downloading; clean the package cache after; refuse and report. |
| 9 | A customer box has a package that ring 0 does not have (other hardware: firmware, CPU microcode, NIC drivers) | It is never approved, so it is never updated. The box reports it as "not covered", and the hub raises it. Fix: add matching hardware to ring 0, or approve it by hand. |
| 9 | A customer box has a package that ring 0 does not have (other hardware: firmware, CPU microcode, NIC drivers) | It is never approved, so it is never updated. The box reports it as "not covered", and the hub raises it. Fix: add matching hardware to ring 0, or approve it by hand. **[FACT] (C12)** The two demo hosts differ by 7 packages: `amd64-microcode`, `proxmox-secure-boot-support`, `felhom-bootstrap` (demo-hp) vs `intel-microcode`, `proxmox-first-boot`, `tailscale`, `tailscale-archive-keyring` (demo-felhom); Secure Boot is ON on demo-hp, OFF on demo-felhom. Ring 0 covers both CPU vendors today. |
| 10 | The package source is down, or its signing key changes (Docker has done this) | The update fails cleanly and the box reports it. A key change is a slow-lane act for a person. |
| 11 | A BYO host | Only the guest and Docker are updated (§1). |
| 12 | Two boxes on ring 0 is a small sample | Accepted for now. When there are customers, the first tester boxes can become a second ring. |
| 13 | A broken-into hub sends a harmful list | The wrapper refuses removals, downgrades and new packages, and the publishers' signatures still apply (§5.4). |
| 14 | `cloudflared` on the host | Not covered by this file yet. How it is installed and updated is OPEN Q9. It faces the internet, so it matters. |
| 14 | `cloudflared` on the host | ~~Not covered by this file yet. How it is installed and updated is OPEN Q9. It faces the internet, so it matters.~~ **[FACT] (C8)** Not on the host: a pinned container in the guest (§1). Four months behind upstream on 2026-10-04 (R-838). |
| 15 | The no-subscription Proxmox repository gets less testing than the enterprise one | Ring 0 is our test. The enterprise repository costs a yearly fee per box. That is a **money** decision for the operator, later (Q10). |
---
@@ -253,6 +377,27 @@ must never overlap a backup, a restore-test or a self-update.
---
### 7.1 Answers — the 2026-10-04 spike `[FACT]`
All evidence: `audits/os-updates-spike-2026-10-04/` (its `README.md` carries every number).
| Q | Answer in one line | Detail |
|---|---|---|
| Q1 | Usually yes: Debian keeps 2 versions; no box package was re-fixed within 14 days in 3 months; the snapshot archive serves a gone version in 2 s. | C2 |
| Q2 | Yes for Proxmox (30–66 versions) and Docker (18–46); Debian only the point-release version. | C2, §5.6 |
| Q3 | Without `live-restore`: every container restarts, ~30 s of silence. With it: none. Switching it off is a trap. | C5 |
| Q4 | `--next-boot` falls back only after a boot that succeeds (measured); a hang keeps the new kernel (code). Software watchdog only. | C4 |
| Q5 | Not by itself; `dpkg --configure -a` + `apt-get -f install` repair it in ~5 s. | C7 |
| Q6 | Hosts 188 pending each, guests 54–59; guest Debian 24 s, host Debian 60 s, kernel 47 s. Three guests, three Docker versions. | audit README |
| Q7 | Few restarts by script; libc leaves PID 1, `lxc-start`, Proxmox daemons and dockerd on the old library. Proxmox packages restart their own daemons. | C11 |
| Q8 | The backup legs are windowed; restore-tests and agent updates are not; host apt timers are inert. | C10 |
| Q9 | Not a host package: a pinned guest container, 4 months behind. | C8 |
| Q10 | €120 (Community) to €1,100 (Premium) per CPU socket per year, net; every tier includes the Enterprise Repository. Money — the operator's. | audit README |
**Sample approved list** built from what ring 0 installed (`partI/sample-approved-list.tsv`: 108 host + 49 guest
packages, with origin) and simulated read-only on demo-felhom: **all 157 would install, exact version, downloadable**;
not covered: 79 Proxmox + 1 Tailscale on the host (slow lane / not ours), 6 Docker + 4 Debian in the guest (C9).
## 8. Build order `[PROPOSAL]`
Each step returns to the operator for go or no-go.