docs: 11 §5.7 System page, §5.8 Docker slow lane BUILT, §5.9 crash restart; 00/03/07/08; decisions 90-94 (CC unattended); runbooks docker-undo + crash-guard; register R-852 R-835 R-848 R-849 R-851 R-854 closed, R-853 R-855 R-856 opened, R-812 R-840 narrowed (334 -> 332); live evidence

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

CONTEXT.md

@@ -16,6 +16,44 @@
 > and holds nothing of its own; this file does hold its own content, namely the standing rulings below.
 
 
+> **2026-10-04 (evening) — the System page, the Docker engine slow lane and the crash guard BUILT.** Agent v0.142.0,
+> hub v0.132.0, installer 1.30.0, `build-golden.sh` 3.1.0. **Decided by CC unattended — operator may reverse:** `09` §3
+> decisions **90** (crash signal = a clean-shutdown marker; pstore saved nothing), **91** (`panic_on_oops` stays 0; an
+> oops is reported), **92** (the 3rd unclean stop in 60 min leaves the box off; 10 s / 24 h), **93** (the wrapper
+> verifies Docker authority against root-owned files), **94** (page colours = alarm thresholds). Live: live-restore on by
+> reload with the same ids (9202 6/6, demo-hp 24/24, demo-felhom 5/5); Docker 29.7 → 29.8.2 on both demo boxes, ids kept;
+> the operator's button approved `os-docker-20261004-142842` (TEST 0-night wait, reverted); a signed undo to 29.7.2 on
+> demo-hp and back; demo-felhom as ring 1 by a signed job; a replay refused; crash 1–2 restarted in 54/53 s, the guard
+> tripped, crash 3 stayed off, the hub mailed the trip, re-armed. `REPORT-os-docker-crash-2026-10-04.md`.
+
+> **Rulings 2026-10-04 (~15:17) — recorded before the work (System page + Docker lane + crash restart brief).** `09` §3
+> decisions **87** (Docker `live-restore` ON fleet-wide, never off by a plain restart), **88** (a crashed host restarts by
+> itself, with a limit — R-851) and **89** (the operator sees the boxes' OS / Proxmox versions in the hub — R-852).
+
+> **2026-10-04 (late afternoon) — OS updates: host fast lane + fleet view + alarms BUILT (`11` §8.2–§8.3); the tunnel
+> status is true (R-841).** Agent v0.141.0 → v0.141.1, hub v0.131.0 → v0.131.1, controller v0.292.0 (cloudflared
+> readiness health check). **Decided by CC unattended — operator may reverse:** `09` §3 decisions **84** (appliance proof
+> = the root-owned install record), **85** (alarm numbers 7/14/7/14 days, configuration), **86** (a same-session patch
+> release of agent and hub for the host "reboot needed" defect, R-846). Live: `tunnel_down`/`tunnel_recovered` on
+> demo-hp; ring 0 host pass on demo-felhom (108 Debian packages); a 605-package host release; ring 1 exact install;
+> by-hand host undo proved; leg 23–32 s with nothing to install. Kernel spike (R-836, narrowed): GRUB's one-shot is not
+> a one-shot on LVM `/boot`; Secure Boot fine; `kernel.panic = 0` (R-851); `sp5100_tco` answers. demo-hp now runs and
+> defaults to kernel 7.0.14-20. Docker slow lane designed (`11` §5.8). Golden 0.292.0 baked + vouched with agent 0.141.1 (min_agent
+> 0.131.0); the golden waiver deleted (not needed). `REPORT-os-host-lane-2026-10-04.md`.
+
+> **Rulings 2026-10-04 (~12:20) — recorded before the work (host fast lane brief).** `09` §3 decisions **81** (R-842 A:
+> the undo is the whole-guest backup by hand; R-842 closed), **82** (R-840 not built now — "There are no older boxes";
+> row kept open with the reviewer's note) and **83** (next: R-841, the host fast lane, OS-update improvements).
+
+> **2026-10-04 (afternoon) — OS updates, guest fast lane BUILT (`11` §8.1).** Agent v0.140.0: `configs/felhom-os-apply`
+> (R1–R13, repair first, snapshot.debian.org fallback; FELHOM_OSAPPLY sudoers), `internal/osupdate` (the leg after a
+> successful primary backup, under the heavy-op gate; health baseline = start of the leg), `--selftest=os-update`.
+> Hub v0.130.0: `internal/osupdates` (rings, switch default ON, candidate = what ALL ring-0 boxes run, approval 24 h +
+> 1 night via `OS_APPROVE_AFTER` / `OS_APPROVE_NIGHTS`, `os_update` desired block, `POST /hosts/{id}/os-report`,
+> operator `/os/*`). Controller v0.291.0: decision 78 (R-726), infra pins raised (R-838) + `scripts/check-infra-pins.py`.
+> Installer 1.29.0 installs the wrapper. **No automatic undo** (R-837 measured → R-842). **Existing boxes need the
+> wrapper + sudoers by hand** (R-840 — the demo boxes got them by hand). `REPORT-os-guest-lane-2026-10-04.md`.
+
 > **Rulings 2026-10-04 (~10:17) — recorded before the work (guest fast lane build).** `09` §3 decisions **78** (R-726
 > option A: a returning household's box sets the old off-site copy aside on night one), **79** (`snapshot.debian.org`
 > fallback for a replaced approved version — option A; the reviewer had picked B) and **80** (next build: the guest

REPORT-os-guest-lane-2026-10-04.md

@@ -0,0 +1,60 @@
+# REPORT — OS updates build step 1: the guest's Debian fast lane; decision 78; the infrastructure images — 2026-10-04
+
+Architecture read: `11-os-updates.md` (with C1–C12, §5.4.1, §7.1 — the design; it won wherever it differed from the
+brief, see below), `03-host-agent.md`, `07` §6.1, `09` §3 decisions 11/12/15/18, `08`. Baselines (re-verified):
+felhom.eu `1b74ddc0c9` (hub 0.129.0), agent `596238cc2e` (0.139.0), controller `99a1497560` (0.290.0), catalog
+`917a779cca`. Register 331, highest R-839. Rulings recorded first: `09` §3 decisions 78–80 (`6ed79cd`). Evidence: `documentation/audits/os-guest-lane-2026-10-04/` (parts A–G).
+
+## The Part table
+
+| Part | Result | Notes |
+|---|---|---|
+| A — the snapshot undo first (R-837) | **done — and it FAILED: no snapshot is possible** | PVE refuses any snapshot not named `vzdump` of a guest with host-path binds (mp8/mp9), as the agent's token (which has `VM.Snapshot` + `VM.Snapshot.Rollback`) and as root. By the brief's rule: **no automatic undo built**; the decision is in STATUS (R-842). Steps 2–5 (apply, roll back, re-apply) had nothing to roll back to; 9201 was brought current by the product's own leg in Part G. Thin pool unchanged. |
+| B — the wrapper | **done** | `felhom-os-apply` (Python 3 stdlib), R1–R13, repair first, snapshot.debian.org fallback, log lines; host layer and slow lane refused. 35 tests; **every refusal red-proved** (13/13). `visudo -cf` OK. **Changed:** Python not shell (a JSON plan cannot be parsed safely in sh — so "shellcheck clean" became `ast`/compile-checked + the suite); one sudoers entry with a plan `mode` instead of a separate `--repair-only`. **The route for existing boxes: none exists** (R-840, with a proposal). |
+| C — the agent's leg | **done** | After a successful primary whole-guest backup, under the heavy-op gate (red-proved: the gate is held), once per 20 h, 90 s settle. Health rule written and pinned (`HealthVerdict`). Report: full installed set with origins, pending, not covered, restart-needed. Debug action `--selftest=os-update`. 7 leg red-proofs + 2 hook red-proofs. |
+| D — the hub | **done** — hub v0.130.0 | Rings, per-box switch (default ON), the candidate/approval rule (24 h + 1 night, config), approve-now, events, fleet JSON. 5 approval red-proofs; the `os_update` wire golden byte-identical in both repos. |
+| E — household line + decision 78 | **done** | Line = hub customer event `os_update_applied` (info: on the household's timeline, not mailed; hu/en in the bundle). **Changed:** there is no box-side event surface, so the hub event is it (R-844). Decision 78 built in controller v0.291.0, red-proved both ways. |
+| F — infrastructure images (R-838) | **done** | traefik v3.7.13, cloudflared 2026.9.3, filebrowser 1.5.6-stable; breaking changes named (none we use). A release moves all three (9202: ≤ 1.9 s / ≤ 1.5 s; demo boxes: public gap ≤ 19.6 s / ≤ 14.7 s incl. the controller restart). `scripts/check-infra-pins.py` + runbook section. **Changed:** the standing brief `claude/MONTHLY-security-retest.md` lives in the claude.ai project, not the repo — the repo half is the runbook; the project file is the operator's to update. `03` corrected (3 lines). |
+| G — live proof | **done, one part changed** | Ring 0 on both boxes (53 packages each, healthy); approval with a 2-minute TEST wait (272 packages, auto), then the ruled values back; ring 1 on demo-felhom (exactly the 3 approved versions, nothing newer); a failed health check → `health_failed`, operator mail, household line. **Changed:** "show the rollback" — there is none (Part A). Teardown: no snapshot, no plan files, test config gone, demo-felhom back to ring 0. |
+| H — release, golden, records | **done** (see Teardown for the golden) | Agent 0.140.0 (signed per box, both demo boxes on it), hub 0.130.0, controller 0.291.0 (floor 0.291.0, MinAgent 0.131.0 declared), installer 1.29.0. `11` §8.1, `00`, `07` §6.1, `03` updated. |
+
+## Claims in the brief that turned out wrong (named)
+
+1. **"The agent's token can snapshot and roll back"** — it HAS the rights, but no snapshot of a customer guest is
+   possible at all (bind mounts). Neither the token nor root can.
+2. **"A snapshot rollback leaves the thin pool clean"** — unmeasurable: there was no snapshot.
+3. **"A new sudoers line can reach an installed box through the product"** — false. Only the installer writes it;
+   the signed agent update replaces the binary only (R-840). The demo boxes got the wrapper + sudoers BY HAND.
+4. **"A controller release moves the infrastructure containers"** — TRUE for all three. (I first wrote the opposite
+   for the file browser and corrected it the same hour: its start-up mount sync renders the new image.)
+5. **"An agent event can reach the household's timeline"** — only through the hub (a hub customer event); the box has
+   no timeline of its own (R-844).
+6. **"Before each guest update, the box takes a snapshot"** (the one-page summary) — impossible (Part A).
+7. `11` vs the brief: `11` §5.4.1's `--repair-only` flag was folded into the plan; `11`'s "a missed night waits" holds.
+
+## Found and fixed live (before the release)
+
+- `--selftest=os-update` was refused by the flag's allow-list — and so was `--selftest=wgtunnel`, since S3 (R-843,
+  opened and closed; a new test pins every dispatched mode).
+- The wrapper logged an UPDATED conffile as "kept" (dpkg's two message shapes; fixed + tested).
+- An app stopped between the inventory and the apply escaped the health check; the baseline is now the start of the
+  leg (fixed + red-proved).
+- My stopped-app test also made the box mail one `app_start_failed` (a second one was held by the cooldown).
+
+## Rows
+
+Closed: **R-837** (measured), **R-838**, **R-726**, **R-843** (opened and closed). Opened: **R-840** (no product route
+to installed boxes, P2), **R-841** (the agent's cloudflared probe reads a host unit that does not exist, P3), **R-842**
+(the undo decision, waiting on the operator), **R-844** (household line only on the hub, P4), **R-845** (a pass takes
+3–4 min, P4). Narrowed: **R-812**. Register **331 → 333**.
+
+## Teardown, three layers
+
+- **Machines:** no snapshot on either 9201; no plan files; privatebin restarted and healthy; demo-felhom back to ring 0;
+  both 9201s fully Debian-current (openssl at the approved u3). 9202 runs controller 0.291.0 (from Part F).
+  **Kept on purpose:** the wrapper + sudoers on both demo hosts (installed by hand; the old sudoers saved as
+  `/root/felhom-agent.sudoers.bak-pre-osapply`); agent 0.140.0 (signed update).
+- **Host (DooPlex):** helper scripts in the scratchpad only; the hub password copy shredded at the end.
+- **Hub:** v0.130.0 at the ruled 24 h + 1 night (the TEST override reverted and the start log shows no override);
+  both demo boxes ring 0, ON; release `os-20261004-091417` approved (it was approved under the TEST wait — ring 1 boxes
+  will install it; every version in it already runs on both demo boxes). Floor 0.291.0.

REPORT-os-host-lane-2026-10-04.md

@@ -0,0 +1,90 @@
+# REPORT — OS updates, build steps 3 + 4 (2026-10-04, afternoon–evening)
+
+Brief: "OS updates, build step 3 + 4" (Parts A–G). Architecture read first: `documentation/architecture/11-os-updates.md`
+(owner), `08-alarm-ladder.md`, `03-host-agent.md`, `07-backup-architecture.md` §6.1. Evidence for every claim:
+`documentation/audits/os-host-lane-2026-10-04/` (partA … partG).
+
+## Part table
+
+| Part | What | Result | Evidence |
+|---|---|---|---|
+| A | The tunnel status is true (R-841) | **DONE.** Three states from the guest container + controller v0.292.0's readiness check; `unknown` never alarms; `tunnel_down` after two `not_running` reports. Live on demo-hp: port 7844 blocked → `tunnel_down` mailed 11:52 UTC (2nd report); unblocked → `tunnel_recovered` 12:07 UTC. No new sudoers line. | `partA/` (red-proofs agent/hub/controller, `live/`) |
+| B | Host fast lane (`11` §8 step 3) | **DONE.** R12 lifted for lane fast / layer host on an appliance only (root-owned install record); R14 refuses kernel/boot/firmware; host step after a healthy guest step; host health rule in `11` §8.2; reboot-needed with first date, never reboots; separate per-layer approved sets. Live: demo-felhom ring 0 → 108 Debian host packages (all Debian origin, checked against apt), healthy; demo-hp ring 0 (nothing pending); 605-package host release approved under a TEST wait (2 m / 0 nights, logged, reverted); demo-felhom as ring 1 installed exactly the 1 version it lacked; back to ring 0 and the ruled 24 h + 1 night. Host undo runbook proved on demo-hp (`tzdata`). | `partB/` (`live/`, `ring1/`, `undo/`, red-proofs) |
+| C | Fleet view + four alarms (`11` §8 step 4) | **DONE.** `GET /os/fleet`: one line per box, ring, switch, tunnel, per layer release/pending/not-covered/last good leg/reboot-since. Alarms `os_update_stale`, `os_reboot_needed`, `os_ring0_stalled`, `os_not_covered`, hourly, operator-only, each red-proved (9 mutations caught). Numbers in `11` §8.3 / `09` decision 85 as CC-decided. | `partC/` |
+| D | The leg is fast (R-845) | **DONE.** Nothing to install, both layers: **23.3 s** (demo-felhom), **31.5 s** (demo-hp). Before (agent 0.140.0, guest only, nothing to install): 14.0 s. A 108-package host pass: 70 s. | `partD/`, `partB/live/` |
+| E | Kernel one-shot spike on demo-hp (R-836), operator's word before each reboot | **DONE (measured).** Secure Boot ON boots the new kernel fine; **`grub-reboot` is NOT a one-shot here** — `/boot` on LVM, GRUB cannot clear `next_entry`, reboot 2 (no command) came back on the NEW kernel. `kernel.panic = 0`. `sp5100_tco` loads and answers (sysfs only, never armed, unloaded). No kernel installed (7.0.14-20 was already there). Left: runs 7.0.14-20, saved default 7.0.14-20, both kernels installed, `GRUB_DEFAULT=saved`. | `partE/` |
+| F | Docker slow lane design | **DONE (design only).** `11` §5.8. One STATUS decision: `live-restore` on, fleet-wide. | `11` §5.8, STATUS |
+| G | Releases, golden, records | See below. Agent **v0.141.0 + v0.141.1**, hub **v0.131.0 + v0.131.1** (decision 86 — a second release in each, for a live-found defect), controller **v0.292.0**. Installer: **no new release** (see below). | `partG/` |
+
+## Claims in the brief that turned out wrong (or only half right)
+
+- **cloudflared readiness** — it EXISTS (`/ready`, 200 only with a connection), but nothing exposed it: the metrics
+  port was random and there was no health check. A container state alone lies (wrong token: `running`, `/ready` 503).
+  Controller v0.292.0 adds the fixed port + Docker health check; the agent reads it with its existing sudoers line.
+- **"Find where the install mode is known"** — known in TWO places, and only one is trustworthy: `agent.json`
+  `deployment_mode` (the agent can write it) and the installer's root-owned `state.json` `mode`. The wrapper trusts
+  only the second (decision 84).
+- **`grub-reboot` with Secure Boot** — Secure Boot was not the problem (it booted fine). The one-shot itself fails on
+  these boxes because GRUB cannot write its state on LVM `/boot`.
+- **Panic auto-restart** — there is none: `kernel.panic = 0`; a panicked host stays down (R-851).
+- **Under 60 s** — met (23–32 s). But the leg was ALREADY under 60 s with nothing to install (14 s, guest only); the
+  3–4-minute passes of R-845 were passes that installed something.
+
+## Releases
+
+- **Agent v0.141.0** (`cfba0d0`, sha256 `6eaad980…`) and **v0.141.1** (`a6bc3f1`, sha256 `b712f577…`) — signed
+  `agent_update` jobs to both demo boxes (both COMPLETED). v0.141.1 fixes R-846 (host "reboot needed" hid `lxc-start`,
+  and a reboot never cleared it).
+- **Hub v0.131.0** (`88b0a2e`) and **v0.131.1** (`fd1f985`) — deployed via ArgoCD; image tag verified on the pod.
+- **Controller v0.292.0** (`09e634d`) — floor raised (`min_controller_version` 0.292.0, `min_agent` 0.131.0); both demo
+  boxes run it; cloudflared recreated once, `healthy`.
+- **Installer — no new release, deliberately** (recommendation not followed, one line why): the installer fetches
+  `configs/felhom-os-apply` from the VOUCHED agent's tag, the file name did not change and the sudoers line did not
+  change, so vouching agent 0.141.1 delivers the new wrapper to every fresh install with no installer change.
+- **Copied by hand to both demo hosts** (`partG/wrapper-copied-by-hand.txt`): `/usr/local/sbin/felhom-os-apply` only —
+  first from v0.141.0 (sha `4729769c…`), then from v0.141.1 (sha `51e100ad…`), installed `0755 root:root`, the old
+  file kept as `/root/felhom-os-apply.bak-0.140.0`. `/etc/sudoers.d/felhom-agent` was NOT copied: it already equals the
+  repo's (`02df92d7…` on both).
+
+## Golden
+
+- **Golden 0.292.0 baked** (by a helper agent, RUNBOOK-manual-build §4.0/§4.1, same `build-golden.sh` bytes as
+  0.291.0; drill VM 9100 destroyed, drill disk back to `virgin`): sha256 `d6cf8b33…16671`, re-hashed by download in the
+  main session — match. Evidence `documentation/tests/golden-0.292.0-2026-10-04/` (commit `f93738d`).
+- **Vouched:** agent 0.141.1 + golden 0.292.0, `min_agent` 0.131.0 → `artifacts_set` (`05-vouch.txt`). Fresh installs
+  now get agent 0.141.1, its wrapper, and controller 0.292.0.
+- **Waiver REMOVED** (`documentation/tests/golden-waiver.yml` deleted): the newest controller (0.292.0) now has its
+  golden, so `golden_currency_gate.py` passes without it. Golden 0.291.0 alone could NOT have retired it — this session
+  released controller 0.292.0, which put 0.291.0 behind.
+
+## Decisions taken by CC unattended (operator may reverse) — `09` §3
+
+- **84** — the appliance proof is the root-owned install record.
+- **85** — alarm numbers 7 / 14 / 7 / 14 days, all configuration.
+- **86** — a second same-session release of agent and hub (R-846).
+
+## Register
+
+Open rows **332 → 333**. Closed: **R-841**, **R-845**; filed and closed the same day: **R-846**, **R-850**. Opened:
+**R-848** (a held host package is invisible to the hub), **R-849** (the guest "reboot needed since" never clears),
+**R-851** (a panicked host stays down — operator). Narrowed: **R-836** (GRUB one-shot measured: not a one-shot),
+**R-812** (host lane built). `unproven.py`: unchanged (35 of 55 not walked).
+
+## Teardown — three layers
+
+- **Machine:** demo-hp guest 9201 — the port-7844 block removed (`DOCKER-USER` empty, verified); cloudflared
+  `healthy`. demo-hp host — `tzdata` back on 2026c, no holds, no snapshot source left; `sp5100_tco` unloaded; GRUB:
+  `GRUB_DEFAULT=saved`, saved default 7.0.14-20, `next_entry` cleared, `/etc/default/grub` backup at
+  `/root/grub.default.bak-2026-10-04`. demo-felhom host — `tzdata` 2026c (re-installed by its ring-1 run), no snapshot
+  source left.
+- **Host:** both demo hosts run agent 0.141.1 + wrapper `51e100ad…`; both ring 0, switch ON.
+- **Hub:** the TEST approval wait reverted (log: 24 h and 1 night); the TEST releases `os-guest-20261004-123933` and
+  `os-host-20261004-124034` stay (they are real approvals of what ring 0 runs). The operator mails `tunnel_down` /
+  `tunnel_recovered` for demo-hp were sent on purpose (Part A). The hub password copy in the scratchpad was shredded.
+
+## CI
+
+Checked by `head_sha` over every page of the Gitea `jobs` endpoint (felhom.eu `CLAUDE.md` recipe):
+agent `cfba0d0` (jobs 1259, 1260), `3bf77c3` (1261), `a6bc3f1` (1262, 1263), `2e2e8f5` (1264) — all **success**;
+controller `09e634d` (1258) **success**; felhom.eu `88b0a2e` (1256), `fd1f985` (1265) **success**. The last docs pushes
+(`31bdb4b` and this report's commit): see the session's final message — checked after the push.

STATUS.md

@@ -2,8 +2,79 @@
 
 **Ready for the first real tester (Tester-2): yes. You confirmed the tunnel route and the connect mails (2026-09-30).**
 
-**Updated 2026-10-04 (day): the off-site topic is closed; the OS-update test is done. Both demo boxes run controller
-0.290.0 and host agent 0.139.0. Hub 0.129.0. New installs get golden 0.290.0; every box's floor is 0.290.0.**
+**Updated 2026-10-04 (evening): the HOST's security fixes install themselves too, the hub has a fleet view and four
+OS alarms, and the tunnel status is true. Both demo boxes run controller 0.292.0 and host agent 0.141.1. Hub 0.131.1.
+New installs get golden 0.292.0 with agent 0.141.1 (vouched).**
+
+## Today (2026-10-04, evening): host fixes, the fleet view, a true tunnel status
+
+**Decisions I took myself (you may reverse each):**
+- Only a box the installer itself recorded as "appliance" gets host updates (a record the agent cannot change).
+- The four alarm times: no update run for 7 days; "reboot needed" for 14 days; the demo boxes approve nothing for 7
+  days; a box has fixes nobody approved for 14 days. All four are settings.
+- I released the host agent and the hub a second time today. The first release said "reboot needed" wrongly; the new
+  14-day alarm would then have mailed you about boxes you had already rebooted.
+
+**One decision for you (a safe default if you say nothing) — the Docker engine updates (designed, not built):**
+1. **Turn on Docker's "live-restore" on every box.** With it, a Docker engine update restarts no app (measured: 0
+   restarts). Without it, every app stops for about 30 seconds per engine update.
+   - **A (my pick):** turn it on — in the new-install image and once on existing boxes. It goes on without restarting
+     anything. Cost: it must never be turned off by a plain restart again (that stops every app and starts none).
+   - **B:** leave it off. Every Docker update then means ~30 seconds of every app being down, at night.
+   - **If you say nothing:** nothing changes; Docker updates stay unbuilt.
+
+**What I did:**
+- **The host's Debian fixes now install themselves**, after the guest's, on the same night run, only on appliances,
+  never a kernel or boot package, never a reboot. Proven: demo-felhom installed 108 host fixes and stayed healthy; all
+  108 came from Debian. A box made "ring 1" for a test installed exactly the one approved host version it lacked.
+- **A way to put one host package back by hand** is written and proven on demo-hp (and the test taught it two fixes).
+- **The fleet view** in the hub: one line per box with its updates, "reboot needed since", and the tunnel.
+- **The tunnel status is now true:** running, not running, or unknown. I blocked demo-hp's tunnel: after two reports
+  (about 30 minutes) you got a "tunnel down" mail; when I unblocked it, "recovered". A simply stopped tunnel heals
+  itself within 5 minutes, before the hub can even see it.
+- **The night run is fast:** 23–32 seconds when there is nothing to install (target was under 60).
+- **The kernel test on demo-hp (your two reboots):** Secure Boot works with it, but GRUB's "boot once" does not work
+  on our boxes: the second plain reboot came up on the NEW kernel again. So a new kernel that hangs would stay. That
+  must be solved before kernel updates. demo-hp now runs the newer kernel, healthy. A watchdog chip exists on demo-hp.
+- **Found and fixed during the live test:** "reboot needed" was wrong in two ways (fixed in the second release).
+- **New-install image baked and vouched** (0.292.0 with host agent 0.141.1). The golden waiver is gone: not needed.
+- **Rows:** 2 closed, 2 opened-and-closed the same day, 3 opened. The list went from 332 to 333.
+
+**Needs you later (nothing breaks if you wait):**
+- **A host that crashes does not restart by itself** (Linux's "panic" setting is off). Changing it changes how every
+  box behaves, so it is your call, together with kernel updates.
+
+## Today (2026-10-04, afternoon): the guest's security fixes install themselves
+
+**One decision for you (a safe default if you say nothing):**
+
+1. **Undo for a guest update that goes wrong.** I tested it first, as you asked: Proxmox cannot take a snapshot of a
+   customer box at all (the box is linked to the household's drives, and Proxmox refuses). So there is no automatic undo.
+   Today a failed check stops, mails you, and last night's whole-box backup (minutes old) is the undo, by hand.
+   - **A (my pick):** keep it so. Nothing new to build. A restore takes about 1–3 minutes plus losing what apps wrote
+     since the backup.
+   - **B:** build our own disk snapshot under Proxmox. Automatic, but a new mechanism nobody has tested, with a risk to
+     the disk pool.
+   - **If you say nothing:** A stays.
+
+**What I did:**
+- **Your two choices are built.** A returning household's new box sets the old off-site copy aside on night one and
+  starts a new one (nothing deleted). An approved version that Debian already replaced comes from Debian's dated archive.
+- **Guest security fixes now install themselves.** Each night, after the whole-box backup, the demo boxes install
+  Debian's fixes. When both demo boxes run the same versions healthy for 24 hours and one night, the hub approves that
+  set; every other box then installs exactly those versions. Docker, the host and the kernel are not touched.
+  - Proven live: each demo box installed 53 fixes and stayed healthy. With a 2-minute test wait the hub approved the
+    set; then I put back 24 hours. demo-felhom, made "ring 1" for the test, installed exactly the 3 approved versions it
+    lacked and nothing newer. A run where I stopped an app failed its check and mailed you (you got that mail).
+  - You can switch OS updates off per box in the hub. They are ON by default. The household sees one line in its
+    timeline: "System security fixes installed".
+- **The tunnel and two other built-in programs are current** (cloudflared was 4 months old). A new monthly check
+  catches them falling behind. The tunnel came back by itself on both demo boxes within 20 seconds.
+- **Three bugs found and fixed during the live test**, before release.
+- **Two gaps found, now rows:** existing boxes cannot receive the new update tool through the product (only new
+  installs, or by hand — the demo boxes got it by hand); and the hub's "tunnel status" for every box always reads
+  "inactive" because it checks the wrong place.
+- **Rows:** 4 closed (one opened and closed the same day), 5 opened. The list went from 331 to 333.
 
 ## Today (2026-10-04, day): off-site closed, operating-system updates measured
 
@@ -148,8 +219,8 @@ Your licence decisions are recorded: Emby, Plex and n8n stay. recipe-importer ne
 
 ## What needs you
 
-0. **The two decisions at the top of today's section** (a returning household's first night; approved OS updates
-   when Debian has moved on). Each has a safe default if you say nothing. The Hetzner key change stays your call (3 steps, in the list).
+0. **The undo choice at the top of today's section** (A: keep the backup as the undo; B: build our own snapshot).
+   If you say nothing, A stays. Earlier today's two decisions are built. The Hetzner key change stays your call.
 1. **plant-it:** keep the hidden template as it is, or remove it entirely (its image no longer exists). **If you say
    nothing:** it stays hidden; nothing runs it.
 2. **Send the SparkyFitness request, and ask the Tandoor authors** (the "Before the first paying customer" list).

documentation/architecture/00-capability-map.md

@@ -232,7 +232,7 @@ likewise silent. Evidence: `audits/DRILL-r361-2026-08-22/evidence/06-part3-decis
 | **The hub reports LOSS OF VISIBILITY into either off-site store (not just how full it is)** | hub **v0.106.0** (R-339) | **IMPLEMENTED — deliberately NOT proven-live** | Both box checkers count consecutive failed fetch windows and emit `pbsdr_box_unreachable` / `offsite_box_unreachable` (severity `warning`) past a default 3 windows (≈30–45 min), each with a paired `*_recovered` all-clear routed via `recoveredPairedDownTypes` — required because the recoveries are severity `info`, which `severityNotifies` drops. Scopes stay customer-less (`pbsdr-box` / `pool-box`) → operator channel only. Fill logic untouched: a degraded read still drives no band transition. Evidence: `internal/monitor/box_reachability_test.go` + the cross-package wiring test in `internal/notify/`, which asserts an actual operator mail rather than a map entry. **Filed BECAUSE of a measured gap**, not a hypothesis: the 2026-08-18 ep0 outage ran 9 h 37 m with the hub silent | **The gap that remains is R-340**, and it is not small: the ep0 read is the `usage` op, which rides the LOCAL API daemon — the daemon that incident explicitly cleared — so this check would have shown GREEN for that entire outage. It closes "ep0 is unreachable as a host"; it does not close what actually happened. **No live or constructed outage has exercised the emit path**, and one cannot be manufactured against ep0 (Tier 2, protected) |
 | Secrets hygiene: bearer in k8s Secret, no secrets in git, single-quote credential store | hub v0.53, conventions | **IMPLEMENTED** | 07-13 closing bundle | |
 | Operator login password changeable from UI | hub v0.54 | **IMPLEMENTED** | 07-13 | |
-| Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | — | **MISSING** | `felhom-host-install.sh:2133-2136` ("No upgrades are run") | Nothing runs them after install → finding R-812, intention R-808 (added 2026-10-03). Design: `architecture/11-os-updates.md` (NOT RATIFIED, 2026-10-04). **Spike done 2026-10-04** (`audits/os-updates-spike-2026-10-04/`): measured, nothing built — still MISSING |
+| Box operating-system security updates (Proxmox host, guest Debian, Docker engine) | agent v0.142.0, hub v0.132.0 | **PARTIAL — the GUEST and HOST Debian fast lanes and the DOCKER engine slow lane are PROVEN-LIVE (2026-10-04), with the System page, the fleet view and the alarms; the KERNEL lane is MISSING** | Guest: `audits/os-guest-lane-2026-10-04/`. Host + fleet + alarms: `audits/os-host-lane-2026-10-04/`. Docker + System page + crash guard: `audits/os-docker-crash-2026-10-04/` — live-restore on with the same container ids on every box; Docker 29.8.2 on both demo boxes; operator-approved Docker release; a signed undo and a signed ring-1 step; a replay refused; the crash guard restarted demo-hp twice and kept it off the third time. Design `architecture/11-os-updates.md` §5.8, §5.9, §8 | **No automatic undo** (guest: last night's backup; host: by-hand runbook; Docker: a signed undo job); existing boxes need the wrapper, the trust files and the guard by hand (R-840); the kernel lane (R-836); facts reach the hub late after a boot (R-853) |
 | **An ENGLISH-SPEAKING household's first hour: download, install, pair, bind, claim, two apps** | controller **v0.259.0** + hub **v0.119.0** + ISO 1.29.0 + the whole catalog | **PROVEN-LIVE on 0.258.0 with one blocker; THE BLOCKER IS FIXED AND PROVEN, THE WALK IS NOT REPEATED** | `audits/DRILL-first-hour-en-0258-2026-09-20.md` — a fresh install 2026-09-20, one intervention (R-494), stop rule not reached. Then `audits/i18n-closing-2026-09-21/live/` — the three blockers fixed and each proven on a live box or in the operator's inbox: the claim page answers English through the real cookie path; the Backup page's tier names follow the language; and the setup mail carries **four plain-ASCII English words** where the drill's carried `képző-szkítia-ásatás`, one day apart in the same inbox. | **R-596, R-597 and R-598 are CLOSED.** What this row still does NOT claim: **the fixed journey has not been walked end to end by a stranger on a fresh install.** Three fixes proven at the endpoint are not an hour proven by a person, and this project's own rule is that fixes are not a journey (see the recovery-journey row). **Also not walked:** the recovery code (needs ep0), backup/restore/remove/power-cut (proven 2026-09-14), and the two Backup-page *warnings* themselves — guest 9201 is healthy and a healthy box renders none, so they are covered by handler render tests, not live. **Verdict: nothing known now stands between an English-speaking tester and their box — and that is a different sentence from "the walk passed".** |
 | **A deletion of a customer's off-site history is NOTICED within a day** | hub **v0.111.0** (R-431) | **IMPLEMENTED — not yet PROVEN-LIVE** | 09-01 | `hub/internal/monitor/offsite.go` — third signal beside FILL and STALENESS. **On the hub deliberately:** a detector on the box is one the deletion can silence. Alarms when the reported count falls by more than HALF and by at least 5, guarded by `StatsKnown` (R-331), the declared `State` (R-204) and run success (R-100). **Threshold reasoned, not invented:** over 12 898 reports every decrease lands on ZERO and predates `stats_known`; in the 380-report `stats_known` window there are none. **ACCEPTANCE: 9 009 real points replayed → ZERO alarms** (`offsite_r431_test.go`, fixture committed). **What PROVEN-LIVE would need and this does NOT have:** a real drop observed on a live box producing a real mail — the live firing done at ship time was driven through the hub's own path with synthetic counts, which is an end-to-end delivery proof, not a proof that a genuine deletion is caught. |
 

documentation/architecture/03-host-agent.md

@@ -47,7 +47,13 @@ Owns:
 1. **Proxmox lifecycle** — create/start/stop/destroy guests, snapshots, storage allocation. Via a scoped Proxmox API token (the **`FelhomAgent` operator role** — `proxmox-platform.md` §3.6, validated Phase 3 B3) for everything the API covers; raw host ops only where unavoidable.
 2. **Storage management** — attach/classify targets, reconcile the storage manifest, mount USB-by-UUID, present mounts into guests.
 3. **Backup/restore orchestration** — vzdump to the tiers, PBS, snapshot management, and the **self-restore-test**.
-4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub.
+4. **Host & tunnel monitoring** — host metrics, guest up/down, storage-target status, and `cloudflared` health; reports the host domain to the hub. **[FACT, 2026-10-04] The "cloudflared health" leg reads a host unit that does not exist:** `internal/hub/cloudflared.go` runs `systemctl is-active cloudflared` on the HOST, but cloudflared is a container in the guest (below), so every box reports `inactive` (`Unit cloudflared.service could not be found`, demo-hp). R-841. **[FACT, FIXED agent v0.141.0 / controller v0.292.0, R-841 CLOSED]** The agent now reads the guest's `cloudflared` container through the existing `pct exec [0-9]* -- docker inspect -f *` sudoers line: state, exit code and the Docker health status of a check the controller adds (`cloudflared tunnel --metrics localhost:20241 ready` → cloudflared's own `/ready`, 200 only with a connection). Three states: `running` (healthy), `not_running` (stopped, absent, or running but NOT connected), `unknown` (could not ask, or the check is still starting) — `unknown` never alarms. No new sudoers line. **[FACT, agent v0.142.0, R-852]** The host report gains `system`: the Proxmox version and kernel from the
+Proxmox API, plus the OS wrapper's read-only `facts` (host Debian, next-boot kernel, held packages, taint, the crash
+guard; guest Debian, Docker engine, containerd, live-restore), at most every 10 min. **Still no new sudoers line** — the
+facts, `live-restore-on` and the Docker layer all ride `FELHOM_OSAPPLY`. New ROOT-owned files instead (installer 1.30.0;
+by hand on installed boxes, R-840): `/etc/felhom/os-trust.json`, `/etc/felhom/operator-signers` (the wrapper verifies
+Docker authority against them, never against the agent-writable config — `09` decision 93), and the crash guard
+(`/usr/local/sbin/felhom-crash-guard`, its units, `/etc/felhom/crash-guard.conf`).
 5. **Provisioning** — provision a guest **by restoring the golden base image** (§9), deploy the controller into it, hand it its bootstrap config; also **build and refresh the golden base image** itself.
 6. **Hub control loop** — poll for desired state + signed jobs, reconcile, execute, report, heartbeat.
 7. **Local API** — the per-guest authorization gate the controller calls.
@@ -64,7 +70,7 @@ Explicitly does **not**:
 
 - **Native Go binary, systemd service** on the host: boot-start, `Restart=always`, systemd watchdog (kill+restart on hang), journald logging, resource limits.
 - **Root-minimized (boundary settled — Phase 3 B3).** The agent runs as a **non-root** service user with the scoped `FelhomAgent` token for all API-covered work + a **narrow `sudoers` allowlist** for true host ops. Per Phase 3 (B3) the boundary is settled: the entire per-customer guest lifecycle — provision (by restore, §9), config, start/stop, snapshot, backup, **restore**, destroy — is token-covered. Genuine OS-root is confined to: (1) building/refreshing the **golden base image** (`keyctl` create is `root@pam`-only — one-time at enrollment + a maintenance cadence, §9); (2) **host mounts** (USB mount-by-UUID, systemd mount units / fstab); (3) **SMART / hardware sensors**. Root therefore never sits on the per-customer path. See `proxmox-platform.md` §3.6 for the role + boundary table.
-- **`cloudflared` is a separate systemd service**, not embedded in the agent. This is what makes the data path survive control-plane death by construction. The agent **manages and health-watches** it (see §5) but the tunnel does not live or die with the agent process.
+- ~~**`cloudflared` is a separate systemd service**, not embedded in the agent. … The agent **manages and health-watches** it (see §5) but the tunnel does not live or die with the agent process.~~ **[FACT, corrected 2026-10-04 — `11-os-updates.md` C8, R-838]** `cloudflared` is a **container in the customer guest** (`cloudflare/cloudflared:<pin>`), rendered and kept up by the in-guest controller (`felhom-controller/controller/internal/infra/infra.go`, `internal/stacks/infra.go`) and baked into the golden; there is no host systemd unit. It is still NOT embedded in the agent, so the data path survives the agent's death — and the agent does not manage it; it READS its health (R-841, agent v0.141.0). Its version moves only by a controller release (the pin), on the monthly re-test (`runbooks/monthly-floating-retest.md` "Infrastructure pins").
 
 ## 4. Control model — reconcile + signed destructive ops
 
@@ -143,7 +149,7 @@ notification) is the control.
 
 **Box-initiated poll.** The hub never connects inbound. Each poll cycle exchanges:
 
-- **Up:** heartbeat + a host-domain state report — host CPU/RAM/disk, per-guest up/down + spec, storage-target status (USB connected? NFS/CIFS reachable? PBS reachable?), last backup per target, last restore-test result, `cloudflared` health, agent + controller versions, audit-log tail.
+- **Up:** heartbeat + a host-domain state report — host CPU/RAM/disk, per-guest up/down + spec, storage-target status (USB connected? NFS/CIFS reachable? PBS reachable?), last backup per target, last restore-test result, `cloudflared` health (`running` / `not_running` / `unknown` from the guest container's readiness check — R-841), agent + controller versions, audit-log tail.
 - **Down:** the current desired state, any pending signed one-shot jobs, and config (poll interval, update window, policy changes).
 
 **Dead-man's-switch (essential, not optional).** In a box-initiated model the heartbeat
@@ -633,6 +639,13 @@ buildable until then; recorded here so the front-half built in slice 7 lands rea
   never — the binary is what flips. Report field `selfupdate_pending` surfaces a runs-but-never-commits
   binary. v1 scope-outs: no hub-floor auto-update, no failed-update auto-retry (the operator re-signs),
   no pending-timeout auto-rollback.
+- **OS updates, guest fast lane (agent v0.140.0, `11-os-updates.md` §8.1).** A second root-owned wrapper,
+  `/usr/local/sbin/felhom-os-apply`, behind ONE sudoers entry (`FELHOM_OSAPPLY`: `felhom-os-apply --plan
+  /var/lib/felhom-agent/os/plan-*.json`). The agent has no `apt` grant of its own for this; every rule (no removal,
+  no downgrade, no new or unlisted package, Debian origin only, the box's own customer guest only) is in the wrapper,
+  red-proved per rule. The leg runs after a successful primary whole-guest backup, under the heavy-op gate.
+  **[FACT] The signed agent update does NOT carry the wrapper or the sudoers line** — only the installer installs
+  them (R-840).
 - **Controller (the easy case — it's a guest).** The agent owns the controller's lifecycle,
   so the **agent updates the controller**: snapshot-before-update (free rollback, because the
   controller *is* a snapshottable guest) → pull new image → redeploy → health-check → rollback

documentation/architecture/07-backup-architecture.md

@@ -362,6 +362,18 @@ never moved by an update or an undo, and the unit restore accepts it.
 > **R-191 (2026-08-04) — this row was RIGHT and the configuration disagreed with it, weekly, for as long as R-89 has been in force.** The contract has not changed: offsite retention is ep0's, the box's token is write-only, and the box cannot delete its own history. What had not followed was the installer's `keep_last: 2` on the offsite tier, so every weekly run uploaded its snapshot successfully and then failed the whole JOB on a prune the token is refused — `whole_guest_backup_failed` in the operator's inbox about a backup that had already succeeded. Fixed in installer **1.25.0** (`keep_last: 0`) and on both live boxes; a gate now asserts it. **Verified before changing it:** ep0's two prune jobs have run every day since 2026-07-27, 18 tasks, all OK. A doc that states the contract does not enforce it — the gate does.
 
 
+**[FACT, 2026-10-04 — agent v0.140.0, `11-os-updates.md` §8.1] The OS leg closes the night.** After the
+whole-guest backup (the controller drives it, inside [W+2h, W+6h)) ends SUCCESSFULLY on the primary tier, the agent
+waits 90 s and runs the guest's Debian fast lane — still holding the host-wide heavy-op gate, so it never overlaps a
+backup or a restore-test; at most once per 20 h. The backup minutes old is the guest's undo (no snapshot is possible,
+R-837). A failed or missed backup → no OS leg that night. **[FACT, 2026-10-04 — agent v0.141.1, `11` §8.2]** On an
+appliance, the HOST step follows under the same gate: after a healthy guest step only (a failed or unhealthy guest
+step skips it), Debian-origin fixes only, never a kernel, boot or firmware package, never a reboot. Measured: both
+steps with nothing to install, 23–32 s; a 108-package host pass, 70 s. **[FACT, 2026-10-04 — agent v0.142.0, `11` §5.8]** On a RING-0 box a third step
+follows, still under the gate: the guest's Docker engine set (live-restore turned on first, once, by reload; every
+container must keep its id). Ring 1 never takes it in the night leg. Measured: the three steps with nothing to install,
+38–51 s; a six-package Docker step, ~50 s.
+
 **[FACT]** The three nightly legs derive from **one** customer-settable window start W at fixed
 offsets — db-dump at W, Tier-2 at W+60m, offsite at W+105m — so they can never be misordered
 (`cmd/controller/main.go:604-607`). Both boxes run W = `02:30`.

documentation/architecture/08-alarm-ladder.md

@@ -310,6 +310,36 @@ message, not a wider cooldown.
 
 ---
 
+## 6.3 Box alarms outside the app ladder: the tunnel and OS updates [DESIGN, hub v0.131.0, 2026-10-04]
+
+These are **operator-only** (the household can act on none of them — except `host_restarted_after_crash`, the household's
+one info line beside `host_crash_restart`; `operatorOnlyEvents`, pinned by
+`TestOSUpdateEvents_OperatorOnlyExceptApplied`). They go through the same dispatcher and severity contract (§6.1):
+`info` is recorded and never mailed; `warning` and `error` are mailed.
+
+| Event | Severity | Raised when | Cleared | Pinned by |
+|---|---|---|---|---|
+| `tunnel_down` | error | the box's two newest host reports say the tunnel is `not_running` (more than one report cycle, 15 min) and the one before did not | the first `running` after it → `tunnel_recovered` (info) | `api/tunnel_test.go` |
+| `os_update_stale` | warning | no successful OS leg for **7 days** while the switch is ON (agents that can run the leg only); the mail names the likely reason (box not reporting / the last leg's failure / no good night backup) | a successful leg | `TestAlarm_StaleLeg`, `TestAlarm_StaleNamesTheReason` |
+| `os_reboot_needed` | warning | the host has needed a reboot for **14 days** (from the FIRST scanned report that said so) | a scanned pass that finds nothing (agent ≥ 0.141.1 scans the host every pass) | `TestAlarm_RebootNeeded`, `TestRebootNeeded_ClearedByAScannedPass` |
+| `os_ring0_stalled` | error | ring 0 approved nothing for **7 days** in a layer while it has pending FAST-lane updates (a pending kernel does not count) | a new release | `TestAlarm_Ring0Stalled` |
+| `os_not_covered` | warning | a ring-1 box has had fast-lane packages no approved release names for **14 days** | the packages are covered or gone | `TestAlarm_NotCovered` |
+| `host_crash_restart` | warning | the box's crash guard reports a NEW unclean boot (a crash, a power cut or a hard reset; hub v0.132.0) | — (one per boot) | `api/crash_test.go` |
+| `host_crash_guard_tripped` | error | the guard tripped: the next crash leaves the box OFF | the re-arm → `host_crash_guard_rearmed` (info) | `api/crash_test.go` |
+| `host_kernel_oops` | warning | a kernel oops this boot (taint D) — the box keeps running | — (once per boot) | `api/crash_test.go` |
+
+- **`unknown` never alarms** (R-96 rule 3): a probe that could not ask is neither up nor down. An `unknown` report
+  breaks a `not_running` run.
+- **A stopped cloudflared heals itself before the hub can see it** (measured 2026-10-04): the controller's
+  protected-container check recreates it within 5 minutes, and the host reports every 15. So `tunnel_down` catches
+  what the box cannot heal — a running container with no connection (wrong token, blocked network).
+- The OS alarms are checked **hourly**, re-sent at most **once a week** while true, and forgotten when false, so the
+  next occurrence is announced again. The four numbers are configuration (`OS_ALARM_STALE_AFTER`,
+  `OS_ALARM_REBOOT_AFTER`, `OS_ALARM_RING0_STALL_AFTER`, `OS_ALARM_NOT_COVERED_AFTER`) — *decided by CC unattended,
+  operator may reverse* (`11` §8.3).
+
+---
+
 ## 7. The intent test [DESIGN, R-386 — CLOSED controller v0.223.0]
 
 **"The customer stopped this" is asked of the FIELD THAT RECORDS IT, never inferred from the state.**

documentation/architecture/09-update-architecture.md

@@ -765,6 +765,67 @@ its length, and both fixes cost something the household would notice — operato
 80. **Next build: the guest fast lane** (`11` §8 step 2), with R-837 (the snapshot undo) measured first and R-838 (the
     infrastructure images) in the same session. *Operator ruling 2026-10-04 ~10:17.*
 
+### 2026-10-04 (~12:20) — three operator rulings (recorded before the work)
+
+81. **The undo for a failed OS update is the whole-guest backup, restored by hand (R-842, option A).** Nothing new is
+    built. **Rejected:** a home-made LVM-thin snapshot behind Proxmox's back (unmeasured; thin-pool risk).
+    *Operator ruling 2026-10-04 ~12:20.*
+82. **R-840 (a route for wrappers/sudoers to installed boxes) is not built now** — *"There are no older boxes."* No
+    installed box exists outside the two demo boxes, and they take wrapper changes by hand. The row stays open, not
+    re-ranked. *Reviewer's note, recorded as written:* the ruling is right for today, but every box installed from now on
+    becomes an "older box" the first time the wrapper or the sudoers line changes again; R-840 must be solved before the
+    first box that cannot be reached by hand. *Operator ruling 2026-10-04 ~12:20.*
+83. **Next: the tunnel status (R-841), the host fast lane (`11` §8 step 3), and other OS-update improvements.**
+    *Operator ruling 2026-10-04 ~12:20.*
+
+### 2026-10-04 (evening) — decided by CC unattended — operator may reverse (System page / Docker / crash-restart brief)
+
+90. **How does a box tell a crash boot from a clean one?** Options: (a) `pstore` — measured on demo-hp: `efi_pstore` is on,
+    yet a real panic saved NOTHING; (b) the previous boot's journal ends without shutdown lines — readable only after
+    the journal is up, and slow; (c) a marker written by an ExecStop at every orderly shutdown. **Chosen (c):** cheap,
+    early, measured to work; cost: a power cut and a hard reset count as a crash too (they cannot be told apart on these
+    boxes). `11` §5.9.
+91. **`kernel.panic_on_oops`?** Options: set it (an oops becomes a restart, which may loop on a bad driver) or leave it
+    0 and REPORT an oops (taint bit D) to the operator. **Chosen: leave 0, report** — a box with an oops usually keeps
+    serving the household; the hub mails `host_kernel_oops`. `11` §5.9.
+92. **The guard numbers.** The brief said both "at 3 it sets panic=0 so the next crash leaves the box off" (= the 4th)
+    and "if it crashes 3 times within one hour, it stays off" (= the 3rd, the operator's own page). **Chosen: the
+    operator's words** — the 3rd unclean stop within 60 minutes leaves the box off (the guard trips at the 2nd crash
+    boot); `kernel.panic` 10 s; re-arm after 24 h of normal running. All four are `/etc/felhom/crash-guard.conf`.
+93. **Who may authorize a Docker step on the box itself?** Options: the agent's word (its config is agent-writable);
+    the wrapper verifying the signed job against ROOT-owned files. **Chosen: the wrapper verifies** — `ssh-keygen -Y
+    verify` against `/etc/felhom/operator-signers`, bound to `/etc/felhom/os-trust.json` `host_id`, a root-owned nonce
+    record; an unsigned ring-0 step only with that file's `ring0_slow_lane: true`, set by hand on the demo boxes only.
+    Cost: two more root files per box (the installer writes them; R-840 for installed boxes). `11` §5.8.
+94. **The System page colours** use the alarm thresholds themselves (red = an `08` alarm would fire; amber = worth a
+    look; `unknown` = amber with its reason). No second set of numbers. `11` §5.7.
+
+### 2026-10-04 (~15:17) — three operator rulings (recorded before the work)
+
+87. **Docker `live-restore` is ON for every box** (`11` §5.8, option A). It is turned on once — by the golden and by a
+    one-time step on each installed box — and never turned off by a plain restart (R-835). *Operator ruling 2026-10-04
+    ~15:17.*
+88. **A crashed host restarts by itself, with a limit** (R-851): *"Yes, but maybe not indefinitely."* After a limit of
+    crashes in a short time the box stays off and the operator is told. *Operator ruling 2026-10-04 ~15:17.*
+89. **The operator must see the boxes' OS and Proxmox versions in the hub** (*"Where should I be able to see the
+    OS/Proxmox versions of the boxes?"* — today: nowhere; R-852). *Operator ruling 2026-10-04 ~15:17.*
+
+### 2026-10-04 (afternoon) — decided by CC unattended — operator may reverse (host fast lane brief)
+
+84. **Which record proves a box is an appliance, for the host fast lane?** Options: (a) `agent.json`
+    `deployment_mode` — the agent can write it, so a compromised agent could claim "appliance" and unlock host
+    updates; (b) the installer's ROOT-owned `/var/lib/felhom-install/state.json` `mode` — written once, as root, at
+    install. **Chosen (b):** the wrapper is the root fence and must not trust a file the agent can change. Cost: a box
+    whose install record is missing gets no host step (fails closed). `11` §8.2.
+85. **The four OS alarm thresholds.** Options: shorter (3/7 days — noisy: a weekend away alarms) or longer (14/30 —
+    a stopped box goes unseen for weeks). **Chosen:** no OS leg 7 days, reboot needed 14, ring 0 stalled 7, not covered
+    14; all four are configuration. Cost: a real stop is seen after a week, not a night. `11` §8.3, `08` §6.3.
+86. **A second release of the agent (v0.141.1) and the hub (v0.131.1) in the same session**, against "one release per
+    repo". Options: (a) keep v0.141.0 / v0.131.0 and file the defect — the host "reboot needed" stays wrong (it hid
+    `lxc-start`, and a reboot never cleared it), so the new 14-day alarm would fire on rebooted hosts; (b) patch now.
+    **Chosen (b):** a known-false operator alarm is worse than an extra release; both patches are small and red-proved.
+    R-846.
+
 ### 2026-09-30 (day) — operator notes, recorded before the work
 
 - **The day brief runs by day.** Every backup and automatic-update test is started by hand — the night chain's debug

documentation/architecture/11-os-updates.md

@@ -2,7 +2,7 @@
 
 > | | |
 > |---|---|
-> | **Status** | **NOT RATIFIED — a PROPOSAL with one operator ruling, corrected by the 2026-10-04 spike (§7.1, corrections C1–C12 below).** Ratification is Viktor's review, not an editor's. |
+> | **Status** | **NOT RATIFIED — a PROPOSAL with operator rulings, corrected by the 2026-10-04 spike (§7.1, C1–C12); §8 step 2 BUILT 2026-10-04 (§8.1).** Ratification is Viktor's review, not an editor's. |
 > | **Written** | 2026-10-04, by the reviewer (project Claude), before any spike. |
 > | **Verified against** | felhom.eu `d07a1a9` · felhom-controller `99a1497` (v0.290.0) · felhom-agent `d766666` (v0.138.0) · hub v0.128.0 |
 > | **Freshness** | **CURRENT** as of 2026-10-04. The spike `TASK-backup-close-and-os-updates-spike-2026-10-04` adds measurements here as `[FACT]` and corrects every claim it disproves. Mark this file STALE when it falls behind what the product does. |
@@ -324,18 +324,81 @@ must never overlap a backup, a restore-test or a self-update.~~
 |---|---|---|
 | Guest packages | Restore the guest snapshot taken just before the update | It also undoes app data written after the snapshot. Use it only inside the health window, before apps have written much. After that, install the previous version (needs OPEN Q1). **[FACT] (C6)** Customer guests are on LVM-thin and can snapshot; scratch 9202 (`dir` storage) cannot (`snapshot feature is not available`), so the measured undo was a whole-guest backup + restore: **73 s down**, all apps healthy, libc back. The snapshot rollback itself is unmeasured (R-837). |
 | Docker engine | Install the previous version (Docker's repository keeps old versions — OPEN Q2) | Every container restarts again. **[FACT] (C5)** Docker keeps 46 `docker-ce` versions. A step (either way) without `live-restore`: 6 of 6 containers restart, the app silent **26.5–30 s**, healthy at +42–45 s. With `live-restore` on: **0 restarts, no gap**, also across a containerd step. But a restart that turns `live-restore` OFF stops every container and starts NONE (`unless-stopped` ignored) — on 9202 they stayed down until restarted by hand; `systemctl reload` turns it on but not off. |
-| Host packages | Install the previous version | Only if the source still has it (OPEN Q1/Q2). **[FACT]** `rsync` back to its pre-update version: `not found`; `libpng16-16t64` back to the point-release version: worked. A Debian undo needs the snapshot archive (C2). |
-| Host kernel | ~~Boot the previous kernel. Proxmox can boot a new kernel **once** (`proxmox-boot-tool kernel pin <ver> --next-boot`). If that boot fails, the next boot uses the old kernel again. Make the new kernel permanent only after a healthy boot.~~ **[FACT] (C4)** Both demo hosts boot UEFI + GRUB (no proxmox-boot-tool ESPs). **Installing a kernel makes it the GRUB default at once.** `--next-boot` on GRUB writes an ordinary `GRUB_DEFAULT` + `update-grub`; `proxmox-boot-cleanup.service` clears it only once a boot reaches userspace. Measured on demo-hp (old kernel pinned permanently FIRST, new pinned for next boot): boot 1 → `7.0.14-20-pve`, healthy, 60 s; boot 2 → back on `7.0.2-6-pve`, healthy, 76 s. **So the fallback works after a boot that succeeds; read from the code, a kernel that hangs before userspace stays the default on every power cycle.** `[PROPOSAL]` use GRUB's own one-shot (`GRUB_DEFAULT=saved` + `grub-reboot`) with the old kernel as the saved default — unmeasured (R-836). | ~~If the new kernel hangs, someone must switch the box off and on. The spike checks whether a hardware watchdog can do that (OPEN Q4).~~ **[FACT]** Only `softdog` runs (loaded by `watchdog-mux`); it cannot rescue a kernel that never boots. demo-hp has an AMD FCH whose `sp5100_tco` driver ships but is not loaded; untested. A hang still needs a person. |
+| Host packages | Install the previous version | Only if the source still has it (OPEN Q1/Q2). **[FACT]** `rsync` back to its pre-update version: `not found`; `libpng16-16t64` back to the point-release version: worked. A Debian undo needs the snapshot archive (C2). **[FACT, 2026-10-04]** Proved by hand: `runbooks/os-updates-host-undo.md` (demo-hp, `tzdata` back one version from `snapshot.debian.org`, held, released). Use the NEW version's `first_seen` as the timestamp when there is no previous host release; a package that pins its siblings (`eject` → `libmount1 =`) goes back only with them. |
+| Host kernel | ~~Boot the previous kernel. Proxmox can boot a new kernel **once** (`proxmox-boot-tool kernel pin <ver> --next-boot`). If that boot fails, the next boot uses the old kernel again. Make the new kernel permanent only after a healthy boot.~~ **[FACT] (C4)** Both demo hosts boot UEFI + GRUB (no proxmox-boot-tool ESPs). **Installing a kernel makes it the GRUB default at once.** `--next-boot` on GRUB writes an ordinary `GRUB_DEFAULT` + `update-grub`; `proxmox-boot-cleanup.service` clears it only once a boot reaches userspace. Measured on demo-hp (old kernel pinned permanently FIRST, new pinned for next boot): boot 1 → `7.0.14-20-pve`, healthy, 60 s; boot 2 → back on `7.0.2-6-pve`, healthy, 76 s. **So the fallback works after a boot that succeeds; read from the code, a kernel that hangs before userspace stays the default on every power cycle.** ~~`[PROPOSAL]` use GRUB's own one-shot (`GRUB_DEFAULT=saved` + `grub-reboot`) with the old kernel as the saved default — unmeasured (R-836).~~ **[FACT, 2026-10-04, demo-hp, operator's word before each reboot] GRUB's one-shot is NOT a one-shot here either.** With `GRUB_DEFAULT=saved` (old 7.0.2-6 saved) and `grub-reboot` 7.0.14-20: boot 1 → 7.0.14-20, **Secure Boot ON, booted fine** (signed kernel, shim → GRUB); but `/boot` is ext4 on LVM, GRUB cannot write its environment block there (`grub-reboot` warns so itself), `next_entry` was never cleared, and boot 2 with no command → **7.0.14-20 again**. A kernel lane needs a writable env block (the ESP) or a userspace "boot good" step — R-836. demo-hp left on 7.0.14-20, saved default 7.0.14-20, both kernels installed (`audits/os-host-lane-2026-10-04/partE/`). | ~~If the new kernel hangs, someone must switch the box off and on. The spike checks whether a hardware watchdog can do that (OPEN Q4).~~ **[FACT]** Only `softdog` runs (loaded by `watchdog-mux`); it cannot rescue a kernel that never boots. demo-hp has an AMD FCH whose `sp5100_tco` driver ships but is not loaded; untested. A hang still needs a person. **[FACT, 2026-10-04]** `sp5100_tco` is blacklisted by the Proxmox kernel package; loaded by hand it answers (`SP5100 TCO timer`, 60 s, inactive, nowayout 0 — read from sysfs, never opened, so never armed; unloaded). `kernel.panic = 0`: a panic leaves the host stopped (R-851). |
 
 ### 5.7 Telling people
 
 - **Operator:** a hub event for each update and each failure; a fleet view showing each box's OS release,
   how far behind it is, and whether it needs a reboot. A box that is more than N days behind the newest
   approved release raises an alarm (`08`).
+- **[FACT, hub v0.132.0] The System page** (`/system`, R-852, decision 89): one row per box — ring and switch with
+  buttons, the tunnel, host Proxmox / running and next-boot kernel / Debian / release / pending / not covered / held /
+  reboot needed / `kernel.panic` / oops / crash restarts / the guard, guest Debian / release / pending / restart needed,
+  Docker engine / containerd / live-restore / release, the last leg — and above it the releases, what ring 0 runs, "Approve
+  now" and "Approve Docker set". Colours are the alarm thresholds (decision 94). Hosts shows Proxmox / kernel too.
 - **Household:** one line on the timeline in both languages, informal voice: what was updated and
   whether the box restarted. Telling households in advance that the box may restart at night is a
   **promise to users**. That is the operator's decision when the slow lane is built.
 
+### 5.8 The Docker engine slow lane — BUILT 2026-10-04 (agent v0.142.0, hub v0.132.0)  `[FACT]`
+
+**As built** (evidence `audits/os-docker-crash-2026-10-04/partB/`; decisions 87, 93):
+- **live-restore ON**: the golden bakes it (`build-golden.sh` 3.1.0, fail-closed assertion); an installed box gets it once
+  by the wrapper's `live-restore-on` (merge into daemon.json + `systemctl reload docker`). Measured: the same container ids
+  after (9202 6/6 by hand — R10 refuses a scratch guest by design —, demo-hp 24/24, demo-felhom 5/5).
+- **The step** runs in the night leg after a healthy guest and host step, **ring 0 only**, `select pending-docker`,
+  allowed by the wrapper only with the root-owned `ring0_slow_lane` mark. **Ring 1 and every undo** only through a signed
+  `os_docker_step` the wrapper re-verifies itself (decision 93). Measured: 29.7.x → 29.8.2 on both demo boxes, every id
+  kept; a signed undo to 29.7.2 on demo-hp (41 s, ids kept) and back; demo-felhom as ring 1 by a signed job; a replayed
+  job refused by the agent (nonce).
+- **Approval**: the hub never approves a Docker set automatically; the System page's button works after every ring-0 box
+  ran the set in 2 healthy night Docker steps (`OS_DOCKER_APPROVE_NIGHTS` TEST override, logged). An approval nudges no
+  box. Undo: `runbooks/os-updates-docker-undo.md`.
+
+**The design as written before the build:**
+
+Built from C5 (measured) and R-835. **The one decision it needs is in STATUS: `live-restore` on, fleet-wide.**
+
+- **What moves:** `docker-ce`, `docker-ce-cli`, `containerd.io`, `docker-buildx-plugin`, `docker-compose-plugin`,
+  `docker-ce-rootless-extras` in the customer guest — today the six "not covered" packages on every box. One
+  approved **engine set** at a time, like a controller floor: the operator approves it (slow lane, §5.2), after ring 0
+  has run it for at least 2 nights healthy. Never two steps in one night.
+- **Precondition: `live-restore` ON.** Without it an engine step restarts every container — 26.5–30 s of silence,
+  healthy at +42–45 s (C5). With it: 0 restarts, no gap, also across a containerd step. Turning it ON is safe
+  (`systemctl reload docker` applies it without a restart, C5); turning it OFF later by a plain restart stops every
+  container and starts none (R-835) — so it is turned on once, by the golden and by a one-time fleet step, and never
+  turned off by the lane.
+- **Who and when:** the agent, through the same wrapper (`lane: slow`, refusal R3: only inside a verified signed
+  operator job, R-530's mechanism), in the guest, after the guest and host fast-lane steps, under the same heavy-op
+  gate, on a night the operator scheduled. Debian origin rule replaced by "origin `Docker CE`, exactly these names".
+- **Health:** the guest rule (§8.1) plus `docker version` reports the approved engine, and every container running
+  at the start is running with the SAME container id (proof that `live-restore` held). A changed id is
+  `health_failed` even if the app is healthy — it means the households' apps restarted when they should not have.
+- **Undo:** install the previous engine set (Docker's repository keeps 46 versions, C2) — by an operator job, with
+  `live-restore` still on, so the undo is also restart-free.
+- **Not covered here:** the golden's own engine (baked weekly; a new golden carries the approved set), and BYO hosts
+  (the guest is ours on both, so the lane applies there too).
+
+### 5.9 A crashed host restarts, with a limit — BUILT 2026-10-04 (agent v0.142.0, installer 1.30.0)  `[FACT]`
+
+Decision 88 (R-851): *"Yes, but maybe not indefinitely."* Evidence `audits/os-docker-crash-2026-10-04/partC/`.
+- **Measured first (demo-hp, the operator's word before each crash):** `kernel.panic = 10` + `echo c >
+  /proc/sysrq-trigger` → the box restarted by itself in 54 s, on the same kernel (the saved default), the agent up 18 s
+  after the boot. `kernel.panic` set by `sysctl -w` is gone after the restart (back to 0) — it must be set at every boot.
+- **The crash signal** (decision 90): `efi_pstore` is on, yet a real panic saved NOTHING; the journal and `last` show
+  only "no shutdown". The guard uses a **clean-stop marker** (the unit's ExecStop at every orderly shutdown); a boot
+  without it followed a crash, a power cut or a hard reset — counted alike.
+- **The guard** (`felhom-crash-guard`, early boot unit + hourly re-arm timer): armed → `kernel.panic = 10`; the 2nd
+  unclean boot within 60 minutes TRIPS it (`kernel.panic = 0`), so **the 3rd crash within the hour leaves the box off**
+  (decision 92 — the operator's words); re-arms after 24 h of normal running or `felhom-crash-guard rearm`. A crash before
+  the unit runs (very early boot) leaves the box off: the safe side. `panic_on_oops` stays 0; an oops is reported
+  (decision 91).
+- **Telling people:** each unclean boot = an operator mail (`host_crash_restart`) + the household's line; the trip = an
+  alarm (`host_crash_guard_tripped`); re-arm and oops announced once (`08` §6.3). The System page shows the guard.
+- **Measured live:** crash 1 → back in 54 s; crash 2 → back in 53 s, guard tripped; crash 3 → **stayed off** until the
+  operator switched it on; the hub mailed the trip (15 min after the boot — R-853); re-armed by hand.
+
 ---
 
 ## 6. Risks and edge cases
@@ -403,15 +466,99 @@ not covered: 79 Proxmox + 1 Tailscale on the host (slow lane / not ours), 6 Dock
 Each step returns to the operator for go or no-go.
 
 1. **Spike** (measure Q1–Q10; no product code).
-2. **Guest Debian, fast lane.** Lowest risk: a snapshot undo exists.
-3. **Host Debian, fast lane** (no kernel, no Proxmox packages).
-4. **Fleet view and alarms** (§5.7).
-5. **Slow lane: Docker engine.**
+2. **Guest Debian, fast lane.** ~~Lowest risk: a snapshot undo exists.~~ **BUILT 2026-10-04** — agent v0.140.0, hub
+   v0.130.0, installer 1.29.0; §8.1. **There is no snapshot undo** (R-837, measured).
+3. **Host Debian, fast lane** (no kernel, no Proxmox packages). **BUILT 2026-10-04** — agent v0.141.1, hub v0.131.1;
+   §8.2.
+4. **Fleet view and alarms** (§5.7). **BUILT 2026-10-04** — hub v0.131.0/v0.131.1; §8.3.
+5. **Slow lane: Docker engine.** **BUILT 2026-10-04** — agent v0.142.0, hub v0.132.0; §5.8.
 6. **Slow lane: host kernel and Proxmox packages, with the reboot.**
 7. **Later:** the Proxmox major upgrade (PVE 9 → 10), drilled on ring 0 first.
 
 ---
 
+### 8.1 Step 2 as BUILT (2026-10-04)  `[FACT]`
+
+Evidence: `audits/os-guest-lane-2026-10-04/` (parts A–G). Brief: guest fast lane, decisions 78–80 (`09` §3).
+
+- **The undo (R-837, measured first): none automatic.** PVE refuses ANY snapshot of a customer guest —
+  `PVE/AbstractConfig.pm:755-757` skips non-snapshot mounts only for a snapshot named `vzdump`, and every customer
+  guest carries the host-path binds mp8/mp9. As the agent's token and as root: `snapshot feature is not available`.
+  The token's role HAS `VM.Snapshot` / `VM.Snapshot.Rollback`. So §5.6's first row does not exist: a failed health
+  check stops, reports `health_failed`, and the hub mails the operator; the whole-guest backup taken minutes earlier
+  is the undo, by hand. The choice of a real undo is in STATUS (R-842).
+- **The wrapper** `felhom-os-apply` (agent repo `configs/`), Python 3 stdlib, per §5.4.1 with refusals R1–R13 (R12:
+  the host layer; R13: dpkg still broken after the repair). **Changed from the draft** *(decided by CC unattended —
+  operator may reverse)*: one sudoers entry (`--plan <file>`); the plan's `mode` field (`inventory` / `apply` /
+  `health`) replaces a separate `--repair-only` (the repair runs first on every apply); Python, because a JSON plan
+  cannot be parsed safely in sh. The box's own customer guest is "the guest that binds `/mnt/felhom-drives`" (R10).
+- **The leg** (agent `internal/osupdate`): after a SUCCESSFUL primary whole-guest backup, inside the backup's
+  goroutine before the host-wide heavy-op gate is released (so never beside a backup or a restore-test, C10), 90 s
+  after the backup, at most once per 20 h. *Decided by CC unattended — operator may reverse:* the 90 s settle and the
+  20 h gap; the controller's own self-update (04:30) is not detected — the 5-minute health wait absorbs a restart.
+- **The health rule** (`HealthVerdict`, pinned): docker answers, the guest resolves `deb.debian.org`, the controller's
+  health check is `healthy`, and every container running at the START of the leg runs again (healthy if it was). The
+  baseline merges the inventory's reading with the apply's own — found live: an app stopped between them escaped the
+  first rule. Wait 5 min, poll 15 s.
+- **Rings and the switch** (hub, per box): ring 0 = demo-hp + demo-felhom, everything else ring 1; switch ON by
+  default; OFF → the box reports, installs nothing. A box with no `os_update` block (older hub) = ring 1, ON, no
+  release.
+- **The approval rule (the ruled "1–2 day wait")**: every Debian / Debian-Security package=version that ALL ring-0
+  boxes having it agree on; approved when, since that set was first seen, **24 h** passed with every ring-0 report
+  healthy and every ring-0 box completed **1** post-backup night run (`OS_APPROVE_AFTER`, `OS_APPROVE_NIGHTS`; an
+  override is logged as a TEST configuration). The approval time is the snapshot.debian.org timestamp (decision 79).
+  "Approve now" is an operator event. *The 24 h / 1 night numbers: decided by CC unattended within the ruled 1–2 days.*
+- **The household's line**: hub event `os_update_applied` (info: on the household's hub timeline, never mailed;
+  hu/en in the bundle). There is no surface on the box itself (R-844).
+- **Measured live**: ring 0 — 53 Debian packages on each demo box (18.7–31.7 s inside the wrapper; 174–226 s for the
+  whole leg incl. the inventory), healthy, 6 Docker updates "not covered"; approval — with a 2-minute TEST wait, a
+  272-package release approved automatically, then the ruled values restored; ring 1 — demo-felhom installed exactly
+  the 3 approved versions it lacked (269 already current) and left the newer Docker packages alone; a deliberately
+  stopped app → `health_failed` after 5 min, the operator mailed, the household line recorded.
+- **Not delivered to existing boxes by the product**: the wrapper and the sudoers line reach a box only through the
+  installer; the signed agent update replaces the binary only (R-840). The demo boxes got them BY HAND.
+
+### 8.2 Step 3 as BUILT (2026-10-04)  `[FACT]`
+
+Evidence: `audits/os-host-lane-2026-10-04/` (parts A–G).
+
+- **Where it runs:** appliances only. The proof is the ROOT-owned install record `/var/lib/felhom-install/state.json`
+  `mode: appliance` (written by the installer as root); the agent-writable `agent.json` `deployment_mode` is not
+  trusted for this. A BYO host gets no host step (wrapper refusal **R12**, now lifted only for lane fast / layer host
+  on an appliance). *Decided by CC unattended — operator may reverse.*
+- **What:** origin `Debian` / `Debian-Security` only, and never a kernel, boot or firmware package (name pattern
+  `HOST_SLOW_RE` — `linux-*`, `proxmox-kernel*`, `pve-kernel*`, `pve-firmware`, `firmware-*`, `grub*`, `shim*`,
+  `systemd-boot*`, `*-microcode`, `efibootmgr`; refusal **R14**). The hub leaves the same names out of the host
+  candidate.
+- **When:** in the same leg, after the guest step, under the same heavy-op gate. A failed or unhealthy guest step
+  skips the host step.
+- **The host health rule** (`HostHealthVerdict`, pinned in `internal/osupdate`): `felhom-agent`, `pveproxy`,
+  `pvedaemon`, `pvestatd` and `pve-cluster` are `active`; the customer guest runs; the guest health rule (§8.1)
+  passes; the tunnel is `running` — an `unknown` tunnel does not fail it, `not_running` does. Same 5-minute wait.
+- **Separate approved sets:** host and guest releases are separate (`os-host-…`, `os-guest-…`), each by the same rule
+  (24 h, 1 night of THAT layer, every ring-0 box). Ring 1 receives `host_release` beside `release`.
+- **Reboot needed:** reported when PID 1 or `lxc-start` maps a replaced file, with the date of the first scanned
+  report that said so. **Never reboots.** The host is scanned on every pass, so a reboot clears it (v0.141.1 — v0.141.0
+  hid `lxc-start` and never cleared, R-846).
+- **Undo:** by hand, `runbooks/os-updates-host-undo.md` (proved). No automatic undo.
+- **Speed (R-845):** one call per layer; both steps with nothing to install 23–32 s; a 108-package host pass 70 s.
+- **Measured live:** demo-felhom ring 0 installed 108 Debian host packages, all Debian origin (checked against apt),
+  healthy; a 605-package host release approved (TEST wait 2 min / 0 nights, logged, reverted to 24 h + 1 night);
+  demo-felhom as ring 1 installed exactly the one version it lacked (604 already current).
+
+### 8.3 Step 4 as BUILT (2026-10-04)  `[FACT]`
+
+- **The fleet view** (`GET /os/fleet`, operator): one line per box — ring, switch, the tunnel, and per layer: the
+  release, the last outcome, the last successful leg, pending, not covered, restart needed, reboot needed since, the
+  wrapper's own seconds.
+- **The four alarms** (`08` §6.3), operator-only, hourly, at most weekly while true: no successful OS leg for
+  **7 days** while the switch is ON (naming the likely reason); reboot needed for **14 days**; ring 0 approved nothing
+  for **7 days** while it has pending fast-lane updates; not-covered fast-lane packages for **14 days**. The four
+  numbers are configuration (`OS_ALARM_*`). *Decided by CC unattended — operator may reverse:* 7 days = a week of
+  missed nights is past any normal hiccup (a box off for a weekend does not alarm); 14 days for reboot and coverage =
+  two weekly golden cycles, both need a person anyway.
+- **The tunnel** (R-841): `running` / `not_running` / `unknown`; `tunnel_down` after two `not_running` reports.
+
 ## 9. Where the rest lives
 
 - The finding: **R-812** (`backlog/OPEN-ITEMS.md`). The intention: **R-808** (`backlog/ROADMAP.md`).

documentation/audits/os-docker-crash-2026-10-04/partA/hub-redproofs.txt

@@ -0,0 +1,31 @@
+# Hub v0.132.0 red-proofs, 2026-10-04T14:04:22Z
+# First run: 2 of 10 NOT CAUGHT (the parser test never fed empty facts fields; the page test counted the word 'unknown' but not its amber class). Both tests strengthened; the rerun below catches all 10.
+[CAUGHT] unknown, never empty (parser): internal/sysfacts/sysfacts.go mutated; TestParse_AbsentAndPartialAreUnknown ran=True failed=True
+    sysfacts_test.go:38: empty facts fields: {Present:true PVEVersion:x KernelVersion:unknown VMID:0 FactsError: ReadAt: Host:{Debian:unknown KernelRunning:unknown KernelNextBoot:unknown KernelNextBootSou
+[CAUGHT] Docker never auto-approved: internal/osupdates/service.go mutated; TestDocker_NeverAutoApproved ran=True failed=True
+    docker_test.go:33: a Docker set was auto-approved: &{ID:os-docker-20261006-140000 Layer:docker Fingerprint:a4ec54235cdd17a5 ApprovedAt:2026-10-06 14:00:00 +0000 UTC ApprovedBy:auto PackagesJSON:[{"nam
+[CAUGHT] Docker button needs 2 nights: internal/osupdates/service.go mutated; TestDocker_ApproveNeedsTwoHealthyNightsOnEveryRing0Box ran=True failed=True
+    docker_test.go:45: approved after one night: <nil>
+[CAUGHT] a Docker approval nudges no box: internal/osupdates/service.go mutated; TestDocker_ApproveNeedsTwoHealthyNightsOnEveryRing0Box ran=True failed=True
+    docker_test.go:62: a Docker approval must nudge no box (ring 1 takes it by a signed job): [cust1]
+[CAUGHT] crash event once per boot: internal/api/crash.go mutated; TestCheckCrash_EventsOncePerFact ran=True failed=True
+    crash_test.go:45: restart events: op=4 household=4
+[CAUGHT] first sighting is quiet: internal/api/crash.go mutated; TestCheckCrash_EventsOncePerFact ran=True failed=True
+    crash_test.go:39: first sighting flooded 1 events
+[CAUGHT] page: unknown cell: internal/web/system.go mutated; TestSystemPage_FullPartialUnknown ran=True failed=True
+    system_test.go:65: an unknown value must be shown amber with its reason, got 0
+[CAUGHT] page: Docker button gate: internal/web/templates/system.html mutated; TestSystemPage_DockerButtonOnlyWhenReady ran=True failed=True
+    system_test.go:87: button shown after ONE night
+[CAUGHT] buttons need the operator login: internal/web/server.go mutated; TestSystemButtons_NeedTheOperatorLogin ran=True failed=True
+    system_test.go:117: POST /os/ring/full-1 with auth "" = 303 — must be refused
+    system_test.go:117: POST /os/ring/full-1 with auth "Bearer k-full-1" = 303 — must be refused
+    system_test.go:117: POST /os/enabled/full-1 with auth "" = 303 — must be refused
+    system_test.go:117: POST /os/enabled/full-1 with auth "Bearer k-full-1" = 303 — must be refused
+    system_test.go:117: POST /os/approve-now with auth "" = 303 — must be refused
+    system_test.go:117: POST /os/approve-now with auth "Bearer k-full-1" = 303 — must be refused
+    system_test.go:117: POST /os/approve-docker with auth "" = 303 — must be refused
+    system_test.go:117: POST /os/approve-docker with auth "Bearer k-full-1" = 303 — must be refused
+    system_test.go:122: an unauthenticated POST changed a box
+[CAUGHT] Hosts column: internal/web/templates/hosts.html mutated; TestHostsPage_ProxmoxKernelColumn ran=True failed=True
+    system_test.go:153: hosts column missing:
+after restore: ok  	gitea.dooplex.hu/admin/felhom-hub/internal/sysfacts	0.003s | ok  	gitea.dooplex.hu/admin/felhom-hub/internal/osupdates	0.682s | ok  	gitea.dooplex.hu/admin/felhom-hub/internal/api	4.765s | ok  	gitea.dooplex.hu/admin/felhom-hub/internal/web	15.831s rc= 0

documentation/audits/os-docker-crash-2026-10-04/partA/hub_rp2.py

@@ -0,0 +1,54 @@
+import subprocess, shutil, sys
+M = [
+ ("unknown, never empty (parser)", "internal/sysfacts/sysfacts.go",
+  "out.Guest.Containerd, out.Guest.LiveRestore = orUnknown(out.Guest.Containerd), orUnknown(out.Guest.LiveRestore)",
+  "out.Guest.Containerd, out.Guest.LiveRestore = out.Guest.Containerd, out.Guest.LiveRestore",
+  "./internal/sysfacts/", "TestParse_AbsentAndPartialAreUnknown"),
+ ("Docker never auto-approved", "internal/osupdates/service.go",
+  "var Layers = []string{LayerGuest, LayerHost}", "var Layers = []string{LayerGuest, LayerHost, LayerDocker}",
+  "./internal/osupdates/", "TestDocker_NeverAutoApproved"),
+ ("Docker button needs 2 nights", "internal/osupdates/service.go",
+  "\tcase need == 0:\n\t\tneed = 2", "\tcase need == 0:\n\t\tneed = 1",
+  "./internal/osupdates/", "TestDocker_ApproveNeedsTwoHealthyNightsOnEveryRing0Box"),
+ ("a Docker approval nudges no box", "internal/osupdates/service.go",
+  "if s.Bump != nil && layer != LayerDocker {", "if s.Bump != nil {",
+  "./internal/osupdates/", "TestDocker_ApproveNeedsTwoHealthyNightsOnEveryRing0Box"),
+ ("crash event once per boot", "internal/api/crash.go",
+  "\t\tif known[b] {\n\t\t\tcontinue\n\t\t}", "\t\tif false {\n\t\t\tcontinue\n\t\t}",
+  "./internal/api/", "TestCheckCrash_EventsOncePerFact"),
+ ("first sighting is quiet", "internal/api/crash.go",
+  "\t\tif first {\n\t\t\treturn\n\t\t}\n\t\td := map", "\t\td := map",
+  "./internal/api/", "TestCheckCrash_EventsOncePerFact"),
+ ("page: unknown cell", "internal/web/system.go",
+  'if s == "" || s == sysfacts.Unknown {', 'if false {',
+  "./internal/web/", "TestSystemPage_FullPartialUnknown"),
+ ("page: Docker button gate", "internal/web/templates/system.html",
+  '(not .Approved) (eq .Waiting "")}}', '(not .Approved)}}',
+  "./internal/web/", "TestSystemPage_DockerButtonOnlyWhenReady"),
+ ("buttons need the operator login", "internal/web/server.go",
+  'if r.URL.Path == "/login" || isPublicBindPath(r.URL.Path) {', 'if r.URL.Path == "/login" || isPublicBindPath(r.URL.Path) || strings.HasPrefix(r.URL.Path, "/os/") {',
+  "./internal/web/", "TestSystemButtons_NeedTheOperatorLogin"),
+ ("Hosts column", "internal/web/templates/hosts.html",
+  "<th>Proxmox / kernel</th>", "<th>Proxmox</th>",
+  "./internal/web/", "TestHostsPage_ProxmoxKernelColumn"),
+]
+ok = True
+for name, f, old, new, pkg, test in M:
+    src = open(f).read()
+    assert src.count(old) == 1, (name, src.count(old))
+    shutil.copy(f, f + ".bak")
+    open(f, "w").write(src.replace(old, new))
+    r = subprocess.run(["go", "test", pkg, "-run", "^" + test + "$", "-v", "-count=1"], capture_output=True, text=True)
+    shutil.move(f + ".bak", f)
+    out = r.stdout + r.stderr
+    ran = ("=== RUN   " + test) in out
+    failed = ("--- FAIL: " + test) in out
+    v = "CAUGHT" if ran and failed else "NOT CAUGHT"
+    ok &= v == "CAUGHT"
+    print(f"[{v}] {name}: {f} mutated; {test} ran={ran} failed={failed}")
+    for l in out.splitlines():
+        if "_test.go:" in l or "build failed" in l or l.startswith("#"):
+            print("    " + l.strip()[:200])
+r = subprocess.run(["go", "test", "./internal/sysfacts/", "./internal/osupdates/", "./internal/api/", "./internal/web/", "-count=1"], capture_output=True, text=True)
+print("after restore:", r.stdout.strip().replace("\n", " | "), "rc=", r.returncode)
+sys.exit(0 if ok and r.returncode == 0 else 1)

documentation/audits/os-docker-crash-2026-10-04/partA/live/facts-demo-hp-rc1.txt

@@ -0,0 +1,39 @@
+  --- facts (vmid 9201, 2.1s) ---
+  {
+    "guest": {
+      "containerd": "2.3.3-1~debian.13~trixie",
+      "debian": "13.7",
+      "docker_engine": "29.7.2",
+      "live_restore": "on"
+    },
+    "host": {
+      "crash_guard": {
+        "armed": true,
+        "boot_id": "0defc390-c295-4e9f-aa6e-13d69a1f24c3",
+        "config": {
+          "LIMIT": 3,
+          "PANIC_SECONDS": 10,
+          "REARM_HOURS": 24,
+          "WINDOW_MINUTES": 60
+        },
+        "kernel_panic": 10,
+        "last_boot_at": "2026-10-04T13:39:06Z",
+        "last_boot_unclean": false,
+        "tripped": false,
+        "unclean_boots": [],
+        "unclean_boots_24h": 0,
+        "unclean_boots_in_window": 0,
+        "updated_at": "2026-10-04T14:08:23Z",
+        "version": 1
+      },
+      "debian": "13.7",
+      "held": [],
+      "kernel_next_boot": "7.0.14-20-pve",
+      "kernel_next_boot_source": "saved default",
+      "kernel_panic": 10,
+      "kernel_running": "7.0.14-20-pve",
+      "oops_this_boot": false,
+      "tainted": 4097,
+      "warn_this_boot": false
+    }
+  }

documentation/audits/os-docker-crash-2026-10-04/partA/live/hosts-page.html

@@ -0,0 +1,143 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>Hosts — Felhom Hub</title>
+    <link rel="stylesheet" href="/style.css?v=0.132.0">
+</head>
+<body>
+    <svg xmlns="http://www.w3.org/2000/svg" style="display:none" aria-hidden="true">
+<symbol id="i-triangle-alert" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="m21.73 18-8-14a2 2 0 0 0-3.48 0l-8 14A2 2 0 0 0 4 21h16a2 2 0 0 0 1.73-3" /> <path d="M12 9v4" /> <path d="M12 17h.01" /></symbol>
+<symbol id="i-check" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M20 6 9 17l-5-5" /></symbol>
+<symbol id="i-server" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="2" rx="2" ry="2" /> <rect width="20" height="8" x="2" y="14" rx="2" ry="2" /> <line x1="6" x2="6.01" y1="6" y2="6" /> <line x1="6" x2="6.01" y1="18" y2="18" /></symbol>
+<symbol id="i-settings" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M9.671 4.136a2.34 2.34 0 0 1 4.659 0 2.34 2.34 0 0 0 3.319 1.915 2.34 2.34 0 0 1 2.33 4.033 2.34 2.34 0 0 0 0 3.831 2.34 2.34 0 0 1-2.33 4.033 2.34 2.34 0 0 0-3.319 1.915 2.34 2.34 0 0 1-4.659 0 2.34 2.34 0 0 0-3.32-1.915 2.34 2.34 0 0 1-2.33-4.033 2.34 2.34 0 0 0 0-3.831A2.34 2.34 0 0 1 6.35 6.051a2.34 2.34 0 0 0 3.319-1.915" /> <circle cx="12" cy="12" r="3" /></symbol>
+<symbol id="i-x" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M18 6 6 18" /> <path d="m6 6 12 12" /></symbol>
+<symbol id="i-info" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10" /> <path d="M12 16v-4" /> <path d="M12 8h.01" /></symbol>
+<symbol id="i-hard-drive" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M10 16h.01" /> <path d="M2.212 11.577a2 2 0 0 0-.212.896V18a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2v-5.527a2 2 0 0 0-.212-.896L18.55 5.11A2 2 0 0 0 16.76 4H7.24a2 2 0 0 0-1.79 1.11z" /> <path d="M21.946 12.013H2.054" /> <path d="M6 16h.01" /></symbol>
+<symbol id="i-cpu" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 20v2" /> <path d="M12 2v2" /> <path d="M17 20v2" /> <path d="M17 2v2" /> <path d="M2 12h2" /> <path d="M2 17h2" /> <path d="M2 7h2" /> <path d="M20 12h2" /> <path d="M20 17h2" /> <path d="M20 7h2" /> <path d="M7 20v2" /> <path d="M7 2v2" /> <rect x="4" y="4" width="16" height="16" rx="2" /> <rect x="8" y="8" width="8" height="8" rx="1" /></symbol>
+<symbol id="i-clock" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10" /> <path d="M12 6v6l4 2" /></symbol>
+<symbol id="i-boxes" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M2.97 12.92A2 2 0 0 0 2 14.63v3.24a2 2 0 0 0 .97 1.71l3 1.8a2 2 0 0 0 2.06 0L12 19v-5.5l-5-3-4.03 2.42Z" /> <path d="m7 16.5-4.74-2.85" /> <path d="m7 16.5 5-3" /> <path d="M7 16.5v5.17" /> <path d="M12 13.5V19l3.97 2.38a2 2 0 0 0 2.06 0l3-1.8a2 2 0 0 0 .97-1.71v-3.24a2 2 0 0 0-.97-1.71L17 10.5l-5 3Z" /> <path d="m17 16.5-5-3" /> <path d="m17 16.5 4.74-2.85" /> <path d="M17 16.5v5.17" /> <path d="M7.97 4.42A2 2 0 0 0 7 6.13v4.37l5 3 5-3V6.13a2 2 0 0 0-.97-1.71l-3-1.8a2 2 0 0 0-2.06 0l-3 1.8Z" /> <path d="M12 8 7.26 5.15" /> <path d="m12 8 4.74-2.85" /> <path d="M12 13.5V8" /></symbol>
+<symbol id="i-users" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M16 21v-2a4 4 0 0 0-4-4H6a4 4 0 0 0-4 4v2" /> <path d="M16 3.128a4 4 0 0 1 0 7.744" /> <path d="M22 21v-2a4 4 0 0 0-3-3.87" /> <circle cx="9" cy="7" r="4" /></symbol>
+</svg>
+    
+<script>
+
+
+function felhomConfirm(el,question,onYes){
+    if(!el||el.dataset.fcOpen)return;
+    el.dataset.fcOpen='1';
+    var wrap=document.createElement('span');wrap.className='inline-confirm';
+    var q=document.createElement('span');q.className='inline-confirm-q';q.textContent=question;
+    var yes=document.createElement('button');yes.type='button';yes.className='btn btn-sm btn-danger';yes.textContent='Igen';
+    var no=document.createElement('button');no.type='button';no.className='btn btn-sm btn-outline';no.textContent='Mégse';
+    wrap.appendChild(q);wrap.appendChild(yes);wrap.appendChild(no);
+    el.style.display='none';el.parentNode.insertBefore(wrap,el.nextSibling);
+    function close(){wrap.remove();el.style.display='';delete el.dataset.fcOpen;}
+    no.addEventListener('click',close);
+    yes.addEventListener('click',function(){close();onYes();});
+}
+document.addEventListener('click',function(e){
+    var btn=e.target.closest?e.target.closest('[data-confirm]'):null;
+    if(!btn)return;
+    e.preventDefault();
+    felhomConfirm(btn,btn.getAttribute('data-confirm'),function(){
+        var form=btn.closest('form');
+        if(form){if(form.requestSubmit)form.requestSubmit(btn);else form.submit();}
+    });
+});
+</script>
+
+    <div class="container">
+        <header>
+            <h1>Felhom <span>Hub</span></h1>
+            <nav class="nav-links">
+                <a href="/" class="nav-link">Dashboard</a>
+                <a href="/configs" class="nav-link">Customers</a>
+                <a href="/apps" class="nav-link">Apps</a>
+                <a href="/hosts" class="nav-link active">Hosts</a>
+                <a href="/system" class="nav-link">System</a>
+                <a href="/offsite" class="nav-link">Offsite</a>
+                <a href="/configuration" class="nav-link">Configuration</a>
+            </nav>
+        </header>
+
+        <h2 style="margin-bottom: 1rem;">Hosts</h2>
+
+        
+
+        
+
+        
+        <section class="card" style="padding: 0; overflow: hidden;">
+            <table class="data-table">
+                <thead>
+                    <tr>
+                        <th>Host</th>
+                        <th>Customer</th>
+                        <th>Agent</th>
+                        <th>Proxmox / kernel</th>
+                        <th>Status</th>
+                        <th>Guests</th>
+                        <th>CPU</th>
+                        <th>Mem</th>
+                        <th>Disk</th>
+                        <th>Cloudflared</th>
+                        <th>Worst Storage</th>
+                    </tr>
+                </thead>
+                <tbody>
+                    
+                    <tr onclick="window.location='/hosts/demo-felhom-8363b5'" style="cursor: pointer;">
+                        <td><a href="/hosts/demo-felhom-8363b5">demo-felhom-8363b5</a></td>
+                        <td>Demo Ügyfél</td>
+                        <td><code>0.142.0</code></td>
+                        <td style="font-size: 0.85em;">9.2.2<br><span class="text-muted">7.0.2-6-pve</span></td>
+                        <td><span class="status-badge status-badge-ok">ONLINE</span></td>
+                        <td>1/2</td>
+                        <td>0%</td>
+                        <td>17%</td>
+                        <td>29%</td>
+                        <td>running (connected)</td>
+                        <td>29% <span class="text-muted">local</span></td>
+                    </tr>
+                    
+                    <tr onclick="window.location='/hosts/demo-hp-bb76ea'" style="cursor: pointer;">
+                        <td><a href="/hosts/demo-hp-bb76ea">demo-hp-bb76ea</a></td>
+                        <td>Demo HP</td>
+                        <td><code>0.142.0</code></td>
+                        <td style="font-size: 0.85em;">9.2.2<br><span class="text-muted">7.0.14-20-pve</span></td>
+                        <td><span class="status-badge status-badge-ok">ONLINE</span></td>
+                        <td>1/2</td>
+                        <td>16%</td>
+                        <td>19%</td>
+                        <td>51%</td>
+                        <td>running (connected)</td>
+                        <td>63% <span class="text-muted">local-lvm</span></td>
+                    </tr>
+                    
+                    <tr onclick="window.location='/hosts/drill-r50-0a4f9a'" style="cursor: pointer;">
+                        <td><a href="/hosts/drill-r50-0a4f9a">drill-r50-0a4f9a</a></td>
+                        <td>drill-r50</td>
+                        <td><code>0.129.0</code> <span class="status-badge status-warn" title="held: agent 0.129.0 &lt; MinAgent 0.131.0">floor held</span></td>
+                        <td style="font-size: 0.85em;">unknown<br><span class="text-muted">unknown</span></td>
+                        <td><span class="status-badge status-badge-down">DOWN</span></td>
+                        <td>1/1</td>
+                        <td>20%</td>
+                        <td>20%</td>
+                        <td>10%</td>
+                        <td>inactive</td>
+                        <td>10% <span class="text-muted">local</span></td>
+                    </tr>
+                    
+                </tbody>
+            </table>
+        </section>
+        
+
+        <footer style="margin-top: 2rem; color: var(--text-muted); font-size: 0.8rem; text-align: center;">
+            Felhom Hub <span style="font-family: var(--font-mono)">0.132.0</span>
+        </footer>
+    </div>
+</body>
+</html>

documentation/audits/os-docker-crash-2026-10-04/partA/live/system-page.html

@@ -0,0 +1,235 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>System — Felhom Hub</title>
+    <link rel="stylesheet" href="/style.css?v=0.132.0">
+    <style>
+        .sys td, .sys th { white-space: nowrap; font-size: 0.82em; vertical-align: top; }
+        .sys .grp { border-left: 2px solid var(--border, #444); }
+        .c-warn { color: var(--warn); font-weight: 600; }
+        .c-bad { color: var(--danger, #e5534b); font-weight: 700; }
+        .sys form { display: inline; }
+        .rel-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(16rem, 1fr)); gap: 0.75rem; }
+    </style>
+</head>
+<body>
+    <svg xmlns="http://www.w3.org/2000/svg" style="display:none" aria-hidden="true">
+<symbol id="i-triangle-alert" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="m21.73 18-8-14a2 2 0 0 0-3.48 0l-8 14A2 2 0 0 0 4 21h16a2 2 0 0 0 1.73-3" /> <path d="M12 9v4" /> <path d="M12 17h.01" /></symbol>
+<symbol id="i-check" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M20 6 9 17l-5-5" /></symbol>
+<symbol id="i-server" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="2" rx="2" ry="2" /> <rect width="20" height="8" x="2" y="14" rx="2" ry="2" /> <line x1="6" x2="6.01" y1="6" y2="6" /> <line x1="6" x2="6.01" y1="18" y2="18" /></symbol>
+<symbol id="i-settings" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M9.671 4.136a2.34 2.34 0 0 1 4.659 0 2.34 2.34 0 0 0 3.319 1.915 2.34 2.34 0 0 1 2.33 4.033 2.34 2.34 0 0 0 0 3.831 2.34 2.34 0 0 1-2.33 4.033 2.34 2.34 0 0 0-3.319 1.915 2.34 2.34 0 0 1-4.659 0 2.34 2.34 0 0 0-3.32-1.915 2.34 2.34 0 0 1-2.33-4.033 2.34 2.34 0 0 0 0-3.831A2.34 2.34 0 0 1 6.35 6.051a2.34 2.34 0 0 0 3.319-1.915" /> <circle cx="12" cy="12" r="3" /></symbol>
+<symbol id="i-x" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M18 6 6 18" /> <path d="m6 6 12 12" /></symbol>
+<symbol id="i-info" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10" /> <path d="M12 16v-4" /> <path d="M12 8h.01" /></symbol>
+<symbol id="i-hard-drive" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M10 16h.01" /> <path d="M2.212 11.577a2 2 0 0 0-.212.896V18a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2v-5.527a2 2 0 0 0-.212-.896L18.55 5.11A2 2 0 0 0 16.76 4H7.24a2 2 0 0 0-1.79 1.11z" /> <path d="M21.946 12.013H2.054" /> <path d="M6 16h.01" /></symbol>
+<symbol id="i-cpu" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 20v2" /> <path d="M12 2v2" /> <path d="M17 20v2" /> <path d="M17 2v2" /> <path d="M2 12h2" /> <path d="M2 17h2" /> <path d="M2 7h2" /> <path d="M20 12h2" /> <path d="M20 17h2" /> <path d="M20 7h2" /> <path d="M7 20v2" /> <path d="M7 2v2" /> <rect x="4" y="4" width="16" height="16" rx="2" /> <rect x="8" y="8" width="8" height="8" rx="1" /></symbol>
+<symbol id="i-clock" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10" /> <path d="M12 6v6l4 2" /></symbol>
+<symbol id="i-boxes" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M2.97 12.92A2 2 0 0 0 2 14.63v3.24a2 2 0 0 0 .97 1.71l3 1.8a2 2 0 0 0 2.06 0L12 19v-5.5l-5-3-4.03 2.42Z" /> <path d="m7 16.5-4.74-2.85" /> <path d="m7 16.5 5-3" /> <path d="M7 16.5v5.17" /> <path d="M12 13.5V19l3.97 2.38a2 2 0 0 0 2.06 0l3-1.8a2 2 0 0 0 .97-1.71v-3.24a2 2 0 0 0-.97-1.71L17 10.5l-5 3Z" /> <path d="m17 16.5-5-3" /> <path d="m17 16.5 4.74-2.85" /> <path d="M17 16.5v5.17" /> <path d="M7.97 4.42A2 2 0 0 0 7 6.13v4.37l5 3 5-3V6.13a2 2 0 0 0-.97-1.71l-3-1.8a2 2 0 0 0-2.06 0l-3 1.8Z" /> <path d="M12 8 7.26 5.15" /> <path d="m12 8 4.74-2.85" /> <path d="M12 13.5V8" /></symbol>
+<symbol id="i-users" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M16 21v-2a4 4 0 0 0-4-4H6a4 4 0 0 0-4 4v2" /> <path d="M16 3.128a4 4 0 0 1 0 7.744" /> <path d="M22 21v-2a4 4 0 0 0-3-3.87" /> <circle cx="9" cy="7" r="4" /></symbol>
+</svg>
+    
+<script>
+
+
+function felhomConfirm(el,question,onYes){
+    if(!el||el.dataset.fcOpen)return;
+    el.dataset.fcOpen='1';
+    var wrap=document.createElement('span');wrap.className='inline-confirm';
+    var q=document.createElement('span');q.className='inline-confirm-q';q.textContent=question;
+    var yes=document.createElement('button');yes.type='button';yes.className='btn btn-sm btn-danger';yes.textContent='Igen';
+    var no=document.createElement('button');no.type='button';no.className='btn btn-sm btn-outline';no.textContent='Mégse';
+    wrap.appendChild(q);wrap.appendChild(yes);wrap.appendChild(no);
+    el.style.display='none';el.parentNode.insertBefore(wrap,el.nextSibling);
+    function close(){wrap.remove();el.style.display='';delete el.dataset.fcOpen;}
+    no.addEventListener('click',close);
+    yes.addEventListener('click',function(){close();onYes();});
+}
+document.addEventListener('click',function(e){
+    var btn=e.target.closest?e.target.closest('[data-confirm]'):null;
+    if(!btn)return;
+    e.preventDefault();
+    felhomConfirm(btn,btn.getAttribute('data-confirm'),function(){
+        var form=btn.closest('form');
+        if(form){if(form.requestSubmit)form.requestSubmit(btn);else form.submit();}
+    });
+});
+</script>
+
+    <div class="container">
+        <header>
+            <h1>Felhom <span>Hub</span></h1>
+            <nav class="nav-links">
+                <a href="/" class="nav-link">Dashboard</a>
+                <a href="/configs" class="nav-link">Customers</a>
+                <a href="/apps" class="nav-link">Apps</a>
+                <a href="/hosts" class="nav-link">Hosts</a>
+                <a href="/system" class="nav-link active">System</a>
+                <a href="/offsite" class="nav-link">Offsite</a>
+                <a href="/configuration" class="nav-link">Configuration</a>
+            </nav>
+        </header>
+
+        <h2 style="margin-bottom: 1rem;">System — versions and OS updates</h2>
+
+        
+        
+
+        <section class="card" style="margin-bottom: 1.5rem;">
+            <h3 style="margin-top: 0;">Approved releases</h3>
+            <div class="rel-grid">
+                
+                <div><strong>guest</strong>: <code>os-guest-20261004-123933</code><br>
+                    <span class="text-muted">272 packages · 2026-10-04 12:39 UTC · by auto</span></div>
+                
+                <div><strong>host</strong>: <code>os-host-20261004-124133</code><br>
+                    <span class="text-muted">607 packages · 2026-10-04 12:41 UTC · by auto</span></div>
+                
+            </div>
+            <h3>What ring 0 runs now</h3>
+            <div class="rel-grid">
+                
+                <div><strong>guest</strong>:
+                    0 packages, first seen 2026-10-04 14:14 UTC<br>
+                    <span class="text-muted">healthy for 10m0s of 24h0m0s</span>
+                    
+                </div>
+                
+                <div><strong>host</strong>:
+                    607 packages, first seen 2026-10-04 12:24 UTC<br>
+                    <span class="text-muted">approved as os-host-20261004-124133</span>
+                    
+                    
+                </div>
+                
+                <div><strong>docker</strong>:
+                    <span class="text-muted">—</span><br>
+                    <span class="text-muted">a ring-0 box has not reported a Docker step</span>
+                    
+                </div>
+                
+            </div>
+            <form method="POST" action="/os/approve-now" style="margin-top: 0.8rem;">
+                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                <button type="submit" class="btn btn-sm btn-danger" data-confirm="Approve the guest and host sets ring 0 runs NOW, without the 24 h + 1 night wait? Every ring-1 box installs them at its next night run.">Approve now (guest + host)</button>
+                <span class="text-muted" style="font-size: 0.85em;">An urgent fix only — normally the hub approves after 24 h and one night.</span>
+            </form>
+        </section>
+
+        
+        <section class="card" style="padding: 0; overflow-x: auto;">
+            <table class="data-table sys">
+                <thead>
+                    <tr>
+                        <th>Box</th><th>Ring / updates</th><th>Tunnel</th>
+                        <th class="grp">Proxmox</th><th>Kernel (running)</th><th>Kernel (next boot)</th><th>Debian</th><th>Felhom release</th><th>Pending</th><th>Not covered</th><th>Held</th><th>Reboot needed</th><th>kernel.panic</th><th>Oops</th><th>Crash restarts 24 h</th><th>Crash guard</th>
+                        <th class="grp">Guest Debian</th><th>Felhom release</th><th>Pending</th><th>Restart needed</th>
+                        <th class="grp">Docker</th><th>containerd</th><th>live-restore</th><th>Docker release</th>
+                        <th class="grp">Last OS leg</th>
+                    </tr>
+                    <tr class="text-muted"><th></th><th></th><th></th><th class="grp" colspan="13">host</th><th class="grp" colspan="4">guest</th><th class="grp" colspan="4">Docker engine</th><th class="grp"></th></tr>
+                </thead>
+                <tbody>
+                    
+                    <tr>
+                        <td><a href="/hosts/demo-felhom-8363b5">demo-felhom-8363b5</a><br><span class="text-muted">Demo Ügyfél</span>
+                            </td>
+                        <td>
+                            ring 0
+                            <form method="POST" action="/os/ring/demo-felhom-8363b5">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="ring" value="1"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Make demo-felhom-8363b5 a normal (ring 1) box? It then installs only approved releases.">→ normal</button>
+                                
+                            </form><br>
+                            updates <strong>ON</strong>
+                            <form method="POST" action="/os/enabled/demo-felhom-8363b5">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="on" value="0"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Switch OS updates OFF for demo-felhom-8363b5? It keeps reporting and installs nothing.">switch off</button>
+                                
+                            </form>
+                        </td>
+                        <td>running</td>
+                        <td class="grp ">9.2.2</td>
+                        <td>7.0.2-6-pve</td><td>7.0.2-6-pve</td><td>13.7</td>
+                        <td>ring0-20261004T141237Z</td><td>80</td><td>0</td>
+                        <td>none</td><td class="c-warn">since 2026-10-04</td><td>10 s</td><td>no</td>
+                        <td>0</td><td>armed</td>
+                        <td class="grp ">13.7</td>
+                        <td>ring0-20261004T141237Z</td><td>0</td><td>10</td>
+                        <td class="grp ">29.8.2</td>
+                        <td>2.3.6-1~debian.13~trixie</td><td>on</td><td>—</td>
+                        <td class="grp " title="last successful leg: 10 min ago">10 min ago · applied · 34 s</td>
+                    </tr>
+                    
+                    <tr>
+                        <td><a href="/hosts/demo-hp-bb76ea">demo-hp-bb76ea</a><br><span class="text-muted">Demo HP</span>
+                            </td>
+                        <td>
+                            ring 0
+                            <form method="POST" action="/os/ring/demo-hp-bb76ea">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="ring" value="1"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Make demo-hp-bb76ea a normal (ring 1) box? It then installs only approved releases.">→ normal</button>
+                                
+                            </form><br>
+                            updates <strong>ON</strong>
+                            <form method="POST" action="/os/enabled/demo-hp-bb76ea">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="on" value="0"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Switch OS updates OFF for demo-hp-bb76ea? It keeps reporting and installs nothing.">switch off</button>
+                                
+                            </form>
+                        </td>
+                        <td>running</td>
+                        <td class="grp ">9.2.2</td>
+                        <td>7.0.14-20-pve</td><td>7.0.14-20-pve</td><td>13.7</td>
+                        <td>ring0-20261004T140947Z</td><td>78</td><td>0</td>
+                        <td>none</td><td>no</td><td>10 s</td><td>no</td>
+                        <td>0</td><td>armed</td>
+                        <td class="grp ">13.7</td>
+                        <td>ring0-20261004T140947Z</td><td>0</td><td>1</td>
+                        <td class="grp ">29.8.2</td>
+                        <td>2.3.6-1~debian.13~trixie</td><td>on</td><td>—</td>
+                        <td class="grp " title="last successful leg: 13 min ago">13 min ago · applied · 52 s</td>
+                    </tr>
+                    
+                    <tr>
+                        <td><a href="/hosts/drill-r50-0a4f9a">drill-r50-0a4f9a</a><br><span class="text-muted">drill-r50</span>
+                            <br><span class="c-warn" style="font-weight: normal;">no versions reported (agent older than v0.142.0)</span></td>
+                        <td>
+                            ring 1
+                            <form method="POST" action="/os/ring/drill-r50-0a4f9a">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="ring" value="0"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Make drill-r50-0a4f9a a DEMO (ring 0) box? It then installs every new fix first and takes unsigned Docker steps if its root-owned ring-0 mark allows.">→ demo</button>
+                            </form><br>
+                            updates <strong>ON</strong>
+                            <form method="POST" action="/os/enabled/drill-r50-0a4f9a">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="on" value="0"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Switch OS updates OFF for drill-r50-0a4f9a? It keeps reporting and installs nothing.">switch off</button>
+                                
+                            </form>
+                        </td>
+                        <td class="c-bad">inactive</td>
+                        <td class="grp c-warn">unknown</td>
+                        <td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td>
+                        <td>—</td><td>0</td><td>0</td>
+                        <td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td>no</td><td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td>no</td>
+                        <td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td class="c-warn">not installed</td>
+                        <td class="grp c-warn">unknown</td>
+                        <td>—</td><td>0</td><td>0</td>
+                        <td class="grp c-warn">unknown</td>
+                        <td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td>—</td>
+                        <td class="grp " title="last successful leg: never">never · — · 0 s</td>
+                    </tr>
+                    
+                </tbody>
+            </table>
+        </section>
+        <p class="text-muted" style="font-size: 0.85em;">Amber: worth a look. Red: an operator alarm fires (`08` §6.3). "unknown": the box could not read the value — never a guess.</p>
+        
+
+        <footer style="margin-top: 2rem; color: var(--text-muted); font-size: 0.8rem; text-align: center;">
+            Felhom Hub <span style="font-family: var(--font-mono)">0.132.0</span>
+        </footer>
+    </div>
+</body>
+</html>
+

documentation/audits/os-docker-crash-2026-10-04/partB/agent-redproofs.txt

@@ -0,0 +1,106 @@
+# Agent v0.142.0 red-proofs, 2026-10-04T13:54:18Z
+[CAUGHT] docker pkg in a fast plan refused (R2): configs/felhom-os-apply mutated; DockerLane.test_docker_package_in_a_fast_plan_is_refused ran=True rc=1
+    test_docker_package_in_a_fast_plan_is_refused (__main__.DockerLane.test_docker_package_in_a_fast_plan_is_refused) ... FAIL
+    FAIL: test_docker_package_in_a_fast_plan_is_refused (__main__.DockerLane.test_docker_package_in_a_fast_plan_is_refused)
+    self.assertEqual(rc, 2, rep)
+    AssertionError: 0 != 2 : {'docker_restart_needed': False, 'health_after': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'running'}, 'felhom-controller': {'health': 'healthy', '
+    FAILED (failures=1)
+[CAUGHT] slow lane without authority refused (R3): configs/felhom-os-apply mutated; DockerLane.test_no_authority_is_refused ran=True rc=1
+    test_no_authority_is_refused (__main__.DockerLane.test_no_authority_is_refused) ... FAIL
+    FAIL: test_no_authority_is_refused (__main__.DockerLane.test_no_authority_is_refused)
+    self.assertEqual(rc, 2, rep)
+    AssertionError: 0 != 2 : {'authority': 'ring0', 'docker_engine': '29.7.2', 'docker_restart_needed': False, 'health_after': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'runnin
+    FAILED (failures=1)
+[CAUGHT] bad signature refused (R3): configs/felhom-os-apply mutated; DockerLane.test_bad_signature_is_refused ran=True rc=1
+    test_bad_signature_is_refused (__main__.DockerLane.test_bad_signature_is_refused) ... FAIL
+    FAIL: test_bad_signature_is_refused (__main__.DockerLane.test_bad_signature_is_refused)
+    self.assertEqual(rc, 2, rep)
+    AssertionError: 0 != 2 : {'authority': 'signed', 'docker_engine': '29.7.2', 'docker_restart_needed': False, 'health_after': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'runni
+    FAILED (failures=1)
+[CAUGHT] replay refused (R3): configs/felhom-os-apply mutated; DockerLane.test_replayed_signed_job_is_refused ran=True rc=1
+    test_replayed_signed_job_is_refused (__main__.DockerLane.test_replayed_signed_job_is_refused) ... FAIL
+    FAIL: test_replayed_signed_job_is_refused (__main__.DockerLane.test_replayed_signed_job_is_refused)
+    self.assertEqual(rc, 2, rep)
+    AssertionError: 0 != 2 : {'authority': 'signed', 'docker_engine': '29.7.2', 'docker_restart_needed': False, 'health_after': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'runni
+    FAILED (failures=1)
+[CAUGHT] agent-writable trust file refused (R3): configs/felhom-os-apply mutated; DockerLane.test_agent_writable_trust_file_is_refused ran=True rc=1
+    test_agent_writable_trust_file_is_refused (__main__.DockerLane.test_agent_writable_trust_file_is_refused) ... FAIL
+    FAIL: test_agent_writable_trust_file_is_refused (__main__.DockerLane.test_agent_writable_trust_file_is_refused)
+    self.assertEqual(rc, 2, rep)
+    AssertionError: 0 != 2 : {'authority': 'ring0', 'docker_engine': '29.7.2', 'docker_restart_needed': False, 'health_after': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'runnin
+    FAILED (failures=1)
+[CAUGHT] live-restore off refused (R15): configs/felhom-os-apply mutated; DockerLane.test_live_restore_off_is_refused ran=True rc=1
+    test_live_restore_off_is_refused (__main__.DockerLane.test_live_restore_off_is_refused) ... FAIL
+    FAIL: test_live_restore_off_is_refused (__main__.DockerLane.test_live_restore_off_is_refused)
+    self.assertEqual(rc, 2, rep)
+    AssertionError: 0 != 2 : {'authority': 'signed', 'docker_engine': '29.7.2', 'docker_restart_needed': False, 'health_after': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'runni
+    FAILED (failures=1)
+[CAUGHT] live-restore uses reload, never restart: configs/felhom-os-apply mutated; LiveRestore.test_turns_it_on_with_a_reload_never_a_restart ran=True rc=1
+    test_turns_it_on_with_a_reload_never_a_restart (__main__.LiveRestore.test_turns_it_on_with_a_reload_never_a_restart) ... FAIL
+    FAIL: test_turns_it_on_with_a_reload_never_a_restart (__main__.LiveRestore.test_turns_it_on_with_a_reload_never_a_restart)
+    self.assertEqual(rc, 0, rep)
+    AssertionError: 3 != 0 : {'failed': {'rc': 3, 'reason': '', 'step': 'live-restore'}, 'layer': 'guest', 'live_restore': {'containers_before': 2, 'reload_rc': 0, 'result': 'failed', 'same_ids': True}, '
+    FAILED (failures=1)
+[CAUGHT] guest scanned every pass (R-849): configs/felhom-os-apply mutated; HostLayer.test_every_layer_scans_every_pass ran=True rc=1
+    test_every_layer_scans_every_pass (__main__.HostLayer.test_every_layer_scans_every_pass) ... FAIL
+    FAIL: test_every_layer_scans_every_pass (__main__.HostLayer.test_every_layer_scans_every_pass)
+    self.assertTrue(rep["reboot_scanned"] and rep["reboot_needed"], (layer, rep))
+    AssertionError: False is not true : ('guest', {'health_after': {'containers': {'app': {'health': 'healthy', 'id': 'bbb222', 'state': 'running'}, 'felhom-controller': {'health': 'healthy', 'id': 'aaa11
+    FAILED (failures=1)
+[CAUGHT] facts: next-boot by dpkg order: configs/felhom-os-apply mutated; Facts.test_next_entry_wins_and_default_zero_is_the_newest ran=True rc=1
+    test_next_entry_wins_and_default_zero_is_the_newest (__main__.Facts.test_next_entry_wins_and_default_zero_is_the_newest) ... FAIL
+    FAIL: test_next_entry_wins_and_default_zero_is_the_newest (__main__.Facts.test_next_entry_wins_and_default_zero_is_the_newest)
+    self.assertEqual(fa["host"]["kernel_next_boot"], "7.0.14-20-pve", "dpkg order, not string order (7.0.2 < 7.0.14)")
+    AssertionError: '7.0.2-6-pve' != '7.0.14-20-pve'
+    FAILED (failures=1)
+[CAUGHT] crash guard: 3rd crash leaves the box off: configs/felhom-crash-guard mutated; Guard.test_third_crash_in_an_hour_leaves_the_box_off ran=True rc=1
+    FAIL
+    FAIL: test_third_crash_in_an_hour_leaves_the_box_off (__main__.Guard.test_third_crash_in_an_hour_leaves_the_box_off)
+    self.assertTrue(s["tripped"], s)
+    AssertionError: False is not true : {'armed': True, 'boot_id': 'unknown', 'config': {'LIMIT': 3, 'PANIC_SECONDS': 10, 'REARM_HOURS': 24, 'WINDOW_MINUTES': 60}, 'kernel_panic': 10, 'last_boot_at': '202
+    FAILED (failures=1)
+[CAUGHT] crash guard: clean reboots never count: configs/felhom-crash-guard mutated; Guard.test_clean_reboots_never_count ran=True rc=1
+    FAIL
+    FAIL: test_clean_reboots_never_count (__main__.Guard.test_clean_reboots_never_count)
+    self.assertEqual(s["unclean_boots_in_window"], 0)
+    AssertionError: 5 != 0
+    FAILED (failures=1)
+[CAUGHT] crash guard: 24 h re-arm not earlier: configs/felhom-crash-guard mutated; Guard.test_rearms_after_24h_of_normal_running ran=True rc=1
+    FAIL
+    FAIL: test_rearms_after_24h_of_normal_running (__main__.Guard.test_rearms_after_24h_of_normal_running)
+    self.assertTrue(self.e.state()["tripped"], "not before 24 h")
+    AssertionError: False is not true : not before 24 h
+    FAILED (failures=1)
+[CAUGHT] docker health: changed id fails: internal/osupdate/leg.go mutated; TestDockerHealthVerdict ran=True failed=True
+    leg_test.go:442: a changed container id passed — live-restore failed and the apps restarted
+    --- FAIL: TestDockerHealthVerdict (0.00s)
+    FAIL
+    FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/osupdate	0.008s
+    FAIL
+[CAUGHT] docker health: changed id -> health_failed: internal/osupdate/leg.go mutated; TestDocker_ChangedIDIsHealthFailed ran=True failed=True
+    leg_test.go:465: docker = {RunID:20261004T040000Z Layer:docker Trigger:night Mode:apply Ring:0 ReleaseID:ring0-20261004T040000Z Outcome:applied Healthy:true HealthReason: VMID:9201 Upgraded:[{Name:doc
+    --- FAIL: TestDocker_ChangedIDIsHealthFailed (0.00s)
+    FAIL
+    FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/osupdate	0.011s
+    FAIL
+[CAUGHT] docker health: engine must move: internal/osupdate/leg.go mutated; TestDockerHealthVerdict ran=True failed=True
+    leg_test.go:445: the engine did not move and the step passed
+    --- FAIL: TestDockerHealthVerdict (0.00s)
+    FAIL
+    FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/osupdate	0.007s
+    FAIL
+[CAUGHT] ring 1 never steps in the night leg: internal/osupdate/leg.go mutated; TestDocker_Ring1NightLegNeverSteps ran=True failed=True
+    leg_test.go:390: ring 1 took a docker step: guest:inventory,host:inventory,guest:live-restore-on,docker:apply
+    --- FAIL: TestDocker_Ring1NightLegNeverSteps (0.00s)
+    FAIL
+    FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/osupdate	0.009s
+    FAIL
+after restore: ok  	gitea.dooplex.hu/admin/felhom-agent/internal/osupdate	0.628s rc=0
+
+[executor] signed envelope reaches the plan — mutation: drop plan["signed"]
+=== RUN   TestDockerStepExecutor_PassesTheSignedEnvelope
+    dockerjob_test.go:31: docker plan = map[lane:slow layer:docker mode:apply packages:[map[name:docker-ce origin:Docker CE version:5:29.8.2-1~debian.13~trixie]] release_id:os-docker-1 select:listed s
+--- FAIL: TestDockerStepExecutor_PassesTheSignedEnvelope (0.00s)
+FAIL
+FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/osupdate	0.009s
+FAIL

documentation/audits/os-docker-crash-2026-10-04/partB/approval/approve-docker-post.txt

@@ -0,0 +1,2 @@
+HTTP/1.1 303 See Other
+Location: /system?flash=approved+os-docker-20261004-142842

documentation/audits/os-docker-crash-2026-10-04/partB/approval/system-before-approve.html

@@ -0,0 +1,240 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>System — Felhom Hub</title>
+    <link rel="stylesheet" href="/style.css?v=0.132.0">
+    <style>
+        .sys td, .sys th { white-space: nowrap; font-size: 0.82em; vertical-align: top; }
+        .sys .grp { border-left: 2px solid var(--border, #444); }
+        .c-warn { color: var(--warn); font-weight: 600; }
+        .c-bad { color: var(--danger, #e5534b); font-weight: 700; }
+        .sys form { display: inline; }
+        .rel-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(16rem, 1fr)); gap: 0.75rem; }
+    </style>
+</head>
+<body>
+    <svg xmlns="http://www.w3.org/2000/svg" style="display:none" aria-hidden="true">
+<symbol id="i-triangle-alert" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="m21.73 18-8-14a2 2 0 0 0-3.48 0l-8 14A2 2 0 0 0 4 21h16a2 2 0 0 0 1.73-3" /> <path d="M12 9v4" /> <path d="M12 17h.01" /></symbol>
+<symbol id="i-check" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M20 6 9 17l-5-5" /></symbol>
+<symbol id="i-server" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="2" rx="2" ry="2" /> <rect width="20" height="8" x="2" y="14" rx="2" ry="2" /> <line x1="6" x2="6.01" y1="6" y2="6" /> <line x1="6" x2="6.01" y1="18" y2="18" /></symbol>
+<symbol id="i-settings" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M9.671 4.136a2.34 2.34 0 0 1 4.659 0 2.34 2.34 0 0 0 3.319 1.915 2.34 2.34 0 0 1 2.33 4.033 2.34 2.34 0 0 0 0 3.831 2.34 2.34 0 0 1-2.33 4.033 2.34 2.34 0 0 0-3.319 1.915 2.34 2.34 0 0 1-4.659 0 2.34 2.34 0 0 0-3.32-1.915 2.34 2.34 0 0 1-2.33-4.033 2.34 2.34 0 0 0 0-3.831A2.34 2.34 0 0 1 6.35 6.051a2.34 2.34 0 0 0 3.319-1.915" /> <circle cx="12" cy="12" r="3" /></symbol>
+<symbol id="i-x" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M18 6 6 18" /> <path d="m6 6 12 12" /></symbol>
+<symbol id="i-info" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10" /> <path d="M12 16v-4" /> <path d="M12 8h.01" /></symbol>
+<symbol id="i-hard-drive" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M10 16h.01" /> <path d="M2.212 11.577a2 2 0 0 0-.212.896V18a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2v-5.527a2 2 0 0 0-.212-.896L18.55 5.11A2 2 0 0 0 16.76 4H7.24a2 2 0 0 0-1.79 1.11z" /> <path d="M21.946 12.013H2.054" /> <path d="M6 16h.01" /></symbol>
+<symbol id="i-cpu" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 20v2" /> <path d="M12 2v2" /> <path d="M17 20v2" /> <path d="M17 2v2" /> <path d="M2 12h2" /> <path d="M2 17h2" /> <path d="M2 7h2" /> <path d="M20 12h2" /> <path d="M20 17h2" /> <path d="M20 7h2" /> <path d="M7 20v2" /> <path d="M7 2v2" /> <rect x="4" y="4" width="16" height="16" rx="2" /> <rect x="8" y="8" width="8" height="8" rx="1" /></symbol>
+<symbol id="i-clock" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10" /> <path d="M12 6v6l4 2" /></symbol>
+<symbol id="i-boxes" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M2.97 12.92A2 2 0 0 0 2 14.63v3.24a2 2 0 0 0 .97 1.71l3 1.8a2 2 0 0 0 2.06 0L12 19v-5.5l-5-3-4.03 2.42Z" /> <path d="m7 16.5-4.74-2.85" /> <path d="m7 16.5 5-3" /> <path d="M7 16.5v5.17" /> <path d="M12 13.5V19l3.97 2.38a2 2 0 0 0 2.06 0l3-1.8a2 2 0 0 0 .97-1.71v-3.24a2 2 0 0 0-.97-1.71L17 10.5l-5 3Z" /> <path d="m17 16.5-5-3" /> <path d="m17 16.5 4.74-2.85" /> <path d="M17 16.5v5.17" /> <path d="M7.97 4.42A2 2 0 0 0 7 6.13v4.37l5 3 5-3V6.13a2 2 0 0 0-.97-1.71l-3-1.8a2 2 0 0 0-2.06 0l-3 1.8Z" /> <path d="M12 8 7.26 5.15" /> <path d="m12 8 4.74-2.85" /> <path d="M12 13.5V8" /></symbol>
+<symbol id="i-users" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M16 21v-2a4 4 0 0 0-4-4H6a4 4 0 0 0-4 4v2" /> <path d="M16 3.128a4 4 0 0 1 0 7.744" /> <path d="M22 21v-2a4 4 0 0 0-3-3.87" /> <circle cx="9" cy="7" r="4" /></symbol>
+</svg>
+    
+<script>
+
+
+function felhomConfirm(el,question,onYes){
+    if(!el||el.dataset.fcOpen)return;
+    el.dataset.fcOpen='1';
+    var wrap=document.createElement('span');wrap.className='inline-confirm';
+    var q=document.createElement('span');q.className='inline-confirm-q';q.textContent=question;
+    var yes=document.createElement('button');yes.type='button';yes.className='btn btn-sm btn-danger';yes.textContent='Igen';
+    var no=document.createElement('button');no.type='button';no.className='btn btn-sm btn-outline';no.textContent='Mégse';
+    wrap.appendChild(q);wrap.appendChild(yes);wrap.appendChild(no);
+    el.style.display='none';el.parentNode.insertBefore(wrap,el.nextSibling);
+    function close(){wrap.remove();el.style.display='';delete el.dataset.fcOpen;}
+    no.addEventListener('click',close);
+    yes.addEventListener('click',function(){close();onYes();});
+}
+document.addEventListener('click',function(e){
+    var btn=e.target.closest?e.target.closest('[data-confirm]'):null;
+    if(!btn)return;
+    e.preventDefault();
+    felhomConfirm(btn,btn.getAttribute('data-confirm'),function(){
+        var form=btn.closest('form');
+        if(form){if(form.requestSubmit)form.requestSubmit(btn);else form.submit();}
+    });
+});
+</script>
+
+    <div class="container">
+        <header>
+            <h1>Felhom <span>Hub</span></h1>
+            <nav class="nav-links">
+                <a href="/" class="nav-link">Dashboard</a>
+                <a href="/configs" class="nav-link">Customers</a>
+                <a href="/apps" class="nav-link">Apps</a>
+                <a href="/hosts" class="nav-link">Hosts</a>
+                <a href="/system" class="nav-link active">System</a>
+                <a href="/offsite" class="nav-link">Offsite</a>
+                <a href="/configuration" class="nav-link">Configuration</a>
+            </nav>
+        </header>
+
+        <h2 style="margin-bottom: 1rem;">System — versions and OS updates</h2>
+
+        
+        
+
+        <section class="card" style="margin-bottom: 1.5rem;">
+            <h3 style="margin-top: 0;">Approved releases</h3>
+            <div class="rel-grid">
+                
+                <div><strong>guest</strong>: <code>os-guest-20261004-123933</code><br>
+                    <span class="text-muted">272 packages · 2026-10-04 12:39 UTC · by auto</span></div>
+                
+                <div><strong>host</strong>: <code>os-host-20261004-124133</code><br>
+                    <span class="text-muted">607 packages · 2026-10-04 12:41 UTC · by auto</span></div>
+                
+            </div>
+            <h3>What ring 0 runs now</h3>
+            <div class="rel-grid">
+                
+                <div><strong>guest</strong>:
+                    272 packages, first seen 2026-10-04 11:07 UTC<br>
+                    <span class="text-muted">approved as os-guest-20261004-123933</span>
+                    
+                    
+                </div>
+                
+                <div><strong>host</strong>:
+                    607 packages, first seen 2026-10-04 12:24 UTC<br>
+                    <span class="text-muted">approved as os-host-20261004-124133</span>
+                    
+                    
+                </div>
+                
+                <div><strong>docker</strong>:
+                    6 packages, first seen 2026-10-04 14:28 UTC<br>
+                    
+                    
+                    <form method="POST" action="/os/approve-docker" style="margin-top: 0.3rem;">
+                        <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                        <button type="submit" class="btn btn-sm" data-confirm="Approve this Docker engine set? Ring-1 boxes take it only through a signed operator job.">Approve Docker set</button>
+                    </form>
+                </div>
+                
+            </div>
+            <form method="POST" action="/os/approve-now" style="margin-top: 0.8rem;">
+                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                <button type="submit" class="btn btn-sm btn-danger" data-confirm="Approve the guest and host sets ring 0 runs NOW, without the 24 h + 1 night wait? Every ring-1 box installs them at its next night run.">Approve now (guest + host)</button>
+                <span class="text-muted" style="font-size: 0.85em;">An urgent fix only — normally the hub approves after 24 h and one night.</span>
+            </form>
+        </section>
+
+        
+        <section class="card" style="padding: 0; overflow-x: auto;">
+            <table class="data-table sys">
+                <thead>
+                    <tr>
+                        <th>Box</th><th>Ring / updates</th><th>Tunnel</th>
+                        <th class="grp">Proxmox</th><th>Kernel (running)</th><th>Kernel (next boot)</th><th>Debian</th><th>Felhom release</th><th>Pending</th><th>Not covered</th><th>Held</th><th>Reboot needed</th><th>kernel.panic</th><th>Oops</th><th>Crash restarts 24 h</th><th>Crash guard</th>
+                        <th class="grp">Guest Debian</th><th>Felhom release</th><th>Pending</th><th>Restart needed</th>
+                        <th class="grp">Docker</th><th>containerd</th><th>live-restore</th><th>Docker release</th>
+                        <th class="grp">Last OS leg</th>
+                    </tr>
+                    <tr class="text-muted"><th></th><th></th><th></th><th class="grp" colspan="13">host</th><th class="grp" colspan="4">guest</th><th class="grp" colspan="4">Docker engine</th><th class="grp"></th></tr>
+                </thead>
+                <tbody>
+                    
+                    <tr>
+                        <td><a href="/hosts/demo-felhom-8363b5">demo-felhom-8363b5</a><br><span class="text-muted">Demo Ügyfél</span>
+                            </td>
+                        <td>
+                            ring 0
+                            <form method="POST" action="/os/ring/demo-felhom-8363b5">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="ring" value="1"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Make demo-felhom-8363b5 a normal (ring 1) box? It then installs only approved releases.">→ normal</button>
+                                
+                            </form><br>
+                            updates <strong>ON</strong>
+                            <form method="POST" action="/os/enabled/demo-felhom-8363b5">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="on" value="0"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Switch OS updates OFF for demo-felhom-8363b5? It keeps reporting and installs nothing.">switch off</button>
+                                
+                            </form>
+                        </td>
+                        <td>running</td>
+                        <td class="grp ">9.2.2</td>
+                        <td>7.0.2-6-pve</td><td>7.0.2-6-pve</td><td>13.7</td>
+                        <td>ring0-20261004T142539Z</td><td>80</td><td>0</td>
+                        <td>none</td><td class="c-warn">since 2026-10-04</td><td>10 s</td><td>no</td>
+                        <td>0</td><td>armed</td>
+                        <td class="grp ">13.7</td>
+                        <td>ring0-20261004T142539Z</td><td>0</td><td>10</td>
+                        <td class="grp ">29.8.2</td>
+                        <td>2.3.6-1~debian.13~trixie</td><td>on</td><td>ring0-20261004T142539Z</td>
+                        <td class="grp " title="last successful leg: 2 min ago">2 min ago · nothing · 13 s</td>
+                    </tr>
+                    
+                    <tr>
+                        <td><a href="/hosts/demo-hp-bb76ea">demo-hp-bb76ea</a><br><span class="text-muted">Demo HP</span>
+                            </td>
+                        <td>
+                            ring 0
+                            <form method="POST" action="/os/ring/demo-hp-bb76ea">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="ring" value="1"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Make demo-hp-bb76ea a normal (ring 1) box? It then installs only approved releases.">→ normal</button>
+                                
+                            </form><br>
+                            updates <strong>ON</strong>
+                            <form method="POST" action="/os/enabled/demo-hp-bb76ea">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="on" value="0"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Switch OS updates OFF for demo-hp-bb76ea? It keeps reporting and installs nothing.">switch off</button>
+                                
+                            </form>
+                        </td>
+                        <td>running</td>
+                        <td class="grp ">9.2.2</td>
+                        <td>7.0.14-20-pve</td><td>7.0.14-20-pve</td><td>13.7</td>
+                        <td>ring0-20261004T142618Z</td><td>78</td><td>0</td>
+                        <td>none</td><td>no</td><td>10 s</td><td>no</td>
+                        <td>0</td><td>armed</td>
+                        <td class="grp ">13.7</td>
+                        <td>ring0-20261004T142618Z</td><td>0</td><td>1</td>
+                        <td class="grp ">29.8.2</td>
+                        <td>2.3.6-1~debian.13~trixie</td><td>on</td><td>ring0-20261004T142618Z</td>
+                        <td class="grp " title="last successful leg: 2 min ago">1 min ago · nothing · 17 s</td>
+                    </tr>
+                    
+                    <tr>
+                        <td><a href="/hosts/drill-r50-0a4f9a">drill-r50-0a4f9a</a><br><span class="text-muted">drill-r50</span>
+                            <br><span class="c-warn" style="font-weight: normal;">no versions reported (agent older than v0.142.0)</span></td>
+                        <td>
+                            ring 1
+                            <form method="POST" action="/os/ring/drill-r50-0a4f9a">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="ring" value="0"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Make drill-r50-0a4f9a a DEMO (ring 0) box? It then installs every new fix first and takes unsigned Docker steps if its root-owned ring-0 mark allows.">→ demo</button>
+                            </form><br>
+                            updates <strong>ON</strong>
+                            <form method="POST" action="/os/enabled/drill-r50-0a4f9a">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="on" value="0"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Switch OS updates OFF for drill-r50-0a4f9a? It keeps reporting and installs nothing.">switch off</button>
+                                
+                            </form>
+                        </td>
+                        <td class="c-bad">inactive</td>
+                        <td class="grp c-warn">unknown</td>
+                        <td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td>
+                        <td>—</td><td>0</td><td>0</td>
+                        <td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td>no</td><td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td>no</td>
+                        <td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td class="c-warn">not installed</td>
+                        <td class="grp c-warn">unknown</td>
+                        <td>—</td><td>0</td><td>0</td>
+                        <td class="grp c-warn">unknown</td>
+                        <td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td>—</td>
+                        <td class="grp " title="last successful leg: never">never · — · 0 s</td>
+                    </tr>
+                    
+                </tbody>
+            </table>
+        </section>
+        <p class="text-muted" style="font-size: 0.85em;">Amber: worth a look. Red: an operator alarm fires (`08` §6.3). "unknown": the box could not read the value — never a guess.</p>
+        
+
+        <footer style="margin-top: 2rem; color: var(--text-muted); font-size: 0.8rem; text-align: center;">
+            Felhom Hub <span style="font-family: var(--font-mono)">0.132.0</span>
+        </footer>
+    </div>
+</body>
+</html>
+

documentation/audits/os-docker-crash-2026-10-04/partB/b1-live-restore-9202.txt

@@ -0,0 +1,80 @@
++ pct exec 9202 -- docker ps --no-trunc --format '{{.ID}} {{.Names}} {{.Status}}'
++ sort
++ cat /root/9202-ids-before.txt
++ cut -c1-12,65-
+1b8dc9032f36 paperless-webserver Up 27 minutes (healthy)
+2d6019eb0042 paperless-redis Up 27 minutes (healthy)
+3f8f9651ee32 felhom-controller Up 27 minutes (healthy)
+468aa784345b paperless-postgres Up 27 minutes (healthy)
+7504fa621b39 filebrowser Up 27 minutes (healthy)
+8d8778d4dcf8 traefik Up 27 minutes
++ pct exec 9202 -- docker info --format 'live={{.LiveRestoreEnabled}}'
+live=false
++ pct exec 9202 -- cat /etc/docker/daemon.json
+{
+  "features": { "containerd-snapshotter": false },
+  "log-driver": "json-file",
+  "log-opts": { "max-size": "10m", "max-file": "3" }
+}
+++ date +%s.%N
++ S=1791122916.729550605
++ sudo -u felhom-agent /usr/local/bin/felhom-agent-0.142.0-rc1 --config /etc/felhom-agent/agent.json --selftest=live-restore -vmid 9202
++ grep -E 'LIVE-RESTORE|live-restore|error|refused'
+selftest=live-restore: refused: {"code": "R10", "reason": "vmid 9202 does not bind /mnt/felhom-drives \u2014 it is not this box's customer guest"}
+++ bc
++++ date +%s.%N
+++ echo 1791122916.827586880-1791122916.729550605
++ echo WALL=.098036275
+WALL=.098036275
++ pct exec 9202 -- docker ps --no-trunc --format '{{.ID}} {{.Names}} {{.Status}}'
++ sort
++ cut '-d ' -f1,2 /root/9202-ids-before.txt
++ cut '-d ' -f1,2 /root/9202-ids-after.txt
++ diff /tmp/a /tmp/b
+++ wc -l
+IDS IDENTICAL (6 containers)
++ echo 'IDS IDENTICAL (6 containers)'
++ pct exec 9202 -- docker info --format 'live={{.LiveRestoreEnabled}}'
+live=false
++ pct exec 9202 -- cat /etc/docker/daemon.json
+{
+  "features": { "containerd-snapshotter": false },
+  "log-driver": "json-file",
+  "log-opts": { "max-size": "10m", "max-file": "3" }
+}
++ pct exec 9202 -- systemctl show docker -p ActiveEnterTimestamp -p NRestarts
+NRestarts=0
+ActiveEnterTimestamp=Sun 2026-10-04 13:40:57 UTC
++ pct exec 9202 -- cp -p /etc/docker/daemon.json /root/daemon.json.bak-2026-10-04
++ pct exec 9202 -- python3 -c '
+import json; p="/etc/docker/daemon.json"; d=json.load(open(p)); d["live-restore"]=True; open(p,"w").write(json.dumps(d,indent=2,sort_keys=True)+"\n")'
+++ date +%s.%N
++ S=1791122938.796302354
++ pct exec 9202 -- systemctl reload docker
+++ bc
++++ date +%s.%N
+++ echo 1791122939.730342275-1791122938.796302354
+reload_rc=0 took=.934039921
++ echo 'reload_rc=0 took=.934039921'
++ sleep 3
++ pct exec 9202 -- docker info --format 'live={{.LiveRestoreEnabled}}'
+live=true
++ pct exec 9202 -- docker ps --no-trunc --format '{{.ID}} {{.Names}} {{.Status}}'
++ sort
++ cut '-d ' -f1,2 /root/9202-ids-before.txt
++ cut '-d ' -f1,2 /root/9202-ids-after.txt
++ diff /tmp/a /tmp/b
+++ wc -l
+IDS IDENTICAL (6 containers)
++ echo 'IDS IDENTICAL (6 containers)'
++ cut -c1-12,65- /root/9202-ids-after.txt
+1b8dc9032f36 paperless-webserver Up 28 minutes (healthy)
+2d6019eb0042 paperless-redis Up 28 minutes (healthy)
+3f8f9651ee32 felhom-controller Up 28 minutes (healthy)
+468aa784345b paperless-postgres Up 28 minutes (healthy)
+7504fa621b39 filebrowser Up 28 minutes (healthy)
+8d8778d4dcf8 traefik Up 28 minutes
++ pct exec 9202 -- systemctl show docker -p ActiveEnterTimestamp -p NRestarts -p ExecMainPID
+NRestarts=0
+ExecMainPID=227
+ActiveEnterTimestamp=Sun 2026-10-04 13:40:57 UTC

documentation/audits/os-docker-crash-2026-10-04/partB/b2-live-restore-demo-hp-9201.txt

@@ -0,0 +1,7 @@
+before: 24 containers, live=false
+os-apply: LIVE-RESTORE reload_rc=0 state=true containers=24 same-ids=yes
+live-restore: on (see the wrapper's LIVE-RESTORE line above for the container ids)
+WALL=7.532254689
+IDS IDENTICAL (24 containers)
+after: live=true
+NRestarts=0 ActiveEnterTimestamp=Sun 2026-10-04 13:41:02 UTC 

documentation/audits/os-docker-crash-2026-10-04/partB/b3-ring0-docker-demo-hp.log

@@ -0,0 +1,202 @@
+=== felhom-agent 0.142.0-rc1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true ===
+time=2026-10-04T16:09:47.878+02:00 level=INFO msg="osupdate: START" run=20261004T140947Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T140947Z
+time=2026-10-04T16:10:04.518+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T140947Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T16:10:04.518+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T16:10:04.518+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T16:10:04.518+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T16:10:04.519+02:00 level=INFO msg="osupdate: DONE" run=20261004T140947Z layer=guest vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=6 not_covered=6 restart_needed=0 reboot_needed=false wrapper_seconds=16.6
+time=2026-10-04T16:10:04.530+02:00 level=INFO msg="osupdate: START" run=20261004T140947Z layer=host vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T140947Z
+time=2026-10-04T16:10:19.645+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T140947Z layer=host lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T16:10:19.645+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T16:10:19.645+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T16:10:19.645+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T16:10:20.691+02:00 level=INFO msg="osupdate: DONE" run=20261004T140947Z layer=host vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=78 not_covered=78 restart_needed=0 reboot_needed=false wrapper_seconds=15
+time=2026-10-04T16:10:22.649+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on"
+time=2026-10-04T16:10:22.649+02:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"result\": \"already on\"}"
+time=2026-10-04T16:10:22.649+02:00 level=INFO msg="osupdate: START" run=20261004T140947Z layer=docker vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T140947Z
+time=2026-10-04T16:11:14.270+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T140947Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0"
+time=2026-10-04T16:11:14.270+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T16:11:14.270+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=6 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T16:11:14.271+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=24.4 upgraded=6 restart-needed=containerd-shim reboot-needed=no"
+time=2026-10-04T16:11:50.070+02:00 level=INFO msg="osupdate: DONE" run=20261004T140947Z layer=docker vmid=9201 ring=0 trigger=debug outcome=applied healthy=true reason="" upgraded=6 pending=0 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=51.5
+  --- os-update report (guest) ---
+  {
+    "authority": "",
+    "docker_engine": "",
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "docker-ce-cli",
+      "containerd.io",
+      "docker-ce",
+      "docker-buildx-plugin",
+      "docker-ce-rootless-extras",
+      "docker-compose-plugin"
+    ],
+    "outcome": "nothing",
+    "pending": 6,
+    "reboot_needed": false,
+    "refused": null,
+    "release_id": "ring0-20261004T140947Z",
+    "restart_needed": [],
+    "ring": 0,
+    "run_id": "20261004T140947Z",
+    "upgraded": [],
+    "wrapper_seconds": 16.6
+  }
+  --- os-update report (host) ---
+  {
+    "authority": "",
+    "docker_engine": "",
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "frr",
+      "shim-signed-common",
+      "proxmox-secure-boot-support",
+      "shim-unsigned",
+      "shim-helpers-amd64-signed",
+      "shim-signed",
+      "amd64-microcode",
+      "libradosstriper1",
+      "librgw2",
+      "ceph-common",
+      "librbd1",
+      "librados2",
+      "python3-cephfs",
+      "libcephfs2",
+      "python3-rgw",
+      "python3-rados",
+      "python3-ceph-argparse",
+      "python3-ceph-common",
+      "python3-rbd",
+      "ceph-fuse",
+      "chrony",
+      "libcorosync-common4",
+      "libcfg7",
+      "libcmap4",
+      "libcpg4",
+      "libknet1t64",
+      "libnozzle1t64",
+      "libquorum5",
+      "libvotequorum8",
+      "corosync",
+      "frr-pythontools",
+      "libjs-extjs",
+      "libnvpair3linux",
+      "libproxmox-acme-plugins",
+      "libproxmox-backup-qemu0",
+      "pve-qemu-kvm",
+      "libpve-notify-perl",
+      "libpve-cluster-api-perl",
+      "libpve-cluster-perl",
+      "pve-cluster",
+      "libpve-access-control",
+      "libpve-apiclient-perl",
+      "librados2-perl",
+      "proxmox-backup-client",
+      "proxmox-backup-file-restore",
+      "pve-manager",
+      "libproxmox-acme-perl",
+      "libpve-common-perl",
+      "libpve-guest-common-perl",
+      "qemu-server",
+      "libpve-storage-perl",
+      "pve-edk2-firmware-legacy",
+      "pve-edk2-firmware-ovmf",
+      "libpve-network-api-perl",
+      "libpve-network-perl",
+      "proxmox-firewall-data",
+      "pve-firewall",
+      "pve-container",
+      "pve-ha-manager",
+      "novnc-pve",
+      "proxmox-enterprise-support-keyring",
+      "proxmox-mini-journalreader",
+      "proxmox-widget-toolkit",
+      "pve-docs",
+      "pve-i18n",
+      "pve-xtermjs",
+      "pve-yew-mobile-i18n",
+      "pve-yew-mobile-gui",
+      "libuutil3linux",
+      "libzfs7linux",
+      "libzpool7linux",
+      "proxmox-kernel-helper",
+      "pve-edk2-firmware-aarch64",
+      "pve-edk2-firmware",
+      "pve-firmware",
+      "zfs-initramfs",
+      "zfsutils-linux",
+      "zfs-zed"
+    ],
+    "outcome": "nothing",
+    "pending": 78,
+    "reboot_needed": false,
+    "refused": null,
+    "release_id": "ring0-20261004T140947Z",
+    "restart_needed": [],
+    "ring": 0,
+    "run_id": "20261004T140947Z",
+    "upgraded": [],
+    "wrapper_seconds": 15
+  }
+  --- os-update report (docker) ---
+  {
+    "authority": "ring0",
+    "docker_engine": "29.8.2",
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": null,
+    "outcome": "applied",
+    "pending": 0,
+    "reboot_needed": false,
+    "refused": null,
+    "release_id": "ring0-20261004T140947Z",
+    "restart_needed": [
+      "containerd-shim"
+    ],
+    "ring": 0,
+    "run_id": "20261004T140947Z",
+    "upgraded": [
+      {
+        "name": "docker-ce-cli",
+        "version": "5:29.8.2-1~debian.13~trixie",
+        "origin": ""
+      },
+      {
+        "name": "containerd.io",
+        "version": "2.3.6-1~debian.13~trixie",
+        "origin": ""
+      },
+      {
+        "name": "docker-ce",
+        "version": "5:29.8.2-1~debian.13~trixie",
+        "origin": ""
+      },
+      {
+        "name": "docker-buildx-plugin",
+        "version": "0.37.1-1~debian.13~trixie",
+        "origin": ""
+      },
+      {
+        "name": "docker-ce-rootless-extras",
+        "version": "5:29.8.2-1~debian.13~trixie",
+        "origin": ""
+      },
+      {
+        "name": "docker-compose-plugin",
+        "version": "5.6.0-1~debian.13~trixie",
+        "origin": ""
+      }
+    ],
+    "wrapper_seconds": 51.5
+  }
+  pass took 2m2.2s
+WALL=122.327970106
+IDS IDENTICAL (24 containers)
+engine=29.8.2
+live=true

documentation/audits/os-docker-crash-2026-10-04/partB/b4-ring0-docker-demo-felhom.log

@@ -0,0 +1,259 @@
+before: 5 containers, engine=29.7.1, live=false
+=== felhom-agent 0.142.0-rc1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true ===
+time=2026-10-04T16:12:37.832+02:00 level=INFO msg="osupdate: START" run=20261004T141237Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T141237Z
+time=2026-10-04T16:12:50.352+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T141237Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T16:12:50.352+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T16:12:50.352+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T16:12:50.352+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T16:12:50.352+02:00 level=INFO msg="osupdate: DONE" run=20261004T141237Z layer=guest vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=6 not_covered=6 restart_needed=10 reboot_needed=true wrapper_seconds=12.5
+time=2026-10-04T16:12:50.364+02:00 level=INFO msg="osupdate: START" run=20261004T141237Z layer=host vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T141237Z
+time=2026-10-04T16:13:00.773+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T141237Z layer=host lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T16:13:00.773+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T16:13:00.773+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T16:13:00.773+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T16:13:01.549+02:00 level=INFO msg="osupdate: DONE" run=20261004T141237Z layer=host vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=80 not_covered=80 restart_needed=34 reboot_needed=true wrapper_seconds=10.4
+time=2026-10-04T16:13:08.002+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE reload_rc=0 state=true containers=5 same-ids=yes"
+time=2026-10-04T16:13:08.002+02:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"containers_before\": 5, \"reload_rc\": 0, \"result\": \"on\", \"same_ids\": true}"
+time=2026-10-04T16:13:08.002+02:00 level=INFO msg="osupdate: START" run=20261004T141237Z layer=docker vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T141237Z
+time=2026-10-04T16:13:42.395+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T141237Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0"
+time=2026-10-04T16:13:42.395+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T16:13:42.395+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=6 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T16:13:42.395+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=13.7 upgraded=6 restart-needed=agetty,containerd-shim,cron,dbus-daemon,dhclient,sshd,systemd,systemd-journal,systemd-logind,systemd-network reboot-needed=yes"
+time=2026-10-04T16:14:16.982+02:00 level=INFO msg="osupdate: DONE" run=20261004T141237Z layer=docker vmid=9201 ring=0 trigger=debug outcome=applied healthy=true reason="" upgraded=6 pending=0 not_covered=0 restart_needed=10 reboot_needed=true wrapper_seconds=34.3
+  --- os-update report (guest) ---
+  {
+    "authority": "",
+    "docker_engine": "",
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "docker-ce-cli",
+      "containerd.io",
+      "docker-ce",
+      "docker-buildx-plugin",
+      "docker-ce-rootless-extras",
+      "docker-compose-plugin"
+    ],
+    "outcome": "nothing",
+    "pending": 6,
+    "reboot_needed": true,
+    "refused": null,
+    "release_id": "ring0-20261004T141237Z",
+    "restart_needed": [
+      "agetty",
+      "containerd",
+      "cron",
+      "dbus-daemon",
+      "dhclient",
+      "sshd",
+      "systemd",
+      "systemd-journal",
+      "systemd-logind",
+      "systemd-network"
+    ],
+    "ring": 0,
+    "run_id": "20261004T141237Z",
+    "upgraded": [],
+    "wrapper_seconds": 12.5
+  }
+  --- os-update report (host) ---
+  {
+    "authority": "",
+    "docker_engine": "",
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "frr",
+      "shim-signed-common",
+      "shim-unsigned",
+      "shim-helpers-amd64-signed",
+      "shim-signed",
+      "libradosstriper1",
+      "librgw2",
+      "ceph-common",
+      "librbd1",
+      "librados2",
+      "python3-cephfs",
+      "libcephfs2",
+      "python3-rgw",
+      "python3-rados",
+      "python3-ceph-argparse",
+      "python3-ceph-common",
+      "python3-rbd",
+      "ceph-fuse",
+      "chrony",
+      "libcorosync-common4",
+      "libcfg7",
+      "libcmap4",
+      "libcpg4",
+      "libknet1t64",
+      "libnozzle1t64",
+      "libquorum5",
+      "libvotequorum8",
+      "corosync",
+      "frr-pythontools",
+      "libjs-extjs",
+      "libnvpair3linux",
+      "libproxmox-acme-plugins",
+      "libproxmox-backup-qemu0",
+      "pve-qemu-kvm",
+      "libpve-notify-perl",
+      "libpve-cluster-api-perl",
+      "libpve-cluster-perl",
+      "pve-cluster",
+      "libpve-access-control",
+      "libpve-apiclient-perl",
+      "librados2-perl",
+      "proxmox-backup-client",
+      "proxmox-backup-file-restore",
+      "pve-manager",
+      "libproxmox-acme-perl",
+      "libpve-common-perl",
+      "libpve-guest-common-perl",
+      "qemu-server",
+      "libpve-storage-perl",
+      "pve-edk2-firmware-legacy",
+      "pve-edk2-firmware-ovmf",
+      "libpve-network-api-perl",
+      "libpve-network-perl",
+      "proxmox-firewall-data",
+      "pve-firewall",
+      "pve-container",
+      "pve-ha-manager",
+      "novnc-pve",
+      "proxmox-enterprise-support-keyring",
+      "proxmox-mini-journalreader",
+      "proxmox-widget-toolkit",
+      "pve-docs",
+      "pve-i18n",
+      "pve-xtermjs",
+      "pve-yew-mobile-i18n",
+      "pve-yew-mobile-gui",
+      "libuutil3linux",
+      "libzfs7linux",
+      "libzpool7linux",
+      "proxmox-first-boot",
+      "pve-firmware",
+      "proxmox-kernel-7.0.14-20-pve-signed",
+      "proxmox-kernel-7.0",
+      "proxmox-kernel-helper",
+      "pve-edk2-firmware-aarch64",
+      "pve-edk2-firmware",
+      "zfs-initramfs",
+      "zfsutils-linux",
+      "zfs-zed",
+      "tailscale"
+    ],
+    "outcome": "nothing",
+    "pending": 80,
+    "reboot_needed": true,
+    "refused": null,
+    "release_id": "ring0-20261004T141237Z",
+    "restart_needed": [
+      "agetty",
+      "blkmapd",
+      "chronyd",
+      "cron",
+      "dbus-daemon",
+      "dmeventd",
+      "ksmtuned",
+      "lxc-monitord",
+      "lxc-start",
+      "lxcfs",
+      "pmxcfs",
+      "proxmox-firewal",
+      "pve-firewall",
+      "pve-ha-crm",
+      "pve-ha-lrm",
+      "pve-lxc-syscall",
+      "pvedaemon",
+      "pvedaemon worke",
+      "pvefw-logger",
+      "pveproxy",
+      "pveproxy worker",
+      "pvescheduler",
+      "pvestatd",
+      "qmeventd",
+      "rpcbind",
+      "rrdcached",
+      "smartd",
+      "spiceproxy",
+      "spiceproxy work",
+      "sshd",
+      "systemd-logind",
+      "systemd-udevd",
+      "watchdog-mux",
+      "zed"
+    ],
+    "ring": 0,
+    "run_id": "20261004T141237Z",
+    "upgraded": [],
+    "wrapper_seconds": 10.4
+  }
+  --- os-update report (docker) ---
+  {
+    "authority": "ring0",
+    "docker_engine": "29.8.2",
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": null,
+    "outcome": "applied",
+    "pending": 0,
+    "reboot_needed": true,
+    "refused": null,
+    "release_id": "ring0-20261004T141237Z",
+    "restart_needed": [
+      "agetty",
+      "containerd-shim",
+      "cron",
+      "dbus-daemon",
+      "dhclient",
+      "sshd",
+      "systemd",
+      "systemd-journal",
+      "systemd-logind",
+      "systemd-network"
+    ],
+    "ring": 0,
+    "run_id": "20261004T141237Z",
+    "upgraded": [
+      {
+        "name": "docker-ce-cli",
+        "version": "5:29.8.2-1~debian.13~trixie",
+        "origin": ""
+      },
+      {
+        "name": "containerd.io",
+        "version": "2.3.6-1~debian.13~trixie",
+        "origin": ""
+      },
+      {
+        "name": "docker-ce",
+        "version": "5:29.8.2-1~debian.13~trixie",
+        "origin": ""
+      },
+      {
+        "name": "docker-buildx-plugin",
+        "version": "0.37.1-1~debian.13~trixie",
+        "origin": ""
+      },
+      {
+        "name": "docker-ce-rootless-extras",
+        "version": "5:29.8.2-1~debian.13~trixie",
+        "origin": ""
+      },
+      {
+        "name": "docker-compose-plugin",
+        "version": "5.6.0-1~debian.13~trixie",
+        "origin": ""
+      }
+    ],
+    "wrapper_seconds": 34.3
+  }
+  pass took 1m39.2s
+WALL=99.343682135
+IDS IDENTICAL (5 containers)
+after: engine=29.8.2 live=true

documentation/audits/os-docker-crash-2026-10-04/partB/b5-pass-0.142.0-demo-hp.log

@@ -0,0 +1,163 @@
+=== felhom-agent 0.142.0 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true ===
+time=2026-10-04T16:26:18.795+02:00 level=INFO msg="osupdate: START" run=20261004T142618Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T142618Z
+time=2026-10-04T16:26:34.572+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T142618Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T16:26:34.572+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T16:26:34.572+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T16:26:34.572+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T16:26:34.574+02:00 level=INFO msg="osupdate: DONE" run=20261004T142618Z layer=guest vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=15.7
+time=2026-10-04T16:26:34.583+02:00 level=INFO msg="osupdate: START" run=20261004T142618Z layer=host vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T142618Z
+time=2026-10-04T16:26:49.369+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T142618Z layer=host lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T16:26:49.369+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T16:26:49.369+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T16:26:49.369+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T16:26:50.399+02:00 level=INFO msg="osupdate: DONE" run=20261004T142618Z layer=host vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=78 not_covered=78 restart_needed=0 reboot_needed=false wrapper_seconds=14.7
+time=2026-10-04T16:26:52.343+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on"
+time=2026-10-04T16:26:52.343+02:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"result\": \"already on\"}"
+time=2026-10-04T16:26:52.343+02:00 level=INFO msg="osupdate: START" run=20261004T142618Z layer=docker vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T142618Z
+time=2026-10-04T16:27:09.714+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T142618Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0"
+time=2026-10-04T16:27:09.715+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T16:27:09.715+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T16:27:09.715+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T16:27:09.716+02:00 level=INFO msg="osupdate: DONE" run=20261004T142618Z layer=docker vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=17.3
+  --- os-update report (guest) ---
+  {
+    "authority": "",
+    "docker_engine": "",
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": null,
+    "outcome": "nothing",
+    "pending": 0,
+    "reboot_needed": false,
+    "refused": null,
+    "release_id": "ring0-20261004T142618Z",
+    "restart_needed": [
+      "containerd-shim"
+    ],
+    "ring": 0,
+    "run_id": "20261004T142618Z",
+    "upgraded": [],
+    "wrapper_seconds": 15.7
+  }
+  --- os-update report (host) ---
+  {
+    "authority": "",
+    "docker_engine": "",
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "frr",
+      "shim-signed-common",
+      "proxmox-secure-boot-support",
+      "shim-unsigned",
+      "shim-helpers-amd64-signed",
+      "shim-signed",
+      "amd64-microcode",
+      "libradosstriper1",
+      "librgw2",
+      "ceph-common",
+      "librbd1",
+      "librados2",
+      "python3-cephfs",
+      "libcephfs2",
+      "python3-rgw",
+      "python3-rados",
+      "python3-ceph-argparse",
+      "python3-ceph-common",
+      "python3-rbd",
+      "ceph-fuse",
+      "chrony",
+      "libcorosync-common4",
+      "libcfg7",
+      "libcmap4",
+      "libcpg4",
+      "libknet1t64",
+      "libnozzle1t64",
+      "libquorum5",
+      "libvotequorum8",
+      "corosync",
+      "frr-pythontools",
+      "libjs-extjs",
+      "libnvpair3linux",
+      "libproxmox-acme-plugins",
+      "libproxmox-backup-qemu0",
+      "pve-qemu-kvm",
+      "libpve-notify-perl",
+      "libpve-cluster-api-perl",
+      "libpve-cluster-perl",
+      "pve-cluster",
+      "libpve-access-control",
+      "libpve-apiclient-perl",
+      "librados2-perl",
+      "proxmox-backup-client",
+      "proxmox-backup-file-restore",
+      "pve-manager",
+      "libproxmox-acme-perl",
+      "libpve-common-perl",
+      "libpve-guest-common-perl",
+      "qemu-server",
+      "libpve-storage-perl",
+      "pve-edk2-firmware-legacy",
+      "pve-edk2-firmware-ovmf",
+      "libpve-network-api-perl",
+      "libpve-network-perl",
+      "proxmox-firewall-data",
+      "pve-firewall",
+      "pve-container",
+      "pve-ha-manager",
+      "novnc-pve",
+      "proxmox-enterprise-support-keyring",
+      "proxmox-mini-journalreader",
+      "proxmox-widget-toolkit",
+      "pve-docs",
+      "pve-i18n",
+      "pve-xtermjs",
+      "pve-yew-mobile-i18n",
+      "pve-yew-mobile-gui",
+      "libuutil3linux",
+      "libzfs7linux",
+      "libzpool7linux",
+      "proxmox-kernel-helper",
+      "pve-edk2-firmware-aarch64",
+      "pve-edk2-firmware",
+      "pve-firmware",
+      "zfs-initramfs",
+      "zfsutils-linux",
+      "zfs-zed"
+    ],
+    "outcome": "nothing",
+    "pending": 78,
+    "reboot_needed": false,
+    "refused": null,
+    "release_id": "ring0-20261004T142618Z",
+    "restart_needed": [],
+    "ring": 0,
+    "run_id": "20261004T142618Z",
+    "upgraded": [],
+    "wrapper_seconds": 14.7
+  }
+  --- os-update report (docker) ---
+  {
+    "authority": "ring0",
+    "docker_engine": "29.8.2",
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": null,
+    "outcome": "nothing",
+    "pending": 0,
+    "reboot_needed": false,
+    "refused": null,
+    "release_id": "ring0-20261004T142618Z",
+    "restart_needed": [
+      "containerd-shim"
+    ],
+    "ring": 0,
+    "run_id": "20261004T142618Z",
+    "upgraded": [],
+    "wrapper_seconds": 17.3
+  }
+  pass took 50.9s
+WALL=51.005070963

documentation/audits/os-docker-crash-2026-10-04/partB/b5-pass-0.142.0-felhom-pve.log

@@ -0,0 +1,218 @@
+=== felhom-agent 0.142.0 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true ===
+time=2026-10-04T16:25:39.774+02:00 level=INFO msg="osupdate: START" run=20261004T142539Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T142539Z
+time=2026-10-04T16:25:52.120+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T142539Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T16:25:52.120+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T16:25:52.120+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T16:25:52.120+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T16:25:52.120+02:00 level=INFO msg="osupdate: DONE" run=20261004T142539Z layer=guest vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=10 reboot_needed=true wrapper_seconds=12.3
+time=2026-10-04T16:25:52.131+02:00 level=INFO msg="osupdate: START" run=20261004T142539Z layer=host vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T142539Z
+time=2026-10-04T16:26:02.540+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T142539Z layer=host lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T16:26:02.540+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T16:26:02.540+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T16:26:02.540+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T16:26:03.376+02:00 level=INFO msg="osupdate: DONE" run=20261004T142539Z layer=host vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=80 not_covered=80 restart_needed=34 reboot_needed=true wrapper_seconds=10.4
+time=2026-10-04T16:26:04.999+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on"
+time=2026-10-04T16:26:04.999+02:00 level=INFO msg="osupdate: live-restore" vmid=9201 result="{\"result\": \"already on\"}"
+time=2026-10-04T16:26:04.999+02:00 level=INFO msg="osupdate: START" run=20261004T142539Z layer=docker vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T142539Z
+time=2026-10-04T16:26:18.174+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T142539Z layer=docker:9201 lane=slow mode=apply select=pending-docker packages=0 authority=ring0"
+time=2026-10-04T16:26:18.174+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T16:26:18.174+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T16:26:18.174+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T16:26:18.174+02:00 level=INFO msg="osupdate: DONE" run=20261004T142539Z layer=docker vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=10 reboot_needed=true wrapper_seconds=13.1
+  --- os-update report (guest) ---
+  {
+    "authority": "",
+    "docker_engine": "",
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": null,
+    "outcome": "nothing",
+    "pending": 0,
+    "reboot_needed": true,
+    "refused": null,
+    "release_id": "ring0-20261004T142539Z",
+    "restart_needed": [
+      "agetty",
+      "containerd-shim",
+      "cron",
+      "dbus-daemon",
+      "dhclient",
+      "sshd",
+      "systemd",
+      "systemd-journal",
+      "systemd-logind",
+      "systemd-network"
+    ],
+    "ring": 0,
+    "run_id": "20261004T142539Z",
+    "upgraded": [],
+    "wrapper_seconds": 12.3
+  }
+  --- os-update report (host) ---
+  {
+    "authority": "",
+    "docker_engine": "",
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "frr",
+      "shim-signed-common",
+      "shim-unsigned",
+      "shim-helpers-amd64-signed",
+      "shim-signed",
+      "libradosstriper1",
+      "librgw2",
+      "ceph-common",
+      "librbd1",
+      "librados2",
+      "python3-cephfs",
+      "libcephfs2",
+      "python3-rgw",
+      "python3-rados",
+      "python3-ceph-argparse",
+      "python3-ceph-common",
+      "python3-rbd",
+      "ceph-fuse",
+      "chrony",
+      "libcorosync-common4",
+      "libcfg7",
+      "libcmap4",
+      "libcpg4",
+      "libknet1t64",
+      "libnozzle1t64",
+      "libquorum5",
+      "libvotequorum8",
+      "corosync",
+      "frr-pythontools",
+      "libjs-extjs",
+      "libnvpair3linux",
+      "libproxmox-acme-plugins",
+      "libproxmox-backup-qemu0",
+      "pve-qemu-kvm",
+      "libpve-notify-perl",
+      "libpve-cluster-api-perl",
+      "libpve-cluster-perl",
+      "pve-cluster",
+      "libpve-access-control",
+      "libpve-apiclient-perl",
+      "librados2-perl",
+      "proxmox-backup-client",
+      "proxmox-backup-file-restore",
+      "pve-manager",
+      "libproxmox-acme-perl",
+      "libpve-common-perl",
+      "libpve-guest-common-perl",
+      "qemu-server",
+      "libpve-storage-perl",
+      "pve-edk2-firmware-legacy",
+      "pve-edk2-firmware-ovmf",
+      "libpve-network-api-perl",
+      "libpve-network-perl",
+      "proxmox-firewall-data",
+      "pve-firewall",
+      "pve-container",
+      "pve-ha-manager",
+      "novnc-pve",
+      "proxmox-enterprise-support-keyring",
+      "proxmox-mini-journalreader",
+      "proxmox-widget-toolkit",
+      "pve-docs",
+      "pve-i18n",
+      "pve-xtermjs",
+      "pve-yew-mobile-i18n",
+      "pve-yew-mobile-gui",
+      "libuutil3linux",
+      "libzfs7linux",
+      "libzpool7linux",
+      "proxmox-first-boot",
+      "pve-firmware",
+      "proxmox-kernel-7.0.14-20-pve-signed",
+      "proxmox-kernel-7.0",
+      "proxmox-kernel-helper",
+      "pve-edk2-firmware-aarch64",
+      "pve-edk2-firmware",
+      "zfs-initramfs",
+      "zfsutils-linux",
+      "zfs-zed",
+      "tailscale"
+    ],
+    "outcome": "nothing",
+    "pending": 80,
+    "reboot_needed": true,
+    "refused": null,
+    "release_id": "ring0-20261004T142539Z",
+    "restart_needed": [
+      "agetty",
+      "blkmapd",
+      "chronyd",
+      "cron",
+      "dbus-daemon",
+      "dmeventd",
+      "ksmtuned",
+      "lxc-monitord",
+      "lxc-start",
+      "lxcfs",
+      "pmxcfs",
+      "proxmox-firewal",
+      "pve-firewall",
+      "pve-ha-crm",
+      "pve-ha-lrm",
+      "pve-lxc-syscall",
+      "pvedaemon",
+      "pvedaemon worke",
+      "pvefw-logger",
+      "pveproxy",
+      "pveproxy worker",
+      "pvescheduler",
+      "pvestatd",
+      "qmeventd",
+      "rpcbind",
+      "rrdcached",
+      "smartd",
+      "spiceproxy",
+      "spiceproxy work",
+      "sshd",
+      "systemd-logind",
+      "systemd-udevd",
+      "watchdog-mux",
+      "zed"
+    ],
+    "ring": 0,
+    "run_id": "20261004T142539Z",
+    "upgraded": [],
+    "wrapper_seconds": 10.4
+  }
+  --- os-update report (docker) ---
+  {
+    "authority": "ring0",
+    "docker_engine": "29.8.2",
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": null,
+    "outcome": "nothing",
+    "pending": 0,
+    "reboot_needed": true,
+    "refused": null,
+    "release_id": "ring0-20261004T142539Z",
+    "restart_needed": [
+      "agetty",
+      "containerd-shim",
+      "cron",
+      "dbus-daemon",
+      "dhclient",
+      "sshd",
+      "systemd",
+      "systemd-journal",
+      "systemd-logind",
+      "systemd-network"
+    ],
+    "ring": 0,
+    "run_id": "20261004T142539Z",
+    "upgraded": [],
+    "wrapper_seconds": 13.1
+  }
+  pass took 38.4s
+WALL=38.491768907

documentation/audits/os-docker-crash-2026-10-04/partB/ring1/nonce.txt

@@ -0,0 +1 @@
+nonce=cb4a902da3007b8f4c3a440d3f2d9a60

documentation/audits/os-docker-crash-2026-10-04/partB/ring1/params.json

@@ -0,0 +1 @@
+{"release_id":"os-docker-20261004-142842","undo":false,"packages":[{"name":"containerd.io","version":"2.3.6-1~debian.13~trixie","origin":"Docker CE"},{"name":"docker-buildx-plugin","version":"0.37.1-1~debian.13~trixie","origin":"Docker CE"},{"name":"docker-ce","version":"5:29.8.2-1~debian.13~trixie","origin":"Docker CE"},{"name":"docker-ce-cli","version":"5:29.8.2-1~debian.13~trixie","origin":"Docker CE"},{"name":"docker-ce-rootless-extras","version":"5:29.8.2-1~debian.13~trixie","origin":"Docker CE"},{"name":"docker-compose-plugin","version":"5.6.0-1~debian.13~trixie","origin":"Docker CE"}]}

documentation/audits/os-docker-crash-2026-10-04/partB/ring1/r0-set-ring1.txt

@@ -0,0 +1,2 @@
+HTTP/1.1 303 See Other
+Location: /system?flash=done

documentation/audits/os-docker-crash-2026-10-04/partB/ring1/r1-ring1-night-pass.txt

@@ -0,0 +1,11 @@
+=== felhom-agent 0.142.0 selftest=os-update vmid=9201 ring=1 enabled=true guest-release=true host-release=true appliance=true ===
+time=2026-10-04T16:42:05.041+02:00 level=INFO msg="osupdate: DONE" run=20261004T144154Z layer=guest vmid=9201 ring=1 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=10 reboot_needed=true wrapper_seconds=10.6
+time=2026-10-04T16:42:15.090+02:00 level=INFO msg="osupdate: DONE" run=20261004T144154Z layer=host vmid=9201 ring=1 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=80 not_covered=80 restart_needed=34 reboot_needed=true wrapper_seconds=9.2
+time=2026-10-04T16:42:15.108+02:00 level=INFO msg="osupdate: docker step skipped — ring 1 takes an engine set only inside a signed operator job (`11` §5.8)" run=20261004T144154Z vmid=9201 trigger=debug ring=1 enabled=true
+  docker step: skipped (see the log line above)
+containerd.io	2.3.6-1~debian.13~trixie
+docker-buildx-plugin	0.37.1-1~debian.13~trixie
+docker-ce	5:29.8.2-1~debian.13~trixie
+docker-ce-cli	5:29.8.2-1~debian.13~trixie
+docker-ce-rootless-extras	5:29.8.2-1~debian.13~trixie
+docker-compose-plugin	5.6.0-1~debian.13~trixie

documentation/audits/os-docker-crash-2026-10-04/partB/ring1/r2-signed-job.txt

@@ -0,0 +1,2 @@
+signed: op=os_docker_step host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=cb4a902da3007b8f4c3a440d3f2d9a60 expires=2026-10-04T15:27:25Z
+uploaded signed op to the hub jobs queue

documentation/audits/os-docker-crash-2026-10-04/partB/ring1/r3-job-result.txt

@@ -0,0 +1,7 @@
+time=2026-10-04T16:52:54.082+02:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=b7956e7339bfda31 op=os_docker_step key_id=felhom-op-1 nonce=cb4a902da3007b8f4c3a440d3f2d9a60
+time=2026-10-04T16:52:55.697+02:00 level=INFO msg="osupdate: START" run=20261004T145254Z layer=docker vmid=9201 ring=1 trigger=signed enabled=true release=os-docker-20261004-142842
+time=2026-10-04T16:53:07.878+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=os-docker-20261004-142842 layer=docker:9201 lane=slow mode=apply select=listed packages=6 authority=signed"
+time=2026-10-04T16:53:07.878+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=6 not-installed=0 from-snapshot=0"
+time=2026-10-04T16:53:07.878+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T16:53:07.879+02:00 level=INFO msg="osupdate: DONE" run=20261004T145254Z layer=docker vmid=9201 ring=1 trigger=signed outcome=nothing healthy=true reason="" upgraded=0 pending=0 not_covered=0 restart_needed=10 reboot_needed=true wrapper_seconds=
+time=2026-10-04T16:53:07.884+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=b7956e7339bfda31 op=os_docker_step

documentation/audits/os-docker-crash-2026-10-04/partB/ring1/r4-replay-signed.txt

@@ -0,0 +1,2 @@
+signed: op=os_docker_step host=demo-felhom-8363b5 guest="" key_id=felhom-op-1 nonce=cb4a902da3007b8f4c3a440d3f2d9a60 expires=2026-10-04T15:38:36Z
+uploaded signed op to the hub jobs queue

documentation/audits/os-docker-crash-2026-10-04/partB/ring1/r5-replay-result.txt

@@ -0,0 +1,3 @@
+time=2026-10-04T16:52:54.082+02:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=b7956e7339bfda31 op=os_docker_step key_id=felhom-op-1 nonce=cb4a902da3007b8f4c3a440d3f2d9a60
+time=2026-10-04T16:53:07.884+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=b7956e7339bfda31 op=os_docker_step
+time=2026-10-04T17:07:54.007+02:00 level=WARN msg="signedjobs: REJECTED signed op — executor not called" job=dec707a162451c5b op=os_docker_step reason=rejected err="authz: replay (nonce already seen): nonce cb4a902da3007b8f4c3a440d3f2d9a60"

documentation/audits/os-docker-crash-2026-10-04/partB/ring1/r6-back-to-ring0.txt

@@ -0,0 +1,2 @@
+HTTP/1.1 303 See Other
+Location: /system?flash=done

documentation/audits/os-docker-crash-2026-10-04/partB/undo/agent-journal-and-after.txt

@@ -0,0 +1,16 @@
+time=2026-10-04T16:39:43.360+02:00 level=INFO msg="audit: gate decision" class=os_docker_step host=demo-hp-bb76ea guest="" source=one_shot_job disposition=destructive allowed=true reason=signed key_id=felhom-op-1 nonce=f21cbf93… durable_id=""
+time=2026-10-04T16:39:43.360+02:00 level=INFO msg="gate decision" class=os_docker_step guest="" source=one_shot_job disposition=destructive allowed=true reason=signed
+time=2026-10-04T16:39:43.360+02:00 level=WARN msg="signedjobs: AUTHORIZED signed op — executing" job=c9f9efc098335bf9 op=os_docker_step key_id=felhom-op-1 nonce=f21cbf9302fb8b655b3ccf3e36f1cbea
+os-apply: LIVE-RESTORE already on
+time=2026-10-04T16:39:45.427+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: LIVE-RESTORE already on"
+time=2026-10-04T16:39:45.427+02:00 level=INFO msg="osupdate: START" run=20261004T143943Z layer=docker vmid=9201 ring=0 trigger=signed enabled=true release=undo-to-29.7.2
+os-apply: START release=undo-to-29.7.2 layer=docker:9201 lane=slow mode=apply select=listed packages=6 authority=signed UNDO
+os-apply: PLAN upgrade=6 already=0 not-installed=0 from-snapshot=0
+os-apply: DONE rc=0 seconds=16.7 upgraded=6 restart-needed=containerd-shim reboot-needed=no
+time=2026-10-04T16:40:26.840+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=undo-to-29.7.2 layer=docker:9201 lane=slow mode=apply select=listed packages=6 authority=signed UNDO"
+time=2026-10-04T16:40:26.840+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=6 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T16:40:26.840+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=16.7 upgraded=6 restart-needed=containerd-shim reboot-needed=no"
+time=2026-10-04T16:41:02.662+02:00 level=INFO msg="osupdate: DONE" run=20261004T143943Z layer=docker vmid=9201 ring=0 trigger=signed outcome=applied healthy=true reason="" upgraded=6 pending=6 not_covered=0 restart_needed=1 reboot_needed=false wrapper_seconds=41.3
+time=2026-10-04T16:41:02.716+02:00 level=WARN msg="signedjobs: signed op COMPLETED" job=c9f9efc098335bf9 op=os_docker_step
+IDS IDENTICAL (24 containers)
+after: engine=29.7.2 live=true

documentation/audits/os-docker-crash-2026-10-04/partB/undo/back-to-newest-demo-hp.txt

@@ -0,0 +1,8 @@
+time=2026-10-04T16:42:55.380+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+guest nothing healthy=true
+time=2026-10-04T16:43:10.212+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+host nothing healthy=true
+time=2026-10-04T16:43:58.115+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=6 already=0 not-installed=0 from-snapshot=0"
+docker applied healthy=true
+IDS IDENTICAL (24 containers)
+engine=29.8.2

documentation/audits/os-docker-crash-2026-10-04/partB/undo/before.txt

@@ -0,0 +1 @@
+before: 24 containers engine=29.8.2

documentation/audits/os-docker-crash-2026-10-04/partB/undo/params.json

@@ -0,0 +1 @@
+{"release_id":"undo-to-29.7.2","undo":true,"packages":[{"name":"containerd.io","version":"2.3.3-1~debian.13~trixie","origin":"Docker CE"},{"name":"docker-buildx-plugin","version":"0.36.1-1~debian.13~trixie","origin":"Docker CE"},{"name":"docker-ce","version":"5:29.7.2-1~debian.13~trixie","origin":"Docker CE"},{"name":"docker-ce-cli","version":"5:29.7.2-1~debian.13~trixie","origin":"Docker CE"},{"name":"docker-ce-rootless-extras","version":"5:29.7.2-1~debian.13~trixie","origin":"Docker CE"},{"name":"docker-compose-plugin","version":"5.5.0-1~debian.13~trixie","origin":"Docker CE"}]}

documentation/audits/os-docker-crash-2026-10-04/partB/undo/signed.txt

@@ -0,0 +1,2 @@
+signed: op=os_docker_step host=demo-hp-bb76ea guest="" key_id=felhom-op-1 nonce=f21cbf9302fb8b655b3ccf3e36f1cbea expires=2026-10-04T15:15:26Z
+uploaded signed op to the hub jobs queue

documentation/audits/os-docker-crash-2026-10-04/partC/c0-readonly-demo-hp.txt

@@ -0,0 +1,53 @@
++ sysctl kernel.panic kernel.panic_on_oops kernel.sysrq kernel.panic_on_warn kernel.softlockup_panic kernel.hung_task_panic
+kernel.panic = 0
+kernel.panic_on_oops = 0
+kernel.sysrq = 438
+kernel.panic_on_warn = 0
+kernel.softlockup_panic = 0
+kernel.hung_task_panic = 0
++ ls -la /sys/fs/pstore/ /var/lib/systemd/pstore/
+/sys/fs/pstore/:
+total 0
+drwxr-x---  2 root root 0 Oct  4 14:36 .
+drwxr-xr-x 10 root root 0 Oct  4 14:36 ..
+
+/var/lib/systemd/pstore/:
+total 8
+drwxr-xr-x  2 root root 4096 May 21 13:32 .
+drwxr-xr-x 12 root root 4096 Aug 21 17:44 ..
++ mount
++ grep pstore
+none on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
++ cat /sys/module/pstore/parameters/backend
+efi_pstore
++ dmesg
++ grep -iE 'pstore|efi_pstore|ramoops|erst'
++ head
+[    4.129612] systemd[1]: Starting modprobe@efi_pstore.service - Load Kernel Module efi_pstore...
+[    4.153714] pstore: Using crash dump compression: deflate
+[    4.206148] pstore: Registered efi_pstore as persistent store backend
+[    4.207174] systemd[1]: modprobe@efi_pstore.service: Deactivated successfully.
+[    4.207514] systemd[1]: Finished modprobe@efi_pstore.service - Load Kernel Module efi_pstore.
+[    4.210265] systemd[1]: systemd-pstore.service - Platform Persistent Storage Archival skipped, unmet condition check ConditionDirectoryNotEmpty=/sys/fs/pstore
++ grep -E '^#?Storage' /etc/systemd/journald.conf
+#Storage=auto
++ ls /var/log/journal
++ head -2
+9756a908fef046fda6a56428d3fefe3c
++ journalctl --list-boots --no-pager
++ tail -4
+ -3 9cfa86adcf9f4dec99e324f5e2645032 Sun 2026-10-04 09:44:31 CEST Sun 2026-10-04 09:45:46 CEST
+ -2 1407ed1ad3ba4a37ac84ddae7cb05b3b Sun 2026-10-04 09:46:18 CEST Sun 2026-10-04 14:26:34 CEST
+ -1 ffd2f9aa02a344399338755a7958219f Sun 2026-10-04 14:27:08 CEST Sun 2026-10-04 14:35:43 CEST
+  0 7e320e5daf1f4ee79760c2de1aaf39ff Sun 2026-10-04 14:36:16 CEST Sun 2026-10-04 15:26:56 CEST
++ systemctl is-enabled systemd-pstore.service
+enabled
++ lsmod
++ grep -iE 'efi_pstore|ramoops'
+efi_pstore             12288  0
++ ls /sys/firmware/efi/efivars
++ grep -ci dump
+0
++ which kdump-config
++ cat /proc/cmdline
+BOOT_IMAGE=/boot/vmlinuz-7.0.14-20-pve root=/dev/mapper/pve-root ro quiet

documentation/audits/os-docker-crash-2026-10-04/partC/c1-crash1.txt

@@ -0,0 +1,85 @@
+== before crash 1, 2026-10-04T13:38:26Z
+kernel.panic = 10
+7e320e5d-af1f-4ee7-9760-c2de1aaf39ff
+7.0.14-20-pve
+2026-10-04 14:36:11
+crash issued 2026-10-04T13:38:27Z
+== after crash 1, 2026-10-04T13:39:27Z
++ uname -r
+7.0.14-20-pve
++ uptime -s
+2026-10-04 15:39:06
++ cat /proc/sys/kernel/random/boot_id
+0defc390-c295-4e9f-aa6e-13d69a1f24c3
++ sysctl kernel.panic
+kernel.panic = 0
++ ls -la /sys/fs/pstore/ /var/lib/systemd/pstore/
+/sys/fs/pstore/:
+total 0
+drwxr-x---  2 root root 0 Oct  4 15:39 .
+drwxr-xr-x 10 root root 0 Oct  4 15:39 ..
+
+/var/lib/systemd/pstore/:
+total 8
+drwxr-xr-x  2 root root 4096 May 21 13:32 .
+drwxr-xr-x 12 root root 4096 Aug 21 17:44 ..
++ find /var/lib/systemd/pstore -type f
++ head
+++ find /var/lib/systemd/pstore -type f
+++ head -2
++ journalctl -b -1 --no-pager -n 8 -o short-iso
+2026-10-04T15:38:27+02:00 demo-hp systemd[205455]: Listening on gpg-agent.socket - GnuPG cryptographic agent and passphrase cache.
+2026-10-04T15:38:27+02:00 demo-hp systemd[205455]: Listening on ssh-agent.socket - OpenSSH Agent socket.
+2026-10-04T15:38:27+02:00 demo-hp systemd[205455]: Reached target sockets.target - Sockets.
+2026-10-04T15:38:27+02:00 demo-hp systemd[205455]: Reached target basic.target - Basic System.
+2026-10-04T15:38:27+02:00 demo-hp systemd[205455]: Reached target default.target - Main User Target.
+2026-10-04T15:38:27+02:00 demo-hp systemd[205455]: Startup finished in 310ms.
+2026-10-04T15:38:27+02:00 demo-hp systemd[1]: Started user@0.service - User Manager for UID 0.
+2026-10-04T15:38:27+02:00 demo-hp systemd[1]: Started session-15.scope - Session 15 of User root.
++ journalctl -b -2 --no-pager -n 4 -o short-iso
+2026-10-04T14:35:43+02:00 demo-hp systemd-shutdown[1]: Syncing filesystems and block devices.
+2026-10-04T14:35:43+02:00 demo-hp systemd-shutdown[1]: Sending SIGTERM to remaining processes...
+2026-10-04T14:35:43+02:00 demo-hp systemd-journald[349]: Received SIGTERM from PID 1 (systemd-shutdow).
+2026-10-04T14:35:43+02:00 demo-hp systemd-journald[349]: Journal stopped
++ journalctl -b 0 -u systemd-pstore --no-pager -o cat
++ head
++ last -x
++ head -5
+reboot   system boot  7.0.14-20-pve    Sun Oct  4 15:39 - still running
+reboot   system boot  7.0.14-20-pve    Sun Oct  4 14:36 - crash 
+reboot   system boot  7.0.14-20-pve    Sun Oct  4 14:27 - 14:35  (00:08)
+shutdown system down  7.0.14-20-pve    Sun Oct  4 14:35 - 14:36  (00:00)
+reboot   system boot  7.0.2-6-pve      Sun Oct  4 09:46 - 14:26  (04:39)
++ cat /sys/module/efi_pstore/parameters/pstore_disable
+N
++ cat /sys/module/pstore/parameters/max_reason
+cat: /sys/module/pstore/parameters/max_reason: No such file or directory
++ cat /sys/module/pstore/parameters/kmsg_bytes
+10240
+++ uname -r
++ grep -E 'EFI_VARS_PSTORE|PSTORE_DEFAULT|CONFIG_PSTORE=' /boot/config-7.0.14-20-pve
+CONFIG_EFI_VARS_PSTORE=m
+# CONFIG_EFI_VARS_PSTORE_DEFAULT_DISABLE is not set
+CONFIG_PSTORE=y
+CONFIG_PSTORE_DEFAULT_KMSG_BYTES=10240
++ cat /sys/module/printk/parameters/always_kmsg_dump
+N
++ journalctl -b 0 -u felhom-agent --no-pager -o short-iso
++ grep -iE 'started|report|hub:'
++ head -5
++ cut -c1-200
+2026-10-04T15:39:22+02:00 demo-hp systemd[1]: Started felhom-agent.service - Felhom host agent (Proxmox host tier; hub control loop + PBS verify + storage watchdog).
+2026-10-04T15:39:23+02:00 demo-hp sudo[1492]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
+2026-10-04T15:39:24+02:00 demo-hp felhom-agent[1304]: time=2026-10-04T15:39:24.149+02:00 level=INFO msg="guest-power: watchdog started" interval=1m0s max_attempts=3
+2026-10-04T15:39:24+02:00 demo-hp felhom-agent[1304]: time=2026-10-04T15:39:24.149+02:00 level=INFO msg="controller-supervisor: started" interval=30s confirm_sweeps=2 crashloop_max=3 crashloop_window=
+2026-10-04T15:39:24+02:00 demo-hp sudo[1585]: felhom-agent : PWD=/ ; USER=root ; COMMAND=/usr/sbin/lvs --reportformat json --units b -o lv_name,data_percent,metadata_percent -- pve/data
++ pct status 9201
+status: running
++ journalctl -k -b -2 --no-pager
++ grep -ciE 'oops|BUG:'
+0
++ journalctl -k --no-pager
++ grep -iE 'Oops|BUG:|Call Trace'
++ head -3
++ cat /proc/sys/kernel/tainted
+4097

documentation/audits/os-docker-crash-2026-10-04/partC/c2-guard-crash1.txt

@@ -0,0 +1,29 @@
+== crash 1 issued 2026-10-04T14:51:28Z
+kernel.panic = 10
+0defc390-c295-4e9f-aa6e-13d69a1f24c3
+back: ping+ssh after 54s
+7.0.14-20-pve
+kernel.panic = 10
+{
+  "armed": true,
+  "boot_id": "cc0aa40b-ac53-4524-a46f-6c5983ce5ed9",
+  "config": {
+    "LIMIT": 3,
+    "PANIC_SECONDS": 10,
+    "REARM_HOURS": 24,
+    "WINDOW_MINUTES": 60
+  },
+  "kernel_panic": 10,
+  "last_boot_at": "2026-10-04T14:52:08Z",
+  "last_boot_unclean": true,
+  "tripped": false,
+  "unclean_boots": [
+    "2026-10-04T14:52:08Z"
+  ],
+  "unclean_boots_24h": 1,
+  "unclean_boots_in_window": 1,
+  "updated_at": "2026-10-04T14:52:18Z",
+  "version": 1
+}
+hub log since the boot:
+2026/10/04 16:52:57 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 0 backups, 2 restore-tests, 2 pbs-snapshots, 13538 bytes)

documentation/audits/os-docker-crash-2026-10-04/partC/c2-guard-crash2.txt

@@ -0,0 +1,33 @@
+== crash 2 issued 2026-10-04T14:53:36Z
+kernel.panic = 10
+cc0aa40b-ac53-4524-a46f-6c5983ce5ed9
+back: ping+ssh after 53s
+7.0.14-20-pve
+kernel.panic = 0
+{
+  "armed": false,
+  "boot_id": "53f69dfb-93ef-44c3-a9ad-803d86dfb0b3",
+  "config": {
+    "LIMIT": 3,
+    "PANIC_SECONDS": 10,
+    "REARM_HOURS": 24,
+    "WINDOW_MINUTES": 60
+  },
+  "kernel_panic": 0,
+  "last_boot_at": "2026-10-04T14:54:33Z",
+  "last_boot_unclean": true,
+  "tripped": true,
+  "tripped_at": "2026-10-04T14:54:43Z",
+  "tripped_reason": "2 unclean boots within 60 minutes \u2014 the next crash leaves the box off (limit 3)",
+  "unclean_boots": [
+    "2026-10-04T14:52:08Z",
+    "2026-10-04T14:54:33Z"
+  ],
+  "unclean_boots_24h": 2,
+  "unclean_boots_in_window": 2,
+  "updated_at": "2026-10-04T14:54:43Z",
+  "version": 1
+}
+hub log since the boot:
+2026/10/04 16:52:57 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 0 backups, 2 restore-tests, 2 pbs-snapshots, 13538 bytes)
+2026/10/04 16:54:53 [INFO] host-report from demo-hp-bb76ea (1 guests, 4 storage targets, 0 backups, 2 restore-tests, 2 pbs-snapshots, 13395 bytes)

documentation/audits/os-docker-crash-2026-10-04/partC/c2-guard-crash3.txt

@@ -0,0 +1,4 @@
+== crash 3 issued 2026-10-04T14:55:13Z
+kernel.panic = 0
+53f69dfb-93ef-44c3-a9ad-803d86dfb0b3
+STAYED OFF for 184s (no ping)

documentation/audits/os-docker-crash-2026-10-04/partC/c3-after-power-on.txt

@@ -0,0 +1,29 @@
+== after the operator's power-on, ssh back 2026-10-04T15:17:41Z (27s after his word)
+7.0.14-20-pve
+kernel.panic = 0
+{
+  "armed": false,
+  "boot_id": "182af911-22a9-41ec-97ac-7ee247a4d470",
+  "config": {
+    "LIMIT": 3,
+    "PANIC_SECONDS": 10,
+    "REARM_HOURS": 24,
+    "WINDOW_MINUTES": 60
+  },
+  "kernel_panic": 0,
+  "last_boot_at": "2026-10-04T15:17:26Z",
+  "last_boot_unclean": true,
+  "tripped": true,
+  "tripped_at": "2026-10-04T14:54:43Z",
+  "tripped_reason": "2 unclean boots within 60 minutes \u2014 the next crash leaves the box off (limit 3)",
+  "unclean_boots": [
+    "2026-10-04T14:52:08Z",
+    "2026-10-04T14:54:33Z",
+    "2026-10-04T15:17:26Z"
+  ],
+  "unclean_boots_24h": 3,
+  "unclean_boots_in_window": 3,
+  "updated_at": "2026-10-04T15:17:36Z",
+  "version": 1
+}
+status: stopped

documentation/audits/os-docker-crash-2026-10-04/partC/c4-hub-events.txt

@@ -0,0 +1,12 @@
+2026/10/04 17:21:00 [INFO] Operator email sent for demo-hp/app_start_failed
+2026/10/04 17:26:30 [INFO] Operator email sent for demo-hp/app_stopped_unhealthy
+2026/10/04 17:26:30 [INFO] Customer email sent to drill@felhom.eu for demo-hp/app_stopped_unhealthy
+2026/10/04 17:32:52 [WARN] host demo-hp-bb76ea crash guard: host_crash_restart
+2026/10/04 17:32:52 [WARN] host demo-hp-bb76ea crash guard: host_restarted_after_crash
+2026/10/04 17:32:52 [WARN] host demo-hp-bb76ea crash guard: host_crash_restart
+2026/10/04 17:32:52 [WARN] host demo-hp-bb76ea crash guard: host_restarted_after_crash
+2026/10/04 17:32:52 [WARN] host demo-hp-bb76ea crash guard: host_crash_restart
+2026/10/04 17:32:52 [WARN] host demo-hp-bb76ea crash guard: host_restarted_after_crash
+2026/10/04 17:32:52 [WARN] host demo-hp-bb76ea crash guard: host_crash_guard_tripped
+2026/10/04 17:32:52 [INFO] Operator email sent for demo-hp/host_crash_restart
+2026/10/04 17:32:52 [INFO] Operator email sent for demo-hp/host_crash_guard_tripped

documentation/audits/os-docker-crash-2026-10-04/partC/c5-system-page-tripped.html

@@ -0,0 +1,240 @@
+<!DOCTYPE html>
+<html lang="en">
+<head>
+    <meta charset="UTF-8">
+    <meta name="viewport" content="width=device-width, initial-scale=1.0">
+    <title>System — Felhom Hub</title>
+    <link rel="stylesheet" href="/style.css?v=0.132.0">
+    <style>
+        .sys td, .sys th { white-space: nowrap; font-size: 0.82em; vertical-align: top; }
+        .sys .grp { border-left: 2px solid var(--border, #444); }
+        .c-warn { color: var(--warn); font-weight: 600; }
+        .c-bad { color: var(--danger, #e5534b); font-weight: 700; }
+        .sys form { display: inline; }
+        .rel-grid { display: grid; grid-template-columns: repeat(auto-fit, minmax(16rem, 1fr)); gap: 0.75rem; }
+    </style>
+</head>
+<body>
+    <svg xmlns="http://www.w3.org/2000/svg" style="display:none" aria-hidden="true">
+<symbol id="i-triangle-alert" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="m21.73 18-8-14a2 2 0 0 0-3.48 0l-8 14A2 2 0 0 0 4 21h16a2 2 0 0 0 1.73-3" /> <path d="M12 9v4" /> <path d="M12 17h.01" /></symbol>
+<symbol id="i-check" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M20 6 9 17l-5-5" /></symbol>
+<symbol id="i-server" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><rect width="20" height="8" x="2" y="2" rx="2" ry="2" /> <rect width="20" height="8" x="2" y="14" rx="2" ry="2" /> <line x1="6" x2="6.01" y1="6" y2="6" /> <line x1="6" x2="6.01" y1="18" y2="18" /></symbol>
+<symbol id="i-settings" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M9.671 4.136a2.34 2.34 0 0 1 4.659 0 2.34 2.34 0 0 0 3.319 1.915 2.34 2.34 0 0 1 2.33 4.033 2.34 2.34 0 0 0 0 3.831 2.34 2.34 0 0 1-2.33 4.033 2.34 2.34 0 0 0-3.319 1.915 2.34 2.34 0 0 1-4.659 0 2.34 2.34 0 0 0-3.32-1.915 2.34 2.34 0 0 1-2.33-4.033 2.34 2.34 0 0 0 0-3.831A2.34 2.34 0 0 1 6.35 6.051a2.34 2.34 0 0 0 3.319-1.915" /> <circle cx="12" cy="12" r="3" /></symbol>
+<symbol id="i-x" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M18 6 6 18" /> <path d="m6 6 12 12" /></symbol>
+<symbol id="i-info" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10" /> <path d="M12 16v-4" /> <path d="M12 8h.01" /></symbol>
+<symbol id="i-hard-drive" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M10 16h.01" /> <path d="M2.212 11.577a2 2 0 0 0-.212.896V18a2 2 0 0 0 2 2h16a2 2 0 0 0 2-2v-5.527a2 2 0 0 0-.212-.896L18.55 5.11A2 2 0 0 0 16.76 4H7.24a2 2 0 0 0-1.79 1.11z" /> <path d="M21.946 12.013H2.054" /> <path d="M6 16h.01" /></symbol>
+<symbol id="i-cpu" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M12 20v2" /> <path d="M12 2v2" /> <path d="M17 20v2" /> <path d="M17 2v2" /> <path d="M2 12h2" /> <path d="M2 17h2" /> <path d="M2 7h2" /> <path d="M20 12h2" /> <path d="M20 17h2" /> <path d="M20 7h2" /> <path d="M7 20v2" /> <path d="M7 2v2" /> <rect x="4" y="4" width="16" height="16" rx="2" /> <rect x="8" y="8" width="8" height="8" rx="1" /></symbol>
+<symbol id="i-clock" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><circle cx="12" cy="12" r="10" /> <path d="M12 6v6l4 2" /></symbol>
+<symbol id="i-boxes" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M2.97 12.92A2 2 0 0 0 2 14.63v3.24a2 2 0 0 0 .97 1.71l3 1.8a2 2 0 0 0 2.06 0L12 19v-5.5l-5-3-4.03 2.42Z" /> <path d="m7 16.5-4.74-2.85" /> <path d="m7 16.5 5-3" /> <path d="M7 16.5v5.17" /> <path d="M12 13.5V19l3.97 2.38a2 2 0 0 0 2.06 0l3-1.8a2 2 0 0 0 .97-1.71v-3.24a2 2 0 0 0-.97-1.71L17 10.5l-5 3Z" /> <path d="m17 16.5-5-3" /> <path d="m17 16.5 4.74-2.85" /> <path d="M17 16.5v5.17" /> <path d="M7.97 4.42A2 2 0 0 0 7 6.13v4.37l5 3 5-3V6.13a2 2 0 0 0-.97-1.71l-3-1.8a2 2 0 0 0-2.06 0l-3 1.8Z" /> <path d="M12 8 7.26 5.15" /> <path d="m12 8 4.74-2.85" /> <path d="M12 13.5V8" /></symbol>
+<symbol id="i-users" viewBox="0 0 24 24" fill="none" stroke="currentColor" stroke-width="2" stroke-linecap="round" stroke-linejoin="round"><path d="M16 21v-2a4 4 0 0 0-4-4H6a4 4 0 0 0-4 4v2" /> <path d="M16 3.128a4 4 0 0 1 0 7.744" /> <path d="M22 21v-2a4 4 0 0 0-3-3.87" /> <circle cx="9" cy="7" r="4" /></symbol>
+</svg>
+    
+<script>
+
+
+function felhomConfirm(el,question,onYes){
+    if(!el||el.dataset.fcOpen)return;
+    el.dataset.fcOpen='1';
+    var wrap=document.createElement('span');wrap.className='inline-confirm';
+    var q=document.createElement('span');q.className='inline-confirm-q';q.textContent=question;
+    var yes=document.createElement('button');yes.type='button';yes.className='btn btn-sm btn-danger';yes.textContent='Igen';
+    var no=document.createElement('button');no.type='button';no.className='btn btn-sm btn-outline';no.textContent='Mégse';
+    wrap.appendChild(q);wrap.appendChild(yes);wrap.appendChild(no);
+    el.style.display='none';el.parentNode.insertBefore(wrap,el.nextSibling);
+    function close(){wrap.remove();el.style.display='';delete el.dataset.fcOpen;}
+    no.addEventListener('click',close);
+    yes.addEventListener('click',function(){close();onYes();});
+}
+document.addEventListener('click',function(e){
+    var btn=e.target.closest?e.target.closest('[data-confirm]'):null;
+    if(!btn)return;
+    e.preventDefault();
+    felhomConfirm(btn,btn.getAttribute('data-confirm'),function(){
+        var form=btn.closest('form');
+        if(form){if(form.requestSubmit)form.requestSubmit(btn);else form.submit();}
+    });
+});
+</script>
+
+    <div class="container">
+        <header>
+            <h1>Felhom <span>Hub</span></h1>
+            <nav class="nav-links">
+                <a href="/" class="nav-link">Dashboard</a>
+                <a href="/configs" class="nav-link">Customers</a>
+                <a href="/apps" class="nav-link">Apps</a>
+                <a href="/hosts" class="nav-link">Hosts</a>
+                <a href="/system" class="nav-link active">System</a>
+                <a href="/offsite" class="nav-link">Offsite</a>
+                <a href="/configuration" class="nav-link">Configuration</a>
+            </nav>
+        </header>
+
+        <h2 style="margin-bottom: 1rem;">System — versions and OS updates</h2>
+
+        
+        
+
+        <section class="card" style="margin-bottom: 1.5rem;">
+            <h3 style="margin-top: 0;">Approved releases</h3>
+            <div class="rel-grid">
+                
+                <div><strong>guest</strong>: <code>os-guest-20261004-123933</code><br>
+                    <span class="text-muted">272 packages · 2026-10-04 12:39 UTC · by auto</span></div>
+                
+                <div><strong>host</strong>: <code>os-host-20261004-124133</code><br>
+                    <span class="text-muted">607 packages · 2026-10-04 12:41 UTC · by auto</span></div>
+                
+                <div><strong>docker</strong>: <code>os-docker-20261004-142842</code><br>
+                    <span class="text-muted">6 packages · 2026-10-04 14:28 UTC · by operator</span></div>
+                
+            </div>
+            <h3>What ring 0 runs now</h3>
+            <div class="rel-grid">
+                
+                <div><strong>guest</strong>:
+                    272 packages, first seen 2026-10-04 11:07 UTC<br>
+                    <span class="text-muted">approved as os-guest-20261004-123933</span>
+                    
+                    
+                </div>
+                
+                <div><strong>host</strong>:
+                    607 packages, first seen 2026-10-04 12:24 UTC<br>
+                    <span class="text-muted">approved as os-host-20261004-124133</span>
+                    
+                    
+                </div>
+                
+                <div><strong>docker</strong>:
+                    6 packages, first seen 2026-10-04 14:28 UTC<br>
+                    <span class="text-muted">approved as os-docker-20261004-142842</span>
+                    
+                    
+                </div>
+                
+            </div>
+            <form method="POST" action="/os/approve-now" style="margin-top: 0.8rem;">
+                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                <button type="submit" class="btn btn-sm btn-danger" data-confirm="Approve the guest and host sets ring 0 runs NOW, without the 24 h + 1 night wait? Every ring-1 box installs them at its next night run.">Approve now (guest + host)</button>
+                <span class="text-muted" style="font-size: 0.85em;">An urgent fix only — normally the hub approves after 24 h and one night.</span>
+            </form>
+        </section>
+
+        
+        <section class="card" style="padding: 0; overflow-x: auto;">
+            <table class="data-table sys">
+                <thead>
+                    <tr>
+                        <th>Box</th><th>Ring / updates</th><th>Tunnel</th>
+                        <th class="grp">Proxmox</th><th>Kernel (running)</th><th>Kernel (next boot)</th><th>Debian</th><th>Felhom release</th><th>Pending</th><th>Not covered</th><th>Held</th><th>Reboot needed</th><th>kernel.panic</th><th>Oops</th><th>Crash restarts 24 h</th><th>Crash guard</th>
+                        <th class="grp">Guest Debian</th><th>Felhom release</th><th>Pending</th><th>Restart needed</th>
+                        <th class="grp">Docker</th><th>containerd</th><th>live-restore</th><th>Docker release</th>
+                        <th class="grp">Last OS leg</th>
+                    </tr>
+                    <tr class="text-muted"><th></th><th></th><th></th><th class="grp" colspan="13">host</th><th class="grp" colspan="4">guest</th><th class="grp" colspan="4">Docker engine</th><th class="grp"></th></tr>
+                </thead>
+                <tbody>
+                    
+                    <tr>
+                        <td><a href="/hosts/demo-felhom-8363b5">demo-felhom-8363b5</a><br><span class="text-muted">Demo Ügyfél</span>
+                            </td>
+                        <td>
+                            ring 0
+                            <form method="POST" action="/os/ring/demo-felhom-8363b5">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="ring" value="1"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Make demo-felhom-8363b5 a normal (ring 1) box? It then installs only approved releases.">→ normal</button>
+                                
+                            </form><br>
+                            updates <strong>ON</strong>
+                            <form method="POST" action="/os/enabled/demo-felhom-8363b5">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="on" value="0"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Switch OS updates OFF for demo-felhom-8363b5? It keeps reporting and installs nothing.">switch off</button>
+                                
+                            </form>
+                        </td>
+                        <td>running</td>
+                        <td class="grp ">9.2.2</td>
+                        <td>7.0.2-6-pve</td><td>7.0.2-6-pve</td><td>13.7</td>
+                        <td>os-host-20261004-124133</td><td>80</td><td>0</td>
+                        <td>none</td><td class="c-warn">since 2026-10-04</td><td>10 s</td><td>no</td>
+                        <td>0</td><td>armed</td>
+                        <td class="grp ">13.7</td>
+                        <td>os-guest-20261004-123933</td><td>0</td><td>10</td>
+                        <td class="grp ">29.8.2</td>
+                        <td>2.3.6-1~debian.13~trixie</td><td>on</td><td>os-docker-20261004-142842</td>
+                        <td class="grp " title="last successful leg: 51 min ago">40 min ago · nothing · 12 s</td>
+                    </tr>
+                    
+                    <tr>
+                        <td><a href="/hosts/demo-hp-bb76ea">demo-hp-bb76ea</a><br><span class="text-muted">Demo HP</span>
+                            </td>
+                        <td>
+                            ring 0
+                            <form method="POST" action="/os/ring/demo-hp-bb76ea">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="ring" value="1"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Make demo-hp-bb76ea a normal (ring 1) box? It then installs only approved releases.">→ normal</button>
+                                
+                            </form><br>
+                            updates <strong>ON</strong>
+                            <form method="POST" action="/os/enabled/demo-hp-bb76ea">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="on" value="0"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Switch OS updates OFF for demo-hp-bb76ea? It keeps reporting and installs nothing.">switch off</button>
+                                
+                            </form>
+                        </td>
+                        <td>running</td>
+                        <td class="grp ">9.2.2</td>
+                        <td>7.0.14-20-pve</td><td>7.0.14-20-pve</td><td>13.7</td>
+                        <td>ring0-20261004T144238Z</td><td>78</td><td>0</td>
+                        <td>none</td><td>no</td><td class="c-warn">0 (stays off)</td><td>no</td>
+                        <td class="c-warn">3</td><td class="c-bad" title="2 unclean boots within 60 minutes — the next crash leaves the box off (limit 3)">TRIPPED 2026-10-04T14:54:43Z</td>
+                        <td class="grp ">13.7</td>
+                        <td>ring0-20261004T144238Z</td><td>6</td><td>1</td>
+                        <td class="grp ">29.8.2</td>
+                        <td>2.3.6-1~debian.13~trixie</td><td>on</td><td>ring0-20261004T144238Z</td>
+                        <td class="grp " title="last successful leg: 50 min ago">48 min ago · applied · 45 s</td>
+                    </tr>
+                    
+                    <tr>
+                        <td><a href="/hosts/drill-r50-0a4f9a">drill-r50-0a4f9a</a><br><span class="text-muted">drill-r50</span>
+                            <br><span class="c-warn" style="font-weight: normal;">no versions reported (agent older than v0.142.0)</span></td>
+                        <td>
+                            ring 1
+                            <form method="POST" action="/os/ring/drill-r50-0a4f9a">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="ring" value="0"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Make drill-r50-0a4f9a a DEMO (ring 0) box? It then installs every new fix first and takes unsigned Docker steps if its root-owned ring-0 mark allows.">→ demo</button>
+                            </form><br>
+                            updates <strong>ON</strong>
+                            <form method="POST" action="/os/enabled/drill-r50-0a4f9a">
+                                <input type="hidden" name="_csrf" value=""><input type="hidden" name="return" value="/system">
+                                <input type="hidden" name="on" value="0"><button type="submit" class="btn btn-sm btn-outline" data-confirm="Switch OS updates OFF for drill-r50-0a4f9a? It keeps reporting and installs nothing.">switch off</button>
+                                
+                            </form>
+                        </td>
+                        <td class="c-bad">inactive</td>
+                        <td class="grp c-warn">unknown</td>
+                        <td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td>
+                        <td>—</td><td>0</td><td>0</td>
+                        <td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td>no</td><td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td>no</td>
+                        <td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td class="c-warn">not installed</td>
+                        <td class="grp c-warn">unknown</td>
+                        <td>—</td><td>0</td><td>0</td>
+                        <td class="grp c-warn">unknown</td>
+                        <td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td class="c-warn" title="the box could not read it (agent older than v0.142.0, or the guest is down)">unknown</td><td>—</td>
+                        <td class="grp " title="last successful leg: never">never · — · 0 s</td>
+                    </tr>
+                    
+                </tbody>
+            </table>
+        </section>
+        <p class="text-muted" style="font-size: 0.85em;">Amber: worth a look. Red: an operator alarm fires (`08` §6.3). "unknown": the box could not read the value — never a guess.</p>
+        
+
+        <footer style="margin-top: 2rem; color: var(--text-muted); font-size: 0.8rem; text-align: center;">
+            Felhom Hub <span style="font-family: var(--font-mono)">0.132.0</span>
+        </footer>
+    </div>
+</body>
+</html>
+

documentation/audits/os-docker-crash-2026-10-04/partC/c6-rearm.txt

@@ -0,0 +1,3 @@
+crash-guard: RE-ARMED by operator (was tripped: True); kernel.panic=10
+kernel.panic = 10
+{'armed': True, 'tripped': False, 'rearmed_at': '2026-10-04T15:33:25Z', 'rearmed_by': 'operator', 'unclean_boots_in_window': 0, 'unclean_boots_24h': 3, 'last_trip': {'at': '2026-10-04T14:54:43Z', 'reason': '2 unclean boots within 60 minutes — the next crash leaves the box off (limit 3)'}}

documentation/audits/os-docker-crash-2026-10-04/partD/copied-by-hand-demo-felhom.txt

@@ -0,0 +1,10 @@
+755 root 41c759e4c195d349 /usr/local/sbin/felhom-os-apply
+755 root f45b99817fe8ff50 /usr/local/sbin/felhom-crash-guard
+644 root 5580ad6cdd51990b /etc/systemd/system/felhom-crash-guard.service
+644 root 93d2c4749fe63ab1 /etc/systemd/system/felhom-crash-guard-check.service
+644 root b214e79e483c339f /etc/systemd/system/felhom-crash-guard-check.timer
+644 root 9b9d305b421f81d2 /etc/felhom/crash-guard.conf
+644 root d17cb8e4456575b1 /etc/felhom/operator-signers
+644 root 338a564cc8392bad /etc/felhom/os-trust.json
+755 root 18acdf69f9c915be /usr/local/bin/felhom-agent-0.142.0-rc1
+kernel.panic = 10

documentation/audits/os-docker-crash-2026-10-04/partD/copied-by-hand-demo-hp.txt

@@ -0,0 +1,33 @@
+Created symlink '/etc/systemd/system/sysinit.target.wants/felhom-crash-guard.service' -> '/etc/systemd/system/felhom-crash-guard.service'.
+Created symlink '/etc/systemd/system/timers.target.wants/felhom-crash-guard-check.timer' -> '/etc/systemd/system/felhom-crash-guard-check.timer'.
+755 root 41c759e4c195d349 /usr/local/sbin/felhom-os-apply
+755 root f45b99817fe8ff50 /usr/local/sbin/felhom-crash-guard
+644 root 5580ad6cdd51990b /etc/systemd/system/felhom-crash-guard.service
+644 root 93d2c4749fe63ab1 /etc/systemd/system/felhom-crash-guard-check.service
+644 root b214e79e483c339f /etc/systemd/system/felhom-crash-guard-check.timer
+644 root 9b9d305b421f81d2 /etc/felhom/crash-guard.conf
+644 root d17cb8e4456575b1 /etc/felhom/operator-signers
+644 root af0890ef7a5778d0 /etc/felhom/os-trust.json
+755 root 18acdf69f9c915be /usr/local/bin/felhom-agent-0.142.0-rc1
+active
+active
+kernel.panic = 10
+{
+  "armed": true,
+  "boot_id": "0defc390-c295-4e9f-aa6e-13d69a1f24c3",
+  "config": {
+    "LIMIT": 3,
+    "PANIC_SECONDS": 10,
+    "REARM_HOURS": 24,
+    "WINDOW_MINUTES": 60
+  },
+  "kernel_panic": 10,
+  "last_boot_at": "2026-10-04T13:39:06Z",
+  "last_boot_unclean": false,
+  "tripped": false,
+  "unclean_boots": [],
+  "unclean_boots_24h": 0,
+  "unclean_boots_in_window": 0,
+  "updated_at": "2026-10-04T14:08:23Z",
+  "version": 1
+}

documentation/audits/os-guest-lane-2026-10-04/partB/redproof.txt

@@ -12,3 +12,7 @@ R10: 4 raise(s) removed -> suite rc=1; failing tests: test_R10_bind_only_in_a_sn
 R11: 9 raise(s) removed -> suite rc=1; failing tests: test_R11_bad_name, test_R11_bad_version_string, test_R11_duplicate; own test(s) failed: YES
 R12: 1 raise(s) removed -> suite rc=1; failing tests: test_R12_host_layer; own test(s) failed: YES
 R13: 1 raise(s) removed -> suite rc=1; failing tests: test_R13_repair_does_not_fix_it; own test(s) failed: YES
+== RP (wrapper conffile): the old wording (every conffile line reported as kept)
+FAIL: test_updated_vs_kept (__main__.Conffiles.test_updated_vs_kept)
+AssertionError: 'os-apply: CONFFILE updated /etc/debian_version (it was not changed locally)' not found in ['os-apply: START release=os-t1 layer=guest:9201 lane=fast mode=apply packages=2', 'os-apply: REPAIR configured=0 fixed=0', 'os-apply: PLAN upgrade=2 already=0 not-installed=0 from-snapshot=0', 'os-apply: CONFFILE kept /etc/debian_version', 'os-apply: CONFFILE kept /etc/ssh/sshd_config', 'os-apply: DONE rc=0 seconds=0.0 upgraded=2']
+FAILED (failures=1)

documentation/audits/os-guest-lane-2026-10-04/partC/agent-leg-redproofs.txt

@@ -33,3 +33,9 @@ FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/localapi	0.563s
 == RP (local API hook): run the leg after a FAILED backup too
         afterbackup_test.go:60: the leg ran after a FAILED backup: [8200]
 FAIL (recorded from the run above)
+== RP (selftest flag): drop the os-update case from selftestFlag.Set
+    selftest_flag_test.go:37: dispatched but refused by --selftest: [os-update]
+FAIL	gitea.dooplex.hu/admin/felhom-agent/cmd/felhom-agent	0.010s
+== RP (leg): baseline = the apply's own before-reading only (the pre-fix rule)
+    leg_test.go:279: an app that stopped during the run passed: {RunID:20261004T040000Z Trigger:night Mode:apply Ring:0 ReleaseID:ring0-20261004T040000Z Outcome:applied Healthy:true HealthReason: VMID:9201 Upgraded:[{Name:libc6 Version: Origin:}] Installed:[] Pending:[] NotCovered:[] RestartNeeded:[] DockerRestartNeeded:false RebootNeeded:false Refused:[]}
+FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/osupdate	0.195s

documentation/audits/os-guest-lane-2026-10-04/partF/tp-demo-felhom.log

@@ -0,0 +1,90 @@
+1791104489.428006469 302
+1791104490.114048259 302
+1791104490.867431883 302
+1791104491.567710242 302
+1791104492.259675188 302
+1791104492.950311346 302
+1791104493.658119300 302
+1791104494.345122536 302
+1791104495.027891977 302
+1791104495.724212868 302
+1791104496.422407509 302
+1791104497.128727616 302
+1791104497.800224130 302
+1791104498.519248669 302
+1791104499.203467396 302
+1791104499.898045533 502
+1791104500.634582730 502
+1791104501.341338498 502
+1791104502.054985899 502
+1791104502.792687510 502
+1791104503.493970117 502
+1791104504.236100842 502
+1791104504.980276912 502
+1791104505.689072396 502
+1791104506.471770326 502
+1791104507.185250940 502
+1791104508.316227102 502
+1791104509.010368824 404
+1791104509.690610573 302
+1791104510.381269521 302
+1791104511.048606319 302
+1791104511.718435404 302
+1791104512.462336250 502
+1791104513.187968905 502
+1791104513.871106880 302
+1791104514.590549839 302
+1791104515.281227748 302
+1791104515.964558604 302
+1791104516.720756559 302
+1791104517.426584135 302
+1791104518.115760697 302
+1791104518.810201702 302
+1791104519.507291973 302
+1791104520.172275341 302
+1791104520.866469463 302
+1791104521.541290853 302
+1791104522.219399888 302
+1791104522.891993819 302
+1791104523.560829339 302
+1791104524.240801665 302
+1791104524.946212414 302
+1791104525.627388820 302
+1791104526.329685610 302
+1791104527.011843100 302
+1791104527.684678463 302
+1791104528.380214494 302
+1791104529.058330821 302
+1791104529.740540446 302
+1791104530.436470651 302
+1791104531.121601771 302
+1791104531.809593181 302
+1791104532.503149505 302
+1791104533.173713227 302
+1791104533.856540265 302
+1791104534.544973430 302
+1791104535.253316994 302
+1791104535.932080467 302
+1791104536.599649531 302
+1791104537.294371521 302
+1791104537.968883762 302
+1791104538.662330307 302
+1791104539.355172110 302
+1791104540.014510174 302
+1791104540.732988090 302
+1791104541.409134391 302
+1791104542.085884593 302
+1791104542.786152655 302
+1791104543.468040923 302
+1791104544.129791932 302
+1791104545.053502785 302
+1791104545.747716347 302
+1791104546.430378570 302
+1791104547.089481515 302
+1791104547.826344317 302
+1791104548.491677889 302
+1791104549.200576500 302
+1791104549.873171847 302
+1791104550.549674984 302
+1791104551.223931502 302
+1791104551.897581518 302

documentation/audits/os-guest-lane-2026-10-04/partF/tp-demo-hp.log

@@ -0,0 +1,93 @@
+1791104489.428473157 302
+1791104490.154456409 302
+1791104490.836711390 302
+1791104491.512992898 302
+1791104492.188921801 302
+1791104492.892582650 302
+1791104493.561025000 302
+1791104494.243801578 302
+1791104494.946931718 302
+1791104495.612770541 302
+1791104496.296371937 302
+1791104496.959464984 302
+1791104497.624069604 302
+1791104498.287369613 302
+1791104498.954218253 302
+1791104499.625720367 302
+1791104500.285823257 302
+1791104500.962562788 302
+1791104501.623569883 302
+1791104502.280153375 502
+1791104502.973783514 502
+1791104503.699803496 502
+1791104504.477656042 502
+1791104505.219746840 502
+1791104505.935741922 502
+1791104506.696658409 502
+1791104507.383980538 502
+1791104508.099312305 502
+1791104508.778478656 502
+1791104509.485583013 502
+1791104510.206577081 502
+1791104510.894142842 502
+1791104511.583745893 502
+1791104512.301754435 502
+1791104512.965882550 502
+1791104513.674358904 502
+1791104514.365515398 302
+1791104515.084994873 302
+1791104515.780146089 302
+1791104516.467673569 302
+1791104517.127579680 302
+1791104517.806574655 302
+1791104518.464909459 302
+1791104519.137104416 302
+1791104519.819783794 502
+1791104520.535296664 530
+1791104521.190614199 302
+1791104521.864351794 302
+1791104522.541227619 302
+1791104523.210468864 302
+1791104523.862415986 302
+1791104524.545740539 302
+1791104525.247498962 302
+1791104525.907280086 302
+1791104526.568936189 302
+1791104527.226485539 302
+1791104527.883240645 302
+1791104528.572125595 302
+1791104529.261611733 302
+1791104529.990339048 302
+1791104530.672950131 302
+1791104531.335651483 302
+1791104532.020457877 302
+1791104532.681731917 302
+1791104533.347279268 302
+1791104533.999334640 302
+1791104534.662664742 302
+1791104535.326620624 302
+1791104535.998590184 302
+1791104536.651396219 302
+1791104537.299040037 302
+1791104537.937915214 302
+1791104538.640295166 302
+1791104539.329815017 302
+1791104539.985113137 302
+1791104540.672365246 302
+1791104541.358812693 302
+1791104542.035139317 302
+1791104542.698601778 302
+1791104543.351804369 302
+1791104544.018269004 302
+1791104544.686076951 302
+1791104545.360390747 302
+1791104546.033087077 302
+1791104546.697562917 302
+1791104547.355845574 302
+1791104548.046312941 302
+1791104548.728344788 302
+1791104549.396103966 302
+1791104550.084478791 302
+1791104550.772128650 302
+1791104551.425494194 302
+1791104552.081996680 302

documentation/audits/os-guest-lane-2026-10-04/partF/tunnel-gap.txt

@@ -0,0 +1,2 @@
+demo-hp (public URL through Cloudflare, 09:01:30-09:06:00Z): samples 92 codes {'302': 73, '502': 18, '530': 1}; first non-302 09:01:42Z, last 09:02:00Z; gap at most 19.6 s
+demo-felhom (public URL through Cloudflare, 09:01:30-09:06:00Z): samples 89 codes {'302': 74, '502': 14, '404': 1}; first non-302 09:01:39Z, last 09:01:53Z; gap at most 14.7 s

documentation/audits/os-guest-lane-2026-10-04/partG/g1-demo-felhom.log

@@ -0,0 +1,306 @@
+=== felhom-agent 0.140.0-rc2 selftest=os-update vmid=9201 ring=0 enabled=true release=false ===
+time=2026-10-04T11:08:52.934+02:00 level=INFO msg="osupdate: START" run=20261004T090852Z vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T090852Z
+time=2026-10-04T11:09:08.241+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T090852Z layer=guest:9201 lane=fast mode=inventory packages=0"
+time=2026-10-04T11:11:46.297+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T090852Z layer=guest:9201 lane=fast mode=apply packages=53"
+time=2026-10-04T11:11:46.297+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T11:11:46.297+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=53 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T11:11:46.297+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: CONFFILE updated /etc/debian_version (it was not changed locally)"
+time=2026-10-04T11:11:46.297+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=18.7 upgraded=53"
+time=2026-10-04T11:11:46.297+02:00 level=INFO msg="osupdate: DONE" run=20261004T090852Z vmid=9201 ring=0 trigger=debug outcome=applied healthy=true reason="" upgraded=53 pending=6 not_covered=6 restart_needed=7
+  --- os-update report ---
+  {
+    "docker_restart_needed": true,
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "docker-ce-cli",
+      "containerd.io",
+      "docker-ce",
+      "docker-buildx-plugin",
+      "docker-ce-rootless-extras",
+      "docker-compose-plugin"
+    ],
+    "outcome": "applied",
+    "pending": 6,
+    "refused": null,
+    "release_id": "ring0-20261004T090852Z",
+    "restart_needed": [
+      "agetty",
+      "containerd",
+      "cron",
+      "dbus-daemon",
+      "dhclient",
+      "sshd",
+      "systemd-logind"
+    ],
+    "ring": 0,
+    "run_id": "20261004T090852Z",
+    "upgraded": [
+      {
+        "name": "libc6",
+        "version": "2.41-12+deb13u4",
+        "origin": ""
+      },
+      {
+        "name": "base-files",
+        "version": "13.8+deb13u7",
+        "origin": ""
+      },
+      {
+        "name": "bash",
+        "version": "5.2.37-2+b10",
+        "origin": ""
+      },
+      {
+        "name": "bsdutils",
+        "version": "1:2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "gzip",
+        "version": "1.13-1+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libperl5.40",
+        "version": "5.40.1-6+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "perl",
+        "version": "5.40.1-6+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "perl-base",
+        "version": "5.40.1-6+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "perl-modules-5.40",
+        "version": "5.40.1-6+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "liblastlog2-2",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "bsdextrautils",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "util-linux-extra",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libblkid1",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libmount1",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libsmartcols1",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "mount",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "fdisk",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libuuid1",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "util-linux",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libfdisk1",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libaudit-common",
+        "version": "1:4.0.2-2+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libaudit1",
+        "version": "1:4.0.2-2+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libsqlite3-0",
+        "version": "3.46.1-7+deb13u2",
+        "origin": ""
+      },
+      {
+        "name": "libc-bin",
+        "version": "2.41-12+deb13u4",
+        "origin": ""
+      },
+      {
+        "name": "login",
+        "version": "1:4.16.0-2+really2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "logsave",
+        "version": "1.47.2-3+b12",
+        "origin": ""
+      },
+      {
+        "name": "libext2fs2t64",
+        "version": "1.47.2-3+b12",
+        "origin": ""
+      },
+      {
+        "name": "e2fsprogs",
+        "version": "1.47.2-3+b12",
+        "origin": ""
+      },
+      {
+        "name": "libexpat1",
+        "version": "2.8.3-1~deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "openssl-provider-legacy",
+        "version": "3.5.7-1~deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "libssl3t64",
+        "version": "3.5.7-1~deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "postfix",
+        "version": "3.10.13-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "python3.13",
+        "version": "3.13.5-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "libpython3.13-stdlib",
+        "version": "3.13.5-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "python3.13-minimal",
+        "version": "3.13.5-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "libpython3.13-minimal",
+        "version": "3.13.5-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "tzdata",
+        "version": "2026c-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libcap2",
+        "version": "1:2.75-10+deb13u1+b3",
+        "origin": ""
+      },
+      {
+        "name": "libpcre2-8-0",
+        "version": "10.46-1~deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "dhcpcd-base",
+        "version": "1:10.1.0-11+deb13u4",
+        "origin": ""
+      },
+      {
+        "name": "bind9-dnsutils",
+        "version": "1:9.20.29-1~deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "bind9-host",
+        "version": "1:9.20.29-1~deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "bind9-libs",
+        "version": "1:9.20.29-1~deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libc-l10n",
+        "version": "2.41-12+deb13u4",
+        "origin": ""
+      },
+      {
+        "name": "locales",
+        "version": "2.41-12+deb13u4",
+        "origin": ""
+      },
+      {
+        "name": "libssh2-1t64",
+        "version": "1.11.1-1+deb13u2",
+        "origin": ""
+      },
+      {
+        "name": "curl",
+        "version": "8.14.1-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "libcurl4t64",
+        "version": "8.14.1-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "libcap2-bin",
+        "version": "1:2.75-10+deb13u1+b3",
+        "origin": ""
+      },
+      {
+        "name": "libcom-err2",
+        "version": "1.47.2-3+b12",
+        "origin": ""
+      },
+      {
+        "name": "libcurl3t64-gnutls",
+        "version": "8.14.1-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "libss2",
+        "version": "1.47.2-3+b12",
+        "origin": ""
+      },
+      {
+        "name": "openssl",
+        "version": "3.5.7-1~deb13u3",
+        "origin": ""
+      }
+    ]
+  }

documentation/audits/os-guest-lane-2026-10-04/partG/g1-demo-hp.log

@@ -0,0 +1,306 @@
+=== felhom-agent 0.140.0-rc2 selftest=os-update vmid=9201 ring=0 enabled=true release=false ===
+time=2026-10-04T11:04:21.448+02:00 level=INFO msg="osupdate: START" run=20261004T090421Z vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T090421Z
+time=2026-10-04T11:04:42.234+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T090421Z layer=guest:9201 lane=fast mode=inventory packages=0"
+time=2026-10-04T11:08:07.757+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T090421Z layer=guest:9201 lane=fast mode=apply packages=53"
+time=2026-10-04T11:08:07.757+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T11:08:07.757+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=53 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T11:08:07.757+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: CONFFILE kept /etc/debian_version"
+time=2026-10-04T11:08:07.757+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=31.7 upgraded=53"
+time=2026-10-04T11:08:07.758+02:00 level=INFO msg="osupdate: DONE" run=20261004T090421Z vmid=9201 ring=0 trigger=debug outcome=applied healthy=true reason="" upgraded=53 pending=6 not_covered=6 restart_needed=7
+  --- os-update report ---
+  {
+    "docker_restart_needed": true,
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "docker-ce-cli",
+      "containerd.io",
+      "docker-ce",
+      "docker-buildx-plugin",
+      "docker-ce-rootless-extras",
+      "docker-compose-plugin"
+    ],
+    "outcome": "applied",
+    "pending": 6,
+    "refused": null,
+    "release_id": "ring0-20261004T090421Z",
+    "restart_needed": [
+      "agetty",
+      "containerd",
+      "cron",
+      "dbus-daemon",
+      "dhclient",
+      "sshd",
+      "systemd-logind"
+    ],
+    "ring": 0,
+    "run_id": "20261004T090421Z",
+    "upgraded": [
+      {
+        "name": "libc6",
+        "version": "2.41-12+deb13u4",
+        "origin": ""
+      },
+      {
+        "name": "base-files",
+        "version": "13.8+deb13u7",
+        "origin": ""
+      },
+      {
+        "name": "bash",
+        "version": "5.2.37-2+b10",
+        "origin": ""
+      },
+      {
+        "name": "bsdutils",
+        "version": "1:2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "gzip",
+        "version": "1.13-1+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libperl5.40",
+        "version": "5.40.1-6+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "perl",
+        "version": "5.40.1-6+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "perl-base",
+        "version": "5.40.1-6+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "perl-modules-5.40",
+        "version": "5.40.1-6+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "liblastlog2-2",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "bsdextrautils",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "util-linux-extra",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libblkid1",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libmount1",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libsmartcols1",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "mount",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "fdisk",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libuuid1",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "util-linux",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libfdisk1",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libaudit-common",
+        "version": "1:4.0.2-2+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libaudit1",
+        "version": "1:4.0.2-2+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libsqlite3-0",
+        "version": "3.46.1-7+deb13u2",
+        "origin": ""
+      },
+      {
+        "name": "libc-bin",
+        "version": "2.41-12+deb13u4",
+        "origin": ""
+      },
+      {
+        "name": "login",
+        "version": "1:4.16.0-2+really2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "logsave",
+        "version": "1.47.2-3+b12",
+        "origin": ""
+      },
+      {
+        "name": "libext2fs2t64",
+        "version": "1.47.2-3+b12",
+        "origin": ""
+      },
+      {
+        "name": "e2fsprogs",
+        "version": "1.47.2-3+b12",
+        "origin": ""
+      },
+      {
+        "name": "libexpat1",
+        "version": "2.8.3-1~deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "openssl-provider-legacy",
+        "version": "3.5.7-1~deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "libssl3t64",
+        "version": "3.5.7-1~deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "postfix",
+        "version": "3.10.13-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "python3.13",
+        "version": "3.13.5-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "libpython3.13-stdlib",
+        "version": "3.13.5-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "python3.13-minimal",
+        "version": "3.13.5-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "libpython3.13-minimal",
+        "version": "3.13.5-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "tzdata",
+        "version": "2026c-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libcap2",
+        "version": "1:2.75-10+deb13u1+b3",
+        "origin": ""
+      },
+      {
+        "name": "libpcre2-8-0",
+        "version": "10.46-1~deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "dhcpcd-base",
+        "version": "1:10.1.0-11+deb13u4",
+        "origin": ""
+      },
+      {
+        "name": "bind9-dnsutils",
+        "version": "1:9.20.29-1~deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "bind9-host",
+        "version": "1:9.20.29-1~deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "bind9-libs",
+        "version": "1:9.20.29-1~deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libc-l10n",
+        "version": "2.41-12+deb13u4",
+        "origin": ""
+      },
+      {
+        "name": "locales",
+        "version": "2.41-12+deb13u4",
+        "origin": ""
+      },
+      {
+        "name": "libssh2-1t64",
+        "version": "1.11.1-1+deb13u2",
+        "origin": ""
+      },
+      {
+        "name": "curl",
+        "version": "8.14.1-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "libcurl4t64",
+        "version": "8.14.1-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "libcap2-bin",
+        "version": "1:2.75-10+deb13u1+b3",
+        "origin": ""
+      },
+      {
+        "name": "libcom-err2",
+        "version": "1.47.2-3+b12",
+        "origin": ""
+      },
+      {
+        "name": "libcurl3t64-gnutls",
+        "version": "8.14.1-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "libss2",
+        "version": "1.47.2-3+b12",
+        "origin": ""
+      },
+      {
+        "name": "openssl",
+        "version": "3.5.7-1~deb13u3",
+        "origin": ""
+      }
+    ]
+  }

documentation/audits/os-guest-lane-2026-10-04/partG/g3-ring1.log

@@ -0,0 +1,58 @@
+=== felhom-agent 0.140.0-rc2 selftest=os-update vmid=9201 ring=1 enabled=true release=true ===
+time=2026-10-04T11:15:59.945+02:00 level=INFO msg="osupdate: START" run=20261004T091559Z vmid=9201 ring=1 trigger=debug enabled=true release=os-20261004-091417
+time=2026-10-04T11:16:10.634+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=os-20261004-091417 layer=guest:9201 lane=fast mode=inventory packages=0"
+time=2026-10-04T11:20:04.641+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=os-20261004-091417 layer=guest:9201 lane=fast mode=apply packages=272"
+time=2026-10-04T11:20:04.641+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T11:20:04.641+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=3 already=269 not-installed=0 from-snapshot=0"
+time=2026-10-04T11:20:04.641+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=9.3 upgraded=3"
+time=2026-10-04T11:20:04.642+02:00 level=INFO msg="osupdate: DONE" run=20261004T091559Z vmid=9201 ring=1 trigger=debug outcome=applied healthy=true reason="" upgraded=3 pending=6 not_covered=6 restart_needed=10
+  --- os-update report ---
+  {
+    "docker_restart_needed": true,
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "docker-ce-cli",
+      "containerd.io",
+      "docker-ce",
+      "docker-buildx-plugin",
+      "docker-ce-rootless-extras",
+      "docker-compose-plugin"
+    ],
+    "outcome": "applied",
+    "pending": 6,
+    "refused": null,
+    "release_id": "os-20261004-091417",
+    "restart_needed": [
+      "agetty",
+      "containerd",
+      "cron",
+      "dbus-daemon",
+      "dhclient",
+      "sshd",
+      "systemd",
+      "systemd-journal",
+      "systemd-logind",
+      "systemd-network"
+    ],
+    "ring": 1,
+    "run_id": "20261004T091559Z",
+    "upgraded": [
+      {
+        "name": "libssl3t64",
+        "version": "3.5.7-1~deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "openssl",
+        "version": "3.5.7-1~deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "openssl-provider-legacy",
+        "version": "3.5.7-1~deb13u3",
+        "origin": ""
+      }
+    ]
+  }

documentation/audits/os-guest-lane-2026-10-04/partG/g4-healthfail.log

@@ -0,0 +1,58 @@
+=== felhom-agent 0.140.0-rc3 selftest=os-update vmid=9201 ring=0 enabled=true release=false ===
+time=2026-10-04T11:22:44.828+02:00 level=INFO msg="osupdate: START" run=20261004T092244Z vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T092244Z
+time=2026-10-04T11:22:59.130+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T092244Z layer=guest:9201 lane=fast mode=inventory packages=0"
+time=2026-10-04T11:23:37.731+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T092244Z layer=guest:9201 lane=fast mode=apply packages=3"
+time=2026-10-04T11:23:37.731+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T11:23:37.731+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=3 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T11:23:37.731+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=3.8 upgraded=3"
+time=2026-10-04T11:28:41.271+02:00 level=INFO msg="osupdate: DONE" run=20261004T092244Z vmid=9201 ring=0 trigger=debug outcome=health_failed healthy=false reason="privatebin was running and is not" upgraded=3 pending=6 not_covered=6 restart_needed=10
+  --- os-update report ---
+  {
+    "docker_restart_needed": true,
+    "health_reason": "privatebin was running and is not",
+    "healthy": false,
+    "mode": "apply",
+    "not_covered": [
+      "docker-ce-cli",
+      "containerd.io",
+      "docker-ce",
+      "docker-buildx-plugin",
+      "docker-ce-rootless-extras",
+      "docker-compose-plugin"
+    ],
+    "outcome": "health_failed",
+    "pending": 6,
+    "refused": null,
+    "release_id": "ring0-20261004T092244Z",
+    "restart_needed": [
+      "agetty",
+      "containerd",
+      "cron",
+      "dbus-daemon",
+      "dhclient",
+      "sshd",
+      "systemd",
+      "systemd-journal",
+      "systemd-logind",
+      "systemd-network"
+    ],
+    "ring": 0,
+    "run_id": "20261004T092244Z",
+    "upgraded": [
+      {
+        "name": "openssl-provider-legacy",
+        "version": "3.5.7-1~deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "libssl3t64",
+        "version": "3.5.7-1~deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "openssl",
+        "version": "3.5.7-1~deb13u3",
+        "origin": ""
+      }
+    ]
+  }

documentation/audits/os-guest-lane-2026-10-04/partG/g5-final.log

@@ -0,0 +1,38 @@
+=== felhom-agent 0.140.0 selftest=os-update vmid=9201 ring=0 enabled=true release=false ===
+time=2026-10-04T11:38:36.515+02:00 level=INFO msg="osupdate: START" run=20261004T093836Z vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T093836Z
+time=2026-10-04T11:38:50.533+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T093836Z layer=guest:9201 lane=fast mode=inventory packages=0"
+time=2026-10-04T11:38:50.534+02:00 level=INFO msg="osupdate: DONE" run=20261004T093836Z vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=6 not_covered=6 restart_needed=10
+  --- os-update report ---
+  {
+    "docker_restart_needed": true,
+    "health_reason": "",
+    "healthy": true,
+    "mode": "inventory",
+    "not_covered": [
+      "docker-ce-cli",
+      "containerd.io",
+      "docker-ce",
+      "docker-buildx-plugin",
+      "docker-ce-rootless-extras",
+      "docker-compose-plugin"
+    ],
+    "outcome": "nothing",
+    "pending": 6,
+    "refused": null,
+    "release_id": "ring0-20261004T093836Z",
+    "restart_needed": [
+      "agetty",
+      "containerd",
+      "cron",
+      "dbus-daemon",
+      "dhclient",
+      "sshd",
+      "systemd",
+      "systemd-journal",
+      "systemd-logind",
+      "systemd-network"
+    ],
+    "ring": 0,
+    "run_id": "20261004T093836Z",
+    "upgraded": null
+  }

documentation/audits/os-guest-lane-2026-10-04/partG/hub-customer-timeline-demo-hp.txt

@@ -0,0 +1,6 @@
+os_update_applied|System security fixes installed (3 package(s)).|hub|Oct 04 09:28
+os_update_health_failed|OS update on demo-hp-bb76ea: the guest was NOT healthy after 3 package(s) were installed (privatebin was running and is not). Nothing was undone automatically (no guest snapshot is possible, R-837); last night's whole-guest backup is th
+os_update_applied|System security fixes installed (3 package(s)).|hub|Oct 04 09:21
+os_update_applied|System security fixes installed (53 package(s)).|hub|Oct 04 09:03
+sent|OS update on demo-hp-bb76ea: the guest was NOT healthy after 3 package(s) were installed (privatebin was running and is not). Nothing was undone automatically (no guest snapshot is possible, R-837); last night's whole-guest backup is the undo.|Oct 04 09:2
+skipped|OS update on demo-hp-bb76ea: the guest was NOT healthy after 3 package(s) were installed (privatebin was running and is not). Nothing was undone automatically (no guest snapshot is possible, R-837); last night's whole-guest backup is the undo.|Oct 04 0

documentation/audits/os-host-lane-2026-10-04/partA/agent-redproofs.txt

@@ -0,0 +1,9 @@
+# agent v0.141.0 tunnel-probe red-proofs (each mutation compiles; result line printed)
+== mutation: container-state-only
+    cloudflared_test.go:32: running but not connected: got "running" (container running but the tunnel is NOT connected (cloudflared /ready fails)), want "not_running" (…NOT connected…)
+FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/hub	0.009s
+== mutation: unknown-as-down
+    cloudflared_test.go:32: guest not running: got "not_running" (could not ask the guest: CT 9201 not running), want "unknown" (…could not ask…)
+    cloudflared_test.go:32: sudo refused: got "not_running" (could not ask the guest: sudo: a password is required), want "unknown" (…could not ask…)
+FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/hub	0.009s
+ok  	gitea.dooplex.hu/admin/felhom-agent/internal/hub	(cached)

documentation/audits/os-host-lane-2026-10-04/partA/controller-redproof.txt

@@ -0,0 +1,3 @@
+== RP (controller): the healthcheck block removed from the cloudflared template
+    infra_tunnel_test.go:300: tunnel=false: compose lacks "test: [\"CMD\", \"cloudflared\", \"tunnel\", \"--metrics\", \"localhost:20241\", \"ready\"]":
+FAIL	gitea.dooplex.hu/admin/felhom-controller/internal/stacks	0.007s

documentation/audits/os-host-lane-2026-10-04/partA/hub-redproofs.txt

@@ -0,0 +1,17 @@
+# hub v0.131.0 tunnel-alarm red-proofs, 2026-10-04 (each mutation compiles; the package result line is printed)
+== mutation: one-report
+    --- FAIL: TestTunnelAlarm_Sequences/a_single_bad_report_does_not_alarm (0.02s)
+        tunnel_test.go:47: tunnel_down=1 tunnel_recovered=0, want 0 / 0
+    --- FAIL: TestTunnelAlarm_Sequences/unknown_never_alarms_and_breaks_the_run (0.03s)
+        tunnel_test.go:47: tunnel_down=1 tunnel_recovered=0, want 0 / 0
+== mutation: unknown-counts-as-down
+    --- FAIL: TestTunnelAlarm_Sequences/unknown_never_alarms_and_breaks_the_run (0.03s)
+        tunnel_test.go:47: tunnel_down=1 tunnel_recovered=0, want 0 / 0
+FAIL	gitea.dooplex.hu/admin/felhom-hub/internal/api	0.164s
+== mutation: no-alarm
+    --- FAIL: TestTunnelAlarm_Sequences/two_in_a_row_alarm_once (0.03s)
+        tunnel_test.go:47: tunnel_down=0 tunnel_recovered=0, want 1 / 0
+    --- FAIL: TestTunnelAlarm_Sequences/recovery_after_an_alarm (0.04s)
+        tunnel_test.go:47: tunnel_down=0 tunnel_recovered=1, want 1 / 1
+== reverted
+ok  	gitea.dooplex.hu/admin/felhom-hub/internal/api	0.168s

documentation/audits/os-host-lane-2026-10-04/partA/live/demo-hp-block-states.txt

@@ -0,0 +1,19 @@
+11:34:43 container=running/healthy hub=running alarm=none
+11:35:45 container=running/healthy hub=running alarm=none
+11:36:47 container=running/unhealthy hub=running alarm=none
+11:37:49 container=running/unhealthy hub=not_running alarm=none
+11:38:51 container=running/unhealthy hub=not_running alarm=none
+11:39:53 container=running/unhealthy hub=not_running alarm=none
+11:40:55 container=running/unhealthy hub=not_running alarm=none
+11:41:57 container=running/unhealthy hub=not_running alarm=none
+11:42:59 container=running/unhealthy hub=not_running alarm=none
+11:44:01 container=running/unhealthy hub=not_running alarm=none
+11:45:03 container=running/unhealthy hub=not_running alarm=none
+11:46:05 container=running/unhealthy hub=not_running alarm=none
+11:47:07 container=running/unhealthy hub=not_running alarm=none
+11:48:09 container=running/unhealthy hub=not_running alarm=none
+11:49:10 container=running/unhealthy hub=not_running alarm=none
+11:50:12 container=running/unhealthy hub=not_running alarm=none
+11:51:14 container=running/unhealthy hub=not_running alarm=none
+11:52:16 container=running/unhealthy hub=not_running alarm=none
+11:53:18 container=running/unhealthy hub=not_running alarm=2026/10/04 13:52:29 [WARN] host demo-hp-bb76ea tunnel: tunnel_down (container running but the tunnel is NOT connected (cloudflared /ready fails))

documentation/audits/os-host-lane-2026-10-04/partA/live/demo-hp-block.txt

@@ -0,0 +1,8 @@
+block at 2026-10-04T11:34:33Z: DOCKER-USER DROP tcp+udp dport 7844 in guest 9201 on demo-hp
+Chain DOCKER-USER (1 references)
+target     prot opt source               destination         
+DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:7844
+DROP       udp  --  0.0.0.0/0            0.0.0.0/0            udp dpt:7844
+unblock at 2026-10-04T11:53:29Z
+Chain DOCKER-USER (1 references)
+target     prot opt source               destination         

documentation/audits/os-host-lane-2026-10-04/partA/live/demo-hp-recover-states.txt

@@ -0,0 +1,14 @@
+11:53:39 container=unhealthy hub=not_running
+11:54:41 container=healthy hub=not_running
+11:55:43 container=healthy hub=not_running
+11:56:45 container=healthy hub=not_running
+11:57:47 container=healthy hub=not_running
+11:58:48 container=healthy hub=not_running
+11:59:50 container=healthy hub=not_running
+12:00:52 container=healthy hub=not_running
+12:01:54 container=healthy hub=not_running
+12:02:56 container=healthy hub=not_running
+12:03:58 container=healthy hub=not_running
+12:05:00 container=healthy hub=not_running
+12:06:02 container=healthy hub=not_running
+12:07:04 container=healthy hub=not_running

documentation/audits/os-host-lane-2026-10-04/partA/live/demo-hp-states.txt

@@ -0,0 +1,22 @@
+11:23:25 running
+11:23:56 running
+11:24:26 running
+11:24:57 running
+11:25:27 running
+11:25:58 running
+11:26:28 running
+11:26:59 running
+11:27:29 running
+11:28:00 running
+11:28:31 running
+11:29:01 running
+11:29:32 running
+11:30:02 running
+11:30:33 running
+11:31:03 running
+11:31:34 running
+11:32:04 running
+11:32:35 running
+11:33:05 running
+11:33:36 running
+11:34:06 running

documentation/audits/os-host-lane-2026-10-04/partA/live/demo-hp-stop.txt

@@ -0,0 +1,3 @@
+stop at 2026-10-04T11:23:17Z
+cloudflared
+exited 0

documentation/audits/os-host-lane-2026-10-04/partA/live/hub-log-tunnel_down.txt

@@ -0,0 +1,5 @@
+2026/10/04 13:52:29 [WARN] host demo-hp-bb76ea tunnel: tunnel_down (container running but the tunnel is NOT connected (cloudflared /ready fails))
+2026/10/04 13:52:30 [INFO] Operator email sent for demo-hp/tunnel_down
+2026/10/04 13:52:29 [WARN] host demo-hp-bb76ea tunnel: tunnel_down (container running but the tunnel is NOT connected (cloudflared /ready fails))
+2026/10/04 13:52:30 [INFO] Operator email sent for demo-hp/tunnel_down
+2026/10/04 14:07:29 [WARN] host demo-hp-bb76ea tunnel: tunnel_recovered (connected)

documentation/audits/os-host-lane-2026-10-04/partB/agent-leg-redproofs.txt

@@ -0,0 +1,22 @@
+# agent v0.141.0 leg red-proofs (each mutation compiles; result line printed)
+== mutation: byo-gets-host
+--- FAIL: TestBYO_NoHostPlan (0.00s)
+FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/osupdate	0.440s
+== mutation: host-after-failed-guest
+--- FAIL: TestGuestFailure_SkipsTheHost (0.00s)
+--- FAIL: TestHealth_FailsAfterTheWait (0.00s)
+FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/osupdate	0.420s
+== mutation: host-ignores-services
+--- FAIL: TestHostHealthVerdict (0.00s)
+FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/osupdate	0.441s
+== mutation: host-ignores-guest
+--- FAIL: TestHostHealthVerdict (0.00s)
+FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/osupdate	0.410s
+== mutation: host-ignores-tunnel
+--- FAIL: TestHost_TunnelDownFailsTheHostStep (0.00s)
+--- FAIL: TestHostHealthVerdict (0.00s)
+FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/osupdate	0.427s
+== mutation: host-uses-guest-release
+--- FAIL: TestRing1_EachLayerItsOwnRelease (0.00s)
+FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/osupdate	0.461s
+ok  	gitea.dooplex.hu/admin/felhom-agent/internal/osupdate	(cached)

documentation/audits/os-host-lane-2026-10-04/partB/live-defects-redproofs.txt

@@ -0,0 +1,36 @@
+# Live-found defects in agent 0.141.0 / hub 0.131.0 and their red-proofs, 2026-10-04T11:39:07Z
+
+## 1. the host restart scan hid lxc-start (cgroup 0::/lxc.monitor/9201 matched the skip pattern "lxc")
+live: demo-felhom after the 108-package host pass: lxc-start had 20 deleted maps, report said reboot_needed=false
+mutation: RESTART_SKIP_CGROUP host ':/lxc/' -> 'lxc'
+    self.assertFalse(self.grep(host, "0::/lxc.monitor/9201"), "lxc-start must NOT be skipped (it runs the guest)")
+    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+AssertionError: True is not false : lxc-start must NOT be skipped (it runs the guest)
+
+----------------------------------------------------------------------
+Ran 1 test in 0.005s
+
+FAILED (failures=1)
+
+## 2. the host scanned only after an install, so a reboot never cleared 'reboot needed'
+mutation: wrapper 'if self.layer == "host" and ...' -> 'if False and ...'
+----------------------------------------------------------------------
+Ran 1 test in 0.002s
+
+FAILED (failures=1)
+
+mutation: agent leg drops rep.RebootScanned copy
+=== RUN   TestHostReport_CarriesRebootScanned
+    leg_test.go:367: hub got [{RunID:20261004T040000Z Layer:guest Trigger:night Mode:apply Ring:0 ReleaseID:ring0-20261004T040000Z Outcome:nothing Healthy:true HealthReason: VMID:9201 Upgraded:[] Inst
+--- FAIL: TestHostReport_CarriesRebootScanned (0.00s)
+FAIL
+FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/osupdate	0.008s
+FAIL
+
+mutation: hub layerLine predicate ignores RebootScanned
+=== RUN   TestRebootNeeded_ClearedByAScannedPass
+    alarms_test.go:323: a scanned pass after the reboot did not clear it: 2026-10-04 12:00:00 +0000 UTC
+--- FAIL: TestRebootNeeded_ClearedByAScannedPass (0.03s)
+FAIL
+FAIL	gitea.dooplex.hu/admin/felhom-hub/internal/osupdates	0.037s
+FAIL

documentation/audits/os-host-lane-2026-10-04/partB/live/b1-demo-felhom-ring0.log

@@ -0,0 +1,716 @@
+=== felhom-agent 0.141.0 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true ===
+time=2026-10-04T13:34:48.263+02:00 level=INFO msg="osupdate: START" run=20261004T113448Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T113448Z
+time=2026-10-04T13:34:59.917+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T113448Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T13:34:59.917+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T13:34:59.917+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T13:34:59.917+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T13:34:59.917+02:00 level=INFO msg="osupdate: DONE" run=20261004T113448Z layer=guest vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=6 not_covered=6 restart_needed=0 reboot_needed=false wrapper_seconds=11.6
+time=2026-10-04T13:34:59.927+02:00 level=INFO msg="osupdate: START" run=20261004T113448Z layer=host vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T113448Z
+time=2026-10-04T13:35:57.810+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T113448Z layer=host lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T13:35:57.810+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T13:35:57.810+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=108 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T13:35:57.810+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: CONFFILE updated /etc/debian_version (it was not changed locally)"
+time=2026-10-04T13:35:57.810+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=41.1 upgraded=108 restart-needed=(sd-pam),agetty,bash,blkmapd,chronyd,cron,dbus-daemon,dmeventd,felhom-agent,felhom-os-apply,ksmtuned,pmxcfs,proxmox-firewal,pve-firewall,pve-ha-crm,pve-ha-lrm,pvedaemon,pvedaemon worke,pvefw-logger,pveproxy,pveproxy worker,pvescheduler,pvestatd,qmeventd,rpcbind,rrdcached,smartd,spiceproxy,spiceproxy work,sshd,sshd-session,sudo,systemd-logind,systemd-udevd,watchdog-mux,zed reboot-needed=no"
+time=2026-10-04T13:35:58.613+02:00 level=INFO msg="osupdate: DONE" run=20261004T113448Z layer=host vmid=9201 ring=0 trigger=debug outcome=applied healthy=true reason="" upgraded=108 pending=80 not_covered=80 restart_needed=36 reboot_needed=false wrapper_seconds=57.8
+  --- os-update report (guest) ---
+  {
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "docker-ce-cli",
+      "containerd.io",
+      "docker-ce",
+      "docker-buildx-plugin",
+      "docker-ce-rootless-extras",
+      "docker-compose-plugin"
+    ],
+    "outcome": "nothing",
+    "pending": 6,
+    "reboot_needed": false,
+    "refused": null,
+    "release_id": "ring0-20261004T113448Z",
+    "restart_needed": null,
+    "ring": 0,
+    "run_id": "20261004T113448Z",
+    "upgraded": [],
+    "wrapper_seconds": 11.6
+  }
+  --- os-update report (host) ---
+  {
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "frr",
+      "shim-signed-common",
+      "shim-unsigned",
+      "shim-helpers-amd64-signed",
+      "shim-signed",
+      "libradosstriper1",
+      "librgw2",
+      "ceph-common",
+      "librbd1",
+      "librados2",
+      "python3-cephfs",
+      "libcephfs2",
+      "python3-rgw",
+      "python3-rados",
+      "python3-ceph-argparse",
+      "python3-ceph-common",
+      "python3-rbd",
+      "ceph-fuse",
+      "chrony",
+      "libcorosync-common4",
+      "libcfg7",
+      "libcmap4",
+      "libcpg4",
+      "libknet1t64",
+      "libnozzle1t64",
+      "libquorum5",
+      "libvotequorum8",
+      "corosync",
+      "frr-pythontools",
+      "libjs-extjs",
+      "libnvpair3linux",
+      "libproxmox-acme-plugins",
+      "libproxmox-backup-qemu0",
+      "pve-qemu-kvm",
+      "libpve-notify-perl",
+      "libpve-cluster-api-perl",
+      "libpve-cluster-perl",
+      "pve-cluster",
+      "libpve-access-control",
+      "libpve-apiclient-perl",
+      "librados2-perl",
+      "proxmox-backup-client",
+      "proxmox-backup-file-restore",
+      "pve-manager",
+      "libproxmox-acme-perl",
+      "libpve-common-perl",
+      "libpve-guest-common-perl",
+      "qemu-server",
+      "libpve-storage-perl",
+      "pve-edk2-firmware-legacy",
+      "pve-edk2-firmware-ovmf",
+      "libpve-network-api-perl",
+      "libpve-network-perl",
+      "proxmox-firewall-data",
+      "pve-firewall",
+      "pve-container",
+      "pve-ha-manager",
+      "novnc-pve",
+      "proxmox-enterprise-support-keyring",
+      "proxmox-mini-journalreader",
+      "proxmox-widget-toolkit",
+      "pve-docs",
+      "pve-i18n",
+      "pve-xtermjs",
+      "pve-yew-mobile-i18n",
+      "pve-yew-mobile-gui",
+      "libuutil3linux",
+      "libzfs7linux",
+      "libzpool7linux",
+      "proxmox-first-boot",
+      "pve-firmware",
+      "proxmox-kernel-7.0.14-20-pve-signed",
+      "proxmox-kernel-7.0",
+      "proxmox-kernel-helper",
+      "pve-edk2-firmware-aarch64",
+      "pve-edk2-firmware",
+      "zfs-initramfs",
+      "zfsutils-linux",
+      "zfs-zed",
+      "tailscale"
+    ],
+    "outcome": "applied",
+    "pending": 80,
+    "reboot_needed": false,
+    "refused": null,
+    "release_id": "ring0-20261004T113448Z",
+    "restart_needed": [
+      "(sd-pam)",
+      "agetty",
+      "bash",
+      "blkmapd",
+      "chronyd",
+      "cron",
+      "dbus-daemon",
+      "dmeventd",
+      "felhom-agent",
+      "felhom-os-apply",
+      "ksmtuned",
+      "pmxcfs",
+      "proxmox-firewal",
+      "pve-firewall",
+      "pve-ha-crm",
+      "pve-ha-lrm",
+      "pvedaemon",
+      "pvedaemon worke",
+      "pvefw-logger",
+      "pveproxy",
+      "pveproxy worker",
+      "pvescheduler",
+      "pvestatd",
+      "qmeventd",
+      "rpcbind",
+      "rrdcached",
+      "smartd",
+      "spiceproxy",
+      "spiceproxy work",
+      "sshd",
+      "sshd-session",
+      "sudo",
+      "systemd-logind",
+      "systemd-udevd",
+      "watchdog-mux",
+      "zed"
+    ],
+    "ring": 0,
+    "run_id": "20261004T113448Z",
+    "upgraded": [
+      {
+        "name": "libc6",
+        "version": "2.41-12+deb13u4",
+        "origin": ""
+      },
+      {
+        "name": "base-files",
+        "version": "13.8+deb13u7",
+        "origin": ""
+      },
+      {
+        "name": "bash",
+        "version": "5.2.37-2+b10",
+        "origin": ""
+      },
+      {
+        "name": "bsdutils",
+        "version": "1:2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "gzip",
+        "version": "1.13-1+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libperl5.40",
+        "version": "5.40.1-6+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "perl",
+        "version": "5.40.1-6+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "perl-base",
+        "version": "5.40.1-6+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "perl-modules-5.40",
+        "version": "5.40.1-6+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "liblastlog2-2",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "eject",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "bsdextrautils",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "util-linux-extra",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "fdisk",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libsmartcols1",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libblkid1",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libmount1",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "mount",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libuuid1",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "util-linux",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libfdisk1",
+        "version": "2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libaudit-common",
+        "version": "1:4.0.2-2+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libaudit1",
+        "version": "1:4.0.2-2+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "sqlite3",
+        "version": "3.46.1-7+deb13u2",
+        "origin": ""
+      },
+      {
+        "name": "libsqlite3-0",
+        "version": "3.46.1-7+deb13u2",
+        "origin": ""
+      },
+      {
+        "name": "libc-bin",
+        "version": "2.41-12+deb13u4",
+        "origin": ""
+      },
+      {
+        "name": "login",
+        "version": "1:4.16.0-2+really2.41.5-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "logsave",
+        "version": "1.47.2-3+b12",
+        "origin": ""
+      },
+      {
+        "name": "libext2fs2t64",
+        "version": "1.47.2-3+b12",
+        "origin": ""
+      },
+      {
+        "name": "e2fsprogs",
+        "version": "1.47.2-3+b12",
+        "origin": ""
+      },
+      {
+        "name": "dnsmasq-base",
+        "version": "2.91-1+deb13u2",
+        "origin": ""
+      },
+      {
+        "name": "dnsmasq",
+        "version": "2.91-1+deb13u2",
+        "origin": ""
+      },
+      {
+        "name": "libcap2",
+        "version": "1:2.75-10+deb13u1+b3",
+        "origin": ""
+      },
+      {
+        "name": "libpcre2-8-0",
+        "version": "10.46-1~deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "libpcre2-posix3",
+        "version": "10.46-1~deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "libexpat1",
+        "version": "2.8.3-1~deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "openssl-provider-legacy",
+        "version": "3.5.7-1~deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "libssl3t64",
+        "version": "3.5.7-1~deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "postfix",
+        "version": "3.10.13-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "python3.13",
+        "version": "3.13.5-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "libpython3.13-stdlib",
+        "version": "3.13.5-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "python3.13-minimal",
+        "version": "3.13.5-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "libpython3.13-minimal",
+        "version": "3.13.5-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "tzdata",
+        "version": "2026c-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "liblzma5",
+        "version": "5.8.1-1+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "rsync",
+        "version": "3.5.0+ds1-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "openssl",
+        "version": "3.5.7-1~deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "dhcpcd-base",
+        "version": "1:10.1.0-11+deb13u4",
+        "origin": ""
+      },
+      {
+        "name": "libgssapi-krb5-2",
+        "version": "1.21.3-5+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libkrb5-3",
+        "version": "1.21.3-5+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libkrb5support0",
+        "version": "1.21.3-5+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libk5crypto3",
+        "version": "1.21.3-5+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libcom-err2",
+        "version": "1.47.2-3+b12",
+        "origin": ""
+      },
+      {
+        "name": "libxml2",
+        "version": "2.12.7+dfsg+really2.9.14-2.1+deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "bind9-dnsutils",
+        "version": "1:9.20.29-1~deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "bind9-host",
+        "version": "1:9.20.29-1~deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "bind9-libs",
+        "version": "1:9.20.29-1~deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "krb5-locales",
+        "version": "1.21.3-5+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libc-l10n",
+        "version": "2.41-12+deb13u4",
+        "origin": ""
+      },
+      {
+        "name": "locales",
+        "version": "2.41-12+deb13u4",
+        "origin": ""
+      },
+      {
+        "name": "xz-utils",
+        "version": "5.8.1-1+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "busybox",
+        "version": "1:1.37.0-6+b9",
+        "origin": ""
+      },
+      {
+        "name": "curl",
+        "version": "8.14.1-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "libssh2-1t64",
+        "version": "1.11.1-1+deb13u2",
+        "origin": ""
+      },
+      {
+        "name": "libcurl4t64",
+        "version": "8.14.1-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "librabbitmq4",
+        "version": "0.15.0-1+deb13u2",
+        "origin": ""
+      },
+      {
+        "name": "libnss3",
+        "version": "2:3.110-1+deb13u4",
+        "origin": ""
+      },
+      {
+        "name": "libgcrypt20",
+        "version": "1.11.0-7+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "gpgsm",
+        "version": "2.4.7-21+deb13u1+b5",
+        "origin": ""
+      },
+      {
+        "name": "dirmngr",
+        "version": "2.4.7-21+deb13u1+b5",
+        "origin": ""
+      },
+      {
+        "name": "gpg",
+        "version": "2.4.7-21+deb13u1+b5",
+        "origin": ""
+      },
+      {
+        "name": "gpgconf",
+        "version": "2.4.7-21+deb13u1+b5",
+        "origin": ""
+      },
+      {
+        "name": "gpg-agent",
+        "version": "2.4.7-21+deb13u1+b5",
+        "origin": ""
+      },
+      {
+        "name": "libasound2t64",
+        "version": "1.2.14-1+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libasound2-data",
+        "version": "1.2.14-1+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libbytes-random-secure-perl",
+        "version": "0.29-4~deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libcap2-bin",
+        "version": "1:2.75-10+deb13u1+b3",
+        "origin": ""
+      },
+      {
+        "name": "libcurl3t64-gnutls",
+        "version": "8.14.1-2+deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "libevent-2.1-7t64",
+        "version": "2.1.13-stable-1~deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libevent-core-2.1-7t64",
+        "version": "2.1.13-stable-1~deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libgbm1",
+        "version": "25.0.7-2+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "mesa-libgallium",
+        "version": "25.0.7-2+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libglib2.0-0t64",
+        "version": "2.84.4-3~deb13u5",
+        "origin": ""
+      },
+      {
+        "name": "libgraphite2-3",
+        "version": "1.3.14-2+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libgstreamer-plugins-base1.0-0",
+        "version": "1.26.2-1+deb13u2",
+        "origin": ""
+      },
+      {
+        "name": "libhtml-parser-perl",
+        "version": "3.83-2~deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libhttp-daemon-perl",
+        "version": "6.16-1+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libtalloc2",
+        "version": "2:2.4.3+samba4.22.11+dfsg-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libtevent0t64",
+        "version": "2:0.16.2+samba4.22.11+dfsg-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libsmbclient0",
+        "version": "2:4.22.11+dfsg-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "samba-common",
+        "version": "2:4.22.11+dfsg-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "smbclient",
+        "version": "2:4.22.11+dfsg-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libtdb1",
+        "version": "2:1.4.13+samba4.22.11+dfsg-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libldb2",
+        "version": "2:2.11.0+samba4.22.11+dfsg-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "samba-libs",
+        "version": "2:4.22.11+dfsg-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libwbclient0",
+        "version": "2:4.22.11+dfsg-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libnet-dns-perl",
+        "version": "1.56-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libpcre2-16-0",
+        "version": "10.46-1~deb13u3",
+        "origin": ""
+      },
+      {
+        "name": "libpng16-16t64",
+        "version": "1.6.48-1+deb13u6",
+        "origin": ""
+      },
+      {
+        "name": "libslirp0",
+        "version": "4.8.0-1+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libxml-libxml-perl",
+        "version": "2.0207+dfsg+really+2.0134-5+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "socat",
+        "version": "1.8.0.3-1+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libss2",
+        "version": "1.47.2-3+b12",
+        "origin": ""
+      },
+      {
+        "name": "libtasn1-6",
+        "version": "4.20.0-2+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "libunbound8",
+        "version": "1.26.1-0+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "python3-idna",
+        "version": "3.10-1+deb13u1",
+        "origin": ""
+      },
+      {
+        "name": "python3-urllib3",
+        "version": "2.3.0-3+deb13u2",
+        "origin": ""
+      },
+      {
+        "name": "xfsprogs",
+        "version": "6.13.0-2+deb13u1",
+        "origin": ""
+      }
+    ],
+    "wrapper_seconds": 57.8
+  }
+  pass took 1m10.4s
+WALL_SECONDS=70.462688397

documentation/audits/os-host-lane-2026-10-04/partB/live/b2-demo-hp-ring0.log

@@ -0,0 +1,135 @@
+=== felhom-agent 0.141.1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true ===
+time=2026-10-04T14:22:52.476+02:00 level=INFO msg="osupdate: START" run=20261004T122252Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T122252Z
+time=2026-10-04T14:23:08.079+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T122252Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T14:23:08.079+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T14:23:08.079+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T14:23:08.079+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T14:23:08.080+02:00 level=INFO msg="osupdate: DONE" run=20261004T122252Z layer=guest vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=6 not_covered=6 restart_needed=0 reboot_needed=false wrapper_seconds=15.5
+time=2026-10-04T14:23:08.089+02:00 level=INFO msg="osupdate: START" run=20261004T122252Z layer=host vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T122252Z
+time=2026-10-04T14:23:22.788+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T122252Z layer=host lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T14:23:22.788+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T14:23:22.788+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T14:23:22.788+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T14:23:23.852+02:00 level=INFO msg="osupdate: DONE" run=20261004T122252Z layer=host vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=78 not_covered=78 restart_needed=0 reboot_needed=false wrapper_seconds=14.6
+  --- os-update report (guest) ---
+  {
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "docker-ce-cli",
+      "containerd.io",
+      "docker-ce",
+      "docker-buildx-plugin",
+      "docker-ce-rootless-extras",
+      "docker-compose-plugin"
+    ],
+    "outcome": "nothing",
+    "pending": 6,
+    "reboot_needed": false,
+    "refused": null,
+    "release_id": "ring0-20261004T122252Z",
+    "restart_needed": null,
+    "ring": 0,
+    "run_id": "20261004T122252Z",
+    "upgraded": [],
+    "wrapper_seconds": 15.5
+  }
+  --- os-update report (host) ---
+  {
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "frr",
+      "shim-signed-common",
+      "proxmox-secure-boot-support",
+      "shim-unsigned",
+      "shim-helpers-amd64-signed",
+      "shim-signed",
+      "amd64-microcode",
+      "libradosstriper1",
+      "librgw2",
+      "ceph-common",
+      "librbd1",
+      "librados2",
+      "python3-cephfs",
+      "libcephfs2",
+      "python3-rgw",
+      "python3-rados",
+      "python3-ceph-argparse",
+      "python3-ceph-common",
+      "python3-rbd",
+      "ceph-fuse",
+      "chrony",
+      "libcorosync-common4",
+      "libcfg7",
+      "libcmap4",
+      "libcpg4",
+      "libknet1t64",
+      "libnozzle1t64",
+      "libquorum5",
+      "libvotequorum8",
+      "corosync",
+      "frr-pythontools",
+      "libjs-extjs",
+      "libnvpair3linux",
+      "libproxmox-acme-plugins",
+      "libproxmox-backup-qemu0",
+      "pve-qemu-kvm",
+      "libpve-notify-perl",
+      "libpve-cluster-api-perl",
+      "libpve-cluster-perl",
+      "pve-cluster",
+      "libpve-access-control",
+      "libpve-apiclient-perl",
+      "librados2-perl",
+      "proxmox-backup-client",
+      "proxmox-backup-file-restore",
+      "pve-manager",
+      "libproxmox-acme-perl",
+      "libpve-common-perl",
+      "libpve-guest-common-perl",
+      "qemu-server",
+      "libpve-storage-perl",
+      "pve-edk2-firmware-legacy",
+      "pve-edk2-firmware-ovmf",
+      "libpve-network-api-perl",
+      "libpve-network-perl",
+      "proxmox-firewall-data",
+      "pve-firewall",
+      "pve-container",
+      "pve-ha-manager",
+      "novnc-pve",
+      "proxmox-enterprise-support-keyring",
+      "proxmox-mini-journalreader",
+      "proxmox-widget-toolkit",
+      "pve-docs",
+      "pve-i18n",
+      "pve-xtermjs",
+      "pve-yew-mobile-i18n",
+      "pve-yew-mobile-gui",
+      "libuutil3linux",
+      "libzfs7linux",
+      "libzpool7linux",
+      "proxmox-kernel-helper",
+      "pve-edk2-firmware-aarch64",
+      "pve-edk2-firmware",
+      "pve-firmware",
+      "zfs-initramfs",
+      "zfsutils-linux",
+      "zfs-zed"
+    ],
+    "outcome": "nothing",
+    "pending": 78,
+    "reboot_needed": false,
+    "refused": null,
+    "release_id": "ring0-20261004T122252Z",
+    "restart_needed": [],
+    "ring": 0,
+    "run_id": "20261004T122252Z",
+    "upgraded": [],
+    "wrapper_seconds": 14.6
+  }
+  pass took 31.4s
+WALL_SECONDS=31.520457543

documentation/audits/os-host-lane-2026-10-04/partB/ring1/r0-set-ring1.txt

@@ -0,0 +1,2 @@
+{"ok":true}
+ 200

documentation/audits/os-host-lane-2026-10-04/partB/ring1/r1-demo-felhom-tzdata-back.txt

@@ -0,0 +1,43 @@
++ PKG=tzdata
+++ grep -oE 'tzdata:amd64 \([^,]+' /var/log/apt/history.log
+++ tail -1
+++ sed 's/.*(//'
++ OLD=2026b-0+deb13u1
+++ dpkg-query -W '-f=${Version}' tzdata
++ NEW=2026c-0+deb13u1
++ echo OLD=2026b-0+deb13u1 NEW=2026c-0+deb13u1
+OLD=2026b-0+deb13u1 NEW=2026c-0+deb13u1
+++ curl -s 'https://snapshot.debian.org/mr/binary/tzdata/2026c-0+deb13u1/binfiles?fileinfo=1'
+++ python3 -c '
+import json,sys; d=json.load(sys.stdin)
+print(sorted(f["first_seen"] for v in d["fileinfo"].values() for f in v)[0])'
++ TS=20260831T204404Z
++ . /etc/os-release
+++ PRETTY_NAME='Debian GNU/Linux 13 (trixie)'
+++ NAME='Debian GNU/Linux'
+++ VERSION_ID=13
+++ VERSION='13 (trixie)'
+++ VERSION_CODENAME=trixie
+++ DEBIAN_VERSION_FULL=13.7
+++ ID=debian
+++ HOME_URL=https://www.debian.org/
+++ SUPPORT_URL=https://www.debian.org/support
+++ BUG_REPORT_URL=https://bugs.debian.org/
++ printf 'deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/%s %s main\ndeb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/%s %s-security main\n' 20260831T204404Z trixie 20260831T204404Z trixie
++ apt-get -q update
++ apt-get -s install --allow-downgrades tzdata=2026b-0+deb13u1
++ grep -E '^(Inst|Remv)|downgraded'
+0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 78 not upgraded.
+Inst tzdata [2026c-0+deb13u1] (2026b-0+deb13u1 Debian:13.6/stable [all])
++ DEBIAN_FRONTEND=noninteractive
++ apt-get -y -q install --allow-downgrades tzdata=2026b-0+deb13u1
++ rm -f /etc/apt/sources.list.d/felhom-undo-snapshot.list
++ apt-get -q update
++ dpkg-query -W tzdata
+tzdata	2026b-0+deb13u1
++ ls /etc/apt/sources.list.d/
+ceph.sources
+debian.sources
+pve-enterprise.sources
+pve-no-subscription.sources
+tailscale.list

documentation/audits/os-host-lane-2026-10-04/partB/ring1/r2-demo-felhom-ring1-leg.log

@@ -0,0 +1,178 @@
+=== felhom-agent 0.141.1 selftest=os-update vmid=9201 ring=1 enabled=true guest-release=true host-release=true appliance=true ===
+time=2026-10-04T14:40:46.795+02:00 level=INFO msg="osupdate: START" run=20261004T124046Z layer=guest vmid=9201 ring=1 trigger=debug enabled=true release=os-guest-20261004-123933
+time=2026-10-04T14:40:56.895+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=os-guest-20261004-123933 layer=guest:9201 lane=fast mode=apply select=listed packages=272"
+time=2026-10-04T14:40:56.895+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T14:40:56.895+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=272 not-installed=0 from-snapshot=0"
+time=2026-10-04T14:40:56.895+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T14:40:56.896+02:00 level=INFO msg="osupdate: DONE" run=20261004T124046Z layer=guest vmid=9201 ring=1 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=6 not_covered=6 restart_needed=0 reboot_needed=false wrapper_seconds=10.1
+time=2026-10-04T14:40:56.907+02:00 level=INFO msg="osupdate: START" run=20261004T124046Z layer=host vmid=9201 ring=1 trigger=debug enabled=true release=os-host-20261004-124034
+time=2026-10-04T14:41:10.034+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=os-host-20261004-124034 layer=host lane=fast mode=apply select=listed packages=605"
+time=2026-10-04T14:41:10.034+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T14:41:10.034+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=1 already=604 not-installed=0 from-snapshot=0"
+time=2026-10-04T14:41:10.034+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=1.4 upgraded=1 restart-needed=agetty,blkmapd,chronyd,cron,dbus-daemon,dmeventd,ksmtuned,lxc-monitord,lxc-start,lxcfs,pmxcfs,proxmox-firewal,pve-firewall,pve-ha-crm,pve-ha-lrm,pve-lxc-syscall,pvedaemon,pvedaemon worke,pvefw-logger,pveproxy,pveproxy worker,pvescheduler,pvestatd,qmeventd,rpcbind,rrdcached,smartd,spiceproxy,spiceproxy work,sshd,systemd-logind,systemd-udevd,watchdog-mux,zed reboot-needed=yes"
+time=2026-10-04T14:41:10.815+02:00 level=INFO msg="osupdate: DONE" run=20261004T124046Z layer=host vmid=9201 ring=1 trigger=debug outcome=applied healthy=true reason="" upgraded=1 pending=80 not_covered=80 restart_needed=34 reboot_needed=true wrapper_seconds=13.1
+  --- os-update report (guest) ---
+  {
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "docker-ce-cli",
+      "containerd.io",
+      "docker-ce",
+      "docker-buildx-plugin",
+      "docker-ce-rootless-extras",
+      "docker-compose-plugin"
+    ],
+    "outcome": "nothing",
+    "pending": 6,
+    "reboot_needed": false,
+    "refused": null,
+    "release_id": "os-guest-20261004-123933",
+    "restart_needed": null,
+    "ring": 1,
+    "run_id": "20261004T124046Z",
+    "upgraded": [],
+    "wrapper_seconds": 10.1
+  }
+  --- os-update report (host) ---
+  {
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "frr",
+      "shim-signed-common",
+      "shim-unsigned",
+      "shim-helpers-amd64-signed",
+      "shim-signed",
+      "libradosstriper1",
+      "librgw2",
+      "ceph-common",
+      "librbd1",
+      "librados2",
+      "python3-cephfs",
+      "libcephfs2",
+      "python3-rgw",
+      "python3-rados",
+      "python3-ceph-argparse",
+      "python3-ceph-common",
+      "python3-rbd",
+      "ceph-fuse",
+      "chrony",
+      "libcorosync-common4",
+      "libcfg7",
+      "libcmap4",
+      "libcpg4",
+      "libknet1t64",
+      "libnozzle1t64",
+      "libquorum5",
+      "libvotequorum8",
+      "corosync",
+      "frr-pythontools",
+      "libjs-extjs",
+      "libnvpair3linux",
+      "libproxmox-acme-plugins",
+      "libproxmox-backup-qemu0",
+      "pve-qemu-kvm",
+      "libpve-notify-perl",
+      "libpve-cluster-api-perl",
+      "libpve-cluster-perl",
+      "pve-cluster",
+      "libpve-access-control",
+      "libpve-apiclient-perl",
+      "librados2-perl",
+      "proxmox-backup-client",
+      "proxmox-backup-file-restore",
+      "pve-manager",
+      "libproxmox-acme-perl",
+      "libpve-common-perl",
+      "libpve-guest-common-perl",
+      "qemu-server",
+      "libpve-storage-perl",
+      "pve-edk2-firmware-legacy",
+      "pve-edk2-firmware-ovmf",
+      "libpve-network-api-perl",
+      "libpve-network-perl",
+      "proxmox-firewall-data",
+      "pve-firewall",
+      "pve-container",
+      "pve-ha-manager",
+      "novnc-pve",
+      "proxmox-enterprise-support-keyring",
+      "proxmox-mini-journalreader",
+      "proxmox-widget-toolkit",
+      "pve-docs",
+      "pve-i18n",
+      "pve-xtermjs",
+      "pve-yew-mobile-i18n",
+      "pve-yew-mobile-gui",
+      "libuutil3linux",
+      "libzfs7linux",
+      "libzpool7linux",
+      "proxmox-first-boot",
+      "pve-firmware",
+      "proxmox-kernel-7.0.14-20-pve-signed",
+      "proxmox-kernel-7.0",
+      "proxmox-kernel-helper",
+      "pve-edk2-firmware-aarch64",
+      "pve-edk2-firmware",
+      "zfs-initramfs",
+      "zfsutils-linux",
+      "zfs-zed",
+      "tailscale"
+    ],
+    "outcome": "applied",
+    "pending": 80,
+    "reboot_needed": true,
+    "refused": null,
+    "release_id": "os-host-20261004-124034",
+    "restart_needed": [
+      "agetty",
+      "blkmapd",
+      "chronyd",
+      "cron",
+      "dbus-daemon",
+      "dmeventd",
+      "ksmtuned",
+      "lxc-monitord",
+      "lxc-start",
+      "lxcfs",
+      "pmxcfs",
+      "proxmox-firewal",
+      "pve-firewall",
+      "pve-ha-crm",
+      "pve-ha-lrm",
+      "pve-lxc-syscall",
+      "pvedaemon",
+      "pvedaemon worke",
+      "pvefw-logger",
+      "pveproxy",
+      "pveproxy worker",
+      "pvescheduler",
+      "pvestatd",
+      "qmeventd",
+      "rpcbind",
+      "rrdcached",
+      "smartd",
+      "spiceproxy",
+      "spiceproxy work",
+      "sshd",
+      "systemd-logind",
+      "systemd-udevd",
+      "watchdog-mux",
+      "zed"
+    ],
+    "ring": 1,
+    "run_id": "20261004T124046Z",
+    "upgraded": [
+      {
+        "name": "tzdata",
+        "version": "2026c-0+deb13u1",
+        "origin": ""
+      }
+    ],
+    "wrapper_seconds": 13.1
+  }
+  pass took 24.1s
+tzdata	2026c-0+deb13u1

documentation/audits/os-host-lane-2026-10-04/partB/undo/u1-find.txt

@@ -0,0 +1,11 @@
++ PKG=eject
++ OLD=2.41-5
++ dpkg -s eject
++ grep -E '^(Status|Version)'
+Status: install ok installed
+Version: 2.41.5-0+deb13u1
++ curl -s 'https://snapshot.debian.org/mr/binary/eject/2.41-5/binfiles?fileinfo=1'
++ python3 -c '
+import json,sys; d=json.load(sys.stdin)
+print(sorted(f["first_seen"] for v in d["fileinfo"].values() for f in v)[0])'
+20250510T015204Z

documentation/audits/os-host-lane-2026-10-04/partB/undo/u2-simulate.txt

@@ -0,0 +1,28 @@
++ PKG=eject
++ OLD=2.41-5
++ TS=20250510T015204Z
++ . /etc/os-release
+++ PRETTY_NAME='Debian GNU/Linux 13 (trixie)'
+++ NAME='Debian GNU/Linux'
+++ VERSION_ID=13
+++ VERSION='13 (trixie)'
+++ VERSION_CODENAME=trixie
+++ DEBIAN_VERSION_FULL=13.7
+++ ID=debian
+++ HOME_URL=https://www.debian.org/
+++ SUPPORT_URL=https://www.debian.org/support
+++ BUG_REPORT_URL=https://bugs.debian.org/
++ cat
+++ date +%s
++ S=1791116624
++ apt-get -q update
++ tail -3
+Get:7 http://snapshot.debian.org/archive/debian/20250510T015204Z trixie/main amd64 Packages [9681 kB]
+Fetched 9904 kB in 5s (2165 kB/s)
+Reading package lists...
+++ date +%s
+update_seconds=6
++ echo update_seconds=6
++ apt-get -s install --allow-downgrades eject=2.41-5
++ grep -E '^(Inst|Remv|Conf)|downgraded|newly|Err|E:'
+E: Version '2.41-5' for 'eject' was not found

documentation/audits/os-host-lane-2026-10-04/partB/undo/u3-simulate-newts.txt

@@ -0,0 +1,35 @@
++ PKG=eject
++ OLD=2.41-5
+++ dpkg-query -W '-f=${Version}' eject
++ NEW=2.41.5-0+deb13u1
+++ curl -s 'https://snapshot.debian.org/mr/binary/eject/2.41.5-0+deb13u1/binfiles?fileinfo=1'
+++ python3 -c '
+import json,sys; d=json.load(sys.stdin)
+print(sorted(f["first_seen"] for v in d["fileinfo"].values() for f in v)[0])'
+TS=20260814T165831Z
++ TS=20260814T165831Z
++ echo TS=20260814T165831Z
++ . /etc/os-release
+++ PRETTY_NAME='Debian GNU/Linux 13 (trixie)'
+++ NAME='Debian GNU/Linux'
+++ VERSION_ID=13
+++ VERSION='13 (trixie)'
+++ VERSION_CODENAME=trixie
+++ DEBIAN_VERSION_FULL=13.7
+++ ID=debian
+++ HOME_URL=https://www.debian.org/
+++ SUPPORT_URL=https://www.debian.org/support
+++ BUG_REPORT_URL=https://bugs.debian.org/
++ cat
++ apt-get -q update
++ tail -1
+Reading package lists...
++ apt-cache madison eject
+     eject | 2.41.5-0+deb13u1 | http://deb.debian.org/debian trixie/main amd64 Packages
+     eject | 2.41.5-0+deb13u1 | http://security.debian.org/debian-security trixie-security/main amd64 Packages
+     eject | 2.41.5-0+deb13u1 | http://snapshot.debian.org/archive/debian-security/20260814T165831Z trixie-security/main amd64 Packages
+     eject |     2.41-5 | http://snapshot.debian.org/archive/debian/20260814T165831Z trixie/main amd64 Packages
++ apt-get -s install --allow-downgrades eject=2.41-5
++ grep -E '^(Inst|Remv|Conf)|downgraded|newly|Err|E:'
+E: Unable to correct problems, you have held broken packages.
+E: The following information from --solver 3.0 may provide additional context:

documentation/audits/os-host-lane-2026-10-04/partB/undo/u4-tzdata-simulate.txt

@@ -0,0 +1,37 @@
++ rm -f /etc/apt/sources.list.d/felhom-undo-snapshot.list
++ PKG=tzdata
++ OLD=2026b-0+deb13u1
+++ dpkg-query -W '-f=${Version}' tzdata
++ NEW=2026c-0+deb13u1
++ echo NEW=2026c-0+deb13u1
+NEW=2026c-0+deb13u1
+++ curl -s 'https://snapshot.debian.org/mr/binary/tzdata/2026c-0+deb13u1/binfiles?fileinfo=1'
+++ python3 -c '
+import json,sys; d=json.load(sys.stdin)
+print(sorted(f["first_seen"] for v in d["fileinfo"].values() for f in v)[0])'
+TS=20260831T204404Z
++ TS=20260831T204404Z
++ echo TS=20260831T204404Z
++ . /etc/os-release
+++ PRETTY_NAME='Debian GNU/Linux 13 (trixie)'
+++ NAME='Debian GNU/Linux'
+++ VERSION_ID=13
+++ VERSION='13 (trixie)'
+++ VERSION_CODENAME=trixie
+++ DEBIAN_VERSION_FULL=13.7
+++ ID=debian
+++ HOME_URL=https://www.debian.org/
+++ SUPPORT_URL=https://www.debian.org/support
+++ BUG_REPORT_URL=https://bugs.debian.org/
++ cat
++ apt-get -q update
++ tail -1
+Reading package lists...
++ apt-cache madison tzdata
+    tzdata | 2026c-0+deb13u1 | http://deb.debian.org/debian trixie/main amd64 Packages
+    tzdata | 2026b-0+deb13u1 | http://snapshot.debian.org/archive/debian/20260831T204404Z trixie/main amd64 Packages
++ apt-get -s install --allow-downgrades tzdata=2026b-0+deb13u1
++ grep -E '^(Inst|Remv|Conf)|downgraded|newly|Err|E:'
+0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 77 not upgraded.
+Inst tzdata [2026c-0+deb13u1] (2026b-0+deb13u1 Debian:13.6/stable [all])
+Conf tzdata (2026b-0+deb13u1 Debian:13.6/stable [all])

documentation/audits/os-host-lane-2026-10-04/partB/undo/u5-tzdata-undo.txt

@@ -0,0 +1,38 @@
++ PKG=tzdata
++ OLD=2026b-0+deb13u1
+++ date +%s
++ S=1791116670
++ DEBIAN_FRONTEND=noninteractive
++ apt-get -y -q install --allow-downgrades tzdata=2026b-0+deb13u1
++ grep -E '^(Unpacking|Setting up)|downgraded'
+0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 77 not upgraded.
+Unpacking tzdata (2026b-0+deb13u1) over (2026c-0+deb13u1) ...
+Setting up tzdata (2026b-0+deb13u1) ...
++ apt-mark hold tzdata
+tzdata set on hold.
++ rm -f /etc/apt/sources.list.d/felhom-undo-snapshot.list
++ apt-get -q update
++ tail -1
+Reading package lists...
+++ date +%s
+undo_seconds=4
++ echo undo_seconds=4
++ dpkg -s tzdata
++ grep -E '^(Status|Version)'
+Status: hold ok installed
+Version: 2026b-0+deb13u1
++ apt-mark showhold
+tzdata
++ systemctl is-active pveproxy pvedaemon pvestatd pve-cluster felhom-agent
+active
+active
+active
+active
+active
++ pct status 9201
+status: running
++ ls /etc/apt/sources.list.d/
+ceph.sources
+debian.sources
+pve-enterprise.sources
+pve-no-subscription.sources

documentation/audits/os-host-lane-2026-10-04/partB/undo/u6-leg-with-hold.log

@@ -0,0 +1,135 @@
+=== felhom-agent 0.141.1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true ===
+time=2026-10-04T14:24:41.910+02:00 level=INFO msg="osupdate: START" run=20261004T122441Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T122441Z
+time=2026-10-04T14:24:57.242+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T122441Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T14:24:57.242+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T14:24:57.242+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T14:24:57.242+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T14:24:57.243+02:00 level=INFO msg="osupdate: DONE" run=20261004T122441Z layer=guest vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=6 not_covered=6 restart_needed=0 reboot_needed=false wrapper_seconds=15.3
+time=2026-10-04T14:24:57.251+02:00 level=INFO msg="osupdate: START" run=20261004T122441Z layer=host vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T122441Z
+time=2026-10-04T14:25:12.127+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T122441Z layer=host lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T14:25:12.127+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T14:25:12.127+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T14:25:12.127+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T14:25:13.136+02:00 level=INFO msg="osupdate: DONE" run=20261004T122441Z layer=host vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=78 not_covered=78 restart_needed=0 reboot_needed=false wrapper_seconds=14.8
+  --- os-update report (guest) ---
+  {
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "docker-ce-cli",
+      "containerd.io",
+      "docker-ce",
+      "docker-buildx-plugin",
+      "docker-ce-rootless-extras",
+      "docker-compose-plugin"
+    ],
+    "outcome": "nothing",
+    "pending": 6,
+    "reboot_needed": false,
+    "refused": null,
+    "release_id": "ring0-20261004T122441Z",
+    "restart_needed": null,
+    "ring": 0,
+    "run_id": "20261004T122441Z",
+    "upgraded": [],
+    "wrapper_seconds": 15.3
+  }
+  --- os-update report (host) ---
+  {
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "frr",
+      "shim-signed-common",
+      "proxmox-secure-boot-support",
+      "shim-unsigned",
+      "shim-helpers-amd64-signed",
+      "shim-signed",
+      "amd64-microcode",
+      "libradosstriper1",
+      "librgw2",
+      "ceph-common",
+      "librbd1",
+      "librados2",
+      "python3-cephfs",
+      "libcephfs2",
+      "python3-rgw",
+      "python3-rados",
+      "python3-ceph-argparse",
+      "python3-ceph-common",
+      "python3-rbd",
+      "ceph-fuse",
+      "chrony",
+      "libcorosync-common4",
+      "libcfg7",
+      "libcmap4",
+      "libcpg4",
+      "libknet1t64",
+      "libnozzle1t64",
+      "libquorum5",
+      "libvotequorum8",
+      "corosync",
+      "frr-pythontools",
+      "libjs-extjs",
+      "libnvpair3linux",
+      "libproxmox-acme-plugins",
+      "libproxmox-backup-qemu0",
+      "pve-qemu-kvm",
+      "libpve-notify-perl",
+      "libpve-cluster-api-perl",
+      "libpve-cluster-perl",
+      "pve-cluster",
+      "libpve-access-control",
+      "libpve-apiclient-perl",
+      "librados2-perl",
+      "proxmox-backup-client",
+      "proxmox-backup-file-restore",
+      "pve-manager",
+      "libproxmox-acme-perl",
+      "libpve-common-perl",
+      "libpve-guest-common-perl",
+      "qemu-server",
+      "libpve-storage-perl",
+      "pve-edk2-firmware-legacy",
+      "pve-edk2-firmware-ovmf",
+      "libpve-network-api-perl",
+      "libpve-network-perl",
+      "proxmox-firewall-data",
+      "pve-firewall",
+      "pve-container",
+      "pve-ha-manager",
+      "novnc-pve",
+      "proxmox-enterprise-support-keyring",
+      "proxmox-mini-journalreader",
+      "proxmox-widget-toolkit",
+      "pve-docs",
+      "pve-i18n",
+      "pve-xtermjs",
+      "pve-yew-mobile-i18n",
+      "pve-yew-mobile-gui",
+      "libuutil3linux",
+      "libzfs7linux",
+      "libzpool7linux",
+      "proxmox-kernel-helper",
+      "pve-edk2-firmware-aarch64",
+      "pve-edk2-firmware",
+      "pve-firmware",
+      "zfs-initramfs",
+      "zfsutils-linux",
+      "zfs-zed"
+    ],
+    "outcome": "nothing",
+    "pending": 78,
+    "reboot_needed": false,
+    "refused": null,
+    "release_id": "ring0-20261004T122441Z",
+    "restart_needed": [],
+    "ring": 0,
+    "run_id": "20261004T122441Z",
+    "upgraded": [],
+    "wrapper_seconds": 14.8
+  }
+  pass took 31.2s
+tzdata	2026b-0+deb13u1

documentation/audits/os-host-lane-2026-10-04/partB/undo/u7-unhold-leg.log

@@ -0,0 +1,143 @@
+Canceled hold on tzdata.
+=== felhom-agent 0.141.1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true ===
+time=2026-10-04T14:25:22.024+02:00 level=INFO msg="osupdate: START" run=20261004T122522Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T122522Z
+time=2026-10-04T14:25:37.515+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T122522Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T14:25:37.515+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T14:25:37.515+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T14:25:37.515+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T14:25:37.516+02:00 level=INFO msg="osupdate: DONE" run=20261004T122522Z layer=guest vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=6 not_covered=6 restart_needed=0 reboot_needed=false wrapper_seconds=15.4
+time=2026-10-04T14:25:37.526+02:00 level=INFO msg="osupdate: START" run=20261004T122522Z layer=host vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T122522Z
+time=2026-10-04T14:25:57.778+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T122522Z layer=host lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T14:25:57.778+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T14:25:57.778+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=1 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T14:25:57.778+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=2.0 upgraded=1 restart-needed=- reboot-needed=no"
+time=2026-10-04T14:25:58.809+02:00 level=INFO msg="osupdate: DONE" run=20261004T122522Z layer=host vmid=9201 ring=0 trigger=debug outcome=applied healthy=true reason="" upgraded=1 pending=78 not_covered=78 restart_needed=0 reboot_needed=false wrapper_seconds=20.2
+  --- os-update report (guest) ---
+  {
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "docker-ce-cli",
+      "containerd.io",
+      "docker-ce",
+      "docker-buildx-plugin",
+      "docker-ce-rootless-extras",
+      "docker-compose-plugin"
+    ],
+    "outcome": "nothing",
+    "pending": 6,
+    "reboot_needed": false,
+    "refused": null,
+    "release_id": "ring0-20261004T122522Z",
+    "restart_needed": null,
+    "ring": 0,
+    "run_id": "20261004T122522Z",
+    "upgraded": [],
+    "wrapper_seconds": 15.4
+  }
+  --- os-update report (host) ---
+  {
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "frr",
+      "shim-signed-common",
+      "proxmox-secure-boot-support",
+      "shim-unsigned",
+      "shim-helpers-amd64-signed",
+      "shim-signed",
+      "amd64-microcode",
+      "libradosstriper1",
+      "librgw2",
+      "ceph-common",
+      "librbd1",
+      "librados2",
+      "python3-cephfs",
+      "libcephfs2",
+      "python3-rgw",
+      "python3-rados",
+      "python3-ceph-argparse",
+      "python3-ceph-common",
+      "python3-rbd",
+      "ceph-fuse",
+      "chrony",
+      "libcorosync-common4",
+      "libcfg7",
+      "libcmap4",
+      "libcpg4",
+      "libknet1t64",
+      "libnozzle1t64",
+      "libquorum5",
+      "libvotequorum8",
+      "corosync",
+      "frr-pythontools",
+      "libjs-extjs",
+      "libnvpair3linux",
+      "libproxmox-acme-plugins",
+      "libproxmox-backup-qemu0",
+      "pve-qemu-kvm",
+      "libpve-notify-perl",
+      "libpve-cluster-api-perl",
+      "libpve-cluster-perl",
+      "pve-cluster",
+      "libpve-access-control",
+      "libpve-apiclient-perl",
+      "librados2-perl",
+      "proxmox-backup-client",
+      "proxmox-backup-file-restore",
+      "pve-manager",
+      "libproxmox-acme-perl",
+      "libpve-common-perl",
+      "libpve-guest-common-perl",
+      "qemu-server",
+      "libpve-storage-perl",
+      "pve-edk2-firmware-legacy",
+      "pve-edk2-firmware-ovmf",
+      "libpve-network-api-perl",
+      "libpve-network-perl",
+      "proxmox-firewall-data",
+      "pve-firewall",
+      "pve-container",
+      "pve-ha-manager",
+      "novnc-pve",
+      "proxmox-enterprise-support-keyring",
+      "proxmox-mini-journalreader",
+      "proxmox-widget-toolkit",
+      "pve-docs",
+      "pve-i18n",
+      "pve-xtermjs",
+      "pve-yew-mobile-i18n",
+      "pve-yew-mobile-gui",
+      "libuutil3linux",
+      "libzfs7linux",
+      "libzpool7linux",
+      "proxmox-kernel-helper",
+      "pve-edk2-firmware-aarch64",
+      "pve-edk2-firmware",
+      "pve-firmware",
+      "zfs-initramfs",
+      "zfsutils-linux",
+      "zfs-zed"
+    ],
+    "outcome": "applied",
+    "pending": 78,
+    "reboot_needed": false,
+    "refused": null,
+    "release_id": "ring0-20261004T122522Z",
+    "restart_needed": [],
+    "ring": 0,
+    "run_id": "20261004T122522Z",
+    "upgraded": [
+      {
+        "name": "tzdata",
+        "version": "2026c-0+deb13u1",
+        "origin": ""
+      }
+    ],
+    "wrapper_seconds": 20.2
+  }
+  pass took 36.9s
+tzdata	2026c-0+deb13u1
+0

documentation/audits/os-host-lane-2026-10-04/partB/wrapper-redproof.txt

@@ -0,0 +1,20 @@
+# felhom-os-apply (agent v0.141.0) red-proof, 2026-10-04: each mutation is applied to a COPY, syntax-checked, and the suite run against it.
+R1 (7 raise(s) removed): compiles=True suite rc=1; failing: test_R1_not_owned_by_the_agent, test_R1_path_outside_the_plan_dir, test_R1_symlink; expected test failed: YES
+R2 (2 raise(s) removed): compiles=True suite rc=1; failing: test_R2_non_debian_origin_in_the_plan, test_R2_non_debian_origin_in_the_simulation; expected test failed: YES
+R3 (1 raise(s) removed): compiles=True suite rc=1; failing: test_R3_slow_lane; expected test failed: YES
+R4 (1 raise(s) removed): compiles=True suite rc=1; failing: test_R4_removal; expected test failed: YES
+R5 (1 raise(s) removed): compiles=True suite rc=1; failing: test_R5_downgrade_exact; expected test failed: YES
+R6 (4 raise(s) removed): compiles=True suite rc=1; failing: test_R6_allow_new_is_slow_lane, test_R6_new_package, test_R6_unlisted_package; expected test failed: YES
+R7 (7 raise(s) removed): compiles=True suite rc=1; failing: test_R7_not_downloadable_and_no_snapshot, test_snapshot_does_not_have_it_either; expected test failed: YES
+R8 (1 raise(s) removed): compiles=True suite rc=1; failing: test_R8_free_space; expected test failed: YES
+R9 (2 raise(s) removed): compiles=True suite rc=1; failing: test_R9_apt_lock_held, test_R9_guest_locked_by_a_backup; expected test failed: YES
+R10 (4 raise(s) removed): compiles=True suite rc=1; failing: test_R10_bind_only_in_a_snapshot_section, test_R10_not_running, test_R10_not_the_boxs_own_guest, test_R10_reserved_vmid; expected test failed: YES
+R11 (12 raise(s) removed): compiles=True suite rc=1; failing: test_R11_bad_name, test_R11_bad_version_string, test_R11_duplicate; expected test failed: YES
+R12 (5 raise(s) removed): compiles=True suite rc=1; failing: test_R12_host_on_a_byo_box, test_R12_host_record_not_root_owned, test_R12_host_without_an_install_record, test_R12_unknown_layer; expected test failed: YES
+R13 (1 raise(s) removed): compiles=True suite rc=1; failing: test_R13_repair_does_not_fix_it; expected test failed: YES
+R14 (2 raise(s) removed): compiles=True suite rc=1; failing: test_R14_kernel_package_in_a_host_plan, test_R14_kernel_package_pulled_by_the_simulation; expected test failed: YES
+host layer runs in the guest instead: compiles=True suite rc=1; failing: test_host_runs_on_the_host_not_in_the_guest; expected test failed: YES
+pending-fast takes every pending upgrade: compiles=True suite rc=1; failing: test_pending_fast_skips_proxmox_docker_and_kernel; expected test failed: YES
+version checks in the target (pct exec per package): compiles=True suite rc=1; failing: test_no_per_package_guest_calls; expected test failed: YES
+lxc-start does not mean reboot: compiles=True suite rc=0; failing: NONE; expected test failed: NO
+lxc-start does not mean reboot (after adding test_reboot_needed_for_lxc_start_alone): suite rc=1; failing: test_reboot_needed_for_lxc_start_alone

documentation/audits/os-host-lane-2026-10-04/partC/agent-golden-redproof.txt

@@ -0,0 +1,8 @@
+# Agent red-proof: the host_release wire tag (contract with the hub golden), 2026-10-04T11:04:25Z
+# mutation: report.go json:"host_release" -> json:"hostrelease"; restored after
+=== RUN   TestOSUpdateGolden_Decodes
+    osupdate_contract_test.go:34: host_release = <nil>
+--- FAIL: TestOSUpdateGolden_Decodes (0.00s)
+FAIL
+FAIL	gitea.dooplex.hu/admin/felhom-agent/internal/hub	0.007s
+FAIL

documentation/audits/os-host-lane-2026-10-04/partC/hub-redproofs.txt

@@ -0,0 +1,30 @@
+# Hub red-proofs, Parts B+C (hub v0.131.0), 2026-10-04T11:03:52Z
+# script: each mutation applied, the named test run with -v, file restored.
+[CAUGHT] host candidate leaves out kernel/boot/firmware: mutated internal/osupdates/service.go; TestCandidate_HostLeavesOutKernelBootFirmware ran=True failed=True rc=1
+    alarms_test.go:27: host candidate = map[firmware-realtek:{Name:firmware-realtek Version:20250410-2 Origin:Debian} grub-efi-amd64:{Name:grub-efi-amd64 Version:2.12-9 Origin:Debian} intel-microcode:{Name:intel-microcode Ve
+    FAIL	gitea.dooplex.hu/admin/felhom-hub/internal/osupdates	0.034s
+[CAUGHT] host nights counted per layer: mutated internal/osupdates/service.go; TestLayers_SeparateSets ran=True failed=True rc=1
+    alarms_test.go:60: host approved on a GUEST night: &{ID:os-host-20261005-130000 Layer:host Fingerprint:535006df2f78491d ApprovedAt:2026-10-05 13:00:00 +0000 UTC ApprovedBy:auto PackagesJSON:[{"name":"libssl3t64","version
+    FAIL	gitea.dooplex.hu/admin/felhom-hub/internal/osupdates	0.042s
+[CAUGHT] alarm 1 stale threshold: mutated internal/osupdates/service.go; TestAlarm_StaleLeg ran=True failed=True rc=1
+    alarms_test.go:151: no alarm at 8 days: []
+    FAIL	gitea.dooplex.hu/admin/felhom-hub/internal/osupdates	0.041s
+[CAUGHT] alarm 1 switch OFF not watched: mutated internal/osupdates/service.go; TestAlarm_StaleLeg ran=True failed=True rc=1
+    alarms_test.go:168: alarmed with the switch OFF
+    FAIL	gitea.dooplex.hu/admin/felhom-hub/internal/osupdates	0.054s
+[CAUGHT] alarm 2 reboot-needed counts from the FIRST report: mutated internal/osupdates/service.go; TestAlarm_RebootNeeded ran=True failed=True rc=1
+    alarms_test.go:211: reboot-needed since = 2026-10-17 12:00:00 +0000 UTC
+    FAIL	gitea.dooplex.hu/admin/felhom-hub/internal/osupdates	0.045s
+[CAUGHT] alarm 3 counts only fast-lane pending: mutated internal/osupdates/service.go; TestAlarm_Ring0Stalled ran=True failed=True rc=1
+    alarms_test.go:235: a pending KERNEL counted as a stalled fast lane
+    FAIL	gitea.dooplex.hu/admin/felhom-hub/internal/osupdates	0.040s
+[CAUGHT] alarm 4 ring 0 never alarms: mutated internal/osupdates/service.go; TestAlarm_NotCovered ran=True failed=True rc=1
+    alarms_test.go:278: not-covered alarms = ["OS updates (guest): cust1 has had 1 fast-lane package(s) no approved release covers since 2026-10-04: libsomething-hw. Ring 0 does not run them (other hardware?) — add matching 
+    FAIL	gitea.dooplex.hu/admin/felhom-hub/internal/osupdates	0.060s
+[CAUGHT] alarm events are operator-only: mutated internal/notify/dispatcher.go; TestOSUpdateEvents_OperatorOnlyExceptApplied ran=True failed=True rc=1
+    os_alarms_operator_test.go:17: os_ring0_stalled is not operator-only
+    FAIL	gitea.dooplex.hu/admin/felhom-hub/internal/notify	0.009s
+[CAUGHT] host_release served to ring 1: mutated internal/osupdates/service.go; TestOSUpdate_DesiredBlockMatchesTheGolden ran=True failed=True rc=1
+    os_updates_test.go:59: served os_update diverged from the golden:
+    FAIL	gitea.dooplex.hu/admin/felhom-hub/internal/api	0.055s
+after restore: ok  	gitea.dooplex.hu/admin/felhom-hub/internal/osupdates	0.544s | ok  	gitea.dooplex.hu/admin/felhom-hub/internal/notify	1.169s | ok  	gitea.dooplex.hu/admin/felhom-hub/internal/api	4.690s rc= 0

documentation/audits/os-host-lane-2026-10-04/partC/hub_rp.py

@@ -0,0 +1,44 @@
+import subprocess, sys, shutil
+M = [
+ ("host candidate leaves out kernel/boot/firmware", "internal/osupdates/service.go",
+  "if layer == LayerHost && hostSlowRE.MatchString(p.Name) {\n\t\t\t\tcontinue", "if false && hostSlowRE.MatchString(p.Name) {\n\t\t\t\tcontinue",
+  "./internal/osupdates/", "TestCandidate_HostLeavesOutKernelBootFirmware"),
+ ("host nights counted per layer", "internal/osupdates/service.go",
+  "reps, err := s.Store.OSReportsSince(h, layer, first)", "reps, err := s.Store.OSReportsSince(h, LayerGuest, first)",
+  "./internal/osupdates/", "TestLayers_SeparateSets"),
+ ("alarm 1 stale threshold", "internal/osupdates/service.go",
+  "now.Sub(ref) >= stale", "now.Sub(ref) >= 10*stale", "./internal/osupdates/", "TestAlarm_StaleLeg"),
+ ("alarm 1 switch OFF not watched", "internal/osupdates/service.go",
+  "holds := st.Enabled && canRun", "holds := canRun", "./internal/osupdates/", "TestAlarm_StaleLeg"),
+ ("alarm 2 reboot-needed counts from the FIRST report", "internal/osupdates/service.go",
+  "\t\tsince = rep.ReceivedAt\n", "\t\tif since.IsZero() {\n\t\t\tsince = rep.ReceivedAt\n\t\t}\n", "./internal/osupdates/", "TestAlarm_RebootNeeded"),
+ ("alarm 3 counts only fast-lane pending", "internal/osupdates/service.go",
+  "pendingAny += len(fastPending(layer, r.Pending))", "pendingAny += len(r.Pending)", "./internal/osupdates/", "TestAlarm_Ring0Stalled"),
+ ("alarm 4 ring 0 never alarms", "internal/osupdates/service.go",
+  "\tif ring == 0 {\n\t\treturn nil\n\t}\n", "", "./internal/osupdates/", "TestAlarm_NotCovered"),
+ ("alarm events are operator-only", "internal/notify/dispatcher.go",
+  '\t"os_ring0_stalled": true,\n', "", "./internal/notify/", "TestOSUpdateEvents_OperatorOnlyExceptApplied"),
+ ("host_release served to ring 1", "internal/osupdates/service.go",
+  "b.Release, b.HostRelease = s.releaseBlock(LayerGuest), s.releaseBlock(LayerHost)", "b.Release = s.releaseBlock(LayerGuest)",
+  "./internal/api/", "TestOSUpdate_DesiredBlockMatchesTheGolden"),
+]
+ok = True
+for name, f, old, new, pkg, test in M:
+    src = open(f).read()
+    assert src.count(old) == 1, (name, src.count(old))
+    shutil.copy(f, f + ".bak")
+    open(f, "w").write(src.replace(old, new))
+    r = subprocess.run(["go", "test", pkg, "-run", "^" + test + "$", "-v", "-count=1"], capture_output=True, text=True)
+    shutil.move(f + ".bak", f)
+    out = r.stdout + r.stderr
+    ran = ("=== RUN   " + test) in out
+    failed = ("--- FAIL: " + test) in out
+    verdict = "CAUGHT" if (ran and failed and r.returncode != 0) else "NOT CAUGHT"
+    if verdict != "CAUGHT": ok = False
+    print(f"[{verdict}] {name}: mutated {f}; {test} ran={ran} failed={failed} rc={r.returncode}")
+    for l in out.splitlines():
+        if "_test.go:" in l or l.startswith("FAIL\t") or "build failed" in l or l.startswith("#"):
+            print("    " + l.strip()[:220])
+r = subprocess.run(["go", "test", "./internal/osupdates/", "./internal/notify/", "./internal/api/", "-count=1"], capture_output=True, text=True)
+print("after restore:", r.stdout.strip().replace("\n", " | "), "rc=", r.returncode)
+sys.exit(0 if ok and r.returncode == 0 else 1)

documentation/audits/os-host-lane-2026-10-04/partC/live/fleet-during-ring1-test.json

@@ -0,0 +1,104 @@
+{
+    "boxes": [
+        {
+            "HostID": "demo-felhom-8363b5",
+            "Ring": 1,
+            "Enabled": true,
+            "Tunnel": "running",
+            "Guest": {
+                "ReleaseID": "os-guest-20261004-123933",
+                "LastOutcome": "nothing",
+                "LastAt": "2026-10-04T12:40:56Z",
+                "LastSuccessfulLeg": "2026-10-04T12:40:56Z",
+                "Pending": 6,
+                "NotCovered": 6,
+                "NotCoveredFast": 0,
+                "RestartNeeded": 0,
+                "RebootNeededSince": "2026-10-04T09:20:04Z",
+                "WrapperPassSeconds": 10.1
+            },
+            "Host": {
+                "ReleaseID": "os-host-20261004-124034",
+                "LastOutcome": "applied",
+                "LastAt": "2026-10-04T12:41:10Z",
+                "LastSuccessfulLeg": "2026-10-04T12:41:10Z",
+                "Pending": 80,
+                "NotCovered": 80,
+                "NotCoveredFast": 0,
+                "RestartNeeded": 34,
+                "RebootNeededSince": "2026-10-04T11:53:20Z",
+                "WrapperPassSeconds": 13.1
+            }
+        },
+        {
+            "HostID": "demo-hp-bb76ea",
+            "Ring": 0,
+            "Enabled": true,
+            "Tunnel": "unknown",
+            "Guest": {
+                "ReleaseID": "ring0-20261004T122522Z",
+                "LastOutcome": "nothing",
+                "LastAt": "2026-10-04T12:25:37Z",
+                "LastSuccessfulLeg": "2026-10-04T12:25:37Z",
+                "Pending": 6,
+                "NotCovered": 6,
+                "NotCoveredFast": 0,
+                "RestartNeeded": 0,
+                "RebootNeededSince": "2026-10-04T09:21:28Z",
+                "WrapperPassSeconds": 15.4
+            },
+            "Host": {
+                "ReleaseID": "ring0-20261004T122522Z",
+                "LastOutcome": "applied",
+                "LastAt": "2026-10-04T12:25:58Z",
+                "LastSuccessfulLeg": "2026-10-04T12:25:58Z",
+                "Pending": 78,
+                "NotCovered": 78,
+                "NotCoveredFast": 0,
+                "RestartNeeded": 0,
+                "RebootNeededSince": "0001-01-01T00:00:00Z",
+                "WrapperPassSeconds": 20.2
+            }
+        },
+        {
+            "HostID": "drill-r50-0a4f9a",
+            "Ring": 1,
+            "Enabled": true,
+            "Tunnel": "inactive",
+            "Guest": {
+                "ReleaseID": "",
+                "LastOutcome": "",
+                "LastAt": "0001-01-01T00:00:00Z",
+                "LastSuccessfulLeg": "0001-01-01T00:00:00Z",
+                "Pending": 0,
+                "NotCovered": 0,
+                "NotCoveredFast": 0,
+                "RestartNeeded": 0,
+                "RebootNeededSince": "0001-01-01T00:00:00Z",
+                "WrapperPassSeconds": 0
+            },
+            "Host": {
+                "ReleaseID": "",
+                "LastOutcome": "",
+                "LastAt": "0001-01-01T00:00:00Z",
+                "LastSuccessfulLeg": "0001-01-01T00:00:00Z",
+                "Pending": 0,
+                "NotCovered": 0,
+                "NotCoveredFast": 0,
+                "RestartNeeded": 0,
+                "RebootNeededSince": "0001-01-01T00:00:00Z",
+                "WrapperPassSeconds": 0
+            }
+        }
+    ],
+    "latest_guest_release": {
+        "approved_at": "2026-10-04T12:39:33Z",
+        "approved_by": "auto",
+        "id": "os-guest-20261004-123933"
+    },
+    "latest_host_release": {
+        "approved_at": "2026-10-04T12:40:34Z",
+        "approved_by": "auto",
+        "id": "os-host-20261004-124034"
+    }
+}

documentation/audits/os-host-lane-2026-10-04/partD/after-demo-felhom-agent0.141.1.log

@@ -0,0 +1,172 @@
+=== felhom-agent 0.141.1 selftest=os-update vmid=9201 ring=0 enabled=true guest-release=false host-release=false appliance=true ===
+time=2026-10-04T13:52:57.534+02:00 level=INFO msg="osupdate: START" run=20261004T115257Z layer=guest vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T115257Z
+time=2026-10-04T13:53:09.479+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T115257Z layer=guest:9201 lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T13:53:09.479+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T13:53:09.479+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T13:53:09.479+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T13:53:09.480+02:00 level=INFO msg="osupdate: DONE" run=20261004T115257Z layer=guest vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=6 not_covered=6 restart_needed=0 reboot_needed=false wrapper_seconds=11.9
+time=2026-10-04T13:53:09.491+02:00 level=INFO msg="osupdate: START" run=20261004T115257Z layer=host vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T115257Z
+time=2026-10-04T13:53:19.927+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T115257Z layer=host lane=fast mode=apply select=pending-fast packages=0"
+time=2026-10-04T13:53:19.927+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: REPAIR configured=0 fixed=0"
+time=2026-10-04T13:53:19.927+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: PLAN upgrade=0 already=0 not-installed=0 from-snapshot=0"
+time=2026-10-04T13:53:19.927+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)"
+time=2026-10-04T13:53:20.705+02:00 level=INFO msg="osupdate: DONE" run=20261004T115257Z layer=host vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=80 not_covered=80 restart_needed=34 reboot_needed=true wrapper_seconds=10.4
+  --- os-update report (guest) ---
+  {
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "docker-ce-cli",
+      "containerd.io",
+      "docker-ce",
+      "docker-buildx-plugin",
+      "docker-ce-rootless-extras",
+      "docker-compose-plugin"
+    ],
+    "outcome": "nothing",
+    "pending": 6,
+    "reboot_needed": false,
+    "refused": null,
+    "release_id": "ring0-20261004T115257Z",
+    "restart_needed": null,
+    "ring": 0,
+    "run_id": "20261004T115257Z",
+    "upgraded": [],
+    "wrapper_seconds": 11.9
+  }
+  --- os-update report (host) ---
+  {
+    "health_reason": "",
+    "healthy": true,
+    "mode": "apply",
+    "not_covered": [
+      "frr",
+      "shim-signed-common",
+      "shim-unsigned",
+      "shim-helpers-amd64-signed",
+      "shim-signed",
+      "libradosstriper1",
+      "librgw2",
+      "ceph-common",
+      "librbd1",
+      "librados2",
+      "python3-cephfs",
+      "libcephfs2",
+      "python3-rgw",
+      "python3-rados",
+      "python3-ceph-argparse",
+      "python3-ceph-common",
+      "python3-rbd",
+      "ceph-fuse",
+      "chrony",
+      "libcorosync-common4",
+      "libcfg7",
+      "libcmap4",
+      "libcpg4",
+      "libknet1t64",
+      "libnozzle1t64",
+      "libquorum5",
+      "libvotequorum8",
+      "corosync",
+      "frr-pythontools",
+      "libjs-extjs",
+      "libnvpair3linux",
+      "libproxmox-acme-plugins",
+      "libproxmox-backup-qemu0",
+      "pve-qemu-kvm",
+      "libpve-notify-perl",
+      "libpve-cluster-api-perl",
+      "libpve-cluster-perl",
+      "pve-cluster",
+      "libpve-access-control",
+      "libpve-apiclient-perl",
+      "librados2-perl",
+      "proxmox-backup-client",
+      "proxmox-backup-file-restore",
+      "pve-manager",
+      "libproxmox-acme-perl",
+      "libpve-common-perl",
+      "libpve-guest-common-perl",
+      "qemu-server",
+      "libpve-storage-perl",
+      "pve-edk2-firmware-legacy",
+      "pve-edk2-firmware-ovmf",
+      "libpve-network-api-perl",
+      "libpve-network-perl",
+      "proxmox-firewall-data",
+      "pve-firewall",
+      "pve-container",
+      "pve-ha-manager",
+      "novnc-pve",
+      "proxmox-enterprise-support-keyring",
+      "proxmox-mini-journalreader",
+      "proxmox-widget-toolkit",
+      "pve-docs",
+      "pve-i18n",
+      "pve-xtermjs",
+      "pve-yew-mobile-i18n",
+      "pve-yew-mobile-gui",
+      "libuutil3linux",
+      "libzfs7linux",
+      "libzpool7linux",
+      "proxmox-first-boot",
+      "pve-firmware",
+      "proxmox-kernel-7.0.14-20-pve-signed",
+      "proxmox-kernel-7.0",
+      "proxmox-kernel-helper",
+      "pve-edk2-firmware-aarch64",
+      "pve-edk2-firmware",
+      "zfs-initramfs",
+      "zfsutils-linux",
+      "zfs-zed",
+      "tailscale"
+    ],
+    "outcome": "nothing",
+    "pending": 80,
+    "reboot_needed": true,
+    "refused": null,
+    "release_id": "ring0-20261004T115257Z",
+    "restart_needed": [
+      "agetty",
+      "blkmapd",
+      "chronyd",
+      "cron",
+      "dbus-daemon",
+      "dmeventd",
+      "ksmtuned",
+      "lxc-monitord",
+      "lxc-start",
+      "lxcfs",
+      "pmxcfs",
+      "proxmox-firewal",
+      "pve-firewall",
+      "pve-ha-crm",
+      "pve-ha-lrm",
+      "pve-lxc-syscall",
+      "pvedaemon",
+      "pvedaemon worke",
+      "pvefw-logger",
+      "pveproxy",
+      "pveproxy worker",
+      "pvescheduler",
+      "pvestatd",
+      "qmeventd",
+      "rpcbind",
+      "rrdcached",
+      "smartd",
+      "spiceproxy",
+      "spiceproxy work",
+      "sshd",
+      "systemd-logind",
+      "systemd-udevd",
+      "watchdog-mux",
+      "zed"
+    ],
+    "ring": 0,
+    "run_id": "20261004T115257Z",
+    "upgraded": [],
+    "wrapper_seconds": 10.4
+  }
+  pass took 23.2s
+WALL_SECONDS=23.268350584

documentation/audits/os-host-lane-2026-10-04/partD/before-demo-hp-agent0.140.0.log

@@ -0,0 +1,8 @@
+=== felhom-agent 0.140.0 selftest=os-update vmid=9201 ring=0 enabled=true release=false ===
+time=2026-10-04T13:19:32.683+02:00 level=INFO msg="osupdate: START" run=20261004T111932Z vmid=9201 ring=0 trigger=debug enabled=true release=ring0-20261004T111932Z
+time=2026-10-04T13:19:46.617+02:00 level=INFO msg="osupdate: wrapper" line="os-apply: START release=ring0-20261004T111932Z layer=guest:9201 lane=fast mode=inventory packages=0"
+time=2026-10-04T13:19:46.618+02:00 level=INFO msg="osupdate: DONE" run=20261004T111932Z vmid=9201 ring=0 trigger=debug outcome=nothing healthy=true reason="" upgraded=0 pending=6 not_covered=6 restart_needed=10
+    "outcome": "nothing",
+    "pending": 6,
+    "upgraded": null
+WALL_SECONDS=14.029320302

documentation/audits/os-host-lane-2026-10-04/partE/e0-readonly-demo-hp.txt

@@ -0,0 +1,125 @@
++ uname -r
+7.0.2-6-pve
++ mokutil --sb-state
+SecureBoot enabled
++ od -An -tx1 /sys/firmware/efi/efivars/SecureBoot-8be4df61-93ca-11d2-aa0d-00e098032b8c
++ head -1
+ 06 00 00 00 01
++ ls /boot/vmlinuz-7.0.14-20-pve /boot/vmlinuz-7.0.2-6-pve
+/boot/vmlinuz-7.0.14-20-pve
+/boot/vmlinuz-7.0.2-6-pve
++ dpkg -l 'proxmox-kernel-*'
++ grep '^ii'
++ awk '{print $2,$3}'
+proxmox-kernel-7.0 7.0.14-20
+proxmox-kernel-7.0.14-20-pve-signed 7.0.14-20
+proxmox-kernel-7.0.2-6-pve-signed 7.0.2-6
+proxmox-kernel-helper 9.1.0+fde2
++ proxmox-boot-tool status
++ tail -5
+Re-executing '/usr/sbin/proxmox-boot-tool' in new private mount namespace..
+E: /etc/kernel/proxmox-boot-uuids does not exist.
++ grep -E '^GRUB_DEFAULT|^GRUB_TIMEOUT|^GRUB_SAVEDEFAULT' /etc/default/grub
+GRUB_DEFAULT=0
+GRUB_TIMEOUT=5
++ grub-editenv list
++ '[' -d /sys/firmware/efi ']'
++ efibootmgr
++ head -6
+BootCurrent: 0003
+Timeout: 0 seconds
+BootOrder: 0003,0029,0001,0002,0006,0007,0019,001A,001B,001C,001D,001E,0008,0009,0016,0017,001F,0020,0021,0022,0023,0024,0025,0026,0027,0028,000A,0004,0005,000B
+Boot0001* USB Floppy/CD	VenMedia(b6fef66f-1495-4584-a836-3492d1984a8d,0500000001)0000424f
+Boot0002* USB Hard Drive	VenMedia(b6fef66f-1495-4584-a836-3492d1984a8d,0200000001)0000424f
+Boot0003* proxmox	HD(2,GPT,175383fb-546d-430e-9b4c-73ec2379169d,0x800,0x200000)/File(\EFI\proxmox\shimx64.efi)
++ sysctl kernel.panic kernel.panic_on_oops
+kernel.panic = 0
+kernel.panic_on_oops = 0
++ lsmod
++ grep -i -E 'sp5100|wdt|watchdog'
++ ls -l /dev/watchdog /dev/watchdog0
+crw------- 1 root root  10, 130 Oct  4 09:46 /dev/watchdog
+crw------- 1 root root 243,   0 Oct  4 09:46 /dev/watchdog0
++ wdctl
++ head -12
+Device:        /dev/watchdog0
+Identity:      Software Watchdog [version 0]
+Timeout:       10 seconds
+Timeleft:       9 seconds
+Pre-timeout:    0 seconds
+Pre-timeout governor: noop
+Available pre-timeout governors: noop
+FLAG           DESCRIPTION               STATUS BOOT-STATUS
+KEEPALIVEPING  Keep alive ping reply          1           0
+MAGICCLOSE     Supports magic close char      0           0
+PRETIMEOUT     Pretimeout (in seconds)        0           0
+SETTIMEOUT     Set timeout (in seconds)       0           0
++ dmesg
++ grep -i -E 'sp5100|watchdog'
++ tail -5
+[    0.353227] NMI watchdog: Enabled. Permanently consumes one hw-PMU counter.
++ apt-cache policy proxmox-default-kernel
++ head -4
+proxmox-default-kernel:
+  Installed: 2.1.0
+  Candidate: 2.1.0
+  Version table:
++ apt-cache search --names-only '^proxmox-kernel-[0-9.]+-[0-9]+-pve-signed$'
++ sort -V
++ tail -3
+proxmox-kernel-7.0.14-18-pve-signed - Proxmox Kernel Image (signed)
+proxmox-kernel-7.0.14-19-pve-signed - Proxmox Kernel Image (signed)
+proxmox-kernel-7.0.14-20-pve-signed - Proxmox Kernel Image (signed)
++ cat /proc/cmdline
+BOOT_IMAGE=/boot/vmlinuz-7.0.2-6-pve root=/dev/mapper/pve-root ro quiet
++ grep -nE '^menuentry|^submenu|^\s+menuentry' /boot/grub/grub.cfg
++ cut -c1-140
++ head -8
+26:  menuentry_id_option="--id"
+28:  menuentry_id_option=""
+110:menuentry 'Proxmox VE GNU/Linux' --class proxmox --class gnu-linux --class gnu --class os $menuentry_id_option 'gnulinux-simple-529c0c3d
+128:submenu 'Advanced options for Proxmox VE GNU/Linux' $menuentry_id_option 'gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43' {
+129:	menuentry 'Proxmox VE GNU/Linux, with Linux 7.0.14-20-pve' --class proxmox --class gnu-linux --class gnu --class os $menuentry_id_optio
+147:	menuentry 'Proxmox VE GNU/Linux, with Linux 7.0.14-20-pve (recovery mode)' --class proxmox --class gnu-linux --class gnu --class os $me
+165:	menuentry 'Proxmox VE GNU/Linux, with Linux 7.0.2-6-pve' --class proxmox --class gnu-linux --class gnu --class os $menuentry_id_option 
+183:	menuentry 'Proxmox VE GNU/Linux, with Linux 7.0.2-6-pve (recovery mode)' --class proxmox --class gnu-linux --class gnu --class os $menu
++ ls -l --time-style=+%F_%T /boot/grub/grub.cfg
+-rw------- 1 root root 13743 2026-10-04_09:47:05 /boot/grub/grub.cfg
++ uptime -s
+2026-10-04 09:46:15
++ last -x reboot
++ head -4
+reboot   system boot  7.0.2-6-pve      Sun Oct  4 09:46 - still running
+reboot   system boot  7.0.14-20-pve    Sun Oct  4 09:44 - 09:45  (00:00)
+shutdown system down  7.0.14-20-pve    Sun Oct  4 09:45 - 09:46  (00:00)
+reboot   system boot  7.0.2-6-pve      Fri Aug 21 17:44 - 09:43 (43+15:58)
++ ls /boot/efi/EFI/proxmox/
+BOOTX64.CSV
+fbx64.efi
+grub.cfg
+grubx64.efi
+mmx64.efi
+shimx64.efi
++ cat /boot/efi/EFI/proxmox/grub.cfg
++ head -5
+search.fs_uuid 529c0c3d-b48e-4d01-989d-43fd5d7dbb43 root lvmid/zVqGDa-V6js-HB26-RyZr-d0ft-fUB3-blL75R/iAs9TN-W8rL-ViG8-jptz-1tcY-3djE-MRqamk 
+set prefix=($root)'/boot/grub'
+configfile $prefix/grub.cfg
++ lspci -nn
++ grep -i -E 'smbus|fch'
+00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61)
+00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51)
+06:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 61)
++ modinfo -F filename sp5100_tco
+/lib/modules/7.0.2-6-pve/kernel/drivers/watchdog/sp5100_tco.ko
++ grep -rl sp5100 /lib/modprobe.d /etc/modprobe.d
+/lib/modprobe.d/blacklist_proxmox-kernel-7.0.14-20-pve.conf
+/lib/modprobe.d/blacklist_proxmox-kernel-7.0.2-6-pve.conf
++ grep -h sp5100 /lib/modprobe.d/aliases.conf /lib/modprobe.d/blacklist_proxmox-kernel-7.0.14-20-pve.conf /lib/modprobe.d/blacklist_proxmox-kernel-7.0.2-6-pve.conf /lib/modprobe.d/fbdev-blacklist.conf /lib/modprobe.d/proxmox_prevent_autoload_proxmox-kernel-7.0.14-20-pve.conf /lib/modprobe.d/proxmox_prevent_autoload_proxmox-kernel-7.0.2-6-pve.conf /lib/modprobe.d/systemd.conf /etc/modprobe.d/amd64-microcode-blacklist.conf /etc/modprobe.d/pve-blacklist.conf /etc/modprobe.d/zfs.conf
++ head -3
+blacklist sp5100_tco
+blacklist sp5100_tco
++ modprobe -n -v sp5100_tco
+insmod /lib/modules/7.0.2-6-pve/kernel/drivers/watchdog/sp5100_tco.ko 
++ ls /sys/class/watchdog/
+watchdog0

documentation/audits/os-host-lane-2026-10-04/partE/e1-saved-default.txt

@@ -0,0 +1,20 @@
++ findmnt -no SOURCE,FSTYPE --target /boot
+/dev/mapper/pve-root ext4
++ ls -l /boot/grub/grubenv
+-rw-r--r-- 1 root root 1024 Oct  4 09:46 /boot/grub/grubenv
++ cp -p /etc/default/grub /root/grub.default.bak-2026-10-04
++ sed -i 's/^GRUB_DEFAULT=0$/GRUB_DEFAULT=saved/' /etc/default/grub
++ grep '^GRUB_DEFAULT' /etc/default/grub
+GRUB_DEFAULT=saved
++ update-grub
++ tail -3
+Found memtest86+ 32bit image: /boot/memtest86+ia32.bin
+Adding boot menu entry for UEFI Firmware Settings ...
+done
++ grub-set-default 'gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.2-6-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43'
++ grub-editenv list
+saved_entry=gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.2-6-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43
++ grep -n 'set default' /boot/grub/grub.cfg
++ head -4
+17:   set default="${next_entry}"
+22:   set default="${saved_entry}"

documentation/audits/os-host-lane-2026-10-04/partE/e2-before-reboot1.txt

@@ -0,0 +1,14 @@
++ grub-reboot 'gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-20-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43'
+
+WARNING: Detected GRUB environment block on lvm device
+gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-20-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43 will remain the default boot entry until manually cleared with:
+    grub-editenv /boot/grub/grubenv unset next_entry
+
++ grub-editenv list
+saved_entry=gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.2-6-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43
+next_entry=gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-20-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43
++ uname -r
+7.0.2-6-pve
++ date -u +%FT%TZ
+2026-10-04T12:26:10Z
+reboot1 issued 2026-10-04T12:26:10Z

documentation/audits/os-host-lane-2026-10-04/partE/e3-after-reboot1.txt

@@ -0,0 +1,19 @@
++ uname -r
+7.0.14-20-pve
++ uptime -s
+2026-10-04 14:27:03
++ mokutil --sb-state
+SecureBoot enabled
++ grub-editenv list
+saved_entry=gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.2-6-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43
+next_entry=gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-20-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43
++ cat /proc/cmdline
+BOOT_IMAGE=/boot/vmlinuz-7.0.14-20-pve root=/dev/mapper/pve-root ro quiet
+active
+active
+active
+active
+active
+status: running
+starting
+starting

documentation/audits/os-host-lane-2026-10-04/partE/e4-reboot2.txt

@@ -0,0 +1,33 @@
+reboot2 issued 2026-10-04T12:35:19Z, no grub command; grubenv as after reboot 1
+ssh back after ~40s
+System is going down. Unprivileged users are not permitted to log in anymore. For technical details, see pam_nologin(8).
+
++ uname -r
+7.0.14-20-pve
++ uptime -s
+2026-10-04 14:27:03
++ grub-editenv list
+saved_entry=gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.2-6-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43
+next_entry=gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-20-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43
++ cat /proc/cmdline
+BOOT_IMAGE=/boot/vmlinuz-7.0.14-20-pve root=/dev/mapper/pve-root ro quiet
+--- after the real restart, 2026-10-04T12:36:27Z
++ uname -r
+7.0.14-20-pve
++ uptime -s
+2026-10-04 14:36:11
++ mokutil --sb-state
+SecureBoot enabled
++ grub-editenv list
+saved_entry=gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.2-6-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43
+next_entry=gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-20-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43
++ cat /proc/cmdline
+BOOT_IMAGE=/boot/vmlinuz-7.0.14-20-pve root=/dev/mapper/pve-root ro quiet
++ systemctl is-active pveproxy pvedaemon pvestatd pve-cluster felhom-agent
+activating
+active
+active
+active
+inactive
++ pct status 9201
+status: stopped

documentation/audits/os-host-lane-2026-10-04/partE/e5-final-state.txt

@@ -0,0 +1,27 @@
++ grub-editenv /boot/grub/grubenv unset next_entry
++ grub-set-default 'gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-20-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43'
++ grub-editenv list
+saved_entry=gnulinux-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43>gnulinux-7.0.14-20-pve-advanced-529c0c3d-b48e-4d01-989d-43fd5d7dbb43
++ grep '^GRUB_DEFAULT' /etc/default/grub
+GRUB_DEFAULT=saved
++ dpkg -l 'proxmox-kernel-*-pve-signed'
++ awk '{print $2}'
++ grep '^ii'
+proxmox-kernel-7.0.14-20-pve-signed
+proxmox-kernel-7.0.2-6-pve-signed
++ uname -r
+7.0.14-20-pve
++ systemctl is-active pveproxy pvedaemon pvestatd pve-cluster felhom-agent
+active
+active
+active
+active
+active
++ pct status 9201
+status: running
++ sleep 45
++ pct exec 9201 -- docker inspect -f '{{.Name}} {{.State.Health.Status}}' cloudflared felhom-controller
+/cloudflared healthy
+/felhom-controller healthy
++ sysctl kernel.panic
+kernel.panic = 0

documentation/audits/os-host-lane-2026-10-04/partE/e6-sp5100.txt

@@ -0,0 +1,74 @@
++ modprobe sp5100_tco
+rc=0
++ echo rc=0
++ lsmod
++ grep sp5100
+sp5100_tco             20480  0
++ dmesg
++ grep -i sp5100
++ tail -5
+[   85.993589] sp5100_tco: SP5100/SB800 TCO WatchDog Timer Driver
+[   85.993780] sp5100-tco sp5100-tco: Using 0xfeb00000 for watchdog MMIO address
+[   85.993918] sp5100-tco sp5100-tco: initialized. heartbeat=60 sec (nowayout=0)
+== /sys/class/watchdog/watchdog0
++ for w in /sys/class/watchdog/watchdog*
++ echo '== /sys/class/watchdog/watchdog0'
++ for f in identity state timeout nowayout bootstatus status
+++ cat /sys/class/watchdog/watchdog0/identity
+identity=Software Watchdog
++ printf '%s=%s\n' identity 'Software Watchdog'
++ for f in identity state timeout nowayout bootstatus status
+++ cat /sys/class/watchdog/watchdog0/state
+state=active
++ printf '%s=%s\n' state active
++ for f in identity state timeout nowayout bootstatus status
+++ cat /sys/class/watchdog/watchdog0/timeout
+timeout=10
++ printf '%s=%s\n' timeout 10
++ for f in identity state timeout nowayout bootstatus status
+++ cat /sys/class/watchdog/watchdog0/nowayout
+nowayout=0
++ printf '%s=%s\n' nowayout 0
++ for f in identity state timeout nowayout bootstatus status
+++ cat /sys/class/watchdog/watchdog0/bootstatus
+bootstatus=0
++ printf '%s=%s\n' bootstatus 0
++ for f in identity state timeout nowayout bootstatus status
+++ cat /sys/class/watchdog/watchdog0/status
+status=0x8000
+== /sys/class/watchdog/watchdog1
++ printf '%s=%s\n' status 0x8000
++ for w in /sys/class/watchdog/watchdog*
++ echo '== /sys/class/watchdog/watchdog1'
++ for f in identity state timeout nowayout bootstatus status
+++ cat /sys/class/watchdog/watchdog1/identity
+identity=SP5100 TCO timer
++ printf '%s=%s\n' identity 'SP5100 TCO timer'
++ for f in identity state timeout nowayout bootstatus status
+++ cat /sys/class/watchdog/watchdog1/state
+state=inactive
++ printf '%s=%s\n' state inactive
++ for f in identity state timeout nowayout bootstatus status
+++ cat /sys/class/watchdog/watchdog1/timeout
+timeout=60
++ printf '%s=%s\n' timeout 60
++ for f in identity state timeout nowayout bootstatus status
+++ cat /sys/class/watchdog/watchdog1/nowayout
+nowayout=0
++ printf '%s=%s\n' nowayout 0
++ for f in identity state timeout nowayout bootstatus status
+++ cat /sys/class/watchdog/watchdog1/bootstatus
+bootstatus=0
++ printf '%s=%s\n' bootstatus 0
++ for f in identity state timeout nowayout bootstatus status
+++ cat /sys/class/watchdog/watchdog1/status
+status=0x0
++ printf '%s=%s\n' status 0x0
++ rmmod sp5100_tco
+rmmod_rc=0
++ echo rmmod_rc=0
++ lsmod
++ grep -c sp5100
+0
++ ls /sys/class/watchdog/
+watchdog0

documentation/audits/os-host-lane-2026-10-04/partG/wrapper-copied-by-hand.txt

@@ -0,0 +1,4 @@
+demo-felhom wrapper 4729769ce32e25e6 755 root; sudoers 02df92d751f1780a
+demo-hp wrapper 4729769ce32e25e6 755 root; sudoers 02df92d751f1780a
+demo-felhom wrapper 51e100ad81945f67
+demo-hp wrapper 51e100ad81945f67

documentation/backlog/CLOSED-ITEMS.md

@@ -26,6 +26,47 @@
 
 ---
 
+## 2026-10-04 (evening) — System page, Docker slow lane, crash guard (agent v0.142.0, hub v0.132.0, installer 1.30.0)
+
+> Evidence: `audits/os-docker-crash-2026-10-04/`.
+
+| Row | What | Closed | Evidence |
+|---|---|---|---|
+| **R-852** | **The operator could see no box's Debian, Proxmox, kernel or Docker version anywhere.** The box reports them (`system` stanza: Proxmox + kernel from the API, the wrapper's read-only `facts`); hub v0.132.0 shows them on the new **System** page (with the ring / switch / approve buttons) and a Proxmox / kernel column on Hosts. **Reasoning kept: a value the box could not read says `unknown`, amber, with the reason — never empty, never guessed.** | CLOSED 2026-10-04 — FIXED (`09` decision 89) | `partA/` |
+| **R-835** | **Turning Docker's live-restore OFF by a restart stops every container and starts none.** Resolved by design (decision 87): live-restore is ON everywhere — the golden bakes it, installed boxes get it once by a RELOAD (measured: same ids on 9202 6/6, demo-hp 24/24, demo-felhom 5/5), and the wrapper refuses a Docker step while it is off (R15). **Rule kept: never turned off by a plain restart.** | CLOSED 2026-10-04 — RESOLVED BY DESIGN | `partB/b1..b4` |
+| **R-848** | **A held host package was invisible to the hub.** The facts report `held`; the System page shows it amber. | CLOSED 2026-10-04 — FIXED | `partA/` |
+| **R-849** | **The guest's "restart needed since" never cleared.** The wrapper scans the guest on every pass too (agent v0.142.0). | CLOSED 2026-10-04 — FIXED | `partB/agent-redproofs.txt` |
+| **R-851** | **A host that panicked stayed stopped (`kernel.panic = 0`).** The crash guard (decision 88): `kernel.panic = 10`; the 3rd unclean stop within 60 minutes leaves the box off; re-arms after 24 h or by `felhom-crash-guard rearm`. MEASURED on demo-hp: crash 1 and 2 restarted by themselves (54 s, 53 s), the guard tripped, crash 3 stayed off until the operator switched it on; the hub mailed the trip. **Reasoning kept: a crash, a power cut and a hard reset cannot be told apart on these boxes (pstore saved nothing for a real panic) — the guard counts every unclean stop.** | CLOSED 2026-10-04 — FIXED | `partC/` |
+| **R-854** | **The first Docker steps were stored as GUEST reports** — they ran (agent rc) before hub v0.132.0 was live, and the older hub maps an unknown layer to `guest`; the guest candidate briefly read 0 packages. One-time; a pass with the released agent re-reported both layers. Nothing to fix. | CLOSED 2026-10-04 — ONE-TIME, NO ACTION | `partB/b5-*` |
+
+## 2026-10-04 (afternoon) — OS updates, host fast lane + fleet view + alarms (agent v0.141.0/v0.141.1, hub v0.131.0/v0.131.1, controller v0.292.0)
+
+> Evidence: `audits/os-host-lane-2026-10-04/`.
+
+| Row | What | Closed | Evidence |
+|---|---|---|---|
+| **R-841** | **Every box reported its tunnel `inactive`** (the agent asked a host unit that does not exist). Agent v0.141.0 reads the guest's `cloudflared` container and controller v0.292.0's Docker health check on cloudflared's own `/ready` (200 only with a connection); three states `running` / `not_running` / `unknown`; hub v0.131.0 alarms `tunnel_down` after two `not_running` reports and `tunnel_recovered` on the next `running`. LIVE on demo-hp: port 7844 blocked → `tunnel_down` mailed after the 2nd report; unblocked → `tunnel_recovered`. **Reasoning kept: `unknown` never alarms; a container state alone says "up" for a dead tunnel (wrong token: running, `/ready` 503). A plain `docker stop` is healed by the controller's protected-container check within 5 min — before the 15-min host report sees it.** | CLOSED 2026-10-04 — FIXED | `partA/` |
+| **R-845** | **The OS leg was slow.** One `pct exec` per package (~0.9 s each) replaced by one call per layer; restart scan only after an install (host: every pass, v0.141.1); repair only when `dpkg --audit` reports. MEASURED: nothing to install, both layers — 23.3 s (demo-felhom), 31.5 s (demo-hp); before, guest only, nothing to install — 14.0 s; the 174–245 s passes of R-845 were passes WITH an install. A 108-package host install pass: 70 s. | CLOSED 2026-10-04 — FIXED | `partD/`, `partB/live/` |
+| **R-846** | **Host "reboot needed" was wrong in agent v0.141.0** (found live on demo-felhom): the restart scan skipped every cgroup containing `lxc`, hiding `lxc-start` (`0::/lxc.monitor/<vmid>`, 20 deleted maps after libc6); and it scanned only after an install, so a reboot never cleared it (the hub's 14-day alarm would fire on a rebooted host). Agent v0.141.1 (`:/lxc/`, host scans every pass, `reboot_scanned`) + hub v0.131.1. Red-proved. | CLOSED 2026-10-04 — FIXED | `partB/live-defects-redproofs.txt` |
+| **R-850** | **Hub v0.131.0 put the layer into the release fingerprint**, so the unchanged guest set (272 packages) counted as new and waited its 24 h again. One-time; ring 1 kept the previous release meanwhile; approved again under the TEST wait (`os-guest-20261004-123933`). Nothing to fix. | CLOSED 2026-10-04 — ONE-TIME, NO ACTION | hub log 2026-10-04 |
+
+## 2026-10-04 (~12:20) — operator ruling
+
+| Row | What | Closed | Evidence |
+|---|---|---|---|
+| **R-842** | **The guest OS update had no automatic undo (no snapshot of a guest with host binds is possible).** Ruled option A (`09` §3 decision 81): the whole-guest backup, minutes old, restored by hand, is the undo. Nothing new built; a home-made LVM-thin snapshot rejected (unmeasured, thin-pool risk). | CLOSED 2026-10-04 — RULED (decision 81) | `audits/os-guest-lane-2026-10-04/partA/README.md` |
+
+## 2026-10-04 (day) — OS updates, guest fast lane (agent v0.140.0, hub v0.130.0, controller v0.291.0)
+
+> Evidence: `audits/os-guest-lane-2026-10-04/`.
+
+| Row | What | Closed | Evidence |
+|---|---|---|---|
+| **R-837** | **The guest snapshot undo was unmeasured.** MEASURED: impossible — PVE refuses any non-`vzdump` snapshot of a guest with host-path binds (mp8/mp9), as the agent's token (which has the rights) and as root (`PVE/AbstractConfig.pm:755-757`). The automatic undo was not built (the brief's stop rule); the decision is R-842. **Rule kept: a guest with host binds has no PVE snapshot; the night's backup is its undo.** | CLOSED 2026-10-04 — MEASURED | `partA/README.md` |
+| **R-838** | **The infrastructure images never moved** (cloudflared four months behind). Controller v0.291.0: traefik v3.7.13, cloudflared 2026.9.3, filebrowser 1.5.6-stable (release notes read; nothing we use breaks). A controller release DOES move all three (bring-up + start-up mount sync; measured on 9202 and both demo boxes, public gap ≤ 19.6 s); `scripts/check-infra-pins.py` + the runbook's "Infrastructure pins" section put them on the monthly re-test. | CLOSED 2026-10-04 — FIXED controller v0.291.0 | `partF/` |
+| **R-726** | **A returning household's new box made no off-site copy on night one.** Decision 78 built (controller v0.291.0): a claimed box that has never made an off-site copy sets the orphaned old copy aside and starts a new one; nothing deleted; a box that has made copies still asks. Red-proved both ways. | CLOSED 2026-10-04 — FIXED controller v0.291.0 | `partE/r726-redproof.txt` |
+| **R-843** | **`--selftest=wgtunnel` (since S3) and `--selftest=os-update` were dispatched but refused by the flag's allow-list.** Found live (the OS debug action could not run); both accepted in agent v0.140.0; `TestSelftestFlag_AcceptsEveryDispatchedMode` pins every dispatched mode. **Rule kept: a dispatch case without a flag case is dead code — the test reads both.** | CLOSED 2026-10-04 — FIXED agent v0.140.0 (opened and closed the same day) | `partC/agent-leg-redproofs.txt` |
+
 ## 2026-10-04 (day) — the off-site topic closed (hub v0.129.0, agent v0.139.0)
 
 > Evidence: `audits/backup-close-2026-10-04/`.

documentation/backlog/OPEN-ITEMS.md

@@ -191,7 +191,7 @@ stopping line that lies.
 | **R-687** | App updates | P4 | **[P3-LOW] Part 7's live proof has four gaps a scratch box cannot close, and one observability gap.** (1) **W+5h reached with steps left** is proven by unit test only (`TestLeg_NoStepAtOrAfterW5h`) — the leg starts at W+105m and would need a 3-hour leg live; (2) **the off-site leg FAILING** before the update leg: 9202 has no off-site tier, so only the "no target" path ran live — failure and panic are `TestChainUpdateLeg_EveryPath`; (3) **a `files_may_change` step WITHOUT a whole copy**: both drill apps given the mark (wishlist, navidrome, romm) turned out whole on 9202 by the backup side's truth table (why, per app, is not logged — see the observability gap), so only "with a copy" ran live; (4) **the full-system gate waiting** cannot run on 9202 (no agent), and **did not occur on the demo boxes' real night either** (2026-09-25: both legs ended by 04:19, before the gate opened at 04:30, and no whole-box backup was due on either box) — unit + red-proof only (`TestD20_GateWaitsForTheLeg`). **Also cosmetic:** a leg with no steps reports `"steps": null` to the hub, not `[]`. **Observability:** when the leg TAKES a `files_may_change` step it does not log which whole copy allowed it (only the skip says why). `audits/night-2026-09-25/C/` **-- NARROWED 2026-09-25 (controller v0.273.0):** the cosmetic `"steps": null` → `[]` and the taken `files_may_change` step's missing log line are FIXED (red-proofed, `audits/night-2026-09-26/F/`). Items (1)–(4) stay; (4) did not occur on 2026-09-25 either (demo-felhom's whole-box backup ran at 07:29, three hours after its leg; demo-hp had none due). **-- 2026-09-28 (night 27/28):** (4) did not occur again — on demo-hp the leg ended 04:23:54 and the whole-guest backup began 04:37:06, after the gate opened at 04:30; demo-felhom's backup ran at 07:36 (`audits/evidence-golden-0276-2026-09-28/phaseD2-night-read.txt`). **-- 2026-09-30 (by day, demo-hp 9201): item (4) PROVEN LIVE.** The night chain pressed by hand, the window moved to W = now − 2h05m the moment the leg started, `quiesce.poll_interval` 1m: `[quiesce] full-system backup due and inside its window, but the automatic update leg is running … deferring` at 11:35:11 and 11:36:11 UTC while bookstack (55.1 s) and kimai (75.1 s) stepped; the leg's end line at 11:36:29; the backup quiesced at 11:37:11 (the first poll after), job done 11:47:19, the agent's `backup: completed` 9.98 GB. Config and window put back and read back (`audits/pg-last-six-2026-09-30/C/`). **Found, cosmetic, manual chain only:** the deferral names the moved window's W+5h (16:29) while the manual leg's own deadline was its start + the leg length (16:49). | **OPEN — P3, gaps (1)–(3) + the manual-chain deferral text; item (4) proven live 2026-09-30; owner: CC** **Re-ranked 2026-10-03: P3→P4: the gaps are covered by unit tests; left is live-proof completeness and one log text.** | — | — | CC |
 | **R-734** | App updates | P4 | **[P3-LOW] The harness marks immich `files_may_change` because immich rewrites six 13-byte `.immich` folder markers at every start.** MEASURED 2026-09-30 on the bench (v3.2.2 → v3.2.4): the bind-tree hash of `appdata/immich` changed; the only changed files were `{encoded-video,library,backups,profile,thumbs,upload}/.immich`, rewritten at each start — no household file. The mark is honest by the harness's rule and the ladder writer copies it (never edited by hand), so immich's v3.2.4 night step needs a fresh WHOLE copy (decision 13); on a box without one the night leg skips it and a person presses. The 2026-09-23 immich entry did not carry it (`files_changed []`). **Needs:** a decision whether app-owned marker files are excluded from the file hash (a per-template ignore list, or a size/name rule), or the mark stays. **-- 2026-09-30 (evening):** the harness now NAMES the files behind the mark (`files_changed_detail`, catalog `5b1972b`); on immich's step `0b82…` re-proof it named exactly the six `.immich` markers again. calibre-web's step (v4.0.6 → v4.0.8) carries the mark too, and the named files are its LIBRARY DATABASE: `media/books/metadata.db`, `metadata.db-shm`, `metadata.db-wal` changed; the book file did not (bench names re-run, `A/calibre-names/`). That is household data (the template's backup class `mandatory` holds DB and books as one unit), so the mark is right there and the night leg takes that step only with a fresh whole copy. On 9202 the same step changed no file in that folder (per-file hashes before/after) — not explained. `audits/more-night-apps-2026-09-30/` | **READY — rank P3-LOW; owner: CC (harness); the rule change needs a word** **Re-ranked 2026-10-03: P3→P4: the effect is an update that waits for a press; no data risk.** | — | — | CC + operator |
 
-## Backup & restore — 55 rows (P2 11, P3 23, P4 21)
+## Backup & restore — 54 rows (P2 10, P3 23, P4 21)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -204,7 +204,6 @@ stopping line that lies.
 | **R-518** | Backup & restore | P2 | **[P2-MEDIUM] „Mentés most" on the whole-system backup stops every app for about eight minutes while the page promises „csak néhány másodpercre".** MEASURED 2026-09-14 (BIGNIGHT, VM 333, 12 apps): the button's call quiesced all 12 stacks at 19:03:23Z (first stopped 19:03:27Z); the local vzdump ran 19:03:49 → 19:09:59Z; the controller then kept the apps stopped for the second (PBS) tier and restarted them at 19:10:09Z after it failed, the last started 19:11:12Z (`phase4/guest-backup-quiesce-log.txt`) — **≈ 7 m 45 s** with every app answering 404. The page under the button: „Pillanatkép-mód: az alkalmazások csak néhány másodpercre állnak le." A household pressing it at dinner loses every app for the length of the dump, and longer on a bigger box. **Fix shape:** state the real expected downtime (it scales with data), or quiesce per tier and not across a second tier's attempt; do not start a tier whose storage is absent (see R-517). **NARROWED 2026-09-15 (controller v0.243.0 + agent v0.131.0):** a tier whose storage the agent reports absent is skipped before anything stops (`backup_tier_skipped`, once per absence; unknown never skipped), and the button copy now says „általában néhány perc, nagyobb adatnál több". Unit-proven with red-proofs. **Still open:** quiesce per tier, so a slow second tier does not keep every app down. **— NIGHT 2026-09-23 (controller v0.267.0):** the copy half is DONE: the page and the confirm now state the measured stop (≈ 8 minutes on a 12-app box), both languages, red-proofed (`audits/night-2026-09-23/A5-*`). The brief's „csak néhány másodpercre" had already gone in v0.243.0. **Still open:** quiesce per tier, so a slow second tier does not keep every app down. | **READY — P2, narrowed to per-tier quiesce; owner: CC (controller)** | — | — | CC |
 | **R-519** | Backup & restore | P2 | **[P2-MEDIUM] After a backup torn by a power cut, an app's restore point carries the new database dump's time while its files are from the previous run — and no customer screen says the run was interrupted.** MEASURED 2026-09-14 (BIGNIGHT F2, VM 333): „Mentés most" 19:40:03Z; the power was cut 19:40:08Z while adventurelog was stopped for its volume dump. On disk afterwards, `backups/primary/adventurelog`: `db-dumps/adventurelog-postgres.sql` **19:40:07**, `volume-dumps/*.tar` **19:00:35**, `manifest.json created_at 19:02:34Z`; bookstack the same shape (sql 19:40:07, tars 19:00:48). `GET /api/backup/snapshots` for both → `time 2026-09-14T19:40:07Z, helyi` (`phase5/F2/units-on-disk.txt`, `backup-honesty.txt`). `/backups/apps` shows „Utolsó adatbázis mentés 2026-09-14 21:40 · … OK" and every app „Utolsó: 5 perce"; `/backups` and `/dashboard` contain no word of an interruption (fragments `megszakad|sikertelen|nem sikerült` = 0; control „hiba" appears in the standing warning text). The controller itself knew: `[appstop] crash recovery: an app-data backup (volume dump) … was interrupted … restarting them: [adventurelog]` and pushed `backup_failed (error)` to the hub. A household restoring „the 21:40 backup" gets 21:00 files for BookStack's uploads. **Fix shape:** date a point by the oldest part it contains (or mark it partial) and show the interrupted run on the backups page until the next complete one. **F6 (drive unplugged 1 s into a backup, 20:37:59Z) adds three facts:** the run skipped four apps' volume dumps („Skipping volume dump for immich — drive disconnected", also jellyfin, nextcloud, paperless-ngx) and still reported `db_dump {"count":6, "success":true}`; nextcloud's point is dated 20:37:59Z (its SQL finished before the unplug) beside 19:01 volume tars; and a torn `immich-postgres.sql.tmp` (20:38:00) plus F3's `pre-restore-…-nextcloud-mariadb.sql.tmp` are left in the units on the drive. Immich's point correctly stayed at 19:02:34Z (the `.tmp` was not promoted). `/backups/apps` fragments `kihagy|sikertelen|részleges` = 0 (`phase5/F6/`). | **READY — rank P2-MEDIUM; owner: CC (controller)** | — | — | CC |
 | **R-638** | Backup & restore | P2 | **[P2-MEDIUM] The product's database loader cannot replay a copy over a NEWER schema: on PostgreSQL it FAILS, on MariaDB it leaves the newer version's tables behind.** MEASURED 2026-09-23 on 9202. `ImportDump` (`appbackup/dbdump.go:719`, `psql -v ON_ERROR_STOP=1 --single-transaction`) replays a `pg_dump --clean --if-exists` file over the live database. After docmost 0.95.0 → 0.96.0 migrated, the replay of the pre-update copy was refused in 0.40 s, rc 3: *cannot drop constraint workspaces_pkey on table public.workspaces because other objects depend on it / DETAIL: constraint oauth_clients_workspace_id_fkey …* — the new version created six tables whose foreign keys point at old ones, and `--clean` only drops what the dump knows. Database unchanged (the transaction rolled back). On MariaDB (`mariadb-dump`, `FOREIGN_KEY_CHECKS=0`) the same replay after romm 5.0.0 → 5.3.0 returned rc 0 in 1.25 s and left **12 base tables** of the new version behind; RomM 5.0.0 happened to ignore them. **What worked:** `DROP SCHEMA public CASCADE; CREATE SCHEMA public;` + the dump in ONE transaction — rc 0 in 1.38 s, every table, index and extension back. **Why this is a row of its own and not only part of R-637:** the SAME loader backs shipped paths — `rollbackSafetyDump` (off-site restore's undo) and the dump replay of the restores — so **any restore of a copy taken BEFORE an update that migrated, replayed over the migrated database, may fail the same way. NOT MEASURED:** whether the unit restore the hold sentence names does this (it also carries the data VOLUME tar, which may make the replay moot). That is the measurement owed, on 9202, before anyone relies on it. Evidence: `audits/update-rulings-2026-09-23/README.md` Part 1, `docmost-45`, `romm-44`. **-- NARROWED 2026-09-23:** the undo no longer touches this loader — it copies folders (decision 19, controller v0.263.0). **What stays open is the part about SHIPPED paths:** `rollbackSafetyDump` and the restores' dump replay still replay over whatever schema is live, and whether the unit restore the hold sentence names works after a real schema migration is STILL UNMEASURED. | **OPEN — P2, narrowed to the restore paths; owner: CC; measure the named restore after a real schema migration first** | — | — | CC |
-| **R-726** | Backup & restore | P2 | **[P2-MEDIUM] A new box for a customer who had one before makes NO off-site copy on night one: the old repository is found orphaned, and the fix is a button nobody pointed the household to.** MEASURED 2026-09-30 00:15 UTC on the new-household drill box (`tester-1`, whose previous box was deleted 2026-09-17): `[offbox] offsite repo ORPHANED — remote holds backups written under a previous, no-longer-available key; runs will skip until reset` → `offbox_repo_orphaned` (warning) to the household's timeline and an operator mail; `offsite-integrity` then checked nothing. The household's page is honest („A távoli tároló másik kulccsal készült mentéseket tartalmaz … Új távoli mentés indítása…", old history set aside, never deleted), but the evening before, the recovery-code ceremony and the off-site page raised nothing, and the guide does not mention it. The hub re-issued the off-site credentials on re-enroll by itself; it could have known the repository would orphan. Customer data was never at risk (the old repository is untouched); the household simply has no off-site copy until someone presses the button. **Fix direction:** offer the reset at the recovery-code ceremony when the repository already holds another key's snapshots, or the hub's re-enroll re-issue sets the old history aside the same way (it is the same move-aside), and the guide says so. | **WAITING-ON-OPERATOR 2026-10-04 — two options in STATUS (A: a fresh box sets the old history aside by itself on its first night, as an unclaimed box already does; B: ask at the recovery-code evening). CC's pick: A. Nothing built.** | — | — | CC + operator |
 | **R-822** | Backup & restore | P2 | **An add-only key does not make retention safe: an attacker who can only ADD snapshots can make the honest pruner erase every real one.** MEASURED 2026-10-03 (lab, restic 0.14.0 from the controller image, rclone `--append-only`): 13 empty snapshots dated in the future with the same host and tag, added through the add-only key (all allowed), make the box's exact policy `forget --group-by host,tags --keep-daily 7 --keep-weekly 4 --keep-monthly 6` keep only the fakes and select **all 3 real snapshots** for removal (`--dry-run`). **Whoever prunes an append-only repo — the box in a window, or a Felhom-side worker — inherits this.** Not a defect today (today the box can simply delete, R-95); a PRECONDITION on the R-95 build, like R-430 was. `audits/offsite-append-only-2026-10-03/lab/C3-retention-poisoning.txt` | **NARROWED 2026-10-03 — the guard ships in controller v0.289.0 (future-dated / newer-than-hub / recent-removal refusals, oldest-first cap, the lab's 13-fake shape refused in a test). RESIDUAL, not closable by a guard: an add-only attacker can plant PAST-dated snapshots interleaved with real ones and so steer weekly/monthly keeps; bounded per window by `MaxRemove` and the hub's count check, not prevented.** | — | Before any `forget`: refuse when a snapshot is dated in the future or newer than the newest the hub has seen reported; dry-run first and abort above the count the policy can remove in a week; hub compares the count before/after (DESIGN.md §3) | CC |
 | **R-49** | Backup & restore | P3 | **[P2] The offsite capture set is ~90% cache and duplication — 1.1 GB of a 1.2 GB immich "photo backup".** Measured 2026-07-19: `immich_ml_cache.tar` **823 660 032 B (~60%)** — re-downloadable ML model weights; `immich_postgres_data.tar` **308 251 136 B (~23%)** — a raw tar of the postgres data dir that DUPLICATES the logical `.sql` dump captured beside it; `upload/backups/` **18 MB** — immich's own nightly dump, a backup inside the backup, growing daily; plus the stranded pre-v3 `dccc13fe…` tree (~36 MB) no DB has ever referenced. Actual irreplaceable content: **72 MB of originals**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-19, size S–M, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** **Re-ranked 2026-10-03: P2→P3: wasted space and transfer, no data risk; needs a capture-set ruling, not a sale blocker.** | — | **Evidence: `audits/DIAG-immich-restore-round2-2026-07-19.md` §4 (full byte breakdown).** This is the customer's offsite quota and transfer cost, and it lands on the Hetzner sub-account they are billed for. **Recorded, deliberately not changed** — a capture-set exclusion is a data-loss-shaped decision and gets its own ruling, not a drive-by edit. Candidates in priority order: (a) `immich_ml_cache` — pure cache, strongest case; (b) the `postgres_data` volume tar where a logical dump of the same DB is already captured (the dump is what the restore path actually replays); (c) `upload/backups/`. Likely generalises past immich into a template-classification rule about cache volumes and self-backup directories, so it should be specified against the catalog, not one app | CC |
 | **R-127** | Backup & restore | P3 | **The catalog's `data_key: true` flag is UNRELIABLE — at least four data-encrypting keys the catalog itself labels as encryption keys are unflagged; and the O4 restore path can regenerate a DB password that then does not match the restored data directory** | **READY (S/M)** | — | **Found by D5's Part 0, and it is why D5's boundary is `type: secret` rather than `data_key`.** Two separable legs. **(a) The misclassification.** Only 5 fields across 4 apps set `data_key: true` (`adventurelog/SECRET_KEY`, `homebox/HBOX_AUTH_API_KEY_PEPPER`, `papra/AUTH_SECRET`, `sparkyfitness/{API_ENCRYPTION_KEY,BETTER_AUTH_SECRET}`), yet `n8n/N8N_ENCRYPTION_KEY` („Titkosítási kulcs"), `wanderer/POCKETBASE_ENCRYPTION_KEY` („Adatbázis titkosítási kulcs"), `calcom/CALENDSO_ENCRYPTION_KEY` and `bookstack/APP_KEY` are unflagged — the catalog's own Hungarian labels contradict the flag. **D5 makes this non-urgent but not harmless:** everything `type: secret` now travels, so the keys DO reach the drive; what stays wrong is the **fail-closed gate**, which only refuses for `data_key` names — so if one of these is missing from both sources the restore proceeds onto data it cannot decrypt instead of refusing. Fix = flag them (`app-catalog-felhom.eu`, a catalog-only change) + a gate/test that the flag set and the label set agree. **(b) The regenerated-DB-password trap.** `internal/backup/restore_unit.go` O4 generates a replacement for any missing non-data-key secret. Proven on `postgres:16-alpine`: with PGDATA restored from the volume tar, `POSTGRES_PASSWORD` is ignored (initdb skipped), so the app fails over the compose network while the dump replay still succeeds through the container's local **trust** socket — success reported, data unreachable. v0.188.0 corrected the WARN's false claim that *"stored data is unaffected"* and scoped it, but did **not** add a guard: D5 shrinks this to the rare case (the secret was empty at capture AND absent from the guest). Real fix = either treat a DB password as fail-closed like a data key, or `ALTER USER` to the regenerated value after the volume restore. 18 DB/root-password fields are in scope; MariaDB fails loudly instead (Access denied), which is the safer half | CC |
@@ -268,7 +267,7 @@ stopping line that lies.
 | **R-368** | Storage & devices | P4 | **The storage default DOES apply at deploy time — the earlier claim that it never does was wrong, and the residual defect is smaller and different.** R-352 and `SPEC-app-data-placement-2026-08-21.md` §2.2 stated *"the deploy route never reads it"*, from `grep -nE 'GetDefaultStoragePath|primaryHDDPath|IsDefault' internal/stacks/deploy.go internal/stacks/manager.go` → nothing. **That grep searched Go files only and never the templates.** `internal/web/templates/deploy.html:612` reads `.IsDefault` directly off each `DeployStoragePath` (which embeds `settings.StoragePath`, `web/handlers.go:89-99`) and **pre-selects the default drive for a new deploy**: `{{else if and .IsDefault (not .NotAllowed)}}selected{{end}}`. So `// new apps use this by default` (`settings.go:453`) is **IMPRECISE ABOUT THE MECHANISM, NOT FALSE** — nobody calls `GetDefaultStoragePath()` on that route, but the value is honoured. The customer-facing label promises exactly this and no more: **„Legyen alapértelmezett új telepítéseknél"** (`storage.html:469`). **THE RESIDUAL, and it is the whole finding:** the default lives in the TEMPLATE, not in the server. `POST /api/stacks/<n>/deploy` accepts `values` verbatim; omit `HDD_PATH` and `withPathVars` (`stacks/deploy.go:584`) receives `""` and no default is applied. **That is why the invariant has no test — there is nothing server-side to test.** | **OPEN — LOW** | corrects R-352(2); supersedes SPEC §2.2 | Either move the default into the server so the API and the form agree and a test can pin it, or reword the comment to say the template owns it. Do not "fix" the behaviour: it is correct on the path customers use. | CC |
 | **R-568** | Storage & devices | P4 | **[P3-LOW] The dashboard's drive-health rows swap order between visits — the same two disks, listed in a different order a minute apart.** MEASURED 2026-09-17 on demo-hp 9201 during slice 1 release C's live proof: `/dashboard` fetched on 0.249.0 listed „KXG50PNV1T02 NVMe TOSHIBA 1024GB” then „SanDisk X600 M.2 2280 SATA 128GB”; fetched on 0.250.0 a minute later, the reverse (`audits/i18n-slice1-2026-09-17/C/live/hu-before-vs-after.txt`). `diskHealthRows` (`disk_health.go` L135–141) keeps the agent's response order and does not sort; the agent's order is therefore not stable. Cosmetic, but a household that reads „the second disk” finds a different one. **Fix shape:** sort the rows controller-side by a durable key (device path or serial), with a test that feeds two orders and expects one. | **READY - rank P3-LOW; owner: CC** **Re-ranked 2026-10-03: P3->P4: cosmetic.** | — | — | CC |
 
-## Security & access — 31 rows (P2 4, P3 24, P4 3)
+## Security & access — 30 rows (P2 3, P3 24, P4 3)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -302,15 +301,14 @@ stopping line that lies.
 | **R-134** | Security & access | P4 | **Two zone-resolvers disagree on depth.** The controller strips labels progressively (`controller/internal/cloudflare/zone.go:18`); the hub's `resolveZone` tries the exact name then `parentDomain`, which strips exactly ONE label (`hub/internal/cloudflare/unblock.go:115,136`) | READY (XS) | — | For a one-label Felhom-issued subdomain both work; for anything deeper the hub silently fails to find the zone while the controller succeeds — the geo-unblock would then no-op with a "no active zone found" error. One concept, two implementations. Same audit §2.6 | CC |
 | **R-525** | Security & access | P4 | **[P3-LOW] FileBrowser has its own login; putting it behind the dashboard session (traefik forwardAuth or Quantum proxy auth) is a new mechanism nobody has measured.** Filed 2026-09-15 by the P1-fixes task (B.5). R-513 closed the default-password hole with a generated password; a household still has two logins. **What it needs:** a spike on a scratch guest — forwardAuth to the controller session, and what FileBrowser Quantum does with a trusted header. | **READY — rank P3-LOW; owner: CC (spike)** **Re-ranked 2026-10-03: P3->P4: comfort feature needing a new unmeasured mechanism; the default password hole is closed.** | — | — | CC |
 | **R-779** | Security & access | P4 | **[P3-LOW] Part A's "two outside addresses seen as two" is proven through the simulated tunnel only; on the REAL tunnel the second outside address (ep0, one request allowed) was refused by Cloudflare's edge with 403 and never reached the box.** Measured 2026-10-01 19:51 UTC (`audits/visitors-2026-10-01/A/L2-demo-hp-real-tunnel.txt`): no log line on demo-hp; demo-hp's box has no geo restriction in its settings, so a Cloudflare ZONE rule (country or bot, not read) refused a German datacenter address. DooPlex's own address on the real tunnel was seen as itself. **Needs:** one sign-in from a second Hungarian address (the operator's phone off wifi) while DooPlex is locked out — 2 minutes; and say which Cloudflare rule refused ep0. | **WAITING-ON-OPERATOR — rank P3-LOW; owner: operator (a phone), CC reads the logs** **Re-ranked 2026-10-03: P3→P4: a proof gap on the real tunnel; operator-only follow-up.** | — | — | CC + operator |
-| **R-838** | Security & access | P2 | **`cloudflared` — the internet-facing tunnel on every box — never moves: pinned at `2026.6.0` since 2026-06-11; upstream is `2026.9.3`.** MEASURED 2026-10-04 (`11` C8, Q9): it is not a host package but a container in the guest, pinned in `controller/internal/infra/infra.go:26` and baked into the golden; traefik and filebrowser are pinned the same way. Nothing re-tests or raises these pins on a schedule; the app-image update arc (`09`) covers catalog apps, not these. Fix direction: put the infrastructure pins on the same monthly re-test as app images, raised by a controller release. `audits/os-updates-spike-2026-10-04/README.md` (Q9) | **READY — owner: CC (pin raise) + operator (cadence)** | — | — | CC |
 
-## Box system & updates — 20 rows (P2 3, P3 14, P4 3)
+## Box system & updates — 20 rows (P2 4, P3 13, P4 3)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
 | **R-530** | Box system & updates | P2 | **[P2-MEDIUM] A floor does not deliver an agent: agents update only by an operator-signed `agent_update` job per box, and nothing records which boxes still run 0.130.0.** MEASURED 2026-09-15: the hub HOLDS a floor whose declared MinAgent is above the box's agent (`api/handler.go` ResolveManagedFloor); the agent's only update path is `signedjobs` + `selfupdate.Executor`. demo-hp reached 0.131.0 by `felhom-opsign -op agent_update` (key `felhom-op-1`) at 08:44:16Z and its controller floor was then SERVED in 3 s. **demo-felhom (N100) and Peti's box still run 0.130.0** — not touched (Peti fenced; N100 not asked). **What it needs:** the operator signs per box, or rules a fleet rollout step. **NARROWED 2026-09-16 (operator ruling 1):** the keys stay on DooPlex owner-only and CC may sign `agent_update` until the first PAYING customer (testers excluded) — recorded in `CONTEXT.md` + `04-control-plane-authorization.md` §3.1. Both demo boxes now run agent 0.131.0 (demo-hp 2026-09-15, demo-felhom 2026-09-16, each by a per-box signed job; Peti's box untouched, still 0.130.0). **What remains:** a fleet rollout step — signing per box does not scale past a handful, and nothing lists which boxes are behind. **2026-09-25:** Peti's box was RETIRED (operator ruling) — it no longer counts as a box behind. | **WAITING-ON-OPERATOR — rank P2-MEDIUM; owner: operator (signing)** | — | — | operator |
 | **R-604** | Box system & updates | P2 | **[P2-MEDIUM] A per-customer controller floor silently excludes that box from every global floor raise, and NOTHING says so — demo-hp missed four of them.** FOUND 2026-09-21 while raising the global floor to 0.259.0 at the operator's request. The raise logged `Global controller-version floor set to "0.259.0"` and then `managed floor SERVED for demo-felhom` — **and nothing at all for demo-hp**, which went on reporting every few minutes and stayed on 0.258.0. Cause: `customer_configs.min_controller_version` for demo-hp held **`0.243.0`**, a per-customer override that wins over the global. It is a **leftover from the 2026-09-16 drill**, whose golden was 0.243.0; **R-343 measured on 2026-08-18 that all five rows were EMPTY and recorded that as a safety property** — it stopped being true and nothing surfaced the change. demo-hp had therefore silently missed the raises to 0.253.0, 0.254.0, 0.257.0 and 0.259.0. **Why it is invisible rather than merely quiet:** `managed floor SERVED` fires **once per CHANGE** (`h.floorNotes`, `api/handler.go:600`), deliberately, because a box reports every few minutes — so a box whose override never changes is silent for ever, and its silence is indistinguishable from the silence of a box that already had the line. A session that raises the floor reads one SERVED line and reasonably concludes the fleet took it. **CLEARED** for demo-hp the same session (rollback line: POST `/customers/demo-hp/floor` with `min_controller_version=0.243.0`, `min_agent=0.131.0`); it then self-updated 0.258.0 → 0.259.0 in **under four minutes**, healthy, `settle-gate: GO — at/above floor 0.259.0`, and its claim page answers **"Wrong or expired code"** in English — the floor delivered the FIX, not a version string, to a box nobody hand-deployed. All five overrides are now empty. **Fix shape (smallest first):** the floor-raise page shows which customers carry an override and would NOT be moved, before the save; or the raise logs one line per customer naming the ones it skipped and why. A raise that quietly reaches half the fleet is worse than one that refuses. | **READY — rank P2-MEDIUM; owner: CC (hub)** | — | — | CC |
-| **R-812** | Box system & updates | P2 | **[P2] A box never receives operating-system security updates — not the Proxmox host, not the guest's Debian, not its Docker engine.** SEARCHED 2026-10-03 (read-only): `felhom-controller`, `felhom-agent`, `app-catalog-felhom.eu` and `felhom.eu` hold no `apt-get upgrade`, `apt full-upgrade`, `unattended-upgrades`, `pveupgrade` or `needrestart` that runs on a box. The installer aligns the host's Proxmox repositories to no-subscription *"so the box can pull security updates"* and then says plainly *"No upgrades are run"* (`scripts/felhom-host-install.sh:2133-2136`). The guest's Docker engine is installed when the golden is BAKED (`felhom-agent/configs/build-golden.sh:124-125`), so a fresh install gets that week's engine and an installed box keeps it forever. The only `apt full-upgrade` in the project is a by-hand step for the off-site endpoint ep0 (`documentation/runbooks/offsite-endpoint.md:41`), not a box. App images ARE updated (the update arc); the layer under them is not. The intention, with its scope, is **R-808** in `ROADMAP.md`. | **NARROWED 2026-10-04 — the SPIKE is done (`11` §7.1, corrections C1–C12, `audits/os-updates-spike-2026-10-04/`); no product code yet. LEFT: the build steps of `11` §8, each with the operator's go; the §5.3 snapshot question is in STATUS; preconditions R-835, R-836, R-837.** | — | — | CC + operator |
+| **R-812** | Box system & updates | P2 | **[P2] A box never receives operating-system security updates — not the Proxmox host, not the guest's Debian, not its Docker engine.** SEARCHED 2026-10-03 (read-only): `felhom-controller`, `felhom-agent`, `app-catalog-felhom.eu` and `felhom.eu` hold no `apt-get upgrade`, `apt full-upgrade`, `unattended-upgrades`, `pveupgrade` or `needrestart` that runs on a box. The installer aligns the host's Proxmox repositories to no-subscription *"so the box can pull security updates"* and then says plainly *"No upgrades are run"* (`scripts/felhom-host-install.sh:2133-2136`). The guest's Docker engine is installed when the golden is BAKED (`felhom-agent/configs/build-golden.sh:124-125`), so a fresh install gets that week's engine and an installed box keeps it forever. The only `apt full-upgrade` in the project is a by-hand step for the off-site endpoint ep0 (`documentation/runbooks/offsite-endpoint.md:41`), not a box. App images ARE updated (the update arc); the layer under them is not. The intention, with its scope, is **R-808** in `ROADMAP.md`. | **NARROWED 2026-10-04 (evening) — the guest's DOCKER engine slow lane is BUILT and proven live (agent v0.142.0, hub v0.132.0; `11` §5.8): live-restore on everywhere, ring 0 steps under a root-owned mark, ring 1 and undo only by a signed job the wrapper re-verifies; the operator approves each engine set on the System page. LEFT: the kernel lane (R-836); existing boxes (R-840).** Earlier: NARROWED 2026-10-04 (afternoon) — the HOST's Debian fast lane is BUILT and proven live too (agent v0.141.1, hub v0.131.1; `11` §8.2, `audits/os-host-lane-2026-10-04/`): appliances only, never kernel/boot/firmware, after a healthy guest step; fleet view and four alarms (§8.3). LEFT: the Docker and kernel slow lanes (R-836); existing boxes (R-840).** Earlier: NARROWED AGAIN 2026-10-04 (day) — the GUEST's Debian fast lane is BUILT and proven live (agent v0.140.0, hub v0.130.0, installer 1.29.0; `11` §8.1, `audits/os-guest-lane-2026-10-04/`). LEFT: the host, Docker and kernel lanes; the undo (R-842); existing boxes (R-840).** Earlier: NARROWED 2026-10-04 — the SPIKE is done (`11` §7.1, corrections C1–C12, `audits/os-updates-spike-2026-10-04/`); no product code yet. LEFT: the build steps of `11` §8, each with the operator's go; the §5.3 snapshot question is in STATUS; preconditions R-835, R-836, R-837.** | — | — | CC + operator |
 | **R-35** | Box system & updates | P3 | **Config-apply should not end the customer's session.** The offsite config push bumped `config_version` 10→11 at 16:54:58 and the controller self-restarted (container `StartedAt` 16:54:59Z, back up 16:55:02); in-memory sessions died with it and **customer zero was force-logged-out mid-flow**. **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-21, size S, roadmap state `idea`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | Direction: **hot-apply the offbox target** (no restart for a config the running process can adopt), or **persist sessions** across restart. The restart itself is by design — the collateral is not. Evidence `controller-log-full.txt` | CC |
 | **R-50b** | Box system & updates | P3 | **[P2] A root-owned privileged host artifact is delivered unversioned from `main` — "which wrapper is on this host?" is unanswerable.** `configs/felhom-pbs-apply` installs to `/usr/local/sbin/felhom-pbs-apply` (0755 root:root) and is the pinned sudoers vector for `create\ |reconcile\|grant` against `/etc/pve/priv/storage`. It is fetched by `felhom-host-install.sh:1914` via `fetch_raw`, which hits `raw/branch/main/<path>` — **no tag, no pin, no checksum, and no record in the Day-0 artifact manifest**, unlike the agent binary (sha256-vouched) and the golden image. Three consequences: (1) two hosts installed a week apart can carry different privileged wrapper code while both reporting the same agent version; (2) a host hotfixed in place (felhom-pve, 2026-07-18) is indistinguishable from one that fetched the same content — the fleet has no inventory of it; (3) an accidental push to `main` reaches the next install of every host with no review gate between commit and root-owned deployment. | **NARROWED** — **(a) SHIPPED 2026-07-21; (b)/(c) open** — moved from `ROADMAP.md` 2026-10-03: it states a checkable fact about the shipped product, so it is a FINDING (the sorting rule). **Re-ranked 2026-10-03: [P2] → P3 — operator-only; leg (a) shipped.** | — | **Surfaced 2026-07-21 while stopping the R-39 v0.90.1 publish** (`felhom-controller/REPORT.md` §5): the publish was cancelled precisely because the version number would have claimed to carry a fix that in fact rides this unversioned channel. Candidate shapes, in increasing cost: (a) record the wrapper's sha256 in the Day-0 artifact manifest beside the agent binary and have the agent report the installed file's hash, so drift is at least *visible*; (b) `fetch_raw` takes a pinned ref (tag or commit) supplied by the manifest rather than `main`; (c) the wrapper becomes a published generic-registry artifact with the same gate ladder as the agent binary. **(a) is the cheap honest first step and would have caught this class already.** Pairs with R-39 (whose remaining fleet half is specced separately)  **(a) SHIPPED 2026-07-21 — hub v0.68.0 + agent v0.91.2.** `ArtifactManifest.WrapperSHA256` + an operator field; agents report the installed wrapper's sha256 each cycle and the host page surfaces a mismatch. **An unknown on EITHER side reads as quiet, never as drift** — lighting every host amber on rollout day is how a warning becomes background noise. Live confirmation of exactly the problem: felhom-pve's July-18 in-place hotfix hashed `2888f2ea…`, matching **no commit anyone could name**; it now reports `104db0a4…` against a vouchable manifest value. **(b)/(c) REMAIN OPEN:** the wrapper is still fetched unversioned from `raw/branch/main` — this makes drift *visible*, it does not fix the channel. Also recorded: the 0440 sudoers file is not agent-readable, so its drift stays invisible. **Flips (2026-10-03):** `00` §A "The installer is PUBLISHED, not pushed" — the same discipline for the privileged wrappers. **Re-ranked 2026-10-03:** [P2] → P3: operator-only; (a) shipped 2026-07-21 (the report carries the wrapper sha256), (b)/(c) open. **Finding-shaped** — an R-424 instance; check against today's product before building. | CC |
 | **R-78** | Box system & updates | P3 | **`local_api` authority ruling — auto-reconcile vs detect-only** **MIGRATED FROM `ROADMAP.md` 2026-08-22 (R-369) — originally filed 2026-07-25, size M, roadmap state `idea (deferred OUT of R-77 on purpose)`.** Moved verbatim; nothing added or reinterpreted. The roadmap keeps its copy as history, marked moved. **An OWED OPERATOR DECISION, not a defect — moved because an owed ruling hidden among feature ideas is the shape this session exists to remove.** | **OPEN — migrated from ROADMAP 2026-08-22, rank unchanged** | — | R-77 ships detection because the fix is genuinely undecided, and **both directions can lose customer-visible function**. **Direction 1 (today):** `controller.yaml` wins and drift is silent → the 2026-07-25 island migration blinded the whole fleet's control plane for 17.5 h (drive gate, guest-reboot recovery, quiesce/backup all degrade). R-77 makes that loud but does not stop it recurring. **Direction 2 (`bootstrap.json` wins, auto-reconcile on boot):** a guest whose `controller.yaml` is CORRECT and whose `bootstrap.json` is stale — a half-completed re-provision, a hand-repaired guest, a setup-wizard box — gets a **working channel clobbered on the next restart**, fleet-wide and silently, during a routine deploy. That is not obviously better than the bug. Needs a spike: which writer is authoritative per field (endpoint vs fingerprint vs token — `mergeLocalAPI` replaces the whole block, so they cannot be reconciled independently today), whether the agent side should stamp a generation/mtime so 'newer wins' is even expressible, and whether reconcile should require an operator ack. Until then the drift alert plus a manual edit is the supported path. | CC |
@@ -325,12 +323,12 @@ stopping line that lies.
 | **R-184** | Box system & updates | P4 | **Nothing prevents the hub from vouching an agent version that was never released.** The R-115 gate proves every RELEASED version is installable, but it works from git tags — so a hub artifact-manifest entry naming a version with no tag and no package is invisible to it. The installer would then die at step 5 on a virgin machine, as root | **READY (S) — NEW 2026-08-03** | — | **Filed BECAUSE the R-115 gate deliberately does not cover it, rather than leaving the gap unstated.** CI cannot check it: the hub's `/api/v1/artifacts/<customer>` answers **401** without a per-customer retrieval passphrase and Gitea's package **listing** api answers **401** without a token (both measured 2026-08-03, P-C), so a credential-free gate can ask *"is this version installable"* but never *"which version is vouched"*. **Two shapes, and the second is better:** (a) give CI a hub credential — expands what CI can reach, and is the operator's call not a gate author's; (b) **validate at vouch time, in the hub**: the operator UI's Day-0 artifact form refuses a version whose package is not downloadable. (b) fails closed at the moment of the decision, needs no new credential anywhere, and puts the check where the mistake is actually made. **Exposure is low and should be said so:** vouching is a deliberate operator action against a version they have just released, and R-115's release path now makes released-but-unpublished nearly impossible. This is the residue, not the main risk | CC |
 | **R-194** | Box system & updates | P4 | **PVE's permission cache delays every grant-state verdict by an unknown amount, so "the agent can read it" and "the ACL exists" are not the same measurement.** Observed twice while validating R-190's self-repair on demo-felhom 2026-08-04: both ACL rows for `/storage/felhom-backup` were deleted, and `GET /access/permissions` continued to report `Datastore.AllocateSpace` present — for **~40 s** in one run and **~16 minutes** in another. During that window the capability probe reads healthy and the self-repair does not fire | **OPEN** | — | **Why it matters beyond the delay:** it puts a floor under how fast a lost grant can be noticed, it makes any single permission read a lagging indicator, and — the interesting part — **it is a candidate contributor to R-190's own timeline**: a grant removed at an unknown moment could keep working until a cache expiry, which is exactly the shape of *worked at 04:44, refused at 09:24*. That does not explain what removed it, but it may explain when the refusal SURFACED, and the two have been treated as the same instant. **Not a defect in our code** — it is PVE behaviour, and the mitigation already tolerates it (the repair fires on the next probe after the cache clears). **What is worth deciding:** whether the store-grant probe should ALSO consult the storage content listing as a second signal, since that appeared to reflect the loss immediately (`{"data":[]}` while the permission read still said present) — two signals disagreeing is itself information, and today only one of them is read | CC |
 | **R-373** | Box system & updates | P4 | **`SysDataGrowGB` is the intended lever for the system-data volume, it works, and nothing sets it.** Written down 2026-08-02 in `audits/SPIKE-recovery-unit-space-2026-08-02.md:230-232`, under an explicit *"### Not filed"* heading: the 20 G / 50 G mismatch was ruled a tier-sizing decision rather than a defect, *"`SysDataGrowGB` is the intended lever and it works; nothing sets it."* A lever with no caller is the same shape as R-368's comment — a setting that names a behaviour nothing invokes. **Age when filed: 20 days.** | **OPEN — LOW** | R-368 (same shape) | Either wire it to something an operator can reach, or remove it and record the sizing decision where a reader will meet it. | CC |
-| **R-835** | Box system & updates | P3 | **Turning Docker's `live-restore` OFF with a restart stops every running container and starts none.** MEASURED 2026-10-04 on scratch 9202: `live-restore` on (via `systemctl reload docker`, which does enable it) kept all 6 containers running across two engine steps; `systemctl reload` with the baked `daemon.json` did NOT turn it off; a `systemctl restart docker` did — and the new daemon stopped every container (`Exited (0)`, `Removing stale sandbox … isRestore=false`) and restarted none, though all are `unless-stopped`. Nothing brought them back for 3.5 min. A precondition for the Docker slow lane (`11` C5): if `live-restore` ships, turning it off must be a guarded act (stop apps first), never a plain restart. `audits/os-updates-spike-2026-10-04/partG/` | **READY — design input, owner: CC** | — | — | CC |
-| **R-836** | Box system & updates | P3 | **A new host kernel that hangs before userspace stays the GRUB default: `--next-boot` is not a one-shot on these hosts.** MEASURED 2026-10-04 on demo-hp (operator's word, 2 reboots): both demo hosts boot UEFI + GRUB without proxmox-boot-tool ESPs; installing a kernel makes it the default at once; `kernel pin <new> --next-boot` writes an ordinary `GRUB_DEFAULT`, and `proxmox-boot-cleanup.service` clears it only after a boot reaches userspace. With the old kernel pinned FIRST, the fallback after a good boot worked (new 60 s, old 76 s). READ FROM THE CODE, not measured: a hang leaves the new kernel default on every power cycle. Only `softdog` runs (useless before userspace); demo-hp's `sp5100_tco` ships unloaded, untested. Fix direction for the slow lane: GRUB's own one-shot (`GRUB_DEFAULT=saved` + `grub-reboot`) with the old kernel saved — to be measured, including Secure Boot (ON on demo-hp). `audits/os-updates-spike-2026-10-04/partH/` | **READY — measure before the kernel slow lane; owner: CC + operator (reboots)** | — | — | CC |
-| **R-837** | Box system & updates | P3 | **The guest snapshot undo (`11` §5.6) is unmeasured.** 2026-10-04: customer guests sit on LVM-thin and can snapshot, but scratch 9202 sits on `dir` storage (`snapshot feature is not available`), so the spike measured a backup-and-restore undo instead (73 s down, all apps back, libc back). The snapshot + rollback time on LVM-thin, and what it does to the thin pool (R-672's shape), were not measured — no fenced LVM-thin scratch guest this session. `audits/os-updates-spike-2026-10-04/partG/SUMMARY.md` | **READY — before the guest fast lane is built; owner: CC** | — | needs an LVM-thin scratch guest on a demo host | CC |
+| **R-836** | Box system & updates | P3 | **A new host kernel that hangs before userspace stays the GRUB default: `--next-boot` is not a one-shot on these hosts.** MEASURED 2026-10-04 on demo-hp (operator's word, 2 reboots): both demo hosts boot UEFI + GRUB without proxmox-boot-tool ESPs; installing a kernel makes it the default at once; `kernel pin <new> --next-boot` writes an ordinary `GRUB_DEFAULT`, and `proxmox-boot-cleanup.service` clears it only after a boot reaches userspace. With the old kernel pinned FIRST, the fallback after a good boot worked (new 60 s, old 76 s). READ FROM THE CODE, not measured: a hang leaves the new kernel default on every power cycle. Only `softdog` runs (useless before userspace); demo-hp's `sp5100_tco` ships unloaded, untested. Fix direction for the slow lane: GRUB's own one-shot (`GRUB_DEFAULT=saved` + `grub-reboot`) with the old kernel saved — to be measured, including Secure Boot (ON on demo-hp). `audits/os-updates-spike-2026-10-04/partH/` | **NARROWED 2026-10-04 (os-host-lane Part E, operator's word before each of 2 reboots) — GRUB's own one-shot is NOT a one-shot here either.** `GRUB_DEFAULT=saved` (old kernel saved) + `grub-reboot <new>`: boot 1 → new kernel, **Secure Boot ON and fine**; but GRUB could not clear `next_entry` (`grub-reboot` itself warns: *environment block on lvm device … will remain the default until manually cleared*; `/boot` is ext4 on LVM `pve-root`), so boot 2 (no command) → **the new kernel again**. `kernel.panic = 0`: a panic leaves the host stopped (R-851). `sp5100_tco` LOADS and answers (`SP5100 TCO timer`, 60 s, inactive, nowayout 0; read from sysfs, never opened, unloaded) — a hardware watchdog exists on demo-hp, but nothing arms it before userspace. demo-hp left on 7.0.14-20 with that as the saved default. **LEFT (fix direction):** a GRUB env block GRUB can write (on the ESP, vfat) or a userspace "boot good" step that rewrites the default, plus arming `sp5100_tco`; to be measured before the kernel slow lane. `audits/os-host-lane-2026-10-04/partE/` **READY — owner: CC + operator (reboots).** | — | — | CC |
+| **R-853** | Box system & updates | P3 | **After a boot the box's versions and crash facts reach the hub up to ~15 minutes late.** MEASURED 2026-10-04 on demo-hp (crash-guard test): the agent's first report after a boot has no `system.facts` — the facts read needs a RUNNING customer guest (`firstGuest`), the guest starts ~1–2 min after the agent, and the failed read is cached for 10 minutes; so the HOST half (the crash guard, the kernel) is lost too. The crash events arrived 15 min after the boot (17:17 → 17:32 CEST); nothing was lost (the guard keeps 7 days). Fix direction: the facts mode reads the host without a guest (guest fields `unknown`), and a failed read is not cached. `audits/os-docker-crash-2026-10-04/partC/c4-hub-events.txt` | **READY — owner: CC** | — | — | CC |
 | **R-839** | Box system & updates | P3 | **After a Docker restart that stopped every container, the boot sweep HELD an app whose `app.yaml` `HDD_PATH` names its user folder instead of the drive.** MEASURED 2026-10-04 on scratch 9202 (paperless-ngx): `bootrecon` logged `drive /mnt/felhom-drives/scratch_hdd/userdata/paperless-ngx is not a live mountpoint — NOT starting it`, although the drive `/mnt/felhom-drives/scratch_hdd` IS a mountpoint; the app stayed down until started by hand. The customer boxes' 9201s showed no hold. **Not diagnosed:** which writer put a per-app path in `HDD_PATH` on 9202, and whether a household box can get it. `audits/os-updates-spike-2026-10-04/partG/SUMMARY.md` | **READY — diagnose; owner: CC** | — | — | CC |
+| **R-840** | Box system & updates | P2 | **A new root wrapper or a new sudoers line cannot reach a box that is already installed — there is no product route.** FOUND 2026-10-04 (guest OS lane, Part B 1): the agent's signed `agent_update` replaces ONLY the binary (`configs/felhom-selfupdate-guarded` swaps `/usr/local/bin/felhom-agent`); wrappers (`/usr/local/sbin/felhom-*`) and `/etc/sudoers.d/felhom-agent` are written only by `felhom-host-install.sh` step 5 on a new box. So agent v0.140.0's OS leg is DEAD on every existing box (the capability probe will show `osapply-run` missing) until someone installs `felhom-os-apply` + the FELHOM_OSAPPLY line by hand — which is what the demo boxes got this session. The `wireguard-tools` line (S3) reached the fleet the same way: by reinstall or by hand. **Proposal:** a signed `agent_config_update` op (operator-signed like `agent_update`) whose params pin the agent TAG and the sha256 of a config bundle (sudoers + wrappers); the self-update wrapper installs it as root after `visudo -cf` and a syntax check, keeps the previous copies, and the capability probe confirms. `audits/os-guest-lane-2026-10-04/README.md` | **READY — design + operator go; owner: CC.** 2026-10-04 (evening): the demo boxes got the v0.142.0 wrapper, the crash guard and the two root-owned trust files BY HAND again (`audits/os-docker-crash-2026-10-04/partD/`). **RULED 2026-10-04 ~12:20 (`09` §3 decision 82): not built now — "There are no older boxes"; no installed box exists outside the two demo boxes, which take wrapper changes by hand. Kept open, not re-ranked. Reviewer's note (as written): every box installed from now on becomes an "older box" the first time the wrapper or the sudoers line changes again (the 2026-10-04 host-lane brief changes the wrapper); R-840 must be solved before the first box that cannot be reached by hand.** | — | — | CC |
 
-## Monitoring & notifications — 24 rows (P2 2, P3 16, P4 6)
+## Monitoring & notifications — 25 rows (P2 2, P3 16, P4 7)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -358,8 +356,9 @@ stopping line that lies.
 | **R-337** | Monitoring & notifications | P4 | **`/backup/status` lagged a completed backup by minutes on one box and not the other — and it RESOLVED ITSELF, which is why this is WATCHING and not a defect.** During the R-336 recovery on 2026-08-18, `demo-hp`'s snapshot landed on ep0 at **03:58:43Z** (complete manifest; the host's own task index says `OK`) — yet `GET /backup/status` was **still serving the superseded 03:27:00Z failure at ~04:03Z**, four-plus minutes later. `demo-felhom` showed its new result within ~40 s of completion. **The lag cleared on its own:** demo-hp's 04:07:35Z host report carries `felhom-pbs success=true, 4.29 GB`, and the hub is green for both boxes. **The first draft of this row claimed the success was "still reported as failed" — that was written before the next report arrived and it was wrong; the corrected claim is a several-minute skew between the two boxes, not a stuck value.** It is recorded because a status field that can trail its own artifact by minutes will, during an incident, be read as a second failure — this session nearly did — and because the asymmetry between the two boxes is unexplained | **WATCHING — NEW 2026-08-18** | another observation, ideally during an incident rather than constructed | **Do not open a fix on this as written.** First establish the intended refresh path for `/backup/status` after an out-of-schedule run; only if the skew is not simply collection cadence is there anything to pin. If it is cadence, close this row and say so | CC |
 | **R-348** | Monitoring & notifications | P4 | **Every agent restart blanks the reported backup list for up to ~18 hours, and the comment that covers it says "unaffected".** Observed 2026-08-20 while deploying R-344: the first host reports after `demo-hp`'s agent restart carry **`0 backups`** (11:15:50 and 11:30:52 CEST, two consecutive), while the box's own `pvesm list` shows archives present on **both** tiers. `internal/backup/store.go`'s `Store` is in-memory and `byTarget` is repopulated only when a backup **runs** — daily for the local tier, weekly for offsite — so the field reads 0 until the next run. `restore_tests` did **not** blank, because that half has a durable on-disk companion (`RestoreTestState`, R-189). **It blinds no alarm, and that was CHECKED rather than assumed.** `hub/internal/monitor/deadline.go` scans back over stored reports with a 7-day `backupEvidenceLookback` whose own comment names this exact case — *"when the LATEST report carries none... and against an agent that stayed restarted for days"* — and `pbs_snapshots` stayed populated at 2 regardless. So this is an observability wart, **not** a safety hole, and it is filed at that severity deliberately. **What is actually wrong is the comment.** The `Store` doc says *"Backups are unaffected — their freshness has a ground truth on the storage (R-84)"*. That is true of the **consequence** and false of the **field**, and it sits three lines below a paragraph explaining that the very same sentence about restore-tests *"used to be here and it is now FALSE"* — so the file already carries one correction of this shape and invites the next reader to trust the surviving half. | **READY (XS) — NEW 2026-08-20** | — | Say what is measured: the field IS lost on restart and repopulates only when a backup runs; the freshness VERDICT is unaffected because the hub looks back 7 days. **Name `backupEvidenceLookback` in the comment** so the cross-repo dependency is visible from the agent side — today the agent's claim of safety rests on a hub constant it does not mention. Per the workspace rule, a comment asserting an invariant needs a test pinning it: the pin belongs on the HUB side, asserting the verdict survives a report carrying `backups: []`. | CC |
 | **R-371** | Monitoring & notifications | P4 | **The off-site tier is the only backup tier that announces nothing on success.** Written down 2026-08-05 in `audits/CAMPAIGN-11-recovery-journey-2026-08-05.md:508-513` and explicitly *"recorded, not filed"*: the off-site run emits **no hub event at all**, while both lesser tiers do (`db_dump_completed`, `crossdrive_completed`). Failures are covered by `backup_run_failures` and staleness by the hub's 8-day tier deadline, which is why it was judged a wrinkle. **Still true 2026-08-22** — the 2026-08-21 drill's own event dump shows `db_dump_completed` and six `crossdrive_completed` rows and no off-site success event. **Age when filed: 17 days.** | **OPEN — LOW** | — | Either emit one, or record deliberately that the highest-value tier is silent on success and say why. | CC |
+| **R-856** | Monitoring & notifications | P4 | **A crash restart reaches the household twice: the hub's "restarted after an unexpected stop" line AND the controller's app mails.** 2026-10-04 crash-guard test on demo-hp: after the third crash and the power-on, the controller sent `app_start_failed` (operator) and `app_stopped_unhealthy` (operator AND the household's address) for apps that were still coming up. Each is true on its own; the app ladder has no "the host just crashed" suppression like its boot grace for an ordinary restart (`08` §5). A design question for the operator, not a defect yet. `audits/os-docker-crash-2026-10-04/partC/c4-hub-events.txt` | **READY — operator decision** | — | — | operator |
 
-## Hub & operator — 21 rows (P2 1, P3 7, P4 13)
+## Hub & operator — 23 rows (P2 1, P3 7, P4 15)
 
 | ID | Category | Sev | What | State | Blocked on | Next action | Owner |
 |---|---|---|---|---|---|---|---|
@@ -384,6 +383,8 @@ stopping line that lies.
 | **R-688** | Hub & operator | P4 | **[P3-LOW] The customer delete says it removes the tunnel and zone, but no leg of it calls Cloudflare.** The dialog's acknowledgement reads "the customer will be RESET — offsite repo DESTROYED, PBS revoked, tunnel/zone removed" (`hub/internal/web/customer_delete.go` `deleteCascadeAcks`), while `commitCustomerReset` has legs for Hetzner, PBS, claim, descriptor and DB only. Seen 2026-09-25 retiring `peti-felhom`, whose config carried a Cloudflare tunnel token and API token (`sajatfelhom.hu`): the tokens went with the record; any tunnel or DNS record on Cloudflare's side was neither listed nor removed. **Fix direction:** either a Cloudflare leg (tunnel + DNS by the customer's ids), or the dialog stops promising it and lists what to remove by hand. `audits/RETIRE-peti-2026-09-25.md` **-- HALF DONE 2026-09-25 (hub v0.125.0):** the dialog no longer promises a Cloudflare removal; the preview lists what the operator removes by hand, by domain (the tunnel, the DNS records), never the token — proven live on the hub (`audits/night-2026-09-26/F/`). The Cloudflare leg itself is NOT built. | **NARROWED — the Cloudflare leg only; owner: operator (decide if it is wanted) / CC (build)** **Re-ranked 2026-10-03: P3→P4: the dialog no longer promises it; what is left is operator comfort.** | — | — | CC + operator |
 | **R-719** | Hub & operator | P4 | **[P2-MEDIUM] A customer who already exists never gets a fresh self-bind link when their new box registers: the last link expires in 7 days and nothing re-sends it.** MEASURED 2026-09-29 (new-household drill, `tester-1`): the previous link went out 2026-09-17 07:25 UTC at a host delete and expired 2026-09-24; the box registered at 19:11:30 UTC and its console told the volunteer to open the link from their e-mail — there was none that worked. Hub source: the link is sent at customer creation, RESET, e-mail set on a box-less customer and host delete (`selfbind_mint.go` callers `hosts.go:908`, `configs.go:850`, `customer_reset.go:162`) — never on appliance registration. The volunteer guide says the operator needs to press nothing. The operator pressed „Send self-bind link" (the mail arrived in 1 s) — an operator step the volunteer depends on, recorded, not an intervention. **Fix direction:** send the link when an unclaimed appliance registers while a box-less customer waits with no live link (R-509's first fix shape), or the guide's operator part says: press it the day the volunteer installs. Evidence: `audits/evidence-drill-new-household-2026-09-30/` `phase0/operator-steps.txt`. **CHANGED AND BUILT 2026-09-30 (hub v0.126.0) — the brief's shape was not buildable:** a box registers UNCLAIMED (uuid, MACs, host keys, hardware — nothing of a customer), so "send the link when their box registers" would mail every waiting customer. Built instead: the expired AND used link pages offer „Új linket kérek" → a fresh link to the address registered for that link's customer, only when it has no box, ≤1/h per customer, identical answer for any token (no oracle). Live: the button on the real hub, the same page for a made-up token, no mail; the mint+send path unit-proven (RP42). Evidence: `audits/evidence-fixes-first-tester-2026-09-30/``partD/`. | **WAITING-ON-OPERATOR** (2026-10-03 triage: the row's verdict was finished, but it names open work no other row carries — the operator has not reviewed the changed page shape ("operator may prefer another"), and mint+send is unit-proven only) — **CLOSED 2026-09-30 — hub v0.126.0 (changed shape; operator may prefer another)** | — | — | operator |
 | **R-814** | Hub & operator | P4 | `PBS-storage-1` (u629193, box 611421) still `status=active`, 19.9 MB | **VERIFY** (2026-10-03 triage: a July watch row with no id; given R-814. WAITING-ON-OPERATOR — no record found that the box was deleted.) — WAITING-ON-OPERATOR | operator console | Delete the box | operator |
+| **R-844** | Hub & operator | P4 | **The household's OS-update line exists only on the hub's customer timeline.** 2026-10-04: the box itself has no event surface for agent results (the controller UI shows no timeline), so `os_update_applied` is a hub customer event (info: recorded, never mailed). Its stored text is the hub's English sentence; the hu/en bundle text (`mail.event.os_update_applied`) is used only if it is ever mailed. Fix direction: a controller-side line (the controller already polls the agent's local API) when the box gets a household timeline. `audits/os-guest-lane-2026-10-04/partG/hub-customer-timeline-demo-hp.txt` | **READY — owner: CC** | — | — | CC |
+| **R-855** | Hub & operator | P4 | **The hub's start log prints "after -1 healthy ring-0 night(s)" for the TEST override `OS_DOCKER_APPROVE_NIGHTS=0`** (the internal "none" value; the WARN line before it is right). Cosmetic, TEST configuration only. Fix: print 0. `audits/os-docker-crash-2026-10-04/partB/` (hub log 16:27:42) | **READY — owner: CC** | — | — | CC |
 
 ## Business & legal — 7 rows (P2 4, P4 3)