os-apply: the host restart scan no longer hides lxc-start (skip ':/lxc/' not 'lxc'), and the host scans on every pass so a reboot clears 'reboot needed' (reboot_scanned reaches the hub) — both found live on demo-felhom

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

.gitignore

@@ -9,3 +9,6 @@
 
 # go
 /vendor/
+
+# Python bytecode written by configs/test_felhom_os_apply.py
+configs/__pycache__/

CHANGELOG.md

@@ -1,3 +1,85 @@
+## v0.141.0 — OS updates: the host fast lane (`11` §8 step 3); the tunnel status is true (R-841); the leg is fast (R-845)
+
+> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.141.0` (`cfba0d0`), sha256
+> `6eaad9809613c4fca64aefc30cc16415528499bf48efe3a1daaa08af8a611aff`, verified by download. Not vouched at release time.
+
+**MinAgent impact:** none required by any controller. **Needs hub v0.131.0** (`host_release`, the tunnel's three
+states, layer-tagged OS reports). With an older hub the host step finds no host release (ring 1 → nothing) and the
+tunnel's `detail` is ignored. Reads the cloudflared health check controller v0.292.0 adds; with an older controller
+the tunnel is judged on the container state alone and `detail` says so.
+
+- **R-841 — the tunnel.** The agent used to run `systemctl is-active cloudflared` on the HOST — a unit that does not
+  exist (cloudflared is a container in the customer guest), so every box reported `inactive`. `GuestTunnelProber`
+  now reads the guest's `cloudflared` container through the EXISTING sudoers line (`pct exec N -- docker inspect -f
+  *`): state, exit code, and the Docker health status. Three states: `running` (healthy), `not_running` (stopped,
+  absent, or running but NOT connected), `unknown` (could not ask, or the check is still starting). `detail` says why.
+- **The host step** (`11` §8 step 3). After a healthy guest step, under the same heavy-op gate, the leg runs the same
+  wrapper with `layer: host`. Debian origin only; never kernel, boot or firmware packages (new refusal **R14**);
+  **only on an appliance** (**R12** lifted for the host fast lane: proof is the ROOT-owned install record
+  `/var/lib/felhom-install/state.json` `mode: appliance` — the agent-writable `agent.json` is not trusted for this).
+  A failed or unhealthy guest step skips it. **Host health rule:** the agent, pveproxy, pvedaemon, pvestatd and
+  pve-cluster are active; the customer guest runs; the guest health rule passes; the tunnel is `running` (an
+  `unknown` tunnel does not fail the rule; `not_running` does). Never reboots: "reboot needed" is reported when PID 1
+  or `lxc-start` runs a replaced library. No automatic undo — the by-hand runbook is `os-updates-host-undo.md`.
+- **R-845 — the leg is fast.** The wrapper asked about each package in its own `pct exec` (about 0.9 s each). It now
+  makes one call per layer for the version checks (`apt-cache madison` for all names at once, `dpkg --compare-versions`
+  on the host), scans for restart-needed only after an install, repairs only when `dpkg --audit` reports something,
+  and reports its own `pass_seconds`.
+- **Wrapper tests:** `configs/test_felhom_os_apply.py` 46 tests (host layer, R12, R14, lxc-start, the speed rules).
+  Red-proofs: `felhom.eu/documentation/audits/os-host-lane-2026-10-04/partA/`, `partB/`, `partC/agent-golden-redproof.txt`.
+- **Contract:** the desired-state golden gains `host_release` (byte-identical with the hub's); `TestOSUpdateGolden_Decodes`
+  checks it.
+
+## v0.140.0 — OS updates, guest fast lane (`11-os-updates.md` §8 step 2; `09` §3 decisions 76, 79, 80)
+
+> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.140.0` (`9cac346`), sha256
+> `ae2d60b794869c51d6b063c8e31e2da98ecdbd36c6b75febd64e2fb4266c1250`, verified by download. Not vouched at release time.
+
+**MinAgent impact:** none required by any controller. **Needs hub v0.130.0** (`os-report`, the `os_update` block); an
+older hub serves no block and the leg then reports and installs nothing (ring 1, no release).
+
+- **`configs/felhom-os-apply`** — the root wrapper (Python 3, stdlib). One sudoers entry, `FELHOM_OSAPPLY`:
+  `felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json`. Modes `inventory` / `apply` / `health`. Refuses (exit
+  2, nothing changed) on R1–R13: plan path/owner/JSON, a non-Debian origin, the slow lane, any removal, any downgrade,
+  a new or unlisted package, a version not downloadable even from the snapshot, low space, a lock (apt or a guest lock
+  such as a backup), a vmid that is not the box's own customer guest (it must bind `/mnt/felhom-drives`), malformed
+  names/versions, the host layer, dpkg still broken after the repair. Repairs first (`dpkg --configure -a`,
+  `apt-get -f install`). A version Debian already replaced comes from `snapshot.debian.org` at the approval time
+  (decision 79). Reports the full installed set with origins, pending, restart-needed (outside containers), health.
+  `configs/test_felhom_os_apply.py`: 35 tests; every refusal red-proved.
+- **`internal/osupdate`** — the leg: after a SUCCESSFUL primary whole-guest backup, still holding the heavy-op gate
+  (never beside another backup or a restore-test), once per night, 90 s after the backup. Ring 0 installs every pending
+  Debian / Debian-Security fix; ring 1 exactly the hub's newest approved release; switched OFF → reports only. The
+  health rule: docker answers, the network resolves, the controller is healthy, every container running at the START
+  of the leg runs (and is healthy if it was) — a 5-minute wait. **No automatic undo:** a customer guest cannot be
+  snapshotted (R-837). A failure is `health_failed` → the hub mails the operator.
+- **`--selftest=os-update -vmid N`** — the debug action (trigger `debug`, never throttled, not a night run).
+- **The `--selftest` flag also accepts `wgtunnel`** — it was dispatched but refused since S3 (found by the new
+  `TestSelftestFlag_AcceptsEveryDispatchedMode`, which also caught `os-update` live).
+- Proven live 2026-10-04 on both demo boxes (ring 0: 53 packages each; ring 1: exactly 3 approved versions; a failed
+  health check → `health_failed`, operator mailed): `felhom.eu/documentation/audits/os-guest-lane-2026-10-04/`.
+
+## v0.139.0 — a DR restore never lands beside a live original (2026-10-04, R-834)
+
+> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.139.0` (`475bdce`), sha256
+> `8534a9be368a6d24d8065db77436e86900443c5c6554c71f6fe91c2bdb9d0b9c`, verified by download. **Not vouched.**
+
+**MinAgent impact:** none required by any controller.
+
+- The DR bring-up (`--selftest=bring-up -mode dr`) keeps the archive's `onboot: 1`, binds the host's REAL drives
+  (`mp8 /mnt/felhom-drives`) and STARTS the guest — right on a replaced host, wrong beside a live original (a second
+  controller for the same household on the same drives). It now REFUSES, before any restore, when the archive's
+  source guest still exists on the host, when any guest binds the drives parent, or when a guest's config cannot be
+  read (fail closed). On a replaced host it proceeds and keeps its binds, unchanged.
+- The restore-test was MEASURED safe live on demo-hp (onboot 0 and throwaway stand-ins for mp8/mp9 from the first
+  config read to teardown); a test now pins its "no host path" half beside the existing onboot test.
+- No sudoers change: the restore-test sets onboot 0 through the API create call, and DR refuses rather than degrade,
+  so no `-onboot 0` line is needed.
+- Tests: `TestRunBringUp_DRRefusesBesideALiveOriginal` (source guest present / drives bind on another guest / an
+  unreadable config refuse; a replaced host proceeds and keeps the drives bind), `TestRunBringUp_ProvisionNotBlockedByADrivesBind`,
+  `TestArchiveSourceVMID`, `TestRestoreTest_NoHostPathBindBesideTheOriginal`. Red-proofs: the DR check returning ""
+  → three refusal cases restore and START; the restore-test's mp8 override set to the host path → fails.
+
 ## v0.138.0 — the restore test takes only THIS box's archives (2026-09-30, R-727, `09` §3 decision 51)
 
 > **RELEASED 2026-09-30** by `scripts/release-agent.sh` — tag `v0.138.0` (`e1b8269`), sha256

REPORT.md

@@ -1,12 +1,9 @@
-# REPORT — 2026-09-30: v0.138.0 (R-727)
+# REPORT — 2026-10-04: v0.141.0, the host fast lane, the true tunnel status, the fast leg
 
-Full session report: `felhom.eu/REPORT-fixes-first-tester-2026-09-30.md`.
+Full session report: `felhom.eu/REPORT-os-host-lane-2026-10-04.md`.
 
-- **Measured:** a PBS archive carries its key FINGERPRINT (PVE content `encrypted`), not a host id; the storage
+- Tunnel (R-841): the agent reads the guest's cloudflared container and its health check; three states.
-  carries its own (`GET /storage` → `encryption-key`). The restore test now skips an archive written with another
+- Host fast lane: the wrapper gains the host layer (R12 appliance proof from the root-owned install record, R14 no
-  key and logs it by name; an unencrypted storage is not filtered; a failed storage read is UNKNOWN.
+  kernel/boot/firmware); the leg runs the host step after a healthy guest step; host health rule.
-- Tests `TestR727_*`; red-proof RP39 (the skip removed → the 2026-09-16 archive of an earlier box is picked).
+- Speed (R-845): one call per layer instead of one per package; measured before/after in the session report.
-- Released by `release-agent.sh` (tag `v0.138.0`, sha256 `55916026…8195`, verified by download); **not vouched**.
+- Tests green; red-proofs in the audit folder.
-  Delivered by signed `agent_update` jobs to `demo-hp-bb76ea` and `demo-felhom-8363b5` (both committed within 340 s);
-  `-selftest=restore-test-due` on both reads each tier normally on 0.138.0.
-- ep0 (decision 51): the three drill archives in `tester-1`'s namespace removed; other namespaces byte-identical.

cmd/felhom-agent/main.go

@@ -48,6 +48,7 @@ import (
 	"gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr"
 	"gitea.dooplex.hu/admin/felhom-agent/internal/poke"
 	"gitea.dooplex.hu/admin/felhom-agent/internal/provision"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/osupdate"
 	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
 	"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
 	"gitea.dooplex.hu/admin/felhom-agent/internal/restorespace"
@@ -242,6 +243,8 @@ func main() {
 		os.Exit(runSelftestRestoreTest(context.Background(), cfg, logger, archive))
 	case "restore-test-due":
 		os.Exit(runSelftestRestoreTestDue(context.Background(), cfg, logger))
+	case "os-update":
+		os.Exit(runSelftestOSUpdate(context.Background(), cfg, logger, vmid))
 	case "pbs-verify":
 		os.Exit(runSelftestPBSVerify(context.Background(), cfg, logger))
 	case "lanresolver":
@@ -782,7 +785,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
 	pbsStore := pbs.NewSnapshotStore()
 	pbsTargets := pbsTargetsFromPVE(cfg, px, logger)
 	pbsReporter := pbs.NewLiveSnapshotReporter(pbsTargets, pbsStore, pbs.DefaultLiveSnapshotTimeout, logger)
-	collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, backupStore, backupStore, pbsReporter, cfg.Hub.HostID, version, logger)
+	collector := hub.NewCollector(px, newTunnelProber(cfg, px), observer, backupStore, backupStore, pbsReporter, cfg.Hub.HostID, version, logger)
 	collector.SetBackupTargetResolver(primaryBackupTargetOf(cfg)) // R-109: the recipe names the live target
 	// Privileged-capability self-check (v0.44.0): probe the sudoers grants the non-root agent
 	// depends on. The probe runs `sudo -n -l` LITERALLY (a policy LIST, never executing the
@@ -839,6 +842,10 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
 	// The "Down" channel sync hook: on each heartbeat, fetch desired-state when the generation
 	// advances. The loop calls it via the EnvelopeObserver seam (hub does not import desired).
 	desiredSyncer := desired.NewSyncer(client, desiredProvider, logger)
+	// OS updates, guest fast lane (agent v0.140.0, `11-os-updates.md` §8 step 2): the leg consumes the hub's
+	// os_update block and runs after each successful primary whole-guest backup (wired on the local API below).
+	osLeg := newOSLeg(cfg, client, px, logger)
+	desiredSyncer.AddConsumer(osLeg)
 	// S5: consume a host_loss restore_directive into an inspectable restore PLAN (derive + surface,
 	// execute nothing). The recipe is fetched on-demand (rare directive) via a fresh Collect.
 	desiredSyncer.AddConsumer(dr.NewConsumer(func(ctx context.Context) *hub.DRRecipeHostHalf {
@@ -1112,6 +1119,17 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
 		},
 	}
 	localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
+	if localSrv != nil {
+		localSrv.SetAfterPrimaryBackup(func(ctx context.Context, vmid int) {
+			// Let the controller finish bringing its apps back after the backup, then run (still under the gate).
+			select {
+			case <-ctx.Done():
+				return
+			case <-time.After(90 * time.Second):
+			}
+			_, _ = osLeg.Run(ctx, vmid, "night")
+		})
+	}
 	if localTokens != nil {
 		defer localTokens.Close()
 	}
@@ -2008,7 +2026,7 @@ func runSelftestHub(ctx context.Context, cfg config.Config, logger *slog.Logger)
 	// pbs coord. The live reporter lists snapshots directly (fresh store, last-known-good fallback) so
 	// the selftest reflects exactly what a freshly-restarted daemon's first collect emits.
 	pbsReporter := pbs.NewLiveSnapshotReporter(pbsTargetsFromPVE(cfg, px, logger), pbs.NewSnapshotStore(), pbs.DefaultLiveSnapshotTimeout, logger)
-	collector := hub.NewCollector(px, hub.SystemctlProber{}, observer, nil, nil, pbsReporter, cfg.Hub.HostID, version, logger)
+	collector := hub.NewCollector(px, newTunnelProber(cfg, px), observer, nil, nil, pbsReporter, cfg.Hub.HostID, version, logger)
 	// R-109: wire the backup-target resolver here TOO. Without it selftest=hub would print a recipe whose
 	// backup_target reads unknown/agent_backup_config_unavailable while the daemon's is resolved — and
 	// this one-shot exists precisely so "the report it would send" can be trusted to match.
@@ -3478,8 +3496,126 @@ func (f *selftestFlag) Set(v string) error {
 		f.mode = "identity-consume"
 	case "controller-swap":
 		f.mode = "controller-swap"
+	case "os-update":
+		f.mode = "os-update"
+	case "wgtunnel": // dispatched since S3 but refused here until 2026-10-04 (TestSelftestFlag_AcceptsEveryDispatchedMode)
+		f.mode = "wgtunnel"
 	default:
-		return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap)", v)
+		return fmt.Errorf("invalid --selftest value %q (want read|task|hub|storage|backup|restore-test|restore-test-due|pbs-verify|lanresolver|wgtunnel|bring-up|provision|escrow-create|escrow-consume|identity-consume|controller-swap|os-update)", v)
 	}
 	return nil
 }
+
+// newOSLeg builds the OS-update leg (agent v0.140.0). The wrapper runs through sudo (FELHOM_OSAPPLY); the plan and
+// the once-per-night marker live in the agent's own os/ dir.
+func newOSLeg(cfg config.Config, client *hub.Client, px *proxmox.Client, logger *slog.Logger) *osupdate.Leg {
+	mode := proxmox.RunnerMode(cfg.Privileged.Mode)
+	if mode == "" {
+		mode = proxmox.RunnerSudo
+	}
+	l := &osupdate.Leg{
+		Runner:    &proxmox.ExecRunner{Mode: mode, SudoPath: cfg.Privileged.SudoPath},
+		Logger:    logger,
+		PlanDir:   osupdate.DefaultPlanDir,
+		StatePath: filepath.Join(osupdate.DefaultPlanDir, "last-night-run"),
+		// The host step (agent v0.141.0) runs only on an appliance install; the wrapper re-checks the ROOT-owned record.
+		Appliance: cfg.IsAppliance(),
+		Tunnel:    newTunnelProber(cfg, px),
+	}
+	if client != nil {
+		l.Hub = client
+	}
+	return l
+}
+
+// runSelftestOSUpdate is the OS leg's DEBUG ACTION (agent v0.140.0): one pass for -vmid, now, exactly as the night
+// runs it after a backup — the hub's os_update block (fetched fresh), the wrapper via sudo, the health wait, the
+// report to the hub — with trigger "debug" (never throttled, and it does NOT count as a night run for approval).
+// Run it as the agent user:  sudo -u felhom-agent felhom-agent --config … --selftest=os-update -vmid 9201
+func runSelftestOSUpdate(ctx context.Context, cfg config.Config, logger *slog.Logger, vmid int) int {
+	if vmid <= 0 {
+		fmt.Fprintln(os.Stderr, "selftest=os-update: -vmid is required")
+		return 2
+	}
+	client, err := hub.NewClient(cfg.Hub, logger)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "selftest=os-update: hub client:", err)
+		return 1
+	}
+	px, perr := newProxmoxClient(cfg)
+	if perr != nil {
+		fmt.Fprintln(os.Stderr, "selftest=os-update: proxmox client:", perr)
+		return 1
+	}
+	leg := newOSLeg(cfg, client, px, logger)
+	resp, err := client.FetchDesiredState(ctx)
+	if err != nil {
+		fmt.Fprintln(os.Stderr, "selftest=os-update: desired state:", err)
+		return 1
+	}
+	leg.SetBlock(resp.DesiredState.OSUpdate)
+	b := leg.Block()
+	fmt.Printf("=== felhom-agent %s selftest=os-update vmid=%d ring=%d enabled=%v guest-release=%v host-release=%v appliance=%v ===\n",
+		version, vmid, b.Ring, b.Enabled, b.Release != nil, b.HostRelease != nil, leg.Appliance)
+	start := time.Now()
+	g, h := leg.Run(ctx, vmid, "debug")
+	for _, rep := range []osupdate.Report{g, h} {
+		if rep.Layer == "" {
+			fmt.Println("  host step: skipped (see the log line above)")
+			continue
+		}
+		printJSON("os-update report ("+rep.Layer+")", map[string]any{"run_id": rep.RunID, "ring": rep.Ring, "release_id": rep.ReleaseID,
+			"mode": rep.Mode, "outcome": rep.Outcome, "healthy": rep.Healthy, "health_reason": rep.HealthReason,
+			"upgraded": rep.Upgraded, "pending": len(rep.Pending), "not_covered": rep.NotCovered,
+			"restart_needed": rep.RestartNeeded, "reboot_needed": rep.RebootNeeded, "wrapper_seconds": rep.PassSeconds, "refused": rep.Refused})
+	}
+	fmt.Printf("  pass took %s\n", time.Since(start).Round(100*time.Millisecond))
+	rep := g
+	if h.Layer != "" && !(h.Outcome == "applied" || h.Outcome == "nothing" || h.Outcome == "inventory") {
+		rep = h
+	}
+	switch rep.Outcome {
+	case "applied", "nothing", "inventory", "skipped":
+		return 0
+	}
+	return 1
+}
+
+// newTunnelProber reads the box's REAL tunnel (R-841, agent v0.141.0): the cloudflared container in each running
+// customer guest — a guest that binds /mnt/felhom-drives, the same rule the OS wrapper's R10 uses — through the
+// existing `pct exec [0-9]* -- docker inspect -f *` sudoers line.
+func newTunnelProber(cfg config.Config, px *proxmox.Client) hub.CloudflaredProber {
+	mode := proxmox.RunnerMode(cfg.Privileged.Mode)
+	if mode == "" {
+		mode = proxmox.RunnerSudo
+	}
+	return hub.GuestTunnelProber{
+		Runner: &proxmox.ExecRunner{Mode: mode, SudoPath: cfg.Privileged.SudoPath},
+		Guests: func(ctx context.Context) ([]int, error) {
+			if px == nil {
+				return nil, fmt.Errorf("no proxmox client")
+			}
+			gs, err := px.ListLXC(ctx)
+			if err != nil {
+				return nil, err
+			}
+			var out []int
+			for _, g := range gs {
+				if g.Status != "running" {
+					continue
+				}
+				gc, err := px.GuestConfig(ctx, g.VMID)
+				if err != nil {
+					continue
+				}
+				for _, v := range gc.MountPoints() {
+					if src, _, _ := strings.Cut(v, ","); src == "/mnt/felhom-drives" {
+						out = append(out, g.VMID)
+						break
+					}
+				}
+			}
+			return out, nil
+		},
+	}
+}

cmd/felhom-agent/selftest_flag_test.go

@@ -0,0 +1,39 @@
+package main
+
+import (
+	"os"
+	"regexp"
+	"strings"
+	"testing"
+)
+
+// Every mode the dispatcher (`switch selftest.mode`) runs must be ACCEPTED by the --selftest flag. Found live
+// 2026-10-04: --selftest=os-update had a dispatch case and a function but the flag's allow-list refused it, so the
+// debug action could not run. Red-proof: drop the "os-update" case from selftestFlag.Set and this fails.
+func TestSelftestFlag_AcceptsEveryDispatchedMode(t *testing.T) {
+	src, err := os.ReadFile("main.go")
+	if err != nil {
+		t.Fatal(err)
+	}
+	s := string(src)
+	i := strings.Index(s, "switch selftest.mode {")
+	if i < 0 {
+		t.Fatal("dispatch switch not found")
+	}
+	block := s[i:]
+	block = block[:strings.Index(block, "\n\t}\n")]
+	modes := regexp.MustCompile(`(?m)^\tcase "([a-z-]+)":`).FindAllStringSubmatch(block, -1)
+	if len(modes) < 5 {
+		t.Fatalf("parsed only %d dispatch cases — the parser is wrong", len(modes))
+	}
+	var bad []string
+	for _, m := range modes {
+		var f selftestFlag
+		if err := f.Set(m[1]); err != nil {
+			bad = append(bad, m[1])
+		}
+	}
+	if len(bad) > 0 {
+		t.Fatalf("dispatched but refused by --selftest: %v", bad)
+	}
+}

configs/felhom-agent.sudoers

@@ -299,10 +299,17 @@ Cmnd_Alias FELHOM_SELFHEAL = \
 # argument after the numeric vmid is a literal, so the grant cannot be widened by anything the guest or
 # the hub says. The address read is deliberately NOT duplicated here — it is already FELHOM_DNSMASQ's,
 # and the same command must not be granted twice under two names.
+# OS updates, guest fast lane (`11-os-updates.md` §5.4.1, agent v0.140.0). The ONLY entry: the root wrapper with
+# one plan file in the agent's own os/ dir. Every safety rule (no removal, no downgrade, no new or unlisted package,
+# Debian origin only, the box's own customer guest only) lives in the wrapper, red-proved per rule
+# (configs/test_felhom_os_apply.py). The agent gets NO apt grant of its own.
+Cmnd_Alias FELHOM_OSAPPLY = \
+  /usr/local/sbin/felhom-os-apply --plan /var/lib/felhom-agent/os/plan-*.json
+
 Cmnd_Alias FELHOM_GUESTNET = \
   /usr/sbin/pct exec [0-9]* -- ip route show default, \
   /usr/sbin/pct exec [0-9]* -- cat /etc/network/interfaces, \
   /usr/sbin/pct exec [0-9]* -- pgrep -x dhclient, \
   /usr/sbin/pct exec [0-9]* -- dhclient -pf /run/dhclient.eth0.pid -lf /var/lib/dhcp/dhclient.eth0.leases eth0
 
-felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN
+felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY

configs/felhom-os-apply

@@ -0,0 +1,593 @@
+#!/usr/bin/python3
+# felhom-os-apply — the ROOT half of the agent's operating-system update leg (`11-os-updates.md` §5.4.1, §8.1–8.2).
+#
+# Install as /usr/local/sbin/felhom-os-apply (0755 root:root). The non-root agent invokes it via `sudo -n`
+# (FELHOM_OSAPPLY alias) with EXACTLY:   felhom-os-apply --plan /var/lib/felhom-agent/os/plan-<id>.json
+# Nothing else on the command line is accepted. Python 3, standard library only (a JSON plan cannot be parsed
+# safely in sh). Tests: configs/test_felhom_os_apply.py (a fake runner; nothing real is executed).
+#
+# THE TRUST MODEL. The plan is written by the agent, so a broken-into agent writes whatever plan it likes. The
+# protection is therefore what this file REFUSES, not where the plan came from: no removal, no downgrade, no new
+# package, no package outside the plan, only Debian origin in the fast lane, no kernel / boot package on the host,
+# only the box's own customer guest, and the host layer only on a box whose ROOT-OWNED install record says
+# "appliance" (a BYO host belongs to its owner, `11` §1). Package signatures stay Debian's: apt checks every Release
+# file, including the snapshot.debian.org fallback (decision 79). Nothing here is overridable from the environment.
+#
+# LAYERS (agent v0.141.0): "guest" (the customer LXC, entered with `pct exec`) and "host" (this Proxmox host, run
+# directly). LANE: "fast" only — the slow lane (kernel, Proxmox, Docker) is REFUSED (R3, R14) until `11` §8 steps 5–6.
+#
+# Modes (plan field "mode"):
+#   inventory   `apt-get update`, then report what is installed (with origin), what is pending, and health.
+#   apply       repair first, pick the packages (select "listed": the plan's name=version list; "pending-fast": every
+#               pending Debian / Debian-Security upgrade, for ring 0), check every refusal on an `apt-get -s`
+#               simulation of EXACTLY name=version, install, clean, scan for restart-needed, report as inventory.
+#   health      report health only (the agent polls it after a run).
+# Output: log lines on stderr and the journal (tag felhom-os-apply); the LAST stdout line is
+#   OSAPPLY-REPORT <one JSON object>
+# which is what the agent parses. Exit 0 = done; 2 = refused (nothing changed); 3 = failed during install.
+#
+# SPEED (R-845, agent v0.141.0). Every `pct exec` costs ~0.9 s (measured on demo-hp), and v0.140.0 made one per
+# package for version comparisons — 272 packages ≈ 4 minutes. Versions are now compared with the HOST's dpkg (the same
+# Debian algorithm), madison/policy run once per pass for all packages, the restart scan runs only after an install,
+# and one wrapper call does the whole pass (no separate inventory call before an apply).
+import json
+import os
+import re
+import stat
+import subprocess
+import sys
+import time
+
+PLAN_DIR = "/var/lib/felhom-agent/os"
+PLAN_RE = re.compile(r"^plan-[A-Za-z0-9._-]{1,80}\.json$")
+AGENT_USER = "felhom-agent"
+FAST_ORIGINS = ("Debian", "Debian-Security")
+# Debian package name and version grammar (Debian policy §5.6.1, §5.6.12).
+NAME_RE = re.compile(r"^[a-z0-9][a-z0-9+.-]+$")
+VERSION_RE = re.compile(r"^(?:[0-9]+:)?[0-9][A-Za-z0-9.+~-]*$")
+SNAP_RE = re.compile(r"^[0-9]{8}T[0-9]{6}Z$")
+RESERVED_VMIDS = set(range(990000, 990010)) | {9999}
+DRIVES_PARENT = "/mnt/felhom-drives"
+SNAPSHOT_LIST = "/etc/apt/sources.list.d/felhom-os-snapshot.list"
+APT_ENV = ["env", "DEBIAN_FRONTEND=noninteractive", "APT_LISTCHANGES_FRONTEND=none", "NEEDRESTART_MODE=l", "LC_ALL=C"]
+DPKG_OPTS = ["-o", "Dpkg::Options::=--force-confold", "-o", "Dpkg::Options::=--force-confdef"]
+MIN_FREE = 500 * 1024 * 1024
+# The installer's ROOT-OWNED record (felhom-host-install.sh `state_set mode`); the agent cannot write it.
+INSTALL_STATE = "/var/lib/felhom-install/state.json"
+# Kernel, boot and firmware packages are the SLOW lane on the host whatever their origin (`11` C3, §5.2): a host
+# reboot is needed for them to take effect, and a bad one can stop the box from booting.
+HOST_SLOW_RE = re.compile(r"^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|"
+                          r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr)")
+# restart_needed() leaves out processes whose cgroup line matches (grep basic regex). Host: the LXC guests' own
+# processes (`0::/lxc/<vmid>/...`) -- NOT lxc-start itself, whose cgroup is `0::/lxc.monitor/<vmid>` (measured
+# 2026-10-04 on demo-felhom: the old pattern "lxc" hid lxc-start with 20 deleted maps, so "reboot needed" stayed false
+# after a libc6 update). Pinned by test_restart_skip_patterns_against_real_cgroups.
+RESTART_SKIP_CGROUP = {"guest": "docker", "host": ":/lxc/"}
+HOST_SERVICES = ["pveproxy", "pvedaemon", "pvestatd", "pve-cluster", "felhom-agent"]
+
+
+class Refused(Exception):
+    def __init__(self, code, reason):
+        super().__init__(f"{code} {reason}")
+        self.code, self.reason = code, reason
+
+
+class Runner:
+    """Runs commands for real. Tests replace it with a fake. `guest` runs inside the container via pct exec."""
+
+    def host(self, argv, timeout=600, stdin=None):
+        p = subprocess.run(argv, capture_output=True, text=True, timeout=timeout, input=stdin)
+        return p.returncode, p.stdout, p.stderr
+
+    def guest(self, vmid, argv, timeout=1800):
+        return self.host(["/usr/sbin/pct", "exec", str(vmid), "--"] + argv, timeout)
+
+    def read_file(self, path):
+        with open(path) as f:
+            return f.read()
+
+    def stat(self, path):
+        return os.lstat(path)
+
+    def agent_uid(self):
+        import pwd
+        return pwd.getpwnam(AGENT_USER).pw_uid
+
+    def write_file(self, layer, vmid, path, body):
+        """Write a small text file in the target layer — never via a shell string."""
+        if layer == "host":
+            with open(path, "w") as f:
+                f.write(body)
+            return
+        rc, _, _ = self.host(["/usr/sbin/pct", "exec", str(vmid), "--", "tee", path], 60, stdin=body)
+        if rc != 0:
+            raise Refused("R7", f"could not write {path} in the guest")
+
+    def log(self, line):
+        print(line, file=sys.stderr, flush=True)
+        try:
+            subprocess.run(["logger", "-t", "felhom-os-apply", line], timeout=10)
+        except Exception:
+            pass
+
+
+class Apply:
+    def __init__(self, runner, plan_path):
+        self.r = runner
+        self.plan_path = plan_path
+        self.report = {"refused": None, "mode": None}
+
+    # ---------- checks ----------
+    def load_plan(self):
+        p = self.plan_path
+        d, base = os.path.dirname(p), os.path.basename(p)
+        if d != PLAN_DIR or not PLAN_RE.match(base) or ".." in p:
+            raise Refused("R1", f"the plan must be {PLAN_DIR}/plan-<id>.json, got {p!r}")
+        try:
+            st = self.r.stat(p)
+        except OSError as e:
+            raise Refused("R1", f"cannot stat the plan: {e}")
+        if not stat.S_ISREG(st.st_mode):
+            raise Refused("R1", "the plan is not a regular file (a symlink or a device is refused)")
+        if st.st_uid != self.r.agent_uid():
+            raise Refused("R1", f"the plan is not owned by {AGENT_USER}")
+        if st.st_size > 2 * 1024 * 1024:
+            raise Refused("R1", "the plan is larger than 2 MB")
+        try:
+            plan = json.loads(self.r.read_file(p))
+        except (OSError, ValueError) as e:
+            raise Refused("R1", f"the plan is not valid JSON: {e}")
+        if not isinstance(plan, dict):
+            raise Refused("R1", "the plan is not a JSON object")
+        return plan
+
+    def check_plan(self, plan):
+        mode = plan.get("mode", "apply")
+        if mode not in ("apply", "inventory", "health"):
+            raise Refused("R11", f"unknown mode {mode!r}")
+        layer = plan.get("layer")
+        if layer not in ("guest", "host"):
+            raise Refused("R12", f"layer {layer!r} is not guest or host")
+        if plan.get("lane", "fast") != "fast":
+            raise Refused("R3", "the slow lane is refused in this release")
+        vmid = plan.get("vmid")
+        if not isinstance(vmid, int) or isinstance(vmid, bool) or vmid <= 0:
+            raise Refused("R11", f"vmid must be a positive integer, got {vmid!r}")
+        rid = plan.get("release_id", "")
+        if not isinstance(rid, str) or not re.match(r"^[A-Za-z0-9._:-]{1,80}$", rid):
+            raise Refused("R11", f"release_id {rid!r} is not a plain id")
+        if plan.get("allow_new"):
+            raise Refused("R6", "allow_new is a slow-lane field; the fast lane never adds a package")
+        select = plan.get("select", "listed")
+        if select not in ("listed", "pending-fast"):
+            raise Refused("R11", f"unknown select {select!r}")
+        pk = plan.get("packages", [])
+        if not isinstance(pk, list):
+            raise Refused("R11", "packages must be a list")
+        if mode == "apply" and select == "listed" and not pk:
+            raise Refused("R11", "packages must be a non-empty list in apply mode (select listed)")
+        if select == "pending-fast" and pk:
+            raise Refused("R11", "select pending-fast takes no package list")
+        seen = set()
+        for e in pk:
+            if not isinstance(e, dict):
+                raise Refused("R11", "every package entry must be an object")
+            n, v, o = e.get("name"), e.get("version"), e.get("origin")
+            if not isinstance(n, str) or not NAME_RE.match(n):
+                raise Refused("R11", f"package name {n!r} is not a Debian package name")
+            if not isinstance(v, str) or not VERSION_RE.match(v):
+                raise Refused("R11", f"version {v!r} of {n} is not a Debian version string")
+            if n in seen:
+                raise Refused("R11", f"package {n} is named twice")
+            seen.add(n)
+            if o not in FAST_ORIGINS:
+                raise Refused("R2", f"{n}: origin {o!r} is not Debian / Debian-Security (the fast lane, `11` C3)")
+            if layer == "host" and HOST_SLOW_RE.match(n):
+                raise Refused("R14", f"{n} is a kernel / boot / firmware package — the host's slow lane")
+        snap = plan.get("snapshot", "")
+        if snap and not SNAP_RE.match(snap):
+            raise Refused("R11", f"snapshot {snap!r} is not YYYYMMDDTHHMMSSZ")
+        return mode, layer, vmid, select
+
+    def check_appliance(self):
+        """R12: the host layer only on a box whose ROOT-OWNED install record says appliance (`11` §1: never BYO)."""
+        try:
+            st = self.r.stat(INSTALL_STATE)
+        except OSError:
+            raise Refused("R12", f"no install record ({INSTALL_STATE}) — this box cannot prove it is an appliance")
+        if st.st_uid != 0 or (st.st_mode & 0o022):
+            raise Refused("R12", f"{INSTALL_STATE} is not root-owned and root-only-writable — it proves nothing")
+        try:
+            mode = json.loads(self.r.read_file(INSTALL_STATE)).get("mode")
+        except (OSError, ValueError, AttributeError):
+            raise Refused("R12", f"{INSTALL_STATE} is unreadable — this box cannot prove it is an appliance")
+        if mode != "appliance":
+            raise Refused("R12", f"this box was installed as {mode!r}, not appliance — its host belongs to its owner")
+
+    def check_guest(self, vmid):
+        if vmid in RESERVED_VMIDS:
+            raise Refused("R10", f"vmid {vmid} is a reserved scratch vmid")
+        try:
+            conf = self.r.read_file(f"/etc/pve/lxc/{vmid}.conf")
+        except OSError:
+            raise Refused("R10", f"vmid {vmid} is not a container on this host")
+        cur = conf.split("\n[", 1)[0]  # the current config, not a snapshot section
+        binds = [l for l in cur.splitlines() if re.match(r"^mp[0-9]+: " + re.escape(DRIVES_PARENT) + r",", l)]
+        if not binds:
+            raise Refused("R10", f"vmid {vmid} does not bind {DRIVES_PARENT} — it is not this box's customer guest")
+        lock = [l for l in cur.splitlines() if l.startswith("lock:")]
+        if lock:
+            raise Refused("R9", f"vmid {vmid} is locked ({lock[0].split(':', 1)[1].strip()}) — a backup or restore is running")
+        rc, out, _ = self.r.host(["/usr/sbin/pct", "status", str(vmid)])
+        if rc != 0 or "running" not in out:
+            raise Refused("R10", f"vmid {vmid} is not running")
+
+    # ---------- target helpers ----------
+    def x(self, argv, timeout=1800):
+        """Run in the TARGET layer: the guest via pct exec, or the host directly."""
+        if self.layer == "host":
+            return self.r.host(argv, timeout)
+        return self.r.guest(self.vmid, argv, timeout)
+
+    def g(self, argv, timeout=1800):
+        """Run in the customer GUEST whatever the layer (its health)."""
+        return self.r.guest(self.vmid, argv, timeout)
+
+    def dpkg_cmp(self, a, op, b):
+        # The HOST's dpkg: the same Debian version algorithm, and no `pct exec` (0.9 s) per comparison (R-845).
+        rc, _, _ = self.r.host(["dpkg", "--compare-versions", a, op, b], 30)
+        return rc == 0
+
+    def installed(self):
+        rc, out, _ = self.x(["dpkg-query", "-W", "-f", "${Package}\t${Version}\t${db:Status-Abbrev}\n"])
+        res = {}
+        for l in out.splitlines():
+            parts = l.split("\t")
+            if len(parts) == 3 and parts[2].startswith("ii"):
+                res[parts[0]] = parts[1]
+        return res
+
+    def madison_all(self, names):
+        """name -> set of downloadable versions, ONE call for all names."""
+        res = {n: set() for n in names}
+        if not names:
+            return res
+        rc, out, _ = self.x(["apt-cache", "madison"] + sorted(names))
+        for l in out.splitlines():
+            f = [x.strip() for x in l.split("|")]
+            if len(f) >= 3 and f[0] in res:
+                res[f[0]].add(f[1])
+        return res
+
+    def simulate(self, args):
+        rc, out, err = self.x(APT_ENV + ["apt-get", "-s", "-q"] + args)
+        inst, remv = [], []
+        for l in out.splitlines():
+            m = re.match(r"^Inst (\S+) (?:\[([^]]*)\] )?\((\S+) (.*?) \[[a-z0-9]+\]\)", l)
+            if m:
+                inst.append({"name": m.group(1), "from": m.group(2), "to": m.group(3), "origin": m.group(4)})
+            m = re.match(r"^Remv (\S+)", l)
+            if m:
+                remv.append(m.group(1))
+        return rc, inst, remv, out + err
+
+    @staticmethod
+    def origin_name(origin):
+        # "Debian:13.7/stable, Debian-Security:13/stable-security" -> {"Debian", "Debian-Security"}
+        return {o.strip().split(":")[0] for o in origin.split(",") if o.strip()}
+
+    def free_bytes(self):
+        rc, out, _ = self.x(["df", "-B1", "--output=avail", "/"])
+        try:
+            return int(out.strip().splitlines()[-1])
+        except (ValueError, IndexError):
+            return -1
+
+    def apt_lock_held(self):
+        rc, out, _ = self.x(["fuser", "/var/lib/dpkg/lock-frontend", "/var/lib/dpkg/lock"])
+        return rc == 0 and out.strip() != ""
+
+    def guest_health(self):
+        """The guest's signals: every container's state + health, the controller's own health, the network."""
+        rc, out, _ = self.g(["docker", "ps", "-a", "--format", "{{.Names}}\t{{.State}}\t{{.Status}}"], timeout=60)
+        cont = {}
+        for l in out.splitlines():
+            p = l.split("\t")
+            if len(p) == 3:
+                h = "healthy" if "(healthy)" in p[2] else "unhealthy" if "(unhealthy)" in p[2] else \
+                    "starting" if "(health: starting)" in p[2] else "none"
+                cont[p[0]] = {"state": p[1], "health": h}
+        nrc, _, _ = self.g(["getent", "hosts", "deb.debian.org"], timeout=30)
+        return {"docker_ok": rc == 0, "containers": cont,
+                "controller": cont.get("felhom-controller", {}).get("health", "absent"),
+                "network_ok": nrc == 0}
+
+    def health(self):
+        if self.layer == "guest":
+            return self.guest_health()
+        rc, out, _ = self.r.host(["systemctl", "is-active"] + HOST_SERVICES, 30)
+        states = out.split()
+        svc = {s: (states[i] if i < len(states) else "unknown") for i, s in enumerate(HOST_SERVICES)}
+        src, sout, _ = self.r.host(["/usr/sbin/pct", "status", str(self.vmid)], 30)
+        running = src == 0 and "running" in sout
+        return {"host_services": svc, "guest_running": running, "guest": self.guest_health() if running else None}
+
+    def restart_needed(self):
+        """Processes still mapping deleted files, OUTSIDE containers (C11). Guest: outside docker; host: outside the
+        LXC guests (the host's /proc shows guest processes too)."""
+        skip = RESTART_SKIP_CGROUP[self.layer]
+        script = ('for p in /proc/[0-9]*; do grep -q "(deleted)" $p/maps 2>/dev/null || continue; '
+                  'grep -q "%s" $p/cgroup 2>/dev/null && continue; echo "${p#/proc/} $(cat $p/comm 2>/dev/null)"; done' % skip)
+        rc, out, _ = self.x(["sh", "-c", script], timeout=120)
+        lines = [l for l in out.splitlines() if " " in l]
+        procs = sorted({l.split(" ", 1)[1] for l in lines})
+        pid1 = any(l.split(" ", 1)[0] == "1" for l in lines)
+        return procs, pid1 or "lxc-start" in procs
+
+    def inventory(self, inst=None):
+        inst = inst if inst is not None else self.installed()
+        names = sorted(inst)
+        origins = {}
+        if names:
+            rc, out, _ = self.x(["apt-cache", "policy"] + names)  # ONE call (R-845)
+            cur, star = None, False
+            for l in out.splitlines():
+                if not l.startswith(" "):
+                    cur, star = l.rstrip(":"), False
+                    continue
+                s = l.strip()
+                if s.startswith("*** "):
+                    star = True
+                    continue
+                if star and cur and re.match(r"^[0-9-]+ ", s):
+                    if "/var/lib/dpkg/status" in s:
+                        origins.setdefault(cur, "local")
+                    else:
+                        origins[cur] = s
+                    continue
+                if star and not re.match(r"^[0-9-]+ ", s):
+                    star = False
+
+        def oname(src):
+            if src in (None, "local"):
+                return "unknown"
+            if "proxmox" in src:
+                return "Proxmox"
+            if "security" in src and "debian" in src:
+                return "Debian-Security"
+            if "docker.com" in src:
+                return "Docker"
+            if "debian" in src:
+                return "Debian"
+            return "other"
+        rc, pend, remv, _ = self.simulate(["dist-upgrade"])
+        self._pending = pend
+        return {
+            "installed": [{"name": n, "version": inst[n], "origin": oname(origins.get(n))} for n in names],
+            "pending": [{"name": p["name"], "from": p["from"], "to": p["to"],
+                         "origin": sorted(self.origin_name(p["origin"]))} for p in pend],
+        }
+
+    # ---------- the run ----------
+    def run(self):
+        plan = self.load_plan()
+        self.mode, self.layer, self.vmid, self.select = self.check_plan(plan)
+        self.report.update(mode=self.mode, layer=self.layer, release_id=plan.get("release_id"), vmid=self.vmid)
+        if self.layer == "host":
+            self.check_appliance()
+        self.check_guest(self.vmid)
+        log = self.r.log
+        if self.mode == "health":
+            self.report["health"] = self.health()
+            return 0
+        log(f"os-apply: START release={plan.get('release_id')} layer={self.layer}" +
+            (f":{self.vmid}" if self.layer == "guest" else "") +
+            f" lane=fast mode={self.mode} select={self.select} packages={len(plan.get('packages', []))}")
+        if self.apt_lock_held():
+            raise Refused("R9", f"another apt/dpkg holds the lock on the {self.layer}")
+        self.report["health_before"] = self.health()
+        if self.mode == "apply":
+            self.repair()
+        rc, out, err = self.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
+        if rc != 0:
+            raise Refused("R7", f"apt-get update failed on the {self.layer}: {(out + err).strip().splitlines()[-1:]}")
+        installed_after = None
+        if self.mode == "apply":
+            rc, installed_after = self.apply(plan)
+            if rc:
+                return rc
+        self.report.update(self.inventory(installed_after))
+        if self.layer == "host" and "reboot_needed" not in self.report:
+            # The host is scanned on EVERY pass (local, no pct exec): a reboot must CLEAR "reboot needed", or the hub's
+            # 14-day alarm fires on a host that was rebooted long ago. The guest still scans only after an install
+            # (R-845; one pct exec, and no alarm reads it). Pinned by test_host_scans_every_pass_guest_only_after_install.
+            self.report["restart_needed"], self.report["reboot_needed"] = self.restart_needed()
+        self.report["reboot_scanned"] = "reboot_needed" in self.report
+        self.report["health_after"] = self.health()
+        return 0
+
+    def repair(self):
+        rc, before, _ = self.x(["dpkg", "--audit"])
+        configured = len([l for l in before.splitlines() if l.startswith(" ")])
+        fixed = 0
+        after = ""
+        if before.strip():  # nothing half-done → nothing to run (R-845: two calls saved on every clean pass)
+            self.x(APT_ENV + ["dpkg", "--configure", "-a", "--force-confold"])
+            rc2, out, err = self.x(APT_ENV + ["apt-get", "-f", "install", "-y", "-q"] + DPKG_OPTS)
+            _, after, _ = self.x(["dpkg", "--audit"])
+            fixed = len(re.findall(r"^Setting up ", out, re.M))
+        self.report["repair"] = {"half_configured_before": configured, "fixed": fixed, "clean_after": after.strip() == ""}
+        self.r.log(f"os-apply: REPAIR configured={configured} fixed={fixed}")
+        if after.strip():
+            raise Refused("R13", "dpkg is still broken after the repair: " + after.strip().splitlines()[0])
+
+    def pending_fast(self):
+        """Ring 0 (select pending-fast): every pending upgrade of an INSTALLED package whose every origin is Debian /
+        Debian-Security — and, on the host, not a kernel / boot / firmware package."""
+        rc, pend, remv, _ = self.simulate(["dist-upgrade"])
+        out = []
+        for p in pend:
+            o = self.origin_name(p["origin"])
+            if p["from"] is None or not o or not o <= set(FAST_ORIGINS):
+                continue
+            if self.layer == "host" and HOST_SLOW_RE.match(p["name"]):
+                continue
+            out.append({"name": p["name"], "version": p["to"], "origin": "Debian-Security" if "Debian-Security" in o else "Debian"})
+        return out
+
+    def apply(self, plan):
+        log = self.r.log
+        packages = plan["packages"] if self.select == "listed" else self.pending_fast()
+        inst = self.installed()
+        upgrade, already, notinst = [], 0, 0
+        for e in packages:
+            n, v = e["name"], e["version"]
+            if n not in inst:
+                notinst += 1
+                continue
+            if not self.dpkg_cmp(v, "gt", inst[n]):
+                already += 1
+                continue
+            upgrade.append((n, v))
+        from_snap = 0
+        if upgrade:
+            avail = self.madison_all([n for n, _ in upgrade])
+            missing = [(n, v) for n, v in upgrade if v not in avail[n]]
+        else:
+            missing = []
+        if missing:
+            snap = plan.get("snapshot", "")
+            if not snap:
+                raise Refused("R7", f"{missing[0][0]}={missing[0][1]} is not downloadable and the plan names no snapshot")
+            self.add_snapshot_sources(snap)
+            avail = self.madison_all([n for n, _ in missing])
+            still = [(n, v) for n, v in missing if v not in avail[n]]
+            if still:
+                self.remove_snapshot_sources()
+                raise Refused("R7", f"{still[0][0]}={still[0][1]} is not downloadable, not even from snapshot {snap}")
+            from_snap = len(missing)
+        try:
+            log(f"os-apply: PLAN upgrade={len(upgrade)} already={already} not-installed={notinst} from-snapshot={from_snap}")
+            self.report["plan"] = {"upgrade": len(upgrade), "already": already, "not_installed": notinst, "from_snapshot": from_snap}
+            if not upgrade:
+                self.report["upgraded"] = []
+                log("os-apply: DONE rc=0 seconds=0 upgraded=0 (nothing to do)")
+                return 0, inst
+            args = ["install", "--only-upgrade", "--no-install-recommends"] + [f"{n}={v}" for n, v in upgrade]
+            rc, sim, remv, text = self.simulate(args)
+            if rc != 0:
+                tail = text.strip().splitlines()[-1] if text.strip() else ""
+                raise Refused("R7", "the simulation failed: " + tail)
+            if remv:
+                raise Refused("R4", f"the plan would remove {', '.join(remv[:5])}")
+            want = dict(upgrade)
+            for p in sim:
+                if p["from"] is None:
+                    raise Refused("R6", f"the plan would add a package that is not installed: {p['name']}")
+                if p["name"] not in want:
+                    raise Refused("R6", f"the plan would touch {p['name']}, which is not in the plan")
+                if p["to"] != want[p["name"]]:
+                    raise Refused("R6", f"{p['name']} would go to {p['to']}, not the approved {want[p['name']]}")
+                if not self.dpkg_cmp(p["to"], "gt", p["from"]):
+                    raise Refused("R5", f"{p['name']} would be downgraded {p['from']} -> {p['to']}")
+                if not self.origin_name(p["origin"]) & set(FAST_ORIGINS):
+                    raise Refused("R2", f"{p['name']} would come from {p['origin']}, not Debian")
+                if self.layer == "host" and HOST_SLOW_RE.match(p["name"]):
+                    raise Refused("R14", f"{p['name']} is a kernel / boot / firmware package — the host's slow lane")
+            need = self.download_bytes(args)
+            free = self.free_bytes()
+            if free >= 0 and free < max(MIN_FREE, 3 * need):
+                raise Refused("R8", f"free space {free} B is below max(500 MB, 3 x download {need} B)")
+            t0 = time.time()
+            rc, out, err = self.x(APT_ENV + ["apt-get", "-y", "-q"] + DPKG_OPTS + args)
+            secs = time.time() - t0
+            # dpkg says "Installing new version of config file X" when X was NOT changed locally (the package's new
+            # version is taken), and "Configuration file 'X'" + "Keeping old config file" when it was (--force-confold
+            # keeps the local one; the package's version lands as X.dpkg-dist). Measured live 2026-10-04 (debian_version).
+            conflict = None
+            for l in (out + err).splitlines():
+                m = re.search(r"Installing new version of config file (\S+?)\s*\.\.\.", l)
+                if m:
+                    log(f"os-apply: CONFFILE updated {m.group(1)} (it was not changed locally)")
+                m = re.search(r"Configuration file '([^']+)'", l)
+                if m:
+                    conflict = m.group(1)
+                if conflict and "Keeping old config file" in l:
+                    log(f"os-apply: CONFFILE kept {conflict} (changed locally; the package's version is {conflict}.dpkg-dist)")
+                    self.report.setdefault("conffiles_kept", []).append(conflict)
+                    conflict = None
+            self.x(["apt-get", "clean"])
+            if rc != 0:
+                _, aud, _ = self.x(["dpkg", "--audit"])
+                first = aud.strip().splitlines()[0] if aud.strip() else "clean"
+                log(f"os-apply: FAILED rc={rc} step=install — dpkg state: {first}")
+                self.report["failed"] = {"rc": rc, "dpkg_audit": first, "tail": (out + err).strip().splitlines()[-3:]}
+                return 3, None
+            self.report["upgraded"] = [{"name": n, "version": v} for n, v in upgrade]
+            self.report["seconds"] = round(secs, 1)
+            procs, reboot = self.restart_needed()  # only after an install (R-845)
+            self.report["restart_needed"] = procs
+            self.report["docker_restart_needed"] = any(p in ("dockerd", "containerd") for p in procs)
+            self.report["reboot_needed"] = reboot
+            log(f"os-apply: DONE rc=0 seconds={secs:.1f} upgraded={len(upgrade)} restart-needed={','.join(procs) or '-'} reboot-needed={'yes' if reboot else 'no'}")
+            return 0, None
+        finally:
+            if from_snap:
+                self.remove_snapshot_sources()
+
+    def download_bytes(self, args):
+        rc, out, _ = self.x(APT_ENV + ["apt-get", "-s", "-o", "Debug::NoLocking=1", "--print-uris", "-q"] + args)
+        total = 0
+        for l in out.splitlines():
+            m = re.match(r"^'[^']+' \S+ ([0-9]+) ", l)
+            if m:
+                total += int(m.group(1))
+        return total
+
+    def add_snapshot_sources(self, snap):
+        rc, out, _ = self.x(["sh", "-c", ". /etc/os-release && echo $VERSION_CODENAME"])
+        code = out.strip()
+        if not re.match(r"^[a-z]+$", code):
+            raise Refused("R7", f"cannot read the {self.layer}'s Debian codename ({code!r})")
+        body = (f"deb [check-valid-until=no] http://snapshot.debian.org/archive/debian/{snap} {code} main\n"
+                f"deb [check-valid-until=no] http://snapshot.debian.org/archive/debian-security/{snap} {code}-security main\n")
+        self.r.write_file(self.layer, self.vmid, SNAPSHOT_LIST, body)
+        self.r.log(f"os-apply: SNAPSHOT using snapshot.debian.org/{snap} for versions no longer published (decision 79)")
+        rc, out, err = self.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
+        if rc != 0:
+            self.remove_snapshot_sources()
+            raise Refused("R7", "apt-get update against snapshot.debian.org failed")
+
+    def remove_snapshot_sources(self):
+        self.x(["rm", "-f", SNAPSHOT_LIST])
+        self.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
+
+
+def main(argv, runner=None):
+    r = runner or Runner()
+    if len(argv) != 3 or argv[1] != "--plan":
+        r.log("os-apply: REFUSED: R1 usage: felhom-os-apply --plan /var/lib/felhom-agent/os/plan-<id>.json")
+        print("OSAPPLY-REPORT " + json.dumps({"refused": {"code": "R1", "reason": "usage"}}))
+        return 2
+    a = Apply(r, argv[2])
+    t0 = time.time()
+    try:
+        rc = a.run()
+    except Refused as e:
+        r.log(f"os-apply: REFUSED: {e.code} {e.reason}")
+        a.report["refused"] = {"code": e.code, "reason": e.reason}
+        rc = 2
+    except subprocess.TimeoutExpired as e:
+        r.log(f"os-apply: FAILED rc=124 step=timeout — {e.cmd}")
+        a.report["failed"] = {"rc": 124, "timeout": str(e.cmd)[:200]}
+        rc = 3
+    a.report["pass_seconds"] = round(time.time() - t0, 1)
+    print("OSAPPLY-REPORT " + json.dumps(a.report, sort_keys=True))
+    return rc
+
+
+if __name__ == "__main__":
+    if os.geteuid() != 0:
+        print("felhom-os-apply: must run as root (via sudo)", file=sys.stderr)
+        sys.exit(2)
+    sys.exit(main(sys.argv))

configs/test_felhom_os_apply.py

@@ -0,0 +1,628 @@
+#!/usr/bin/env python3
+"""Tests for configs/felhom-os-apply (`11` §5.4.1). A fake runner plays the host and the guest: nothing is executed
+for real except the local `dpkg --compare-versions` (pure, no network). Every refusal R1–R13 has a test; the red-proof
+(each test fails when its rule is removed) is `audits/os-guest-lane-2026-10-04/partB/redproof.txt`.
+
+Run: python3 configs/test_felhom_os_apply.py   (also run by internal/osupdate's Go test)
+"""
+import importlib.machinery
+import importlib.util
+import json
+import os
+import pathlib
+import re
+import stat as statmod
+import subprocess
+import unittest
+
+HERE = pathlib.Path(__file__).resolve().parent
+_loader = importlib.machinery.SourceFileLoader("osapply", os.environ.get("OSAPPLY_UNDER_TEST", str(HERE / "felhom-os-apply")))  # red-proof seam
+_spec = importlib.util.spec_from_loader("osapply", _loader)
+osapply = importlib.util.module_from_spec(_spec)
+_loader.exec_module(osapply)
+
+PLAN = "/var/lib/felhom-agent/os/plan-t1.json"
+CONF_OK = ("arch: amd64\nmp0: local-lvm:vm-9201-disk-1,mp=/var/lib/felhom,backup=1,size=70G\n"
+           "mp8: /mnt/felhom-drives,mp=/mnt/felhom-drives\nrootfs: local-lvm:vm-9201-disk-0,size=32G\n")
+DEB = "Debian:13.7/stable"
+SEC = "Debian-Security:13/stable-security"
+
+
+def dpkg_cmp(a, op, b):
+    return subprocess.run(["dpkg", "--compare-versions", a, op, b]).returncode == 0
+
+
+class St:
+    def __init__(self, mode=statmod.S_IFREG | 0o600, uid=999, size=100):
+        self.st_mode, self.st_uid, self.st_size = mode, uid, size
+
+
+class Fake:
+    """The host + one guest. `installed` / `live` (name -> versions in the live archive) / `snapshot` (versions
+    the snapshot archive adds) / `extra_sim` (lines the simulation adds) / `dpkg_audit` / `free`."""
+
+    def __init__(self):
+        self.plan = {"release_id": "os-t1", "layer": "guest", "lane": "fast", "vmid": 9201, "mode": "apply",
+                     "snapshot": "20261004T080000Z",
+                     "packages": [{"name": "libc6", "version": "2.41-12+deb13u4", "origin": "Debian"},
+                                  {"name": "openssl", "version": "3.5.7-1~deb13u3", "origin": "Debian-Security"}]}
+        self.files = {"/etc/pve/lxc/9201.conf": CONF_OK}
+        self.stats = {PLAN: St()}
+        self.installed = {"libc6": "2.41-12+deb13u3", "openssl": "3.5.6-1~deb13u1", "bash": "5.2.37-2+b9"}
+        self.live = {"libc6": {"2.41-12+deb13u4"}, "openssl": {"3.5.7-1~deb13u3"}}
+        self.snapshot = {}
+        self.snap_active = False
+        self.extra_sim = []
+        self.dpkg_audit = ""
+        self.free = 10 * 1024 ** 3
+        self.install_rc = 0
+        self.calls = []
+        self.logs = []
+        self.written = {}
+        self.status = "status: running"
+        self.lock_held = False
+        self.services = {}
+        self.files[osapply.INSTALL_STATE] = json.dumps({"mode": "appliance"})
+        self.stats[osapply.INSTALL_STATE] = St(mode=statmod.S_IFREG | 0o644, uid=0)
+
+    # Runner interface
+    def read_file(self, p):
+        if p == PLAN:
+            return json.dumps(self.plan)
+        if p not in self.files:
+            raise OSError("no such file")
+        return self.files[p]
+
+    def stat(self, p):
+        if p not in self.stats:
+            raise OSError("no such file")
+        return self.stats[p]
+
+    def agent_uid(self):
+        return 999
+
+    def log(self, line):
+        self.logs.append(line)
+
+    def host(self, argv, timeout=600, stdin=None):
+        self.calls.append(("host", argv))
+        if argv[0] == "/usr/sbin/pct" and argv[1] == "status":
+            return 0, self.status + "\n", ""
+        if argv[0] == "dpkg" and argv[1] == "--compare-versions":
+            return (0 if dpkg_cmp(argv[2], argv[3], argv[4]) else 1), "", ""
+        if argv[0] == "systemctl" and argv[1] == "is-active":
+            return 0, "\n".join(self.services.get(s, "active") for s in argv[2:]) + "\n", ""
+        return self.emulate(argv)
+
+    def write_file(self, layer, vmid, path, body):
+        self.written[path] = body
+        self.write_layer = layer
+        if path == osapply.SNAPSHOT_LIST:
+            self.snap_active = True
+
+    def avail(self, n):
+        v = set(self.live.get(n, set()))
+        if self.snap_active:
+            v |= self.snapshot.get(n, set())
+        return v
+
+    def guest(self, vmid, argv, timeout=1800):
+        self.calls.append(("guest", vmid, argv))
+        return self.emulate(argv)
+
+    def emulate(self, argv):
+        a = [x for x in argv if not re.match(r"^[A-Z_]+=", x) and x != "env"]
+        cmd = a[0]
+        if cmd == "dpkg-query":
+            return 0, "".join(f"{n}\t{v}\tii \n" for n, v in self.installed.items()), ""
+        if cmd == "dpkg" and a[1] == "--compare-versions":
+            return (0 if dpkg_cmp(a[2], a[3], a[4]) else 1), "", ""
+        if cmd == "dpkg" and a[1] == "--audit":
+            return 0, self.dpkg_audit, ""
+        if cmd == "dpkg" and a[1] == "--configure":
+            return 0, "", ""
+        if cmd == "fuser":
+            return (0, " 123", "") if self.lock_held else (1, "", "")
+        if cmd == "apt-cache" and a[1] == "madison":
+            self.madison_calls = getattr(self, "madison_calls", 0) + 1
+            return 0, "".join(f" {n} | {v} | http://deb.debian.org trixie/main amd64 Packages\n" for n in a[2:] for v in self.avail(n)), ""
+        if cmd == "apt-cache" and a[1] == "policy":
+            out = ""
+            for n in a[2:]:
+                out += f"{n}:\n  Installed: {self.installed.get(n)}\n  Version table:\n *** {self.installed.get(n)} 500\n        500 http://deb.debian.org/debian trixie/main amd64 Packages\n"
+            return 0, out, ""
+        if cmd == "apt-get":
+            if "update" in a:
+                return 0, "", ""
+            if "clean" in a:
+                return 0, "", ""
+            if "-f" in a:
+                self.dpkg_audit = ""
+                return 0, "Setting up x (1) ...\n" if getattr(self, "repaired", False) else "", ""
+            if "-s" in a:
+                return self.sim(a)
+            if "install" in a:
+                if self.install_rc:
+                    return self.install_rc, "", "E: boom"
+                for x in a:
+                    if "=" in x and not x.startswith("-") and "::" not in x:
+                        n, v = x.split("=", 1)
+                        self.installed[n] = v
+                return 0, getattr(self, "install_out", "Setting up libc6 ...\n"), ""
+        if cmd == "df":
+            return 0, f"Avail\n{self.free}\n", ""
+        if cmd == "docker":
+            return 0, "felhom-controller\trunning\tUp 1 hour (healthy)\napp\trunning\tUp 1 hour (healthy)\n", ""
+        if cmd == "getent":
+            return 0, "1.2.3.4 deb.debian.org\n", ""
+        if cmd == "sh":
+            if "os-release" in a[2]:
+                return 0, "trixie\n", ""
+            if "(deleted)" in a[2]:
+                return 0, getattr(self, "restart_out", ""), ""
+            return 0, "", ""
+        if cmd == "rm":
+            self.snap_active = False
+            return 0, "", ""
+        return 1, "", f"unexpected guest call {a}"
+
+    def sim(self, a):
+        if "--print-uris" in a:
+            return 0, "'http://x/libc6.deb' libc6.deb 4000000 SHA256:x\n", ""
+        if "dist-upgrade" in a:
+            if getattr(self, "pending_sim", None) is not None and not getattr(self, "_pending_used", False):
+                self._pending_used = True
+                return 0, "\n".join(self.pending_sim) + "\n", ""
+            return 0, "Inst bash [5.2.37-2+b9] (5.2.37-2+b10 Debian:13.7/stable [amd64])\n", ""
+        out = ""
+        for x in a:
+            if "=" in x and not x.startswith("-") and "::" not in x:
+                n, v = x.split("=", 1)
+                if v not in self.avail(n):
+                    return 100, "", f"E: Version '{v}' for '{n}' was not found"
+                origin = SEC if n == "openssl" else DEB
+                out += f"Inst {n} [{self.installed[n]}] ({v} {origin} [amd64])\n"
+        out += "".join(l + "\n" for l in self.extra_sim)
+        return 0, out, ""
+
+
+def run(f):
+    import io
+    import contextlib
+    buf = io.StringIO()
+    with contextlib.redirect_stdout(buf):
+        rc = osapply.main(["felhom-os-apply", "--plan", PLAN], runner=f)
+    line = [l for l in buf.getvalue().splitlines() if l.startswith("OSAPPLY-REPORT ")][-1]
+    return rc, json.loads(line[len("OSAPPLY-REPORT "):])
+
+
+class Happy(unittest.TestCase):
+    def test_apply_installs_exactly_the_plan(self):
+        f = Fake()
+        rc, rep = run(f)
+        self.assertEqual(rc, 0, rep)
+        self.assertEqual(f.installed["libc6"], "2.41-12+deb13u4")
+        self.assertEqual(f.installed["openssl"], "3.5.7-1~deb13u3")
+        self.assertEqual(f.installed["bash"], "5.2.37-2+b9", "a package outside the plan was changed")
+        self.assertEqual(rep["plan"]["upgrade"], 2)
+        self.assertIn("installed", rep)
+        self.assertEqual(rep["pending"][0]["name"], "bash")
+        self.assertTrue(any(l.startswith("os-apply: REPAIR ") for l in f.logs), "the repair line must always print")
+        self.assertTrue(any(l.startswith("os-apply: DONE rc=0") for l in f.logs))
+        inst = [c for c in f.calls if c[0] == "guest" and "install" in c[2] and "-s" not in c[2] and "-f" not in c[2]]
+        self.assertTrue(inst and "Dpkg::Options::=--force-confold" in inst[0][2], "must keep existing config files")
+
+    def test_already_current_is_a_no_op(self):
+        f = Fake()
+        f.installed.update(libc6="2.41-12+deb13u4", openssl="3.5.7-1~deb13u3")
+        rc, rep = run(f)
+        self.assertEqual(rc, 0)
+        self.assertEqual(rep["plan"]["upgrade"], 0)
+
+    def test_inventory_installs_nothing(self):
+        f = Fake()
+        f.plan["mode"] = "inventory"
+        rc, rep = run(f)
+        self.assertEqual(rc, 0, rep)
+        self.assertEqual(f.installed["libc6"], "2.41-12+deb13u3")
+        self.assertIn("installed", rep)
+        self.assertNotIn("restart_needed", rep, "the restart scan runs only after an install (R-845)")
+
+    def test_health_mode(self):
+        f = Fake()
+        f.plan["mode"] = "health"
+        rc, rep = run(f)
+        self.assertEqual(rc, 0)
+        self.assertEqual(rep["health"]["controller"], "healthy")
+
+
+class Repair(unittest.TestCase):
+    def test_repair_runs_first_and_is_reported(self):
+        f = Fake()
+        f.dpkg_audit = "The following packages have been unpacked but not yet configured.\n perl  Larry Wall's\n"
+        f.repaired = True
+        rc, rep = run(f)
+        self.assertEqual(rc, 0, rep)
+        self.assertEqual(rep["repair"]["half_configured_before"], 1)
+        self.assertEqual(rep["repair"]["fixed"], 1)
+        order = [i for i, c in enumerate(f.calls) if c[0] == "guest" and c[2][-1:] != ["update"]]
+        first_cfg = next(i for i, c in enumerate(f.calls) if c[0] == "guest" and "--configure" in c[2])
+        first_upd = next(i for i, c in enumerate(f.calls) if c[0] == "guest" and "update" in c[2])
+        self.assertLess(first_cfg, first_upd, "the repair must run before anything else touches apt")
+        self.assertTrue(order)
+
+
+class Snapshot(unittest.TestCase):
+    def test_a_replaced_version_comes_from_the_snapshot(self):
+        f = Fake()
+        f.live["openssl"] = {"3.5.7-1~deb13u4"}  # Debian moved on
+        f.snapshot["openssl"] = {"3.5.7-1~deb13u3"}
+        rc, rep = run(f)
+        self.assertEqual(rc, 0, rep)
+        self.assertEqual(rep["plan"]["from_snapshot"], 1)
+        self.assertEqual(f.installed["openssl"], "3.5.7-1~deb13u3", "must install the APPROVED version, not the newer one")
+        body = f.written[osapply.SNAPSHOT_LIST]
+        self.assertIn("snapshot.debian.org/archive/debian/20261004T080000Z trixie main", body)
+        self.assertIn("debian-security/20261004T080000Z trixie-security main", body)
+        self.assertFalse(f.snap_active, "the temporary snapshot sources must be removed after the run")
+
+    def test_snapshot_does_not_have_it_either(self):
+        f = Fake()
+        f.live["openssl"] = set()
+        rc, rep = run(f)
+        self.assertEqual((rc, rep["refused"]["code"]), (2, "R7"))
+        self.assertFalse(f.snap_active)
+
+
+class Refusals(unittest.TestCase):
+    def refused(self, f, code):
+        rc, rep = run(f)
+        self.assertEqual(rc, 2, rep)
+        self.assertEqual(rep["refused"]["code"], code, rep)
+        self.assertTrue(any(l.startswith(f"os-apply: REFUSED: {code} ") for l in f.logs), f.logs)
+        inst = [c for c in f.calls if c[0] == "guest" and "install" in c[2] and "-s" not in c[2] and "-f" not in c[2]]
+        self.assertEqual(inst, [], "a refusal must install nothing")
+        return rep
+
+    def test_R1_usage(self):
+        import io
+        import contextlib
+        f = Fake()
+        with contextlib.redirect_stdout(io.StringIO()):
+            self.assertEqual(osapply.main(["felhom-os-apply", "--plan", PLAN, "--extra"], runner=f), 2)
+            self.assertEqual(osapply.main(["felhom-os-apply", "--plan"], runner=f), 2)
+
+    def test_R1_path_outside_the_plan_dir(self):
+        import io
+        import contextlib
+        f = Fake()
+        with contextlib.redirect_stdout(io.StringIO()):
+            rc = osapply.main(["felhom-os-apply", "--plan", "/tmp/plan-x.json"], runner=f)
+        self.assertEqual(rc, 2)
+        self.assertTrue(any("R1" in l for l in f.logs))
+
+    def test_R1_symlink(self):
+        f = Fake()
+        f.stats[PLAN] = St(mode=statmod.S_IFLNK | 0o777)
+        self.refused(f, "R1")
+
+    def test_R1_not_owned_by_the_agent(self):
+        f = Fake()
+        f.stats[PLAN] = St(uid=0)
+        self.refused(f, "R1")
+
+    def test_R2_non_debian_origin_in_the_plan(self):
+        f = Fake()
+        f.plan["packages"][0]["origin"] = "Proxmox"
+        self.refused(f, "R2")
+
+    def test_R2_non_debian_origin_in_the_simulation(self):
+        f = Fake()
+        f.installed["libc6"] = "2.41-12+deb13u3"
+        orig = f.sim
+
+        def sim(a):
+            rc, out, err = orig(a)
+            return rc, out.replace("Debian:13.7/stable", "Proxmox Debian Repository:stable"), err
+        f.sim = sim
+        self.refused(f, "R2")
+
+    def test_R3_slow_lane(self):
+        f = Fake()
+        f.plan["lane"] = "slow"
+        self.refused(f, "R3")
+
+    def test_R4_removal(self):
+        f = Fake()
+        f.extra_sim = ["Remv bash [5.2.37-2+b9]"]
+        self.refused(f, "R4")
+
+    def test_R5_downgrade(self):
+        f = Fake()
+        f.installed["libc6"] = "2.41-12+deb13u4"
+        f.plan["packages"] = [{"name": "openssl", "version": "3.5.7-1~deb13u3", "origin": "Debian-Security"}]
+        f.extra_sim = ["Inst openssl [3.5.6-1~deb13u1] (3.5.5-1 Debian:13.7/stable [amd64])"]
+        orig = f.sim
+
+        def sim(a):  # the simulation answers with a LOWER version than installed
+            rc, out, err = orig(a)
+            return rc, "\n".join(l for l in out.splitlines() if not l.startswith("Inst openssl [3.5.6-1~deb13u1] (3.5.7")) + "\n", err
+        f.sim = sim
+        f.plan["packages"][0]["version"] = "3.5.7-1~deb13u3"
+        rep = run(f)[1]
+        # The plan asks 3.5.7; the simulation goes to 3.5.5: that is BOTH a wrong version (R6) and a downgrade.
+        self.assertIn(rep["refused"]["code"], ("R5", "R6"))
+
+    def test_R5_downgrade_exact(self):
+        f = Fake()
+        f.plan["packages"] = [{"name": "openssl", "version": "3.5.7-1~deb13u3", "origin": "Debian-Security"}]
+        f.installed["openssl"] = "3.5.6-1~deb13u1"
+        orig = f.sim
+
+        def sim(a):
+            rc, out, err = orig(a)
+            return rc, out.replace("[3.5.6-1~deb13u1]", "[3.5.8-1]"), err
+        f.sim = sim
+        self.refused(f, "R5")
+
+    def test_R6_new_package(self):
+        f = Fake()
+        f.extra_sim = ["Inst newthing (1.0 Debian:13.7/stable [amd64])"]
+        self.refused(f, "R6")
+
+    def test_R6_unlisted_package(self):
+        f = Fake()
+        f.extra_sim = ["Inst bash [5.2.37-2+b9] (5.2.37-2+b10 Debian:13.7/stable [amd64])"]
+        self.refused(f, "R6")
+
+    def test_R6_allow_new_is_slow_lane(self):
+        f = Fake()
+        f.plan["allow_new"] = ["proxmox-kernel-x"]
+        self.refused(f, "R6")
+
+    def test_R7_not_downloadable_and_no_snapshot(self):
+        f = Fake()
+        f.live["openssl"] = set()
+        f.plan["snapshot"] = ""
+        self.refused(f, "R7")
+
+    def test_R8_free_space(self):
+        f = Fake()
+        f.free = 100 * 1024 * 1024
+        self.refused(f, "R8")
+
+    def test_R9_guest_locked_by_a_backup(self):
+        f = Fake()
+        f.files["/etc/pve/lxc/9201.conf"] = CONF_OK + "lock: backup\n"
+        self.refused(f, "R9")
+
+    def test_R9_apt_lock_held(self):
+        f = Fake()
+        f.lock_held = True
+        self.refused(f, "R9")
+
+    def test_R10_not_the_boxs_own_guest(self):
+        f = Fake()
+        f.files["/etc/pve/lxc/9201.conf"] = CONF_OK.replace("mp8: /mnt/felhom-drives,", "mp8: /mnt/hdd_1/scratch,")
+        self.refused(f, "R10")
+
+    def test_R10_reserved_vmid(self):
+        f = Fake()
+        f.plan["vmid"] = 990003
+        self.refused(f, "R10")
+
+    def test_R10_bind_only_in_a_snapshot_section(self):
+        f = Fake()
+        f.files["/etc/pve/lxc/9201.conf"] = "rootfs: x\n[snap1]\nmp8: /mnt/felhom-drives,mp=/mnt/felhom-drives\n"
+        self.refused(f, "R10")
+
+    def test_R10_not_running(self):
+        f = Fake()
+        f.status = "status: stopped"
+        self.refused(f, "R10")
+
+    def test_R11_duplicate(self):
+        f = Fake()
+        f.plan["packages"].append(dict(f.plan["packages"][0]))
+        self.refused(f, "R11")
+
+    def test_R11_bad_version_string(self):
+        f = Fake()
+        f.plan["packages"][0]["version"] = "1.0; rm -rf /"
+        self.refused(f, "R11")
+
+    def test_R11_bad_name(self):
+        f = Fake()
+        f.plan["packages"][0]["name"] = "--purge"
+        self.refused(f, "R11")
+
+    def test_R12_unknown_layer(self):
+        f = Fake()
+        f.plan["layer"] = "vm"
+        self.refused(f, "R12")
+
+    def test_R12_host_on_a_byo_box(self):
+        f = Fake()
+        f.plan["layer"] = "host"
+        f.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
+        self.refused(f, "R12")
+
+    def test_R12_host_without_an_install_record(self):
+        f = Fake()
+        f.plan["layer"] = "host"
+        del f.stats[osapply.INSTALL_STATE]
+        self.refused(f, "R12")
+
+    def test_R12_host_record_not_root_owned(self):
+        # the agent can write agent.json's deployment_mode; only a ROOT-owned record proves anything
+        f = Fake()
+        f.plan["layer"] = "host"
+        f.stats[osapply.INSTALL_STATE] = St(mode=statmod.S_IFREG | 0o644, uid=999)
+        self.refused(f, "R12")
+
+    def test_R14_kernel_package_in_a_host_plan(self):
+        f = Fake()
+        f.plan["layer"] = "host"
+        f.plan["packages"].append({"name": "linux-image-amd64", "version": "6.12.1-1", "origin": "Debian"})
+        self.refused(f, "R14")
+
+    def test_R14_kernel_package_pulled_by_the_simulation(self):
+        f = Fake()
+        f.plan["layer"] = "host"
+        f.extra_sim = ["Inst grub-common [2.12-9] (2.12-10 Debian:13.7/stable [amd64])"]
+        f.installed["grub-common"] = "2.12-9"
+        f.plan["packages"].append({"name": "grub-common", "version": "2.12-10", "origin": "Debian"})
+        self.refused(f, "R14")
+
+    def test_R13_repair_does_not_fix_it(self):
+        f = Fake()
+        f.dpkg_audit = "The following packages are broken\n perl\n"
+        orig = f.guest
+
+        def guest(vmid, argv, timeout=1800):
+            rc, out, err = orig(vmid, argv, timeout)
+            if "-f" in argv:
+                f.dpkg_audit = "The following packages are broken\n perl\n"
+            return rc, out, err
+        f.guest = guest
+        self.refused(f, "R13")
+
+
+class Conffiles(unittest.TestCase):
+    # dpkg's two shapes, measured live 2026-10-04: an unchanged file is UPDATED; a locally changed one is KEPT.
+    def test_updated_vs_kept(self):
+        f = Fake()
+        f.install_out = ("Installing new version of config file /etc/debian_version ...\n"
+                         "Configuration file '/etc/ssh/sshd_config'\n ==> Modified (by you or by a script) since installation.\n"
+                         " ==> Keeping old config file as default.\n")
+        rc, rep = run(f)
+        self.assertEqual(rc, 0, rep)
+        self.assertIn("os-apply: CONFFILE updated /etc/debian_version (it was not changed locally)", f.logs)
+        self.assertTrue(any(l.startswith("os-apply: CONFFILE kept /etc/ssh/sshd_config") for l in f.logs), f.logs)
+        self.assertEqual(rep["conffiles_kept"], ["/etc/ssh/sshd_config"])
+        self.assertFalse(any("kept /etc/debian_version" in l for l in f.logs), "an updated file must not be reported as kept")
+
+
+class HostLayer(unittest.TestCase):
+    def test_host_runs_on_the_host_not_in_the_guest(self):
+        f = Fake()
+        f.plan["layer"] = "host"
+        rc, rep = run(f)
+        self.assertEqual(rc, 0, rep)
+        inst = [c for c in f.calls if c[0] == "host" and "install" in c[1] and "-s" not in c[1] and "-f" not in c[1]]
+        self.assertTrue(inst, "the host install must run on the host")
+        self.assertFalse([c for c in f.calls if c[0] == "guest" and "install" in c[2]], "nothing installed in the guest")
+        self.assertEqual(sorted(rep["health_after"]["host_services"]), sorted(osapply.HOST_SERVICES))
+        self.assertTrue(rep["health_after"]["guest_running"])
+
+    def test_pending_fast_skips_proxmox_docker_and_kernel(self):
+        f = Fake()
+        f.plan["layer"] = "host"
+        f.plan["select"] = "pending-fast"
+        f.plan["packages"] = []
+        f.installed.update({"pve-manager": "9.2.2", "linux-image-amd64": "6.12.1", "docker-ce": "29.7"})
+        f.pending_sim = [
+            "Inst libc6 [2.41-12+deb13u3] (2.41-12+deb13u4 Debian:13.7/stable [amd64])",
+            "Inst openssl [3.5.6-1~deb13u1] (3.5.7-1~deb13u3 Debian:13.7/stable, Debian-Security:13/stable-security [amd64])",
+            "Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
+            "Inst linux-image-amd64 [6.12.1] (6.12.9 Debian:13.7/stable [amd64])",
+            "Inst docker-ce [29.7] (29.8 Docker CE:trixie [amd64])",
+            "Inst brand-new (1.0 Debian:13.7/stable [amd64])",
+        ]
+        rc, rep = run(f)
+        self.assertEqual(rc, 0, rep)
+        got = sorted(u["name"] for u in rep["upgraded"])
+        self.assertEqual(got, ["libc6", "openssl"], "pending-fast must take only installed, Debian-origin, non-kernel packages")
+
+    def test_reboot_needed_when_pid1_or_lxc_start(self):
+        f = Fake()
+        f.plan["layer"] = "host"
+        f.restart_out = "1 systemd\n2101 lxc-start\n530 sshd\n"
+        rc, rep = run(f)
+        self.assertEqual(rc, 0, rep)
+        self.assertTrue(rep["reboot_needed"])
+        self.assertIn("lxc-start", rep["restart_needed"])
+
+    def test_reboot_needed_for_lxc_start_alone(self):
+        # lxc-start runs the guest; only a guest restart (or a host reboot) replaces it
+        f = Fake()
+        f.plan["layer"] = "host"
+        f.restart_out = "2101 lxc-start\n530 sshd\n"
+        rc, rep = run(f)
+        self.assertTrue(rep["reboot_needed"], rep)
+
+    def test_host_scans_every_pass_guest_only_after_install(self):
+        # A host pass that installs nothing still scans, so a reboot clears the flag (the hub alarm reads it).
+        f = Fake()
+        f.plan["layer"] = "host"
+        f.plan["mode"] = "inventory"
+        f.restart_out = "2101 lxc-start\n"
+        rc, rep = run(f)
+        self.assertEqual(rc, 0, rep)
+        self.assertTrue(rep["reboot_scanned"], rep)
+        self.assertTrue(rep["reboot_needed"], rep)
+        f = Fake()
+        f.plan["layer"] = "host"
+        f.plan["mode"] = "inventory"
+        f.restart_out = ""  # after the reboot: nothing maps a deleted file
+        rc, rep = run(f)
+        self.assertTrue(rep["reboot_scanned"] and rep["reboot_needed"] is False, rep)
+        f = Fake()
+        f.plan["mode"] = "inventory"
+        rc, rep = run(f)
+        self.assertFalse(rep["reboot_scanned"], "the guest scans only after an install (R-845)")
+
+    def test_no_reboot_for_ordinary_daemons(self):
+        f = Fake()
+        f.plan["layer"] = "host"
+        f.restart_out = "530 sshd\n611 cron\n"
+        rc, rep = run(f)
+        self.assertFalse(rep["reboot_needed"], rep)
+
+
+class Speed(unittest.TestCase):
+    # R-845: no `pct exec` per package — version checks on the host, madison once, the restart scan only after an install.
+    def test_no_per_package_guest_calls(self):
+        f = Fake()
+        for i in range(40):
+            f.installed[f"pkg{i}"] = "1.0-1"
+            f.live[f"pkg{i}"] = {"1.0-2"}
+            f.plan["packages"].append({"name": f"pkg{i}", "version": "1.0-2", "origin": "Debian"})
+        rc, rep = run(f)
+        self.assertEqual(rc, 0, rep)
+        guest_cmp = [c for c in f.calls if c[0] == "guest" and "--compare-versions" in c[2]]
+        self.assertEqual(guest_cmp, [], "version comparisons must run on the host")
+        self.assertEqual(f.madison_calls, 1, "madison must run once for all packages")
+        guest_calls = len([c for c in f.calls if c[0] == "guest"])
+        self.assertLess(guest_calls, 30, f"{guest_calls} guest calls for 42 packages — something is per-package again")
+
+
+class Failure(unittest.TestCase):
+    def test_install_failure_is_rc3_with_dpkg_state(self):
+        f = Fake()
+        f.install_rc = 100
+        rc, rep = run(f)
+        self.assertEqual(rc, 3)
+        self.assertEqual(rep["failed"]["rc"], 100)
+        self.assertTrue(any(l.startswith("os-apply: FAILED rc=100 step=install") for l in f.logs))
+
+
+
+class RestartSkipPattern(unittest.TestCase):
+    """The cgroup filter runs as `grep -q PATTERN /proc/<pid>/cgroup`; check it with grep itself against the cgroup
+    lines measured on demo-felhom 2026-10-04."""
+
+    def grep(self, pattern, line):
+        return subprocess.run(["grep", "-q", pattern], input=line + "\n", text=True).returncode == 0
+
+    def test_restart_skip_patterns_against_real_cgroups(self):
+        host = osapply.RESTART_SKIP_CGROUP["host"]
+        self.assertTrue(self.grep(host, "0::/lxc/9201/ns/system.slice/docker.service"), "a guest process must be skipped")
+        self.assertFalse(self.grep(host, "0::/lxc.monitor/9201"), "lxc-start must NOT be skipped (it runs the guest)")
+        self.assertFalse(self.grep(host, "0::/system.slice/pve-cluster.service"), "a host daemon must NOT be skipped")
+        guest = osapply.RESTART_SKIP_CGROUP["guest"]
+        self.assertTrue(self.grep(guest, "0::/system.slice/docker-0123abcd.scope"))
+        self.assertFalse(self.grep(guest, "0::/system.slice/cron.service"))
+
+if __name__ == "__main__":
+    unittest.main()

internal/capability/manifest.go

@@ -123,6 +123,9 @@ var manifest = []Capability{
 	{"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""},
 	{"dnsmasq-write", "dnsmasq drop-in write", "/usr/bin/install", []string{"-m", "0644", "/tmp/felhom-resolver-x.conf", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},
 	{"dnsmasq-enable", "dnsmasq enable", "/usr/bin/systemctl", []string{"enable", "--now", "dnsmasq"}, false, ""},
+
+	// ---- OS updates, guest fast lane (`11` §5.4.1; the wrapper holds every rule) ----
+	{"osapply-run", "OS update wrapper (guest fast lane)", "/usr/local/sbin/felhom-os-apply", []string{"--plan", "/var/lib/felhom-agent/os/plan-x.json"}, false, ""},
 	{"dnsmasq-reload", "dnsmasq reload", "/usr/bin/systemctl", []string{"reload", "dnsmasq"}, false, ""},
 	{"dnsmasq-restart", "dnsmasq restart (LAN-DNS self-heal)", "/usr/bin/systemctl", []string{"restart", "dnsmasq"}, false, ""},
 	{"dnsmasq-rm", "dnsmasq drop-in remove (decommission)", "/usr/bin/rm", []string{"-f", "/etc/dnsmasq.d/felhom-x.conf"}, false, ""},

internal/hub/client.go

@@ -425,3 +425,27 @@ func (c *Client) FetchRetainedIdentityEscrow(ctx context.Context) (*RetainedEscr
 	}
 	return &out, nil
 }
+
+// PostOSReport sends the OS-update leg's report after every run (hub v0.130.0): POST /api/v1/hosts/{id}/os-report.
+// Per-host key, self-scoped on the hub. Errors are typed like RegisterWG's and never include the bearer.
+func (c *Client) PostOSReport(ctx context.Context, body []byte) error {
+	if c.hostID == "" {
+		return fmt.Errorf("hub: PostOSReport requires a configured host_id")
+	}
+	req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.baseURL+"/api/v1/hosts/"+c.hostID+"/os-report", bytes.NewReader(body))
+	if err != nil {
+		return fmt.Errorf("hub: building os-report request: %w", err)
+	}
+	req.Header.Set("Authorization", "Bearer "+c.apiKey)
+	req.Header.Set("Content-Type", "application/json")
+	resp, err := c.hc.Do(req)
+	if err != nil {
+		return &TransportError{Err: err}
+	}
+	defer resp.Body.Close()
+	raw, _ := io.ReadAll(io.LimitReader(resp.Body, 64<<10))
+	if resp.StatusCode < 200 || resp.StatusCode >= 300 {
+		return &HTTPError{StatusCode: resp.StatusCode, BodyTail: tail(raw, 256)}
+	}
+	return nil
+}

internal/hub/cloudflared.go

@@ -2,45 +2,111 @@ package hub
 
 import (
 	"context"
-	"os/exec"
+	"fmt"
 	"strings"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
 )
 
-// CloudflaredProber reports the cloudflared tunnel service health. It is a
+// Tunnel states the agent reports (R-841, agent v0.141.0). THREE, never two: a probe that could not ask is
-// READ-ONLY probe: the agent does NOT manage or restart cloudflared in this slice
+// `unknown`, which the hub never shows as up or down and never alarms on (R-96 rule 3).
-// (that is the tunnel-management slice — this is the seam for it). Injectable so
+const (
-// tests use a fake and never exec.
+	TunnelRunning    = "running"     // the cloudflared container runs AND its readiness check says CONNECTED
+	TunnelNotRunning = "not_running" // stopped / exited / absent, or running but NOT connected (Detail says which)
+	TunnelUnknown    = "unknown"     // the probe could not ask (guest down, pct/sudo error, health still starting)
+)
+
+// CloudflaredProber reports the box's tunnel. Injectable so tests use a fake and never exec.
 type CloudflaredProber interface {
-	// Status returns one of: "active" | "inactive" | "failed" | "unknown".
+	// Status returns one of the Tunnel* states and a short detail (why not_running / why unknown).
-	Status(ctx context.Context) (string, error)
+	Status(ctx context.Context) (status, detail string)
 }
 
-// SystemctlProber runs `systemctl is-active cloudflared`. This is NOT a Privileged
+// GuestTunnelProber reads the REAL tunnel: the `cloudflared` container in the box's own customer guest.
-// (root-CLI) op — `is-active` is non-root readable and is not one of the three
+//
-// proven root exceptions, so it does not go through internal/proxmox.Privileged.
+// Before v0.141.0 the agent ran `systemctl is-active cloudflared` on the HOST — a unit that does not exist (cloudflared
-type SystemctlProber struct {
+// is a guest container, `11-os-updates.md` C8), so every box reported `inactive` (R-841).
-	Unit string // defaults to "cloudflared"
+//
+// It uses ONLY the existing sudoers line `pct exec [0-9]* -- docker inspect -f *` (03 §3): the container's state, exit
+// code and Docker health status. The health status comes from the compose health check controller v0.292.0 adds
+// (`cloudflared tunnel --metrics localhost:20241 ready` → /ready: 200 only with ≥ 1 connection). Measured 2026-10-04:
+// with a wrong token the container stays "running" while /ready answers 503 — so the container state alone would lie.
+// A container with no health check (an older controller) is judged on its state alone, and Detail says so.
+type GuestTunnelProber struct {
+	Runner proxmox.Runner
+	// Guests returns the box's customer guest vmids (running pool guests that bind /mnt/felhom-drives).
+	Guests func(ctx context.Context) ([]int, error)
 }
 
-// Status maps `systemctl is-active` output to the report vocabulary. systemctl
+const tunnelInspect = `{{.State.Status}}|{{.State.ExitCode}}|{{if .State.Health}}{{.State.Health.Status}}{{else}}none{{end}}`
-// exits non-zero for inactive/failed, so the output string is authoritative over
+
-// the exit code; any exec error (binary missing, etc.) maps to "unknown".
+// Status probes every customer guest and reports the worst state (normally there is exactly one guest).
-func (p SystemctlProber) Status(ctx context.Context) (string, error) {
+func (p GuestTunnelProber) Status(ctx context.Context) (string, string) {
-	unit := p.Unit
+	if p.Runner == nil || p.Guests == nil {
-	if unit == "" {
+		return TunnelUnknown, "no probe wired"
-		unit = "cloudflared"
 	}
-	out, _ := exec.CommandContext(ctx, "systemctl", "is-active", unit).Output()
+	vmids, err := p.Guests(ctx)
-	switch strings.TrimSpace(string(out)) {
+	if err != nil {
-	case "active":
+		return TunnelUnknown, "could not list the customer guest: " + err.Error()
-		return "active", nil
-	case "failed":
-		return "failed", nil
-	case "inactive", "deactivating", "activating":
-		return "inactive", nil
-	case "":
-		return "unknown", nil // no output → systemctl/exec problem
-	default:
-		return "unknown", nil
 	}
+	if len(vmids) == 0 {
+		return TunnelUnknown, "no running customer guest"
+	}
+	worst, wdetail := "", ""
+	rank := map[string]int{TunnelRunning: 0, TunnelUnknown: 1, TunnelNotRunning: 2}
+	for _, v := range vmids {
+		out, errOut, err := p.Runner.Run(ctx, "/usr/sbin/pct", "exec", fmt.Sprint(v), "--", "docker", "inspect", "-f", tunnelInspect, "cloudflared")
+		st, d := ClassifyTunnel(string(out), string(errOut), err)
+		if len(vmids) > 1 {
+			d = fmt.Sprintf("guest %d: %s", v, d)
+		}
+		if worst == "" || rank[st] > rank[worst] {
+			worst, wdetail = st, d
+		}
+	}
+	return worst, wdetail
+}
+
+// ClassifyTunnel maps one `docker inspect` answer to a state. Pure; pinned by TestClassifyTunnel.
+func ClassifyTunnel(stdout, stderr string, err error) (string, string) {
+	out := strings.TrimSpace(stdout)
+	if err != nil || out == "" {
+		if strings.Contains(stderr, "No such object") || strings.Contains(stderr, "No such container") {
+			return TunnelNotRunning, "no cloudflared container in the guest"
+		}
+		return TunnelUnknown, "could not ask the guest: " + firstLine(stderr, err)
+	}
+	parts := strings.Split(out, "|")
+	if len(parts) != 3 {
+		return TunnelUnknown, "unreadable docker answer: " + out
+	}
+	state, code, health := parts[0], parts[1], parts[2]
+	if state != "running" {
+		return TunnelNotRunning, fmt.Sprintf("container %s, exit code %s", state, code)
+	}
+	switch health {
+	case "healthy":
+		return TunnelRunning, "connected"
+	case "unhealthy":
+		return TunnelNotRunning, "container running but the tunnel is NOT connected (cloudflared /ready fails)"
+	case "starting":
+		return TunnelUnknown, "container running, readiness check still starting"
+	case "none":
+		return TunnelRunning, "container running (no readiness check on this controller — connection not checked)"
+	}
+	return TunnelUnknown, "unknown health state " + health
+}
+
+func firstLine(stderr string, err error) string {
+	s := strings.TrimSpace(stderr)
+	if i := strings.IndexByte(s, '\n'); i >= 0 {
+		s = s[:i]
+	}
+	if s == "" && err != nil {
+		s = err.Error()
+	}
+	if len(s) > 160 {
+		s = s[:160]
+	}
+	return s
 }

internal/hub/cloudflared_test.go

@@ -0,0 +1,67 @@
+package hub
+
+import (
+	"context"
+	"errors"
+	"io"
+	"strings"
+	"testing"
+)
+
+// R-841: the three states from one `docker inspect` answer. Red-proof: map "unhealthy" to running (the container
+// state alone — what a plain "is it running" probe would say) and the "running but not connected" case fails.
+func TestClassifyTunnel(t *testing.T) {
+	cases := []struct {
+		name, out, errOut string
+		err               error
+		want              string
+		detail            string
+	}{
+		{"connected", "running|0|healthy\n", "", nil, TunnelRunning, "connected"},
+		{"running but not connected", "running|0|unhealthy\n", "", nil, TunnelNotRunning, "NOT connected"},
+		{"stopped", "exited|137|unhealthy\n", "", nil, TunnelNotRunning, "exit code 137"},
+		{"absent", "", "Error: No such object: cloudflared", errors.New("exit status 1"), TunnelNotRunning, "no cloudflared container"},
+		{"still starting", "running|0|starting\n", "", nil, TunnelUnknown, "starting"},
+		{"no health check (older controller)", "running|0|none\n", "", nil, TunnelRunning, "connection not checked"},
+		{"guest not running", "", "CT 9201 not running", errors.New("exit status 255"), TunnelUnknown, "could not ask"},
+		{"sudo refused", "", "sudo: a password is required", errors.New("exit status 1"), TunnelUnknown, "could not ask"},
+	}
+	for _, c := range cases {
+		st, d := ClassifyTunnel(c.out, c.errOut, c.err)
+		if st != c.want || !strings.Contains(d, c.detail) {
+			t.Errorf("%s: got %q (%s), want %q (…%s…)", c.name, st, d, c.want, c.detail)
+		}
+	}
+}
+
+type tunnelRunner struct {
+	calls []string
+	out   map[string]string
+}
+
+func (r *tunnelRunner) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
+	line := name + " " + strings.Join(args, " ")
+	r.calls = append(r.calls, line)
+	return []byte(r.out[args[1]]), nil, nil
+}
+func (r *tunnelRunner) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
+	return r.Run(ctx, name, args...)
+}
+
+// The probe uses EXACTLY the existing sudoers shape `pct exec <vmid> -- docker inspect -f <tmpl> cloudflared`, and
+// with no customer guest it is unknown, never down.
+func TestGuestTunnelProber(t *testing.T) {
+	r := &tunnelRunner{out: map[string]string{"9201": "running|0|healthy"}}
+	p := GuestTunnelProber{Runner: r, Guests: func(context.Context) ([]int, error) { return []int{9201}, nil }}
+	if st, d := p.Status(context.Background()); st != TunnelRunning || d != "connected" {
+		t.Fatalf("got %q %q", st, d)
+	}
+	want := "/usr/sbin/pct exec 9201 -- docker inspect -f " + tunnelInspect + " cloudflared"
+	if len(r.calls) != 1 || r.calls[0] != want {
+		t.Fatalf("command = %q, want %q", r.calls, want)
+	}
+	none := GuestTunnelProber{Runner: r, Guests: func(context.Context) ([]int, error) { return nil, nil }}
+	if st, _ := none.Status(context.Background()); st != TunnelUnknown {
+		t.Fatalf("no guest → %q, want unknown", st)
+	}
+}

internal/hub/collect.go

@@ -280,7 +280,7 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
 		PBSSnapshots:   c.collectPBSSnapshots(ctx),
 
 		AuditTail:       []AuditEntry{},
-		Cloudflared:     Cloudflared{Status: c.cloudflaredStatus(ctx)},
+		Cloudflared:     c.cloudflared(ctx),
 		Capabilities:    c.capabilities(ctx),
 		LeafFingerprint: c.leafFP,
 		Addresses:       c.collectAddresses(),
@@ -555,16 +555,18 @@ func (c *Collector) collectPBSSnapshots(ctx context.Context) []PBSSnapshot {
 	return []PBSSnapshot{}
 }
 
-func (c *Collector) cloudflaredStatus(ctx context.Context) string {
+func (c *Collector) cloudflared(ctx context.Context) Cloudflared {
 	if c.cf == nil {
-		return "unknown"
+		return Cloudflared{Status: TunnelUnknown, Detail: "no probe wired"}
 	}
-	st, err := c.cf.Status(ctx)
+	st, d := c.cf.Status(ctx)
-	if err != nil || st == "" {
+	if st == "" {
-		c.logger.Warn("hub: cloudflared probe failed", "err", err)
+		st = TunnelUnknown
-		return "unknown"
 	}
-	return st
+	if st == TunnelUnknown {
+		c.logger.Debug("hub: tunnel probe could not decide", "detail", d)
+	}
+	return Cloudflared{Status: st, Detail: d}
 }
 
 func percent(used, total int64) float64 {

internal/hub/collect_guestnet_test.go

@@ -16,7 +16,7 @@ func (f fakeGuestNet) GuestNetStatus(context.Context) *GuestNetStatus { return f
 
 func TestCollect_GuestNetOmittedWhenReporterNil(t *testing.T) {
 	px := &fakePx{node: "n", ns: newTestNodeStatus()}
-	c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.92.0", quietLogger())
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{}, nil, nil, nil, "h", "0.92.0", quietLogger())
 	r, err := c.Collect(context.Background())
 	if err != nil {
 		t.Fatalf("Collect: %v", err)
@@ -38,7 +38,7 @@ func TestCollect_GuestNetOmittedWhenReporterNil(t *testing.T) {
 
 func TestCollect_GuestNetPopulatedWhenWired(t *testing.T) {
 	px := &fakePx{node: "n", ns: newTestNodeStatus()}
-	c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.92.0", quietLogger())
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{}, nil, nil, nil, "h", "0.92.0", quietLogger())
 	c.SetGuestNetReporter(fakeGuestNet{st: &GuestNetStatus{
 		CheckedAt: "2026-07-21T10:00:00Z",
 		Guests: []GuestNetGuest{{

internal/hub/collect_mgmtplane_test.go

@@ -12,7 +12,7 @@ func (f fakeMgmtPlane) MgmtPlaneStatus(context.Context) *MgmtPlaneStatus { retur
 
 func TestCollect_MgmtPlaneOmittedWhenReporterNil(t *testing.T) {
 	px := &fakePx{node: "n", ns: newTestNodeStatus()}
-	c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.71.0", quietLogger())
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{}, nil, nil, nil, "h", "0.71.0", quietLogger())
 	r, err := c.Collect(context.Background())
 	if err != nil {
 		t.Fatalf("Collect: %v", err)
@@ -24,7 +24,7 @@ func TestCollect_MgmtPlaneOmittedWhenReporterNil(t *testing.T) {
 
 func TestCollect_MgmtPlanePopulatedWhenWired(t *testing.T) {
 	px := &fakePx{node: "n", ns: newTestNodeStatus()}
-	c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.71.0", quietLogger())
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{}, nil, nil, nil, "h", "0.71.0", quietLogger())
 	c.SetMgmtPlaneReporter(fakeMgmtPlane{st: &MgmtPlaneStatus{
 		PrivsepDirOK: true, SshdReachable: true, HealedRecently: true, PrivsepHealedAt: "2026-07-05T16:42:17Z",
 	}})
@@ -47,7 +47,7 @@ func (f fakeOOB) OOBStatus(context.Context) *OOBStatus { return f.st }
 
 func TestCollect_OOBOmittedWhenNil(t *testing.T) {
 	px := &fakePx{node: "n", ns: newTestNodeStatus()}
-	c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.72.0", quietLogger())
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{}, nil, nil, nil, "h", "0.72.0", quietLogger())
 	r, _ := c.Collect(context.Background())
 	if r.OOB != nil {
 		t.Fatalf("no reporter → oob omitted, got %+v", r.OOB)
@@ -56,7 +56,7 @@ func TestCollect_OOBOmittedWhenNil(t *testing.T) {
 
 func TestCollect_OOBPopulatedWhenWired(t *testing.T) {
 	px := &fakePx{node: "n", ns: newTestNodeStatus()}
-	c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{}, nil, nil, nil, "h", "0.72.0", quietLogger())
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{}, nil, nil, nil, "h", "0.72.0", quietLogger())
 	c.SetOOBReporter(fakeOOB{st: &OOBStatus{FelhomSshdActive: true, FelhomSshdPort: 8822, Reachable: true}})
 	r, _ := c.Collect(context.Background())
 	if r.OOB == nil || r.OOB.FelhomSshdPort != 8822 || !r.OOB.Reachable {

internal/hub/collect_test.go

@@ -33,7 +33,7 @@ func TestCollect_StorageTargetsFromObserver(t *testing.T) {
 	obs := fakeObserver{targets: []StorageTarget{
 		{Name: "local-lvm", Type: StorageTypeLVMThin, State: StorageStateAttached, Reachable: true},
 	}}
-	c := NewCollector(px, fakeProber{status: "active"}, obs, nil, nil, nil, "h", "0.5.0", quietLogger())
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, obs, nil, nil, nil, "h", "0.5.0", quietLogger())
 	r, err := c.Collect(context.Background())
 	if err != nil {
 		t.Fatalf("Collect: %v", err)
@@ -45,7 +45,7 @@ func TestCollect_StorageTargetsFromObserver(t *testing.T) {
 
 func TestCollect_StorageObserverErrorDegradesToEmpty(t *testing.T) {
 	px := &fakePx{node: "n", ns: newTestNodeStatus()}
-	c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{err: errors.New("proxmox down")}, nil, nil, nil, "h", "0.5.0", quietLogger())
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{err: errors.New("proxmox down")}, nil, nil, nil, "h", "0.5.0", quietLogger())
 	r, err := c.Collect(context.Background())
 	if err != nil {
 		t.Fatalf("a storage observe error must not sink the heartbeat: %v", err)
@@ -64,7 +64,7 @@ func TestCollect_HostAndGuests(t *testing.T) {
 		},
 		cfg: map[int]proxmox.GuestConfig{100: {Cores: 2, Memory: 2048}},
 	}
-	c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "demo-host-01", "0.3.0", quietLogger())
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "demo-host-01", "0.3.0", quietLogger())
 	r, err := c.Collect(context.Background())
 	if err != nil {
 		t.Fatalf("Collect: %v", err)
@@ -88,7 +88,7 @@ func TestCollect_HostAndGuests(t *testing.T) {
 	if g.Spec.Cores != 2 || g.Spec.MemoryBytes != 2147483648 || g.Spec.DiskBytes != 21474836480 {
 		t.Errorf("spec = %+v", g.Spec)
 	}
-	if r.Cloudflared.Status != "active" {
+	if r.Cloudflared.Status != "running" || r.Cloudflared.Detail != "connected" {
 		t.Errorf("cloudflared = %q", r.Cloudflared.Status)
 	}
 }
@@ -104,7 +104,7 @@ func TestCollect_GuestConfigFailureKeepsStatusOmitsSpec(t *testing.T) {
 		cfg:    map[int]proxmox.GuestConfig{100: {Cores: 2}},
 		cfgErr: map[int]error{200: errors.New("config read failed")},
 	}
-	c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.3.1", quietLogger())
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "h", "0.3.1", quietLogger())
 	r, err := c.Collect(context.Background())
 	if err != nil {
 		t.Fatalf("a per-guest failure must NOT fail the whole report: %v", err)
@@ -125,7 +125,7 @@ func TestCollect_GuestConfigFailureKeepsStatusOmitsSpec(t *testing.T) {
 
 func TestCollect_NodeStatusFailureIsHardError(t *testing.T) {
 	px := &fakePx{node: "n", nsErr: errors.New("proxmox down")}
-	c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.3.0", quietLogger())
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "h", "0.3.0", quietLogger())
 	if _, err := c.Collect(context.Background()); err == nil {
 		t.Fatal("NodeStatus failure must be a hard error (no useful report)")
 	}
@@ -133,7 +133,7 @@ func TestCollect_NodeStatusFailureIsHardError(t *testing.T) {
 
 func TestCollect_CloudflaredProbeErrorIsUnknown(t *testing.T) {
 	px := &fakePx{node: "n", ns: newTestNodeStatus()}
-	c := NewCollector(px, fakeProber{err: errors.New("no systemctl")}, nil, nil, nil, nil, "h", "0.3.0", quietLogger())
+	c := NewCollector(px, fakeProber{status: "", detail: "could not ask"}, nil, nil, nil, nil, "h", "0.3.0", quietLogger())
 	r, err := c.Collect(context.Background())
 	if err != nil {
 		t.Fatalf("cloudflared failure must not be fatal: %v", err)
@@ -153,7 +153,7 @@ func TestCollect_LeafFingerprint(t *testing.T) {
 	px := &fakePx{node: "n", ns: newTestNodeStatus()}
 	const fp = "60b5974d586f5f3c8ec41eb998d0f07406178219c36bf6d3ff377570279d8245"
 
-	c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.48.0", quietLogger())
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "h", "0.48.0", quietLogger())
 	c.SetLeafFingerprint(fp)
 	r, err := c.Collect(context.Background())
 	if err != nil {
@@ -164,7 +164,7 @@ func TestCollect_LeafFingerprint(t *testing.T) {
 	}
 
 	// Companion: no SetLeafFingerprint (local API disabled) → empty, never a fabricated value.
-	c2 := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.48.0", quietLogger())
+	c2 := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "h", "0.48.0", quietLogger())
 	r2, _ := c2.Collect(context.Background())
 	if r2.LeafFingerprint != "" {
 		t.Fatalf("unset leaf_fingerprint = %q, want empty", r2.LeafFingerprint)

internal/hub/dr_recipe_test.go

@@ -384,7 +384,7 @@ func TestCollectDRRecipe_ProductionPath(t *testing.T) {
 	obs := fakeObserver{targets: capturedDemoFelhomTargets()}
 	pbsRep := fakePBSReporter{snaps: capturedDemoFelhomSnapshots()}
 
-	c := NewCollector(px, fakeProber{status: "active"}, obs, nil, nil, pbsRep, "h", "0.118.0", quietLogger())
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, obs, nil, nil, pbsRep, "h", "0.118.0", quietLogger())
 	c.SetBackupTargetResolver(func() ConfiguredBackupTarget {
 		return ConfiguredBackupTarget{StorageID: "felhom-backup", Known: true}
 	})
@@ -409,7 +409,7 @@ func TestCollectDRRecipe_ProductionPath(t *testing.T) {
 // test that would have caught shipping the seam without wiring it.
 func TestCollectDRRecipe_UnwiredSeamReportsUnknown(t *testing.T) {
 	px := &fakePx{node: "n", ns: newTestNodeStatus()}
-	c := NewCollector(px, fakeProber{status: "active"}, fakeObserver{targets: capturedDemoFelhomTargets()},
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, fakeObserver{targets: capturedDemoFelhomTargets()},
 		nil, nil, nil, "h", "0.118.0", quietLogger())
 
 	r, err := c.Collect(context.Background())

internal/hub/hostmetrics_test.go

@@ -16,7 +16,7 @@ func intp(v int) *int { return &v }
 // HostMetricsNow returns a fresh host block with cpu% from NodeStatus and the temp from the reader.
 func TestHostMetricsNow_PopulatesTemp(t *testing.T) {
 	px := &fakePx{node: "demo-felhom", ns: newTestNodeStatus()}
-	c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.14.0", quietLogger()).
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "h", "0.14.0", quietLogger()).
 		SetTempReader(fakeTemp{c: intp(46)})
 	h, err := c.HostMetricsNow(context.Background())
 	if err != nil {
@@ -36,7 +36,7 @@ func TestHostMetricsNow_PopulatesTemp(t *testing.T) {
 // A missing temp sensor gracefully nulls cpu_temp_c without failing the host read.
 func TestHostMetricsNow_GracefulNullTemp(t *testing.T) {
 	px := &fakePx{node: "n", ns: newTestNodeStatus()}
-	c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.14.0", quietLogger()).
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "h", "0.14.0", quietLogger()).
 		SetTempReader(fakeTemp{c: nil})
 	h, err := c.HostMetricsNow(context.Background())
 	if err != nil {
@@ -50,7 +50,7 @@ func TestHostMetricsNow_GracefulNullTemp(t *testing.T) {
 // A NodeStatus failure is a hard error (no useful host view).
 func TestHostMetricsNow_NodeStatusErrorIsHard(t *testing.T) {
 	px := &fakePx{node: "n", nsErr: errors.New("proxmox down")}
-	c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.14.0", quietLogger())
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "h", "0.14.0", quietLogger())
 	if _, err := c.HostMetricsNow(context.Background()); err == nil {
 		t.Fatal("NodeStatus failure must be a hard error")
 	}
@@ -59,7 +59,7 @@ func TestHostMetricsNow_NodeStatusErrorIsHard(t *testing.T) {
 // Collect() (the hub report) also carries the temp now — the operator freebie.
 func TestCollect_HostReportCarriesTemp(t *testing.T) {
 	px := &fakePx{node: "n", ns: newTestNodeStatus()}
-	c := NewCollector(px, fakeProber{status: "active"}, nil, nil, nil, nil, "h", "0.14.0", quietLogger()).
+	c := NewCollector(px, fakeProber{status: "running", detail: "connected"}, nil, nil, nil, nil, "h", "0.14.0", quietLogger()).
 		SetTempReader(fakeTemp{c: intp(51)})
 	r, err := c.Collect(context.Background())
 	if err != nil {

internal/hub/mock_test.go

@@ -56,7 +56,7 @@ func (f *fakePx) GuestConfig(ctx context.Context, vmid int) (proxmox.GuestConfig
 // fakeProber is a fake CloudflaredProber.
 type fakeProber struct {
 	status string
-	err    error
+	detail string
 }
 
-func (p fakeProber) Status(ctx context.Context) (string, error) { return p.status, p.err }
+func (p fakeProber) Status(ctx context.Context) (string, string) { return p.status, p.detail }

internal/hub/osupdate_contract_test.go

@@ -0,0 +1,36 @@
+package hub
+
+import (
+	"encoding/json"
+	"os"
+	"testing"
+)
+
+// The os_update block is a cross-repo contract: testdata/desired-state-osupdate.golden.json is byte-identical with
+// felhom.eu/hub/internal/api/testdata (the hub's TestOSUpdate_DesiredBlockMatchesTheGolden proves the hub SERVES
+// it). Here: the agent DECODES every field. A renamed json tag on either side fails one of the two tests.
+func TestOSUpdateGolden_Decodes(t *testing.T) {
+	raw, err := os.ReadFile("testdata/desired-state-osupdate.golden.json")
+	if err != nil {
+		t.Fatal(err)
+	}
+	var resp DesiredStateResponse
+	if err := json.Unmarshal(raw, &resp); err != nil {
+		t.Fatal(err)
+	}
+	o := resp.DesiredState.OSUpdate
+	if o == nil || o.Ring != 1 || !o.Enabled || o.Release == nil {
+		t.Fatalf("os_update = %+v", o)
+	}
+	r := o.Release
+	if r.ID != "os-guest-20261004-120000" || r.Snapshot != "20261004T120000Z" || len(r.Packages) != 2 ||
+		r.Packages[1].Name != "openssl" || r.Packages[1].Version != "3.5.7-1~deb13u3" || r.Packages[1].Origin != "Debian-Security" {
+		t.Fatalf("release = %+v", r)
+	}
+	// v0.141.0: the host layer's own approved set (`11` §8 step 3).
+	h := o.HostRelease
+	if h == nil || h.ID != "os-host-20261004-120000" || h.Snapshot != "20261004T120000Z" || len(h.Packages) != 1 ||
+		h.Packages[0].Name != "libssl3t64" || h.Packages[0].Origin != "Debian-Security" {
+		t.Fatalf("host_release = %+v", h)
+	}
+}

internal/hub/report.go

@@ -289,9 +289,10 @@ type GuestSpec struct {
 	DiskBytes   int64 `json:"disk_bytes"`
 }
 
-// Cloudflared is the tunnel service health (read-only probe this slice).
+// Cloudflared is the box's tunnel (R-841, agent v0.141.0): the cloudflared container in the customer guest.
 type Cloudflared struct {
-	Status string `json:"status"` // active | inactive | failed | unknown
+	Status string `json:"status"`           // running | not_running | unknown (TunnelRunning …)
+	Detail string `json:"detail,omitempty"` // why not_running / unknown, or "connected"
 }
 
 // The following element types are declared now so the empty collections above are
@@ -548,6 +549,35 @@ type WireDesiredState struct {
 	RestoreDirective *WireRestoreDirective `json:"restore_directive,omitempty"` // slice 10D (forward-compat)
 	Wireguard        *WireWireguard        `json:"wireguard,omitempty"`         // S3 (doc 06 §3.2; golden-pinned)
 	PBSDR            *WirePBSDR            `json:"pbs_dr,omitempty"`            // PBS DR tier (slice 2 consumer)
+	OSUpdate         *WireOSUpdate         `json:"os_update,omitempty"`         // OS updates, guest fast lane (agent v0.140.0)
+}
+
+// WireOSUpdate is the hub-OWNED OS-update block (hub v0.130.0, `11-os-updates.md` §5.3), merged into the served
+// document at read time. Ring 0 installs every pending Debian / Debian-Security fix; ring 1 installs exactly the
+// newest approved release. Absent (older hub) → the agent treats the box as ring 1, ON, no release: it reports
+// and installs nothing. Golden: testdata/desired-state-osupdate.golden.json (byte-identical with the hub's).
+type WireOSUpdate struct {
+	Ring    int            `json:"ring"`
+	Enabled bool           `json:"enabled"`
+	Release *WireOSRelease `json:"release,omitempty"`
+	// HostRelease is the newest approved HOST release (hub v0.131.0, `11` §8 step 3) — a separate set: a version
+	// approved for the guest is not approved for the host by that fact alone.
+	HostRelease *WireOSRelease `json:"host_release,omitempty"`
+}
+
+// WireOSRelease is an approved version set; Snapshot is the approval time (YYYYMMDDTHHMMSSZ) the wrapper uses
+// for snapshot.debian.org when Debian has already replaced a version (decision 79).
+type WireOSRelease struct {
+	ID       string          `json:"id"`
+	Snapshot string          `json:"snapshot"`
+	Packages []WireOSPackage `json:"packages"`
+}
+
+// WireOSPackage is one approved name=version and its origin ("Debian" | "Debian-Security").
+type WireOSPackage struct {
+	Name    string `json:"name"`
+	Version string `json:"version"`
+	Origin  string `json:"origin"`
 }
 
 // WirePBSDR is the hub's PBS-DR-tier descriptor (PBS DR slice 1, hub/internal/web/pbsdr.go

internal/hub/testdata/desired-state-osupdate.golden.json

@@ -0,0 +1,24 @@
+{
+  "generation": 1,
+  "desired_state": {
+    "os_update": {
+      "ring": 1,
+      "enabled": true,
+      "release": {
+        "id": "os-guest-20261004-120000",
+        "snapshot": "20261004T120000Z",
+        "packages": [
+          {"name": "libc6", "version": "2.41-12+deb13u4", "origin": "Debian"},
+          {"name": "openssl", "version": "3.5.7-1~deb13u3", "origin": "Debian-Security"}
+        ]
+      },
+      "host_release": {
+        "id": "os-host-20261004-120000",
+        "snapshot": "20261004T120000Z",
+        "packages": [
+          {"name": "libssl3t64", "version": "3.5.7-1~deb13u3", "origin": "Debian-Security"}
+        ]
+      }
+    }
+  }
+}

internal/localapi/afterbackup_test.go

@@ -0,0 +1,63 @@
+package localapi
+
+import (
+	"context"
+	"net/http"
+	"sync"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
+)
+
+// The OS leg (agent v0.140.0) runs after a SUCCESSFUL primary backup, and only then; and it runs BEFORE the
+// host-wide heavy-op gate is released, so a restore-test cannot start in the middle of it (`11` C10).
+// Red-proof: drop the `b.Success &&` guard and the failed-backup sub-case fails; move the call after release()
+// and the gate sub-case fails.
+func TestAfterPrimaryBackup(t *testing.T) {
+	run := func(t *testing.T, failErr string) (calls []int, gateHeld bool) {
+		gate := &backup.InFlight{}
+		b := &fakeBackups{failErr: failErr}
+		srv := newTestServerS(t, &fakeGuests{}, b, &fakeStore{}, nil)
+		srv.inFlight = gate
+		var mu sync.Mutex
+		done := make(chan struct{}, 1)
+		srv.SetAfterPrimaryBackup(func(_ context.Context, vmid int) {
+			rel, _, ok := gate.TryAcquire("probe")
+			mu.Lock()
+			calls = append(calls, vmid)
+			gateHeld = !ok
+			mu.Unlock()
+			if ok {
+				rel()
+			}
+			done <- struct{}{}
+		})
+		h := srv.Handler()
+		if do(t, h, "POST", "/backup", "A", "").Code != http.StatusAccepted {
+			t.Fatal("POST /backup not accepted")
+		}
+		select {
+		case <-done:
+		case <-time.After(500 * time.Millisecond):
+		}
+		time.Sleep(20 * time.Millisecond)
+		mu.Lock()
+		defer mu.Unlock()
+		return calls, gateHeld
+	}
+	t.Run("success runs the leg under the gate", func(t *testing.T) {
+		calls, held := run(t, "")
+		if len(calls) != 1 {
+			t.Fatalf("the leg ran %d time(s), want 1", len(calls))
+		}
+		if !held {
+			t.Fatal("the heavy-op gate was free while the leg ran — a restore-test could overlap it")
+		}
+	})
+	t.Run("a failed backup runs nothing", func(t *testing.T) {
+		if calls, _ := run(t, "vzdump exploded"); len(calls) != 0 {
+			t.Fatalf("the leg ran after a FAILED backup: %v", calls)
+		}
+	})
+}

internal/localapi/server.go

@@ -160,6 +160,10 @@ type Options struct {
 	// NetStorage is the privileged network-mount (NAS) surface (Part A1). OPTIONAL — when nil, the
 	// /netstorage endpoints report "not configured". Satisfied by *storage.SudoHostOps.
 	NetStorage NetworkStorageOps
+	// AfterPrimaryBackup (agent v0.140.0, `11-os-updates.md` §8 step 2) runs right after a SUCCESSFUL backup on the
+	// PRIMARY tier, inside the backup goroutine and BEFORE the host-wide heavy-op gate is released — so the OS leg
+	// that it starts can never overlap another backup or a restore-test (`11` C10). OPTIONAL — nil → nothing runs.
+	AfterPrimaryBackup func(ctx context.Context, vmid int)
 	// Privileged runs the fenced root wrappers (E-2a: felhom-backup-target-apply). OPTIONAL — when
 	// nil, POST /backup/target reports "not configured". Satisfied by *proxmox.ExecRunner.
 	Privileged PrivilegedRunner
@@ -276,6 +280,7 @@ type Server struct {
 	tiers []BackupTier
 	// inFlight (R-85) is shared with the restore-test scheduler so the two never run together.
 	inFlight *backup.InFlight
+	afterPrimaryBackup func(ctx context.Context, vmid int) // the OS leg (agent v0.140.0); nil = none
 	logger   *slog.Logger
 	now      func() time.Time
 
@@ -469,6 +474,7 @@ func NewServer(o Options) (*Server, error) {
 	// the primary is always first, because that is what the untargeted endpoints act on.
 	s.tiers = normalizeBackupTiers(o.BackupTiers, o.Backups, cadence)
 	s.inFlight = o.InFlight
+	s.afterPrimaryBackup = o.AfterPrimaryBackup
 	if s.backups == nil && len(s.tiers) > 0 {
 		s.backups = s.tiers[0].Service
 	}
@@ -894,6 +900,10 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
 		}
 		s.store.RecordBackup(b)
 		s.finishJob(key, jobID, b)
+		// OS leg (agent v0.140.0): after the night's whole-guest copy exists, still holding the heavy-op gate.
+		if b.Success && tier.Primary && s.afterPrimaryBackup != nil {
+			s.afterPrimaryBackup(base, vmid)
+		}
 	}()
 	writeStatus(w, http.StatusAccepted, true, BackupResponse{VMID: vmid, JobID: jobID, Phase: PhaseRunning}, "")
 }
@@ -1465,3 +1475,6 @@ func writeStatus(w http.ResponseWriter, code int, ok bool, data any, errMsg stri
 	w.WriteHeader(code)
 	_ = json.NewEncoder(w).Encode(apiResponse{OK: ok, Data: data, Error: errMsg})
 }
+
+// SetAfterPrimaryBackup wires the hook that runs after a successful primary-tier backup (the OS leg, agent v0.140.0).
+func (s *Server) SetAfterPrimaryBackup(fn func(ctx context.Context, vmid int)) { s.afterPrimaryBackup = fn }

internal/osupdate/leg.go

@@ -0,0 +1,496 @@
+// Package osupdate is the agent's OS-update leg (`11-os-updates.md` §8 steps 2–3): the customer GUEST's Debian fast
+// lane (agent v0.140.0) and, after it in the same pass, the HOST's (agent v0.141.0).
+//
+// It runs right after the night's successful whole-guest backup, while the backup goroutine still holds the host-wide
+// heavy-op gate (so it never overlaps a backup or a restore-test, `11` C10), at most once per night. All root work is
+// the wrapper `felhom-os-apply` (configs/, its own tests); this package only builds plans, calls the wrapper through
+// sudo, judges health and reports to the hub — one report per layer.
+//
+// NO AUTOMATIC UNDO (R-837 measured; `09` §3 decision 81): a failed health check stops, reports `health_failed` and the
+// hub mails the operator; the whole-guest backup taken minutes earlier is the guest's undo, by hand; a host package is
+// put back by hand from the previous release's snapshot (runbook). The host is NEVER rebooted by this package.
+package osupdate
+
+import (
+	"context"
+	"encoding/json"
+	"fmt"
+	"log/slog"
+	"os"
+	"path/filepath"
+	"sort"
+	"strings"
+	"sync"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+	"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
+)
+
+// WrapperPath is the pinned sudoers vector (configs/felhom-agent.sudoers FELHOM_OSAPPLY).
+const WrapperPath = "/usr/local/sbin/felhom-os-apply"
+
+// DefaultPlanDir is where plans are written (the sudoers glob names it).
+const DefaultPlanDir = "/var/lib/felhom-agent/os"
+
+// Layers.
+const (
+	LayerGuest = "guest"
+	LayerHost  = "host"
+)
+
+// Package is one name=version with its origin.
+type Package struct {
+	Name    string `json:"name"`
+	Version string `json:"version"`
+	Origin  string `json:"origin"`
+}
+
+// Pending is one update the sources offer (origin as apt names it, possibly several).
+type Pending struct {
+	Name   string   `json:"name"`
+	From   string   `json:"from"`
+	To     string   `json:"to"`
+	Origin []string `json:"origin"`
+}
+
+// Container is one container's state as the wrapper saw it.
+type Container struct {
+	State  string `json:"state"`
+	Health string `json:"health"` // healthy | unhealthy | starting | none
+}
+
+// Health is one health reading. Guest layer: DockerOK..Containers. Host layer: HostServices, GuestRunning and the
+// guest's own reading in Guest.
+type Health struct {
+	DockerOK     bool                 `json:"docker_ok"`
+	NetworkOK    bool                 `json:"network_ok"`
+	Controller   string               `json:"controller"`
+	Containers   map[string]Container `json:"containers"`
+	HostServices map[string]string    `json:"host_services,omitempty"`
+	GuestRunning *bool                `json:"guest_running,omitempty"`
+	Guest        *Health              `json:"guest,omitempty"`
+}
+
+// WrapperReport is the wrapper's OSAPPLY-REPORT object.
+type WrapperReport struct {
+	Mode                string          `json:"mode"`
+	Layer               string          `json:"layer"`
+	Refused             json.RawMessage `json:"refused"`
+	Failed              json.RawMessage `json:"failed"`
+	Upgraded            []Package       `json:"upgraded"`
+	Installed           []Package       `json:"installed"`
+	Pending             []Pending       `json:"pending"`
+	RestartNeeded       []string        `json:"restart_needed"`
+	DockerRestartNeeded bool            `json:"docker_restart_needed"`
+	RebootNeeded        bool            `json:"reboot_needed"`
+	RebootScanned       bool            `json:"reboot_scanned"`
+	HealthBefore        *Health         `json:"health_before"`
+	HealthAfter         *Health         `json:"health_after"`
+	Health              *Health         `json:"health"`
+	PassSeconds         float64         `json:"pass_seconds"`
+}
+
+func (w WrapperReport) refused() bool { return len(w.Refused) > 0 && string(w.Refused) != "null" }
+func (w WrapperReport) failed() bool  { return len(w.Failed) > 0 && string(w.Failed) != "null" }
+
+// Report is what the hub receives per layer (hub osupdates.Report — field-exact).
+type Report struct {
+	RunID               string          `json:"run_id"`
+	Layer               string          `json:"layer"`
+	Trigger             string          `json:"trigger"`
+	Mode                string          `json:"mode"`
+	Ring                int             `json:"ring"`
+	ReleaseID           string          `json:"release_id"`
+	Outcome             string          `json:"outcome"`
+	Healthy             bool            `json:"healthy"`
+	HealthReason        string          `json:"health_reason,omitempty"`
+	VMID                int             `json:"vmid"`
+	Upgraded            []Package       `json:"upgraded,omitempty"`
+	Installed           []Package       `json:"installed,omitempty"`
+	Pending             []Pending       `json:"pending,omitempty"`
+	NotCovered          []string        `json:"not_covered,omitempty"`
+	RestartNeeded       []string        `json:"restart_needed,omitempty"`
+	DockerRestartNeeded bool            `json:"docker_restart_needed,omitempty"`
+	RebootNeeded        bool            `json:"reboot_needed,omitempty"`
+	RebootScanned       bool            `json:"reboot_scanned,omitempty"` // the pass looked (host: every pass) — a false RebootNeeded then means "not needed"
+	Refused             json.RawMessage `json:"refused,omitempty"`
+	PassSeconds         float64         `json:"pass_seconds,omitempty"`
+}
+
+// Reporter posts a report to the hub (*hub.Client).
+type Reporter interface {
+	PostOSReport(ctx context.Context, body []byte) error
+}
+
+// Leg runs one OS-update pass for the customer guest and then the host.
+type Leg struct {
+	Runner     proxmox.Runner
+	Hub        Reporter
+	Tunnel     hub.CloudflaredProber // the host health rule needs the tunnel `running` (R-841)
+	Appliance  bool                  // agent.json deployment_mode; the wrapper re-checks the ROOT-owned record (R12)
+	Logger     *slog.Logger
+	PlanDir    string
+	StatePath  string        // last night run (once per night)
+	HealthWait time.Duration // how long health may take to come back (default 5 min)
+	HealthPoll time.Duration // default 15 s
+	MinGap     time.Duration // between night runs (default 20 h)
+	Now        func() time.Time
+	Sleep      func(context.Context, time.Duration)
+
+	mu    sync.Mutex
+	block *hub.WireOSUpdate
+}
+
+// OnDesiredState stores the hub's os_update block (desired.RawConsumer — store only, never block).
+func (l *Leg) OnDesiredState(_ context.Context, resp *hub.DesiredStateResponse) {
+	if resp == nil {
+		return
+	}
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	l.block = resp.DesiredState.OSUpdate
+}
+
+// Block returns the newest os_update block. No block (an older hub, or nothing fetched yet) = ring 1, ON, no
+// release: the box reports and installs nothing.
+func (l *Leg) Block() hub.WireOSUpdate {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	if l.block == nil {
+		return hub.WireOSUpdate{Ring: 1, Enabled: true}
+	}
+	return *l.block
+}
+
+// SetBlock sets the block directly (the selftest fetches the desired state itself).
+func (l *Leg) SetBlock(b *hub.WireOSUpdate) {
+	l.mu.Lock()
+	defer l.mu.Unlock()
+	l.block = b
+}
+
+func (l *Leg) now() time.Time {
+	if l.Now != nil {
+		return l.Now()
+	}
+	return time.Now()
+}
+
+func (l *Leg) log() *slog.Logger {
+	if l.Logger != nil {
+		return l.Logger
+	}
+	return slog.Default()
+}
+
+func (l *Leg) sleep(ctx context.Context, d time.Duration) {
+	if l.Sleep != nil {
+		l.Sleep(ctx, d)
+		return
+	}
+	select {
+	case <-ctx.Done():
+	case <-time.After(d):
+	}
+}
+
+// IsFast reports whether every origin apt names is Debian / Debian-Security (the fast lane, `11` C3).
+func IsFast(origins []string) bool {
+	if len(origins) == 0 {
+		return false
+	}
+	for _, o := range origins {
+		if o != "Debian" && o != "Debian-Security" {
+			return false
+		}
+	}
+	return true
+}
+
+// HealthVerdict is THE guest health rule (`11` §8.1; pinned by TestHealthVerdict*): docker answers, the guest's
+// network resolves, the controller's own health check is `healthy`, and every container that was running at the
+// start of the pass runs again — and healthy again if it was. "starting" is not yet healthy.
+func HealthVerdict(before, after *Health) (bool, string) {
+	if after == nil {
+		return false, "no health reading"
+	}
+	if !after.DockerOK {
+		return false, "docker does not answer"
+	}
+	if !after.NetworkOK {
+		return false, "the guest cannot resolve deb.debian.org"
+	}
+	if after.Controller != "healthy" {
+		return false, "the controller is " + after.Controller
+	}
+	if before == nil {
+		return true, ""
+	}
+	names := make([]string, 0, len(before.Containers))
+	for n := range before.Containers {
+		names = append(names, n)
+	}
+	sort.Strings(names)
+	for _, n := range names {
+		b := before.Containers[n]
+		if b.State != "running" {
+			continue
+		}
+		a, ok := after.Containers[n]
+		if !ok || a.State != "running" {
+			return false, n + " was running and is not"
+		}
+		if b.Health == "healthy" && a.Health != "healthy" {
+			return false, n + " was healthy and is " + a.Health
+		}
+	}
+	return true, ""
+}
+
+// HostHealthVerdict is THE host health rule (`11` §8.2; pinned by TestHostHealthVerdict): the Proxmox daemons and the
+// agent are active, the customer guest still runs, the guest's own rule still passes against the start of the pass,
+// and the tunnel is `running` (R-841).
+func HostHealthVerdict(before, after *Health, tunnel string) (bool, string) {
+	if after == nil {
+		return false, "no health reading"
+	}
+	svcs := make([]string, 0, len(after.HostServices))
+	for s := range after.HostServices {
+		svcs = append(svcs, s)
+	}
+	sort.Strings(svcs)
+	if len(svcs) == 0 {
+		return false, "no host service reading"
+	}
+	for _, s := range svcs {
+		if after.HostServices[s] != "active" {
+			return false, s + " is " + after.HostServices[s]
+		}
+	}
+	if after.GuestRunning == nil || !*after.GuestRunning {
+		return false, "the customer guest is not running"
+	}
+	var gb *Health
+	if before != nil {
+		gb = before.Guest
+	}
+	if ok, why := HealthVerdict(gb, after.Guest); !ok {
+		return false, "guest: " + why
+	}
+	if tunnel != hub.TunnelRunning {
+		return false, "the tunnel is " + tunnel
+	}
+	return true, ""
+}
+
+// call writes the plan and runs the wrapper once.
+func (l *Leg) call(ctx context.Context, runID string, plan map[string]any) (WrapperReport, error) {
+	dir := l.PlanDir
+	if dir == "" {
+		dir = DefaultPlanDir
+	}
+	if err := os.MkdirAll(dir, 0o700); err != nil {
+		return WrapperReport{}, fmt.Errorf("osupdate: plan dir: %w", err)
+	}
+	b, _ := json.Marshal(plan)
+	path := filepath.Join(dir, fmt.Sprintf("plan-%s-%s-%s.json", runID, plan["layer"], plan["mode"]))
+	if err := os.WriteFile(path, b, 0o600); err != nil {
+		return WrapperReport{}, fmt.Errorf("osupdate: write plan: %w", err)
+	}
+	defer os.Remove(path)
+	stdout, stderr, err := l.Runner.Run(ctx, WrapperPath, "--plan", path)
+	for _, line := range strings.Split(strings.TrimSpace(string(stderr)), "\n") {
+		if strings.HasPrefix(line, "os-apply: ") {
+			l.log().Info("osupdate: wrapper", "line", line)
+		}
+	}
+	var rep WrapperReport
+	found := false
+	for _, line := range strings.Split(string(stdout), "\n") {
+		if strings.HasPrefix(line, "OSAPPLY-REPORT ") {
+			if jerr := json.Unmarshal([]byte(strings.TrimPrefix(line, "OSAPPLY-REPORT ")), &rep); jerr == nil {
+				found = true
+			}
+		}
+	}
+	if !found {
+		return rep, fmt.Errorf("osupdate: wrapper gave no report (err %v): %s", err, strings.TrimSpace(string(stderr)))
+	}
+	return rep, nil // a refusal / failure is IN the report (exit 2 / 3), not an error here
+}
+
+// Run is one pass: the guest layer, then (on an appliance, after a good guest step) the host layer. Returns both
+// reports (host empty when skipped). trigger is "night" or "debug".
+func (l *Leg) Run(ctx context.Context, vmid int, trigger string) (guest Report, host Report) {
+	runID := l.now().UTC().Format("20060102T150405Z")
+	lg := l.log().With("run", runID, "vmid", vmid, "trigger", trigger)
+	if trigger == "night" && l.StatePath != "" {
+		gap := l.MinGap
+		if gap == 0 {
+			gap = 20 * time.Hour
+		}
+		if b, err := os.ReadFile(l.StatePath); err == nil {
+			if last, perr := time.Parse(time.RFC3339, strings.TrimSpace(string(b))); perr == nil && l.now().Sub(last) < gap {
+				lg.Info("osupdate: skipped — already ran tonight", "last", last.UTC().Format(time.RFC3339))
+				return Report{RunID: runID, Layer: LayerGuest, Outcome: "skipped"}, Report{}
+			}
+		}
+	}
+	blk := l.Block()
+	guest = l.runLayer(ctx, runID, LayerGuest, vmid, trigger, blk)
+	if trigger == "night" && l.StatePath != "" {
+		_ = os.WriteFile(l.StatePath, []byte(l.now().UTC().Format(time.RFC3339)), 0o600)
+	}
+	switch {
+	case !l.Appliance:
+		lg.Info("osupdate: host step skipped — not an appliance install (a BYO host belongs to its owner, `11` §1)")
+	case !(guest.Outcome == "applied" || guest.Outcome == "nothing" || guest.Outcome == "inventory") || !guest.Healthy:
+		lg.Warn("osupdate: host step skipped — the guest step did not end healthy", "guest_outcome", guest.Outcome, "reason", guest.HealthReason)
+	default:
+		host = l.runLayer(ctx, runID, LayerHost, vmid, trigger, blk)
+	}
+	return guest, host
+}
+
+func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigger string, blk hub.WireOSUpdate) Report {
+	rel := hub.WireOSRelease{ID: "ring0-" + runID}
+	var wire *hub.WireOSRelease
+	if layer == LayerGuest {
+		wire = blk.Release
+	} else {
+		wire = blk.HostRelease
+	}
+	if blk.Ring == 1 {
+		rel = hub.WireOSRelease{}
+		if wire != nil {
+			rel = *wire
+		}
+	}
+	rep := Report{RunID: runID, Layer: layer, Trigger: trigger, Ring: blk.Ring, VMID: vmid, ReleaseID: rel.ID}
+	lg := l.log().With("run", runID, "layer", layer, "vmid", vmid, "ring", blk.Ring, "trigger", trigger)
+	lg.Info("osupdate: START", "enabled", blk.Enabled, "release", rel.ID)
+
+	plan := map[string]any{"release_id": rel.ID, "layer": layer, "lane": "fast", "vmid": vmid, "snapshot": rel.Snapshot,
+		"packages": []Package{}, "mode": "apply", "select": "listed"}
+	if rel.ID == "" {
+		plan["release_id"] = "none"
+	}
+	planned := map[string]bool{}
+	switch {
+	case !blk.Enabled:
+		plan["mode"] = "inventory"
+		lg.Info("osupdate: switched OFF for this box — reporting only")
+	case blk.Ring == 0:
+		plan["select"] = "pending-fast" // the wrapper picks every pending Debian / Debian-Security upgrade
+	case len(rel.Packages) == 0:
+		plan["mode"] = "inventory" // ring 1 with no approved release for this layer: nothing to install
+	default:
+		var pk []Package
+		for _, p := range rel.Packages {
+			pk = append(pk, Package{Name: p.Name, Version: p.Version, Origin: p.Origin})
+			planned[p.Name] = true
+		}
+		plan["packages"] = pk
+	}
+	rep.Mode = plan["mode"].(string)
+	wr, err := l.call(ctx, runID, plan)
+	switch {
+	case err != nil:
+		rep.Outcome, rep.HealthReason = "failed", err.Error()
+		return l.finish(ctx, lg, rep)
+	case wr.refused():
+		rep.Outcome, rep.Refused = "refused", wr.Refused
+		return l.finish(ctx, lg, rep)
+	case wr.failed():
+		rep.Outcome, rep.Refused = "failed", wr.Failed
+	}
+	rep.Upgraded, rep.PassSeconds = wr.Upgraded, wr.PassSeconds
+	if rep.Outcome == "" {
+		switch {
+		case rep.Mode == "inventory" && !blk.Enabled:
+			rep.Outcome = "inventory"
+		case len(wr.Upgraded) == 0:
+			rep.Outcome = "nothing"
+		default:
+			rep.Outcome = "applied"
+		}
+	}
+	if blk.Ring == 0 {
+		for _, u := range wr.Upgraded {
+			planned[u.Name] = true
+		}
+	}
+	// Health: compare with the start of the pass; give restarted services time (only after an install).
+	cur := wr.HealthAfter
+	verdict := func(h *Health) (bool, string) {
+		if layer == LayerHost {
+			t := hub.TunnelUnknown
+			if l.Tunnel != nil {
+				t, _ = l.Tunnel.Status(ctx)
+			}
+			return HostHealthVerdict(wr.HealthBefore, h, t)
+		}
+		return HealthVerdict(wr.HealthBefore, h)
+	}
+	if len(wr.Upgraded) > 0 {
+		wait, poll := l.HealthWait, l.HealthPoll
+		if wait == 0 {
+			wait = 5 * time.Minute
+		}
+		if poll == 0 {
+			poll = 15 * time.Second
+		}
+		deadline := l.now().Add(wait)
+		for {
+			ok, why := verdict(cur)
+			rep.Healthy, rep.HealthReason = ok, why
+			if ok || !l.now().Before(deadline) || ctx.Err() != nil {
+				break
+			}
+			l.sleep(ctx, poll)
+			hp := map[string]any{"release_id": plan["release_id"], "layer": layer, "lane": "fast", "vmid": vmid, "mode": "health", "packages": []Package{}}
+			hr, herr := l.call(ctx, runID, hp)
+			if herr == nil && hr.Health != nil {
+				cur = hr.Health
+			}
+		}
+		if !rep.Healthy && rep.Outcome == "applied" {
+			rep.Outcome = "health_failed"
+		}
+	} else {
+		rep.Healthy, rep.HealthReason = verdict(cur)
+	}
+	rep.Installed, rep.Pending = wr.Installed, wr.Pending
+	rep.RestartNeeded, rep.DockerRestartNeeded, rep.RebootNeeded = wr.RestartNeeded, wr.DockerRestartNeeded, wr.RebootNeeded
+	rep.RebootScanned = wr.RebootScanned
+	rep.NotCovered = notCovered(wr.Pending, blk.Ring, planned)
+	return l.finish(ctx, lg, rep)
+}
+
+// notCovered lists pending updates no approved release covers: in ring 0 everything outside the fast lane; in ring 1
+// also every fast-lane update the release did not name.
+func notCovered(pending []Pending, ring int, planned map[string]bool) []string {
+	var out []string
+	for _, p := range pending {
+		if !IsFast(p.Origin) || (ring == 1 && !planned[p.Name]) {
+			out = append(out, p.Name)
+		}
+	}
+	return out
+}
+
+func (l *Leg) finish(ctx context.Context, lg *slog.Logger, rep Report) Report {
+	lg.Info("osupdate: DONE", "outcome", rep.Outcome, "healthy", rep.Healthy, "reason", rep.HealthReason,
+		"upgraded", len(rep.Upgraded), "pending", len(rep.Pending), "not_covered", len(rep.NotCovered),
+		"restart_needed", len(rep.RestartNeeded), "reboot_needed", rep.RebootNeeded, "wrapper_seconds", rep.PassSeconds)
+	if l.Hub != nil {
+		body, _ := json.Marshal(rep)
+		rctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), time.Minute)
+		defer cancel()
+		if err := l.Hub.PostOSReport(rctx, body); err != nil {
+			lg.Warn("osupdate: reporting to the hub failed (the run itself is done)", "err", err)
+		}
+	}
+	return rep
+}

internal/osupdate/leg_test.go

@@ -0,0 +1,369 @@
+package osupdate
+
+import (
+	"context"
+	"encoding/json"
+	"io"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
+	"testing"
+	"time"
+
+	"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
+)
+
+// fakeWrapper plays /usr/local/sbin/felhom-os-apply: it reads the plan the leg wrote and answers per layer and mode.
+type fakeWrapper struct {
+	t         *testing.T
+	pending   []Pending
+	applyRep  map[string]WrapperReport // per layer
+	healthSeq map[string][]*Health     // per layer: answers to successive "health" calls
+	plans     []map[string]any
+}
+
+func yes() *bool { b := true; return &b }
+
+func guestOK() *Health {
+	return &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{
+		"felhom-controller": {State: "running", Health: "healthy"}, "app": {State: "running", Health: "healthy"}}}
+}
+
+func hostOK() *Health {
+	return &Health{HostServices: map[string]string{"pveproxy": "active", "pvedaemon": "active", "pvestatd": "active",
+		"pve-cluster": "active", "felhom-agent": "active"}, GuestRunning: yes(), Guest: guestOK()}
+}
+
+func (f *fakeWrapper) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
+	if name != WrapperPath || len(args) != 2 || args[0] != "--plan" {
+		f.t.Fatalf("unexpected command %s %v", name, args)
+	}
+	b, err := os.ReadFile(args[1])
+	if err != nil {
+		f.t.Fatal(err)
+	}
+	var plan map[string]any
+	json.Unmarshal(b, &plan)
+	f.plans = append(f.plans, plan)
+	layer := plan["layer"].(string)
+	ok := guestOK()
+	if layer == LayerHost {
+		ok = hostOK()
+	}
+	var rep WrapperReport
+	switch plan["mode"] {
+	case "inventory":
+		rep = WrapperReport{Mode: "inventory", Pending: f.pending, HealthBefore: ok, HealthAfter: ok,
+			Installed: []Package{{Name: "libc6", Version: "u3", Origin: "Debian"}}}
+	case "apply":
+		rep = f.applyRep[layer]
+		rep.Mode = "apply"
+		if rep.HealthBefore == nil {
+			rep.HealthBefore = ok
+		}
+		if rep.HealthAfter == nil {
+			rep.HealthAfter = ok
+		}
+	case "health":
+		if seq := f.healthSeq[layer]; len(seq) > 0 {
+			rep.Health, f.healthSeq[layer] = seq[0], seq[1:]
+		} else {
+			rep.Health = ok
+		}
+	}
+	out, _ := json.Marshal(rep)
+	return []byte("OSAPPLY-REPORT " + string(out) + "\n"), []byte("os-apply: DONE rc=0\n"), nil
+}
+
+func (f *fakeWrapper) RunStdin(ctx context.Context, _ io.Reader, name string, args ...string) ([]byte, []byte, error) {
+	return f.Run(ctx, name, args...)
+}
+
+type fakeHub struct{ reports []Report }
+
+func (h *fakeHub) PostOSReport(_ context.Context, body []byte) error {
+	var r Report
+	json.Unmarshal(body, &r)
+	h.reports = append(h.reports, r)
+	return nil
+}
+
+type fakeTunnel struct{ st string }
+
+func (t fakeTunnel) Status(context.Context) (string, string) { return t.st, "" }
+
+func newLeg(t *testing.T, w *fakeWrapper, blk *hub.WireOSUpdate) (*Leg, *fakeHub) {
+	h := &fakeHub{}
+	now := time.Date(2026, 10, 4, 4, 0, 0, 0, time.UTC)
+	if w.applyRep == nil {
+		w.applyRep = map[string]WrapperReport{}
+	}
+	if w.healthSeq == nil {
+		w.healthSeq = map[string][]*Health{}
+	}
+	l := &Leg{Runner: w, Hub: h, PlanDir: t.TempDir(), StatePath: filepath.Join(t.TempDir(), "last"),
+		HealthWait: time.Minute, HealthPoll: 10 * time.Second, Appliance: true, Tunnel: fakeTunnel{hub.TunnelRunning},
+		Now:   func() time.Time { return now },
+		Sleep: func(_ context.Context, d time.Duration) { now = now.Add(d) }}
+	if blk != nil {
+		l.SetBlock(blk)
+	}
+	return l, h
+}
+
+var pend = []Pending{
+	{Name: "libc6", From: "u3", To: "u4", Origin: []string{"Debian"}},
+	{Name: "openssl", From: "u1", To: "u3", Origin: []string{"Debian-Security", "Debian"}},
+	{Name: "docker-ce", From: "29.7", To: "29.8", Origin: []string{"Docker CE"}},
+}
+
+func calls(w *fakeWrapper) string {
+	var m []string
+	for _, p := range w.plans {
+		m = append(m, p["layer"].(string)+":"+p["mode"].(string))
+	}
+	return strings.Join(m, ",")
+}
+
+// Ring 0: ONE wrapper call per layer (R-845), select pending-fast (the wrapper picks every Debian / Debian-Security
+// upgrade), from live sources; the guest step first, then the host step.
+func TestRing0_OneCallPerLayer(t *testing.T) {
+	w := &fakeWrapper{t: t, pending: pend, applyRep: map[string]WrapperReport{
+		LayerGuest: {Upgraded: []Package{{Name: "libc6", Version: "u4"}, {Name: "openssl", Version: "u3"}}, Pending: pend[2:]},
+		LayerHost:  {Upgraded: []Package{{Name: "openssl", Version: "u3"}}},
+	}}
+	l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
+	g, ho := l.Run(context.Background(), 9201, "night")
+	if g.Outcome != "applied" || !g.Healthy || ho.Outcome != "applied" || !ho.Healthy {
+		t.Fatalf("guest %+v\nhost %+v", g, ho)
+	}
+	if calls(w) != "guest:apply,host:apply" {
+		t.Fatalf("calls = %s, want one apply per layer, guest first", calls(w))
+	}
+	for _, p := range w.plans {
+		if p["select"] != "pending-fast" || p["snapshot"] != "" || len(p["packages"].([]any)) != 0 {
+			t.Fatalf("ring-0 plan = %v", p)
+		}
+	}
+	if len(g.NotCovered) != 1 || g.NotCovered[0] != "docker-ce" {
+		t.Fatalf("not covered = %v", g.NotCovered)
+	}
+	if len(h.reports) != 2 || h.reports[0].Layer != LayerGuest || h.reports[1].Layer != LayerHost {
+		t.Fatalf("hub got %+v", h.reports)
+	}
+}
+
+// Ring 1 installs EXACTLY each layer's own approved release (a guest release is not a host release).
+func TestRing1_EachLayerItsOwnRelease(t *testing.T) {
+	w := &fakeWrapper{t: t, pending: pend, applyRep: map[string]WrapperReport{
+		LayerGuest: {Upgraded: []Package{{Name: "libc6", Version: "g-u4"}}, Pending: pend[1:]},
+		LayerHost:  {Upgraded: []Package{{Name: "openssl", Version: "h-u3"}}},
+	}}
+	gr := &hub.WireOSRelease{ID: "os-g", Snapshot: "20261004T080000Z", Packages: []hub.WireOSPackage{{Name: "libc6", Version: "g-u4", Origin: "Debian"}}}
+	hr := &hub.WireOSRelease{ID: "os-h", Snapshot: "20261004T090000Z", Packages: []hub.WireOSPackage{{Name: "openssl", Version: "h-u3", Origin: "Debian-Security"}}}
+	l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true, Release: gr, HostRelease: hr})
+	g, ho := l.Run(context.Background(), 9201, "night")
+	if g.ReleaseID != "os-g" || ho.ReleaseID != "os-h" {
+		t.Fatalf("release ids %q %q", g.ReleaseID, ho.ReleaseID)
+	}
+	gp, hp := w.plans[0], w.plans[1]
+	if gp["snapshot"] != "20261004T080000Z" || gp["packages"].([]any)[0].(map[string]any)["version"] != "g-u4" {
+		t.Fatalf("guest plan %v", gp)
+	}
+	if hp["layer"] != LayerHost || hp["snapshot"] != "20261004T090000Z" || hp["packages"].([]any)[0].(map[string]any)["name"] != "openssl" {
+		t.Fatalf("host plan %v", hp)
+	}
+	if strings.Join(g.NotCovered, ",") != "openssl,docker-ce" {
+		t.Fatalf("guest not covered = %v", g.NotCovered)
+	}
+}
+
+func TestRing1_NoReleaseIsInventory(t *testing.T) {
+	w := &fakeWrapper{t: t, pending: pend}
+	l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
+	g, ho := l.Run(context.Background(), 9201, "night")
+	if g.Outcome != "nothing" || ho.Outcome != "nothing" || calls(w) != "guest:inventory,host:inventory" {
+		t.Fatalf("g=%+v h=%+v calls=%s", g, ho, calls(w))
+	}
+}
+
+// No block from the hub (an older hub): ring 1, ON, no release → reports, installs nothing.
+func TestNoBlock_IsRing1Nothing(t *testing.T) {
+	w := &fakeWrapper{t: t, pending: pend}
+	l, _ := newLeg(t, w, nil)
+	if g, _ := l.Run(context.Background(), 9201, "night"); g.Outcome != "nothing" || g.Ring != 1 {
+		t.Fatalf("g=%+v calls=%s", g, calls(w))
+	}
+}
+
+// Switched OFF: the box reports but installs nothing, on both layers.
+func TestSwitchOff_ReportsOnly(t *testing.T) {
+	w := &fakeWrapper{t: t, pending: pend}
+	l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: false})
+	g, ho := l.Run(context.Background(), 9201, "night")
+	if g.Outcome != "inventory" || ho.Outcome != "inventory" || calls(w) != "guest:inventory,host:inventory" || len(h.reports) != 2 {
+		t.Fatalf("g=%+v h=%+v calls=%s", g, ho, calls(w))
+	}
+}
+
+// Not an appliance (BYO, `11` §1): the host step never runs — no host plan at all.
+func TestBYO_NoHostPlan(t *testing.T) {
+	w := &fakeWrapper{t: t, pending: pend, applyRep: map[string]WrapperReport{LayerGuest: {Upgraded: []Package{{Name: "libc6"}}}}}
+	l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
+	l.Appliance = false
+	_, ho := l.Run(context.Background(), 9201, "night")
+	if ho.Outcome != "" || calls(w) != "guest:apply" || len(h.reports) != 1 {
+		t.Fatalf("a BYO box got a host step: host=%+v calls=%s", ho, calls(w))
+	}
+}
+
+// A failed guest step skips the host step that night.
+func TestGuestFailure_SkipsTheHost(t *testing.T) {
+	w := &fakeWrapper{t: t, pending: pend, applyRep: map[string]WrapperReport{
+		LayerGuest: {Refused: json.RawMessage(`{"code":"R6","reason":"x"}`)}}}
+	l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
+	g, ho := l.Run(context.Background(), 9201, "night")
+	if g.Outcome != "refused" || ho.Outcome != "" || calls(w) != "guest:apply" {
+		t.Fatalf("g=%+v h=%+v calls=%s", g, ho, calls(w))
+	}
+}
+
+// Unhealthy after the run, and still unhealthy at the end of the wait → health_failed; the host step is skipped.
+func TestHealth_FailsAfterTheWait(t *testing.T) {
+	bad := &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{
+		"felhom-controller": {State: "running", Health: "healthy"}, "app": {State: "exited"}}}
+	w := &fakeWrapper{t: t, pending: pend, applyRep: map[string]WrapperReport{LayerGuest: {Upgraded: []Package{{Name: "libc6"}}, HealthAfter: bad}},
+		healthSeq: map[string][]*Health{LayerGuest: {bad, bad, bad, bad, bad, bad, bad, bad}}}
+	l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
+	g, ho := l.Run(context.Background(), 9201, "night")
+	if g.Outcome != "health_failed" || g.Healthy || !strings.Contains(g.HealthReason, "app was running") || ho.Outcome != "" {
+		t.Fatalf("g=%+v h=%+v", g, ho)
+	}
+	if h.reports[0].Outcome != "health_failed" {
+		t.Fatal("the hub was not told")
+	}
+}
+
+// A service that takes a moment to come back is not a failure: the poll sees it recover inside the wait.
+func TestHealth_RecoversInsideTheWait(t *testing.T) {
+	starting := &Health{DockerOK: true, NetworkOK: true, Controller: "starting"}
+	w := &fakeWrapper{t: t, pending: pend, applyRep: map[string]WrapperReport{LayerGuest: {Upgraded: []Package{{Name: "libc6"}}, HealthAfter: starting}},
+		healthSeq: map[string][]*Health{LayerGuest: {starting}}}
+	l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
+	if g, _ := l.Run(context.Background(), 9201, "night"); g.Outcome != "applied" || !g.Healthy {
+		t.Fatalf("g = %+v", g)
+	}
+}
+
+// The host step judged unhealthy when the tunnel is down after it.
+func TestHost_TunnelDownFailsTheHostStep(t *testing.T) {
+	w := &fakeWrapper{t: t, pending: pend, applyRep: map[string]WrapperReport{LayerHost: {Upgraded: []Package{{Name: "openssl"}}}}}
+	l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
+	l.Tunnel = fakeTunnel{hub.TunnelNotRunning}
+	_, ho := l.Run(context.Background(), 9201, "night")
+	if ho.Outcome != "health_failed" || !strings.Contains(ho.HealthReason, "tunnel") {
+		t.Fatalf("host = %+v", ho)
+	}
+}
+
+func TestHealthVerdict(t *testing.T) {
+	ok := &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{"a": {State: "running", Health: "healthy"}}}
+	cases := []struct {
+		name   string
+		before *Health
+		after  *Health
+		want   bool
+	}{
+		{"all good", ok, ok, true},
+		{"no reading", ok, nil, false},
+		{"docker down", ok, &Health{NetworkOK: true, Controller: "healthy"}, false},
+		{"no network", ok, &Health{DockerOK: true, Controller: "healthy"}, false},
+		{"controller starting", ok, &Health{DockerOK: true, NetworkOK: true, Controller: "starting"}, false},
+		{"only the controller differs", ok, &Health{DockerOK: true, NetworkOK: true, Controller: "unhealthy", Containers: map[string]Container{"a": {State: "running", Health: "healthy"}}}, false},
+		{"app gone", ok, &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{}}, false},
+		{"app unhealthy", ok, &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{"a": {State: "running", Health: "unhealthy"}}}, false},
+		{"stopped before stays stopped", &Health{Containers: map[string]Container{"x": {State: "exited"}}}, &Health{DockerOK: true, NetworkOK: true, Controller: "healthy"}, true},
+	}
+	for _, c := range cases {
+		if got, why := HealthVerdict(c.before, c.after); got != c.want {
+			t.Errorf("%s: got %v (%s), want %v", c.name, got, why, c.want)
+		}
+	}
+}
+
+// The host rule: every listed daemon active, the guest running and passing its own rule, the tunnel running.
+// Red-proofs: drop any one check and its case fails.
+func TestHostHealthVerdict(t *testing.T) {
+	no := false
+	svcDown := hostOK()
+	svcDown.HostServices["pveproxy"] = "failed"
+	guestDown := hostOK()
+	guestDown.GuestRunning = &no
+	guestApp := hostOK()
+	guestApp.Guest = &Health{DockerOK: true, NetworkOK: true, Controller: "healthy", Containers: map[string]Container{"felhom-controller": {State: "running", Health: "healthy"}}}
+	cases := []struct {
+		name   string
+		after  *Health
+		tunnel string
+		want   bool
+		why    string
+	}{
+		{"all good", hostOK(), hub.TunnelRunning, true, ""},
+		{"a daemon down", svcDown, hub.TunnelRunning, false, "pveproxy"},
+		{"the guest stopped", guestDown, hub.TunnelRunning, false, "guest is not running"},
+		{"an app in the guest gone", guestApp, hub.TunnelRunning, false, "app was running"},
+		{"the tunnel down", hostOK(), hub.TunnelNotRunning, false, "tunnel"},
+		{"the tunnel unknown", hostOK(), hub.TunnelUnknown, false, "tunnel"},
+		{"no services read", &Health{GuestRunning: yes(), Guest: guestOK()}, hub.TunnelRunning, false, "no host service"},
+	}
+	for _, c := range cases {
+		got, why := HostHealthVerdict(hostOK(), c.after, c.tunnel)
+		if got != c.want || !strings.Contains(why, c.why) {
+			t.Errorf("%s: got %v (%s), want %v (…%s…)", c.name, got, why, c.want, c.why)
+		}
+	}
+}
+
+func TestOncePerNight(t *testing.T) {
+	w := &fakeWrapper{t: t, pending: pend, applyRep: map[string]WrapperReport{LayerGuest: {Upgraded: []Package{{Name: "libc6"}}}}}
+	l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
+	l.Run(context.Background(), 9201, "night")
+	n := len(w.plans)
+	if g, _ := l.Run(context.Background(), 9201, "night"); g.Outcome != "skipped" || len(w.plans) != n {
+		t.Fatalf("a second night run in the same night ran: %+v", g)
+	}
+	if g, _ := l.Run(context.Background(), 9201, "debug"); g.Outcome == "skipped" {
+		t.Fatal("the debug action must not be throttled")
+	}
+}
+
+// The root wrapper's own suite (configs/test_felhom_os_apply.py) runs with `go test ./...` so CI covers it.
+func TestWrapperSuite(t *testing.T) {
+	py, err := exec.LookPath("python3")
+	if err != nil {
+		t.Skip("python3 not available")
+	}
+	cmd := exec.Command(py, "-B", "../../configs/test_felhom_os_apply.py")
+	out, err := cmd.CombinedOutput()
+	if err != nil {
+		t.Fatalf("wrapper suite failed: %v\n%s", err, out)
+	}
+	if !strings.Contains(string(out), "OK") {
+		t.Fatalf("wrapper suite did not report OK:\n%s", out)
+	}
+}
+
+// The host's restart scan result reaches the hub with reboot_scanned, so the hub can tell "looked: not needed" (a
+// reboot cleared it) from "did not look". Red-proof: drop the RebootScanned copy in runLayer and this fails.
+func TestHostReport_CarriesRebootScanned(t *testing.T) {
+	w := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{
+		LayerGuest: {},
+		LayerHost:  {RebootScanned: true, RebootNeeded: true, RestartNeeded: []string{"lxc-start"}},
+	}}
+	l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
+	l.Run(context.Background(), 9201, "night")
+	if len(h.reports) != 2 || !h.reports[1].RebootScanned || !h.reports[1].RebootNeeded || h.reports[0].RebootScanned {
+		t.Fatalf("hub got %+v", h.reports)
+	}
+}