os-apply: the host restart scan no longer hides lxc-start (skip ':/lxc/' not 'lxc'), and the host scans on every pass so a reboot clears 'reboot needed' (reboot_scanned reaches the hub) — both found live on demo-felhom

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS

CHANGELOG.md

@@ -1,3 +1,35 @@
+## v0.141.0 — OS updates: the host fast lane (`11` §8 step 3); the tunnel status is true (R-841); the leg is fast (R-845)
+
+> **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.141.0` (`cfba0d0`), sha256
+> `6eaad9809613c4fca64aefc30cc16415528499bf48efe3a1daaa08af8a611aff`, verified by download. Not vouched at release time.
+
+**MinAgent impact:** none required by any controller. **Needs hub v0.131.0** (`host_release`, the tunnel's three
+states, layer-tagged OS reports). With an older hub the host step finds no host release (ring 1 → nothing) and the
+tunnel's `detail` is ignored. Reads the cloudflared health check controller v0.292.0 adds; with an older controller
+the tunnel is judged on the container state alone and `detail` says so.
+
+- **R-841 — the tunnel.** The agent used to run `systemctl is-active cloudflared` on the HOST — a unit that does not
+  exist (cloudflared is a container in the customer guest), so every box reported `inactive`. `GuestTunnelProber`
+  now reads the guest's `cloudflared` container through the EXISTING sudoers line (`pct exec N -- docker inspect -f
+  *`): state, exit code, and the Docker health status. Three states: `running` (healthy), `not_running` (stopped,
+  absent, or running but NOT connected), `unknown` (could not ask, or the check is still starting). `detail` says why.
+- **The host step** (`11` §8 step 3). After a healthy guest step, under the same heavy-op gate, the leg runs the same
+  wrapper with `layer: host`. Debian origin only; never kernel, boot or firmware packages (new refusal **R14**);
+  **only on an appliance** (**R12** lifted for the host fast lane: proof is the ROOT-owned install record
+  `/var/lib/felhom-install/state.json` `mode: appliance` — the agent-writable `agent.json` is not trusted for this).
+  A failed or unhealthy guest step skips it. **Host health rule:** the agent, pveproxy, pvedaemon, pvestatd and
+  pve-cluster are active; the customer guest runs; the guest health rule passes; the tunnel is `running` (an
+  `unknown` tunnel does not fail the rule; `not_running` does). Never reboots: "reboot needed" is reported when PID 1
+  or `lxc-start` runs a replaced library. No automatic undo — the by-hand runbook is `os-updates-host-undo.md`.
+- **R-845 — the leg is fast.** The wrapper asked about each package in its own `pct exec` (about 0.9 s each). It now
+  makes one call per layer for the version checks (`apt-cache madison` for all names at once, `dpkg --compare-versions`
+  on the host), scans for restart-needed only after an install, repairs only when `dpkg --audit` reports something,
+  and reports its own `pass_seconds`.
+- **Wrapper tests:** `configs/test_felhom_os_apply.py` 46 tests (host layer, R12, R14, lxc-start, the speed rules).
+  Red-proofs: `felhom.eu/documentation/audits/os-host-lane-2026-10-04/partA/`, `partB/`, `partC/agent-golden-redproof.txt`.
+- **Contract:** the desired-state golden gains `host_release` (byte-identical with the hub's); `TestOSUpdateGolden_Decodes`
+  checks it.
+
 ## v0.140.0 — OS updates, guest fast lane (`11-os-updates.md` §8 step 2; `09` §3 decisions 76, 79, 80)
 
 > **RELEASED 2026-10-04** by `scripts/release-agent.sh` — tag `v0.140.0` (`9cac346`), sha256

REPORT.md

@@ -1,11 +1,9 @@
-# REPORT — 2026-10-04: v0.140.0, OS updates (guest fast lane)
+# REPORT — 2026-10-04: v0.141.0, the host fast lane, the true tunnel status, the fast leg
 
-Full session report: `felhom.eu/REPORT-os-guest-lane-2026-10-04.md`.
+Full session report: `felhom.eu/REPORT-os-host-lane-2026-10-04.md`.
 
-- `felhom-os-apply` wrapper (R1–R13, repair first, snapshot.debian.org fallback), `FELHOM_OSAPPLY` sudoers, the OS leg
-  after the primary backup, `--selftest=os-update`. Released `9cac346`, sha256 `ae2d60b7…1250`, verified by download.
-- Live on both demo boxes: ring 0 installed 53 packages each, healthy; ring 1 installed exactly the 3 approved versions;
-  a deliberately failed health check reported `health_failed` and mailed the operator.
-- Found and fixed live: the `--selftest` flag refused `os-update` (and `wgtunnel`, since S3); the conffile log line
-  called an updated file "kept"; an app stopped between the inventory and the apply escaped the health check.
-- No automatic undo (R-837: PVE refuses a snapshot of a guest with host-path binds).
+- Tunnel (R-841): the agent reads the guest's cloudflared container and its health check; three states.
+- Host fast lane: the wrapper gains the host layer (R12 appliance proof from the root-owned install record, R14 no
+  kernel/boot/firmware); the leg runs the host step after a healthy guest step; host health rule.
+- Speed (R-845): one call per layer instead of one per package; measured before/after in the session report.
+- Tests green; red-proofs in the audit folder.

configs/felhom-os-apply

@@ -58,6 +58,11 @@ INSTALL_STATE = "/var/lib/felhom-install/state.json"
 # reboot is needed for them to take effect, and a bad one can stop the box from booting.
 HOST_SLOW_RE = re.compile(r"^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|"
                           r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr)")
+# restart_needed() leaves out processes whose cgroup line matches (grep basic regex). Host: the LXC guests' own
+# processes (`0::/lxc/<vmid>/...`) -- NOT lxc-start itself, whose cgroup is `0::/lxc.monitor/<vmid>` (measured
+# 2026-10-04 on demo-felhom: the old pattern "lxc" hid lxc-start with 20 deleted maps, so "reboot needed" stayed false
+# after a libc6 update). Pinned by test_restart_skip_patterns_against_real_cgroups.
+RESTART_SKIP_CGROUP = {"guest": "docker", "host": ":/lxc/"}
 HOST_SERVICES = ["pveproxy", "pvedaemon", "pvestatd", "pve-cluster", "felhom-agent"]
 
 
@@ -310,7 +315,7 @@ class Apply:
     def restart_needed(self):
         """Processes still mapping deleted files, OUTSIDE containers (C11). Guest: outside docker; host: outside the
         LXC guests (the host's /proc shows guest processes too)."""
-        skip = "docker" if self.layer == "guest" else "lxc"
+        skip = RESTART_SKIP_CGROUP[self.layer]
         script = ('for p in /proc/[0-9]*; do grep -q "(deleted)" $p/maps 2>/dev/null || continue; '
                   'grep -q "%s" $p/cgroup 2>/dev/null && continue; echo "${p#/proc/} $(cat $p/comm 2>/dev/null)"; done' % skip)
         rc, out, _ = self.x(["sh", "-c", script], timeout=120)
@@ -392,6 +397,12 @@ class Apply:
             if rc:
                 return rc
         self.report.update(self.inventory(installed_after))
+        if self.layer == "host" and "reboot_needed" not in self.report:
+            # The host is scanned on EVERY pass (local, no pct exec): a reboot must CLEAR "reboot needed", or the hub's
+            # 14-day alarm fires on a host that was rebooted long ago. The guest still scans only after an install
+            # (R-845; one pct exec, and no alarm reads it). Pinned by test_host_scans_every_pass_guest_only_after_install.
+            self.report["restart_needed"], self.report["reboot_needed"] = self.restart_needed()
+        self.report["reboot_scanned"] = "reboot_needed" in self.report
         self.report["health_after"] = self.health()
         return 0
 

configs/test_felhom_os_apply.py

@@ -551,6 +551,27 @@ class HostLayer(unittest.TestCase):
         rc, rep = run(f)
         self.assertTrue(rep["reboot_needed"], rep)
 
+    def test_host_scans_every_pass_guest_only_after_install(self):
+        # A host pass that installs nothing still scans, so a reboot clears the flag (the hub alarm reads it).
+        f = Fake()
+        f.plan["layer"] = "host"
+        f.plan["mode"] = "inventory"
+        f.restart_out = "2101 lxc-start\n"
+        rc, rep = run(f)
+        self.assertEqual(rc, 0, rep)
+        self.assertTrue(rep["reboot_scanned"], rep)
+        self.assertTrue(rep["reboot_needed"], rep)
+        f = Fake()
+        f.plan["layer"] = "host"
+        f.plan["mode"] = "inventory"
+        f.restart_out = ""  # after the reboot: nothing maps a deleted file
+        rc, rep = run(f)
+        self.assertTrue(rep["reboot_scanned"] and rep["reboot_needed"] is False, rep)
+        f = Fake()
+        f.plan["mode"] = "inventory"
+        rc, rep = run(f)
+        self.assertFalse(rep["reboot_scanned"], "the guest scans only after an install (R-845)")
+
     def test_no_reboot_for_ordinary_daemons(self):
         f = Fake()
         f.plan["layer"] = "host"
@@ -586,5 +607,22 @@ class Failure(unittest.TestCase):
         self.assertTrue(any(l.startswith("os-apply: FAILED rc=100 step=install") for l in f.logs))
 
 
+
+class RestartSkipPattern(unittest.TestCase):
+    """The cgroup filter runs as `grep -q PATTERN /proc/<pid>/cgroup`; check it with grep itself against the cgroup
+    lines measured on demo-felhom 2026-10-04."""
+
+    def grep(self, pattern, line):
+        return subprocess.run(["grep", "-q", pattern], input=line + "\n", text=True).returncode == 0
+
+    def test_restart_skip_patterns_against_real_cgroups(self):
+        host = osapply.RESTART_SKIP_CGROUP["host"]
+        self.assertTrue(self.grep(host, "0::/lxc/9201/ns/system.slice/docker.service"), "a guest process must be skipped")
+        self.assertFalse(self.grep(host, "0::/lxc.monitor/9201"), "lxc-start must NOT be skipped (it runs the guest)")
+        self.assertFalse(self.grep(host, "0::/system.slice/pve-cluster.service"), "a host daemon must NOT be skipped")
+        guest = osapply.RESTART_SKIP_CGROUP["guest"]
+        self.assertTrue(self.grep(guest, "0::/system.slice/docker-0123abcd.scope"))
+        self.assertFalse(self.grep(guest, "0::/system.slice/cron.service"))
+
 if __name__ == "__main__":
     unittest.main()

internal/osupdate/leg.go

@@ -84,6 +84,7 @@ type WrapperReport struct {
 	RestartNeeded       []string        `json:"restart_needed"`
 	DockerRestartNeeded bool            `json:"docker_restart_needed"`
 	RebootNeeded        bool            `json:"reboot_needed"`
+	RebootScanned       bool            `json:"reboot_scanned"`
 	HealthBefore        *Health         `json:"health_before"`
 	HealthAfter         *Health         `json:"health_after"`
 	Health              *Health         `json:"health"`
@@ -112,6 +113,7 @@ type Report struct {
 	RestartNeeded       []string        `json:"restart_needed,omitempty"`
 	DockerRestartNeeded bool            `json:"docker_restart_needed,omitempty"`
 	RebootNeeded        bool            `json:"reboot_needed,omitempty"`
+	RebootScanned       bool            `json:"reboot_scanned,omitempty"` // the pass looked (host: every pass) — a false RebootNeeded then means "not needed"
 	Refused             json.RawMessage `json:"refused,omitempty"`
 	PassSeconds         float64         `json:"pass_seconds,omitempty"`
 }
@@ -461,6 +463,7 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
 	}
 	rep.Installed, rep.Pending = wr.Installed, wr.Pending
 	rep.RestartNeeded, rep.DockerRestartNeeded, rep.RebootNeeded = wr.RestartNeeded, wr.DockerRestartNeeded, wr.RebootNeeded
+	rep.RebootScanned = wr.RebootScanned
 	rep.NotCovered = notCovered(wr.Pending, blk.Ring, planned)
 	return l.finish(ctx, lg, rep)
 }

internal/osupdate/leg_test.go

@@ -353,3 +353,17 @@ func TestWrapperSuite(t *testing.T) {
 		t.Fatalf("wrapper suite did not report OK:\n%s", out)
 	}
 }
+
+// The host's restart scan result reaches the hub with reboot_scanned, so the hub can tell "looked: not needed" (a
+// reboot cleared it) from "did not look". Red-proof: drop the RebootScanned copy in runLayer and this fails.
+func TestHostReport_CarriesRebootScanned(t *testing.T) {
+	w := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{
+		LayerGuest: {},
+		LayerHost:  {RebootScanned: true, RebootNeeded: true, RestartNeeded: []string{"lxc-start"}},
+	}}
+	l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
+	l.Run(context.Background(), 9201, "night")
+	if len(h.reports) != 2 || !h.reports[1].RebootScanned || !h.reports[1].RebootNeeded || h.reports[0].RebootScanned {
+		t.Fatalf("hub got %+v", h.reports)
+	}
+}