diff --git a/CHANGELOG.md b/CHANGELOG.md index 494aba7..363e8c8 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,45 @@ +## Unreleased — part of v0.152.0: the kernel lane (R-836; `09` §3 decisions 164, 172; `11` §5.11) (2026-10-07) + +**Delivery: agent binary, then the STEP bundle `0.152.0-step1`, then the bundle `0.152.0`** — the bundle ADDS two paths +(the GRUB generators), and an installed `felhom-os-apply` refuses a path its own table lacks (R16, R-880). + +A new kernel boots ONCE; if it crashes the box comes back on the old kernel by itself; it becomes the default only after +a healthy boot; a booted-but-unhealthy kernel is reverted ONCE by the agent with no person. Built on the spike's +candidate 2 (`audits/kernel-spike-2026-10-07/`), with option C on the one-shot entry. + +- `configs/felhom-grub-oneshot.sh` → `/etc/grub.d/01_felhom_oneshot` (bundle): reads `felhom_next` from a GRUB env block + on the ESP (`EFI/felhom/oneshot.env`), clears and saves it BEFORE the menu, and sets the default to that kernel's + one-shot entry only when the name is an installed kernel. No vfat ESP → prints nothing. +- `configs/felhom-grub-oneshot-entries.sh` → `/etc/grub.d/42_felhom_oneshot` (bundle): one entry per installed kernel, + id `felhom-oneshot-`, the normal entry plus `softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10` + (option C). Sorted after `10_linux`: never entry 0, never the default. +- `configs/felhom-os-apply`: layer `kernel` (lane slow; an appliance; authority = a signed `os_kernel_step` or the + root-owned ring-0 mark). Modes: `apply` STAGES (select `pending-kernel` or a signed `listed` set; `expect_kver` = the + kernel the household was told about): pins the GRUB default to the RUNNING kernel in + `/etc/default/grub.d/zz-felhom-kernel-default.cfg` and proves it from grub.cfg, installs, proves the default did not + move and the one-shot entry exists, writes the flag; never reboots. `kernel-reboot` (a staged step only), + `kernel-boot` (judging | fell_back | self_reverted | revert_failed), `kernel-good` (the new kernel becomes the + default, proved), `kernel-revert` (ONE per step; refused when the default is not the old kernel), `kernel-cancel`, + `kernel-status`. New refusals: R20 (the box cannot do a one-shot: not UEFI, no vfat ESP, a separate /boot, GRUB + without fat/loadenv, the generators missing, a hand pin), R21 (the crash guard tripped or an unclean boot in its + window), R22 (the phase does not allow the mode; never two steps within 20 h), R23 (not exactly one newer kernel, or + not the one signed / told). State `/var/lib/felhom-kernel/state.json`. Facts carry `kernel_lane`; the next-boot + kernel reads the flag and the grub.cfg default. Tests: `KernelLane` (27), red-proof + `felhom.eu/documentation/audits/kernel-lane-2026-10-07/A/redproof.txt`. +- `configs/felhom-crash-guard` unchanged; `KernelStepCannotLeaveTheBoxOff` (3 tests) pins that a step's planned reboot, + one crash and one self-revert add ONE unclean boot (a panic before userspace adds none), so the box cannot stay off. +- `internal/osupdate/kernel.go`: the night leg ends with the kernel step — after a healthy host step (and a healthy + Proxmox step when one ran), trigger `night` only, on a night the hub's `os_update.kernel` block marks `tonight` (the + household was mailed the day before — no mail, no step). Ring 0 stages + reboots; ring 1 reboots only a kernel a + signed `os_kernel_step` staged (`KernelStepExecutor`: stage only, under the heavy-op gate). The hub hears `staged` + BEFORE the reboot. At every start `KernelAfterBoot`: on the new kernel it JUDGES the boot — `KernelVerdict` = the + host health rule (`11` §8.2) AND the box reached the hub (the `judging` report itself) — for 20 minutes (measured: + everything healthy 68 s after the reboot on demo-felhom, 272 s on demo-hp; under the hub's 30-minute `host_stale`). + Healthy → `kernel-good`, outcome `applied`; not healthy → outcome `health_failed`, then ONE `kernel-revert`. + Tests: `TestKernel*` (13); red-proofs in the same file. +- `internal/hub`: `WireOSUpdate.Kernel` {kver, tonight, notified_at}. `internal/reconcile`: `os_kernel_step` is + destructive-class. `cmd/felhom-opsign`: the op is listed. + ## v0.151.0 — the agent can no longer hand the guest any image; the Proxmox package lane; the other-key archives reported; the DR directive retired (R-861, R-812 A, R-366, R-105; `09` §3 163, 165, 168, 169) (2026-10-07) Released by `scripts/release-agent.sh`: binary sha256 `0464354f2cdf452a7c5d2a74d9191fe91415fcfa244154480d26d5b30e10b194` diff --git a/REUSE.md b/REUSE.md index 3265fa8..12aa6ad 100644 --- a/REUSE.md +++ b/REUSE.md @@ -89,6 +89,7 @@ | Symbol | File | Short signature | Use for | Gotchas | |---|---|---|---|---| | `pvegate.Write` / `pvegate.Step` | internal/pvegate/pvegate.go | `Write(ctx) (release, waited, err)` / `Step(ctx) (end, err)` | R-812 option A: keep the agent's own /etc/pve writes out of a Proxmox package step (pmxcfs restarts) | Already wired at the two chokepoints — `Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile). A new root CLI that writes /etc/pve goes into `WritesEtcPVE`, never its own lock. Never take `Step` around anything but the wrapper call (`Leg.runPVE`) — a `Write` inside a `Step` deadlocks until its context ends. | +| `osupdate.KernelVerdict` / `Leg.KernelAfterBoot` | internal/osupdate/kernel.go | `KernelVerdict(before, after, tunnel, hubReached) (ok, why)` | R-836: THE one-shot-boot rule — the host health rule (`HostHealthVerdict`) AND the box reached the hub since the boot | The only judge of a new kernel. Never reboot the host from Go: every reboot is the wrapper's (`kernel-reboot`, `kernel-revert`), and only for a staged step. A new kernel-lane state lives in the wrapper's `/var/lib/felhom-kernel/state.json`, never in the agent's own files (the agent can write those). | | `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in | | `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them | | `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc | diff --git a/cmd/felhom-agent/main.go b/cmd/felhom-agent/main.go index ef60639..d42ce8a 100644 --- a/cmd/felhom-agent/main.go +++ b/cmd/felhom-agent/main.go @@ -877,6 +877,12 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int go osLeg.SendUnsentLoop(ctx, 5*time.Minute, func(n int) { logger.Info("osupdate: sent kept report(s)", "count", n) }) + // R-836 (`09` §3 decision 172): what became of a kernel step across this boot; on a one-shot boot of a new kernel, + // judge it (the host health rule + the hub reached) for KernelJudgeWait, then make it the default or revert ONCE. + // The wait: measured 2026-10-07 (`audits/kernel-lane-2026-10-07/B/`) — every container healthy 68 s after the + // reboot on demo-felhom and 272 s on demo-hp (the hub reached at 63 s / 189 s); 20 minutes leaves room for a slow + // network and stays under the hub's 30-minute host_stale. On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged. + go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait}) // Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue // (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving @@ -1119,6 +1125,16 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int } return release, nil }} + // R-836 (`09` §3 decision 172): a signed kernel step STAGES a kernel on a ring-1 box (install + the one-shot flag, + // never a reboot — the night leg reboots it on a night the household was told about); under the heavy-op gate. + kernelExec := osupdate.KernelStepExecutor{Leg: osLeg, Guest: firstGuest(px), + Gate: func(ctx context.Context) (func(), error) { + release, busy, ok := heavyOps.TryAcquire("os-kernel-step") + if !ok { + return nil, fmt.Errorf("busy: %s", busy) + } + return release, nil + }} // Agent v0.143.0 (R-840): the config bundle — the box's root-owned files by a signed job; the wrapper verifies it. bundleExec := osupdate.ConfigUpdateExecutor{Leg: osLeg, URLTemplate: suCfg.URLTemplate, Username: suCfg.Username, Token: suCfg.Token, // The capability probe confirms from the agent's side: `sudo -l` lists every command the new sudoers grants. @@ -1130,7 +1146,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int } logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ",")) }} - jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, pveExec, bundleExec}, cfg.Hub.HostID, logger) + jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, pveExec, kernelExec, bundleExec}, cfg.Hub.HostID, logger) loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner)) // Controller-driven escrow ceremony (v0.88.0): static config facts + the LATE-BOUND DR gate — diff --git a/cmd/felhom-opsign/main.go b/cmd/felhom-opsign/main.go index d62c920..0eafb63 100644 --- a/cmd/felhom-opsign/main.go +++ b/cmd/felhom-opsign/main.go @@ -43,7 +43,7 @@ func main() { func run() error { var ( - op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | os_pve_step | agent_config_update") + op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | os_pve_step | os_kernel_step | agent_config_update") host = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)") guest = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)") keyID = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)") diff --git a/configs/felhom-grub-oneshot-entries.sh b/configs/felhom-grub-oneshot-entries.sh new file mode 100644 index 0000000..2466e82 --- /dev/null +++ b/configs/felhom-grub-oneshot-entries.sh @@ -0,0 +1,39 @@ +#!/bin/sh +# /etc/grub.d/42_felhom_oneshot — the kernel lane's one-shot ENTRIES (R-836, `09` §3 decision 172, `11` §5.11). +# Installed by the config bundle (felhom-os-apply BUNDLE_FILES), 0755 root. update-grub runs it. +# +# One menu entry per installed Proxmox kernel, id `felhom-oneshot-`, booted ONLY when 01_felhom_oneshot found +# the flag naming it. It is the normal entry plus option C (decision 172): softlockup_panic=1 hardlockup_panic=1 +# hung_task_panic=1 panic=10 — a lockup the kernel can detect becomes a panic, and a panic restarts the box in 10 s into +# the default (the old kernel). A true dead freeze still needs a person (spike candidate 3 failed on all three boxes). +# It sorts AFTER 10_linux, so it is never entry 0 and never the default. +# +# No vfat ESP at /boot/efi → prints nothing (no flag can name these entries). +set -e +prefix="/usr" +exec_prefix="/usr" +datarootdir="/usr/share" +. "$datarootdir/grub/grub-mkconfig_lib" +esp_uuid=$(findmnt -n -o UUID,FSTYPE /boot/efi 2>/dev/null | awk '$2 == "vfat" { print $1 }') +[ -n "$esp_uuid" ] || exit 0 +kernels=$(ls /boot/vmlinuz-*-pve 2>/dev/null | sed 's#^/boot/vmlinuz-##' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$' || true) +[ -n "$kernels" ] || exit 0 +case "${GRUB_DEVICE}" in + /dev/mapper/*|/dev/dm-*|"") root_arg="root=${GRUB_DEVICE}" ;; + *) if [ -n "${GRUB_DEVICE_UUID}" ]; then root_arg="root=UUID=${GRUB_DEVICE_UUID}"; else root_arg="root=${GRUB_DEVICE}"; fi ;; +esac +[ -n "${GRUB_DEVICE}" ] || root_arg="root=$(findmnt -n -o SOURCE /)" +rel=$(make_system_path_relative_to_its_root /boot) +prep=$(prepare_grub_to_access_device "$(${grub_probe:-grub-probe} --target=device /boot)" | sed 's/^/ /') +for k in $kernels; do + [ -f "/boot/initrd.img-$k" ] || continue + cat </dev/null | awk '$2 == "vfat" { print $1 }') +[ -n "$esp_uuid" ] || exit 0 +kernels=$(ls /boot/vmlinuz-*-pve 2>/dev/null | sed 's#^/boot/vmlinuz-##' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$' || true) +[ -n "$kernels" ] || exit 0 +cat </dev/null || continue; ' 'grep -q "%s" $p/cgroup 2>/dev/null && continue; echo "${p#/proc/} $(cat $p/comm 2>/dev/null)"; done' % skip) rc, out, _ = self.x(["sh", "-c", script], timeout=120) @@ -987,8 +1047,12 @@ class Apply: return Bundle(self).from_plan(plan) if self.mode == "agent_update": return self.agent_update(plan) - if self.layer in ("host", "pve"): + if self.layer in ("host", "pve", "kernel"): self.check_appliance() + if self.layer == "kernel" and self.mode != "health": + # the kernel lane's own modes (R-836): most of them run while the guest is still starting after a boot, so + # the guest check is the stage's and the reboot's own (Kernel.run), not every mode's + return Kernel(self, plan).run() self.check_guest(self.vmid) log = self.r.log if self.mode == "live-restore-on": @@ -1364,6 +1428,422 @@ class Apply: self.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600) +def _iso(t): + return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(t)) + + +def _parse_iso(s): + try: + return calendar.timegm(time.strptime(s, "%Y-%m-%dT%H:%M:%SZ")) + except (TypeError, ValueError): + return None + + +class Kernel: + """The kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11). Refusal codes: R3 (authority), R4 (removal), + R6 (a package outside the step), R8 (space), R9 (locks), R12 (appliance), R20 (the box's boot setup cannot do a + one-shot), R21 (the crash guard is tripped or saw an unclean boot within its window), R22 (the step's phase does not + allow this mode), R23 (the kernel set is not one exact new kernel). + + Modes (plan "mode", layer "kernel", lane "slow"): + apply STAGE: install the kernel set, keep the GRUB default on the kernel the box runs, write the flag. + Never reboots. select "pending-kernel" (ring 0, the root-owned mark) or "listed" (a signed + os_kernel_step). expect_kver: the kernel the hub told the household about — any other is R23. + kernel-reboot a STAGED step's reboot (the night leg, after the household was told): phase oneshot, then reboot. + kernel-boot after a boot: what became of the step (judging | fell_back | self_reverted | revert_failed). + kernel-good the one-shot boot was healthy: the new kernel becomes the GRUB default. + kernel-revert the one-shot boot was NOT healthy: reboot ONCE into the old kernel (still the default). + kernel-cancel drop a staged step: clear the flag (the package stays installed, the default never moved). + kernel-status read only.""" + + def __init__(self, apply, plan): + self.a, self.r, self.plan = apply, apply.r, plan + self.report = apply.report + self.log = apply.r.log + + # ---------- reading the box ---------- + def running(self): + rc, out, _ = self.r.host(["uname", "-r"], 30) + v = out.strip() if rc == 0 else "" + return v if KVER_RE.match(v) else "" + + def state(self): + try: + s = json.loads(self.r.read_file(KERNEL_STATE)) + return s if isinstance(s, dict) else {} + except (OSError, ValueError): + return {} + + def save_state(self, s): + s["updated_at"] = _iso(self.r.now()) + self.r.put_file(KERNEL_STATE, (json.dumps(s, indent=2, sort_keys=True) + "\n").encode(), 0o644) + + def flag(self): + """The one-shot flag: the kernel named, "" when the env block holds none, None when there is no env block.""" + rc, out, _ = self.r.host(["grub-editenv", ONESHOT_ENV, "list"], 30) + if rc != 0: + return None + for l in out.splitlines(): + if l.startswith("felhom_next="): + return l.split("=", 1)[1].strip() + return "" + + def grub_cfg(self): + try: + return self.r.read_file(GRUB_CFG) + except OSError: + return "" + + @staticmethod + def default_kver(cfg): + """The kernel grub.cfg boots by default (00_header's `set default=`), or "unknown".""" + m = re.search(r'^\s*set default="(?:gnulinux-advanced-[^>"]*>)?gnulinux-([0-9][^"]*?-pve)-advanced-[^"]*"', cfg, re.M) + return m.group(1) if m else "unknown" + + @staticmethod + def entry_id(cfg, kver): + """The GRUB_DEFAULT value that names kver's normal entry, read from grub.cfg itself (10_linux's ids).""" + sub = re.search(r"\$menuentry_id_option '(gnulinux-advanced-[^']+)'", cfg) + m = re.search(r"\$menuentry_id_option '(gnulinux-" + re.escape(kver) + r"-advanced-[^']+)'", cfg) + if not m: + return None + return f"{sub.group(1)}>{m.group(1)}" if sub else m.group(1) + + def setup_problems(self): + """R20: why this box cannot do a one-shot boot (empty = it can). Measured shape: UEFI, a vfat ESP at /boot/efi, + /boot on the root filesystem, GRUB with fat + loadenv, the bundle's two generators, no hand pin.""" + why = [] + if not self.r.lexists("/sys/firmware/efi"): + why.append("the box does not boot UEFI") + rc, out, _ = self.r.host(["findmnt", "-n", "-o", "FSTYPE", ESP_MOUNT], 30) + if rc != 0 or out.strip() != "vfat": + why.append(f"{ESP_MOUNT} is not a mounted vfat ESP ({out.strip() or 'not mounted'})") + rc, out, _ = self.r.host(["findmnt", "-n", "-o", "TARGET", "/boot"], 30) + if rc == 0 and out.strip(): + why.append("/boot is a separate filesystem (the one-shot entries assume /boot on the root filesystem)") + for m in ("fat", "loadenv"): + if not self.r.lexists(f"/usr/lib/grub/x86_64-efi/{m}.mod"): + why.append(f"GRUB has no {m} module") + for p in (ONESHOT_SNIPPET, ONESHOT_ENTRIES): + if not self.r.lexists(p): + why.append(f"{p} is missing (the config bundle installs it)") + if self.r.lexists(KERNEL_PIN_FILE): + why.append("a kernel is pinned by hand (proxmox-boot-tool kernel pin) — the lane never fights it") + return why + + def guard(self): + try: + return json.loads(self.r.read_file(CRASH_GUARD_STATE)) + except (OSError, ValueError): + return None + + def check_guard(self): + """R21: a kernel step only on a box whose crash guard is armed and saw no unclean boot within its window — so + the step's own reboots (clean), one crash and one self-revert (clean) can never reach the 3rd unclean boot that + leaves the box off (`11` §5.9). Pinned by test_felhom_crash_guard KernelStepCannotLeaveTheBoxOff.""" + g = self.guard() + if not isinstance(g, dict): + raise Refused("R21", f"no crash guard state ({CRASH_GUARD_STATE}) — a kernel step needs the guard") + if g.get("tripped") or not g.get("armed"): + raise Refused("R21", "the crash guard is tripped — no kernel step until it re-arms") + if (g.get("unclean_boots_in_window") or 0) > 0: + raise Refused("R21", f"{g.get('unclean_boots_in_window')} unclean boot(s) within the guard's window — wait") + + def view(self, st=None, cfg=None): + st = self.state() if st is None else st + cfg = self.grub_cfg() if cfg is None else cfg + return {"running": self.running() or "unknown", "default": self.default_kver(cfg), "flag": self.flag(), + "phase": st.get("phase", "none"), "from": st.get("from"), "to": st.get("to"), + "step_id": st.get("step_id"), "self_revert_used": bool(st.get("self_revert_used")), "vmid": st.get("vmid"), + "staged_at": st.get("staged_at"), "rebooted_at": st.get("rebooted_at"), + "result_at": st.get("result_at"), "reason": st.get("reason")} + + # ---------- writing the box ---------- + def write_default(self, kver): + """Pin the GRUB default to kver's normal entry, regenerate grub.cfg, and PROVE it (the default read back).""" + cfg = self.grub_cfg() + eid = self.entry_id(cfg, kver) + if not eid: + raise Refused("R20", f"grub.cfg has no normal entry for {kver}") + body = ("# felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily\n" + f'GRUB_DEFAULT="{eid}"\n') + self.r.put_file(KERNEL_DEFAULT_CFG, body.encode(), 0o644) + rc, out, err = self.r.host(["update-grub"], 300) + got = self.default_kver(self.grub_cfg()) + if rc != 0 or got != kver: + raise Refused("R20", f"update-grub rc={rc}: the default reads {got}, not {kver}: {(out + err).strip()[-200:]}") + self.log(f"os-apply: KERNEL default = {kver} (proved from grub.cfg)") + + def set_flag(self, kver): + self.r.host(["mkdir", "-p", os.path.dirname(ONESHOT_ENV)], 30) + if self.flag() is None: + rc, out, err = self.r.host(["grub-editenv", ONESHOT_ENV, "create"], 30) + if rc != 0: + raise Refused("R20", f"cannot create the one-shot env block on the ESP: {(out + err).strip()[-200:]}") + rc, out, err = self.r.host(["grub-editenv", ONESHOT_ENV, "set", f"felhom_next={kver}"], 30) + if rc != 0 or self.flag() != kver: + raise Refused("R20", f"the one-shot flag did not read back as {kver}: {(out + err).strip()[-200:]}") + + def clear_flag(self): + if self.flag(): + self.r.host(["grub-editenv", ONESHOT_ENV, "unset", "felhom_next"], 30) + + def reboot(self, why): + self.log(f"os-apply: KERNEL REBOOT — {why}") + rc, out, err = self.r.host(["systemctl", "reboot"], 60) + self.report["reboot_rc"] = rc + if rc != 0: + self.report["failed"] = {"rc": 3, "step": "reboot", "reason": (out + err).strip()[-200:]} + return 3 + return 0 + + # ---------- the modes ---------- + def run(self): + mode = self.a.mode + self.report["kernel_mode"] = mode + if mode == "kernel-status": + v = self.view() + v["setup_problems"] = self.setup_problems() + self.report["kernel"] = v + return 0 + fn = {"apply": self.stage, "kernel-reboot": self.reboot_staged, "kernel-boot": self.after_boot, + "kernel-good": self.good, "kernel-revert": self.revert, "kernel-cancel": self.cancel}[mode] + rc = fn() + self.report["kernel"] = self.view() + return rc + + def phase_is(self, st, *phases): + if st.get("phase") not in phases: + raise Refused("R22", f"the kernel step is {st.get('phase', 'none')!r}, not {' or '.join(phases)} — " + f"{self.a.mode} does not apply") + + def stage(self): + a = self.a + st = self.state() + if st.get("phase") in KERNEL_ACTIVE: + raise Refused("R22", f"a kernel step is already {st['phase']} ({st.get('from')} -> {st.get('to')})") + last = _parse_iso(st.get("staged_at")) + if last is not None and self.r.now() - last < KERNEL_MIN_GAP: + raise Refused("R22", "a kernel step was staged within the last 20 hours — never two in one night") + why = self.setup_problems() + if why: + raise Refused("R20", "; ".join(why)) + self.check_guard() + a.check_guest(a.vmid) + who, _ = a.docker_authority(self.plan, op_name=KERNEL_OP) + self.report["authority"] = who + old = self.running() + if not old: + raise Refused("R20", "the running kernel is not a Proxmox kernel version") + self.log(f"os-apply: START release={self.plan.get('release_id')} layer=kernel lane=slow mode=apply " + f"select={a.select} authority={who} running={old}") + if a.apt_lock_held(): + raise Refused("R9", "another apt/dpkg holds the lock on the host") + self.report["health_before"] = a.health() + a.repair() + rc, out, err = a.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600) + if rc != 0: + raise Refused("R7", f"apt-get update failed on the host: {(out + err).strip().splitlines()[-1:]}") + inst = a.installed() + if a.select == "pending-kernel": + _, pend, _, _ = a.simulate(["dist-upgrade"]) + want = [(p["name"], p["to"]) for p in pend if p["from"] is not None and KERNEL_UPGRADE_RE.match(p["name"]) + and a.origin_name(p["origin"]) == {PVE_ORIGIN}] + else: + want = [(e["name"], e["version"]) for e in self.plan["packages"]] + # upgrades of installed names (never a downgrade), and at most the listed new kernel image + args, upg = [], {} + for n, v in want: + if n in inst: + if a.dpkg_cmp(v, "gt", inst[n]): + upg[n] = v + args.append(f"{n}={v}") + elif KERNEL_IMAGE_RE.match(n): + args.append(f"{n}={v}") + else: + raise Refused("R6", f"{n} is not installed and is not a kernel image") + if not args: + self.report["upgraded"], self.report["outcome_hint"] = [], "nothing" + self.log("os-apply: DONE rc=0 upgraded=0 (no pending kernel)") + return 0 + rc, sim, remv, text = a.simulate(["install", "--no-install-recommends"] + args) + if rc != 0: + raise Refused("R7", "the simulation failed: " + (text.strip().splitlines()[-1] if text.strip() else "")) + if remv: + raise Refused("R4", f"the kernel step would remove {', '.join(remv[:5])}") + images = [] + listed = dict(want) + for p in sim: + if a.origin_name(p["origin"]) != {PVE_ORIGIN}: + raise Refused("R2", f"{p['name']} would come from {p['origin']}, not {PVE_ORIGIN!r}") + m = KERNEL_IMAGE_RE.match(p["name"]) + if p["from"] is None: + if not m: + raise Refused("R6", f"the kernel step would add {p['name']}, which is not a kernel image") + if a.select == "listed" and listed.get(p["name"]) != p["to"]: + raise Refused("R23", f"the kernel step would add {p['name']}={p['to']}, not the signed set") + images.append((m.group(1), p["to"])) + continue + if p["name"] not in upg or p["to"] != upg[p["name"]]: + raise Refused("R6", f"the kernel step would touch {p['name']} ({p['to']}), which is not in the step") + if not a.dpkg_cmp(p["to"], "gt", p["from"]): + raise Refused("R5", f"{p['name']} would be downgraded {p['from']} -> {p['to']}") + if len(images) > 1: + raise Refused("R23", f"the kernel step would add {len(images)} kernels — one at a time") + # the target kernel: the new image, else the series meta-package's version (its image is already installed) + if images: + new = images[0][0] + if images[0][1] + "-pve" != new: + raise Refused("R23", f"the image version {images[0][1]} does not name the kernel {new}") + else: + metas = [(n, v) for n, v in upg.items() if re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n)] + if len(metas) != 1: + self.report["upgraded"], self.report["outcome_hint"] = [], "nothing" + self.log("os-apply: DONE rc=0 upgraded=0 (the pending set names no kernel to boot)") + return 0 + new = metas[0][1] + "-pve" + if not KVER_RE.match(new) or not a.dpkg_cmp(new[:-4], "gt", old[:-4]): + raise Refused("R23", f"the kernel {new} is not newer than the running {old}") + ek = self.plan.get("expect_kver") + if ek and ek != new: + raise Refused("R23", f"the step would boot {new}, but the household was told about {ek}") + need = a.download_bytes(["install", "--no-install-recommends"] + args) + free = a.free_bytes() + if free >= 0 and free < max(MIN_FREE, 3 * need): + raise Refused("R8", f"free space {free} B is below max(500 MB, 3 x download {need} B)") + # 1. the default = the kernel the box RUNS (it booted healthily), proved from grub.cfg BEFORE the install + default_before = self.default_kver(self.grub_cfg()) + self.write_default(old) + # 2. install (the kernel's own postinst runs update-grub; our default file keeps the default on `old`) + t0 = time.time() + rc, out, err = a.x(APT_ENV + ["apt-get", "-y", "-q"] + DPKG_OPTS + ["install", "--no-install-recommends"] + args) + a.x(["apt-get", "clean"]) + if rc != 0: + _, aud, _ = a.x(["dpkg", "--audit"]) + self.report["failed"] = {"rc": rc, "step": "install", "dpkg_audit": (aud.strip().splitlines() or ["clean"])[0], + "tail": (out + err).strip().splitlines()[-3:]} + self.log(f"os-apply: FAILED rc={rc} step=install (the default stays {old}; no flag written)") + return 3 + self.report["upgraded"] = [{"name": x.split("=", 1)[0], "version": x.split("=", 1)[1]} for x in args] + self.report["seconds"] = round(time.time() - t0, 1) + # 3. prove: the image is there, the default is still `old`, the one-shot entry for `new` exists + cfg = self.grub_cfg() + if f"felhom-oneshot-{new}" not in cfg or self.default_kver(cfg) != old: + self.r.host(["update-grub"], 300) + cfg = self.grub_cfg() + for f in (f"/boot/vmlinuz-{new}", f"/boot/initrd.img-{new}"): + if not self.r.lexists(f): + self.report["failed"] = {"rc": 3, "step": "verify", "reason": f"{f} is missing after the install"} + return 3 + if f"felhom-oneshot-{new}" not in cfg or "felhom_next" not in cfg: + self.report["failed"] = {"rc": 3, "step": "verify", "reason": f"grub.cfg has no one-shot entry for {new}"} + return 3 + if self.default_kver(cfg) != old: + self.report["failed"] = {"rc": 3, "step": "verify", + "reason": f"the install moved the default to {self.default_kver(cfg)} — no flag written"} + return 3 + # 4. the flag — the ONLY thing that makes the next boot use `new`, and only once + self.set_flag(new) + self.save_state({"phase": "staged", "step_id": self.plan.get("release_id"), "from": old, "to": new, "vmid": a.vmid, + "staged_at": _iso(self.r.now()), "authority": who, "default_before": default_before, + "packages": self.report["upgraded"], "self_revert_used": False}) + self.report["reboot_needed"] = True + self.log(f"os-apply: KERNEL STAGED {old} -> {new} (default {old}, one-shot flag {new}); upgraded={len(args)} " + f"seconds={self.report['seconds']}") + return 0 + + def reboot_staged(self): + st = self.state() + self.phase_is(st, "staged") + if self.flag() != st.get("to"): + raise Refused("R22", f"the one-shot flag reads {self.flag()!r}, not {st.get('to')!r}") + if self.running() != st.get("from"): + raise Refused("R22", f"the box runs {self.running()!r}, not the step's old kernel {st.get('from')!r}") + cfg = self.grub_cfg() + if self.default_kver(cfg) != st["from"] or f"felhom-oneshot-{st['to']}" not in cfg: + raise Refused("R20", "grub.cfg no longer keeps the old default with a one-shot entry for the new kernel") + if self.setup_problems(): + raise Refused("R20", "; ".join(self.setup_problems())) + self.check_guard() + self.a.check_guest(self.a.vmid) + st["health_before"] = self.a.health() + st["phase"], st["rebooted_at"] = "oneshot", _iso(self.r.now()) + self.save_state(st) + return self.reboot(f"one-shot boot of {st['to']} (the default stays {st['from']})") + + def after_boot(self): + st = self.state() + ph, run = st.get("phase"), self.running() + old, new = st.get("from"), st.get("to") + event = "none" + if ph in ("oneshot", "staged", "judging") and run == new and new: + if ph != "judging": + st["phase"], st["judging_since"], event = "judging", _iso(self.r.now()), "judging" + else: + event = "judging" + elif ph in ("oneshot", "judging") and run == old: + # the new kernel did not come up, or crashed: GRUB already booted the default (the old kernel) + self.clear_flag() + st["phase"], st["result_at"], event = "fell_back", _iso(self.r.now()), "fell_back" + st["reason"] = "the box came back on the old kernel by itself (the new one did not boot, or crashed)" + elif ph == "reverting" and run == old: + st["phase"], st["result_at"], event = "self_reverted", _iso(self.r.now()), "self_reverted" + elif ph == "reverting" and run == new: + st["phase"], st["result_at"], event = "revert_failed", _iso(self.r.now()), "revert_failed" + st["reason"] = "the self-revert came back on the NEW kernel — never retried (one self-revert per step)" + if event not in ("none",) and st.get("phase") != ph: + self.save_state(st) + self.log(f"os-apply: KERNEL AFTER-BOOT {ph} -> {st['phase']} running={run} ({old} -> {new})") + self.report["kernel_event"] = event + if event == "judging": + self.report["health_before"] = st.get("health_before") # the agent judges the boot against it + return 0 + + def good(self): + st = self.state() + self.phase_is(st, "judging") + if self.running() != st.get("to"): + raise Refused("R22", f"the box runs {self.running()!r}, not the new kernel {st.get('to')!r}") + try: + self.write_default(st["to"]) + except Refused: + # put the old default back — the box must never be left without a proved default + self.write_default(st["from"]) + raise + self.clear_flag() + st["phase"], st["result_at"] = "good", _iso(self.r.now()) + self.save_state(st) + self.log(f"os-apply: KERNEL GOOD {st['to']} is the default now (was {st['from']})") + return 0 + + def revert(self): + st = self.state() + self.phase_is(st, "judging") + if st.get("self_revert_used"): + raise Refused("R22", "this kernel step already used its one self-revert") + if self.running() != st.get("to"): + raise Refused("R22", f"the box runs {self.running()!r}, not the new kernel {st.get('to')!r}") + self.clear_flag() + cfg = self.grub_cfg() + if self.default_kver(cfg) != st.get("from"): + raise Refused("R20", f"the GRUB default reads {self.default_kver(cfg)}, not the old {st.get('from')} — " + f"no self-revert into an unknown kernel") + reason = self.plan.get("reason") + st["reason"] = reason[:300] if isinstance(reason, str) else "the one-shot boot was not healthy" + st["self_revert_used"], st["phase"], st["reverted_at"] = True, "reverting", _iso(self.r.now()) + self.save_state(st) + return self.reboot(f"self-revert to {st['from']}: {st['reason']}") + + def cancel(self): + st = self.state() + self.phase_is(st, "staged") + self.clear_flag() + st["phase"], st["result_at"], st["reason"] = "cancelled", _iso(self.r.now()), "cancelled before the reboot" + self.save_state(st) + self.log(f"os-apply: KERNEL CANCELLED {st.get('to')} (installed, never the default; flag cleared)") + return 0 + + class Bundle: """The config bundle (R-840, `11` §5.4.2): every root-owned file the installer's step 5 writes, installed as ONE signed unit. Every check runs before the first write; a failed write or a failed self-check puts every previous diff --git a/configs/test_felhom_crash_guard.py b/configs/test_felhom_crash_guard.py index 120b720..f67f5ef 100644 --- a/configs/test_felhom_crash_guard.py +++ b/configs/test_felhom_crash_guard.py @@ -139,5 +139,62 @@ class Guard(unittest.TestCase): self.assertEqual(mode, 0o644) +class KernelStepCannotLeaveTheBoxOff(unittest.TestCase): + """R-836 / `11` §5.11 Part B 4: a kernel step's planned reboot, one crash and one self-revert cannot add up to the + box staying off. The wrapper starts a step only when the guard is armed with NO unclean boot in its window + (felhom-os-apply Kernel.check_guard, R21); the planned reboot and the self-revert are orderly (`systemctl reboot` — + the clean-stop marker); so the step adds at most ONE unclean boot, and the box stays off only after the LIMIT-th + (3rd) within the hour. Red-proof: audits/kernel-lane-2026-10-07/A/redproof.txt.""" + + def setUp(self): + self.d = tempfile.TemporaryDirectory() + self.e = FakeEnv(self.d.name) + cg.main(["x", "boot"], self.e) + s = self.e.state() + self.assertTrue(s["armed"]) + self.assertEqual(s["unclean_boots_in_window"], 0, "the wrapper's precondition (R21)") + + def tearDown(self): + self.d.cleanup() + + def planned(self, minutes): + cg.main(["x", "clean-stop"], self.e) + self.e.t += minutes * 60 + cg.main(["x", "boot"], self.e) + + def crash(self, minutes): + self.e.t += minutes * 60 + cg.main(["x", "boot"], self.e) + + def test_planned_reboot_one_crash_one_self_revert(self): + self.planned(2) # the step's one-shot reboot (orderly) + self.crash(3) # the new kernel crashes after the guard ran; the box restarts (panic=10) + self.planned(2) # the self-revert (orderly) + s = self.e.state() + self.assertFalse(s["tripped"], s) + self.assertTrue(s["armed"]) + self.assertEqual(self.e.panic(), 10, "the box still restarts after a crash") + self.assertEqual(s["unclean_boots_in_window"], 1, "the step added exactly one unclean boot") + + def test_a_panic_before_userspace_is_not_even_counted(self): + # the one-shot kernel panics before the guard's unit runs (measured: rdinit= and init= missing): the planned + # reboot's clean-stop marker is still there when the old kernel boots, so this boot counts as clean. + cg.main(["x", "clean-stop"], self.e) + self.e.t += 120 # the panicking boot: no userspace, the guard never ran + cg.main(["x", "boot"], self.e) + s = self.e.state() + self.assertEqual(s["unclean_boots_in_window"], 0, s) + self.assertEqual(self.e.panic(), 10) + + def test_the_box_stays_off_only_after_two_more_crashes_than_the_step_makes(self): + self.planned(2) + self.crash(3) # the step's one crash + self.planned(2) # the self-revert + self.crash(5) # an UNRELATED crash within the hour: the guard trips (the 3rd would leave it off) + s = self.e.state() + self.assertTrue(s["tripped"]) + self.assertEqual(s["unclean_boots_in_window"], 2, "two unclean boots: one from the step, one not") + + if __name__ == "__main__": unittest.main() diff --git a/configs/test_felhom_os_apply.py b/configs/test_felhom_os_apply.py index 0b40aa3..9e2b9be 100644 --- a/configs/test_felhom_os_apply.py +++ b/configs/test_felhom_os_apply.py @@ -1476,5 +1476,490 @@ class PVELane(unittest.TestCase): self.assertEqual((rc, rep["refused"]["code"]), (2, "R11"), rep) +# ---------- the kernel lane (R-836, `09` §3 decision 172, `11` §5.11) ---------- +U = "1af1fcc6-639c-416b-a7e5-c4470d41a502" +OLD, NEW = "7.0.2-6-pve", "7.0.14-22-pve" +KERNEL_SET = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-22", "origin": "Proxmox Debian Repository"}, + {"name": "proxmox-kernel-7.0.14-22-pve-signed", "version": "7.0.14-22", "origin": "Proxmox Debian Repository"}] + + +class KFake(Fake): + """A Proxmox host with GRUB on UEFI: /boot on the root LV, a vfat ESP, the bundle's two generators. update-grub is + emulated like the real 10_linux: WITHOUT the felhom default file the NEWEST kernel becomes the default (measured, + R-836 — that is the defect the lane exists for); with it, the kernel it names.""" + + def __init__(self): + super().__init__() + self.running = OLD + self.boot = {OLD} + self.efi, self.esp_fs, self.boot_mount, self.mods, self.snippets, self.pin = True, "vfat", "", True, True, False + self.env = None # None = no env block on the ESP; else {"felhom_next": ...} + self.tree = {} # files put_file wrote + self.reboots = [] + self.update_grubs = 0 + self.grub_runs_in_postinst = True + self.installed.update({"proxmox-kernel-7.0": "7.0.2-6", "proxmox-default-kernel": "2.1.0", + "pve-firmware": "3.18-3", "proxmox-kernel-7.0.2-6-pve-signed": "7.0.2-6", + "pve-manager": "9.2.21"}) + self.live.update({"proxmox-kernel-7.0": {"7.0.14-22", "7.0.2-6"}, + "proxmox-kernel-7.0.14-22-pve-signed": {"7.0.14-22"}, + "proxmox-kernel-7.0.14-23-pve-signed": {"7.0.14-23"}, + "pve-firmware": {"3.18-7", "3.18-3"}}) + self.origins = {} + self.kernel_pending = ["Inst pve-firmware [3.18-3] (3.18-7 Proxmox Debian Repository:stable [all])", + "Inst proxmox-kernel-7.0.14-22-pve-signed (7.0.14-22 Proxmox Debian Repository:stable [amd64])", + "Inst proxmox-kernel-7.0 [7.0.2-6] (7.0.14-22 Proxmox Debian Repository:stable [amd64])", + "Inst pve-manager [9.2.21] (9.2.22 Proxmox Debian Repository:stable [amd64])", + "Inst libc6 [2.41-12+deb13u3] (2.41-12+deb13u4 Debian:13.7/stable [amd64])"] + self.plan = {"release_id": "kernel-t1", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": "apply", + "select": "pending-kernel", "packages": []} + self.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": True}) + self.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": True, "tripped": False, "unclean_boots_in_window": 0}) + self.cfg = self.render() + + # -- GRUB -- + def newest(self): + best = None + for k in self.boot: + if best is None or dpkg_cmp(k[:-4], "gt", best[:-4]): + best = k + return best + + def render(self): + d = self.tree.get(osapply.KERNEL_DEFAULT_CFG) + if d: + dflt = re.search(r'GRUB_DEFAULT="([^"]+)"', d).group(1) + else: + dflt = f"gnulinux-advanced-{U}>gnulinux-{self.newest()}-advanced-{U}" + cfg = ('if [ "${next_entry}" ] ; then\n set default="${next_entry}"\nelse\n' + f' set default="{dflt}"\nfi\n' + f"submenu 'Advanced options' $menuentry_id_option 'gnulinux-advanced-{U}' {{\n") + for k in sorted(self.boot): + cfg += f" menuentry 'Proxmox VE, with Linux {k}' $menuentry_id_option 'gnulinux-{k}-advanced-{U}' {{ }}\n" + cfg += "}\n" + if self.snippets: + cfg += "### BEGIN /etc/grub.d/01_felhom_oneshot ###\nload_env felhom_next\n" + cfg += "".join(f"menuentry 'Felhom one-shot: {k}' --id felhom-oneshot-{k} {{ }}\n" for k in sorted(self.boot)) + return cfg + + def lexists(self, p): + if p == "/sys/firmware/efi": + return self.efi + if p.startswith("/usr/lib/grub/x86_64-efi/"): + return self.mods + if p in (osapply.ONESHOT_SNIPPET, osapply.ONESHOT_ENTRIES): + return self.snippets + if p == osapply.KERNEL_PIN_FILE: + return self.pin + m = re.match(r"^/boot/(vmlinuz|initrd\.img)-(.+)$", p) + if m: + return m.group(2) in self.boot + return p in self.tree + + def put_file(self, path, data, mode): + self.tree[path] = data.decode() + + def read_file(self, p): + if p == osapply.GRUB_CFG: + return self.cfg + if p in self.tree: + return self.tree[p] + return super().read_file(p) + + def host(self, argv, timeout=600, stdin=None): + if argv[0] == "uname": + self.calls.append(("host", argv)) + return 0, self.running + "\n", "" + if argv[0] == "findmnt": + self.calls.append(("host", argv)) + if argv[-1] == osapply.ESP_MOUNT: + return (0, self.esp_fs + "\n", "") if self.esp_fs else (1, "", "") + return (0, self.boot_mount + "\n", "") if self.boot_mount else (1, "", "") + if argv[0] == "grub-editenv": + self.calls.append(("host", argv)) + if argv[2] == "list": + return (1, "", "no such file") if self.env is None else \ + (0, "".join(f"{k}={v}\n" for k, v in self.env.items()), "") + if argv[2] == "create": + self.env = {} + return 0, "", "" + if argv[2] == "set": + k, v = argv[3].split("=", 1) + self.env[k] = v + return 0, "", "" + if argv[2] == "unset": + self.env.pop(argv[3], None) + return 0, "", "" + if argv[0] == "update-grub": + self.calls.append(("host", argv)) + self.update_grubs += 1 + self.cfg = self.render() + return 0, "", "" + if argv[0] == "mkdir": + self.calls.append(("host", argv)) + return 0, "", "" + if argv[:2] == ["systemctl", "reboot"]: + self.calls.append(("host", argv)) + self.reboots.append(self.running) + return 0, "", "" + return super().host(argv, timeout, stdin) + + def emulate(self, argv): + a = [x for x in argv if not re.match(r"^[A-Z_]+=", x) and x != "env"] + if a[0] == "apt-get" and "install" in a and "-s" not in a and "--print-uris" not in a and "-f" not in a: + if self.install_rc: + return self.install_rc, "", "E: boom" + for x in a: + if "=" in x and not x.startswith("-") and "::" not in x: + n, v = x.split("=", 1) + self.installed[n] = v + m = re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n) + if m: + self.installed[f"proxmox-kernel-{v}-pve-signed"] = v + if m or osapply.KERNEL_IMAGE_RE.match(n): + self.boot.add(v + "-pve") + if self.grub_runs_in_postinst: + self.cfg = self.render() # the kernel's postinst runs update-grub (zz-update-grub) + return 0, "Setting up proxmox-kernel ...\n", "" + return super().emulate(argv) + + def sim(self, a): + if "--print-uris" in a: + return 0, "", "" + if "dist-upgrade" in a: + return 0, "\n".join(self.kernel_pending) + "\n", "" + out, named = "", set() + for x in a: + if "=" in x and not x.startswith("-") and "::" not in x: + named.add(x.split("=", 1)[0]) + for x in a: + if "=" in x and not x.startswith("-") and "::" not in x: + n, v = x.split("=", 1) + if v not in self.avail(n): + return 100, "", f"E: Version '{v}' for '{n}' was not found" + o = self.origins.get(n, PVE) + out += f"Inst {n} [{self.installed[n]}] ({v} {o} [amd64])\n" if n in self.installed else f"Inst {n} ({v} {o} [amd64])\n" + if re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n): + img = f"proxmox-kernel-{v}-pve-signed" + if img not in self.installed and img not in named: + out += f"Inst {img} ({v} {PVE} [amd64])\n" + out += "".join(l + "\n" for l in self.extra_sim) + return 0, out, "" + + +def kfake(**kw): + f = KFake() + for k, v in kw.items(): + setattr(f, k, v) + return f + + +def kmode(f, mode, **extra): + f.plan = {"release_id": "kernel-t1", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": mode, "packages": []} + f.plan.update(extra) + return run(f) + + +def staged(f=None): + f = f or kfake() + rc, rep = run(f) + assert rc == 0, rep + return f + + +class KernelLane(unittest.TestCase): + """R-836 / `09` §3 decision 172: stage a kernel once through the ESP flag; the default moves only after a healthy + one-shot boot. Red-proof: audits/kernel-lane-2026-10-07/A/redproof.txt.""" + + def refused(self, f, code, mode=None, **extra): + rc, rep = kmode(f, mode, **extra) if mode else run(f) + self.assertEqual(rc, 2, rep) + self.assertEqual(rep["refused"]["code"], code, rep) + return rep + + # --- the four red tests the brief names (Part A 3) --- + def test_installing_a_kernel_does_not_change_the_grub_default(self): + f = kfake() + rc, rep = run(f) + self.assertEqual(rc, 0, rep) + self.assertIn(NEW, f.boot, "the new kernel was installed") + self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD, + "the default must stay the kernel the box runs (R-836: an install made the new one default)") + self.assertIn(f"gnulinux-{OLD}-advanced-{U}", f.tree[osapply.KERNEL_DEFAULT_CFG]) + self.assertEqual(f.env, {"felhom_next": NEW}, "the ONLY way to the new kernel is the one-shot flag") + self.assertEqual(f.reboots, [], "staging never reboots") + st = json.loads(f.tree[osapply.KERNEL_STATE]) + self.assertEqual((st["phase"], st["from"], st["to"]), ("staged", OLD, NEW)) + self.assertEqual(rep["kernel"]["default"], OLD) + + def test_a_box_without_the_esp_flag_is_refused(self): + for attr, val, frag in (("esp_fs", "ext4", "vfat"), ("efi", False, "UEFI"), ("snippets", False, "bundle"), + ("mods", False, "module"), ("boot_mount", "/boot", "separate"), ("pin", True, "pinned")): + f = kfake(**{attr: val}) + rep = self.refused(f, "R20") + self.assertIn(frag, rep["refused"]["reason"], attr) + self.assertNotIn(NEW, f.boot, f"{attr}: nothing may be installed on a refusal") + self.assertIsNone(f.env, f"{attr}: no flag on a refusal") + + def test_a_reboot_without_the_flag_is_refused(self): + f = staged() + f.env = {"felhom_next": ""} + self.refused(f, "R22", mode="kernel-reboot") + self.assertEqual(f.reboots, []) + + def test_a_kernel_outside_the_approved_set_is_refused(self): + # a signed set that names only the meta-package: the image the sources pull in was never signed for + f = kfake() + f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False}) + f.plan.update(select="listed", packages=[dict(KERNEL_SET[0])], + signed=signed_job(packages=[KERNEL_SET[0]], op="os_kernel_step")) + self.refused(f, "R23") + self.assertNotIn(NEW, f.boot) + # a signed set names 7.0.14-22; the sources would ALSO bring 7.0.14-23 + f2 = kfake() + f2.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False}) + f2.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET], + signed=signed_job(packages=KERNEL_SET, op="os_kernel_step")) + f2.extra_sim = ["Inst proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23 Proxmox Debian Repository:stable [amd64])"] + self.refused(f2, "R23") + # a name that is not a kernel package at all + g = kfake() + g.plan.update(select="listed", packages=[{"name": "pve-manager", "version": "9.2.22", "origin": "Proxmox Debian Repository"}]) + self.refused(g, "R23") + # the household was told about another kernel + h = kfake() + h.plan["expect_kver"] = "7.0.14-23-pve" + self.refused(h, "R23") + self.assertNotIn(NEW, h.boot) + + def test_the_snippet_clears_the_flag_on_use(self): + """01_felhom_oneshot: the flag is copied, CLEARED and SAVED before any `set default`, all inside the one branch + that runs only when the flag is set; a default is set only for an installed kernel's own entry.""" + text = (HERE / "felhom-grub-oneshot.sh").read_text() + body = text[text.index("cat <gnulinux-9.9.9-1-pve-advanced-{U}"\n' + f.cfg = f.render() + self.refused(f, "R20", mode="kernel-revert") + self.assertEqual(len(f.reboots), 1) + + def test_cancel_clears_the_flag(self): + f = staged() + rc, rep = kmode(f, "kernel-cancel") + self.assertEqual(rc, 0, rep) + self.assertFalse(f.env.get("felhom_next"), "the flag is gone") + self.assertEqual(rep["kernel"]["phase"], "cancelled") + + def test_status_is_read_only_and_names_setup_problems(self): + f = kfake(esp_fs="ext4") + rc, rep = kmode(f, "kernel-status") + self.assertEqual(rc, 0, rep) + self.assertTrue(rep["kernel"]["setup_problems"]) + self.assertEqual(f.tree, {}) + self.assertFalse([c for c in f.calls if c[1][0] in ("update-grub", "systemctl", "apt-get")]) + + def test_facts_carry_the_kernel_lane(self): + f = staged() + f.plan = {"release_id": "facts", "layer": "host", "lane": "fast", "vmid": 9201, "mode": "facts", "packages": []} + rc, rep = run(f) + self.assertEqual(rc, 0, rep) + h = rep["facts"]["host"] + self.assertEqual(h["kernel_lane"]["phase"], "staged") + self.assertEqual(h["kernel_next_boot"], NEW) + self.assertIn("one-shot", h["kernel_next_boot_source"]) + + if __name__ == "__main__": unittest.main() diff --git a/internal/hub/report.go b/internal/hub/report.go index 2dbc971..aed7fec 100644 --- a/internal/hub/report.go +++ b/internal/hub/report.go @@ -636,6 +636,17 @@ type WireOSUpdate struct { // HostRelease is the newest approved HOST release (hub v0.131.0, `11` §8 step 3) — a separate set: a version // approved for the guest is not approved for the host by that fact alone. HostRelease *WireOSRelease `json:"host_release,omitempty"` + // Kernel is the kernel lane's instruction for THIS box (R-836, `09` §3 decision 172, `11` §5.11): the kernel the + // household was told about, and whether tonight is a told night. Nil (an older hub, or no kernel due) = no kernel + // step. The hub sets Tonight only after the household's mail the day before went out — no mail, no step. + Kernel *WireKernelStep `json:"kernel,omitempty"` +} + +// WireKernelStep is the hub's kernel-lane instruction (hub osupdates.KernelBlock — field-exact, cross-repo). +type WireKernelStep struct { + Kver string `json:"kver"` // e.g. "7.0.14-22-pve" — the wrapper refuses any other (R23) + Tonight bool `json:"tonight"` // the household was mailed the day before: tonight's leg may reboot + NotifiedAt string `json:"notified_at,omitempty"` // when that mail went out (RFC 3339), for the log } // WireOSRelease is an approved version set; Snapshot is the approval time (YYYYMMDDTHHMMSSZ) the wrapper uses diff --git a/internal/osupdate/kernel.go b/internal/osupdate/kernel.go new file mode 100644 index 0000000..bbbcdd9 --- /dev/null +++ b/internal/osupdate/kernel.go @@ -0,0 +1,409 @@ +package osupdate + +// The kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11). The root half is felhom-os-apply's layer "kernel" +// (configs/, its own tests); this file decides WHEN and judges the boot: +// +// - the night leg (Run → runKernel): after a healthy host step, on a night the hub marks as told (the household was +// mailed the day before — no mail, no step): ring 0 STAGES the pending kernel (select pending-kernel, the root-owned +// ring-0 mark) and reboots; ring 1 reboots only a step a signed os_kernel_step staged earlier (KernelStepExecutor). +// - after a boot (KernelAfterBoot, at daemon start): the wrapper says what became of the step. On the new kernel the +// agent JUDGES the boot — the host health rule (`11` §8.2: the Proxmox daemons, the guest running and healthy, the +// tunnel) AND the box reaching the hub — for KernelJudgeWait. Healthy → kernel-good (the new kernel becomes the +// default). Not healthy by the deadline → ONE self-revert (kernel-revert: a reboot into the old kernel, still the +// default). A crash on the new kernel needs nothing from the agent: GRUB already boots the old default. +// +// The host is rebooted by this file only through the wrapper (kernel-reboot, kernel-revert), and only for a staged step. + +import ( + "context" + "encoding/base64" + "encoding/json" + "fmt" + "log/slog" + "regexp" + "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/hub" + "gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs" +) + +// OpKernelStep is the signed op class that STAGES a kernel set on a ring-1 box (it never reboots: the night leg does, +// once the household was told). CC may sign it until the first paying customer (R-530 ruling). +const OpKernelStep = "os_kernel_step" + +// DefaultKernelJudgeWait is how long a one-shot boot may take to come back healthy before the agent reverts it ONCE. +// Measured 2026-10-07 (`audits/kernel-lane-2026-10-07/B/`): every container healthy 68 s after a reboot on demo-felhom, +// 272 s on demo-hp; 20 minutes stays under the hub's 30-minute host_stale (a box that never comes back alarms after it). +const DefaultKernelJudgeWait = 20 * time.Minute + +var kverRE = regexp.MustCompile(`^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$`) + +// KernelView is the wrapper's kernel object (felhom-os-apply Kernel.view). +type KernelView struct { + Running string `json:"running"` + Default string `json:"default"` + Flag *string `json:"flag"` + Phase string `json:"phase"` + From string `json:"from"` + To string `json:"to"` + SelfRevertUsed bool `json:"self_revert_used"` + Reason string `json:"reason"` + VMID int `json:"vmid"` // the customer guest the step was staged for (the health rule's guest) +} + +func parseKernel(raw json.RawMessage) KernelView { + var v KernelView + _ = json.Unmarshal(raw, &v) + return v +} + +// kernelPlan is one kernel-layer wrapper call. +func kernelPlan(mode string, vmid int, extra map[string]any) map[string]any { + p := map[string]any{"release_id": "kernel", "layer": LayerKernel, "lane": "slow", "vmid": vmid, "mode": mode, + "packages": []Package{}} + for k, v := range extra { + p[k] = v + } + return p +} + +// KernelStatus reads the kernel lane's state (read only). +func (l *Leg) KernelStatus(ctx context.Context, vmid int) (KernelView, error) { + wr, err := l.call(ctx, "kstatus"+l.now().UTC().Format("150405"), kernelPlan("kernel-status", vmid, nil)) + if err != nil { + return KernelView{}, err + } + if wr.refused() { + return KernelView{}, fmt.Errorf("kernel-status refused: %s", wr.Refused) + } + return parseKernel(wr.Kernel), nil +} + +// kernelDue reports whether tonight's leg may take a kernel step, and why not. +func kernelDue(blk hub.WireOSUpdate, trigger string) (bool, string) { + switch { + case trigger != "night": + return false, "a kernel step runs only in the night leg (never a debug pass)" + case blk.Kernel == nil: + return false, "the hub names no kernel step for this box" + case !blk.Enabled: + return false, "OS updates are switched off for this box" + case !kverRE.MatchString(blk.Kernel.Kver): + return false, "the hub's kernel " + blk.Kernel.Kver + " is not a kernel version" + case !blk.Kernel.Tonight: + return false, "the household has not been told about tonight (no mail, no step — `09` §3 decision 172)" + } + return true, "" +} + +// runKernel is the night leg's last step. It returns the stage report (Layer "" when nothing ran). On success the box +// is rebooting when it returns. +func (l *Leg) runKernel(ctx context.Context, runID string, vmid int, trigger string, blk hub.WireOSUpdate) Report { + lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid, "trigger", trigger, "ring", blk.Ring) + if ok, why := kernelDue(blk, trigger); !ok { + lg.Info("osupdate: kernel step skipped — " + why) + return Report{} + } + want := blk.Kernel.Kver + st, err := l.KernelStatus(ctx, vmid) + if err != nil { + return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid, + Mode: "apply", Outcome: "failed", HealthReason: "kernel status unreadable: " + err.Error()}) + } + rep := Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid, Mode: "apply", ReleaseID: want} + switch { + case st.Phase == "staged" && st.To == want: + lg.Info("osupdate: kernel step — a staged kernel waits for tonight", "from", st.From, "to", st.To) + rep.Outcome, rep.Healthy = "staged", true + case blk.Ring != 0: + lg.Info("osupdate: kernel step skipped — ring 1 boots only a kernel a signed os_kernel_step staged", "phase", st.Phase, "staged", st.To, "want", want) + return Report{} + default: + wr, cerr := l.call(ctx, runID, kernelPlan("apply", vmid, map[string]any{"release_id": "ring0-" + runID, + "select": "pending-kernel", "expect_kver": want, "run_id": runID, "trigger": trigger, "ring": blk.Ring})) + rep.unsent = reportFile(l.planDir(), runID, LayerKernel, "apply") + rep.Kernel = rawOrNil(wr.Kernel) + switch { + case cerr != nil: + rep.Outcome, rep.HealthReason = "failed", cerr.Error() + return l.finish(ctx, lg, rep) + case wr.refused(): + rep.Outcome, rep.Refused = "refused", wr.Refused + return l.finish(ctx, lg, rep) + case wr.failed(): + rep.Outcome, rep.Refused = "failed", wr.Failed + return l.finish(ctx, lg, rep) + case wr.OutcomeHint == "nothing" || len(wr.Upgraded) == 0: + rep.Outcome, rep.Healthy = "nothing", true + return l.finish(ctx, lg, rep) + } + rep.Outcome, rep.Healthy, rep.Upgraded, rep.Authority, rep.PassSeconds = "staged", true, wr.Upgraded, wr.Authority, wr.PassSeconds + rep.RebootNeeded = true + } + rep = l.finish(ctx, lg, rep) // the hub hears "staged" BEFORE the box goes down + wr, err := l.call(ctx, runID, kernelPlan("kernel-reboot", vmid, nil)) + switch { + case err != nil: + return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid, + Mode: "kernel-reboot", ReleaseID: want, Outcome: "failed", HealthReason: "kernel-reboot: " + err.Error()}) + case wr.refused() || wr.failed(): + return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid, + Mode: "kernel-reboot", ReleaseID: want, Outcome: "refused", Refused: firstRaw(wr.Refused, wr.Failed), + Kernel: rawOrNil(wr.Kernel)}) + } + lg.Warn("osupdate: kernel step — the box restarts now for its one-shot boot", "to", want) + return rep +} + +func firstRaw(a, b json.RawMessage) json.RawMessage { + if r := rawOrNil(a); r != nil { + return r + } + return rawOrNil(b) +} + +// KernelJudge is what KernelAfterBoot needs besides the leg: the hub reachability probe is the "judging" report itself. +type KernelJudge struct { + Wait time.Duration // default DefaultKernelJudgeWait + Poll time.Duration // default 30 s +} + +// KernelAfterBoot runs once at daemon start: what became of a kernel step across the boot. On the new kernel it judges +// the boot (blocking up to the wait — run it in a goroutine). vmid 0 = the guest the step recorded (it may not run yet). +func (l *Leg) KernelAfterBoot(ctx context.Context, vmid int, j KernelJudge) Report { + runID := "boot-" + l.now().UTC().Format("20060102T150405Z") + lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid) + wr, err := l.call(ctx, runID, kernelPlan("kernel-boot", vmid, nil)) + if err != nil { + lg.Warn("osupdate: kernel after-boot check failed", "err", err) + return Report{} + } + if wr.refused() { + lg.Info("osupdate: kernel after-boot check refused (an older wrapper, or a BYO host)", "refused", string(wr.Refused)) + return Report{} + } + v := parseKernel(wr.Kernel) + if vmid <= 0 { + vmid = v.VMID // after a boot the guest may not run yet — the step's own record names it + } + rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid, Mode: "kernel-boot", + ReleaseID: v.To, Kernel: rawOrNil(wr.Kernel)} + switch wr.KernelEvent { + case "fell_back": + rep.Outcome, rep.HealthReason = "fell_back", v.Reason + lg.Warn("osupdate: kernel step FELL BACK — the new kernel did not come up; the box runs the old one", "from", v.From, "to", v.To) + return l.finish(ctx, lg, rep) + case "self_reverted": + rep.Outcome, rep.HealthReason = "self_reverted", v.Reason + lg.Warn("osupdate: kernel step SELF-REVERTED — back on the old kernel", "from", v.From, "to", v.To, "reason", v.Reason) + return l.finish(ctx, lg, rep) + case "revert_failed": + rep.Outcome, rep.HealthReason = "revert_failed", v.Reason + lg.Error("osupdate: kernel self-revert came back on the NEW kernel — no second revert; the operator decides", "to", v.To) + return l.finish(ctx, lg, rep) + case "judging": + return l.judgeKernel(ctx, runID, vmid, v, wr.HealthBefore, j, lg) + } + return Report{} +} + +// KernelVerdict is THE one-shot boot rule (R-836; pinned by TestKernelVerdict): the host health rule (`11` §8.2 — +// the Proxmox daemons and the agent active, the customer guest running and its own rule passing, the tunnel running) +// AND the box reached the hub since this boot. +func KernelVerdict(before, after *Health, tunnel string, hubReached bool) (bool, string) { + if ok, why := HostHealthVerdict(before, after, tunnel); !ok { + return false, why + } + if !hubReached { + return false, "the box has not reached the hub since the boot" + } + return true, "" +} + +func (l *Leg) judgeKernel(ctx context.Context, runID string, vmid int, v KernelView, before *Health, j KernelJudge, lg *slog.Logger) Report { + wait, poll := j.Wait, j.Poll + if wait <= 0 { + wait = DefaultKernelJudgeWait + } + if poll <= 0 { + poll = 30 * time.Second + } + lg.Info("osupdate: kernel step — judging the one-shot boot", "from", v.From, "to", v.To, "wait", wait.String()) + start := l.now() + deadline := start.Add(wait) + hubReached := false + var why string + for { + if !hubReached && l.Hub != nil { + // the hub's reachability IS this report reaching it (and the operator sees the box is back on the new kernel) + body, _ := json.Marshal(Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid, + Mode: "kernel-boot", ReleaseID: v.To, Outcome: "judging", Kernel: mustRaw(v)}) + rctx, cancel := context.WithTimeout(ctx, 30*time.Second) + if err := l.Hub.PostOSReport(rctx, body); err == nil { + hubReached = true + lg.Info("osupdate: kernel step — the box reached the hub on the new kernel", "after", l.now().Sub(start).Round(time.Second).String()) + } + cancel() + } + var h *Health + hr, err := l.call(ctx, runID, kernelPlan("health", vmid, nil)) + switch { + case err != nil: + why = "no health reading: " + err.Error() + case hr.refused(): + why = "no health reading: " + string(hr.Refused) + default: + h = hr.Health + } + if h != nil { + t := hub.TunnelUnknown + if l.Tunnel != nil { + t, _ = l.Tunnel.Status(ctx) + } + var ok bool + ok, why = KernelVerdict(before, h, t, hubReached) + if ok { + return l.kernelGood(ctx, runID, vmid, v, start, lg) + } + } + if !l.now().Before(deadline) || ctx.Err() != nil { + break + } + l.sleep(ctx, poll) + } + if ctx.Err() != nil { + lg.Warn("osupdate: kernel judging stopped (the agent is stopping) — the next start judges again", "reason", why) + return Report{} + } + // not healthy by the deadline: tell the hub (best effort), then ONE self-revert into the old kernel + rep := l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid, + Mode: "kernel-revert", ReleaseID: v.To, Outcome: "health_failed", HealthReason: why + " — reverting to " + v.From, + Kernel: mustRaw(v)}) + lg.Error("osupdate: kernel step — the one-shot boot is NOT healthy; restarting ONCE into the old kernel", "reason", why, + "waited", wait.String(), "from", v.From, "to", v.To) + wr, err := l.call(ctx, runID, kernelPlan("kernel-revert", vmid, map[string]any{"reason": truncate(why, 280)})) + if err != nil || wr.refused() || wr.failed() { + lg.Error("osupdate: kernel self-revert did not start — the box stays on the new kernel; the operator decides", + "err", err, "refused", string(firstRaw(wr.Refused, wr.Failed))) + return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid, + Mode: "kernel-revert", ReleaseID: v.To, Outcome: "revert_failed", Refused: firstRaw(wr.Refused, wr.Failed), + HealthReason: "the self-revert did not start"}) + } + return rep +} + +func (l *Leg) kernelGood(ctx context.Context, runID string, vmid int, v KernelView, start time.Time, lg *slog.Logger) Report { + wr, err := l.call(ctx, runID, kernelPlan("kernel-good", vmid, nil)) + rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid, Mode: "kernel-good", + ReleaseID: v.To} + switch { + case err != nil: + rep.Outcome, rep.HealthReason = "failed", "kernel-good: "+err.Error() + case wr.refused() || wr.failed(): + rep.Outcome, rep.Refused, rep.HealthReason = "failed", firstRaw(wr.Refused, wr.Failed), "kernel-good did not move the default" + default: + rep.Outcome, rep.Healthy = "applied", true + rep.HealthReason = fmt.Sprintf("healthy %s after the agent started; the new kernel is the default", l.now().Sub(start).Round(time.Second)) + } + rep.Kernel = rawOrNil(wr.Kernel) + lg.Info("osupdate: kernel step — "+rep.Outcome, "to", v.To, "reason", rep.HealthReason) + return l.finish(ctx, lg, rep) +} + +func mustRaw(v any) json.RawMessage { + b, _ := json.Marshal(v) + return b +} + +func truncate(s string, n int) string { + if len(s) <= n { + return s + } + return s[:n] +} + +// KernelStepParams are a signed os_kernel_step's params: the exact kernel set (the wrapper compares it with the plan). +type KernelStepParams struct { + ReleaseID string `json:"release_id"` + Packages []Package `json:"packages"` + Kver string `json:"kver"` + VMID int `json:"vmid,omitempty"` +} + +// KernelStepExecutor STAGES a verified os_kernel_step (signedjobs.Executor) under the heavy-op gate. It never reboots: +// the night leg reboots a staged kernel on a night the household was told about. +type KernelStepExecutor struct { + Leg *Leg + Guest func(ctx context.Context) (int, error) + Gate func(ctx context.Context) (release func(), err error) +} + +// Execute implements signedjobs.Executor. +func (e KernelStepExecutor) Execute(ctx context.Context, op string, params json.RawMessage) error { + if op != OpKernelStep { + return signedjobs.ErrNoExecutor + } + so, ok := signedjobs.SignedOpFrom(ctx) + if !ok { + return fmt.Errorf("os_kernel_step: no signed envelope in the context — the wrapper could not verify it") + } + var p KernelStepParams + if err := json.Unmarshal(params, &p); err != nil || len(p.Packages) == 0 || !kverRE.MatchString(p.Kver) { + return fmt.Errorf("os_kernel_step: params must name the kernel set and its kver: %v", err) + } + vmid := p.VMID + if vmid == 0 { + if e.Guest == nil { + return fmt.Errorf("os_kernel_step: no vmid and no guest finder") + } + v, err := e.Guest(ctx) + if err != nil { + return fmt.Errorf("os_kernel_step: find the customer guest: %w", err) + } + vmid = v + } + if e.Gate != nil { + release, err := e.Gate(ctx) + if err != nil { + return fmt.Errorf("os_kernel_step: heavy-op gate busy (a backup or restore-test runs): %w", err) + } + defer release() + } + rep := e.Leg.StageKernelSigned(ctx, vmid, p, so.Blob, string(so.Sig)) + if rep.Outcome == "staged" || rep.Outcome == "nothing" { + return nil + } + return fmt.Errorf("os_kernel_step: %s (%s) %s", rep.Outcome, rep.HealthReason, string(rep.Refused)) +} + +// StageKernelSigned stages a signed kernel set (ring 1): install + flag, no reboot. +func (l *Leg) StageKernelSigned(ctx context.Context, vmid int, p KernelStepParams, blob []byte, sig string) Report { + unlock := l.lockPass(true) + defer unlock() + l.sendUnsentLocked(ctx) // R-868 + runID := l.now().UTC().Format("20060102T150405Z") + rid := p.ReleaseID + if rid == "" { + rid = "signed-" + runID + } + lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid, "trigger", "signed", "release", rid) + wr, err := l.call(ctx, runID, kernelPlan("apply", vmid, map[string]any{"release_id": rid, "select": "listed", + "packages": p.Packages, "expect_kver": p.Kver, "run_id": runID, "trigger": "signed", "ring": l.Block().Ring, + "signed": map[string]string{"blob_b64": base64.StdEncoding.EncodeToString(blob), "sig": sig}})) + rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "signed", Ring: l.Block().Ring, VMID: vmid, Mode: "apply", + ReleaseID: rid, Kernel: rawOrNil(wr.Kernel), unsent: reportFile(l.planDir(), runID, LayerKernel, "apply")} + switch { + case err != nil: + rep.Outcome, rep.HealthReason = "failed", err.Error() + case wr.refused(): + rep.Outcome, rep.Refused = "refused", wr.Refused + case wr.failed(): + rep.Outcome, rep.Refused = "failed", wr.Failed + case wr.OutcomeHint == "nothing" || len(wr.Upgraded) == 0: + rep.Outcome, rep.Healthy = "nothing", true + default: + rep.Outcome, rep.Healthy, rep.Upgraded, rep.Authority, rep.PassSeconds = "staged", true, wr.Upgraded, wr.Authority, wr.PassSeconds + rep.RebootNeeded = true + } + return l.finish(ctx, lg, rep) +} diff --git a/internal/osupdate/kernel_test.go b/internal/osupdate/kernel_test.go new file mode 100644 index 0000000..6e25f1a --- /dev/null +++ b/internal/osupdate/kernel_test.go @@ -0,0 +1,314 @@ +package osupdate + +import ( + "context" + "encoding/base64" + "encoding/json" + "strings" + "testing" + "time" + + "gitea.dooplex.hu/admin/felhom-agent/internal/hub" + "gitea.dooplex.hu/admin/felhom-agent/internal/reconcile" + "gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs" +) + +// ---- the kernel lane (R-836, `09` §3 decision 172, `11` §5.11) ---- + +const kOld, kNew = "7.0.2-6-pve", "7.0.14-22-pve" + +func kview(phase string) json.RawMessage { + return mustRaw(KernelView{Running: kOld, Default: kOld, Phase: phase, From: kOld, To: kNew, VMID: 9201}) +} + +func tonight(ring int) *hub.WireOSUpdate { + return &hub.WireOSUpdate{Ring: ring, Enabled: true, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: true}} +} + +func kernelCalls(w *fakeWrapper) []string { + var m []string + for _, p := range w.plans { + if p["layer"] == LayerKernel { + m = append(m, p["mode"].(string)) + } + } + return m +} + +// Ring 0, a told night: after the healthy host step the leg stages the pending kernel (select pending-kernel, the +// kernel the household was told about), tells the hub "staged", THEN reboots — the kernel step ends the night. +func TestKernel_Ring0ToldNightStagesThenReboots(t *testing.T) { + w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{ + "kernel-status": {{Kernel: kview("none")}}, + "apply": {{Upgraded: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22"}}, Authority: "ring0", Kernel: kview("staged")}}, + "kernel-reboot": {{Kernel: kview("oneshot")}}, + }} + l, h := newLeg(t, w, tonight(0)) + p := l.Run(context.Background(), 9201, "night") + if got := strings.Join(kernelCalls(w), ","); got != "kernel-status,apply,kernel-reboot" { + t.Fatalf("kernel calls = %s", got) + } + var ap map[string]any + for _, x := range w.plans { + if x["layer"] == LayerKernel && x["mode"] == "apply" { + ap = x + } + } + if ap["select"] != "pending-kernel" || ap["expect_kver"] != kNew || ap["lane"] != "slow" { + t.Fatalf("stage plan = %v", ap) + } + if p.Kernel.Outcome != "staged" || !p.Kernel.Healthy { + t.Fatalf("kernel report = %+v", p.Kernel) + } + last := h.reports[len(h.reports)-1] + if last.Layer != LayerKernel || last.Outcome != "staged" { + t.Fatalf("the hub must hear 'staged' before the reboot: %+v", h.reports) + } + // the kernel step is the LAST wrapper call of the night + if lp := w.plans[len(w.plans)-1]; lp["layer"] != LayerKernel || lp["mode"] != "kernel-reboot" { + t.Fatalf("the reboot must end the night, last call = %v", lp) + } +} + +// No mail, no step: a kernel the household was NOT told about never runs; nor in a debug pass; nor without a block. +// COMPANION RED-PROOF (observed): drop the `!blk.Kernel.Tonight` case in kernelDue → the first sub-case fails. +func TestKernel_NoMailNoStep(t *testing.T) { + cases := map[string]struct { + blk *hub.WireOSUpdate + trigger string + }{ + "not told": {&hub.WireOSUpdate{Ring: 0, Enabled: true, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: false}}, "night"}, + "debug pass": {tonight(0), "debug"}, + "no block": {&hub.WireOSUpdate{Ring: 0, Enabled: true}, "night"}, + "switch off": {&hub.WireOSUpdate{Ring: 0, Enabled: false, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: true}}, "night"}, + "bad kver": {&hub.WireOSUpdate{Ring: 0, Enabled: true, Kernel: &hub.WireKernelStep{Kver: "7.0; reboot", Tonight: true}}, "night"}, + } + for name, c := range cases { + w := &fakeWrapper{t: t} + l, _ := newLeg(t, w, c.blk) + p := l.Run(context.Background(), 9201, c.trigger) + if len(kernelCalls(w)) != 0 || p.Kernel.Layer != "" { + t.Fatalf("%s: a kernel step ran: %v", name, kernelCalls(w)) + } + } +} + +// The kernel step needs a healthy host step on an appliance, and a healthy Proxmox step when one ran. +func TestKernel_SkippedWithoutHealthyEarlierSteps(t *testing.T) { + w := &fakeWrapper{t: t} + l, _ := newLeg(t, w, tonight(0)) + l.Appliance = false + l.Run(context.Background(), 9201, "night") + if len(kernelCalls(w)) != 0 { + t.Fatalf("a BYO box took a kernel step: %v", kernelCalls(w)) + } + w2 := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerPVE: {Upgraded: []Package{{Name: "pve-manager", Version: "9.2.21"}}, + PVEManager: "9.2.2"}}} // pveversion still old → the pve step is unhealthy + l2, _ := newLeg(t, w2, tonight(0)) + l2.Run(context.Background(), 9201, "night") + if len(kernelCalls(w2)) != 0 { + t.Fatalf("a kernel step ran after an unhealthy Proxmox step: %v", kernelCalls(w2)) + } +} + +// Ring 1 reboots only a kernel a signed job staged — never stages one itself in the night leg. +func TestKernel_Ring1RebootsOnlyASignedStage(t *testing.T) { + w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}}}} + l, _ := newLeg(t, w, tonight(1)) + l.Run(context.Background(), 9201, "night") + if got := strings.Join(kernelCalls(w), ","); got != "kernel-status" { + t.Fatalf("ring 1 without a staged kernel: calls = %s", got) + } + w2 := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("staged")}}, + "kernel-reboot": {{Kernel: kview("oneshot")}}}} + l2, _ := newLeg(t, w2, tonight(1)) + p := l2.Run(context.Background(), 9201, "night") + if got := strings.Join(kernelCalls(w2), ","); got != "kernel-status,kernel-reboot" || p.Kernel.Outcome != "staged" { + t.Fatalf("ring 1 with a staged kernel: calls = %s report = %+v", got, p.Kernel) + } +} + +// A refused stage never reboots. +func TestKernel_RefusedStageNeverReboots(t *testing.T) { + w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}}, + "apply": {{Refused: json.RawMessage(`{"code":"R20","reason":"/boot/efi is not a mounted vfat ESP"}`)}}}} + l, h := newLeg(t, w, tonight(0)) + p := l.Run(context.Background(), 9201, "night") + if got := strings.Join(kernelCalls(w), ","); got != "kernel-status,apply" || p.Kernel.Outcome != "refused" { + t.Fatalf("calls = %s report = %+v", got, p.Kernel) + } + if last := h.reports[len(h.reports)-1]; last.Layer != LayerKernel || last.Outcome != "refused" { + t.Fatalf("the hub must hear the refusal: %+v", last) + } +} + +// THE one-shot boot rule: the host rule AND the hub reached. COMPANION RED-PROOF (observed): drop the hubReached +// check in KernelVerdict → the second case fails. +func TestKernelVerdict(t *testing.T) { + if ok, why := KernelVerdict(hostOK(), hostOK(), hub.TunnelRunning, true); !ok { + t.Fatalf("a healthy boot read unhealthy: %s", why) + } + if ok, why := KernelVerdict(hostOK(), hostOK(), hub.TunnelRunning, false); ok || !strings.Contains(why, "hub") { + t.Fatalf("a box that has not reached the hub must not pass: ok=%v %q", ok, why) + } + down := hostOK() + down.GuestRunning = new(bool) + if ok, _ := KernelVerdict(hostOK(), down, hub.TunnelRunning, true); ok { + t.Fatal("a guest that does not run must fail") + } + if ok, _ := KernelVerdict(hostOK(), hostOK(), hub.TunnelUnknown, true); ok { + t.Fatal("an unknown tunnel must fail (the host rule)") + } +} + +func judgingLeg(t *testing.T, w *fakeWrapper) (*Leg, *fakeHub) { + if w.kernelRep == nil { + w.kernelRep = map[string][]WrapperReport{} + } + if _, ok := w.kernelRep["kernel-boot"]; !ok { + w.kernelRep["kernel-boot"] = []WrapperReport{{KernelEvent: "judging", Kernel: mustRaw(KernelView{Running: kNew, + Default: kOld, Phase: "judging", From: kOld, To: kNew, VMID: 9201}), HealthBefore: hostOK()}} + } + return newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true}) +} + +// A healthy one-shot boot: the hub hears "judging", then kernel-good, then "applied". +func TestKernelAfterBoot_HealthyBecomesTheDefault(t *testing.T) { + w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-good": {{Kernel: kview("good")}}}} + l, h := judgingLeg(t, w) + r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 10 * time.Minute, Poll: 30 * time.Second}) + if got := strings.Join(kernelCalls(w), ","); got != "kernel-boot,health,kernel-good" { + t.Fatalf("calls = %s", got) + } + if r.Outcome != "applied" || !r.Healthy { + t.Fatalf("report = %+v", r) + } + if len(h.reports) != 2 || h.reports[0].Outcome != "judging" || h.reports[1].Outcome != "applied" { + t.Fatalf("hub reports = %+v", h.reports) + } + if w.plans[1]["vmid"] != float64(9201) { + t.Fatalf("the health reading must use the step's own guest, got %v", w.plans[1]["vmid"]) + } +} + +// An unhealthy one-shot boot: wait the full judge time, tell the hub, then ONE kernel-revert. +// COMPANION RED-PROOF (observed): return before the kernel-revert call in judgeKernel → "calls" fails. +func TestKernelAfterBoot_UnhealthyRevertsOnceAfterTheWait(t *testing.T) { + down := hostOK() + down.GuestRunning = new(bool) + w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"health": {{Health: down}}, + "kernel-revert": {{Kernel: kview("reverting")}}}} + l, h := judgingLeg(t, w) + start := l.now() + r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 10 * time.Minute, Poll: time.Minute}) + calls := kernelCalls(w) + if calls[len(calls)-1] != "kernel-revert" || strings.Count(strings.Join(calls, ","), "kernel-revert") != 1 { + t.Fatalf("calls = %v", calls) + } + if waited := l.now().Sub(start); waited < 10*time.Minute { + t.Fatalf("reverted after %s — before the judge wait", waited) + } + if r.Outcome != "health_failed" || !strings.Contains(r.HealthReason, "not running") { + t.Fatalf("report = %+v", r) + } + if last := h.reports[len(h.reports)-1]; last.Outcome != "health_failed" { + t.Fatalf("the hub must hear health_failed before the revert reboot: %+v", h.reports) + } + for _, p := range w.plans { + if p["mode"] == "kernel-revert" && !strings.Contains(p["reason"].(string), "not running") { + t.Fatalf("the revert must carry the reason: %v", p) + } + } +} + +// A box that never reaches the hub is not "healthy" — it reverts too. +func TestKernelAfterBoot_NoHubMeansRevert(t *testing.T) { + w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-revert": {{Kernel: kview("reverting")}}}} + l, _ := judgingLeg(t, w) + l.Hub = unreachableHub{} + l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 5 * time.Minute, Poll: time.Minute}) + if c := kernelCalls(w); c[len(c)-1] != "kernel-revert" { + t.Fatalf("calls = %v", c) + } +} + +type unreachableHub struct{} + +func (unreachableHub) PostOSReport(context.Context, []byte) error { return context.DeadlineExceeded } + +// What kernel-boot found becomes the hub's outcome, with no judging and no reboot. +func TestKernelAfterBoot_FallBackAndRevertResultsAreReported(t *testing.T) { + for ev, want := range map[string]string{"fell_back": "fell_back", "self_reverted": "self_reverted", "revert_failed": "revert_failed"} { + w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-boot": {{KernelEvent: ev, + Kernel: mustRaw(KernelView{Running: kOld, Default: kOld, Phase: ev, From: kOld, To: kNew, Reason: "r", VMID: 9201})}}}} + l, h := judgingLeg(t, w) + r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{}) + if r.Outcome != want || len(h.reports) != 1 || h.reports[0].Outcome != want { + t.Fatalf("%s: report %+v hub %+v", ev, r, h.reports) + } + if got := strings.Join(kernelCalls(w), ","); got != "kernel-boot" { + t.Fatalf("%s: calls = %s", ev, got) + } + } + // nothing to do → nothing reported + w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-boot": {{KernelEvent: "none", Kernel: kview("good")}}}} + l, h := judgingLeg(t, w) + if r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{}); r.Layer != "" || len(h.reports) != 0 { + t.Fatalf("an ordinary boot must report nothing: %+v %+v", r, h.reports) + } +} + +// The signed executor STAGES (listed + the raw envelope + the kver) and never reboots. +func TestKernelStepExecutor_StagesNeverReboots(t *testing.T) { + w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"apply": {{Upgraded: []Package{{Name: "proxmox-kernel-7.0", + Version: "7.0.14-22"}}, Authority: "signed", Kernel: kview("staged")}}}} + l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true}) + e := KernelStepExecutor{Leg: l, Guest: func(context.Context) (int, error) { return 9201, nil }} + params, _ := json.Marshal(KernelStepParams{ReleaseID: "os-kernel-1", Kver: kNew, + Packages: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22", Origin: PVEOrigin}}}) + ctx := signedjobs.WithSignedOp(context.Background(), &reconcile.SignedOp{Blob: []byte(`{"op":"os_kernel_step"}`), Sig: []byte("SIG")}) + if err := e.Execute(ctx, OpKernelStep, params); err != nil { + t.Fatal(err) + } + pp := w.plans[len(w.plans)-1] + sg, _ := pp["signed"].(map[string]any) + if pp["mode"] != "apply" || pp["select"] != "listed" || pp["expect_kver"] != kNew || sg == nil || + sg["blob_b64"] != base64.StdEncoding.EncodeToString([]byte(`{"op":"os_kernel_step"}`)) { + t.Fatalf("plan = %v", pp) + } + if got := strings.Join(kernelCalls(w), ","); got != "apply" { + t.Fatalf("a signed stage must never reboot: %s", got) + } + if len(h.reports) != 1 || h.reports[0].Outcome != "staged" { + t.Fatalf("hub = %+v", h.reports) + } + if err := e.Execute(context.Background(), OpKernelStep, params); err == nil { + t.Fatal("no envelope must refuse") + } + bad, _ := json.Marshal(KernelStepParams{Kver: "x", Packages: []Package{{Name: "a"}}}) + if err := e.Execute(ctx, OpKernelStep, bad); err == nil { + t.Fatal("a bad kver must refuse") + } + if err := e.Execute(ctx, OpPVEStep, params); err != signedjobs.ErrNoExecutor { + t.Fatalf("another op must pass through the chain: %v", err) + } +} + +// os_kernel_step is never benign. +func TestKernelStep_IsDestructiveClass(t *testing.T) { + if reconcile.Classify(reconcile.ClassOSKernelStep, reconcile.Provenance{}) != reconcile.Destructive { + t.Fatal("os_kernel_step must be destructive-class (signed, operational key)") + } +} + +// A kept stage report (the agent was killed mid-stage) reaches the hub as "staged" with its kernel view. +func TestKernel_KeptStageReportIsStaged(t *testing.T) { + w := &fakeWrapper{t: t} + l, _ := newLeg(t, w, tonight(0)) + ring := 0 + rep := l.reportFromKept(context.Background(), WrapperReport{Layer: LayerKernel, Mode: "apply", Ring: &ring, + Upgraded: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22"}}, Kernel: kview("staged")}, "/x/report-r-kernel-apply.json") + if rep.Outcome != "staged" || !rep.Healthy || len(rep.Kernel) == 0 { + t.Fatalf("kept = %+v", rep) + } +} diff --git a/internal/osupdate/leg.go b/internal/osupdate/leg.go index 49f029e..318979b 100644 --- a/internal/osupdate/leg.go +++ b/internal/osupdate/leg.go @@ -45,6 +45,9 @@ const ( // LayerPVE is the HOST's Proxmox userspace packages — slow lane (R-812 option A, `09` §3 decision 163, `11` §5.10): // ring 0 in the night leg after a healthy host step, ring 1 only inside a signed os_pve_step. Never a kernel. LayerPVE = "pve" + // LayerKernel is the HOST's kernel — the kernel lane (R-836, `09` §3 decision 172, `11` §5.11): a one-shot boot of + // the new kernel through the ESP flag, the default moved only after a healthy boot (kernel.go). + LayerKernel = "kernel" ) // PVEOrigin is the origin apt prints for download.proxmox.com (the wrapper's PVE_ORIGIN); InstalledPVEOrigin is how @@ -126,6 +129,11 @@ type WrapperReport struct { OOMCheck json.RawMessage `json:"oom_check"` // PVEManager: the pve layer — pveversion's pve-manager version after the step ("unknown" = unreadable). PVEManager string `json:"pve_manager"` + // Kernel (R-836): the kernel layer's view {running, default, flag, phase, from, to, …}; KernelEvent what kernel-boot + // found after a boot; OutcomeHint "nothing" when no kernel was pending. + Kernel json.RawMessage `json:"kernel"` + KernelEvent string `json:"kernel_event"` + OutcomeHint string `json:"outcome_hint"` // R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the // agent process that started the pass. ReleaseID / VMID were always in the report. RunID string `json:"run_id"` @@ -168,6 +176,10 @@ type Report struct { OOMCheck json.RawMessage `json:"oom_check,omitempty"` // PVEManager: pve layer — pve-manager's version after the step (the hub's System page; R-812 option A). PVEManager string `json:"pve_manager,omitempty"` + // Kernel: kernel layer — the wrapper's kernel view, byte for byte (running, default, flag, phase, from, to). The + // outcomes of this layer: staged | applied (the new kernel is the default) | fell_back | health_failed (self-revert + // started) | self_reverted | revert_failed | judging | nothing | refused | failed. + Kernel json.RawMessage `json:"kernel,omitempty"` unsent string // R-868: the wrapper's kept copy of this pass's report — deleted once the hub has it } @@ -500,7 +512,7 @@ func (l *Leg) call(ctx context.Context, runID string, plan map[string]any) (Wrap // Pass is one leg's reports; an empty Layer means the step did not run. type Pass struct { - Guest, Host, Docker, PVE Report + Guest, Host, Docker, PVE, Kernel Report } // Run is one pass: the guest layer, then (on an appliance, after a good guest step) the host layer, then (ring 0 @@ -542,6 +554,16 @@ func (l *Leg) Run(ctx context.Context, vmid int, trigger string) Pass { default: p.PVE = l.runPVE(ctx, g.RunID, vmid, trigger, blk, dockerOpts{}) } + // R-836 (`09` §3 decision 172): the kernel step ENDS the night — an appliance, after a healthy host step (and a + // healthy Proxmox step when one ran), only on a night the hub marks as told. It reboots the box. Pinned by TestKernel_*. + switch { + case !l.Appliance || h.Layer == "" || !okStep(h): + lg.Info("osupdate: kernel step skipped — no healthy host step this pass (an appliance only)", "appliance", l.Appliance, "host_outcome", h.Outcome) + case p.PVE.Layer != "" && !okStep(p.PVE): + lg.Warn("osupdate: kernel step skipped — the Proxmox step did not end healthy", "pve_outcome", p.PVE.Outcome) + default: + p.Kernel = l.runKernel(ctx, g.RunID, vmid, trigger, blk) + } return p } diff --git a/internal/osupdate/leg_test.go b/internal/osupdate/leg_test.go index e21512d..6703635 100644 --- a/internal/osupdate/leg_test.go +++ b/internal/osupdate/leg_test.go @@ -25,6 +25,7 @@ type fakeWrapper struct { plans []map[string]any keep bool // R-868: like the real wrapper, keep an apply report beside the plan pveGateHeld bool + kernelRep map[string][]WrapperReport // R-836: per kernel-layer mode, successive answers (the last one repeats) } func yes() *bool { b := true; return &b } @@ -52,9 +53,19 @@ func (f *fakeWrapper) Run(_ context.Context, name string, args ...string) ([]byt f.plans = append(f.plans, plan) layer := plan["layer"].(string) ok := guestOK() - if layer == LayerHost || layer == LayerPVE { + if layer == LayerHost || layer == LayerPVE || layer == LayerKernel { ok = hostOK() } + if layer == LayerKernel { + if seq := f.kernelRep[plan["mode"].(string)]; len(seq) > 0 { + rep := seq[0] + if len(seq) > 1 { + f.kernelRep[plan["mode"].(string)] = seq[1:] + } + out, _ := json.Marshal(rep) + return []byte("OSAPPLY-REPORT " + string(out) + "\n"), []byte("os-apply: DONE rc=0\n"), nil + } + } if layer == LayerPVE && plan["mode"] == "apply" { f.pveGateHeld = pvegate.Stepping() // R-812: the /etc/pve write gate must be held while the pve step runs } diff --git a/internal/osupdate/unsent.go b/internal/osupdate/unsent.go index 656d945..7fc87c8 100644 --- a/internal/osupdate/unsent.go +++ b/internal/osupdate/unsent.go @@ -142,6 +142,17 @@ func (l *Leg) reportFromKept(ctx context.Context, wr WrapperReport, path string) default: rep.Outcome = "applied" } + if wr.Layer == LayerKernel { + // a kernel STAGE changes nothing the box runs (the new kernel only boots once, at the night's reboot), so its + // kept copy needs no fresh health reading (R-836) + rep.Kernel = rawOrNil(wr.Kernel) + rep.Upgraded, rep.PassSeconds, rep.Authority = wr.Upgraded, wr.PassSeconds, wr.Authority + if rep.Outcome == "applied" { + rep.Outcome = "staged" + } + rep.Healthy, rep.HealthReason = rep.Outcome == "staged" || rep.Outcome == "nothing", prefix + return rep + } rep.Upgraded, rep.PassSeconds = wr.Upgraded, wr.PassSeconds rep.DockerEngine, rep.Authority, rep.Undo = wr.DockerEngine, wr.Authority, wr.Undo rep.OOMCheck = rawOrNil(wr.OOMCheck) diff --git a/internal/reconcile/classify.go b/internal/reconcile/classify.go index 029e83b..adb37e0 100644 --- a/internal/reconcile/classify.go +++ b/internal/reconcile/classify.go @@ -56,6 +56,11 @@ const ( // key) like os_docker_step; the root wrapper re-verifies the same signature itself. ClassOSPVEStep OpClass = "os_pve_step" + // A kernel step on the host (R-836, `09` §3 decision 172, `11` §5.11) — ring 1: it STAGES a kernel (install + the + // one-shot flag; the night leg reboots it). Destructive-class (signed, operational key) like os_pve_step; the root + // wrapper re-verifies the same signature itself. + ClassOSKernelStep OpClass = "os_kernel_step" + // The config bundle (agent v0.143.0, R-840, `11` §5.4.2): the box's ROOT-OWNED files (sudoers, wrappers, units). // Destructive-class (signed, operational key) like agent_update; the root wrapper re-verifies the signature itself. ClassAgentConfigUpdate OpClass = "agent_config_update" @@ -127,7 +132,7 @@ func Classify(class OpClass, prov Provenance) Disposition { return Destructive case ClassKeyRotation: return Destructive - case ClassAgentUpdate, ClassOSDockerStep, ClassOSPVEStep, ClassAgentConfigUpdate: + case ClassAgentUpdate, ClassOSDockerStep, ClassOSPVEStep, ClassOSKernelStep, ClassAgentConfigUpdate: // Never benign — no agent-internal provenance can make replacing the agent binary // unsigned-safe (a compromised process must not be able to self-bless an update). return Destructive