Compare commits
7 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 admin
|
c9013bb47d | ||
|
admin
|
2d1e5d0774 | ||
|
admin
|
f09c53efc1 | ||
|
admin
|
92d647a6f6 | ||
|
admin
|
d03ab7f1f5 | ||
|
admin
|
b17d1c597d | ||
|
admin
|
0ae01dfb52 |
+74
-3
@@ -1,4 +1,75 @@
|
||||
## Unreleased (2026-10-07) — the agent can no longer hand the guest any image; felhom-op's pct lines are exact (R-861 (a) A1, (b) B2; `09` §3 decision 165)
|
||||
## v0.153.0 — ring 0 stages exactly the told kernel (R-898; `09` §3 decision 176) (2026-10-07)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `b204ebe6d65944f43b70dcfa4ae0dfb90da38c92e2ba6a38a1826e488eeb6630`, bundle
|
||||
`6db216275eb0dd39188d93481a2045998a69e8cb7838ad908a58d65d9a6a9b57` (tag `v0.153.0` = `2d1e5d0`). Delivered (binary only — the
|
||||
bundle files are unchanged from 0.152.0) to demo-hp, demo-felhom, Tester 1 on 2026-10-07 19:03.
|
||||
|
||||
**Delivery: the agent binary only** — no root file changed (the wrapper is unchanged; its tests gained two cases).
|
||||
|
||||
- `internal/osupdate/kernel.go`: ring 0's night kernel step stages EXACTLY the kernel the household was told about
|
||||
(select `listed`, `KernelSet(kver)` = the series meta-package and the signed image at the kernel's own version) instead
|
||||
of "whatever is pending tonight". Seen 2026-10-07: demo-felhom was told about 7.0.14-20 while its sources offered
|
||||
7.0.14-22 by night — the old code staged `pending-kernel` and the wrapper refused it (R23), losing the night. A told
|
||||
version that is no longer installable is refused by the wrapper before any change (R7) and the hub tells the household
|
||||
again for the newer kernel (hub v0.143.1). Tests `TestKernel_Ring0ToldNightStagesThenReboots` (red-proved against the
|
||||
old select), `TestKernelSet`; wrapper `test_ring0_listed_installs_the_told_kernel_not_the_newest`,
|
||||
`test_ring0_told_kernel_gone_is_refused_before_any_change`.
|
||||
|
||||
## v0.152.0 — the kernel lane (R-836; `09` §3 decisions 164, 172; `11` §5.11) (2026-10-07)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `95ff42208e36ba49b6e2b09a97e81a6fa11562ecc8042f1ed18d378b8b1f88b1`,
|
||||
config bundle sha256 `f0c2cec374b711b3c131c955012b33fd0ad495337d049b7a743c3eef9e85c20b` (tag `v0.152.0` = `d03ab7f`).
|
||||
Step bundle `0.152.0-step1` sha256 `0b71d32b054cf3b7ade0234ffcbb0df159901f542cde540adaee411db466f48e` (the 0.151.0 bundle
|
||||
with only `felhom-os-apply` replaced; `scripts/build-step-bundle.py`), published as package version `0.152.0-step1`.
|
||||
|
||||
**Delivery: agent binary, then the STEP bundle `0.152.0-step1`, then the bundle `0.152.0`** — the bundle ADDS two paths
|
||||
(the GRUB generators), and an installed `felhom-os-apply` refuses a path its own table lacks (R16, R-880).
|
||||
|
||||
A new kernel boots ONCE; if it crashes the box comes back on the old kernel by itself; it becomes the default only after
|
||||
a healthy boot; a booted-but-unhealthy kernel is reverted ONCE by the agent with no person. Built on the spike's
|
||||
candidate 2 (`audits/kernel-spike-2026-10-07/`), with option C on the one-shot entry.
|
||||
|
||||
- `configs/felhom-grub-oneshot.sh` → `/etc/grub.d/01_felhom_oneshot` (bundle): reads `felhom_next` from a GRUB env block
|
||||
on the ESP (`EFI/felhom/oneshot.env`), clears and saves it BEFORE the menu, and sets the default to that kernel's
|
||||
one-shot entry only when the name is an installed kernel. No vfat ESP → prints nothing.
|
||||
- `configs/felhom-grub-oneshot-entries.sh` → `/etc/grub.d/42_felhom_oneshot` (bundle): one entry per installed kernel,
|
||||
id `felhom-oneshot-<ver>`, the normal entry plus `softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10`
|
||||
(option C). Sorted after `10_linux`: never entry 0, never the default.
|
||||
- `configs/felhom-os-apply`: layer `kernel` (lane slow; an appliance; authority = a signed `os_kernel_step` or the
|
||||
root-owned ring-0 mark). Modes: `apply` STAGES (select `pending-kernel` or a signed `listed` set; `expect_kver` = the
|
||||
kernel the household was told about): pins the GRUB default to the RUNNING kernel in
|
||||
`/etc/default/grub.d/zz-felhom-kernel-default.cfg` and proves it from grub.cfg, installs, proves the default did not
|
||||
move and the one-shot entry exists, writes the flag; never reboots. `kernel-reboot` (a staged step only),
|
||||
`kernel-boot` (judging | fell_back | self_reverted | revert_failed), `kernel-good` (the new kernel becomes the
|
||||
default, proved), `kernel-revert` (ONE per step; refused when the default is not the old kernel), `kernel-cancel`,
|
||||
`kernel-status`. New refusals: R20 (the box cannot do a one-shot: not UEFI, no vfat ESP, a separate /boot, GRUB
|
||||
without fat/loadenv, the generators missing, a hand pin), R21 (the crash guard tripped or an unclean boot in its
|
||||
window), R22 (the phase does not allow the mode; never two steps within 20 h), R23 (not exactly one newer kernel, or
|
||||
not the one signed / told). State `/var/lib/felhom-kernel/state.json`. Facts carry `kernel_lane`; the next-boot
|
||||
kernel reads the flag and the grub.cfg default. Tests: `KernelLane` (27), red-proof
|
||||
`felhom.eu/documentation/audits/kernel-lane-2026-10-07/A/redproof.txt`.
|
||||
- `configs/felhom-crash-guard` unchanged; `KernelStepCannotLeaveTheBoxOff` (3 tests) pins that a step's planned reboot,
|
||||
one crash and one self-revert add ONE unclean boot (a panic before userspace adds none), so the box cannot stay off.
|
||||
- `internal/osupdate/kernel.go`: the night leg ends with the kernel step — after a healthy host step (and a healthy
|
||||
Proxmox step when one ran), trigger `night` only, on a night the hub's `os_update.kernel` block marks `tonight` (the
|
||||
household was mailed the day before — no mail, no step). Ring 0 stages + reboots; ring 1 reboots only a kernel a
|
||||
signed `os_kernel_step` staged (`KernelStepExecutor`: stage only, under the heavy-op gate). The hub hears `staged`
|
||||
BEFORE the reboot. At every start `KernelAfterBoot`: on the new kernel it JUDGES the boot — `KernelVerdict` = the
|
||||
host health rule (`11` §8.2) AND the box reached the hub (the `judging` report itself) — for 20 minutes (measured:
|
||||
everything healthy 68 s after the reboot on demo-felhom, 272 s on demo-hp; under the hub's 45-minute `host_stale` (`alerting.stale_threshold`)).
|
||||
Healthy → `kernel-good`, outcome `applied`; not healthy → outcome `health_failed`, then ONE `kernel-revert`.
|
||||
Tests: `TestKernel*` (13); red-proofs in the same file.
|
||||
- `internal/hub`: `WireOSUpdate.Kernel` {kver, tonight, notified_at}. `internal/reconcile`: `os_kernel_step` is
|
||||
destructive-class. `cmd/felhom-opsign`: the op is listed.
|
||||
|
||||
## v0.151.0 — the agent can no longer hand the guest any image; the Proxmox package lane; the other-key archives reported; the DR directive retired (R-861, R-812 A, R-366, R-105; `09` §3 163, 165, 168, 169) (2026-10-07)
|
||||
|
||||
Released by `scripts/release-agent.sh`: binary sha256 `0464354f2cdf452a7c5d2a74d9191fe91415fcfa244154480d26d5b30e10b194`
|
||||
config bundle sha256 `bacd1d175a9392bc1755341d01a106abbd72aba48ab575e7a32dd819d8f5da4c` (tag `v0.151.0` = `dd7cdc0`).
|
||||
**Deliver the binary FIRST, then the bundle:** the bundle's sudoers removes the `tee` grant the 0.150.0 binary still uses.
|
||||
No path added (26 → 26), so no step bundle.
|
||||
|
||||
### Part of v0.151.0 — the agent can no longer hand the guest any image; felhom-op's pct lines are exact (R-861 (a) A1, (b) B2; `09` §3 decision 165)
|
||||
|
||||
**Delivery order: agent binary FIRST, then the config bundle.** The new sudoers drops the agent's in-guest `tee`
|
||||
grant; an older binary still calls `tee`, so a bundle that lands before the binary would stop managed controller
|
||||
@@ -20,7 +91,7 @@ updates (and the old binary's capability probe would read `controllerswap-write`
|
||||
`TestSudoersAllowsTheControllerImageVerb`, `TestFelhomOpSudoersPctIsExact`, `TestR861_WriteControllerImageUsesTheRootVerb`,
|
||||
`TestControllerSwap_WriteViaRootVerb_NoShell`. Red-proofs: `felhom.eu/documentation/audits/day-2026-10-07/C/`.
|
||||
- `README.md`: the controller-swap paragraph described the removed `tee` path — corrected.
|
||||
## Unreleased (2026-10-07) — the Proxmox package lane (R-812 option A, `09` §3 decision 163)
|
||||
### Part of v0.151.0 — the Proxmox package lane (R-812 option A, `09` §3 decision 163)
|
||||
|
||||
**MinAgent impact: none** (a new layer; an older hub ignores the pve report). **The bundle carries the new
|
||||
`felhom-os-apply` — deliver it with the binary** (signed `agent_update`, then signed `agent_config_update`).
|
||||
@@ -29,7 +100,7 @@ updates (and the old binary's capability probe would read `controllerswap-write`
|
||||
- `internal/pvegate` (new): the agent's own writes to /etc/pve wait while a pve step runs (pmxcfs restarts); the step waits for writes in flight (bounded, 2 min — then it fails and does not run). Wired at `proxmox.Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile).
|
||||
- `internal/osupdate`: `LayerPVE`; the night leg runs the pve step in ring 0 after a healthy host step (an appliance; ring 1 never in the night leg); `PVEHealthVerdict` = the host rule + every running container keeps its id + pveversion reads the installed pve-manager; the pve report carries Proxmox userspace only (the hub's candidate set). `PVEStepExecutor` (signed `os_pve_step`, ring 1, under the heavy-op gate and the /etc/pve gate); `reconcile.ClassOSPVEStep` (destructive-class); `felhom-opsign -op os_pve_step` (params by `-params`).
|
||||
- Tests: wrapper `PVELane` (17; red first — the `pve-manager` plan was refused R12 on the old code), `pvegate` (5), `TestPVEGate_*` + `TestWritesEtcPVE`, `TestPVE_*`, `TestPVEHealthVerdict`, `TestPVEStepExecutor_*`. Red-proofs: `felhom.eu/documentation/audits/day-2026-10-07/B/`.
|
||||
## Unreleased (2026-10-07)
|
||||
### Part of v0.151.0
|
||||
|
||||
- R-366 slice 2 (`09` §3 decision 168): the restore-test pick records, per tier, the archives it skipped as written with another key (count, oldest, newest — no key material) in a `ForeignKeyLedger`; the host report carries it as `foreign_key_archives.tiers` (the stanza absent until a tier was evaluated since start, `tiers: []` when none — no null on the wire, the report contract forbids it). The hub turns a change into one operator line. Tests `TestR366_PickRecordsArchivesWrittenWithAnotherKey`, `TestR366_EvaluatedWithNoneIsAnEmptyList` (red-proved, `felhom.eu/documentation/audits/day-2026-10-07/E/`).
|
||||
- R-105 option A (`09` §3 decision 169): the `--selftest=escrow-create -directive <file>` flag and the escrow upload's `directive` field are removed — nothing read the directive; the DR path reads the recipe, tenantsync and the escrow blob. The hub ignores a `directive` from an older agent.
|
||||
|
||||
@@ -1,5 +1,7 @@
|
||||
# CONTEXT — felhom-agent working state
|
||||
|
||||
> **2026-10-07 (evening) — v0.152.0, the kernel lane (R-836, decision 172, `11` §5.11).** Wrapper layer `kernel` + two GRUB generators in the bundle (delivered with step bundle `0.152.0-step1` — the bundle adds paths, R-880); `osupdate/kernel.go`: night step on told nights only, after-boot judge (host rule + hub reached, 20 min), ONE self-revert, `os_kernel_step` stages only. Proven on Tester 1 (panic → fell_back, held guest → self_reverted, healthy → 7.0.14-22 default). Open: R-898, R-897; the ring-0 night run.
|
||||
|
||||
> **2026-10-04 night — v0.143.0 RELEASED + vouched (R-840, decision 96): the config bundle.** `felhom-os-apply` mode
|
||||
> `bundle` (signed `agent_config_update`, verified by the wrapper itself; trust files never bundle paths) +
|
||||
> `--install-bundle` (installer 1.31.0); `BUNDLE_FILES` is the one table; `scripts/build-config-bundle.py`;
|
||||
|
||||
@@ -1,8 +1,9 @@
|
||||
# REPORT — v0.150.0 released and delivered (2026-10-07)
|
||||
# REPORT — agent v0.153.0: ring 0 stages exactly the told kernel (2026-10-07, late evening)
|
||||
|
||||
On the operator's word (`09` §3 decision 161). `scripts/release-agent.sh 0.150.0`: sha `a23d1c90…`, bundle `88456b38…`,
|
||||
tag `v0.150.0` = `3a72a48`, verified by download. Vouched with golden 0.301.0 and MinAgent 0.131.0 (unchanged). No bundle
|
||||
path added (26 → 26), so no step bundle. Signed `agent_update` → demo-hp, demo-felhom, Tester 1 on 0.150.0 (07:06–07:07Z);
|
||||
signed `agent_config_update` → `BUNDLE DONE written=1 same=24 self-check=ok`, capability probe 68/68 on all three
|
||||
(07:21Z). Tester 2 not touched. Carries R-528 (the memory-kill check), R-894 (the last backup per tier on disk), R-330
|
||||
(SMART counters on the wire). Evidence: `felhom.eu/documentation/audits/readback-2026-10-07/delivery/`.
|
||||
- R-898 (closed): the night kernel step on ring 0 stages the kernel the household was told about (`select listed`,
|
||||
`osupdate.KernelSet(kver)`), never "whatever is pending tonight". A told version no longer installable → the wrapper
|
||||
refuses before any change (R7); the hub re-tells the household for the newer kernel.
|
||||
- Tests: `TestKernel_Ring0ToldNightStagesThenReboots` (red-proved against the old select), `TestKernelSet`; wrapper
|
||||
cases `test_ring0_listed_installs_the_told_kernel_not_the_newest`, `test_ring0_told_kernel_gone_is_refused_before_any_change`.
|
||||
- Released `v0.153.0`, binary `b204ebe6…`; delivered (binary only) to demo-hp, demo-felhom, Tester 1 at 19:03.
|
||||
- Tonight (7→8): demo-felhom stages 7.0.14-20, demo-hp 7.0.14-22 — both households told. Read back 2026-10-08.
|
||||
|
||||
@@ -89,6 +89,7 @@
|
||||
| Symbol | File | Short signature | Use for | Gotchas |
|
||||
|---|---|---|---|---|
|
||||
| `pvegate.Write` / `pvegate.Step` | internal/pvegate/pvegate.go | `Write(ctx) (release, waited, err)` / `Step(ctx) (end, err)` | R-812 option A: keep the agent's own /etc/pve writes out of a Proxmox package step (pmxcfs restarts) | Already wired at the two chokepoints — `Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile). A new root CLI that writes /etc/pve goes into `WritesEtcPVE`, never its own lock. Never take `Step` around anything but the wrapper call (`Leg.runPVE`) — a `Write` inside a `Step` deadlocks until its context ends. |
|
||||
| `osupdate.KernelVerdict` / `Leg.KernelAfterBoot` | internal/osupdate/kernel.go | `KernelVerdict(before, after, tunnel, hubReached) (ok, why)` | R-836: THE one-shot-boot rule — the host health rule (`HostHealthVerdict`) AND the box reached the hub since the boot | The only judge of a new kernel. Never reboot the host from Go: every reboot is the wrapper's (`kernel-reboot`, `kernel-revert`), and only for a staged step. A new kernel-lane state lives in the wrapper's `/var/lib/felhom-kernel/state.json`, never in the agent's own files (the agent can write those). |
|
||||
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
|
||||
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
|
||||
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
|
||||
|
||||
@@ -877,6 +877,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
go osLeg.SendUnsentLoop(ctx, 5*time.Minute, func(n int) {
|
||||
logger.Info("osupdate: sent kept report(s)", "count", n)
|
||||
})
|
||||
// R-836 (`09` §3 decision 172): what became of a kernel step across this boot; on a one-shot boot of a new kernel,
|
||||
// judge it (the host health rule + the hub reached) for KernelJudgeWait, then make it the default or revert ONCE.
|
||||
// The wait: measured 2026-10-07 (`audits/kernel-lane-2026-10-07/B/`) — every container healthy 68 s after the
|
||||
// reboot on demo-felhom and 272 s on demo-hp (the hub reached at 63 s / 189 s); 20 minutes leaves room for a slow
|
||||
// network and stays under the hub's 45-minute host_stale (its
|
||||
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
|
||||
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
|
||||
|
||||
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
|
||||
// (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving
|
||||
@@ -1119,6 +1126,16 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
return release, nil
|
||||
}}
|
||||
// R-836 (`09` §3 decision 172): a signed kernel step STAGES a kernel on a ring-1 box (install + the one-shot flag,
|
||||
// never a reboot — the night leg reboots it on a night the household was told about); under the heavy-op gate.
|
||||
kernelExec := osupdate.KernelStepExecutor{Leg: osLeg, Guest: firstGuest(px),
|
||||
Gate: func(ctx context.Context) (func(), error) {
|
||||
release, busy, ok := heavyOps.TryAcquire("os-kernel-step")
|
||||
if !ok {
|
||||
return nil, fmt.Errorf("busy: %s", busy)
|
||||
}
|
||||
return release, nil
|
||||
}}
|
||||
// Agent v0.143.0 (R-840): the config bundle — the box's root-owned files by a signed job; the wrapper verifies it.
|
||||
bundleExec := osupdate.ConfigUpdateExecutor{Leg: osLeg, URLTemplate: suCfg.URLTemplate, Username: suCfg.Username, Token: suCfg.Token,
|
||||
// The capability probe confirms from the agent's side: `sudo -l` lists every command the new sudoers grants.
|
||||
@@ -1130,7 +1147,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
|
||||
}
|
||||
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
|
||||
}}
|
||||
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, pveExec, bundleExec}, cfg.Hub.HostID, logger)
|
||||
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, pveExec, kernelExec, bundleExec}, cfg.Hub.HostID, logger)
|
||||
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner))
|
||||
|
||||
// Controller-driven escrow ceremony (v0.88.0): static config facts + the LATE-BOUND DR gate —
|
||||
|
||||
@@ -43,7 +43,7 @@ func main() {
|
||||
|
||||
func run() error {
|
||||
var (
|
||||
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | os_pve_step | agent_config_update")
|
||||
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | os_pve_step | os_kernel_step | agent_config_update")
|
||||
host = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)")
|
||||
guest = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)")
|
||||
keyID = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)")
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
#!/bin/sh
|
||||
# /etc/grub.d/42_felhom_oneshot — the kernel lane's one-shot ENTRIES (R-836, `09` §3 decision 172, `11` §5.11).
|
||||
# Installed by the config bundle (felhom-os-apply BUNDLE_FILES), 0755 root. update-grub runs it.
|
||||
#
|
||||
# One menu entry per installed Proxmox kernel, id `felhom-oneshot-<version>`, booted ONLY when 01_felhom_oneshot found
|
||||
# the flag naming it. It is the normal entry plus option C (decision 172): softlockup_panic=1 hardlockup_panic=1
|
||||
# hung_task_panic=1 panic=10 — a lockup the kernel can detect becomes a panic, and a panic restarts the box in 10 s into
|
||||
# the default (the old kernel). A true dead freeze still needs a person (spike candidate 3 failed on all three boxes).
|
||||
# It sorts AFTER 10_linux, so it is never entry 0 and never the default.
|
||||
#
|
||||
# No vfat ESP at /boot/efi → prints nothing (no flag can name these entries).
|
||||
set -e
|
||||
prefix="/usr"
|
||||
exec_prefix="/usr"
|
||||
datarootdir="/usr/share"
|
||||
. "$datarootdir/grub/grub-mkconfig_lib"
|
||||
esp_uuid=$(findmnt -n -o UUID,FSTYPE /boot/efi 2>/dev/null | awk '$2 == "vfat" { print $1 }')
|
||||
[ -n "$esp_uuid" ] || exit 0
|
||||
kernels=$(ls /boot/vmlinuz-*-pve 2>/dev/null | sed 's#^/boot/vmlinuz-##' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$' || true)
|
||||
[ -n "$kernels" ] || exit 0
|
||||
case "${GRUB_DEVICE}" in
|
||||
/dev/mapper/*|/dev/dm-*|"") root_arg="root=${GRUB_DEVICE}" ;;
|
||||
*) if [ -n "${GRUB_DEVICE_UUID}" ]; then root_arg="root=UUID=${GRUB_DEVICE_UUID}"; else root_arg="root=${GRUB_DEVICE}"; fi ;;
|
||||
esac
|
||||
[ -n "${GRUB_DEVICE}" ] || root_arg="root=$(findmnt -n -o SOURCE /)"
|
||||
rel=$(make_system_path_relative_to_its_root /boot)
|
||||
prep=$(prepare_grub_to_access_device "$(${grub_probe:-grub-probe} --target=device /boot)" | sed 's/^/ /')
|
||||
for k in $kernels; do
|
||||
[ -f "/boot/initrd.img-$k" ] || continue
|
||||
cat <<EOF
|
||||
menuentry 'Felhom one-shot: $k' --class proxmox --id felhom-oneshot-$k {
|
||||
insmod gzio
|
||||
$prep
|
||||
echo 'Loading Linux $k (felhom one-shot) ...'
|
||||
linux $rel/vmlinuz-$k $root_arg ro ${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT} softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10
|
||||
initrd $rel/initrd.img-$k
|
||||
}
|
||||
EOF
|
||||
done
|
||||
@@ -0,0 +1,36 @@
|
||||
#!/bin/sh
|
||||
# /etc/grub.d/01_felhom_oneshot — the kernel lane's ONE-SHOT boot (R-836, `09` §3 decisions 164 + 172, `11` §5.11).
|
||||
# Installed by the config bundle (felhom-os-apply BUNDLE_FILES), 0755 root. update-grub runs it; it prints GRUB script.
|
||||
#
|
||||
# At boot, GRUB reads `felhom_next` from an environment block on the ESP (vfat — GRUB can rewrite a file there; it
|
||||
# cannot on the LVM /boot, R-836), CLEARS it, and — only if it names an installed kernel — boots that kernel's one-shot
|
||||
# entry (42_felhom_oneshot) instead of the default. The next boot uses the default again whatever happens: a new kernel
|
||||
# that panics comes back on the old one by itself. felhom-os-apply writes the flag (mode apply) and never the default
|
||||
# for a new kernel. Measured on the Tester 1 VM, demo-felhom and demo-hp (Secure Boot on):
|
||||
# `audits/kernel-spike-2026-10-07/` (candidate 2).
|
||||
#
|
||||
# No vfat ESP at /boot/efi, or no Proxmox kernel → prints nothing (the box boots exactly as before).
|
||||
set -e
|
||||
esp_uuid=$(findmnt -n -o UUID,FSTYPE /boot/efi 2>/dev/null | awk '$2 == "vfat" { print $1 }')
|
||||
[ -n "$esp_uuid" ] || exit 0
|
||||
kernels=$(ls /boot/vmlinuz-*-pve 2>/dev/null | sed 's#^/boot/vmlinuz-##' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$' || true)
|
||||
[ -n "$kernels" ] || exit 0
|
||||
cat <<EOF
|
||||
# felhom kernel lane: a one-shot kernel named on the ESP, read and cleared before the menu
|
||||
insmod part_gpt
|
||||
insmod fat
|
||||
search --no-floppy --fs-uuid --set=felhom_esp $esp_uuid
|
||||
if [ -f (\$felhom_esp)/EFI/felhom/oneshot.env ]; then
|
||||
load_env -f (\$felhom_esp)/EFI/felhom/oneshot.env felhom_next
|
||||
if [ "\${felhom_next}" ]; then
|
||||
set felhom_boot="\${felhom_next}"
|
||||
set felhom_next=
|
||||
save_env -f (\$felhom_esp)/EFI/felhom/oneshot.env felhom_next
|
||||
EOF
|
||||
for k in $kernels; do
|
||||
printf ' if [ "${felhom_boot}" = "%s" ]; then set default="felhom-oneshot-%s"; fi\n' "$k" "$k"
|
||||
done
|
||||
cat <<EOF
|
||||
fi
|
||||
fi
|
||||
EOF
|
||||
+491
-11
@@ -124,6 +124,32 @@ DAEMON_JSON = "/etc/docker/daemon.json"
|
||||
# wrapper restarts exactly the containers that mount one of these paths (never the apps, never the engine).
|
||||
DOCKER_SOCKETS = ("/var/run/docker.sock", "/run/docker.sock")
|
||||
CRASH_GUARD_STATE = "/var/lib/felhom-crash-guard/state.json"
|
||||
# ---------- the kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11) ----------
|
||||
# A new kernel boots ONCE through a flag in a GRUB environment block on the ESP (vfat — GRUB can rewrite it there; on
|
||||
# the LVM /boot it cannot, R-836). The bundle's two GRUB generators read and clear the flag (01_felhom_oneshot) and
|
||||
# give each installed kernel a one-shot entry with the lockup-to-panic options (42_felhom_oneshot, option C). The GRUB
|
||||
# default is the kernel the box RUNS, pinned in KERNEL_DEFAULT_CFG; only "kernel-good" (after a healthy one-shot boot)
|
||||
# moves it to the new kernel. Measured in the spike on the Tester 1 VM, demo-felhom and demo-hp (Secure Boot on):
|
||||
# `audits/kernel-spike-2026-10-07/`.
|
||||
KERNEL_OP = "os_kernel_step"
|
||||
KERNEL_STATE = "/var/lib/felhom-kernel/state.json" # 0644 root: the step's phase (the agent reads it)
|
||||
KERNEL_DEFAULT_CFG = "/etc/default/grub.d/zz-felhom-kernel-default.cfg" # sourced last: GRUB_DEFAULT = this kernel
|
||||
ONESHOT_SNIPPET = "/etc/grub.d/01_felhom_oneshot" # bundle-owned: read + clear the flag, pick the entry
|
||||
ONESHOT_ENTRIES = "/etc/grub.d/42_felhom_oneshot" # bundle-owned: the one-shot entries (option C options)
|
||||
GRUB_CFG = "/boot/grub/grub.cfg"
|
||||
ESP_MOUNT = "/boot/efi"
|
||||
ONESHOT_ENV = ESP_MOUNT + "/EFI/felhom/oneshot.env"
|
||||
ONESHOT_ARGS = "softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10"
|
||||
KERNEL_PIN_FILE = "/etc/kernel/proxmox-boot-pin" # an operator's `proxmox-boot-tool kernel pin` — never fought
|
||||
KVER_RE = re.compile(r"^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$")
|
||||
KERNEL_IMAGE_RE = re.compile(r"^proxmox-kernel-([0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve)(-signed)?$")
|
||||
# the packages a kernel step may UPGRADE (never add): the kernel series meta-package, the default-kernel meta, the boot
|
||||
# helper and the firmware the kernel loads. Everything else in HOST_SLOW_RE (grub, shim, microcode, efibootmgr) stays out.
|
||||
KERNEL_UPGRADE_RE = re.compile(r"^(proxmox-default-kernel|proxmox-kernel-[0-9]+\.[0-9]+|proxmox-kernel-helper|pve-firmware)$")
|
||||
KERNEL_MIN_GAP = 20 * 3600 # never two kernel steps in one night
|
||||
KERNEL_MODES = ("kernel-status", "kernel-reboot", "kernel-boot", "kernel-good", "kernel-revert", "kernel-cancel")
|
||||
# phases: staged → oneshot → judging → good | reverting → self_reverted | revert_failed; oneshot → fell_back; staged → cancelled
|
||||
KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
|
||||
|
||||
# ---------- the config bundle (R-840, agent v0.143.0, `11` §5.4.2) ----------
|
||||
# A signed `agent_config_update` job carries {agent_version, bundle_sha256}; the bundle is ONE JSON file built from this
|
||||
@@ -170,6 +196,10 @@ BUNDLE_FILES = [
|
||||
("/etc/systemd/system/felhom-crash-guard-check.service", "felhom-crash-guard-check.service", 0o644, "unit", "replace"),
|
||||
("/etc/systemd/system/felhom-crash-guard-check.timer", "felhom-crash-guard-check.timer", 0o644, "unit", "replace"),
|
||||
("/etc/felhom/crash-guard.conf", "crash-guard.conf", 0o644, "plain", "if-absent"),
|
||||
# the kernel lane's two GRUB generators (R-836, `11` §5.11): they take effect at the next update-grub, which the
|
||||
# first kernel step runs itself; with no flag on the ESP they change nothing about how the box boots.
|
||||
("/etc/grub.d/01_felhom_oneshot", "felhom-grub-oneshot.sh", 0o755, "sh", "replace"),
|
||||
("/etc/grub.d/42_felhom_oneshot", "felhom-grub-oneshot-entries.sh", 0o755, "sh", "replace"),
|
||||
("/etc/systemd/system/felhom-agent.service", "felhom-agent.service", 0o644, "agent-unit", "replace"),
|
||||
("/etc/systemd/system/felhom-agent-rollback.service", "felhom-agent-rollback.service", 0o644, "unit", "replace"),
|
||||
("/etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf", "felhom-agent-limits.conf", 0o644, "dropin", "replace"),
|
||||
@@ -434,7 +464,8 @@ class Apply:
|
||||
|
||||
def check_plan(self, plan):
|
||||
mode = plan.get("mode", "apply")
|
||||
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update", "oom-check"):
|
||||
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update", "oom-check") \
|
||||
+ KERNEL_MODES:
|
||||
raise Refused("R11", f"unknown mode {mode!r}")
|
||||
if mode == "agent_update":
|
||||
if plan.get("layer") != "host":
|
||||
@@ -445,14 +476,18 @@ class Apply:
|
||||
raise Refused("R11", "bundle is a host-layer mode")
|
||||
return mode, "host", 0, "bundle"
|
||||
layer = plan.get("layer")
|
||||
if layer not in ("guest", "host", "docker", "pve"):
|
||||
raise Refused("R12", f"layer {layer!r} is not guest, host, docker or pve")
|
||||
if layer not in ("guest", "host", "docker", "pve", "kernel"):
|
||||
raise Refused("R12", f"layer {layer!r} is not guest, host, docker, pve or kernel")
|
||||
if (mode in KERNEL_MODES) != (layer == "kernel" and mode not in ("apply", "health")):
|
||||
raise Refused("R11", f"mode {mode!r} does not fit layer {layer!r}")
|
||||
lane = plan.get("lane", "fast")
|
||||
if layer == "docker" and lane != "slow":
|
||||
raise Refused("R3", "the Docker engine is the slow lane (`11` §5.8); a fast-lane Docker plan is refused")
|
||||
if layer == "pve" and lane != "slow":
|
||||
raise Refused("R3", "the Proxmox packages are the slow lane (`11` §5.10); a fast-lane pve plan is refused")
|
||||
if layer not in ("docker", "pve") and lane != "fast":
|
||||
if layer == "kernel" and lane != "slow":
|
||||
raise Refused("R3", "the kernel is the slow lane (`11` §5.11); a fast-lane kernel plan is refused")
|
||||
if layer not in ("docker", "pve", "kernel") and lane != "fast":
|
||||
raise Refused("R3", f"the {layer} layer has no slow lane in this release (kernel, Proxmox: `11` §8 step 6)")
|
||||
if mode == "facts" and layer != "host":
|
||||
raise Refused("R11", "facts is a host-layer mode (it reads the host and the guest)")
|
||||
@@ -463,7 +498,9 @@ class Apply:
|
||||
if plan.get("undo") and layer != "docker": # the pve layer has no undo in this release (R-812 option A)
|
||||
raise Refused("R5", "an undo (downgrade) exists only for the Docker layer, inside a signed job")
|
||||
vmid = plan.get("vmid")
|
||||
if not isinstance(vmid, int) or isinstance(vmid, bool) or vmid <= 0:
|
||||
if layer == "kernel" and mode in KERNEL_MODES and mode != "kernel-reboot" and vmid == 0:
|
||||
pass # after a boot the guest may not run (that is what is judged); these modes never touch it
|
||||
elif not isinstance(vmid, int) or isinstance(vmid, bool) or vmid <= 0:
|
||||
raise Refused("R11", f"vmid must be a positive integer, got {vmid!r}")
|
||||
rid = plan.get("release_id", "")
|
||||
if not isinstance(rid, str) or not re.match(r"^[A-Za-z0-9._:-]{1,80}$", rid):
|
||||
@@ -471,19 +508,24 @@ class Apply:
|
||||
if plan.get("allow_new"):
|
||||
raise Refused("R6", "allow_new is a slow-lane field; the fast lane never adds a package")
|
||||
select = plan.get("select", "listed")
|
||||
if select not in ("listed", "pending-fast", "pending-docker", "pending-pve"):
|
||||
if select not in ("listed", "pending-fast", "pending-docker", "pending-pve", "pending-kernel"):
|
||||
raise Refused("R11", f"unknown select {select!r}")
|
||||
if (select == "pending-docker") != (layer == "docker" and select != "listed"):
|
||||
if select == "pending-docker" or layer == "docker":
|
||||
raise Refused("R11", f"select {select!r} does not fit layer {layer!r}")
|
||||
if (select == "pending-pve") != (layer == "pve" and select != "listed"):
|
||||
raise Refused("R11", f"select {select!r} does not fit layer {layer!r}")
|
||||
if (select == "pending-kernel") != (layer == "kernel" and select != "listed"):
|
||||
raise Refused("R11", f"select {select!r} does not fit layer {layer!r}")
|
||||
ek = plan.get("expect_kver")
|
||||
if ek is not None and (layer != "kernel" or not isinstance(ek, str) or not KVER_RE.match(ek)):
|
||||
raise Refused("R11", f"expect_kver {ek!r} is not a kernel version of the kernel layer")
|
||||
pk = plan.get("packages", [])
|
||||
if not isinstance(pk, list):
|
||||
raise Refused("R11", "packages must be a list")
|
||||
if mode == "apply" and select == "listed" and not pk:
|
||||
raise Refused("R11", "packages must be a non-empty list in apply mode (select listed)")
|
||||
if select in ("pending-fast", "pending-docker", "pending-pve") and pk:
|
||||
if select in ("pending-fast", "pending-docker", "pending-pve", "pending-kernel") and pk:
|
||||
raise Refused("R11", f"select {select} takes no package list")
|
||||
seen = set()
|
||||
for e in pk:
|
||||
@@ -503,6 +545,13 @@ class Apply:
|
||||
continue
|
||||
if n in DOCKER_NAMES:
|
||||
raise Refused("R2", f"{n} is a Docker package — the slow lane (`11` §5.8), never in a {layer} plan")
|
||||
if layer == "kernel":
|
||||
if o != PVE_ORIGIN:
|
||||
raise Refused("R2", f"{n}: origin {o!r} is not {PVE_ORIGIN!r} (the kernel layer)")
|
||||
if not (KERNEL_UPGRADE_RE.match(n) or KERNEL_IMAGE_RE.match(n)):
|
||||
raise Refused("R23", f"{n} is not a kernel package (a kernel image, the kernel meta-packages, "
|
||||
f"proxmox-kernel-helper or pve-firmware)")
|
||||
continue
|
||||
if layer == "pve":
|
||||
if o != PVE_ORIGIN:
|
||||
raise Refused("R2", f"{n}: origin {o!r} is not {PVE_ORIGIN!r} (the pve layer)")
|
||||
@@ -759,6 +808,17 @@ class Apply:
|
||||
return v or "unknown"
|
||||
h = {"debian": first(["cat", "/etc/debian_version"]), "kernel_running": first(["uname", "-r"])}
|
||||
h["kernel_next_boot"], h["kernel_next_boot_source"] = self.kernel_next_boot()
|
||||
try:
|
||||
# the kernel lane (R-836): the default and the one-shot flag, read from grub.cfg and the ESP themselves
|
||||
kv = Kernel(self, {}).view()
|
||||
kv["setup_problems"] = Kernel(self, {}).setup_problems()
|
||||
h["kernel_lane"] = kv
|
||||
if kv.get("flag"):
|
||||
h["kernel_next_boot"], h["kernel_next_boot_source"] = kv["flag"], "felhom one-shot flag (once; then the default)"
|
||||
elif kv.get("default") not in (None, "unknown"):
|
||||
h["kernel_next_boot"], h["kernel_next_boot_source"] = kv["default"], "grub.cfg default"
|
||||
except Exception as e: # never cost the System page its other facts
|
||||
h["kernel_lane"] = {"error": str(e)[:200]}
|
||||
rc, out, _ = self.r.host(["apt-mark", "showhold"], 60)
|
||||
h["held"] = sorted(out.split()) if rc == 0 else None
|
||||
try:
|
||||
@@ -817,8 +877,8 @@ class Apply:
|
||||
|
||||
# ---------- target helpers ----------
|
||||
def x(self, argv, timeout=1800):
|
||||
"""Run in the TARGET layer: the guest via pct exec, or the host directly (host and pve)."""
|
||||
if self.layer in ("host", "pve"):
|
||||
"""Run in the TARGET layer: the guest via pct exec, or the host directly (host, pve and kernel)."""
|
||||
if self.layer in ("host", "pve", "kernel"):
|
||||
return self.r.host(argv, timeout)
|
||||
return self.r.guest(self.vmid, argv, timeout) # guest and docker both live in the customer guest
|
||||
|
||||
@@ -913,7 +973,7 @@ class Apply:
|
||||
def restart_needed(self):
|
||||
"""Processes still mapping deleted files, OUTSIDE containers (C11). Guest: outside docker; host: outside the
|
||||
LXC guests (the host's /proc shows guest processes too)."""
|
||||
skip = RESTART_SKIP_CGROUP["host" if self.layer in ("host", "pve") else "guest"]
|
||||
skip = RESTART_SKIP_CGROUP["host" if self.layer in ("host", "pve", "kernel") else "guest"]
|
||||
script = ('for p in /proc/[0-9]*; do grep -q "(deleted)" $p/maps 2>/dev/null || continue; '
|
||||
'grep -q "%s" $p/cgroup 2>/dev/null && continue; echo "${p#/proc/} $(cat $p/comm 2>/dev/null)"; done' % skip)
|
||||
rc, out, _ = self.x(["sh", "-c", script], timeout=120)
|
||||
@@ -987,8 +1047,12 @@ class Apply:
|
||||
return Bundle(self).from_plan(plan)
|
||||
if self.mode == "agent_update":
|
||||
return self.agent_update(plan)
|
||||
if self.layer in ("host", "pve"):
|
||||
if self.layer in ("host", "pve", "kernel"):
|
||||
self.check_appliance()
|
||||
if self.layer == "kernel" and self.mode != "health":
|
||||
# the kernel lane's own modes (R-836): most of them run while the guest is still starting after a boot, so
|
||||
# the guest check is the stage's and the reboot's own (Kernel.run), not every mode's
|
||||
return Kernel(self, plan).run()
|
||||
self.check_guest(self.vmid)
|
||||
log = self.r.log
|
||||
if self.mode == "live-restore-on":
|
||||
@@ -1364,6 +1428,422 @@ class Apply:
|
||||
self.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
|
||||
|
||||
|
||||
def _iso(t):
|
||||
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(t))
|
||||
|
||||
|
||||
def _parse_iso(s):
|
||||
try:
|
||||
return calendar.timegm(time.strptime(s, "%Y-%m-%dT%H:%M:%SZ"))
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
class Kernel:
|
||||
"""The kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11). Refusal codes: R3 (authority), R4 (removal),
|
||||
R6 (a package outside the step), R8 (space), R9 (locks), R12 (appliance), R20 (the box's boot setup cannot do a
|
||||
one-shot), R21 (the crash guard is tripped or saw an unclean boot within its window), R22 (the step's phase does not
|
||||
allow this mode), R23 (the kernel set is not one exact new kernel).
|
||||
|
||||
Modes (plan "mode", layer "kernel", lane "slow"):
|
||||
apply STAGE: install the kernel set, keep the GRUB default on the kernel the box runs, write the flag.
|
||||
Never reboots. select "pending-kernel" (ring 0, the root-owned mark) or "listed" (a signed
|
||||
os_kernel_step). expect_kver: the kernel the hub told the household about — any other is R23.
|
||||
kernel-reboot a STAGED step's reboot (the night leg, after the household was told): phase oneshot, then reboot.
|
||||
kernel-boot after a boot: what became of the step (judging | fell_back | self_reverted | revert_failed).
|
||||
kernel-good the one-shot boot was healthy: the new kernel becomes the GRUB default.
|
||||
kernel-revert the one-shot boot was NOT healthy: reboot ONCE into the old kernel (still the default).
|
||||
kernel-cancel drop a staged step: clear the flag (the package stays installed, the default never moved).
|
||||
kernel-status read only."""
|
||||
|
||||
def __init__(self, apply, plan):
|
||||
self.a, self.r, self.plan = apply, apply.r, plan
|
||||
self.report = apply.report
|
||||
self.log = apply.r.log
|
||||
|
||||
# ---------- reading the box ----------
|
||||
def running(self):
|
||||
rc, out, _ = self.r.host(["uname", "-r"], 30)
|
||||
v = out.strip() if rc == 0 else ""
|
||||
return v if KVER_RE.match(v) else ""
|
||||
|
||||
def state(self):
|
||||
try:
|
||||
s = json.loads(self.r.read_file(KERNEL_STATE))
|
||||
return s if isinstance(s, dict) else {}
|
||||
except (OSError, ValueError):
|
||||
return {}
|
||||
|
||||
def save_state(self, s):
|
||||
s["updated_at"] = _iso(self.r.now())
|
||||
self.r.put_file(KERNEL_STATE, (json.dumps(s, indent=2, sort_keys=True) + "\n").encode(), 0o644)
|
||||
|
||||
def flag(self):
|
||||
"""The one-shot flag: the kernel named, "" when the env block holds none, None when there is no env block."""
|
||||
rc, out, _ = self.r.host(["grub-editenv", ONESHOT_ENV, "list"], 30)
|
||||
if rc != 0:
|
||||
return None
|
||||
for l in out.splitlines():
|
||||
if l.startswith("felhom_next="):
|
||||
return l.split("=", 1)[1].strip()
|
||||
return ""
|
||||
|
||||
def grub_cfg(self):
|
||||
try:
|
||||
return self.r.read_file(GRUB_CFG)
|
||||
except OSError:
|
||||
return ""
|
||||
|
||||
@staticmethod
|
||||
def default_kver(cfg):
|
||||
"""The kernel grub.cfg boots by default (00_header's `set default=`), or "unknown"."""
|
||||
m = re.search(r'^\s*set default="(?:gnulinux-advanced-[^>"]*>)?gnulinux-([0-9][^"]*?-pve)-advanced-[^"]*"', cfg, re.M)
|
||||
return m.group(1) if m else "unknown"
|
||||
|
||||
@staticmethod
|
||||
def entry_id(cfg, kver):
|
||||
"""The GRUB_DEFAULT value that names kver's normal entry, read from grub.cfg itself (10_linux's ids)."""
|
||||
sub = re.search(r"\$menuentry_id_option '(gnulinux-advanced-[^']+)'", cfg)
|
||||
m = re.search(r"\$menuentry_id_option '(gnulinux-" + re.escape(kver) + r"-advanced-[^']+)'", cfg)
|
||||
if not m:
|
||||
return None
|
||||
return f"{sub.group(1)}>{m.group(1)}" if sub else m.group(1)
|
||||
|
||||
def setup_problems(self):
|
||||
"""R20: why this box cannot do a one-shot boot (empty = it can). Measured shape: UEFI, a vfat ESP at /boot/efi,
|
||||
/boot on the root filesystem, GRUB with fat + loadenv, the bundle's two generators, no hand pin."""
|
||||
why = []
|
||||
if not self.r.lexists("/sys/firmware/efi"):
|
||||
why.append("the box does not boot UEFI")
|
||||
rc, out, _ = self.r.host(["findmnt", "-n", "-o", "FSTYPE", ESP_MOUNT], 30)
|
||||
if rc != 0 or out.strip() != "vfat":
|
||||
why.append(f"{ESP_MOUNT} is not a mounted vfat ESP ({out.strip() or 'not mounted'})")
|
||||
rc, out, _ = self.r.host(["findmnt", "-n", "-o", "TARGET", "/boot"], 30)
|
||||
if rc == 0 and out.strip():
|
||||
why.append("/boot is a separate filesystem (the one-shot entries assume /boot on the root filesystem)")
|
||||
for m in ("fat", "loadenv"):
|
||||
if not self.r.lexists(f"/usr/lib/grub/x86_64-efi/{m}.mod"):
|
||||
why.append(f"GRUB has no {m} module")
|
||||
for p in (ONESHOT_SNIPPET, ONESHOT_ENTRIES):
|
||||
if not self.r.lexists(p):
|
||||
why.append(f"{p} is missing (the config bundle installs it)")
|
||||
if self.r.lexists(KERNEL_PIN_FILE):
|
||||
why.append("a kernel is pinned by hand (proxmox-boot-tool kernel pin) — the lane never fights it")
|
||||
return why
|
||||
|
||||
def guard(self):
|
||||
try:
|
||||
return json.loads(self.r.read_file(CRASH_GUARD_STATE))
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
def check_guard(self):
|
||||
"""R21: a kernel step only on a box whose crash guard is armed and saw no unclean boot within its window — so
|
||||
the step's own reboots (clean), one crash and one self-revert (clean) can never reach the 3rd unclean boot that
|
||||
leaves the box off (`11` §5.9). Pinned by test_felhom_crash_guard KernelStepCannotLeaveTheBoxOff."""
|
||||
g = self.guard()
|
||||
if not isinstance(g, dict):
|
||||
raise Refused("R21", f"no crash guard state ({CRASH_GUARD_STATE}) — a kernel step needs the guard")
|
||||
if g.get("tripped") or not g.get("armed"):
|
||||
raise Refused("R21", "the crash guard is tripped — no kernel step until it re-arms")
|
||||
if (g.get("unclean_boots_in_window") or 0) > 0:
|
||||
raise Refused("R21", f"{g.get('unclean_boots_in_window')} unclean boot(s) within the guard's window — wait")
|
||||
|
||||
def view(self, st=None, cfg=None):
|
||||
st = self.state() if st is None else st
|
||||
cfg = self.grub_cfg() if cfg is None else cfg
|
||||
return {"running": self.running() or "unknown", "default": self.default_kver(cfg), "flag": self.flag(),
|
||||
"phase": st.get("phase", "none"), "from": st.get("from"), "to": st.get("to"),
|
||||
"step_id": st.get("step_id"), "self_revert_used": bool(st.get("self_revert_used")), "vmid": st.get("vmid"),
|
||||
"staged_at": st.get("staged_at"), "rebooted_at": st.get("rebooted_at"),
|
||||
"result_at": st.get("result_at"), "reason": st.get("reason")}
|
||||
|
||||
# ---------- writing the box ----------
|
||||
def write_default(self, kver):
|
||||
"""Pin the GRUB default to kver's normal entry, regenerate grub.cfg, and PROVE it (the default read back)."""
|
||||
cfg = self.grub_cfg()
|
||||
eid = self.entry_id(cfg, kver)
|
||||
if not eid:
|
||||
raise Refused("R20", f"grub.cfg has no normal entry for {kver}")
|
||||
body = ("# felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily\n"
|
||||
f'GRUB_DEFAULT="{eid}"\n')
|
||||
self.r.put_file(KERNEL_DEFAULT_CFG, body.encode(), 0o644)
|
||||
rc, out, err = self.r.host(["update-grub"], 300)
|
||||
got = self.default_kver(self.grub_cfg())
|
||||
if rc != 0 or got != kver:
|
||||
raise Refused("R20", f"update-grub rc={rc}: the default reads {got}, not {kver}: {(out + err).strip()[-200:]}")
|
||||
self.log(f"os-apply: KERNEL default = {kver} (proved from grub.cfg)")
|
||||
|
||||
def set_flag(self, kver):
|
||||
self.r.host(["mkdir", "-p", os.path.dirname(ONESHOT_ENV)], 30)
|
||||
if self.flag() is None:
|
||||
rc, out, err = self.r.host(["grub-editenv", ONESHOT_ENV, "create"], 30)
|
||||
if rc != 0:
|
||||
raise Refused("R20", f"cannot create the one-shot env block on the ESP: {(out + err).strip()[-200:]}")
|
||||
rc, out, err = self.r.host(["grub-editenv", ONESHOT_ENV, "set", f"felhom_next={kver}"], 30)
|
||||
if rc != 0 or self.flag() != kver:
|
||||
raise Refused("R20", f"the one-shot flag did not read back as {kver}: {(out + err).strip()[-200:]}")
|
||||
|
||||
def clear_flag(self):
|
||||
if self.flag():
|
||||
self.r.host(["grub-editenv", ONESHOT_ENV, "unset", "felhom_next"], 30)
|
||||
|
||||
def reboot(self, why):
|
||||
self.log(f"os-apply: KERNEL REBOOT — {why}")
|
||||
rc, out, err = self.r.host(["systemctl", "reboot"], 60)
|
||||
self.report["reboot_rc"] = rc
|
||||
if rc != 0:
|
||||
self.report["failed"] = {"rc": 3, "step": "reboot", "reason": (out + err).strip()[-200:]}
|
||||
return 3
|
||||
return 0
|
||||
|
||||
# ---------- the modes ----------
|
||||
def run(self):
|
||||
mode = self.a.mode
|
||||
self.report["kernel_mode"] = mode
|
||||
if mode == "kernel-status":
|
||||
v = self.view()
|
||||
v["setup_problems"] = self.setup_problems()
|
||||
self.report["kernel"] = v
|
||||
return 0
|
||||
fn = {"apply": self.stage, "kernel-reboot": self.reboot_staged, "kernel-boot": self.after_boot,
|
||||
"kernel-good": self.good, "kernel-revert": self.revert, "kernel-cancel": self.cancel}[mode]
|
||||
rc = fn()
|
||||
self.report["kernel"] = self.view()
|
||||
return rc
|
||||
|
||||
def phase_is(self, st, *phases):
|
||||
if st.get("phase") not in phases:
|
||||
raise Refused("R22", f"the kernel step is {st.get('phase', 'none')!r}, not {' or '.join(phases)} — "
|
||||
f"{self.a.mode} does not apply")
|
||||
|
||||
def stage(self):
|
||||
a = self.a
|
||||
st = self.state()
|
||||
if st.get("phase") in KERNEL_ACTIVE:
|
||||
raise Refused("R22", f"a kernel step is already {st['phase']} ({st.get('from')} -> {st.get('to')})")
|
||||
last = _parse_iso(st.get("staged_at"))
|
||||
if last is not None and self.r.now() - last < KERNEL_MIN_GAP:
|
||||
raise Refused("R22", "a kernel step was staged within the last 20 hours — never two in one night")
|
||||
why = self.setup_problems()
|
||||
if why:
|
||||
raise Refused("R20", "; ".join(why))
|
||||
self.check_guard()
|
||||
a.check_guest(a.vmid)
|
||||
who, _ = a.docker_authority(self.plan, op_name=KERNEL_OP)
|
||||
self.report["authority"] = who
|
||||
old = self.running()
|
||||
if not old:
|
||||
raise Refused("R20", "the running kernel is not a Proxmox kernel version")
|
||||
self.log(f"os-apply: START release={self.plan.get('release_id')} layer=kernel lane=slow mode=apply "
|
||||
f"select={a.select} authority={who} running={old}")
|
||||
if a.apt_lock_held():
|
||||
raise Refused("R9", "another apt/dpkg holds the lock on the host")
|
||||
self.report["health_before"] = a.health()
|
||||
a.repair()
|
||||
rc, out, err = a.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
|
||||
if rc != 0:
|
||||
raise Refused("R7", f"apt-get update failed on the host: {(out + err).strip().splitlines()[-1:]}")
|
||||
inst = a.installed()
|
||||
if a.select == "pending-kernel":
|
||||
_, pend, _, _ = a.simulate(["dist-upgrade"])
|
||||
want = [(p["name"], p["to"]) for p in pend if p["from"] is not None and KERNEL_UPGRADE_RE.match(p["name"])
|
||||
and a.origin_name(p["origin"]) == {PVE_ORIGIN}]
|
||||
else:
|
||||
want = [(e["name"], e["version"]) for e in self.plan["packages"]]
|
||||
# upgrades of installed names (never a downgrade), and at most the listed new kernel image
|
||||
args, upg = [], {}
|
||||
for n, v in want:
|
||||
if n in inst:
|
||||
if a.dpkg_cmp(v, "gt", inst[n]):
|
||||
upg[n] = v
|
||||
args.append(f"{n}={v}")
|
||||
elif KERNEL_IMAGE_RE.match(n):
|
||||
args.append(f"{n}={v}")
|
||||
else:
|
||||
raise Refused("R6", f"{n} is not installed and is not a kernel image")
|
||||
if not args:
|
||||
self.report["upgraded"], self.report["outcome_hint"] = [], "nothing"
|
||||
self.log("os-apply: DONE rc=0 upgraded=0 (no pending kernel)")
|
||||
return 0
|
||||
rc, sim, remv, text = a.simulate(["install", "--no-install-recommends"] + args)
|
||||
if rc != 0:
|
||||
raise Refused("R7", "the simulation failed: " + (text.strip().splitlines()[-1] if text.strip() else ""))
|
||||
if remv:
|
||||
raise Refused("R4", f"the kernel step would remove {', '.join(remv[:5])}")
|
||||
images = []
|
||||
listed = dict(want)
|
||||
for p in sim:
|
||||
if a.origin_name(p["origin"]) != {PVE_ORIGIN}:
|
||||
raise Refused("R2", f"{p['name']} would come from {p['origin']}, not {PVE_ORIGIN!r}")
|
||||
m = KERNEL_IMAGE_RE.match(p["name"])
|
||||
if p["from"] is None:
|
||||
if not m:
|
||||
raise Refused("R6", f"the kernel step would add {p['name']}, which is not a kernel image")
|
||||
if a.select == "listed" and listed.get(p["name"]) != p["to"]:
|
||||
raise Refused("R23", f"the kernel step would add {p['name']}={p['to']}, not the signed set")
|
||||
images.append((m.group(1), p["to"]))
|
||||
continue
|
||||
if p["name"] not in upg or p["to"] != upg[p["name"]]:
|
||||
raise Refused("R6", f"the kernel step would touch {p['name']} ({p['to']}), which is not in the step")
|
||||
if not a.dpkg_cmp(p["to"], "gt", p["from"]):
|
||||
raise Refused("R5", f"{p['name']} would be downgraded {p['from']} -> {p['to']}")
|
||||
if len(images) > 1:
|
||||
raise Refused("R23", f"the kernel step would add {len(images)} kernels — one at a time")
|
||||
# the target kernel: the new image, else the series meta-package's version (its image is already installed)
|
||||
if images:
|
||||
new = images[0][0]
|
||||
if images[0][1] + "-pve" != new:
|
||||
raise Refused("R23", f"the image version {images[0][1]} does not name the kernel {new}")
|
||||
else:
|
||||
metas = [(n, v) for n, v in upg.items() if re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n)]
|
||||
if len(metas) != 1:
|
||||
self.report["upgraded"], self.report["outcome_hint"] = [], "nothing"
|
||||
self.log("os-apply: DONE rc=0 upgraded=0 (the pending set names no kernel to boot)")
|
||||
return 0
|
||||
new = metas[0][1] + "-pve"
|
||||
if not KVER_RE.match(new) or not a.dpkg_cmp(new[:-4], "gt", old[:-4]):
|
||||
raise Refused("R23", f"the kernel {new} is not newer than the running {old}")
|
||||
ek = self.plan.get("expect_kver")
|
||||
if ek and ek != new:
|
||||
raise Refused("R23", f"the step would boot {new}, but the household was told about {ek}")
|
||||
need = a.download_bytes(["install", "--no-install-recommends"] + args)
|
||||
free = a.free_bytes()
|
||||
if free >= 0 and free < max(MIN_FREE, 3 * need):
|
||||
raise Refused("R8", f"free space {free} B is below max(500 MB, 3 x download {need} B)")
|
||||
# 1. the default = the kernel the box RUNS (it booted healthily), proved from grub.cfg BEFORE the install
|
||||
default_before = self.default_kver(self.grub_cfg())
|
||||
self.write_default(old)
|
||||
# 2. install (the kernel's own postinst runs update-grub; our default file keeps the default on `old`)
|
||||
t0 = time.time()
|
||||
rc, out, err = a.x(APT_ENV + ["apt-get", "-y", "-q"] + DPKG_OPTS + ["install", "--no-install-recommends"] + args)
|
||||
a.x(["apt-get", "clean"])
|
||||
if rc != 0:
|
||||
_, aud, _ = a.x(["dpkg", "--audit"])
|
||||
self.report["failed"] = {"rc": rc, "step": "install", "dpkg_audit": (aud.strip().splitlines() or ["clean"])[0],
|
||||
"tail": (out + err).strip().splitlines()[-3:]}
|
||||
self.log(f"os-apply: FAILED rc={rc} step=install (the default stays {old}; no flag written)")
|
||||
return 3
|
||||
self.report["upgraded"] = [{"name": x.split("=", 1)[0], "version": x.split("=", 1)[1]} for x in args]
|
||||
self.report["seconds"] = round(time.time() - t0, 1)
|
||||
# 3. prove: the image is there, the default is still `old`, the one-shot entry for `new` exists
|
||||
cfg = self.grub_cfg()
|
||||
if f"felhom-oneshot-{new}" not in cfg or self.default_kver(cfg) != old:
|
||||
self.r.host(["update-grub"], 300)
|
||||
cfg = self.grub_cfg()
|
||||
for f in (f"/boot/vmlinuz-{new}", f"/boot/initrd.img-{new}"):
|
||||
if not self.r.lexists(f):
|
||||
self.report["failed"] = {"rc": 3, "step": "verify", "reason": f"{f} is missing after the install"}
|
||||
return 3
|
||||
if f"felhom-oneshot-{new}" not in cfg or "felhom_next" not in cfg:
|
||||
self.report["failed"] = {"rc": 3, "step": "verify", "reason": f"grub.cfg has no one-shot entry for {new}"}
|
||||
return 3
|
||||
if self.default_kver(cfg) != old:
|
||||
self.report["failed"] = {"rc": 3, "step": "verify",
|
||||
"reason": f"the install moved the default to {self.default_kver(cfg)} — no flag written"}
|
||||
return 3
|
||||
# 4. the flag — the ONLY thing that makes the next boot use `new`, and only once
|
||||
self.set_flag(new)
|
||||
self.save_state({"phase": "staged", "step_id": self.plan.get("release_id"), "from": old, "to": new, "vmid": a.vmid,
|
||||
"staged_at": _iso(self.r.now()), "authority": who, "default_before": default_before,
|
||||
"packages": self.report["upgraded"], "self_revert_used": False})
|
||||
self.report["reboot_needed"] = True
|
||||
self.log(f"os-apply: KERNEL STAGED {old} -> {new} (default {old}, one-shot flag {new}); upgraded={len(args)} "
|
||||
f"seconds={self.report['seconds']}")
|
||||
return 0
|
||||
|
||||
def reboot_staged(self):
|
||||
st = self.state()
|
||||
self.phase_is(st, "staged")
|
||||
if self.flag() != st.get("to"):
|
||||
raise Refused("R22", f"the one-shot flag reads {self.flag()!r}, not {st.get('to')!r}")
|
||||
if self.running() != st.get("from"):
|
||||
raise Refused("R22", f"the box runs {self.running()!r}, not the step's old kernel {st.get('from')!r}")
|
||||
cfg = self.grub_cfg()
|
||||
if self.default_kver(cfg) != st["from"] or f"felhom-oneshot-{st['to']}" not in cfg:
|
||||
raise Refused("R20", "grub.cfg no longer keeps the old default with a one-shot entry for the new kernel")
|
||||
if self.setup_problems():
|
||||
raise Refused("R20", "; ".join(self.setup_problems()))
|
||||
self.check_guard()
|
||||
self.a.check_guest(self.a.vmid)
|
||||
st["health_before"] = self.a.health()
|
||||
st["phase"], st["rebooted_at"] = "oneshot", _iso(self.r.now())
|
||||
self.save_state(st)
|
||||
return self.reboot(f"one-shot boot of {st['to']} (the default stays {st['from']})")
|
||||
|
||||
def after_boot(self):
|
||||
st = self.state()
|
||||
ph, run = st.get("phase"), self.running()
|
||||
old, new = st.get("from"), st.get("to")
|
||||
event = "none"
|
||||
if ph in ("oneshot", "staged", "judging") and run == new and new:
|
||||
if ph != "judging":
|
||||
st["phase"], st["judging_since"], event = "judging", _iso(self.r.now()), "judging"
|
||||
else:
|
||||
event = "judging"
|
||||
elif ph in ("oneshot", "judging") and run == old:
|
||||
# the new kernel did not come up, or crashed: GRUB already booted the default (the old kernel)
|
||||
self.clear_flag()
|
||||
st["phase"], st["result_at"], event = "fell_back", _iso(self.r.now()), "fell_back"
|
||||
st["reason"] = "the box came back on the old kernel by itself (the new one did not boot, or crashed)"
|
||||
elif ph == "reverting" and run == old:
|
||||
st["phase"], st["result_at"], event = "self_reverted", _iso(self.r.now()), "self_reverted"
|
||||
elif ph == "reverting" and run == new:
|
||||
st["phase"], st["result_at"], event = "revert_failed", _iso(self.r.now()), "revert_failed"
|
||||
st["reason"] = "the self-revert came back on the NEW kernel — never retried (one self-revert per step)"
|
||||
if event not in ("none",) and st.get("phase") != ph:
|
||||
self.save_state(st)
|
||||
self.log(f"os-apply: KERNEL AFTER-BOOT {ph} -> {st['phase']} running={run} ({old} -> {new})")
|
||||
self.report["kernel_event"] = event
|
||||
if event == "judging":
|
||||
self.report["health_before"] = st.get("health_before") # the agent judges the boot against it
|
||||
return 0
|
||||
|
||||
def good(self):
|
||||
st = self.state()
|
||||
self.phase_is(st, "judging")
|
||||
if self.running() != st.get("to"):
|
||||
raise Refused("R22", f"the box runs {self.running()!r}, not the new kernel {st.get('to')!r}")
|
||||
try:
|
||||
self.write_default(st["to"])
|
||||
except Refused:
|
||||
# put the old default back — the box must never be left without a proved default
|
||||
self.write_default(st["from"])
|
||||
raise
|
||||
self.clear_flag()
|
||||
st["phase"], st["result_at"] = "good", _iso(self.r.now())
|
||||
self.save_state(st)
|
||||
self.log(f"os-apply: KERNEL GOOD {st['to']} is the default now (was {st['from']})")
|
||||
return 0
|
||||
|
||||
def revert(self):
|
||||
st = self.state()
|
||||
self.phase_is(st, "judging")
|
||||
if st.get("self_revert_used"):
|
||||
raise Refused("R22", "this kernel step already used its one self-revert")
|
||||
if self.running() != st.get("to"):
|
||||
raise Refused("R22", f"the box runs {self.running()!r}, not the new kernel {st.get('to')!r}")
|
||||
self.clear_flag()
|
||||
cfg = self.grub_cfg()
|
||||
if self.default_kver(cfg) != st.get("from"):
|
||||
raise Refused("R20", f"the GRUB default reads {self.default_kver(cfg)}, not the old {st.get('from')} — "
|
||||
f"no self-revert into an unknown kernel")
|
||||
reason = self.plan.get("reason")
|
||||
st["reason"] = reason[:300] if isinstance(reason, str) else "the one-shot boot was not healthy"
|
||||
st["self_revert_used"], st["phase"], st["reverted_at"] = True, "reverting", _iso(self.r.now())
|
||||
self.save_state(st)
|
||||
return self.reboot(f"self-revert to {st['from']}: {st['reason']}")
|
||||
|
||||
def cancel(self):
|
||||
st = self.state()
|
||||
self.phase_is(st, "staged")
|
||||
self.clear_flag()
|
||||
st["phase"], st["result_at"], st["reason"] = "cancelled", _iso(self.r.now()), "cancelled before the reboot"
|
||||
self.save_state(st)
|
||||
self.log(f"os-apply: KERNEL CANCELLED {st.get('to')} (installed, never the default; flag cleared)")
|
||||
return 0
|
||||
|
||||
|
||||
class Bundle:
|
||||
"""The config bundle (R-840, `11` §5.4.2): every root-owned file the installer's step 5 writes, installed as ONE
|
||||
signed unit. Every check runs before the first write; a failed write or a failed self-check puts every previous
|
||||
|
||||
@@ -139,5 +139,62 @@ class Guard(unittest.TestCase):
|
||||
self.assertEqual(mode, 0o644)
|
||||
|
||||
|
||||
class KernelStepCannotLeaveTheBoxOff(unittest.TestCase):
|
||||
"""R-836 / `11` §5.11 Part B 4: a kernel step's planned reboot, one crash and one self-revert cannot add up to the
|
||||
box staying off. The wrapper starts a step only when the guard is armed with NO unclean boot in its window
|
||||
(felhom-os-apply Kernel.check_guard, R21); the planned reboot and the self-revert are orderly (`systemctl reboot` —
|
||||
the clean-stop marker); so the step adds at most ONE unclean boot, and the box stays off only after the LIMIT-th
|
||||
(3rd) within the hour. Red-proof: audits/kernel-lane-2026-10-07/A/redproof.txt."""
|
||||
|
||||
def setUp(self):
|
||||
self.d = tempfile.TemporaryDirectory()
|
||||
self.e = FakeEnv(self.d.name)
|
||||
cg.main(["x", "boot"], self.e)
|
||||
s = self.e.state()
|
||||
self.assertTrue(s["armed"])
|
||||
self.assertEqual(s["unclean_boots_in_window"], 0, "the wrapper's precondition (R21)")
|
||||
|
||||
def tearDown(self):
|
||||
self.d.cleanup()
|
||||
|
||||
def planned(self, minutes):
|
||||
cg.main(["x", "clean-stop"], self.e)
|
||||
self.e.t += minutes * 60
|
||||
cg.main(["x", "boot"], self.e)
|
||||
|
||||
def crash(self, minutes):
|
||||
self.e.t += minutes * 60
|
||||
cg.main(["x", "boot"], self.e)
|
||||
|
||||
def test_planned_reboot_one_crash_one_self_revert(self):
|
||||
self.planned(2) # the step's one-shot reboot (orderly)
|
||||
self.crash(3) # the new kernel crashes after the guard ran; the box restarts (panic=10)
|
||||
self.planned(2) # the self-revert (orderly)
|
||||
s = self.e.state()
|
||||
self.assertFalse(s["tripped"], s)
|
||||
self.assertTrue(s["armed"])
|
||||
self.assertEqual(self.e.panic(), 10, "the box still restarts after a crash")
|
||||
self.assertEqual(s["unclean_boots_in_window"], 1, "the step added exactly one unclean boot")
|
||||
|
||||
def test_a_panic_before_userspace_is_not_even_counted(self):
|
||||
# the one-shot kernel panics before the guard's unit runs (measured: rdinit= and init= missing): the planned
|
||||
# reboot's clean-stop marker is still there when the old kernel boots, so this boot counts as clean.
|
||||
cg.main(["x", "clean-stop"], self.e)
|
||||
self.e.t += 120 # the panicking boot: no userspace, the guard never ran
|
||||
cg.main(["x", "boot"], self.e)
|
||||
s = self.e.state()
|
||||
self.assertEqual(s["unclean_boots_in_window"], 0, s)
|
||||
self.assertEqual(self.e.panic(), 10)
|
||||
|
||||
def test_the_box_stays_off_only_after_two_more_crashes_than_the_step_makes(self):
|
||||
self.planned(2)
|
||||
self.crash(3) # the step's one crash
|
||||
self.planned(2) # the self-revert
|
||||
self.crash(5) # an UNRELATED crash within the hour: the guard trips (the 3rd would leave it off)
|
||||
s = self.e.state()
|
||||
self.assertTrue(s["tripped"])
|
||||
self.assertEqual(s["unclean_boots_in_window"], 2, "two unclean boots: one from the step, one not")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -1476,5 +1476,517 @@ class PVELane(unittest.TestCase):
|
||||
self.assertEqual((rc, rep["refused"]["code"]), (2, "R11"), rep)
|
||||
|
||||
|
||||
# ---------- the kernel lane (R-836, `09` §3 decision 172, `11` §5.11) ----------
|
||||
U = "1af1fcc6-639c-416b-a7e5-c4470d41a502"
|
||||
OLD, NEW = "7.0.2-6-pve", "7.0.14-22-pve"
|
||||
KERNEL_SET = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-22", "origin": "Proxmox Debian Repository"},
|
||||
{"name": "proxmox-kernel-7.0.14-22-pve-signed", "version": "7.0.14-22", "origin": "Proxmox Debian Repository"}]
|
||||
|
||||
|
||||
class KFake(Fake):
|
||||
"""A Proxmox host with GRUB on UEFI: /boot on the root LV, a vfat ESP, the bundle's two generators. update-grub is
|
||||
emulated like the real 10_linux: WITHOUT the felhom default file the NEWEST kernel becomes the default (measured,
|
||||
R-836 — that is the defect the lane exists for); with it, the kernel it names."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__()
|
||||
self.running = OLD
|
||||
self.boot = {OLD}
|
||||
self.efi, self.esp_fs, self.boot_mount, self.mods, self.snippets, self.pin = True, "vfat", "", True, True, False
|
||||
self.env = None # None = no env block on the ESP; else {"felhom_next": ...}
|
||||
self.tree = {} # files put_file wrote
|
||||
self.reboots = []
|
||||
self.update_grubs = 0
|
||||
self.grub_runs_in_postinst = True
|
||||
self.installed.update({"proxmox-kernel-7.0": "7.0.2-6", "proxmox-default-kernel": "2.1.0",
|
||||
"pve-firmware": "3.18-3", "proxmox-kernel-7.0.2-6-pve-signed": "7.0.2-6",
|
||||
"pve-manager": "9.2.21"})
|
||||
self.live.update({"proxmox-kernel-7.0": {"7.0.14-22", "7.0.2-6"},
|
||||
"proxmox-kernel-7.0.14-22-pve-signed": {"7.0.14-22"},
|
||||
"proxmox-kernel-7.0.14-23-pve-signed": {"7.0.14-23"},
|
||||
"pve-firmware": {"3.18-7", "3.18-3"}})
|
||||
self.origins = {}
|
||||
self.kernel_pending = ["Inst pve-firmware [3.18-3] (3.18-7 Proxmox Debian Repository:stable [all])",
|
||||
"Inst proxmox-kernel-7.0.14-22-pve-signed (7.0.14-22 Proxmox Debian Repository:stable [amd64])",
|
||||
"Inst proxmox-kernel-7.0 [7.0.2-6] (7.0.14-22 Proxmox Debian Repository:stable [amd64])",
|
||||
"Inst pve-manager [9.2.21] (9.2.22 Proxmox Debian Repository:stable [amd64])",
|
||||
"Inst libc6 [2.41-12+deb13u3] (2.41-12+deb13u4 Debian:13.7/stable [amd64])"]
|
||||
self.plan = {"release_id": "kernel-t1", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": "apply",
|
||||
"select": "pending-kernel", "packages": []}
|
||||
self.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": True})
|
||||
self.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": True, "tripped": False, "unclean_boots_in_window": 0})
|
||||
self.cfg = self.render()
|
||||
|
||||
# -- GRUB --
|
||||
def newest(self):
|
||||
best = None
|
||||
for k in self.boot:
|
||||
if best is None or dpkg_cmp(k[:-4], "gt", best[:-4]):
|
||||
best = k
|
||||
return best
|
||||
|
||||
def render(self):
|
||||
d = self.tree.get(osapply.KERNEL_DEFAULT_CFG)
|
||||
if d:
|
||||
dflt = re.search(r'GRUB_DEFAULT="([^"]+)"', d).group(1)
|
||||
else:
|
||||
dflt = f"gnulinux-advanced-{U}>gnulinux-{self.newest()}-advanced-{U}"
|
||||
cfg = ('if [ "${next_entry}" ] ; then\n set default="${next_entry}"\nelse\n'
|
||||
f' set default="{dflt}"\nfi\n'
|
||||
f"submenu 'Advanced options' $menuentry_id_option 'gnulinux-advanced-{U}' {{\n")
|
||||
for k in sorted(self.boot):
|
||||
cfg += f" menuentry 'Proxmox VE, with Linux {k}' $menuentry_id_option 'gnulinux-{k}-advanced-{U}' {{ }}\n"
|
||||
cfg += "}\n"
|
||||
if self.snippets:
|
||||
cfg += "### BEGIN /etc/grub.d/01_felhom_oneshot ###\nload_env felhom_next\n"
|
||||
cfg += "".join(f"menuentry 'Felhom one-shot: {k}' --id felhom-oneshot-{k} {{ }}\n" for k in sorted(self.boot))
|
||||
return cfg
|
||||
|
||||
def lexists(self, p):
|
||||
if p == "/sys/firmware/efi":
|
||||
return self.efi
|
||||
if p.startswith("/usr/lib/grub/x86_64-efi/"):
|
||||
return self.mods
|
||||
if p in (osapply.ONESHOT_SNIPPET, osapply.ONESHOT_ENTRIES):
|
||||
return self.snippets
|
||||
if p == osapply.KERNEL_PIN_FILE:
|
||||
return self.pin
|
||||
m = re.match(r"^/boot/(vmlinuz|initrd\.img)-(.+)$", p)
|
||||
if m:
|
||||
return m.group(2) in self.boot
|
||||
return p in self.tree
|
||||
|
||||
def put_file(self, path, data, mode):
|
||||
self.tree[path] = data.decode()
|
||||
|
||||
def read_file(self, p):
|
||||
if p == osapply.GRUB_CFG:
|
||||
return self.cfg
|
||||
if p in self.tree:
|
||||
return self.tree[p]
|
||||
return super().read_file(p)
|
||||
|
||||
def host(self, argv, timeout=600, stdin=None):
|
||||
if argv[0] == "uname":
|
||||
self.calls.append(("host", argv))
|
||||
return 0, self.running + "\n", ""
|
||||
if argv[0] == "findmnt":
|
||||
self.calls.append(("host", argv))
|
||||
if argv[-1] == osapply.ESP_MOUNT:
|
||||
return (0, self.esp_fs + "\n", "") if self.esp_fs else (1, "", "")
|
||||
return (0, self.boot_mount + "\n", "") if self.boot_mount else (1, "", "")
|
||||
if argv[0] == "grub-editenv":
|
||||
self.calls.append(("host", argv))
|
||||
if argv[2] == "list":
|
||||
return (1, "", "no such file") if self.env is None else \
|
||||
(0, "".join(f"{k}={v}\n" for k, v in self.env.items()), "")
|
||||
if argv[2] == "create":
|
||||
self.env = {}
|
||||
return 0, "", ""
|
||||
if argv[2] == "set":
|
||||
k, v = argv[3].split("=", 1)
|
||||
self.env[k] = v
|
||||
return 0, "", ""
|
||||
if argv[2] == "unset":
|
||||
self.env.pop(argv[3], None)
|
||||
return 0, "", ""
|
||||
if argv[0] == "update-grub":
|
||||
self.calls.append(("host", argv))
|
||||
self.update_grubs += 1
|
||||
self.cfg = self.render()
|
||||
return 0, "", ""
|
||||
if argv[0] == "mkdir":
|
||||
self.calls.append(("host", argv))
|
||||
return 0, "", ""
|
||||
if argv[:2] == ["systemctl", "reboot"]:
|
||||
self.calls.append(("host", argv))
|
||||
self.reboots.append(self.running)
|
||||
return 0, "", ""
|
||||
return super().host(argv, timeout, stdin)
|
||||
|
||||
def emulate(self, argv):
|
||||
a = [x for x in argv if not re.match(r"^[A-Z_]+=", x) and x != "env"]
|
||||
if a[0] == "apt-get" and "install" in a and "-s" not in a and "--print-uris" not in a and "-f" not in a:
|
||||
if self.install_rc:
|
||||
return self.install_rc, "", "E: boom"
|
||||
for x in a:
|
||||
if "=" in x and not x.startswith("-") and "::" not in x:
|
||||
n, v = x.split("=", 1)
|
||||
self.installed[n] = v
|
||||
m = re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n)
|
||||
if m:
|
||||
self.installed[f"proxmox-kernel-{v}-pve-signed"] = v
|
||||
if m or osapply.KERNEL_IMAGE_RE.match(n):
|
||||
self.boot.add(v + "-pve")
|
||||
if self.grub_runs_in_postinst:
|
||||
self.cfg = self.render() # the kernel's postinst runs update-grub (zz-update-grub)
|
||||
return 0, "Setting up proxmox-kernel ...\n", ""
|
||||
return super().emulate(argv)
|
||||
|
||||
def sim(self, a):
|
||||
if "--print-uris" in a:
|
||||
return 0, "", ""
|
||||
if "dist-upgrade" in a:
|
||||
return 0, "\n".join(self.kernel_pending) + "\n", ""
|
||||
out, named = "", set()
|
||||
for x in a:
|
||||
if "=" in x and not x.startswith("-") and "::" not in x:
|
||||
named.add(x.split("=", 1)[0])
|
||||
for x in a:
|
||||
if "=" in x and not x.startswith("-") and "::" not in x:
|
||||
n, v = x.split("=", 1)
|
||||
if v not in self.avail(n):
|
||||
return 100, "", f"E: Version '{v}' for '{n}' was not found"
|
||||
o = self.origins.get(n, PVE)
|
||||
out += f"Inst {n} [{self.installed[n]}] ({v} {o} [amd64])\n" if n in self.installed else f"Inst {n} ({v} {o} [amd64])\n"
|
||||
if re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n):
|
||||
img = f"proxmox-kernel-{v}-pve-signed"
|
||||
if img not in self.installed and img not in named:
|
||||
out += f"Inst {img} ({v} {PVE} [amd64])\n"
|
||||
out += "".join(l + "\n" for l in self.extra_sim)
|
||||
return 0, out, ""
|
||||
|
||||
|
||||
def kfake(**kw):
|
||||
f = KFake()
|
||||
for k, v in kw.items():
|
||||
setattr(f, k, v)
|
||||
return f
|
||||
|
||||
|
||||
def kmode(f, mode, **extra):
|
||||
f.plan = {"release_id": "kernel-t1", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": mode, "packages": []}
|
||||
f.plan.update(extra)
|
||||
return run(f)
|
||||
|
||||
|
||||
def staged(f=None):
|
||||
f = f or kfake()
|
||||
rc, rep = run(f)
|
||||
assert rc == 0, rep
|
||||
return f
|
||||
|
||||
|
||||
class KernelLane(unittest.TestCase):
|
||||
"""R-836 / `09` §3 decision 172: stage a kernel once through the ESP flag; the default moves only after a healthy
|
||||
one-shot boot. Red-proof: audits/kernel-lane-2026-10-07/A/redproof.txt."""
|
||||
|
||||
def refused(self, f, code, mode=None, **extra):
|
||||
rc, rep = kmode(f, mode, **extra) if mode else run(f)
|
||||
self.assertEqual(rc, 2, rep)
|
||||
self.assertEqual(rep["refused"]["code"], code, rep)
|
||||
return rep
|
||||
|
||||
# --- the four red tests the brief names (Part A 3) ---
|
||||
def test_installing_a_kernel_does_not_change_the_grub_default(self):
|
||||
f = kfake()
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertIn(NEW, f.boot, "the new kernel was installed")
|
||||
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD,
|
||||
"the default must stay the kernel the box runs (R-836: an install made the new one default)")
|
||||
self.assertIn(f"gnulinux-{OLD}-advanced-{U}", f.tree[osapply.KERNEL_DEFAULT_CFG])
|
||||
self.assertEqual(f.env, {"felhom_next": NEW}, "the ONLY way to the new kernel is the one-shot flag")
|
||||
self.assertEqual(f.reboots, [], "staging never reboots")
|
||||
st = json.loads(f.tree[osapply.KERNEL_STATE])
|
||||
self.assertEqual((st["phase"], st["from"], st["to"]), ("staged", OLD, NEW))
|
||||
self.assertEqual(rep["kernel"]["default"], OLD)
|
||||
|
||||
def test_a_box_without_the_esp_flag_is_refused(self):
|
||||
for attr, val, frag in (("esp_fs", "ext4", "vfat"), ("efi", False, "UEFI"), ("snippets", False, "bundle"),
|
||||
("mods", False, "module"), ("boot_mount", "/boot", "separate"), ("pin", True, "pinned")):
|
||||
f = kfake(**{attr: val})
|
||||
rep = self.refused(f, "R20")
|
||||
self.assertIn(frag, rep["refused"]["reason"], attr)
|
||||
self.assertNotIn(NEW, f.boot, f"{attr}: nothing may be installed on a refusal")
|
||||
self.assertIsNone(f.env, f"{attr}: no flag on a refusal")
|
||||
|
||||
def test_a_reboot_without_the_flag_is_refused(self):
|
||||
f = staged()
|
||||
f.env = {"felhom_next": ""}
|
||||
self.refused(f, "R22", mode="kernel-reboot")
|
||||
self.assertEqual(f.reboots, [])
|
||||
|
||||
def test_a_kernel_outside_the_approved_set_is_refused(self):
|
||||
# a signed set that names only the meta-package: the image the sources pull in was never signed for
|
||||
f = kfake()
|
||||
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
|
||||
f.plan.update(select="listed", packages=[dict(KERNEL_SET[0])],
|
||||
signed=signed_job(packages=[KERNEL_SET[0]], op="os_kernel_step"))
|
||||
self.refused(f, "R23")
|
||||
self.assertNotIn(NEW, f.boot)
|
||||
# a signed set names 7.0.14-22; the sources would ALSO bring 7.0.14-23
|
||||
f2 = kfake()
|
||||
f2.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
|
||||
f2.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
|
||||
signed=signed_job(packages=KERNEL_SET, op="os_kernel_step"))
|
||||
f2.extra_sim = ["Inst proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23 Proxmox Debian Repository:stable [amd64])"]
|
||||
self.refused(f2, "R23")
|
||||
# a name that is not a kernel package at all
|
||||
g = kfake()
|
||||
g.plan.update(select="listed", packages=[{"name": "pve-manager", "version": "9.2.22", "origin": "Proxmox Debian Repository"}])
|
||||
self.refused(g, "R23")
|
||||
# the household was told about another kernel
|
||||
h = kfake()
|
||||
h.plan["expect_kver"] = "7.0.14-23-pve"
|
||||
self.refused(h, "R23")
|
||||
self.assertNotIn(NEW, h.boot)
|
||||
|
||||
def test_the_snippet_clears_the_flag_on_use(self):
|
||||
"""01_felhom_oneshot: the flag is copied, CLEARED and SAVED before any `set default`, all inside the one branch
|
||||
that runs only when the flag is set; a default is set only for an installed kernel's own entry."""
|
||||
text = (HERE / "felhom-grub-oneshot.sh").read_text()
|
||||
body = text[text.index("cat <<EOF"):]
|
||||
i_copy = body.index('set felhom_boot="\\${felhom_next}"')
|
||||
i_clear = body.index("set felhom_next=\n")
|
||||
i_save = body.index("save_env -f (\\$felhom_esp)/EFI/felhom/oneshot.env felhom_next")
|
||||
i_default = body.index('set default="felhom-oneshot-%s"')
|
||||
self.assertLess(i_copy, i_clear)
|
||||
self.assertLess(i_clear, i_save)
|
||||
self.assertLess(i_save, i_default, "the flag must be cleared on disk BEFORE GRUB boots anything")
|
||||
self.assertIn('if [ "${felhom_boot}" = "%s" ]', body, "a default only for a kernel that is installed")
|
||||
self.assertIn(osapply.ONESHOT_ENV[len(osapply.ESP_MOUNT):], text, "the wrapper and GRUB name the same env file")
|
||||
|
||||
# --- the rest of the stage ---
|
||||
def test_option_c_options_are_on_the_one_shot_entry_only(self):
|
||||
text = (HERE / "felhom-grub-oneshot-entries.sh").read_text()
|
||||
self.assertIn(osapply.ONESHOT_ARGS, text)
|
||||
self.assertIn("--id felhom-oneshot-$k", text)
|
||||
self.assertNotIn("set default", text, "the entries file never sets the default")
|
||||
|
||||
def test_both_generators_ride_the_bundle(self):
|
||||
self.assertEqual(osapply.BUNDLE_DESTS[osapply.ONESHOT_SNIPPET][1:4], ("felhom-grub-oneshot.sh", 0o755, "sh"))
|
||||
self.assertEqual(osapply.BUNDLE_DESTS[osapply.ONESHOT_ENTRIES][1:4], ("felhom-grub-oneshot-entries.sh", 0o755, "sh"))
|
||||
|
||||
def test_ring0_pending_kernel_takes_only_the_kernel_set(self):
|
||||
f = kfake()
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(rep["authority"], "ring0")
|
||||
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["proxmox-kernel-7.0", "pve-firmware"])
|
||||
self.assertEqual(f.installed["pve-manager"], "9.2.21", "a Proxmox userspace package is the pve lane's")
|
||||
self.assertEqual(f.installed["libc6"], "2.41-12+deb13u3", "a Debian package is the fast lane's")
|
||||
|
||||
def test_signed_listed_set_is_installed(self):
|
||||
f = kfake()
|
||||
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
|
||||
f.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
|
||||
signed=signed_job(packages=KERNEL_SET, op="os_kernel_step"))
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(rep["authority"], "signed")
|
||||
self.assertEqual(f.env, {"felhom_next": NEW})
|
||||
|
||||
def test_no_authority_and_the_wrong_op_are_refused(self):
|
||||
f = kfake()
|
||||
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
|
||||
self.refused(f, "R3")
|
||||
g = kfake()
|
||||
g.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
|
||||
g.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
|
||||
signed=signed_job(packages=KERNEL_SET, op="os_pve_step"))
|
||||
self.refused(g, "R3")
|
||||
|
||||
def test_fast_lane_byo_and_wrong_select_are_refused(self):
|
||||
f = kfake()
|
||||
f.plan["lane"] = "fast"
|
||||
self.refused(f, "R3")
|
||||
g = kfake()
|
||||
g.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
|
||||
self.refused(g, "R12")
|
||||
h = kfake()
|
||||
h.plan["select"] = "pending-pve"
|
||||
self.refused(h, "R11")
|
||||
k = kfake()
|
||||
k.plan.update(layer="host", lane="fast", mode="kernel-status")
|
||||
self.refused(k, "R11")
|
||||
|
||||
def test_the_crash_guard_must_be_armed_and_quiet(self):
|
||||
f = kfake()
|
||||
f.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": False, "tripped": True, "unclean_boots_in_window": 2})
|
||||
self.refused(f, "R21")
|
||||
g = kfake()
|
||||
g.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": True, "tripped": False, "unclean_boots_in_window": 1})
|
||||
self.refused(g, "R21")
|
||||
h = kfake()
|
||||
del h.files[osapply.CRASH_GUARD_STATE]
|
||||
self.refused(h, "R21")
|
||||
self.assertNotIn(NEW, h.boot)
|
||||
|
||||
def test_never_two_steps_in_one_night(self):
|
||||
f = staged()
|
||||
self.refused(f, "R22") # still staged
|
||||
g = staged()
|
||||
st = json.loads(g.tree[osapply.KERNEL_STATE])
|
||||
st["phase"] = "good"
|
||||
g.tree[osapply.KERNEL_STATE] = json.dumps(st)
|
||||
g.clock += 3600
|
||||
self.refused(g, "R22") # done, but within 20 h
|
||||
|
||||
def test_removal_new_non_kernel_two_kernels_and_older_are_refused(self):
|
||||
f = kfake(extra_sim=["Remv pve-firmware [3.18-3]"])
|
||||
self.refused(f, "R4")
|
||||
g = kfake(extra_sim=["Inst proxmox-new-thing (1.0 Proxmox Debian Repository:stable [all])"])
|
||||
self.refused(g, "R6")
|
||||
h = kfake(extra_sim=["Inst proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23 Proxmox Debian Repository:stable [amd64])"])
|
||||
self.refused(h, "R23")
|
||||
k = kfake(running="7.0.14-23-pve")
|
||||
k.boot = {"7.0.14-23-pve"}
|
||||
k.cfg = k.render()
|
||||
self.refused(k, "R23")
|
||||
m = kfake()
|
||||
m.origins = {"proxmox-kernel-7.0": DEB}
|
||||
self.refused(m, "R2")
|
||||
|
||||
# R-898: ring 0 stages EXACTLY the told kernel (select listed, the set derived from it) — even when the sources
|
||||
# offer a newer one by night; a told version that can no longer be installed is refused BEFORE any change (R7).
|
||||
def test_ring0_listed_installs_the_told_kernel_not_the_newest(self):
|
||||
f = kfake()
|
||||
f.live["proxmox-kernel-7.0"] = {"7.0.14-20", "7.0.14-22", "7.0.2-6"}
|
||||
f.live["proxmox-kernel-7.0.14-20-pve-signed"] = {"7.0.14-20"}
|
||||
told = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"},
|
||||
{"name": "proxmox-kernel-7.0.14-20-pve-signed", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"}]
|
||||
f.plan.update(select="listed", packages=told, expect_kver="7.0.14-20-pve")
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(rep["authority"], "ring0")
|
||||
self.assertEqual(f.env, {"felhom_next": "7.0.14-20-pve"})
|
||||
self.assertEqual(f.installed["proxmox-kernel-7.0"], "7.0.14-20")
|
||||
self.assertNotIn("7.0.14-22-pve", f.boot)
|
||||
|
||||
def test_ring0_told_kernel_gone_is_refused_before_any_change(self):
|
||||
f = kfake()
|
||||
f.live["proxmox-kernel-7.0"] = {"7.0.14-22"} # 7.0.14-20 is no longer in the archive
|
||||
told = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"},
|
||||
{"name": "proxmox-kernel-7.0.14-20-pve-signed", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"}]
|
||||
f.plan.update(select="listed", packages=told, expect_kver="7.0.14-20-pve")
|
||||
rep = self.refused(f, "R7")
|
||||
self.assertIsNone(f.env)
|
||||
self.assertNotIn(osapply.KERNEL_DEFAULT_CFG, f.tree, "refused before the default was even pinned")
|
||||
self.assertEqual(f.installed["proxmox-kernel-7.0"], "7.0.2-6")
|
||||
|
||||
def test_nothing_pending_changes_nothing(self):
|
||||
f = kfake(kernel_pending=[])
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(rep["upgraded"], [])
|
||||
self.assertIsNone(f.env)
|
||||
self.assertNotIn(osapply.KERNEL_STATE, f.tree)
|
||||
|
||||
def test_a_failed_install_writes_no_flag(self):
|
||||
f = kfake(install_rc=100)
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 3, rep)
|
||||
self.assertIsNone(f.env)
|
||||
self.assertNotIn(osapply.KERNEL_STATE, f.tree)
|
||||
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
|
||||
|
||||
def test_a_postinst_without_update_grub_is_regenerated_and_proved(self):
|
||||
f = kfake(grub_runs_in_postinst=False)
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertIn(f"felhom-oneshot-{NEW}", f.cfg)
|
||||
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
|
||||
|
||||
# --- the reboot, the boot, good / revert ---
|
||||
def test_reboot_of_a_staged_step(self):
|
||||
f = staged()
|
||||
rc, rep = kmode(f, "kernel-reboot")
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(f.reboots, [OLD])
|
||||
st = json.loads(f.tree[osapply.KERNEL_STATE])
|
||||
self.assertEqual(st["phase"], "oneshot")
|
||||
self.assertIn("host_services", st["health_before"])
|
||||
|
||||
def test_reboot_needs_a_staged_step(self):
|
||||
self.refused(kfake(), "R22", mode="kernel-reboot")
|
||||
|
||||
def boot_into(self, f, kver):
|
||||
f.running = kver
|
||||
if f.env and f.env.get("felhom_next"):
|
||||
f.env["felhom_next"] = "" # GRUB cleared it (01_felhom_oneshot)
|
||||
return kmode(f, "kernel-boot")
|
||||
|
||||
def test_healthy_one_shot_becomes_the_default(self):
|
||||
f = staged()
|
||||
kmode(f, "kernel-reboot")
|
||||
rc, rep = self.boot_into(f, NEW)
|
||||
self.assertEqual((rc, rep["kernel_event"], rep["kernel"]["phase"]), (0, "judging", "judging"), rep)
|
||||
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD, "judging does not move the default")
|
||||
rc, rep = kmode(f, "kernel-good")
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(osapply.Kernel.default_kver(f.cfg), NEW)
|
||||
self.assertEqual(rep["kernel"]["phase"], "good")
|
||||
|
||||
def test_a_panic_falls_back_and_is_recorded(self):
|
||||
f = staged()
|
||||
kmode(f, "kernel-reboot")
|
||||
rc, rep = self.boot_into(f, OLD)
|
||||
self.assertEqual((rc, rep["kernel_event"]), (0, "fell_back"), rep)
|
||||
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
|
||||
self.refused(f, "R22", mode="kernel-good")
|
||||
|
||||
def test_one_self_revert_then_never_again(self):
|
||||
f = staged()
|
||||
kmode(f, "kernel-reboot")
|
||||
self.boot_into(f, NEW)
|
||||
rc, rep = kmode(f, "kernel-revert", reason="the customer guest is not running")
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertEqual(f.reboots, [OLD, NEW])
|
||||
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD, "the self-revert boots the default, the old kernel")
|
||||
rc, rep = self.boot_into(f, OLD)
|
||||
self.assertEqual(rep["kernel_event"], "self_reverted")
|
||||
# the same step can never revert again
|
||||
st = json.loads(f.tree[osapply.KERNEL_STATE])
|
||||
st["phase"] = "judging"
|
||||
f.tree[osapply.KERNEL_STATE] = json.dumps(st)
|
||||
f.running = NEW
|
||||
self.refused(f, "R22", mode="kernel-revert")
|
||||
self.assertEqual(len(f.reboots), 2)
|
||||
|
||||
def test_a_revert_that_comes_back_new_is_never_retried(self):
|
||||
f = staged()
|
||||
kmode(f, "kernel-reboot")
|
||||
self.boot_into(f, NEW)
|
||||
kmode(f, "kernel-revert")
|
||||
rc, rep = self.boot_into(f, NEW)
|
||||
self.assertEqual(rep["kernel_event"], "revert_failed")
|
||||
self.refused(f, "R22", mode="kernel-revert")
|
||||
|
||||
def test_revert_refuses_an_unknown_default(self):
|
||||
f = staged()
|
||||
kmode(f, "kernel-reboot")
|
||||
self.boot_into(f, NEW)
|
||||
f.tree[osapply.KERNEL_DEFAULT_CFG] = f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-9.9.9-1-pve-advanced-{U}"\n'
|
||||
f.cfg = f.render()
|
||||
self.refused(f, "R20", mode="kernel-revert")
|
||||
self.assertEqual(len(f.reboots), 1)
|
||||
|
||||
def test_cancel_clears_the_flag(self):
|
||||
f = staged()
|
||||
rc, rep = kmode(f, "kernel-cancel")
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertFalse(f.env.get("felhom_next"), "the flag is gone")
|
||||
self.assertEqual(rep["kernel"]["phase"], "cancelled")
|
||||
|
||||
def test_status_is_read_only_and_names_setup_problems(self):
|
||||
f = kfake(esp_fs="ext4")
|
||||
rc, rep = kmode(f, "kernel-status")
|
||||
self.assertEqual(rc, 0, rep)
|
||||
self.assertTrue(rep["kernel"]["setup_problems"])
|
||||
self.assertEqual(f.tree, {})
|
||||
self.assertFalse([c for c in f.calls if c[1][0] in ("update-grub", "systemctl", "apt-get")])
|
||||
|
||||
def test_facts_carry_the_kernel_lane(self):
|
||||
f = staged()
|
||||
f.plan = {"release_id": "facts", "layer": "host", "lane": "fast", "vmid": 9201, "mode": "facts", "packages": []}
|
||||
rc, rep = run(f)
|
||||
self.assertEqual(rc, 0, rep)
|
||||
h = rep["facts"]["host"]
|
||||
self.assertEqual(h["kernel_lane"]["phase"], "staged")
|
||||
self.assertEqual(h["kernel_next_boot"], NEW)
|
||||
self.assertIn("one-shot", h["kernel_next_boot_source"])
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
unittest.main()
|
||||
|
||||
@@ -636,6 +636,17 @@ type WireOSUpdate struct {
|
||||
// HostRelease is the newest approved HOST release (hub v0.131.0, `11` §8 step 3) — a separate set: a version
|
||||
// approved for the guest is not approved for the host by that fact alone.
|
||||
HostRelease *WireOSRelease `json:"host_release,omitempty"`
|
||||
// Kernel is the kernel lane's instruction for THIS box (R-836, `09` §3 decision 172, `11` §5.11): the kernel the
|
||||
// household was told about, and whether tonight is a told night. Nil (an older hub, or no kernel due) = no kernel
|
||||
// step. The hub sets Tonight only after the household's mail the day before went out — no mail, no step.
|
||||
Kernel *WireKernelStep `json:"kernel,omitempty"`
|
||||
}
|
||||
|
||||
// WireKernelStep is the hub's kernel-lane instruction (hub osupdates.KernelBlock — field-exact, cross-repo).
|
||||
type WireKernelStep struct {
|
||||
Kver string `json:"kver"` // e.g. "7.0.14-22-pve" — the wrapper refuses any other (R23)
|
||||
Tonight bool `json:"tonight"` // the household was mailed the day before: tonight's leg may reboot
|
||||
NotifiedAt string `json:"notified_at,omitempty"` // when that mail went out (RFC 3339), for the log
|
||||
}
|
||||
|
||||
// WireOSRelease is an approved version set; Snapshot is the approval time (YYYYMMDDTHHMMSSZ) the wrapper uses
|
||||
|
||||
@@ -0,0 +1,428 @@
|
||||
package osupdate
|
||||
|
||||
// The kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11). The root half is felhom-os-apply's layer "kernel"
|
||||
// (configs/, its own tests); this file decides WHEN and judges the boot:
|
||||
//
|
||||
// - the night leg (Run → runKernel): after a healthy host step, on a night the hub marks as told (the household was
|
||||
// mailed the day before — no mail, no step): ring 0 STAGES the pending kernel (select pending-kernel, the root-owned
|
||||
// ring-0 mark) and reboots; ring 1 reboots only a step a signed os_kernel_step staged earlier (KernelStepExecutor).
|
||||
// - after a boot (KernelAfterBoot, at daemon start): the wrapper says what became of the step. On the new kernel the
|
||||
// agent JUDGES the boot — the host health rule (`11` §8.2: the Proxmox daemons, the guest running and healthy, the
|
||||
// tunnel) AND the box reaching the hub — for KernelJudgeWait. Healthy → kernel-good (the new kernel becomes the
|
||||
// default). Not healthy by the deadline → ONE self-revert (kernel-revert: a reboot into the old kernel, still the
|
||||
// default). A crash on the new kernel needs nothing from the agent: GRUB already boots the old default.
|
||||
//
|
||||
// The host is rebooted by this file only through the wrapper (kernel-reboot, kernel-revert), and only for a staged step.
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"regexp"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
|
||||
)
|
||||
|
||||
// OpKernelStep is the signed op class that STAGES a kernel set on a ring-1 box (it never reboots: the night leg does,
|
||||
// once the household was told). CC may sign it until the first paying customer (R-530 ruling).
|
||||
const OpKernelStep = "os_kernel_step"
|
||||
|
||||
// DefaultKernelJudgeWait is how long a one-shot boot may take to come back healthy before the agent reverts it ONCE.
|
||||
// Measured 2026-10-07 (`audits/kernel-lane-2026-10-07/B/`): every container healthy 68 s after a reboot on demo-felhom,
|
||||
// 272 s on demo-hp; 20 minutes stays under the hub's 45-minute host_stale (a box that never comes back alarms after it).
|
||||
const DefaultKernelJudgeWait = 20 * time.Minute
|
||||
|
||||
var kverRE = regexp.MustCompile(`^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$`)
|
||||
|
||||
// KernelView is the wrapper's kernel object (felhom-os-apply Kernel.view).
|
||||
type KernelView struct {
|
||||
Running string `json:"running"`
|
||||
Default string `json:"default"`
|
||||
Flag *string `json:"flag"`
|
||||
Phase string `json:"phase"`
|
||||
From string `json:"from"`
|
||||
To string `json:"to"`
|
||||
SelfRevertUsed bool `json:"self_revert_used"`
|
||||
Reason string `json:"reason"`
|
||||
VMID int `json:"vmid"` // the customer guest the step was staged for (the health rule's guest)
|
||||
}
|
||||
|
||||
func parseKernel(raw json.RawMessage) KernelView {
|
||||
var v KernelView
|
||||
_ = json.Unmarshal(raw, &v)
|
||||
return v
|
||||
}
|
||||
|
||||
// kernelPlan is one kernel-layer wrapper call.
|
||||
func kernelPlan(mode string, vmid int, extra map[string]any) map[string]any {
|
||||
p := map[string]any{"release_id": "kernel", "layer": LayerKernel, "lane": "slow", "vmid": vmid, "mode": mode,
|
||||
"packages": []Package{}}
|
||||
for k, v := range extra {
|
||||
p[k] = v
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
// KernelStatus reads the kernel lane's state (read only).
|
||||
func (l *Leg) KernelStatus(ctx context.Context, vmid int) (KernelView, error) {
|
||||
wr, err := l.call(ctx, "kstatus"+l.now().UTC().Format("150405"), kernelPlan("kernel-status", vmid, nil))
|
||||
if err != nil {
|
||||
return KernelView{}, err
|
||||
}
|
||||
if wr.refused() {
|
||||
return KernelView{}, fmt.Errorf("kernel-status refused: %s", wr.Refused)
|
||||
}
|
||||
return parseKernel(wr.Kernel), nil
|
||||
}
|
||||
|
||||
// kernelDue reports whether tonight's leg may take a kernel step, and why not.
|
||||
func kernelDue(blk hub.WireOSUpdate, trigger string) (bool, string) {
|
||||
switch {
|
||||
case trigger != "night":
|
||||
return false, "a kernel step runs only in the night leg (never a debug pass)"
|
||||
case blk.Kernel == nil:
|
||||
return false, "the hub names no kernel step for this box"
|
||||
case !blk.Enabled:
|
||||
return false, "OS updates are switched off for this box"
|
||||
case !kverRE.MatchString(blk.Kernel.Kver):
|
||||
return false, "the hub's kernel " + blk.Kernel.Kver + " is not a kernel version"
|
||||
case !blk.Kernel.Tonight:
|
||||
return false, "the household has not been told about tonight (no mail, no step — `09` §3 decision 172)"
|
||||
}
|
||||
return true, ""
|
||||
}
|
||||
|
||||
// runKernel is the night leg's last step. It returns the stage report (Layer "" when nothing ran). On success the box
|
||||
// is rebooting when it returns.
|
||||
func (l *Leg) runKernel(ctx context.Context, runID string, vmid int, trigger string, blk hub.WireOSUpdate) Report {
|
||||
lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid, "trigger", trigger, "ring", blk.Ring)
|
||||
if ok, why := kernelDue(blk, trigger); !ok {
|
||||
lg.Info("osupdate: kernel step skipped — " + why)
|
||||
return Report{}
|
||||
}
|
||||
want := blk.Kernel.Kver
|
||||
st, err := l.KernelStatus(ctx, vmid)
|
||||
if err != nil {
|
||||
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid,
|
||||
Mode: "apply", Outcome: "failed", HealthReason: "kernel status unreadable: " + err.Error()})
|
||||
}
|
||||
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid, Mode: "apply", ReleaseID: want}
|
||||
switch {
|
||||
case st.Phase == "staged" && st.To == want:
|
||||
lg.Info("osupdate: kernel step — a staged kernel waits for tonight", "from", st.From, "to", st.To)
|
||||
rep.Outcome, rep.Healthy = "staged", true
|
||||
case blk.Ring != 0:
|
||||
lg.Info("osupdate: kernel step skipped — ring 1 boots only a kernel a signed os_kernel_step staged", "phase", st.Phase, "staged", st.To, "want", want)
|
||||
return Report{}
|
||||
default:
|
||||
// R-898: EXACTLY the kernel the household was told about — never "whatever is pending tonight" (the sources can
|
||||
// offer a newer one by night; the step then refused, R23, and the night was lost). A version no longer
|
||||
// installable is refused by the wrapper before any change (R7) and the hub tells the household again.
|
||||
wr, cerr := l.call(ctx, runID, kernelPlan("apply", vmid, map[string]any{"release_id": "ring0-" + runID,
|
||||
"select": "listed", "packages": KernelSet(want), "expect_kver": want, "run_id": runID, "trigger": trigger,
|
||||
"ring": blk.Ring}))
|
||||
rep.unsent = reportFile(l.planDir(), runID, LayerKernel, "apply")
|
||||
rep.Kernel = rawOrNil(wr.Kernel)
|
||||
switch {
|
||||
case cerr != nil:
|
||||
rep.Outcome, rep.HealthReason = "failed", cerr.Error()
|
||||
return l.finish(ctx, lg, rep)
|
||||
case wr.refused():
|
||||
rep.Outcome, rep.Refused = "refused", wr.Refused
|
||||
return l.finish(ctx, lg, rep)
|
||||
case wr.failed():
|
||||
rep.Outcome, rep.Refused = "failed", wr.Failed
|
||||
return l.finish(ctx, lg, rep)
|
||||
case wr.OutcomeHint == "nothing" || len(wr.Upgraded) == 0:
|
||||
rep.Outcome, rep.Healthy = "nothing", true
|
||||
return l.finish(ctx, lg, rep)
|
||||
}
|
||||
rep.Outcome, rep.Healthy, rep.Upgraded, rep.Authority, rep.PassSeconds = "staged", true, wr.Upgraded, wr.Authority, wr.PassSeconds
|
||||
rep.RebootNeeded = true
|
||||
}
|
||||
rep = l.finish(ctx, lg, rep) // the hub hears "staged" BEFORE the box goes down
|
||||
wr, err := l.call(ctx, runID, kernelPlan("kernel-reboot", vmid, nil))
|
||||
switch {
|
||||
case err != nil:
|
||||
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid,
|
||||
Mode: "kernel-reboot", ReleaseID: want, Outcome: "failed", HealthReason: "kernel-reboot: " + err.Error()})
|
||||
case wr.refused() || wr.failed():
|
||||
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid,
|
||||
Mode: "kernel-reboot", ReleaseID: want, Outcome: "refused", Refused: firstRaw(wr.Refused, wr.Failed),
|
||||
Kernel: rawOrNil(wr.Kernel)})
|
||||
}
|
||||
lg.Warn("osupdate: kernel step — the box restarts now for its one-shot boot", "to", want)
|
||||
return rep
|
||||
}
|
||||
|
||||
func firstRaw(a, b json.RawMessage) json.RawMessage {
|
||||
if r := rawOrNil(a); r != nil {
|
||||
return r
|
||||
}
|
||||
return rawOrNil(b)
|
||||
}
|
||||
|
||||
// KernelJudge is what KernelAfterBoot needs besides the leg: the hub reachability probe is the "judging" report itself.
|
||||
type KernelJudge struct {
|
||||
Wait time.Duration // default DefaultKernelJudgeWait
|
||||
Poll time.Duration // default 30 s
|
||||
}
|
||||
|
||||
// KernelAfterBoot runs once at daemon start: what became of a kernel step across the boot. On the new kernel it judges
|
||||
// the boot (blocking up to the wait — run it in a goroutine). vmid 0 = the guest the step recorded (it may not run yet).
|
||||
func (l *Leg) KernelAfterBoot(ctx context.Context, vmid int, j KernelJudge) Report {
|
||||
runID := "boot-" + l.now().UTC().Format("20060102T150405Z")
|
||||
lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid)
|
||||
wr, err := l.call(ctx, runID, kernelPlan("kernel-boot", vmid, nil))
|
||||
if err != nil {
|
||||
lg.Warn("osupdate: kernel after-boot check failed", "err", err)
|
||||
return Report{}
|
||||
}
|
||||
if wr.refused() {
|
||||
lg.Info("osupdate: kernel after-boot check refused (an older wrapper, or a BYO host)", "refused", string(wr.Refused))
|
||||
return Report{}
|
||||
}
|
||||
v := parseKernel(wr.Kernel)
|
||||
if vmid <= 0 {
|
||||
vmid = v.VMID // after a boot the guest may not run yet — the step's own record names it
|
||||
}
|
||||
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid, Mode: "kernel-boot",
|
||||
ReleaseID: v.To, Kernel: rawOrNil(wr.Kernel)}
|
||||
switch wr.KernelEvent {
|
||||
case "fell_back":
|
||||
rep.Outcome, rep.HealthReason = "fell_back", v.Reason
|
||||
lg.Warn("osupdate: kernel step FELL BACK — the new kernel did not come up; the box runs the old one", "from", v.From, "to", v.To)
|
||||
return l.finish(ctx, lg, rep)
|
||||
case "self_reverted":
|
||||
rep.Outcome, rep.HealthReason = "self_reverted", v.Reason
|
||||
lg.Warn("osupdate: kernel step SELF-REVERTED — back on the old kernel", "from", v.From, "to", v.To, "reason", v.Reason)
|
||||
return l.finish(ctx, lg, rep)
|
||||
case "revert_failed":
|
||||
rep.Outcome, rep.HealthReason = "revert_failed", v.Reason
|
||||
lg.Error("osupdate: kernel self-revert came back on the NEW kernel — no second revert; the operator decides", "to", v.To)
|
||||
return l.finish(ctx, lg, rep)
|
||||
case "judging":
|
||||
return l.judgeKernel(ctx, runID, vmid, v, wr.HealthBefore, j, lg)
|
||||
}
|
||||
return Report{}
|
||||
}
|
||||
|
||||
// KernelVerdict is THE one-shot boot rule (R-836; pinned by TestKernelVerdict): the host health rule (`11` §8.2 —
|
||||
// the Proxmox daemons and the agent active, the customer guest running and its own rule passing, the tunnel running)
|
||||
// AND the box reached the hub since this boot.
|
||||
func KernelVerdict(before, after *Health, tunnel string, hubReached bool) (bool, string) {
|
||||
if ok, why := HostHealthVerdict(before, after, tunnel); !ok {
|
||||
return false, why
|
||||
}
|
||||
if !hubReached {
|
||||
return false, "the box has not reached the hub since the boot"
|
||||
}
|
||||
return true, ""
|
||||
}
|
||||
|
||||
func (l *Leg) judgeKernel(ctx context.Context, runID string, vmid int, v KernelView, before *Health, j KernelJudge, lg *slog.Logger) Report {
|
||||
wait, poll := j.Wait, j.Poll
|
||||
if wait <= 0 {
|
||||
wait = DefaultKernelJudgeWait
|
||||
}
|
||||
if poll <= 0 {
|
||||
poll = 30 * time.Second
|
||||
}
|
||||
lg.Info("osupdate: kernel step — judging the one-shot boot", "from", v.From, "to", v.To, "wait", wait.String())
|
||||
start := l.now()
|
||||
deadline := start.Add(wait)
|
||||
hubReached := false
|
||||
var why string
|
||||
for {
|
||||
if !hubReached && l.Hub != nil {
|
||||
// the hub's reachability IS this report reaching it (and the operator sees the box is back on the new kernel)
|
||||
body, _ := json.Marshal(Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid,
|
||||
Mode: "kernel-boot", ReleaseID: v.To, Outcome: "judging", Kernel: mustRaw(v)})
|
||||
rctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
if err := l.Hub.PostOSReport(rctx, body); err == nil {
|
||||
hubReached = true
|
||||
lg.Info("osupdate: kernel step — the box reached the hub on the new kernel", "after", l.now().Sub(start).Round(time.Second).String())
|
||||
}
|
||||
cancel()
|
||||
}
|
||||
var h *Health
|
||||
hr, err := l.call(ctx, runID, kernelPlan("health", vmid, nil))
|
||||
switch {
|
||||
case err != nil:
|
||||
why = "no health reading: " + err.Error()
|
||||
case hr.refused():
|
||||
why = "no health reading: " + string(hr.Refused)
|
||||
default:
|
||||
h = hr.Health
|
||||
}
|
||||
if h != nil {
|
||||
t := hub.TunnelUnknown
|
||||
if l.Tunnel != nil {
|
||||
t, _ = l.Tunnel.Status(ctx)
|
||||
}
|
||||
var ok bool
|
||||
ok, why = KernelVerdict(before, h, t, hubReached)
|
||||
if ok {
|
||||
return l.kernelGood(ctx, runID, vmid, v, start, lg)
|
||||
}
|
||||
}
|
||||
if !l.now().Before(deadline) || ctx.Err() != nil {
|
||||
break
|
||||
}
|
||||
l.sleep(ctx, poll)
|
||||
}
|
||||
if ctx.Err() != nil {
|
||||
lg.Warn("osupdate: kernel judging stopped (the agent is stopping) — the next start judges again", "reason", why)
|
||||
return Report{}
|
||||
}
|
||||
// not healthy by the deadline: tell the hub (best effort), then ONE self-revert into the old kernel
|
||||
rep := l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid,
|
||||
Mode: "kernel-revert", ReleaseID: v.To, Outcome: "health_failed", HealthReason: why + " — reverting to " + v.From,
|
||||
Kernel: mustRaw(v)})
|
||||
lg.Error("osupdate: kernel step — the one-shot boot is NOT healthy; restarting ONCE into the old kernel", "reason", why,
|
||||
"waited", wait.String(), "from", v.From, "to", v.To)
|
||||
wr, err := l.call(ctx, runID, kernelPlan("kernel-revert", vmid, map[string]any{"reason": truncate(why, 280)}))
|
||||
if err != nil || wr.refused() || wr.failed() {
|
||||
lg.Error("osupdate: kernel self-revert did not start — the box stays on the new kernel; the operator decides",
|
||||
"err", err, "refused", string(firstRaw(wr.Refused, wr.Failed)))
|
||||
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid,
|
||||
Mode: "kernel-revert", ReleaseID: v.To, Outcome: "revert_failed", Refused: firstRaw(wr.Refused, wr.Failed),
|
||||
HealthReason: "the self-revert did not start"})
|
||||
}
|
||||
return rep
|
||||
}
|
||||
|
||||
func (l *Leg) kernelGood(ctx context.Context, runID string, vmid int, v KernelView, start time.Time, lg *slog.Logger) Report {
|
||||
wr, err := l.call(ctx, runID, kernelPlan("kernel-good", vmid, nil))
|
||||
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.Block().Ring, VMID: vmid, Mode: "kernel-good",
|
||||
ReleaseID: v.To}
|
||||
switch {
|
||||
case err != nil:
|
||||
rep.Outcome, rep.HealthReason = "failed", "kernel-good: "+err.Error()
|
||||
case wr.refused() || wr.failed():
|
||||
rep.Outcome, rep.Refused, rep.HealthReason = "failed", firstRaw(wr.Refused, wr.Failed), "kernel-good did not move the default"
|
||||
default:
|
||||
rep.Outcome, rep.Healthy = "applied", true
|
||||
rep.HealthReason = fmt.Sprintf("healthy %s after the agent started; the new kernel is the default", l.now().Sub(start).Round(time.Second))
|
||||
}
|
||||
rep.Kernel = rawOrNil(wr.Kernel)
|
||||
lg.Info("osupdate: kernel step — "+rep.Outcome, "to", v.To, "reason", rep.HealthReason)
|
||||
return l.finish(ctx, lg, rep)
|
||||
}
|
||||
|
||||
func mustRaw(v any) json.RawMessage {
|
||||
b, _ := json.Marshal(v)
|
||||
return b
|
||||
}
|
||||
|
||||
func truncate(s string, n int) string {
|
||||
if len(s) <= n {
|
||||
return s
|
||||
}
|
||||
return s[:n]
|
||||
}
|
||||
|
||||
// KernelSet is the package set that installs exactly kver (R-898; the hub's kernelSet, field-exact): the series
|
||||
// meta-package and the signed image, both at the kernel's own version. Proxmox keeps old kernel versions in its archive.
|
||||
// nil for a string that is not a kernel version.
|
||||
func KernelSet(kver string) []Package {
|
||||
m := kverSeriesRE.FindStringSubmatch(kver)
|
||||
if m == nil {
|
||||
return nil
|
||||
}
|
||||
v := kver[:len(kver)-len("-pve")]
|
||||
return []Package{{Name: "proxmox-kernel-" + m[1], Version: v, Origin: PVEOrigin},
|
||||
{Name: "proxmox-kernel-" + kver + "-signed", Version: v, Origin: PVEOrigin}}
|
||||
}
|
||||
|
||||
var kverSeriesRE = regexp.MustCompile(`^([0-9]+\.[0-9]+)\.[0-9]+-[0-9]+-pve$`)
|
||||
|
||||
// KernelStepParams are a signed os_kernel_step's params: the exact kernel set (the wrapper compares it with the plan).
|
||||
type KernelStepParams struct {
|
||||
ReleaseID string `json:"release_id"`
|
||||
Packages []Package `json:"packages"`
|
||||
Kver string `json:"kver"`
|
||||
VMID int `json:"vmid,omitempty"`
|
||||
}
|
||||
|
||||
// KernelStepExecutor STAGES a verified os_kernel_step (signedjobs.Executor) under the heavy-op gate. It never reboots:
|
||||
// the night leg reboots a staged kernel on a night the household was told about.
|
||||
type KernelStepExecutor struct {
|
||||
Leg *Leg
|
||||
Guest func(ctx context.Context) (int, error)
|
||||
Gate func(ctx context.Context) (release func(), err error)
|
||||
}
|
||||
|
||||
// Execute implements signedjobs.Executor.
|
||||
func (e KernelStepExecutor) Execute(ctx context.Context, op string, params json.RawMessage) error {
|
||||
if op != OpKernelStep {
|
||||
return signedjobs.ErrNoExecutor
|
||||
}
|
||||
so, ok := signedjobs.SignedOpFrom(ctx)
|
||||
if !ok {
|
||||
return fmt.Errorf("os_kernel_step: no signed envelope in the context — the wrapper could not verify it")
|
||||
}
|
||||
var p KernelStepParams
|
||||
if err := json.Unmarshal(params, &p); err != nil || len(p.Packages) == 0 || !kverRE.MatchString(p.Kver) {
|
||||
return fmt.Errorf("os_kernel_step: params must name the kernel set and its kver: %v", err)
|
||||
}
|
||||
vmid := p.VMID
|
||||
if vmid == 0 {
|
||||
if e.Guest == nil {
|
||||
return fmt.Errorf("os_kernel_step: no vmid and no guest finder")
|
||||
}
|
||||
v, err := e.Guest(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("os_kernel_step: find the customer guest: %w", err)
|
||||
}
|
||||
vmid = v
|
||||
}
|
||||
if e.Gate != nil {
|
||||
release, err := e.Gate(ctx)
|
||||
if err != nil {
|
||||
return fmt.Errorf("os_kernel_step: heavy-op gate busy (a backup or restore-test runs): %w", err)
|
||||
}
|
||||
defer release()
|
||||
}
|
||||
rep := e.Leg.StageKernelSigned(ctx, vmid, p, so.Blob, string(so.Sig))
|
||||
if rep.Outcome == "staged" || rep.Outcome == "nothing" {
|
||||
return nil
|
||||
}
|
||||
return fmt.Errorf("os_kernel_step: %s (%s) %s", rep.Outcome, rep.HealthReason, string(rep.Refused))
|
||||
}
|
||||
|
||||
// StageKernelSigned stages a signed kernel set (ring 1): install + flag, no reboot.
|
||||
func (l *Leg) StageKernelSigned(ctx context.Context, vmid int, p KernelStepParams, blob []byte, sig string) Report {
|
||||
unlock := l.lockPass(true)
|
||||
defer unlock()
|
||||
l.sendUnsentLocked(ctx) // R-868
|
||||
runID := l.now().UTC().Format("20060102T150405Z")
|
||||
rid := p.ReleaseID
|
||||
if rid == "" {
|
||||
rid = "signed-" + runID
|
||||
}
|
||||
lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid, "trigger", "signed", "release", rid)
|
||||
wr, err := l.call(ctx, runID, kernelPlan("apply", vmid, map[string]any{"release_id": rid, "select": "listed",
|
||||
"packages": p.Packages, "expect_kver": p.Kver, "run_id": runID, "trigger": "signed", "ring": l.Block().Ring,
|
||||
"signed": map[string]string{"blob_b64": base64.StdEncoding.EncodeToString(blob), "sig": sig}}))
|
||||
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "signed", Ring: l.Block().Ring, VMID: vmid, Mode: "apply",
|
||||
ReleaseID: rid, Kernel: rawOrNil(wr.Kernel), unsent: reportFile(l.planDir(), runID, LayerKernel, "apply")}
|
||||
switch {
|
||||
case err != nil:
|
||||
rep.Outcome, rep.HealthReason = "failed", err.Error()
|
||||
case wr.refused():
|
||||
rep.Outcome, rep.Refused = "refused", wr.Refused
|
||||
case wr.failed():
|
||||
rep.Outcome, rep.Refused = "failed", wr.Failed
|
||||
case wr.OutcomeHint == "nothing" || len(wr.Upgraded) == 0:
|
||||
rep.Outcome, rep.Healthy = "nothing", true
|
||||
default:
|
||||
rep.Outcome, rep.Healthy, rep.Upgraded, rep.Authority, rep.PassSeconds = "staged", true, wr.Upgraded, wr.Authority, wr.PassSeconds
|
||||
rep.RebootNeeded = true
|
||||
}
|
||||
return l.finish(ctx, lg, rep)
|
||||
}
|
||||
@@ -0,0 +1,328 @@
|
||||
package osupdate
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
|
||||
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
|
||||
)
|
||||
|
||||
// ---- the kernel lane (R-836, `09` §3 decision 172, `11` §5.11) ----
|
||||
|
||||
const kOld, kNew = "7.0.2-6-pve", "7.0.14-22-pve"
|
||||
|
||||
func kview(phase string) json.RawMessage {
|
||||
return mustRaw(KernelView{Running: kOld, Default: kOld, Phase: phase, From: kOld, To: kNew, VMID: 9201})
|
||||
}
|
||||
|
||||
func tonight(ring int) *hub.WireOSUpdate {
|
||||
return &hub.WireOSUpdate{Ring: ring, Enabled: true, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: true}}
|
||||
}
|
||||
|
||||
func kernelCalls(w *fakeWrapper) []string {
|
||||
var m []string
|
||||
for _, p := range w.plans {
|
||||
if p["layer"] == LayerKernel {
|
||||
m = append(m, p["mode"].(string))
|
||||
}
|
||||
}
|
||||
return m
|
||||
}
|
||||
|
||||
// Ring 0, a told night: after the healthy host step the leg stages the pending kernel (select pending-kernel, the
|
||||
// kernel the household was told about), tells the hub "staged", THEN reboots — the kernel step ends the night.
|
||||
func TestKernel_Ring0ToldNightStagesThenReboots(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{
|
||||
"kernel-status": {{Kernel: kview("none")}},
|
||||
"apply": {{Upgraded: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22"}}, Authority: "ring0", Kernel: kview("staged")}},
|
||||
"kernel-reboot": {{Kernel: kview("oneshot")}},
|
||||
}}
|
||||
l, h := newLeg(t, w, tonight(0))
|
||||
p := l.Run(context.Background(), 9201, "night")
|
||||
if got := strings.Join(kernelCalls(w), ","); got != "kernel-status,apply,kernel-reboot" {
|
||||
t.Fatalf("kernel calls = %s", got)
|
||||
}
|
||||
var ap map[string]any
|
||||
for _, x := range w.plans {
|
||||
if x["layer"] == LayerKernel && x["mode"] == "apply" {
|
||||
ap = x
|
||||
}
|
||||
}
|
||||
// R-898: EXACTLY the told kernel — the listed set derived from it, never "pending" (red before the fix: select was
|
||||
// pending-kernel, so a newer kernel in the sources by night was refused R23 and the night was lost)
|
||||
pk, _ := json.Marshal(ap["packages"])
|
||||
if ap["select"] != "listed" || ap["expect_kver"] != kNew || ap["lane"] != "slow" ||
|
||||
string(pk) != `[{"name":"proxmox-kernel-7.0","origin":"Proxmox Debian Repository","version":"7.0.14-22"},{"name":"proxmox-kernel-7.0.14-22-pve-signed","origin":"Proxmox Debian Repository","version":"7.0.14-22"}]` {
|
||||
t.Fatalf("stage plan = %v (packages %s)", ap, pk)
|
||||
}
|
||||
if p.Kernel.Outcome != "staged" || !p.Kernel.Healthy {
|
||||
t.Fatalf("kernel report = %+v", p.Kernel)
|
||||
}
|
||||
last := h.reports[len(h.reports)-1]
|
||||
if last.Layer != LayerKernel || last.Outcome != "staged" {
|
||||
t.Fatalf("the hub must hear 'staged' before the reboot: %+v", h.reports)
|
||||
}
|
||||
// the kernel step is the LAST wrapper call of the night
|
||||
if lp := w.plans[len(w.plans)-1]; lp["layer"] != LayerKernel || lp["mode"] != "kernel-reboot" {
|
||||
t.Fatalf("the reboot must end the night, last call = %v", lp)
|
||||
}
|
||||
}
|
||||
|
||||
// No mail, no step: a kernel the household was NOT told about never runs; nor in a debug pass; nor without a block.
|
||||
// COMPANION RED-PROOF (observed): drop the `!blk.Kernel.Tonight` case in kernelDue → the first sub-case fails.
|
||||
func TestKernel_NoMailNoStep(t *testing.T) {
|
||||
cases := map[string]struct {
|
||||
blk *hub.WireOSUpdate
|
||||
trigger string
|
||||
}{
|
||||
"not told": {&hub.WireOSUpdate{Ring: 0, Enabled: true, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: false}}, "night"},
|
||||
"debug pass": {tonight(0), "debug"},
|
||||
"no block": {&hub.WireOSUpdate{Ring: 0, Enabled: true}, "night"},
|
||||
"switch off": {&hub.WireOSUpdate{Ring: 0, Enabled: false, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: true}}, "night"},
|
||||
"bad kver": {&hub.WireOSUpdate{Ring: 0, Enabled: true, Kernel: &hub.WireKernelStep{Kver: "7.0; reboot", Tonight: true}}, "night"},
|
||||
}
|
||||
for name, c := range cases {
|
||||
w := &fakeWrapper{t: t}
|
||||
l, _ := newLeg(t, w, c.blk)
|
||||
p := l.Run(context.Background(), 9201, c.trigger)
|
||||
if len(kernelCalls(w)) != 0 || p.Kernel.Layer != "" {
|
||||
t.Fatalf("%s: a kernel step ran: %v", name, kernelCalls(w))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// The kernel step needs a healthy host step on an appliance, and a healthy Proxmox step when one ran.
|
||||
func TestKernel_SkippedWithoutHealthyEarlierSteps(t *testing.T) {
|
||||
w := &fakeWrapper{t: t}
|
||||
l, _ := newLeg(t, w, tonight(0))
|
||||
l.Appliance = false
|
||||
l.Run(context.Background(), 9201, "night")
|
||||
if len(kernelCalls(w)) != 0 {
|
||||
t.Fatalf("a BYO box took a kernel step: %v", kernelCalls(w))
|
||||
}
|
||||
w2 := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerPVE: {Upgraded: []Package{{Name: "pve-manager", Version: "9.2.21"}},
|
||||
PVEManager: "9.2.2"}}} // pveversion still old → the pve step is unhealthy
|
||||
l2, _ := newLeg(t, w2, tonight(0))
|
||||
l2.Run(context.Background(), 9201, "night")
|
||||
if len(kernelCalls(w2)) != 0 {
|
||||
t.Fatalf("a kernel step ran after an unhealthy Proxmox step: %v", kernelCalls(w2))
|
||||
}
|
||||
}
|
||||
|
||||
// Ring 1 reboots only a kernel a signed job staged — never stages one itself in the night leg.
|
||||
func TestKernel_Ring1RebootsOnlyASignedStage(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}}}}
|
||||
l, _ := newLeg(t, w, tonight(1))
|
||||
l.Run(context.Background(), 9201, "night")
|
||||
if got := strings.Join(kernelCalls(w), ","); got != "kernel-status" {
|
||||
t.Fatalf("ring 1 without a staged kernel: calls = %s", got)
|
||||
}
|
||||
w2 := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("staged")}},
|
||||
"kernel-reboot": {{Kernel: kview("oneshot")}}}}
|
||||
l2, _ := newLeg(t, w2, tonight(1))
|
||||
p := l2.Run(context.Background(), 9201, "night")
|
||||
if got := strings.Join(kernelCalls(w2), ","); got != "kernel-status,kernel-reboot" || p.Kernel.Outcome != "staged" {
|
||||
t.Fatalf("ring 1 with a staged kernel: calls = %s report = %+v", got, p.Kernel)
|
||||
}
|
||||
}
|
||||
|
||||
// A refused stage never reboots.
|
||||
func TestKernel_RefusedStageNeverReboots(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}},
|
||||
"apply": {{Refused: json.RawMessage(`{"code":"R20","reason":"/boot/efi is not a mounted vfat ESP"}`)}}}}
|
||||
l, h := newLeg(t, w, tonight(0))
|
||||
p := l.Run(context.Background(), 9201, "night")
|
||||
if got := strings.Join(kernelCalls(w), ","); got != "kernel-status,apply" || p.Kernel.Outcome != "refused" {
|
||||
t.Fatalf("calls = %s report = %+v", got, p.Kernel)
|
||||
}
|
||||
if last := h.reports[len(h.reports)-1]; last.Layer != LayerKernel || last.Outcome != "refused" {
|
||||
t.Fatalf("the hub must hear the refusal: %+v", last)
|
||||
}
|
||||
}
|
||||
|
||||
// THE one-shot boot rule: the host rule AND the hub reached. COMPANION RED-PROOF (observed): drop the hubReached
|
||||
// check in KernelVerdict → the second case fails.
|
||||
func TestKernelVerdict(t *testing.T) {
|
||||
if ok, why := KernelVerdict(hostOK(), hostOK(), hub.TunnelRunning, true); !ok {
|
||||
t.Fatalf("a healthy boot read unhealthy: %s", why)
|
||||
}
|
||||
if ok, why := KernelVerdict(hostOK(), hostOK(), hub.TunnelRunning, false); ok || !strings.Contains(why, "hub") {
|
||||
t.Fatalf("a box that has not reached the hub must not pass: ok=%v %q", ok, why)
|
||||
}
|
||||
down := hostOK()
|
||||
down.GuestRunning = new(bool)
|
||||
if ok, _ := KernelVerdict(hostOK(), down, hub.TunnelRunning, true); ok {
|
||||
t.Fatal("a guest that does not run must fail")
|
||||
}
|
||||
if ok, _ := KernelVerdict(hostOK(), hostOK(), hub.TunnelUnknown, true); ok {
|
||||
t.Fatal("an unknown tunnel must fail (the host rule)")
|
||||
}
|
||||
}
|
||||
|
||||
func judgingLeg(t *testing.T, w *fakeWrapper) (*Leg, *fakeHub) {
|
||||
if w.kernelRep == nil {
|
||||
w.kernelRep = map[string][]WrapperReport{}
|
||||
}
|
||||
if _, ok := w.kernelRep["kernel-boot"]; !ok {
|
||||
w.kernelRep["kernel-boot"] = []WrapperReport{{KernelEvent: "judging", Kernel: mustRaw(KernelView{Running: kNew,
|
||||
Default: kOld, Phase: "judging", From: kOld, To: kNew, VMID: 9201}), HealthBefore: hostOK()}}
|
||||
}
|
||||
return newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
|
||||
}
|
||||
|
||||
// A healthy one-shot boot: the hub hears "judging", then kernel-good, then "applied".
|
||||
func TestKernelAfterBoot_HealthyBecomesTheDefault(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-good": {{Kernel: kview("good")}}}}
|
||||
l, h := judgingLeg(t, w)
|
||||
r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 10 * time.Minute, Poll: 30 * time.Second})
|
||||
if got := strings.Join(kernelCalls(w), ","); got != "kernel-boot,health,kernel-good" {
|
||||
t.Fatalf("calls = %s", got)
|
||||
}
|
||||
if r.Outcome != "applied" || !r.Healthy {
|
||||
t.Fatalf("report = %+v", r)
|
||||
}
|
||||
if len(h.reports) != 2 || h.reports[0].Outcome != "judging" || h.reports[1].Outcome != "applied" {
|
||||
t.Fatalf("hub reports = %+v", h.reports)
|
||||
}
|
||||
if w.plans[1]["vmid"] != float64(9201) {
|
||||
t.Fatalf("the health reading must use the step's own guest, got %v", w.plans[1]["vmid"])
|
||||
}
|
||||
}
|
||||
|
||||
// An unhealthy one-shot boot: wait the full judge time, tell the hub, then ONE kernel-revert.
|
||||
// COMPANION RED-PROOF (observed): return before the kernel-revert call in judgeKernel → "calls" fails.
|
||||
func TestKernelAfterBoot_UnhealthyRevertsOnceAfterTheWait(t *testing.T) {
|
||||
down := hostOK()
|
||||
down.GuestRunning = new(bool)
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"health": {{Health: down}},
|
||||
"kernel-revert": {{Kernel: kview("reverting")}}}}
|
||||
l, h := judgingLeg(t, w)
|
||||
start := l.now()
|
||||
r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 10 * time.Minute, Poll: time.Minute})
|
||||
calls := kernelCalls(w)
|
||||
if calls[len(calls)-1] != "kernel-revert" || strings.Count(strings.Join(calls, ","), "kernel-revert") != 1 {
|
||||
t.Fatalf("calls = %v", calls)
|
||||
}
|
||||
if waited := l.now().Sub(start); waited < 10*time.Minute {
|
||||
t.Fatalf("reverted after %s — before the judge wait", waited)
|
||||
}
|
||||
if r.Outcome != "health_failed" || !strings.Contains(r.HealthReason, "not running") {
|
||||
t.Fatalf("report = %+v", r)
|
||||
}
|
||||
if last := h.reports[len(h.reports)-1]; last.Outcome != "health_failed" {
|
||||
t.Fatalf("the hub must hear health_failed before the revert reboot: %+v", h.reports)
|
||||
}
|
||||
for _, p := range w.plans {
|
||||
if p["mode"] == "kernel-revert" && !strings.Contains(p["reason"].(string), "not running") {
|
||||
t.Fatalf("the revert must carry the reason: %v", p)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A box that never reaches the hub is not "healthy" — it reverts too.
|
||||
func TestKernelAfterBoot_NoHubMeansRevert(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-revert": {{Kernel: kview("reverting")}}}}
|
||||
l, _ := judgingLeg(t, w)
|
||||
l.Hub = unreachableHub{}
|
||||
l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 5 * time.Minute, Poll: time.Minute})
|
||||
if c := kernelCalls(w); c[len(c)-1] != "kernel-revert" {
|
||||
t.Fatalf("calls = %v", c)
|
||||
}
|
||||
}
|
||||
|
||||
type unreachableHub struct{}
|
||||
|
||||
func (unreachableHub) PostOSReport(context.Context, []byte) error { return context.DeadlineExceeded }
|
||||
|
||||
// What kernel-boot found becomes the hub's outcome, with no judging and no reboot.
|
||||
func TestKernelAfterBoot_FallBackAndRevertResultsAreReported(t *testing.T) {
|
||||
for ev, want := range map[string]string{"fell_back": "fell_back", "self_reverted": "self_reverted", "revert_failed": "revert_failed"} {
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-boot": {{KernelEvent: ev,
|
||||
Kernel: mustRaw(KernelView{Running: kOld, Default: kOld, Phase: ev, From: kOld, To: kNew, Reason: "r", VMID: 9201})}}}}
|
||||
l, h := judgingLeg(t, w)
|
||||
r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{})
|
||||
if r.Outcome != want || len(h.reports) != 1 || h.reports[0].Outcome != want {
|
||||
t.Fatalf("%s: report %+v hub %+v", ev, r, h.reports)
|
||||
}
|
||||
if got := strings.Join(kernelCalls(w), ","); got != "kernel-boot" {
|
||||
t.Fatalf("%s: calls = %s", ev, got)
|
||||
}
|
||||
}
|
||||
// nothing to do → nothing reported
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-boot": {{KernelEvent: "none", Kernel: kview("good")}}}}
|
||||
l, h := judgingLeg(t, w)
|
||||
if r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{}); r.Layer != "" || len(h.reports) != 0 {
|
||||
t.Fatalf("an ordinary boot must report nothing: %+v %+v", r, h.reports)
|
||||
}
|
||||
}
|
||||
|
||||
// The signed executor STAGES (listed + the raw envelope + the kver) and never reboots.
|
||||
func TestKernelStepExecutor_StagesNeverReboots(t *testing.T) {
|
||||
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"apply": {{Upgraded: []Package{{Name: "proxmox-kernel-7.0",
|
||||
Version: "7.0.14-22"}}, Authority: "signed", Kernel: kview("staged")}}}}
|
||||
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
|
||||
e := KernelStepExecutor{Leg: l, Guest: func(context.Context) (int, error) { return 9201, nil }}
|
||||
params, _ := json.Marshal(KernelStepParams{ReleaseID: "os-kernel-1", Kver: kNew,
|
||||
Packages: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22", Origin: PVEOrigin}}})
|
||||
ctx := signedjobs.WithSignedOp(context.Background(), &reconcile.SignedOp{Blob: []byte(`{"op":"os_kernel_step"}`), Sig: []byte("SIG")})
|
||||
if err := e.Execute(ctx, OpKernelStep, params); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
pp := w.plans[len(w.plans)-1]
|
||||
sg, _ := pp["signed"].(map[string]any)
|
||||
if pp["mode"] != "apply" || pp["select"] != "listed" || pp["expect_kver"] != kNew || sg == nil ||
|
||||
sg["blob_b64"] != base64.StdEncoding.EncodeToString([]byte(`{"op":"os_kernel_step"}`)) {
|
||||
t.Fatalf("plan = %v", pp)
|
||||
}
|
||||
if got := strings.Join(kernelCalls(w), ","); got != "apply" {
|
||||
t.Fatalf("a signed stage must never reboot: %s", got)
|
||||
}
|
||||
if len(h.reports) != 1 || h.reports[0].Outcome != "staged" {
|
||||
t.Fatalf("hub = %+v", h.reports)
|
||||
}
|
||||
if err := e.Execute(context.Background(), OpKernelStep, params); err == nil {
|
||||
t.Fatal("no envelope must refuse")
|
||||
}
|
||||
bad, _ := json.Marshal(KernelStepParams{Kver: "x", Packages: []Package{{Name: "a"}}})
|
||||
if err := e.Execute(ctx, OpKernelStep, bad); err == nil {
|
||||
t.Fatal("a bad kver must refuse")
|
||||
}
|
||||
if err := e.Execute(ctx, OpPVEStep, params); err != signedjobs.ErrNoExecutor {
|
||||
t.Fatalf("another op must pass through the chain: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
// os_kernel_step is never benign.
|
||||
func TestKernelStep_IsDestructiveClass(t *testing.T) {
|
||||
if reconcile.Classify(reconcile.ClassOSKernelStep, reconcile.Provenance{}) != reconcile.Destructive {
|
||||
t.Fatal("os_kernel_step must be destructive-class (signed, operational key)")
|
||||
}
|
||||
}
|
||||
|
||||
// A kept stage report (the agent was killed mid-stage) reaches the hub as "staged" with its kernel view.
|
||||
func TestKernel_KeptStageReportIsStaged(t *testing.T) {
|
||||
w := &fakeWrapper{t: t}
|
||||
l, _ := newLeg(t, w, tonight(0))
|
||||
ring := 0
|
||||
rep := l.reportFromKept(context.Background(), WrapperReport{Layer: LayerKernel, Mode: "apply", Ring: &ring,
|
||||
Upgraded: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22"}}, Kernel: kview("staged")}, "/x/report-r-kernel-apply.json")
|
||||
if rep.Outcome != "staged" || !rep.Healthy || len(rep.Kernel) == 0 {
|
||||
t.Fatalf("kept = %+v", rep)
|
||||
}
|
||||
}
|
||||
|
||||
func TestKernelSet(t *testing.T) {
|
||||
if got := KernelSet("7.0.14-20-pve"); len(got) != 2 || got[0].Name != "proxmox-kernel-7.0" || got[0].Version != "7.0.14-20" ||
|
||||
got[1].Name != "proxmox-kernel-7.0.14-20-pve-signed" {
|
||||
t.Fatalf("%+v", got)
|
||||
}
|
||||
if KernelSet("7.0; reboot") != nil || KernelSet("") != nil {
|
||||
t.Fatal("a non-kernel string must give no set")
|
||||
}
|
||||
}
|
||||
@@ -45,6 +45,9 @@ const (
|
||||
// LayerPVE is the HOST's Proxmox userspace packages — slow lane (R-812 option A, `09` §3 decision 163, `11` §5.10):
|
||||
// ring 0 in the night leg after a healthy host step, ring 1 only inside a signed os_pve_step. Never a kernel.
|
||||
LayerPVE = "pve"
|
||||
// LayerKernel is the HOST's kernel — the kernel lane (R-836, `09` §3 decision 172, `11` §5.11): a one-shot boot of
|
||||
// the new kernel through the ESP flag, the default moved only after a healthy boot (kernel.go).
|
||||
LayerKernel = "kernel"
|
||||
)
|
||||
|
||||
// PVEOrigin is the origin apt prints for download.proxmox.com (the wrapper's PVE_ORIGIN); InstalledPVEOrigin is how
|
||||
@@ -126,6 +129,11 @@ type WrapperReport struct {
|
||||
OOMCheck json.RawMessage `json:"oom_check"`
|
||||
// PVEManager: the pve layer — pveversion's pve-manager version after the step ("unknown" = unreadable).
|
||||
PVEManager string `json:"pve_manager"`
|
||||
// Kernel (R-836): the kernel layer's view {running, default, flag, phase, from, to, …}; KernelEvent what kernel-boot
|
||||
// found after a boot; OutcomeHint "nothing" when no kernel was pending.
|
||||
Kernel json.RawMessage `json:"kernel"`
|
||||
KernelEvent string `json:"kernel_event"`
|
||||
OutcomeHint string `json:"outcome_hint"`
|
||||
// R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the
|
||||
// agent process that started the pass. ReleaseID / VMID were always in the report.
|
||||
RunID string `json:"run_id"`
|
||||
@@ -168,6 +176,10 @@ type Report struct {
|
||||
OOMCheck json.RawMessage `json:"oom_check,omitempty"`
|
||||
// PVEManager: pve layer — pve-manager's version after the step (the hub's System page; R-812 option A).
|
||||
PVEManager string `json:"pve_manager,omitempty"`
|
||||
// Kernel: kernel layer — the wrapper's kernel view, byte for byte (running, default, flag, phase, from, to). The
|
||||
// outcomes of this layer: staged | applied (the new kernel is the default) | fell_back | health_failed (self-revert
|
||||
// started) | self_reverted | revert_failed | judging | nothing | refused | failed.
|
||||
Kernel json.RawMessage `json:"kernel,omitempty"`
|
||||
|
||||
unsent string // R-868: the wrapper's kept copy of this pass's report — deleted once the hub has it
|
||||
}
|
||||
@@ -500,7 +512,7 @@ func (l *Leg) call(ctx context.Context, runID string, plan map[string]any) (Wrap
|
||||
|
||||
// Pass is one leg's reports; an empty Layer means the step did not run.
|
||||
type Pass struct {
|
||||
Guest, Host, Docker, PVE Report
|
||||
Guest, Host, Docker, PVE, Kernel Report
|
||||
}
|
||||
|
||||
// Run is one pass: the guest layer, then (on an appliance, after a good guest step) the host layer, then (ring 0
|
||||
@@ -542,6 +554,16 @@ func (l *Leg) Run(ctx context.Context, vmid int, trigger string) Pass {
|
||||
default:
|
||||
p.PVE = l.runPVE(ctx, g.RunID, vmid, trigger, blk, dockerOpts{})
|
||||
}
|
||||
// R-836 (`09` §3 decision 172): the kernel step ENDS the night — an appliance, after a healthy host step (and a
|
||||
// healthy Proxmox step when one ran), only on a night the hub marks as told. It reboots the box. Pinned by TestKernel_*.
|
||||
switch {
|
||||
case !l.Appliance || h.Layer == "" || !okStep(h):
|
||||
lg.Info("osupdate: kernel step skipped — no healthy host step this pass (an appliance only)", "appliance", l.Appliance, "host_outcome", h.Outcome)
|
||||
case p.PVE.Layer != "" && !okStep(p.PVE):
|
||||
lg.Warn("osupdate: kernel step skipped — the Proxmox step did not end healthy", "pve_outcome", p.PVE.Outcome)
|
||||
default:
|
||||
p.Kernel = l.runKernel(ctx, g.RunID, vmid, trigger, blk)
|
||||
}
|
||||
return p
|
||||
}
|
||||
|
||||
|
||||
@@ -25,6 +25,7 @@ type fakeWrapper struct {
|
||||
plans []map[string]any
|
||||
keep bool // R-868: like the real wrapper, keep an apply report beside the plan
|
||||
pveGateHeld bool
|
||||
kernelRep map[string][]WrapperReport // R-836: per kernel-layer mode, successive answers (the last one repeats)
|
||||
}
|
||||
|
||||
func yes() *bool { b := true; return &b }
|
||||
@@ -52,9 +53,19 @@ func (f *fakeWrapper) Run(_ context.Context, name string, args ...string) ([]byt
|
||||
f.plans = append(f.plans, plan)
|
||||
layer := plan["layer"].(string)
|
||||
ok := guestOK()
|
||||
if layer == LayerHost || layer == LayerPVE {
|
||||
if layer == LayerHost || layer == LayerPVE || layer == LayerKernel {
|
||||
ok = hostOK()
|
||||
}
|
||||
if layer == LayerKernel {
|
||||
if seq := f.kernelRep[plan["mode"].(string)]; len(seq) > 0 {
|
||||
rep := seq[0]
|
||||
if len(seq) > 1 {
|
||||
f.kernelRep[plan["mode"].(string)] = seq[1:]
|
||||
}
|
||||
out, _ := json.Marshal(rep)
|
||||
return []byte("OSAPPLY-REPORT " + string(out) + "\n"), []byte("os-apply: DONE rc=0\n"), nil
|
||||
}
|
||||
}
|
||||
if layer == LayerPVE && plan["mode"] == "apply" {
|
||||
f.pveGateHeld = pvegate.Stepping() // R-812: the /etc/pve write gate must be held while the pve step runs
|
||||
}
|
||||
|
||||
@@ -142,6 +142,17 @@ func (l *Leg) reportFromKept(ctx context.Context, wr WrapperReport, path string)
|
||||
default:
|
||||
rep.Outcome = "applied"
|
||||
}
|
||||
if wr.Layer == LayerKernel {
|
||||
// a kernel STAGE changes nothing the box runs (the new kernel only boots once, at the night's reboot), so its
|
||||
// kept copy needs no fresh health reading (R-836)
|
||||
rep.Kernel = rawOrNil(wr.Kernel)
|
||||
rep.Upgraded, rep.PassSeconds, rep.Authority = wr.Upgraded, wr.PassSeconds, wr.Authority
|
||||
if rep.Outcome == "applied" {
|
||||
rep.Outcome = "staged"
|
||||
}
|
||||
rep.Healthy, rep.HealthReason = rep.Outcome == "staged" || rep.Outcome == "nothing", prefix
|
||||
return rep
|
||||
}
|
||||
rep.Upgraded, rep.PassSeconds = wr.Upgraded, wr.PassSeconds
|
||||
rep.DockerEngine, rep.Authority, rep.Undo = wr.DockerEngine, wr.Authority, wr.Undo
|
||||
rep.OOMCheck = rawOrNil(wr.OOMCheck)
|
||||
|
||||
@@ -56,6 +56,11 @@ const (
|
||||
// key) like os_docker_step; the root wrapper re-verifies the same signature itself.
|
||||
ClassOSPVEStep OpClass = "os_pve_step"
|
||||
|
||||
// A kernel step on the host (R-836, `09` §3 decision 172, `11` §5.11) — ring 1: it STAGES a kernel (install + the
|
||||
// one-shot flag; the night leg reboots it). Destructive-class (signed, operational key) like os_pve_step; the root
|
||||
// wrapper re-verifies the same signature itself.
|
||||
ClassOSKernelStep OpClass = "os_kernel_step"
|
||||
|
||||
// The config bundle (agent v0.143.0, R-840, `11` §5.4.2): the box's ROOT-OWNED files (sudoers, wrappers, units).
|
||||
// Destructive-class (signed, operational key) like agent_update; the root wrapper re-verifies the signature itself.
|
||||
ClassAgentConfigUpdate OpClass = "agent_config_update"
|
||||
@@ -127,7 +132,7 @@ func Classify(class OpClass, prov Provenance) Disposition {
|
||||
return Destructive
|
||||
case ClassKeyRotation:
|
||||
return Destructive
|
||||
case ClassAgentUpdate, ClassOSDockerStep, ClassOSPVEStep, ClassAgentConfigUpdate:
|
||||
case ClassAgentUpdate, ClassOSDockerStep, ClassOSPVEStep, ClassOSKernelStep, ClassAgentConfigUpdate:
|
||||
// Never benign — no agent-internal provenance can make replacing the agent binary
|
||||
// unsigned-safe (a compromised process must not be able to self-bless an update).
|
||||
return Destructive
|
||||
|
||||
Reference in New Issue
Block a user