Compare commits

..

18 Commits

Author SHA1 Message Date
admin bdf25f019c Host power-off: a guest started after acceptance cancels it; escrow reply data race fixed
gates / gates (push) Successful in 30s
Security review of c6d9435: the other-guests check ran only at acceptance. waitJobsClear now
looks again before each step and CANCELS at once when another guest or VM runs (gate released).
Red-proofed. Also: go test -race found a data race in the escrow ceremony start reply (reads
job.Phase while the job goroutine writes it) — the reply fields are copied before the goroutine
starts; -race was red before, green 3/3 after.
2026-10-13 19:38:23 +02:00
admin c6d9435316 Host power-off: re-check jobs before each step; cancel if one is still running (R-998)
gates / gates (push) Successful in 29s
Security review of f38b03c: a controller update, drive/NAS job, escrow ceremony or Proxmox
vzdump could start between acceptance and the power-off (the heavy gate does not cover them).
runHostPowerOff now asks jobBlocker again before the guest shutdown and before the host
power-off, waits up to 15 min, and CANCELS (gate released, box stays on) if still busy.
Red-proofed: no wait, and a gate kept after a cancel, each convicted.
2026-10-13 19:36:01 +02:00
admin f38b03cdb8 Host power-off: refuse while another guest or VM runs; hold the heavy gate; fail closed (R-998)
gates / gates (push) Successful in 33s
Security review of 3403c87: (1) any guest's controller could switch off a shared host — demo-hp
runs a scratch guest and the Tester 1 VM. Now refused (other_guests_running) when /proc shows any
other lxc-start -n N or kvm -id N; an unreadable /proc refuses. (2) the checks ran once, so a
backup could start before the power-off: the heavy-operation gate is now taken on acceptance and
kept (released only if the power-off command fails). (3) an unwired gate or task reader refused
nothing; now refuses. Each red-proofed.
2026-10-13 19:33:09 +02:00
admin 3403c87c80 Host power-off for the household (R-998): POST /host/poweroff
gates / gates (push) Successful in 34s
The guest's controller asks; the agent refuses in the first 10 min after boot, a second time in
one boot (persisted, keyed by /proc/stat btime), and while a backup, restore test, OS step,
controller update, drive format, NAS verify, escrow ceremony or vzdump runs (each a named 409
reason). On acceptance: the record first, a host report carrying deliberate_off accepted by the
hub (bounded 60 s), then pct shutdown --timeout 120, then systemctl poweroff.

Sudoers: two exact vectors (pct ^shutdown N --timeout N$, systemctl poweroff) in FELHOM_POWEROFF;
nine injection cases. New TestEveryAliasIsGranted: a defined alias missing from the grant line
passed the coverage check before. Every new test red-proofed.
2026-10-13 19:29:52 +02:00
admin 86adbcaf8a rules: "The system poster stays true" (section 6, all five copies identical)
gates / gates (push) Successful in 57s
The shared rule file carries the same wording in felhom.eu, felhom-controller,
felhom-agent, app-catalog-felhom.eu and the workspace root on DooPlex --
"change all five or none" -- so this is this repo's copy of a rule added in
felhom.eu a08bd3cbd5.

A session that changes a fact listed in
architecture/felhom-system-poster.facts.md (a machine, a role, a traffic path,
a backup tier, a time, a retention, a key, a known gap) updates that file in
the same commit; fixes the poster's text if the change is text only; or adds
"System poster needs a refresh: <what changed>" to STATUS. The report names
which of the three it did.

The poster is drawn in Claude Design and nothing in the repo renders it, so
scripts/poster_facts_gate.py only WARNS when the facts outrun the drawing --
it never fails a push, because a refresh needs the operator and another tool.

Verified identical to the other four copies by diff, before and after.
2026-10-09 18:10:19 +02:00
admin 24ea960229 REPORT: v0.154.0 released and delivered (2026-10-09)
gates / gates (push) Successful in 1m7s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-09 08:36:51 +02:00
admin cff6fb5a45 v0.154.0: CHANGELOG release entry (sha, bundle, tag)
gates / gates (push) Successful in 1m11s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-09 06:58:00 +02:00
admin db25b469ba Host report keeps the last backup across a restart; Secure Boot meta-package leaves the Proxmox lane
gates / gates (push) Successful in 1m7s
Found in the 2026-10-09 kernel-night read-back:
- demo-felhom: the kernel step restarted the host 4 minutes after the night
  backup; the in-memory backup list was empty after the restart, so the hub
  alarmed "host tier: newest backup is 48h old". The report now adds R-894's
  saved newest success per tier (one shared instance).
- demo-hp: proxmox-secure-boot-support pulled shim-signed-common into the
  ring-0 Proxmox plan; the step was refused R6 and the kernel step skipped.
  It joins HOST_SLOW_RE with shim and GRUB.

Red-proved: TestCollectBackups_SavedSuccessSurvivesARestart,
TestR894_LastKnownBackupsIsWiredIntoTheDaemon,
test_ring0_pending_pve_leaves_the_secure_boot_meta_out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-09 06:47:58 +02:00
admin b228b44f0e REPORT/CONTEXT: 2026-10-08 day (R-899, R-304 on main, unreleased)
gates / gates (push) Successful in 52s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-08 08:08:53 +02:00
admin 91b9405c81 R-304: no 'wrong code' when earlier sealed packages were not all checked (424 older_unchecked)
gates / gates (push) Successful in 55s
Unreleased; ships with tomorrow's release.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-08 07:53:01 +02:00
admin 4c69c25b48 R-899: no OS leg after a household press (trigger=manual); after-boot kernel reports carry the saved ring
gates / gates (push) Successful in 50s
Unreleased; ships with tomorrow's release.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-08 07:34:19 +02:00
admin c9013bb47d CHANGELOG/REPORT: v0.153.0 released and delivered
gates / gates (push) Successful in 1m33s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 19:06:04 +02:00
admin 2d1e5d0774 R-898: ring 0 stages exactly the told kernel (select listed, KernelSet(kver))
gates / gates (push) Successful in 1m6s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 18:45:28 +02:00
admin f09c53efc1 REPORT/CONTEXT: v0.152.0 the kernel lane
gates / gates (push) Successful in 1m9s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 18:20:11 +02:00
admin 92d647a6f6 CHANGELOG: v0.152.0 released (shas, step bundle); host_stale is 45 min (comment fix)
gates / gates (push) Successful in 1m0s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 15:42:26 +02:00
admin d03ab7f1f5 R-836: the kernel lane — one-shot boot through the ESP flag, boot good / one self-revert, night step on a told night
gates / gates (push) Successful in 44s
Wrapper layer kernel (stage / reboot / boot / good / revert / cancel / status;
R20-R23), the two GRUB generators in the bundle (option C on the one-shot
entry), the agent's night step and after-boot judge (host health rule + hub
reached, 20 min measured), the signed os_kernel_step (stage only).
Red-proofs: felhom.eu audits/kernel-lane-2026-10-07/A/redproof.txt.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 15:18:24 +02:00
admin b17d1c597d REPORT/CHANGELOG: 2026-10-07 day
gates / gates (push) Successful in 59s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 14:21:01 +02:00
admin 0ae01dfb52 CHANGELOG: v0.151.0 released (R-861, R-812 A, R-366, R-105)
gates / gates (push) Successful in 58s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 13:14:47 +02:00
43 changed files with 3633 additions and 54 deletions
+16
View File
@@ -73,3 +73,19 @@ that no longer exists. Each edit is named in the report (file, line, before, aft
operator's word: loosen a safety rule, a fence, a „never", a protected machine, a secret rule, or a review step; or
remove a rule. When in doubt, it is a rule change, and it goes to the operator. If Claude Code's own permission check
asks before such an edit, wait for the operator's click; if it refuses, record that and file the exact line.
## 6. The system poster stays true
**A session that changes a fact listed in `architecture/felhom-system-poster.facts.md` (a machine, a
role, a traffic path, a backup tier, a time, a retention, a key, a known gap) updates that file in
the same commit.** If the change is text only, it also edits the matching text in
`felhom-system-poster.html`. If it needs a new drawing, it adds the line
**"System poster needs a refresh: <what changed>"** to `STATUS.md`'s "waiting on the operator" list.
**The session report names which of the three it did.**
Why the three-way split: the poster is a drawing made in Claude Design, and **nothing in this
repository renders it** — unlike `where-felhom-stands.html`, which has `render_stands.py`. So the
facts file is the source of truth and the drawing trails it. `scripts/poster_facts_gate.py` warns
when the facts file has a newer commit than the poster; it **never fails a push**, because a refresh
needs the operator and another tool, and a gate nobody can clear is a gate people learn to route
around.
+116 -3
View File
@@ -1,4 +1,117 @@
## Unreleased (2026-10-07) — the agent can no longer hand the guest any image; felhom-op's pct lines are exact (R-861 (a) A1, (b) B2; `09` §3 decision 165)
## v0.154.0 — no OS leg after a household press; the after-boot kernel report carries the real ring (R-899); no „wrong code" when older packages were not checked (R-304); the last backup survives a restart in the host report; the Secure Boot meta-package leaves the Proxmox lane (2026-10-09)
Released by `scripts/release-agent.sh`: binary sha256 `36a54ad20e896cd61d7b9944e2a521f4209c2b76940d8f1827af05b645463226`, bundle
`93487989f50d4a3039a05cec0f91ec478d62489a414705e49403e9b1340d20e0` (tag `v0.154.0` = `db25b46`).
**Delivery: the agent binary AND the config bundle** — the wrapper `felhom-os-apply` changed (2026-10-09).
- **The hub's backup evidence survives a restart (2026-10-09, found in the kernel-night read-back).** The host report's
`backups` list came only from memory, which is empty after an agent restart. On demo-felhom the night backup landed
02:40 UTC and the kernel step restarted the host at 02:44 — no report fell in between, two nights running — so the hub
alarmed „host tier: newest backup is 48h old" at 03:00 after a good backup. The report now adds R-894's saved newest
success per tier and guest (`backup-success-state.json`, ONE instance shared with the local API) where memory holds no
success at or after it. Only successes are saved, so a failure is never hidden. `hub.KnownBackupReporter`,
`BackupSuccessState.KnownBackupSuccesses`; tests `TestCollectBackups_SavedSuccessSurvivesARestart` (red-proved: the
restart case reported nothing), `TestBackupSuccessState_KnownBackupSuccessesSurviveAReopen`, and
`TestR894_LastKnownBackupsIsWiredIntoTheDaemon` now also asserts the collector gets the same instance (red-proved).
- **`proxmox-secure-boot-support` is a boot-chain package (2026-10-09).** On demo-hp (Secure Boot on) its upgrade pulled
`shim-signed-common`; in the ring-0 Proxmox plan that refused the whole step (R6) and skipped the night's kernel step,
after the household had been told „tonight". It joins `HOST_SLOW_RE` with shim and GRUB (they stay out of every lane,
as before). `test_ring0_pending_pve_leaves_the_secure_boot_meta_out` (red-proved).
- **R-899 (operator ruling 2026-10-08, option A):** `POST /backup?trigger=manual` (a „Mentés most" press, sent by the
controller from its next release) runs no OS leg after it: the leg belongs to the night, after the night's own copy.
Before, a press ran the leg at once (in the day; on 2026-10-07 08:51 demo-hp skipped it only by the 20-hour rule). A
request without the parameter (an older controller, the scheduled path) behaves as before.
`internal/localapi/server.go`; `TestAfterPrimaryBackup` gained two sub-cases (red-proved: ignoring `trigger`, the leg
ran once after a press).
- **R-304 — „wrong code" only when every earlier package was tried.** `POST /escrow/recover-offsite-password` answered
400 („the recovery code did not open the sealed bundle") whenever the current package and the TRIED earlier packages
refused the code — even when the hub withheld earlier packages (rows with no key material, rows over its serve cap), a
served package was malformed, the 6-attempt cap stopped the loop, or the retained list could not be read at all. Now
those cases answer **424** with `older_unchecked` (the count, -1 = unknown) and a sentence that does not call the code
wrong. `escrow.ErrRetainedUnchecked` / `RetainedUncheckedError`; the retained fetcher's second value is now every
withheld package (unopenable + truncated + malformed). Tests `TestR304_*` (real age crypto; red-proved: without the
check, four cases returned the wrong-code error) and a 424 row in `TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus`.
An older controller maps the unknown 424 to its neutral „we do not know why" sentence.
- **The „ring 1" label after a boot:** in the first second after a reboot the agent has not fetched the hub's block, and
the after-boot kernel reports (`judging`, `good`, `revert`, `fell_back`…) said ring 1 on a ring-0 box (demo-felhom,
2026-10-08 night). Now they read the fetched block, else the block the daemon saved on disk before the reboot (R-866),
else ring 1 as before. A label only — the hub's approval reads its own ring list. `kernelReportRing`,
`TestKernelReportRing_BeforeFirstFetch` (red-proved: ring 1, want 0).
## v0.153.0 — ring 0 stages exactly the told kernel (R-898; `09` §3 decision 176) (2026-10-07)
Released by `scripts/release-agent.sh`: binary sha256 `b204ebe6d65944f43b70dcfa4ae0dfb90da38c92e2ba6a38a1826e488eeb6630`, bundle
`6db216275eb0dd39188d93481a2045998a69e8cb7838ad908a58d65d9a6a9b57` (tag `v0.153.0` = `2d1e5d0`). Delivered (binary only — the
bundle files are unchanged from 0.152.0) to demo-hp, demo-felhom, Tester 1 on 2026-10-07 19:03.
**Delivery: the agent binary only** — no root file changed (the wrapper is unchanged; its tests gained two cases).
- `internal/osupdate/kernel.go`: ring 0's night kernel step stages EXACTLY the kernel the household was told about
(select `listed`, `KernelSet(kver)` = the series meta-package and the signed image at the kernel's own version) instead
of "whatever is pending tonight". Seen 2026-10-07: demo-felhom was told about 7.0.14-20 while its sources offered
7.0.14-22 by night — the old code staged `pending-kernel` and the wrapper refused it (R23), losing the night. A told
version that is no longer installable is refused by the wrapper before any change (R7) and the hub tells the household
again for the newer kernel (hub v0.143.1). Tests `TestKernel_Ring0ToldNightStagesThenReboots` (red-proved against the
old select), `TestKernelSet`; wrapper `test_ring0_listed_installs_the_told_kernel_not_the_newest`,
`test_ring0_told_kernel_gone_is_refused_before_any_change`.
## v0.152.0 — the kernel lane (R-836; `09` §3 decisions 164, 172; `11` §5.11) (2026-10-07)
Released by `scripts/release-agent.sh`: binary sha256 `95ff42208e36ba49b6e2b09a97e81a6fa11562ecc8042f1ed18d378b8b1f88b1`,
config bundle sha256 `f0c2cec374b711b3c131c955012b33fd0ad495337d049b7a743c3eef9e85c20b` (tag `v0.152.0` = `d03ab7f`).
Step bundle `0.152.0-step1` sha256 `0b71d32b054cf3b7ade0234ffcbb0df159901f542cde540adaee411db466f48e` (the 0.151.0 bundle
with only `felhom-os-apply` replaced; `scripts/build-step-bundle.py`), published as package version `0.152.0-step1`.
**Delivery: agent binary, then the STEP bundle `0.152.0-step1`, then the bundle `0.152.0`** — the bundle ADDS two paths
(the GRUB generators), and an installed `felhom-os-apply` refuses a path its own table lacks (R16, R-880).
A new kernel boots ONCE; if it crashes the box comes back on the old kernel by itself; it becomes the default only after
a healthy boot; a booted-but-unhealthy kernel is reverted ONCE by the agent with no person. Built on the spike's
candidate 2 (`audits/kernel-spike-2026-10-07/`), with option C on the one-shot entry.
- `configs/felhom-grub-oneshot.sh` → `/etc/grub.d/01_felhom_oneshot` (bundle): reads `felhom_next` from a GRUB env block
on the ESP (`EFI/felhom/oneshot.env`), clears and saves it BEFORE the menu, and sets the default to that kernel's
one-shot entry only when the name is an installed kernel. No vfat ESP → prints nothing.
- `configs/felhom-grub-oneshot-entries.sh` → `/etc/grub.d/42_felhom_oneshot` (bundle): one entry per installed kernel,
id `felhom-oneshot-<ver>`, the normal entry plus `softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10`
(option C). Sorted after `10_linux`: never entry 0, never the default.
- `configs/felhom-os-apply`: layer `kernel` (lane slow; an appliance; authority = a signed `os_kernel_step` or the
root-owned ring-0 mark). Modes: `apply` STAGES (select `pending-kernel` or a signed `listed` set; `expect_kver` = the
kernel the household was told about): pins the GRUB default to the RUNNING kernel in
`/etc/default/grub.d/zz-felhom-kernel-default.cfg` and proves it from grub.cfg, installs, proves the default did not
move and the one-shot entry exists, writes the flag; never reboots. `kernel-reboot` (a staged step only),
`kernel-boot` (judging | fell_back | self_reverted | revert_failed), `kernel-good` (the new kernel becomes the
default, proved), `kernel-revert` (ONE per step; refused when the default is not the old kernel), `kernel-cancel`,
`kernel-status`. New refusals: R20 (the box cannot do a one-shot: not UEFI, no vfat ESP, a separate /boot, GRUB
without fat/loadenv, the generators missing, a hand pin), R21 (the crash guard tripped or an unclean boot in its
window), R22 (the phase does not allow the mode; never two steps within 20 h), R23 (not exactly one newer kernel, or
not the one signed / told). State `/var/lib/felhom-kernel/state.json`. Facts carry `kernel_lane`; the next-boot
kernel reads the flag and the grub.cfg default. Tests: `KernelLane` (27), red-proof
`felhom.eu/documentation/audits/kernel-lane-2026-10-07/A/redproof.txt`.
- `configs/felhom-crash-guard` unchanged; `KernelStepCannotLeaveTheBoxOff` (3 tests) pins that a step's planned reboot,
one crash and one self-revert add ONE unclean boot (a panic before userspace adds none), so the box cannot stay off.
- `internal/osupdate/kernel.go`: the night leg ends with the kernel step — after a healthy host step (and a healthy
Proxmox step when one ran), trigger `night` only, on a night the hub's `os_update.kernel` block marks `tonight` (the
household was mailed the day before — no mail, no step). Ring 0 stages + reboots; ring 1 reboots only a kernel a
signed `os_kernel_step` staged (`KernelStepExecutor`: stage only, under the heavy-op gate). The hub hears `staged`
BEFORE the reboot. At every start `KernelAfterBoot`: on the new kernel it JUDGES the boot — `KernelVerdict` = the
host health rule (`11` §8.2) AND the box reached the hub (the `judging` report itself) — for 20 minutes (measured:
everything healthy 68 s after the reboot on demo-felhom, 272 s on demo-hp; under the hub's 45-minute `host_stale` (`alerting.stale_threshold`)).
Healthy → `kernel-good`, outcome `applied`; not healthy → outcome `health_failed`, then ONE `kernel-revert`.
Tests: `TestKernel*` (13); red-proofs in the same file.
- `internal/hub`: `WireOSUpdate.Kernel` {kver, tonight, notified_at}. `internal/reconcile`: `os_kernel_step` is
destructive-class. `cmd/felhom-opsign`: the op is listed.
## v0.151.0 — the agent can no longer hand the guest any image; the Proxmox package lane; the other-key archives reported; the DR directive retired (R-861, R-812 A, R-366, R-105; `09` §3 163, 165, 168, 169) (2026-10-07)
Released by `scripts/release-agent.sh`: binary sha256 `0464354f2cdf452a7c5d2a74d9191fe91415fcfa244154480d26d5b30e10b194`
config bundle sha256 `bacd1d175a9392bc1755341d01a106abbd72aba48ab575e7a32dd819d8f5da4c` (tag `v0.151.0` = `dd7cdc0`).
**Deliver the binary FIRST, then the bundle:** the bundle's sudoers removes the `tee` grant the 0.150.0 binary still uses.
No path added (26 → 26), so no step bundle.
### Part of v0.151.0 — the agent can no longer hand the guest any image; felhom-op's pct lines are exact (R-861 (a) A1, (b) B2; `09` §3 decision 165)
**Delivery order: agent binary FIRST, then the config bundle.** The new sudoers drops the agent's in-guest `tee`
grant; an older binary still calls `tee`, so a bundle that lands before the binary would stop managed controller
@@ -20,7 +133,7 @@ updates (and the old binary's capability probe would read `controllerswap-write`
`TestSudoersAllowsTheControllerImageVerb`, `TestFelhomOpSudoersPctIsExact`, `TestR861_WriteControllerImageUsesTheRootVerb`,
`TestControllerSwap_WriteViaRootVerb_NoShell`. Red-proofs: `felhom.eu/documentation/audits/day-2026-10-07/C/`.
- `README.md`: the controller-swap paragraph described the removed `tee` path — corrected.
## Unreleased (2026-10-07) — the Proxmox package lane (R-812 option A, `09` §3 decision 163)
### Part of v0.151.0 — the Proxmox package lane (R-812 option A, `09` §3 decision 163)
**MinAgent impact: none** (a new layer; an older hub ignores the pve report). **The bundle carries the new
`felhom-os-apply` — deliver it with the binary** (signed `agent_update`, then signed `agent_config_update`).
@@ -29,7 +142,7 @@ updates (and the old binary's capability probe would read `controllerswap-write`
- `internal/pvegate` (new): the agent's own writes to /etc/pve wait while a pve step runs (pmxcfs restarts); the step waits for writes in flight (bounded, 2 min — then it fails and does not run). Wired at `proxmox.Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile).
- `internal/osupdate`: `LayerPVE`; the night leg runs the pve step in ring 0 after a healthy host step (an appliance; ring 1 never in the night leg); `PVEHealthVerdict` = the host rule + every running container keeps its id + pveversion reads the installed pve-manager; the pve report carries Proxmox userspace only (the hub's candidate set). `PVEStepExecutor` (signed `os_pve_step`, ring 1, under the heavy-op gate and the /etc/pve gate); `reconcile.ClassOSPVEStep` (destructive-class); `felhom-opsign -op os_pve_step` (params by `-params`).
- Tests: wrapper `PVELane` (17; red first — the `pve-manager` plan was refused R12 on the old code), `pvegate` (5), `TestPVEGate_*` + `TestWritesEtcPVE`, `TestPVE_*`, `TestPVEHealthVerdict`, `TestPVEStepExecutor_*`. Red-proofs: `felhom.eu/documentation/audits/day-2026-10-07/B/`.
## Unreleased (2026-10-07)
### Part of v0.151.0
- R-366 slice 2 (`09` §3 decision 168): the restore-test pick records, per tier, the archives it skipped as written with another key (count, oldest, newest — no key material) in a `ForeignKeyLedger`; the host report carries it as `foreign_key_archives.tiers` (the stanza absent until a tier was evaluated since start, `tiers: []` when none — no null on the wire, the report contract forbids it). The hub turns a change into one operator line. Tests `TestR366_PickRecordsArchivesWrittenWithAnotherKey`, `TestR366_EvaluatedWithNoneIsAnEmptyList` (red-proved, `felhom.eu/documentation/audits/day-2026-10-07/E/`).
- R-105 option A (`09` §3 decision 169): the `--selftest=escrow-create -directive <file>` flag and the escrow upload's `directive` field are removed — nothing read the directive; the DR path reads the recipe, tenantsync and the escrow blob. The hub ignores a `directive` from an older agent.
+7
View File
@@ -1,5 +1,12 @@
# CONTEXT — felhom-agent working state
> **2026-10-08 (day) — UNRELEASED on main, ships with tomorrow's release (decision 178).** R-899: `POST /backup?trigger=manual`
> runs no OS leg (`localapi/server.go`); after-boot kernel reports carry the saved block's ring (`kernelReportRing`). R-304:
> `escrow.ErrRetainedUnchecked` → HTTP 424 `older_unchecked` when not every earlier package was tried (withheld, caps,
> malformed, unreadable list). Binary-only delivery; no wrapper or root file changed.
> **2026-10-07 (evening) — v0.152.0, the kernel lane (R-836, decision 172, `11` §5.11).** Wrapper layer `kernel` + two GRUB generators in the bundle (delivered with step bundle `0.152.0-step1` — the bundle adds paths, R-880); `osupdate/kernel.go`: night step on told nights only, after-boot judge (host rule + hub reached, 20 min), ONE self-revert, `os_kernel_step` stages only. Proven on Tester 1 (panic → fell_back, held guest → self_reverted, healthy → 7.0.14-22 default). Open: R-898, R-897; the ring-0 night run.
> **2026-10-04 night — v0.143.0 RELEASED + vouched (R-840, decision 96): the config bundle.** `felhom-os-apply` mode
> `bundle` (signed `agent_config_update`, verified by the wrapper itself; trust files never bundle paths) +
> `--install-bundle` (installer 1.31.0); `BUNDLE_FILES` is the one table; `scripts/build-config-bundle.py`;
+14 -7
View File
@@ -1,8 +1,15 @@
# REPORT — v0.150.0 released and delivered (2026-10-07)
# REPORT — v0.154.0 released and delivered (2026-10-09)
On the operator's word (`09` §3 decision 161). `scripts/release-agent.sh 0.150.0`: sha `a23d1c90…`, bundle `88456b38…`,
tag `v0.150.0` = `3a72a48`, verified by download. Vouched with golden 0.301.0 and MinAgent 0.131.0 (unchanged). No bundle
path added (26 → 26), so no step bundle. Signed `agent_update` → demo-hp, demo-felhom, Tester 1 on 0.150.0 (07:06–07:07Z);
signed `agent_config_update` → `BUNDLE DONE written=1 same=24 self-check=ok`, capability probe 68/68 on all three
(07:21Z). Tester 2 not touched. Carries R-528 (the memory-kill check), R-894 (the last backup per tier on disk), R-330
(SMART counters on the wire). Evidence: `felhom.eu/documentation/audits/readback-2026-10-07/delivery/`.
Built from the 2026-10-09 kernel-night read-back: (1) the host report now adds the saved newest backup success per tier
(R-894's file, one shared instance), so a restart right after a night backup no longer empties the hub's evidence — on
demo-felhom it had caused a false „newest backup is 48h old" alarm; (2) `proxmox-secure-boot-support` joins the
boot-chain lane, so it no longer pulls `shim-signed-common` into the ring-0 Proxmox plan (demo-hp's step was refused R6
and its kernel step skipped). Red-proofs: `TestCollectBackups_SavedSuccessSurvivesARestart`,
`TestR894_LastKnownBackupsIsWiredIntoTheDaemon`, `test_ring0_pending_pve_leaves_the_secure_boot_meta_out`.
`scripts/release-agent.sh 0.154.0`: sha `36a54ad2…`, bundle `93487989…`, tag `v0.154.0` = `db25b46`, verified by
download. CI 1576 (code) success; 1579 (CHANGELOG) failed in its fetch step (Gitea answered 500), re-run → success.
Signed `agent_update` → demo-hp, demo-felhom, Tester 1 (07:16–07:18 local); signed `agent_config_update` → `BUNDLE DONE
written=1 same=26 self-check=ok`, capability probe 68/68 on all three. NOT vouched (the vouch needs a golden at or above
the fleet's controller; no golden today). Live: demo-felhom's first report after the update carries the 02:40Z backup.
Evidence: `felhom.eu/documentation/audits/release-2026-10-09/delivery/`.
+1
View File
@@ -89,6 +89,7 @@
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `pvegate.Write` / `pvegate.Step` | internal/pvegate/pvegate.go | `Write(ctx) (release, waited, err)` / `Step(ctx) (end, err)` | R-812 option A: keep the agent's own /etc/pve writes out of a Proxmox package step (pmxcfs restarts) | Already wired at the two chokepoints — `Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile). A new root CLI that writes /etc/pve goes into `WritesEtcPVE`, never its own lock. Never take `Step` around anything but the wrapper call (`Leg.runPVE`) — a `Write` inside a `Step` deadlocks until its context ends. |
| `osupdate.KernelVerdict` / `Leg.KernelAfterBoot` | internal/osupdate/kernel.go | `KernelVerdict(before, after, tunnel, hubReached) (ok, why)` | R-836: THE one-shot-boot rule — the host health rule (`HostHealthVerdict`) AND the box reached the hub since the boot | The only judge of a new kernel. Never reboot the host from Go: every reboot is the wrapper's (`kernel-reboot`, `kernel-revert`), and only for a staged step. A new kernel-lane state lives in the wrapper's `/var/lib/felhom-kernel/state.json`, never in the agent's own files (the agent can write those). |
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
+75 -3
View File
@@ -50,6 +50,7 @@ import (
"gitea.dooplex.hu/admin/felhom-agent/internal/pbs"
"gitea.dooplex.hu/admin/felhom-agent/internal/pbsdr"
"gitea.dooplex.hu/admin/felhom-agent/internal/poke"
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
"gitea.dooplex.hu/admin/felhom-agent/internal/provision"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
@@ -833,7 +834,25 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// (The startup self-check log runs AFTER the pbsdr manager is wired below, so its snapshot
// already carries the gated view — v0.86.0.)
collector.SetCapabilityProber(probeAll)
// R-998 (doc 14): the household power-off record of THIS boot rides every report until the box is down.
powerOffState := poweroff.New(filepath.Join(cfg.OOB.WithDefaults().StateDir, "poweroff.json"), poweroff.ProcBtime)
collector.SetDeliberateOffSource(func() *hub.DeliberateOff {
if r := powerOffState.Current(); r != nil {
return &hub.DeliberateOff{RequestedAt: r.RequestedAt.UTC().Format(time.RFC3339)}
}
return nil
})
loop := hub.NewLoop(collector, client, time.Duration(hcfg.PollSeconds)*time.Second, logger)
// R-998: the power-off path waits until the hub ACCEPTED a report carrying the record.
powerOffAck := make(chan struct{}, 1)
loop.SetReportedHook(func(r *hub.HostReport) {
if r != nil && r.DeliberateOff != nil {
select {
case powerOffAck <- struct{}{}:
default:
}
}
})
interval := time.Duration(hcfg.PollSeconds) * time.Second
// Heartbeat log-pull (v0.83.0): when the control envelope requests it, the NEXT
// heartbeat carries the debug ring's tail (newest-kept, byte-capped in the loop).
@@ -877,6 +896,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
go osLeg.SendUnsentLoop(ctx, 5*time.Minute, func(n int) {
logger.Info("osupdate: sent kept report(s)", "count", n)
})
// R-836 (`09` §3 decision 172): what became of a kernel step across this boot; on a one-shot boot of a new kernel,
// judge it (the host health rule + the hub reached) for KernelJudgeWait, then make it the default or revert ONCE.
// The wait: measured 2026-10-07 (`audits/kernel-lane-2026-10-07/B/`) — every container healthy 68 s after the
// reboot on demo-felhom and 272 s on demo-hp (the hub reached at 63 s / 189 s); 20 minutes leaves room for a slow
// network and stays under the hub's 45-minute host_stale (its
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
// (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving
@@ -1119,6 +1145,16 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
}
return release, nil
}}
// R-836 (`09` §3 decision 172): a signed kernel step STAGES a kernel on a ring-1 box (install + the one-shot flag,
// never a reboot — the night leg reboots it on a night the household was told about); under the heavy-op gate.
kernelExec := osupdate.KernelStepExecutor{Leg: osLeg, Guest: firstGuest(px),
Gate: func(ctx context.Context) (func(), error) {
release, busy, ok := heavyOps.TryAcquire("os-kernel-step")
if !ok {
return nil, fmt.Errorf("busy: %s", busy)
}
return release, nil
}}
// Agent v0.143.0 (R-840): the config bundle — the box's root-owned files by a signed job; the wrapper verifies it.
bundleExec := osupdate.ConfigUpdateExecutor{Leg: osLeg, URLTemplate: suCfg.URLTemplate, Username: suCfg.Username, Token: suCfg.Token,
// The capability probe confirms from the agent's side: `sudo -l` lists every command the new sudoers grants.
@@ -1130,7 +1166,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
}
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
}}
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, pveExec, bundleExec}, cfg.Hub.HostID, logger)
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, pveExec, kernelExec, bundleExec}, cfg.Hub.HostID, logger)
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner))
// Controller-driven escrow ceremony (v0.88.0): static config facts + the LATE-BOUND DR gate —
@@ -1163,6 +1199,34 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
}
localSrv := buildLocalAPIServer(cfg, px, backupStore, heavyOps, observer, driveKnown, hostOps, gate, collector, client, intentRec, guestBindStore, formatJobStore, logRing, escrowCeremonyCfg, logger, &localTokens)
if localSrv != nil {
// R-998 (doc 14): POST /host/poweroff — guest shutdown, then host power-off; guards in hostpoweroff.go.
pwMode := proxmox.RunnerMode(cfg.Privileged.Mode)
if pwMode == "" {
pwMode = proxmox.RunnerSudo
}
localSrv.SetHostPowerOff(localapi.HostPowerOffConfig{
State: powerOffState,
Power: localapi.NewGuestBinder(&proxmox.ExecRunner{Mode: pwMode, SudoPath: cfg.Privileged.SudoPath}, logger),
ReportNow: func(ctx context.Context) bool {
select {
case storageTrigger <- struct{}{}:
default: // a report is already queued; it runs after the record was written
}
select {
case <-powerOffAck:
return true
case <-ctx.Done():
return false
}
},
Uptime: func() (time.Duration, bool) {
b, err := strconv.ParseInt(poweroff.ProcBtime(), 10, 64)
if err != nil || b <= 0 {
return 0, false
}
return time.Since(time.Unix(b, 0)), true
},
})
localSrv.SetAfterPrimaryBackup(func(ctx context.Context, vmid int) {
// Let the controller finish bringing its apps back after the backup, then run (still under the gate).
select {
@@ -1887,11 +1951,15 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
return nil, 0, ferr
}
out := make([]escrow.RetainedBlob, 0, len(resp.Packages))
// R-304: every package the hub holds and the code will NOT be tried against — no key material,
// over the hub's cap, or malformed here. A refusal may call the code wrong only when this is 0.
withheld := resp.UnopenableCount + resp.TruncatedCount
for _, p := range resp.Packages {
blob, derr := base64.StdEncoding.DecodeString(p.IdentityEscrowB64)
if derr != nil || len(blob) == 0 {
// One malformed package must not sink the rest — the customer's code may open a
// later one, and a skipped entry is strictly better than a refusal we cannot justify.
withheld++
continue
}
out = append(out, escrow.RetainedBlob{
@@ -1901,9 +1969,13 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
Index: p.Index,
})
}
return out, resp.UnopenableCount, nil
return out, withheld, nil
},
}
// R-894: ONE instance — the local API writes it, the host report reads it (2026-10-09: a restart right
// after the night backup erased the hub's evidence of it).
lastKnownBackups := backup.NewBackupSuccessState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "backup-success-state.json"))
collector.SetKnownBackupReporter(lastKnownBackups)
srv, err := localapi.NewServer(localapi.Options{
EscrowRecovery: escrowRecoverer,
ListenAddr: cfg.LocalAPI.ListenAddr,
@@ -1916,7 +1988,7 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
Store: store,
// R-894: the newest success per tier on disk — the due-check's fallback when the storage cannot be
// read right after a restart. Same state dir as restore-test-state.json.
LastKnownBackups: backup.NewBackupSuccessState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "backup-success-state.json")),
LastKnownBackups: lastKnownBackups,
Storage: observer,
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
+31
View File
@@ -12,12 +12,18 @@ import (
//
// COMPANION RED-PROOF (observed): delete the `LastKnownBackups:` line from buildLocalAPIServer → this
// fails with "localapi.Options in buildLocalAPIServer has no LastKnownBackups field". Restored.
//
// 2026-10-09: the SAME instance also feeds the host report (collector.SetKnownBackupReporter), so a
// restart right after a backup no longer erases the hub's evidence of it. The field may name a local
// variable; the variable must be built by backup.NewBackupSuccessState and be the one handed to the
// collector. RED-PROOF (observed): drop the SetKnownBackupReporter call → "not handed to the collector".
func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
_, f := parseMain(t)
if !callsWithin(f, "main")["runDaemon"] || !callsWithin(f, "runDaemon")["buildLocalAPIServer"] {
t.Fatal("main → runDaemon → buildLocalAPIServer is broken — the path this test asserts is not the live one")
}
var field, built bool
var fieldVar, handed string
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
@@ -45,6 +51,28 @@ func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
if callsIn(kv.Value)["backup.NewBackupSuccessState"] {
built = true
}
if id, ok := kv.Value.(*ast.Ident); ok {
fieldVar = id.Name
}
}
}
return true
})
// a local variable: built by NewBackupSuccessState, and handed to the collector
ast.Inspect(fd.Body, func(n ast.Node) bool {
switch x := n.(type) {
case *ast.AssignStmt:
for i, l := range x.Lhs {
if id, ok := l.(*ast.Ident); ok && fieldVar != "" && id.Name == fieldVar && i < len(x.Rhs) &&
callsIn(x.Rhs[i])["backup.NewBackupSuccessState"] {
built = true
}
}
case *ast.CallExpr:
if fn, ok := x.Fun.(*ast.SelectorExpr); ok && fn.Sel.Name == "SetKnownBackupReporter" && len(x.Args) == 1 {
if id, ok := x.Args[0].(*ast.Ident); ok {
handed = id.Name
}
}
}
return true
@@ -56,6 +84,9 @@ func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
if !built {
t.Fatal("LastKnownBackups is not built by backup.NewBackupSuccessState")
}
if handed == "" || handed != fieldVar {
t.Fatalf("the saved backups (%q) are not handed to the collector (SetKnownBackupReporter got %q)", fieldVar, handed)
}
}
func callsIn(n ast.Node) map[string]bool {
+1 -1
View File
@@ -43,7 +43,7 @@ func main() {
func run() error {
var (
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | os_pve_step | agent_config_update")
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | os_pve_step | os_kernel_step | agent_config_update")
host = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)")
guest = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)")
keyID = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)")
+8 -2
View File
@@ -80,7 +80,13 @@ Cmnd_Alias FELHOM_DNSMASQ = \
Cmnd_Alias FELHOM_GUESTHOOK = \
/usr/sbin/pct ^set [0-9]+ --hookscript local\:snippets/felhom-guest-hook\.sh$, \
/usr/sbin/pct ^set [0-9]+ --delete mp[0-9]+$, \
/usr/sbin/pct ^reboot [0-9]+$
/usr/sbin/pct ^reboot [0-9]+$, \
/usr/sbin/pct ^shutdown [0-9]+ --timeout [0-9]+$
# Host power-off (R-998, doc 14): the household's „A gép kikapcsolása" — guest shutdown above, then this one fixed
# vector. Guarded agent-side (10 min after boot, one per boot, never during a backup/restore/update/drive job).
Cmnd_Alias FELHOM_POWEROFF = \
/usr/bin/systemctl poweroff
# Intermediary mount model (the drive hot-swap re-architecture). The agent keeps a SHARED host parent
# /mnt/felhom-drives (self-bind + make-shared) and binds/unbinds each drive's felhom-data namespace UNDERNEATH it so the
@@ -314,4 +320,4 @@ Cmnd_Alias FELHOM_GUESTNET = \
/usr/sbin/pct ^exec [0-9]+ -- pgrep -x dhclient$, \
/usr/sbin/pct ^exec [0-9]+ -- dhclient -pf /run/dhclient\.eth0\.pid -lf /var/lib/dhcp/dhclient\.eth0\.leases eth0$
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_FSTRIM, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY
felhom-agent ALL=(root) NOPASSWD: FELHOM_MOUNT, FELHOM_DISK, FELHOM_PROVISION, FELHOM_FORMAT, FELHOM_DNSMASQ, FELHOM_GUESTHOOK, FELHOM_INTERMEDIARY, FELHOM_CONTROLLERSWAP, FELHOM_STALELOCK, FELHOM_FSTRIM, FELHOM_NETMOUNT, FELHOM_WG, FELHOM_SELFUPDATE, FELHOM_SSHD, FELHOM_OOB, FELHOM_PBSDR, FELHOM_BACKUPTARGET, FELHOM_SELFHEAL, FELHOM_ESCROW, FELHOM_GUESTNET, FELHOM_SCRATCH_TEARDOWN, FELHOM_OSAPPLY, FELHOM_POWEROFF
+39
View File
@@ -0,0 +1,39 @@
#!/bin/sh
# /etc/grub.d/42_felhom_oneshot — the kernel lane's one-shot ENTRIES (R-836, `09` §3 decision 172, `11` §5.11).
# Installed by the config bundle (felhom-os-apply BUNDLE_FILES), 0755 root. update-grub runs it.
#
# One menu entry per installed Proxmox kernel, id `felhom-oneshot-<version>`, booted ONLY when 01_felhom_oneshot found
# the flag naming it. It is the normal entry plus option C (decision 172): softlockup_panic=1 hardlockup_panic=1
# hung_task_panic=1 panic=10 — a lockup the kernel can detect becomes a panic, and a panic restarts the box in 10 s into
# the default (the old kernel). A true dead freeze still needs a person (spike candidate 3 failed on all three boxes).
# It sorts AFTER 10_linux, so it is never entry 0 and never the default.
#
# No vfat ESP at /boot/efi → prints nothing (no flag can name these entries).
set -e
prefix="/usr"
exec_prefix="/usr"
datarootdir="/usr/share"
. "$datarootdir/grub/grub-mkconfig_lib"
esp_uuid=$(findmnt -n -o UUID,FSTYPE /boot/efi 2>/dev/null | awk '$2 == "vfat" { print $1 }')
[ -n "$esp_uuid" ] || exit 0
kernels=$(ls /boot/vmlinuz-*-pve 2>/dev/null | sed 's#^/boot/vmlinuz-##' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$' || true)
[ -n "$kernels" ] || exit 0
case "${GRUB_DEVICE}" in
/dev/mapper/*|/dev/dm-*|"") root_arg="root=${GRUB_DEVICE}" ;;
*) if [ -n "${GRUB_DEVICE_UUID}" ]; then root_arg="root=UUID=${GRUB_DEVICE_UUID}"; else root_arg="root=${GRUB_DEVICE}"; fi ;;
esac
[ -n "${GRUB_DEVICE}" ] || root_arg="root=$(findmnt -n -o SOURCE /)"
rel=$(make_system_path_relative_to_its_root /boot)
prep=$(prepare_grub_to_access_device "$(${grub_probe:-grub-probe} --target=device /boot)" | sed 's/^/ /')
for k in $kernels; do
[ -f "/boot/initrd.img-$k" ] || continue
cat <<EOF
menuentry 'Felhom one-shot: $k' --class proxmox --id felhom-oneshot-$k {
insmod gzio
$prep
echo 'Loading Linux $k (felhom one-shot) ...'
linux $rel/vmlinuz-$k $root_arg ro ${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT} softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10
initrd $rel/initrd.img-$k
}
EOF
done
+36
View File
@@ -0,0 +1,36 @@
#!/bin/sh
# /etc/grub.d/01_felhom_oneshot — the kernel lane's ONE-SHOT boot (R-836, `09` §3 decisions 164 + 172, `11` §5.11).
# Installed by the config bundle (felhom-os-apply BUNDLE_FILES), 0755 root. update-grub runs it; it prints GRUB script.
#
# At boot, GRUB reads `felhom_next` from an environment block on the ESP (vfat — GRUB can rewrite a file there; it
# cannot on the LVM /boot, R-836), CLEARS it, and — only if it names an installed kernel — boots that kernel's one-shot
# entry (42_felhom_oneshot) instead of the default. The next boot uses the default again whatever happens: a new kernel
# that panics comes back on the old one by itself. felhom-os-apply writes the flag (mode apply) and never the default
# for a new kernel. Measured on the Tester 1 VM, demo-felhom and demo-hp (Secure Boot on):
# `audits/kernel-spike-2026-10-07/` (candidate 2).
#
# No vfat ESP at /boot/efi, or no Proxmox kernel → prints nothing (the box boots exactly as before).
set -e
esp_uuid=$(findmnt -n -o UUID,FSTYPE /boot/efi 2>/dev/null | awk '$2 == "vfat" { print $1 }')
[ -n "$esp_uuid" ] || exit 0
kernels=$(ls /boot/vmlinuz-*-pve 2>/dev/null | sed 's#^/boot/vmlinuz-##' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$' || true)
[ -n "$kernels" ] || exit 0
cat <<EOF
# felhom kernel lane: a one-shot kernel named on the ESP, read and cleared before the menu
insmod part_gpt
insmod fat
search --no-floppy --fs-uuid --set=felhom_esp $esp_uuid
if [ -f (\$felhom_esp)/EFI/felhom/oneshot.env ]; then
load_env -f (\$felhom_esp)/EFI/felhom/oneshot.env felhom_next
if [ "\${felhom_next}" ]; then
set felhom_boot="\${felhom_next}"
set felhom_next=
save_env -f (\$felhom_esp)/EFI/felhom/oneshot.env felhom_next
EOF
for k in $kernels; do
printf ' if [ "${felhom_boot}" = "%s" ]; then set default="felhom-oneshot-%s"; fi\n' "$k" "$k"
done
cat <<EOF
fi
fi
EOF
+497 -12
View File
@@ -93,8 +93,13 @@ OOMCHECK_TIMEOUTS = {"image": 10, "clock": 5, "run": 30, "inspect": 10, "events"
OOMCHECK_SETTLE = 2
# Kernel, boot and firmware packages are the SLOW lane on the host whatever their origin (`11` C3, §5.2): a host
# reboot is needed for them to take effect, and a bad one can stop the box from booting.
# proxmox-secure-boot-support is the Secure Boot meta-package: its only job is to pull shim-signed and the signed GRUB,
# so it belongs with them. Measured 2026-10-09 on demo-hp (Secure Boot on): left in the pve lane, its upgrade pulled
# shim-signed-common, the whole pve step was refused R6, and the night's kernel step was skipped with it
# (`audits/kernel-night-2026-10-08/`). Pinned by test_ring0_pending_pve_leaves_the_secure_boot_meta_out.
HOST_SLOW_RE = re.compile(r"^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|"
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr)")
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr|"
r"proxmox-secure-boot-support)")
# restart_needed() leaves out processes whose cgroup line matches (grep basic regex). Host: the LXC guests' own
# processes (`0::/lxc/<vmid>/...`) -- NOT lxc-start itself, whose cgroup is `0::/lxc.monitor/<vmid>` (measured
# 2026-10-04 on demo-felhom: the old pattern "lxc" hid lxc-start with 20 deleted maps, so "reboot needed" stayed false
@@ -124,6 +129,32 @@ DAEMON_JSON = "/etc/docker/daemon.json"
# wrapper restarts exactly the containers that mount one of these paths (never the apps, never the engine).
DOCKER_SOCKETS = ("/var/run/docker.sock", "/run/docker.sock")
CRASH_GUARD_STATE = "/var/lib/felhom-crash-guard/state.json"
# ---------- the kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11) ----------
# A new kernel boots ONCE through a flag in a GRUB environment block on the ESP (vfat — GRUB can rewrite it there; on
# the LVM /boot it cannot, R-836). The bundle's two GRUB generators read and clear the flag (01_felhom_oneshot) and
# give each installed kernel a one-shot entry with the lockup-to-panic options (42_felhom_oneshot, option C). The GRUB
# default is the kernel the box RUNS, pinned in KERNEL_DEFAULT_CFG; only "kernel-good" (after a healthy one-shot boot)
# moves it to the new kernel. Measured in the spike on the Tester 1 VM, demo-felhom and demo-hp (Secure Boot on):
# `audits/kernel-spike-2026-10-07/`.
KERNEL_OP = "os_kernel_step"
KERNEL_STATE = "/var/lib/felhom-kernel/state.json" # 0644 root: the step's phase (the agent reads it)
KERNEL_DEFAULT_CFG = "/etc/default/grub.d/zz-felhom-kernel-default.cfg" # sourced last: GRUB_DEFAULT = this kernel
ONESHOT_SNIPPET = "/etc/grub.d/01_felhom_oneshot" # bundle-owned: read + clear the flag, pick the entry
ONESHOT_ENTRIES = "/etc/grub.d/42_felhom_oneshot" # bundle-owned: the one-shot entries (option C options)
GRUB_CFG = "/boot/grub/grub.cfg"
ESP_MOUNT = "/boot/efi"
ONESHOT_ENV = ESP_MOUNT + "/EFI/felhom/oneshot.env"
ONESHOT_ARGS = "softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10"
KERNEL_PIN_FILE = "/etc/kernel/proxmox-boot-pin" # an operator's `proxmox-boot-tool kernel pin` — never fought
KVER_RE = re.compile(r"^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$")
KERNEL_IMAGE_RE = re.compile(r"^proxmox-kernel-([0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve)(-signed)?$")
# the packages a kernel step may UPGRADE (never add): the kernel series meta-package, the default-kernel meta, the boot
# helper and the firmware the kernel loads. Everything else in HOST_SLOW_RE (grub, shim, microcode, efibootmgr) stays out.
KERNEL_UPGRADE_RE = re.compile(r"^(proxmox-default-kernel|proxmox-kernel-[0-9]+\.[0-9]+|proxmox-kernel-helper|pve-firmware)$")
KERNEL_MIN_GAP = 20 * 3600 # never two kernel steps in one night
KERNEL_MODES = ("kernel-status", "kernel-reboot", "kernel-boot", "kernel-good", "kernel-revert", "kernel-cancel")
# phases: staged → oneshot → judging → good | reverting → self_reverted | revert_failed; oneshot → fell_back; staged → cancelled
KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
# ---------- the config bundle (R-840, agent v0.143.0, `11` §5.4.2) ----------
# A signed `agent_config_update` job carries {agent_version, bundle_sha256}; the bundle is ONE JSON file built from this
@@ -170,6 +201,10 @@ BUNDLE_FILES = [
("/etc/systemd/system/felhom-crash-guard-check.service", "felhom-crash-guard-check.service", 0o644, "unit", "replace"),
("/etc/systemd/system/felhom-crash-guard-check.timer", "felhom-crash-guard-check.timer", 0o644, "unit", "replace"),
("/etc/felhom/crash-guard.conf", "crash-guard.conf", 0o644, "plain", "if-absent"),
# the kernel lane's two GRUB generators (R-836, `11` §5.11): they take effect at the next update-grub, which the
# first kernel step runs itself; with no flag on the ESP they change nothing about how the box boots.
("/etc/grub.d/01_felhom_oneshot", "felhom-grub-oneshot.sh", 0o755, "sh", "replace"),
("/etc/grub.d/42_felhom_oneshot", "felhom-grub-oneshot-entries.sh", 0o755, "sh", "replace"),
("/etc/systemd/system/felhom-agent.service", "felhom-agent.service", 0o644, "agent-unit", "replace"),
("/etc/systemd/system/felhom-agent-rollback.service", "felhom-agent-rollback.service", 0o644, "unit", "replace"),
("/etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf", "felhom-agent-limits.conf", 0o644, "dropin", "replace"),
@@ -434,7 +469,8 @@ class Apply:
def check_plan(self, plan):
mode = plan.get("mode", "apply")
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update", "oom-check"):
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update", "oom-check") \
+ KERNEL_MODES:
raise Refused("R11", f"unknown mode {mode!r}")
if mode == "agent_update":
if plan.get("layer") != "host":
@@ -445,14 +481,18 @@ class Apply:
raise Refused("R11", "bundle is a host-layer mode")
return mode, "host", 0, "bundle"
layer = plan.get("layer")
if layer not in ("guest", "host", "docker", "pve"):
raise Refused("R12", f"layer {layer!r} is not guest, host, docker or pve")
if layer not in ("guest", "host", "docker", "pve", "kernel"):
raise Refused("R12", f"layer {layer!r} is not guest, host, docker, pve or kernel")
if (mode in KERNEL_MODES) != (layer == "kernel" and mode not in ("apply", "health")):
raise Refused("R11", f"mode {mode!r} does not fit layer {layer!r}")
lane = plan.get("lane", "fast")
if layer == "docker" and lane != "slow":
raise Refused("R3", "the Docker engine is the slow lane (`11` §5.8); a fast-lane Docker plan is refused")
if layer == "pve" and lane != "slow":
raise Refused("R3", "the Proxmox packages are the slow lane (`11` §5.10); a fast-lane pve plan is refused")
if layer not in ("docker", "pve") and lane != "fast":
if layer == "kernel" and lane != "slow":
raise Refused("R3", "the kernel is the slow lane (`11` §5.11); a fast-lane kernel plan is refused")
if layer not in ("docker", "pve", "kernel") and lane != "fast":
raise Refused("R3", f"the {layer} layer has no slow lane in this release (kernel, Proxmox: `11` §8 step 6)")
if mode == "facts" and layer != "host":
raise Refused("R11", "facts is a host-layer mode (it reads the host and the guest)")
@@ -463,7 +503,9 @@ class Apply:
if plan.get("undo") and layer != "docker": # the pve layer has no undo in this release (R-812 option A)
raise Refused("R5", "an undo (downgrade) exists only for the Docker layer, inside a signed job")
vmid = plan.get("vmid")
if not isinstance(vmid, int) or isinstance(vmid, bool) or vmid <= 0:
if layer == "kernel" and mode in KERNEL_MODES and mode != "kernel-reboot" and vmid == 0:
pass # after a boot the guest may not run (that is what is judged); these modes never touch it
elif not isinstance(vmid, int) or isinstance(vmid, bool) or vmid <= 0:
raise Refused("R11", f"vmid must be a positive integer, got {vmid!r}")
rid = plan.get("release_id", "")
if not isinstance(rid, str) or not re.match(r"^[A-Za-z0-9._:-]{1,80}$", rid):
@@ -471,19 +513,24 @@ class Apply:
if plan.get("allow_new"):
raise Refused("R6", "allow_new is a slow-lane field; the fast lane never adds a package")
select = plan.get("select", "listed")
if select not in ("listed", "pending-fast", "pending-docker", "pending-pve"):
if select not in ("listed", "pending-fast", "pending-docker", "pending-pve", "pending-kernel"):
raise Refused("R11", f"unknown select {select!r}")
if (select == "pending-docker") != (layer == "docker" and select != "listed"):
if select == "pending-docker" or layer == "docker":
raise Refused("R11", f"select {select!r} does not fit layer {layer!r}")
if (select == "pending-pve") != (layer == "pve" and select != "listed"):
raise Refused("R11", f"select {select!r} does not fit layer {layer!r}")
if (select == "pending-kernel") != (layer == "kernel" and select != "listed"):
raise Refused("R11", f"select {select!r} does not fit layer {layer!r}")
ek = plan.get("expect_kver")
if ek is not None and (layer != "kernel" or not isinstance(ek, str) or not KVER_RE.match(ek)):
raise Refused("R11", f"expect_kver {ek!r} is not a kernel version of the kernel layer")
pk = plan.get("packages", [])
if not isinstance(pk, list):
raise Refused("R11", "packages must be a list")
if mode == "apply" and select == "listed" and not pk:
raise Refused("R11", "packages must be a non-empty list in apply mode (select listed)")
if select in ("pending-fast", "pending-docker", "pending-pve") and pk:
if select in ("pending-fast", "pending-docker", "pending-pve", "pending-kernel") and pk:
raise Refused("R11", f"select {select} takes no package list")
seen = set()
for e in pk:
@@ -503,6 +550,13 @@ class Apply:
continue
if n in DOCKER_NAMES:
raise Refused("R2", f"{n} is a Docker package — the slow lane (`11` §5.8), never in a {layer} plan")
if layer == "kernel":
if o != PVE_ORIGIN:
raise Refused("R2", f"{n}: origin {o!r} is not {PVE_ORIGIN!r} (the kernel layer)")
if not (KERNEL_UPGRADE_RE.match(n) or KERNEL_IMAGE_RE.match(n)):
raise Refused("R23", f"{n} is not a kernel package (a kernel image, the kernel meta-packages, "
f"proxmox-kernel-helper or pve-firmware)")
continue
if layer == "pve":
if o != PVE_ORIGIN:
raise Refused("R2", f"{n}: origin {o!r} is not {PVE_ORIGIN!r} (the pve layer)")
@@ -759,6 +813,17 @@ class Apply:
return v or "unknown"
h = {"debian": first(["cat", "/etc/debian_version"]), "kernel_running": first(["uname", "-r"])}
h["kernel_next_boot"], h["kernel_next_boot_source"] = self.kernel_next_boot()
try:
# the kernel lane (R-836): the default and the one-shot flag, read from grub.cfg and the ESP themselves
kv = Kernel(self, {}).view()
kv["setup_problems"] = Kernel(self, {}).setup_problems()
h["kernel_lane"] = kv
if kv.get("flag"):
h["kernel_next_boot"], h["kernel_next_boot_source"] = kv["flag"], "felhom one-shot flag (once; then the default)"
elif kv.get("default") not in (None, "unknown"):
h["kernel_next_boot"], h["kernel_next_boot_source"] = kv["default"], "grub.cfg default"
except Exception as e: # never cost the System page its other facts
h["kernel_lane"] = {"error": str(e)[:200]}
rc, out, _ = self.r.host(["apt-mark", "showhold"], 60)
h["held"] = sorted(out.split()) if rc == 0 else None
try:
@@ -817,8 +882,8 @@ class Apply:
# ---------- target helpers ----------
def x(self, argv, timeout=1800):
"""Run in the TARGET layer: the guest via pct exec, or the host directly (host and pve)."""
if self.layer in ("host", "pve"):
"""Run in the TARGET layer: the guest via pct exec, or the host directly (host, pve and kernel)."""
if self.layer in ("host", "pve", "kernel"):
return self.r.host(argv, timeout)
return self.r.guest(self.vmid, argv, timeout) # guest and docker both live in the customer guest
@@ -913,7 +978,7 @@ class Apply:
def restart_needed(self):
"""Processes still mapping deleted files, OUTSIDE containers (C11). Guest: outside docker; host: outside the
LXC guests (the host's /proc shows guest processes too)."""
skip = RESTART_SKIP_CGROUP["host" if self.layer in ("host", "pve") else "guest"]
skip = RESTART_SKIP_CGROUP["host" if self.layer in ("host", "pve", "kernel") else "guest"]
script = ('for p in /proc/[0-9]*; do grep -q "(deleted)" $p/maps 2>/dev/null || continue; '
'grep -q "%s" $p/cgroup 2>/dev/null && continue; echo "${p#/proc/} $(cat $p/comm 2>/dev/null)"; done' % skip)
rc, out, _ = self.x(["sh", "-c", script], timeout=120)
@@ -987,8 +1052,12 @@ class Apply:
return Bundle(self).from_plan(plan)
if self.mode == "agent_update":
return self.agent_update(plan)
if self.layer in ("host", "pve"):
if self.layer in ("host", "pve", "kernel"):
self.check_appliance()
if self.layer == "kernel" and self.mode != "health":
# the kernel lane's own modes (R-836): most of them run while the guest is still starting after a boot, so
# the guest check is the stage's and the reboot's own (Kernel.run), not every mode's
return Kernel(self, plan).run()
self.check_guest(self.vmid)
log = self.r.log
if self.mode == "live-restore-on":
@@ -1364,6 +1433,422 @@ class Apply:
self.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
def _iso(t):
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(t))
def _parse_iso(s):
try:
return calendar.timegm(time.strptime(s, "%Y-%m-%dT%H:%M:%SZ"))
except (TypeError, ValueError):
return None
class Kernel:
"""The kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11). Refusal codes: R3 (authority), R4 (removal),
R6 (a package outside the step), R8 (space), R9 (locks), R12 (appliance), R20 (the box's boot setup cannot do a
one-shot), R21 (the crash guard is tripped or saw an unclean boot within its window), R22 (the step's phase does not
allow this mode), R23 (the kernel set is not one exact new kernel).
Modes (plan "mode", layer "kernel", lane "slow"):
apply STAGE: install the kernel set, keep the GRUB default on the kernel the box runs, write the flag.
Never reboots. select "pending-kernel" (ring 0, the root-owned mark) or "listed" (a signed
os_kernel_step). expect_kver: the kernel the hub told the household about — any other is R23.
kernel-reboot a STAGED step's reboot (the night leg, after the household was told): phase oneshot, then reboot.
kernel-boot after a boot: what became of the step (judging | fell_back | self_reverted | revert_failed).
kernel-good the one-shot boot was healthy: the new kernel becomes the GRUB default.
kernel-revert the one-shot boot was NOT healthy: reboot ONCE into the old kernel (still the default).
kernel-cancel drop a staged step: clear the flag (the package stays installed, the default never moved).
kernel-status read only."""
def __init__(self, apply, plan):
self.a, self.r, self.plan = apply, apply.r, plan
self.report = apply.report
self.log = apply.r.log
# ---------- reading the box ----------
def running(self):
rc, out, _ = self.r.host(["uname", "-r"], 30)
v = out.strip() if rc == 0 else ""
return v if KVER_RE.match(v) else ""
def state(self):
try:
s = json.loads(self.r.read_file(KERNEL_STATE))
return s if isinstance(s, dict) else {}
except (OSError, ValueError):
return {}
def save_state(self, s):
s["updated_at"] = _iso(self.r.now())
self.r.put_file(KERNEL_STATE, (json.dumps(s, indent=2, sort_keys=True) + "\n").encode(), 0o644)
def flag(self):
"""The one-shot flag: the kernel named, "" when the env block holds none, None when there is no env block."""
rc, out, _ = self.r.host(["grub-editenv", ONESHOT_ENV, "list"], 30)
if rc != 0:
return None
for l in out.splitlines():
if l.startswith("felhom_next="):
return l.split("=", 1)[1].strip()
return ""
def grub_cfg(self):
try:
return self.r.read_file(GRUB_CFG)
except OSError:
return ""
@staticmethod
def default_kver(cfg):
"""The kernel grub.cfg boots by default (00_header's `set default=`), or "unknown"."""
m = re.search(r'^\s*set default="(?:gnulinux-advanced-[^>"]*>)?gnulinux-([0-9][^"]*?-pve)-advanced-[^"]*"', cfg, re.M)
return m.group(1) if m else "unknown"
@staticmethod
def entry_id(cfg, kver):
"""The GRUB_DEFAULT value that names kver's normal entry, read from grub.cfg itself (10_linux's ids)."""
sub = re.search(r"\$menuentry_id_option '(gnulinux-advanced-[^']+)'", cfg)
m = re.search(r"\$menuentry_id_option '(gnulinux-" + re.escape(kver) + r"-advanced-[^']+)'", cfg)
if not m:
return None
return f"{sub.group(1)}>{m.group(1)}" if sub else m.group(1)
def setup_problems(self):
"""R20: why this box cannot do a one-shot boot (empty = it can). Measured shape: UEFI, a vfat ESP at /boot/efi,
/boot on the root filesystem, GRUB with fat + loadenv, the bundle's two generators, no hand pin."""
why = []
if not self.r.lexists("/sys/firmware/efi"):
why.append("the box does not boot UEFI")
rc, out, _ = self.r.host(["findmnt", "-n", "-o", "FSTYPE", ESP_MOUNT], 30)
if rc != 0 or out.strip() != "vfat":
why.append(f"{ESP_MOUNT} is not a mounted vfat ESP ({out.strip() or 'not mounted'})")
rc, out, _ = self.r.host(["findmnt", "-n", "-o", "TARGET", "/boot"], 30)
if rc == 0 and out.strip():
why.append("/boot is a separate filesystem (the one-shot entries assume /boot on the root filesystem)")
for m in ("fat", "loadenv"):
if not self.r.lexists(f"/usr/lib/grub/x86_64-efi/{m}.mod"):
why.append(f"GRUB has no {m} module")
for p in (ONESHOT_SNIPPET, ONESHOT_ENTRIES):
if not self.r.lexists(p):
why.append(f"{p} is missing (the config bundle installs it)")
if self.r.lexists(KERNEL_PIN_FILE):
why.append("a kernel is pinned by hand (proxmox-boot-tool kernel pin) — the lane never fights it")
return why
def guard(self):
try:
return json.loads(self.r.read_file(CRASH_GUARD_STATE))
except (OSError, ValueError):
return None
def check_guard(self):
"""R21: a kernel step only on a box whose crash guard is armed and saw no unclean boot within its window — so
the step's own reboots (clean), one crash and one self-revert (clean) can never reach the 3rd unclean boot that
leaves the box off (`11` §5.9). Pinned by test_felhom_crash_guard KernelStepCannotLeaveTheBoxOff."""
g = self.guard()
if not isinstance(g, dict):
raise Refused("R21", f"no crash guard state ({CRASH_GUARD_STATE}) — a kernel step needs the guard")
if g.get("tripped") or not g.get("armed"):
raise Refused("R21", "the crash guard is tripped — no kernel step until it re-arms")
if (g.get("unclean_boots_in_window") or 0) > 0:
raise Refused("R21", f"{g.get('unclean_boots_in_window')} unclean boot(s) within the guard's window — wait")
def view(self, st=None, cfg=None):
st = self.state() if st is None else st
cfg = self.grub_cfg() if cfg is None else cfg
return {"running": self.running() or "unknown", "default": self.default_kver(cfg), "flag": self.flag(),
"phase": st.get("phase", "none"), "from": st.get("from"), "to": st.get("to"),
"step_id": st.get("step_id"), "self_revert_used": bool(st.get("self_revert_used")), "vmid": st.get("vmid"),
"staged_at": st.get("staged_at"), "rebooted_at": st.get("rebooted_at"),
"result_at": st.get("result_at"), "reason": st.get("reason")}
# ---------- writing the box ----------
def write_default(self, kver):
"""Pin the GRUB default to kver's normal entry, regenerate grub.cfg, and PROVE it (the default read back)."""
cfg = self.grub_cfg()
eid = self.entry_id(cfg, kver)
if not eid:
raise Refused("R20", f"grub.cfg has no normal entry for {kver}")
body = ("# felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily\n"
f'GRUB_DEFAULT="{eid}"\n')
self.r.put_file(KERNEL_DEFAULT_CFG, body.encode(), 0o644)
rc, out, err = self.r.host(["update-grub"], 300)
got = self.default_kver(self.grub_cfg())
if rc != 0 or got != kver:
raise Refused("R20", f"update-grub rc={rc}: the default reads {got}, not {kver}: {(out + err).strip()[-200:]}")
self.log(f"os-apply: KERNEL default = {kver} (proved from grub.cfg)")
def set_flag(self, kver):
self.r.host(["mkdir", "-p", os.path.dirname(ONESHOT_ENV)], 30)
if self.flag() is None:
rc, out, err = self.r.host(["grub-editenv", ONESHOT_ENV, "create"], 30)
if rc != 0:
raise Refused("R20", f"cannot create the one-shot env block on the ESP: {(out + err).strip()[-200:]}")
rc, out, err = self.r.host(["grub-editenv", ONESHOT_ENV, "set", f"felhom_next={kver}"], 30)
if rc != 0 or self.flag() != kver:
raise Refused("R20", f"the one-shot flag did not read back as {kver}: {(out + err).strip()[-200:]}")
def clear_flag(self):
if self.flag():
self.r.host(["grub-editenv", ONESHOT_ENV, "unset", "felhom_next"], 30)
def reboot(self, why):
self.log(f"os-apply: KERNEL REBOOT — {why}")
rc, out, err = self.r.host(["systemctl", "reboot"], 60)
self.report["reboot_rc"] = rc
if rc != 0:
self.report["failed"] = {"rc": 3, "step": "reboot", "reason": (out + err).strip()[-200:]}
return 3
return 0
# ---------- the modes ----------
def run(self):
mode = self.a.mode
self.report["kernel_mode"] = mode
if mode == "kernel-status":
v = self.view()
v["setup_problems"] = self.setup_problems()
self.report["kernel"] = v
return 0
fn = {"apply": self.stage, "kernel-reboot": self.reboot_staged, "kernel-boot": self.after_boot,
"kernel-good": self.good, "kernel-revert": self.revert, "kernel-cancel": self.cancel}[mode]
rc = fn()
self.report["kernel"] = self.view()
return rc
def phase_is(self, st, *phases):
if st.get("phase") not in phases:
raise Refused("R22", f"the kernel step is {st.get('phase', 'none')!r}, not {' or '.join(phases)} — "
f"{self.a.mode} does not apply")
def stage(self):
a = self.a
st = self.state()
if st.get("phase") in KERNEL_ACTIVE:
raise Refused("R22", f"a kernel step is already {st['phase']} ({st.get('from')} -> {st.get('to')})")
last = _parse_iso(st.get("staged_at"))
if last is not None and self.r.now() - last < KERNEL_MIN_GAP:
raise Refused("R22", "a kernel step was staged within the last 20 hours — never two in one night")
why = self.setup_problems()
if why:
raise Refused("R20", "; ".join(why))
self.check_guard()
a.check_guest(a.vmid)
who, _ = a.docker_authority(self.plan, op_name=KERNEL_OP)
self.report["authority"] = who
old = self.running()
if not old:
raise Refused("R20", "the running kernel is not a Proxmox kernel version")
self.log(f"os-apply: START release={self.plan.get('release_id')} layer=kernel lane=slow mode=apply "
f"select={a.select} authority={who} running={old}")
if a.apt_lock_held():
raise Refused("R9", "another apt/dpkg holds the lock on the host")
self.report["health_before"] = a.health()
a.repair()
rc, out, err = a.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
if rc != 0:
raise Refused("R7", f"apt-get update failed on the host: {(out + err).strip().splitlines()[-1:]}")
inst = a.installed()
if a.select == "pending-kernel":
_, pend, _, _ = a.simulate(["dist-upgrade"])
want = [(p["name"], p["to"]) for p in pend if p["from"] is not None and KERNEL_UPGRADE_RE.match(p["name"])
and a.origin_name(p["origin"]) == {PVE_ORIGIN}]
else:
want = [(e["name"], e["version"]) for e in self.plan["packages"]]
# upgrades of installed names (never a downgrade), and at most the listed new kernel image
args, upg = [], {}
for n, v in want:
if n in inst:
if a.dpkg_cmp(v, "gt", inst[n]):
upg[n] = v
args.append(f"{n}={v}")
elif KERNEL_IMAGE_RE.match(n):
args.append(f"{n}={v}")
else:
raise Refused("R6", f"{n} is not installed and is not a kernel image")
if not args:
self.report["upgraded"], self.report["outcome_hint"] = [], "nothing"
self.log("os-apply: DONE rc=0 upgraded=0 (no pending kernel)")
return 0
rc, sim, remv, text = a.simulate(["install", "--no-install-recommends"] + args)
if rc != 0:
raise Refused("R7", "the simulation failed: " + (text.strip().splitlines()[-1] if text.strip() else ""))
if remv:
raise Refused("R4", f"the kernel step would remove {', '.join(remv[:5])}")
images = []
listed = dict(want)
for p in sim:
if a.origin_name(p["origin"]) != {PVE_ORIGIN}:
raise Refused("R2", f"{p['name']} would come from {p['origin']}, not {PVE_ORIGIN!r}")
m = KERNEL_IMAGE_RE.match(p["name"])
if p["from"] is None:
if not m:
raise Refused("R6", f"the kernel step would add {p['name']}, which is not a kernel image")
if a.select == "listed" and listed.get(p["name"]) != p["to"]:
raise Refused("R23", f"the kernel step would add {p['name']}={p['to']}, not the signed set")
images.append((m.group(1), p["to"]))
continue
if p["name"] not in upg or p["to"] != upg[p["name"]]:
raise Refused("R6", f"the kernel step would touch {p['name']} ({p['to']}), which is not in the step")
if not a.dpkg_cmp(p["to"], "gt", p["from"]):
raise Refused("R5", f"{p['name']} would be downgraded {p['from']} -> {p['to']}")
if len(images) > 1:
raise Refused("R23", f"the kernel step would add {len(images)} kernels — one at a time")
# the target kernel: the new image, else the series meta-package's version (its image is already installed)
if images:
new = images[0][0]
if images[0][1] + "-pve" != new:
raise Refused("R23", f"the image version {images[0][1]} does not name the kernel {new}")
else:
metas = [(n, v) for n, v in upg.items() if re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n)]
if len(metas) != 1:
self.report["upgraded"], self.report["outcome_hint"] = [], "nothing"
self.log("os-apply: DONE rc=0 upgraded=0 (the pending set names no kernel to boot)")
return 0
new = metas[0][1] + "-pve"
if not KVER_RE.match(new) or not a.dpkg_cmp(new[:-4], "gt", old[:-4]):
raise Refused("R23", f"the kernel {new} is not newer than the running {old}")
ek = self.plan.get("expect_kver")
if ek and ek != new:
raise Refused("R23", f"the step would boot {new}, but the household was told about {ek}")
need = a.download_bytes(["install", "--no-install-recommends"] + args)
free = a.free_bytes()
if free >= 0 and free < max(MIN_FREE, 3 * need):
raise Refused("R8", f"free space {free} B is below max(500 MB, 3 x download {need} B)")
# 1. the default = the kernel the box RUNS (it booted healthily), proved from grub.cfg BEFORE the install
default_before = self.default_kver(self.grub_cfg())
self.write_default(old)
# 2. install (the kernel's own postinst runs update-grub; our default file keeps the default on `old`)
t0 = time.time()
rc, out, err = a.x(APT_ENV + ["apt-get", "-y", "-q"] + DPKG_OPTS + ["install", "--no-install-recommends"] + args)
a.x(["apt-get", "clean"])
if rc != 0:
_, aud, _ = a.x(["dpkg", "--audit"])
self.report["failed"] = {"rc": rc, "step": "install", "dpkg_audit": (aud.strip().splitlines() or ["clean"])[0],
"tail": (out + err).strip().splitlines()[-3:]}
self.log(f"os-apply: FAILED rc={rc} step=install (the default stays {old}; no flag written)")
return 3
self.report["upgraded"] = [{"name": x.split("=", 1)[0], "version": x.split("=", 1)[1]} for x in args]
self.report["seconds"] = round(time.time() - t0, 1)
# 3. prove: the image is there, the default is still `old`, the one-shot entry for `new` exists
cfg = self.grub_cfg()
if f"felhom-oneshot-{new}" not in cfg or self.default_kver(cfg) != old:
self.r.host(["update-grub"], 300)
cfg = self.grub_cfg()
for f in (f"/boot/vmlinuz-{new}", f"/boot/initrd.img-{new}"):
if not self.r.lexists(f):
self.report["failed"] = {"rc": 3, "step": "verify", "reason": f"{f} is missing after the install"}
return 3
if f"felhom-oneshot-{new}" not in cfg or "felhom_next" not in cfg:
self.report["failed"] = {"rc": 3, "step": "verify", "reason": f"grub.cfg has no one-shot entry for {new}"}
return 3
if self.default_kver(cfg) != old:
self.report["failed"] = {"rc": 3, "step": "verify",
"reason": f"the install moved the default to {self.default_kver(cfg)} — no flag written"}
return 3
# 4. the flag — the ONLY thing that makes the next boot use `new`, and only once
self.set_flag(new)
self.save_state({"phase": "staged", "step_id": self.plan.get("release_id"), "from": old, "to": new, "vmid": a.vmid,
"staged_at": _iso(self.r.now()), "authority": who, "default_before": default_before,
"packages": self.report["upgraded"], "self_revert_used": False})
self.report["reboot_needed"] = True
self.log(f"os-apply: KERNEL STAGED {old} -> {new} (default {old}, one-shot flag {new}); upgraded={len(args)} "
f"seconds={self.report['seconds']}")
return 0
def reboot_staged(self):
st = self.state()
self.phase_is(st, "staged")
if self.flag() != st.get("to"):
raise Refused("R22", f"the one-shot flag reads {self.flag()!r}, not {st.get('to')!r}")
if self.running() != st.get("from"):
raise Refused("R22", f"the box runs {self.running()!r}, not the step's old kernel {st.get('from')!r}")
cfg = self.grub_cfg()
if self.default_kver(cfg) != st["from"] or f"felhom-oneshot-{st['to']}" not in cfg:
raise Refused("R20", "grub.cfg no longer keeps the old default with a one-shot entry for the new kernel")
if self.setup_problems():
raise Refused("R20", "; ".join(self.setup_problems()))
self.check_guard()
self.a.check_guest(self.a.vmid)
st["health_before"] = self.a.health()
st["phase"], st["rebooted_at"] = "oneshot", _iso(self.r.now())
self.save_state(st)
return self.reboot(f"one-shot boot of {st['to']} (the default stays {st['from']})")
def after_boot(self):
st = self.state()
ph, run = st.get("phase"), self.running()
old, new = st.get("from"), st.get("to")
event = "none"
if ph in ("oneshot", "staged", "judging") and run == new and new:
if ph != "judging":
st["phase"], st["judging_since"], event = "judging", _iso(self.r.now()), "judging"
else:
event = "judging"
elif ph in ("oneshot", "judging") and run == old:
# the new kernel did not come up, or crashed: GRUB already booted the default (the old kernel)
self.clear_flag()
st["phase"], st["result_at"], event = "fell_back", _iso(self.r.now()), "fell_back"
st["reason"] = "the box came back on the old kernel by itself (the new one did not boot, or crashed)"
elif ph == "reverting" and run == old:
st["phase"], st["result_at"], event = "self_reverted", _iso(self.r.now()), "self_reverted"
elif ph == "reverting" and run == new:
st["phase"], st["result_at"], event = "revert_failed", _iso(self.r.now()), "revert_failed"
st["reason"] = "the self-revert came back on the NEW kernel — never retried (one self-revert per step)"
if event not in ("none",) and st.get("phase") != ph:
self.save_state(st)
self.log(f"os-apply: KERNEL AFTER-BOOT {ph} -> {st['phase']} running={run} ({old} -> {new})")
self.report["kernel_event"] = event
if event == "judging":
self.report["health_before"] = st.get("health_before") # the agent judges the boot against it
return 0
def good(self):
st = self.state()
self.phase_is(st, "judging")
if self.running() != st.get("to"):
raise Refused("R22", f"the box runs {self.running()!r}, not the new kernel {st.get('to')!r}")
try:
self.write_default(st["to"])
except Refused:
# put the old default back — the box must never be left without a proved default
self.write_default(st["from"])
raise
self.clear_flag()
st["phase"], st["result_at"] = "good", _iso(self.r.now())
self.save_state(st)
self.log(f"os-apply: KERNEL GOOD {st['to']} is the default now (was {st['from']})")
return 0
def revert(self):
st = self.state()
self.phase_is(st, "judging")
if st.get("self_revert_used"):
raise Refused("R22", "this kernel step already used its one self-revert")
if self.running() != st.get("to"):
raise Refused("R22", f"the box runs {self.running()!r}, not the new kernel {st.get('to')!r}")
self.clear_flag()
cfg = self.grub_cfg()
if self.default_kver(cfg) != st.get("from"):
raise Refused("R20", f"the GRUB default reads {self.default_kver(cfg)}, not the old {st.get('from')} — "
f"no self-revert into an unknown kernel")
reason = self.plan.get("reason")
st["reason"] = reason[:300] if isinstance(reason, str) else "the one-shot boot was not healthy"
st["self_revert_used"], st["phase"], st["reverted_at"] = True, "reverting", _iso(self.r.now())
self.save_state(st)
return self.reboot(f"self-revert to {st['from']}: {st['reason']}")
def cancel(self):
st = self.state()
self.phase_is(st, "staged")
self.clear_flag()
st["phase"], st["result_at"], st["reason"] = "cancelled", _iso(self.r.now()), "cancelled before the reboot"
self.save_state(st)
self.log(f"os-apply: KERNEL CANCELLED {st.get('to')} (installed, never the default; flag cleared)")
return 0
class Bundle:
"""The config bundle (R-840, `11` §5.4.2): every root-owned file the installer's step 5 writes, installed as ONE
signed unit. Every check runs before the first write; a failed write or a failed self-check puts every previous
+57
View File
@@ -139,5 +139,62 @@ class Guard(unittest.TestCase):
self.assertEqual(mode, 0o644)
class KernelStepCannotLeaveTheBoxOff(unittest.TestCase):
"""R-836 / `11` §5.11 Part B 4: a kernel step's planned reboot, one crash and one self-revert cannot add up to the
box staying off. The wrapper starts a step only when the guard is armed with NO unclean boot in its window
(felhom-os-apply Kernel.check_guard, R21); the planned reboot and the self-revert are orderly (`systemctl reboot` —
the clean-stop marker); so the step adds at most ONE unclean boot, and the box stays off only after the LIMIT-th
(3rd) within the hour. Red-proof: audits/kernel-lane-2026-10-07/A/redproof.txt."""
def setUp(self):
self.d = tempfile.TemporaryDirectory()
self.e = FakeEnv(self.d.name)
cg.main(["x", "boot"], self.e)
s = self.e.state()
self.assertTrue(s["armed"])
self.assertEqual(s["unclean_boots_in_window"], 0, "the wrapper's precondition (R21)")
def tearDown(self):
self.d.cleanup()
def planned(self, minutes):
cg.main(["x", "clean-stop"], self.e)
self.e.t += minutes * 60
cg.main(["x", "boot"], self.e)
def crash(self, minutes):
self.e.t += minutes * 60
cg.main(["x", "boot"], self.e)
def test_planned_reboot_one_crash_one_self_revert(self):
self.planned(2) # the step's one-shot reboot (orderly)
self.crash(3) # the new kernel crashes after the guard ran; the box restarts (panic=10)
self.planned(2) # the self-revert (orderly)
s = self.e.state()
self.assertFalse(s["tripped"], s)
self.assertTrue(s["armed"])
self.assertEqual(self.e.panic(), 10, "the box still restarts after a crash")
self.assertEqual(s["unclean_boots_in_window"], 1, "the step added exactly one unclean boot")
def test_a_panic_before_userspace_is_not_even_counted(self):
# the one-shot kernel panics before the guard's unit runs (measured: rdinit= and init= missing): the planned
# reboot's clean-stop marker is still there when the old kernel boots, so this boot counts as clean.
cg.main(["x", "clean-stop"], self.e)
self.e.t += 120 # the panicking boot: no userspace, the guard never ran
cg.main(["x", "boot"], self.e)
s = self.e.state()
self.assertEqual(s["unclean_boots_in_window"], 0, s)
self.assertEqual(self.e.panic(), 10)
def test_the_box_stays_off_only_after_two_more_crashes_than_the_step_makes(self):
self.planned(2)
self.crash(3) # the step's one crash
self.planned(2) # the self-revert
self.crash(5) # an UNRELATED crash within the hour: the guard trips (the 3rd would leave it off)
s = self.e.state()
self.assertTrue(s["tripped"])
self.assertEqual(s["unclean_boots_in_window"], 2, "two unclean boots: one from the step, one not")
if __name__ == "__main__":
unittest.main()
+529
View File
@@ -1469,6 +1469,23 @@ class PVELane(unittest.TestCase):
"pending-pve: installed, Proxmox-origin, never kernel/boot/firmware, never Debian or other origins")
self.assertEqual(f.installed["libc6"], "2.41-12+deb13u3")
# 2026-10-09, demo-hp (Secure Boot on): proxmox-secure-boot-support's upgrade pulls shim-signed-common; in the pve
# plan that refused the whole step (R6) and skipped the night's kernel step. It is a boot-chain package: left out.
# RED-PROOF: drop proxmox-secure-boot-support from HOST_SLOW_RE -> it is in the plan -> this test fails.
def test_ring0_pending_pve_leaves_the_secure_boot_meta_out(self):
f = pve_fake(ring0=True)
f.plan["select"], f.plan["packages"] = "pending-pve", []
f.installed.update({"proxmox-secure-boot-support": "9.0.1"})
f.pending_sim = [
"Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
"Inst proxmox-secure-boot-support [9.0.1] (9.0.2 Proxmox Debian Repository:stable [all])",
"Inst shim-signed-common [1.48+pmx1+16.1-1+pmx1] (1.51+pmx1+16.1-2+pmx1 Proxmox Debian Repository:stable [all])",
]
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
self.assertEqual(f.installed["proxmox-secure-boot-support"], "9.0.1")
def test_select_pending_pve_needs_the_pve_layer(self):
f = Fake()
f.plan["layer"], f.plan["select"], f.plan["packages"] = "host", "pending-pve", []
@@ -1476,5 +1493,517 @@ class PVELane(unittest.TestCase):
self.assertEqual((rc, rep["refused"]["code"]), (2, "R11"), rep)
# ---------- the kernel lane (R-836, `09` §3 decision 172, `11` §5.11) ----------
U = "1af1fcc6-639c-416b-a7e5-c4470d41a502"
OLD, NEW = "7.0.2-6-pve", "7.0.14-22-pve"
KERNEL_SET = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-22", "origin": "Proxmox Debian Repository"},
{"name": "proxmox-kernel-7.0.14-22-pve-signed", "version": "7.0.14-22", "origin": "Proxmox Debian Repository"}]
class KFake(Fake):
"""A Proxmox host with GRUB on UEFI: /boot on the root LV, a vfat ESP, the bundle's two generators. update-grub is
emulated like the real 10_linux: WITHOUT the felhom default file the NEWEST kernel becomes the default (measured,
R-836 — that is the defect the lane exists for); with it, the kernel it names."""
def __init__(self):
super().__init__()
self.running = OLD
self.boot = {OLD}
self.efi, self.esp_fs, self.boot_mount, self.mods, self.snippets, self.pin = True, "vfat", "", True, True, False
self.env = None # None = no env block on the ESP; else {"felhom_next": ...}
self.tree = {} # files put_file wrote
self.reboots = []
self.update_grubs = 0
self.grub_runs_in_postinst = True
self.installed.update({"proxmox-kernel-7.0": "7.0.2-6", "proxmox-default-kernel": "2.1.0",
"pve-firmware": "3.18-3", "proxmox-kernel-7.0.2-6-pve-signed": "7.0.2-6",
"pve-manager": "9.2.21"})
self.live.update({"proxmox-kernel-7.0": {"7.0.14-22", "7.0.2-6"},
"proxmox-kernel-7.0.14-22-pve-signed": {"7.0.14-22"},
"proxmox-kernel-7.0.14-23-pve-signed": {"7.0.14-23"},
"pve-firmware": {"3.18-7", "3.18-3"}})
self.origins = {}
self.kernel_pending = ["Inst pve-firmware [3.18-3] (3.18-7 Proxmox Debian Repository:stable [all])",
"Inst proxmox-kernel-7.0.14-22-pve-signed (7.0.14-22 Proxmox Debian Repository:stable [amd64])",
"Inst proxmox-kernel-7.0 [7.0.2-6] (7.0.14-22 Proxmox Debian Repository:stable [amd64])",
"Inst pve-manager [9.2.21] (9.2.22 Proxmox Debian Repository:stable [amd64])",
"Inst libc6 [2.41-12+deb13u3] (2.41-12+deb13u4 Debian:13.7/stable [amd64])"]
self.plan = {"release_id": "kernel-t1", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": "apply",
"select": "pending-kernel", "packages": []}
self.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": True})
self.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": True, "tripped": False, "unclean_boots_in_window": 0})
self.cfg = self.render()
# -- GRUB --
def newest(self):
best = None
for k in self.boot:
if best is None or dpkg_cmp(k[:-4], "gt", best[:-4]):
best = k
return best
def render(self):
d = self.tree.get(osapply.KERNEL_DEFAULT_CFG)
if d:
dflt = re.search(r'GRUB_DEFAULT="([^"]+)"', d).group(1)
else:
dflt = f"gnulinux-advanced-{U}>gnulinux-{self.newest()}-advanced-{U}"
cfg = ('if [ "${next_entry}" ] ; then\n set default="${next_entry}"\nelse\n'
f' set default="{dflt}"\nfi\n'
f"submenu 'Advanced options' $menuentry_id_option 'gnulinux-advanced-{U}' {{\n")
for k in sorted(self.boot):
cfg += f" menuentry 'Proxmox VE, with Linux {k}' $menuentry_id_option 'gnulinux-{k}-advanced-{U}' {{ }}\n"
cfg += "}\n"
if self.snippets:
cfg += "### BEGIN /etc/grub.d/01_felhom_oneshot ###\nload_env felhom_next\n"
cfg += "".join(f"menuentry 'Felhom one-shot: {k}' --id felhom-oneshot-{k} {{ }}\n" for k in sorted(self.boot))
return cfg
def lexists(self, p):
if p == "/sys/firmware/efi":
return self.efi
if p.startswith("/usr/lib/grub/x86_64-efi/"):
return self.mods
if p in (osapply.ONESHOT_SNIPPET, osapply.ONESHOT_ENTRIES):
return self.snippets
if p == osapply.KERNEL_PIN_FILE:
return self.pin
m = re.match(r"^/boot/(vmlinuz|initrd\.img)-(.+)$", p)
if m:
return m.group(2) in self.boot
return p in self.tree
def put_file(self, path, data, mode):
self.tree[path] = data.decode()
def read_file(self, p):
if p == osapply.GRUB_CFG:
return self.cfg
if p in self.tree:
return self.tree[p]
return super().read_file(p)
def host(self, argv, timeout=600, stdin=None):
if argv[0] == "uname":
self.calls.append(("host", argv))
return 0, self.running + "\n", ""
if argv[0] == "findmnt":
self.calls.append(("host", argv))
if argv[-1] == osapply.ESP_MOUNT:
return (0, self.esp_fs + "\n", "") if self.esp_fs else (1, "", "")
return (0, self.boot_mount + "\n", "") if self.boot_mount else (1, "", "")
if argv[0] == "grub-editenv":
self.calls.append(("host", argv))
if argv[2] == "list":
return (1, "", "no such file") if self.env is None else \
(0, "".join(f"{k}={v}\n" for k, v in self.env.items()), "")
if argv[2] == "create":
self.env = {}
return 0, "", ""
if argv[2] == "set":
k, v = argv[3].split("=", 1)
self.env[k] = v
return 0, "", ""
if argv[2] == "unset":
self.env.pop(argv[3], None)
return 0, "", ""
if argv[0] == "update-grub":
self.calls.append(("host", argv))
self.update_grubs += 1
self.cfg = self.render()
return 0, "", ""
if argv[0] == "mkdir":
self.calls.append(("host", argv))
return 0, "", ""
if argv[:2] == ["systemctl", "reboot"]:
self.calls.append(("host", argv))
self.reboots.append(self.running)
return 0, "", ""
return super().host(argv, timeout, stdin)
def emulate(self, argv):
a = [x for x in argv if not re.match(r"^[A-Z_]+=", x) and x != "env"]
if a[0] == "apt-get" and "install" in a and "-s" not in a and "--print-uris" not in a and "-f" not in a:
if self.install_rc:
return self.install_rc, "", "E: boom"
for x in a:
if "=" in x and not x.startswith("-") and "::" not in x:
n, v = x.split("=", 1)
self.installed[n] = v
m = re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n)
if m:
self.installed[f"proxmox-kernel-{v}-pve-signed"] = v
if m or osapply.KERNEL_IMAGE_RE.match(n):
self.boot.add(v + "-pve")
if self.grub_runs_in_postinst:
self.cfg = self.render() # the kernel's postinst runs update-grub (zz-update-grub)
return 0, "Setting up proxmox-kernel ...\n", ""
return super().emulate(argv)
def sim(self, a):
if "--print-uris" in a:
return 0, "", ""
if "dist-upgrade" in a:
return 0, "\n".join(self.kernel_pending) + "\n", ""
out, named = "", set()
for x in a:
if "=" in x and not x.startswith("-") and "::" not in x:
named.add(x.split("=", 1)[0])
for x in a:
if "=" in x and not x.startswith("-") and "::" not in x:
n, v = x.split("=", 1)
if v not in self.avail(n):
return 100, "", f"E: Version '{v}' for '{n}' was not found"
o = self.origins.get(n, PVE)
out += f"Inst {n} [{self.installed[n]}] ({v} {o} [amd64])\n" if n in self.installed else f"Inst {n} ({v} {o} [amd64])\n"
if re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n):
img = f"proxmox-kernel-{v}-pve-signed"
if img not in self.installed and img not in named:
out += f"Inst {img} ({v} {PVE} [amd64])\n"
out += "".join(l + "\n" for l in self.extra_sim)
return 0, out, ""
def kfake(**kw):
f = KFake()
for k, v in kw.items():
setattr(f, k, v)
return f
def kmode(f, mode, **extra):
f.plan = {"release_id": "kernel-t1", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": mode, "packages": []}
f.plan.update(extra)
return run(f)
def staged(f=None):
f = f or kfake()
rc, rep = run(f)
assert rc == 0, rep
return f
class KernelLane(unittest.TestCase):
"""R-836 / `09` §3 decision 172: stage a kernel once through the ESP flag; the default moves only after a healthy
one-shot boot. Red-proof: audits/kernel-lane-2026-10-07/A/redproof.txt."""
def refused(self, f, code, mode=None, **extra):
rc, rep = kmode(f, mode, **extra) if mode else run(f)
self.assertEqual(rc, 2, rep)
self.assertEqual(rep["refused"]["code"], code, rep)
return rep
# --- the four red tests the brief names (Part A 3) ---
def test_installing_a_kernel_does_not_change_the_grub_default(self):
f = kfake()
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertIn(NEW, f.boot, "the new kernel was installed")
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD,
"the default must stay the kernel the box runs (R-836: an install made the new one default)")
self.assertIn(f"gnulinux-{OLD}-advanced-{U}", f.tree[osapply.KERNEL_DEFAULT_CFG])
self.assertEqual(f.env, {"felhom_next": NEW}, "the ONLY way to the new kernel is the one-shot flag")
self.assertEqual(f.reboots, [], "staging never reboots")
st = json.loads(f.tree[osapply.KERNEL_STATE])
self.assertEqual((st["phase"], st["from"], st["to"]), ("staged", OLD, NEW))
self.assertEqual(rep["kernel"]["default"], OLD)
def test_a_box_without_the_esp_flag_is_refused(self):
for attr, val, frag in (("esp_fs", "ext4", "vfat"), ("efi", False, "UEFI"), ("snippets", False, "bundle"),
("mods", False, "module"), ("boot_mount", "/boot", "separate"), ("pin", True, "pinned")):
f = kfake(**{attr: val})
rep = self.refused(f, "R20")
self.assertIn(frag, rep["refused"]["reason"], attr)
self.assertNotIn(NEW, f.boot, f"{attr}: nothing may be installed on a refusal")
self.assertIsNone(f.env, f"{attr}: no flag on a refusal")
def test_a_reboot_without_the_flag_is_refused(self):
f = staged()
f.env = {"felhom_next": ""}
self.refused(f, "R22", mode="kernel-reboot")
self.assertEqual(f.reboots, [])
def test_a_kernel_outside_the_approved_set_is_refused(self):
# a signed set that names only the meta-package: the image the sources pull in was never signed for
f = kfake()
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
f.plan.update(select="listed", packages=[dict(KERNEL_SET[0])],
signed=signed_job(packages=[KERNEL_SET[0]], op="os_kernel_step"))
self.refused(f, "R23")
self.assertNotIn(NEW, f.boot)
# a signed set names 7.0.14-22; the sources would ALSO bring 7.0.14-23
f2 = kfake()
f2.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
f2.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
signed=signed_job(packages=KERNEL_SET, op="os_kernel_step"))
f2.extra_sim = ["Inst proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23 Proxmox Debian Repository:stable [amd64])"]
self.refused(f2, "R23")
# a name that is not a kernel package at all
g = kfake()
g.plan.update(select="listed", packages=[{"name": "pve-manager", "version": "9.2.22", "origin": "Proxmox Debian Repository"}])
self.refused(g, "R23")
# the household was told about another kernel
h = kfake()
h.plan["expect_kver"] = "7.0.14-23-pve"
self.refused(h, "R23")
self.assertNotIn(NEW, h.boot)
def test_the_snippet_clears_the_flag_on_use(self):
"""01_felhom_oneshot: the flag is copied, CLEARED and SAVED before any `set default`, all inside the one branch
that runs only when the flag is set; a default is set only for an installed kernel's own entry."""
text = (HERE / "felhom-grub-oneshot.sh").read_text()
body = text[text.index("cat <<EOF"):]
i_copy = body.index('set felhom_boot="\\${felhom_next}"')
i_clear = body.index("set felhom_next=\n")
i_save = body.index("save_env -f (\\$felhom_esp)/EFI/felhom/oneshot.env felhom_next")
i_default = body.index('set default="felhom-oneshot-%s"')
self.assertLess(i_copy, i_clear)
self.assertLess(i_clear, i_save)
self.assertLess(i_save, i_default, "the flag must be cleared on disk BEFORE GRUB boots anything")
self.assertIn('if [ "${felhom_boot}" = "%s" ]', body, "a default only for a kernel that is installed")
self.assertIn(osapply.ONESHOT_ENV[len(osapply.ESP_MOUNT):], text, "the wrapper and GRUB name the same env file")
# --- the rest of the stage ---
def test_option_c_options_are_on_the_one_shot_entry_only(self):
text = (HERE / "felhom-grub-oneshot-entries.sh").read_text()
self.assertIn(osapply.ONESHOT_ARGS, text)
self.assertIn("--id felhom-oneshot-$k", text)
self.assertNotIn("set default", text, "the entries file never sets the default")
def test_both_generators_ride_the_bundle(self):
self.assertEqual(osapply.BUNDLE_DESTS[osapply.ONESHOT_SNIPPET][1:4], ("felhom-grub-oneshot.sh", 0o755, "sh"))
self.assertEqual(osapply.BUNDLE_DESTS[osapply.ONESHOT_ENTRIES][1:4], ("felhom-grub-oneshot-entries.sh", 0o755, "sh"))
def test_ring0_pending_kernel_takes_only_the_kernel_set(self):
f = kfake()
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["authority"], "ring0")
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["proxmox-kernel-7.0", "pve-firmware"])
self.assertEqual(f.installed["pve-manager"], "9.2.21", "a Proxmox userspace package is the pve lane's")
self.assertEqual(f.installed["libc6"], "2.41-12+deb13u3", "a Debian package is the fast lane's")
def test_signed_listed_set_is_installed(self):
f = kfake()
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
f.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
signed=signed_job(packages=KERNEL_SET, op="os_kernel_step"))
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["authority"], "signed")
self.assertEqual(f.env, {"felhom_next": NEW})
def test_no_authority_and_the_wrong_op_are_refused(self):
f = kfake()
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
self.refused(f, "R3")
g = kfake()
g.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
g.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
signed=signed_job(packages=KERNEL_SET, op="os_pve_step"))
self.refused(g, "R3")
def test_fast_lane_byo_and_wrong_select_are_refused(self):
f = kfake()
f.plan["lane"] = "fast"
self.refused(f, "R3")
g = kfake()
g.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
self.refused(g, "R12")
h = kfake()
h.plan["select"] = "pending-pve"
self.refused(h, "R11")
k = kfake()
k.plan.update(layer="host", lane="fast", mode="kernel-status")
self.refused(k, "R11")
def test_the_crash_guard_must_be_armed_and_quiet(self):
f = kfake()
f.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": False, "tripped": True, "unclean_boots_in_window": 2})
self.refused(f, "R21")
g = kfake()
g.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": True, "tripped": False, "unclean_boots_in_window": 1})
self.refused(g, "R21")
h = kfake()
del h.files[osapply.CRASH_GUARD_STATE]
self.refused(h, "R21")
self.assertNotIn(NEW, h.boot)
def test_never_two_steps_in_one_night(self):
f = staged()
self.refused(f, "R22") # still staged
g = staged()
st = json.loads(g.tree[osapply.KERNEL_STATE])
st["phase"] = "good"
g.tree[osapply.KERNEL_STATE] = json.dumps(st)
g.clock += 3600
self.refused(g, "R22") # done, but within 20 h
def test_removal_new_non_kernel_two_kernels_and_older_are_refused(self):
f = kfake(extra_sim=["Remv pve-firmware [3.18-3]"])
self.refused(f, "R4")
g = kfake(extra_sim=["Inst proxmox-new-thing (1.0 Proxmox Debian Repository:stable [all])"])
self.refused(g, "R6")
h = kfake(extra_sim=["Inst proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23 Proxmox Debian Repository:stable [amd64])"])
self.refused(h, "R23")
k = kfake(running="7.0.14-23-pve")
k.boot = {"7.0.14-23-pve"}
k.cfg = k.render()
self.refused(k, "R23")
m = kfake()
m.origins = {"proxmox-kernel-7.0": DEB}
self.refused(m, "R2")
# R-898: ring 0 stages EXACTLY the told kernel (select listed, the set derived from it) — even when the sources
# offer a newer one by night; a told version that can no longer be installed is refused BEFORE any change (R7).
def test_ring0_listed_installs_the_told_kernel_not_the_newest(self):
f = kfake()
f.live["proxmox-kernel-7.0"] = {"7.0.14-20", "7.0.14-22", "7.0.2-6"}
f.live["proxmox-kernel-7.0.14-20-pve-signed"] = {"7.0.14-20"}
told = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"},
{"name": "proxmox-kernel-7.0.14-20-pve-signed", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"}]
f.plan.update(select="listed", packages=told, expect_kver="7.0.14-20-pve")
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["authority"], "ring0")
self.assertEqual(f.env, {"felhom_next": "7.0.14-20-pve"})
self.assertEqual(f.installed["proxmox-kernel-7.0"], "7.0.14-20")
self.assertNotIn("7.0.14-22-pve", f.boot)
def test_ring0_told_kernel_gone_is_refused_before_any_change(self):
f = kfake()
f.live["proxmox-kernel-7.0"] = {"7.0.14-22"} # 7.0.14-20 is no longer in the archive
told = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"},
{"name": "proxmox-kernel-7.0.14-20-pve-signed", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"}]
f.plan.update(select="listed", packages=told, expect_kver="7.0.14-20-pve")
rep = self.refused(f, "R7")
self.assertIsNone(f.env)
self.assertNotIn(osapply.KERNEL_DEFAULT_CFG, f.tree, "refused before the default was even pinned")
self.assertEqual(f.installed["proxmox-kernel-7.0"], "7.0.2-6")
def test_nothing_pending_changes_nothing(self):
f = kfake(kernel_pending=[])
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["upgraded"], [])
self.assertIsNone(f.env)
self.assertNotIn(osapply.KERNEL_STATE, f.tree)
def test_a_failed_install_writes_no_flag(self):
f = kfake(install_rc=100)
rc, rep = run(f)
self.assertEqual(rc, 3, rep)
self.assertIsNone(f.env)
self.assertNotIn(osapply.KERNEL_STATE, f.tree)
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
def test_a_postinst_without_update_grub_is_regenerated_and_proved(self):
f = kfake(grub_runs_in_postinst=False)
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertIn(f"felhom-oneshot-{NEW}", f.cfg)
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
# --- the reboot, the boot, good / revert ---
def test_reboot_of_a_staged_step(self):
f = staged()
rc, rep = kmode(f, "kernel-reboot")
self.assertEqual(rc, 0, rep)
self.assertEqual(f.reboots, [OLD])
st = json.loads(f.tree[osapply.KERNEL_STATE])
self.assertEqual(st["phase"], "oneshot")
self.assertIn("host_services", st["health_before"])
def test_reboot_needs_a_staged_step(self):
self.refused(kfake(), "R22", mode="kernel-reboot")
def boot_into(self, f, kver):
f.running = kver
if f.env and f.env.get("felhom_next"):
f.env["felhom_next"] = "" # GRUB cleared it (01_felhom_oneshot)
return kmode(f, "kernel-boot")
def test_healthy_one_shot_becomes_the_default(self):
f = staged()
kmode(f, "kernel-reboot")
rc, rep = self.boot_into(f, NEW)
self.assertEqual((rc, rep["kernel_event"], rep["kernel"]["phase"]), (0, "judging", "judging"), rep)
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD, "judging does not move the default")
rc, rep = kmode(f, "kernel-good")
self.assertEqual(rc, 0, rep)
self.assertEqual(osapply.Kernel.default_kver(f.cfg), NEW)
self.assertEqual(rep["kernel"]["phase"], "good")
def test_a_panic_falls_back_and_is_recorded(self):
f = staged()
kmode(f, "kernel-reboot")
rc, rep = self.boot_into(f, OLD)
self.assertEqual((rc, rep["kernel_event"]), (0, "fell_back"), rep)
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
self.refused(f, "R22", mode="kernel-good")
def test_one_self_revert_then_never_again(self):
f = staged()
kmode(f, "kernel-reboot")
self.boot_into(f, NEW)
rc, rep = kmode(f, "kernel-revert", reason="the customer guest is not running")
self.assertEqual(rc, 0, rep)
self.assertEqual(f.reboots, [OLD, NEW])
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD, "the self-revert boots the default, the old kernel")
rc, rep = self.boot_into(f, OLD)
self.assertEqual(rep["kernel_event"], "self_reverted")
# the same step can never revert again
st = json.loads(f.tree[osapply.KERNEL_STATE])
st["phase"] = "judging"
f.tree[osapply.KERNEL_STATE] = json.dumps(st)
f.running = NEW
self.refused(f, "R22", mode="kernel-revert")
self.assertEqual(len(f.reboots), 2)
def test_a_revert_that_comes_back_new_is_never_retried(self):
f = staged()
kmode(f, "kernel-reboot")
self.boot_into(f, NEW)
kmode(f, "kernel-revert")
rc, rep = self.boot_into(f, NEW)
self.assertEqual(rep["kernel_event"], "revert_failed")
self.refused(f, "R22", mode="kernel-revert")
def test_revert_refuses_an_unknown_default(self):
f = staged()
kmode(f, "kernel-reboot")
self.boot_into(f, NEW)
f.tree[osapply.KERNEL_DEFAULT_CFG] = f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-9.9.9-1-pve-advanced-{U}"\n'
f.cfg = f.render()
self.refused(f, "R20", mode="kernel-revert")
self.assertEqual(len(f.reboots), 1)
def test_cancel_clears_the_flag(self):
f = staged()
rc, rep = kmode(f, "kernel-cancel")
self.assertEqual(rc, 0, rep)
self.assertFalse(f.env.get("felhom_next"), "the flag is gone")
self.assertEqual(rep["kernel"]["phase"], "cancelled")
def test_status_is_read_only_and_names_setup_problems(self):
f = kfake(esp_fs="ext4")
rc, rep = kmode(f, "kernel-status")
self.assertEqual(rc, 0, rep)
self.assertTrue(rep["kernel"]["setup_problems"])
self.assertEqual(f.tree, {})
self.assertFalse([c for c in f.calls if c[1][0] in ("update-grub", "systemctl", "apt-get")])
def test_facts_carry_the_kernel_lane(self):
f = staged()
f.plan = {"release_id": "facts", "layer": "host", "lane": "fast", "vmid": 9201, "mode": "facts", "packages": []}
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
h = rep["facts"]["host"]
self.assertEqual(h["kernel_lane"]["phase"], "staged")
self.assertEqual(h["kernel_next_boot"], NEW)
self.assertIn("one-shot", h["kernel_next_boot_source"])
if __name__ == "__main__":
unittest.main()
+23
View File
@@ -103,6 +103,29 @@ func (s *BackupSuccessState) LastKnownSuccess(target string, vmid int) (time.Tim
return e.at, ok
}
// KnownBackupSuccesses returns every saved success as a host-report record (the collector's
// KnownBackupReporter, so the hub keeps its evidence across a restart). Only the tier, guest and start
// time are known; the archive name is not saved and stays empty. Sorted for a stable report.
func (s *BackupSuccessState) KnownBackupSuccesses() []hub.Backup {
if s == nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
out := make([]hub.Backup, 0, len(s.last))
for _, e := range s.last {
out = append(out, hub.Backup{TargetID: e.target, VMID: e.vmid, Success: true, CrashConsistent: true,
StartedAt: e.at.Format(time.RFC3339), UncoveredVolumes: []string{}})
}
sort.Slice(out, func(i, j int) bool {
if out[i].TargetID != out[j].TargetID {
return out[i].TargetID < out[j].TargetID
}
return out[i].VMID < out[j].VMID
})
return out
}
func (s *BackupSuccessState) saveLocked() error {
entries := make([]backupSuccessJSON, 0, len(s.last))
for _, e := range s.last {
+17
View File
@@ -57,3 +57,20 @@ func TestBackupSuccessState_CorruptFileIsEmpty(t *testing.T) {
t.Fatal("a corrupt file must read as nothing known")
}
}
// The saved successes come back as host-report records: success, tier, guest, start time (R-894 → the
// host report, 2026-10-09). A new state read from the same file gives the same records (the restart case).
func TestBackupSuccessState_KnownBackupSuccessesSurviveAReopen(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup-success-state.json")
s := NewBackupSuccessState(path)
if err := s.RecordBackupSuccess("felhom-backup", hub.Backup{VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z"}); err != nil {
t.Fatal(err)
}
got := NewBackupSuccessState(path).KnownBackupSuccesses()
if len(got) != 1 || !got[0].Success || got[0].TargetID != "felhom-backup" || got[0].VMID != 9201 || got[0].StartedAt != "2026-10-09T02:40:16Z" {
t.Fatalf("got %+v", got)
}
if got[0].UncoveredVolumes == nil {
t.Fatal("uncovered_volumes must marshal as [], not null")
}
}
+2
View File
@@ -115,6 +115,8 @@ var manifest = []Capability{
{"guesthook-register", "pre-start hook register", "/usr/sbin/pct", []string{"set", "9201", "--hookscript", "local:snippets/felhom-guest-hook.sh"}, false, ""},
{"guesthook-delete-mp", "dead mountpoint slot delete (C1 net)", "/usr/sbin/pct", []string{"set", "9201", "--delete", "mp0"}, false, ""},
{"guest-reboot", "enroll activate-binds reboot", "/usr/sbin/pct", []string{"reboot", "9201"}, false, ""},
{"guest-shutdown", "household power-off: guest shutdown (R-998)", "/usr/sbin/pct", []string{"shutdown", "9201", "--timeout", "120"}, false, ""},
{"host-poweroff", "household power-off: host power-off (R-998)", "/usr/bin/systemctl", []string{"poweroff"}, false, ""},
// ---- LAN split-horizon resolver (dnsmasq) ----
{"dnsmasq-install", "dnsmasq package install", "/usr/bin/apt-get", []string{"install", "-y", "-q", "dnsmasq"}, false, ""},
+36
View File
@@ -311,3 +311,39 @@ func TestWGCapabilityCriticality(t *testing.T) {
}
}
}
// TestEveryAliasIsGranted closes a gap found while adding FELHOM_POWEROFF (R-998): the coverage check above reads
// every Cmnd_Alias body whether or not the alias appears on the grant line, so a manifest command whose alias was
// never GRANTED still passed — and failed with „a password is required" on the box. Every defined alias must be
// named on the `felhom-agent ALL=(root) NOPASSWD:` line. Red-proof: drop FELHOM_POWEROFF from that line.
func TestEveryAliasIsGranted(t *testing.T) {
data, err := os.ReadFile(sudoersPath)
if err != nil {
t.Fatalf("read sudoers: %v", err)
}
var defined []string
granted := map[string]bool{}
grantLines := 0
for _, ln := range strings.Split(string(data), "\n") {
tr := strings.TrimSpace(ln)
if strings.HasPrefix(tr, "Cmnd_Alias ") {
if f := strings.Fields(tr); len(f) >= 2 {
defined = append(defined, f[1])
}
}
if strings.HasPrefix(tr, "felhom-agent ") && strings.Contains(tr, "NOPASSWD:") {
grantLines++
for _, a := range strings.Split(tr[strings.Index(tr, "NOPASSWD:")+len("NOPASSWD:"):], ",") {
granted[strings.TrimSpace(a)] = true
}
}
}
if grantLines == 0 || len(defined) < 10 {
t.Fatalf("parsed %d grant line(s), %d aliases — parser broke", grantLines, len(defined))
}
for _, a := range defined {
if !granted[a] {
t.Errorf("Cmnd_Alias %s is defined but never granted to felhom-agent", a)
}
}
}
@@ -53,6 +53,16 @@ var r861Injections = []string{
"/usr/sbin/pct exec 9201 -- tee /etc/felhom-controller-image",
"/usr/local/sbin/felhom-priv-apply controller-image 9201 9202",
"/usr/local/sbin/felhom-priv-apply controller-image 9201;id",
// R-998 (host power-off): the two new grants take exactly their fixed shape — no extra option, no other target
"/usr/sbin/pct shutdown 9201 --timeout 120 --forceStop 1",
"/usr/sbin/pct shutdown 9201 --forceStop 1 --timeout 120",
"/usr/sbin/pct shutdown 9201",
"/usr/sbin/pct shutdown 9201 --timeout -1",
"/usr/bin/systemctl poweroff --force",
"/usr/bin/systemctl poweroff -ff",
"/usr/bin/systemctl reboot",
"/usr/bin/systemctl halt",
"/usr/bin/systemctl kexec",
}
func TestSudoersRefusesTheR861Injections(t *testing.T) {
+50 -10
View File
@@ -59,8 +59,26 @@ var (
// It carries no material and no code: only WHICH earlier package opened, by its supersession date,
// which is the one fact the customer needs to recognise it.
ErrCodeOpensRetained = errors.New("escrow: the recovery code did not open the CURRENT sealed package, but it DID open a retained earlier one")
// ErrRetainedUnchecked — the code did NOT open the current package, and NOT every earlier package the hub
// holds for this host was tried (R-304, 2026-10-08): the retained list could not be fetched, the hub
// withheld rows (over its serve cap, or rows with no key material), a package was malformed, or the
// attempt cap stopped the loop. So „the code is wrong" is NOT known — it may be right for a package
// nobody tried. Distinct from a mistype for exactly the R-224 reason: never accuse the customer of
// something we did not check.
ErrRetainedUnchecked = errors.New("escrow: the recovery code did not open the current sealed package, and some earlier packages were NOT checked")
)
// RetainedUncheckedError wraps ErrRetainedUnchecked with how many earlier packages went unchecked
// (-1 = unknown: the retained list itself could not be read). No secret.
type RetainedUncheckedError struct {
Unchecked int
}
func (e *RetainedUncheckedError) Error() string {
return fmt.Sprintf("%s (unchecked=%d)", ErrRetainedUnchecked.Error(), e.Unchecked)
}
func (e *RetainedUncheckedError) Unwrap() error { return ErrRetainedUnchecked }
// RetainedMatch says which retained package a code opened. Returned inside RetainedOpenedError; it
// carries no secret — not the code, not the bundle, not the repository password.
type RetainedMatch struct {
@@ -102,8 +120,10 @@ type RetainedBlob struct {
}
// RetainedFetcher yields this host's RETAINED sealed packages, newest-superseded first. An empty
// slice is a clean "none". R-311.
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, unopenable int, err error)
// slice is a clean "none". R-311. `withheld` counts the earlier packages the hub holds that are NOT in
// blobs — rows with no key material, rows over the hub's serve cap, and packages dropped as malformed
// (R-304): each is a package the code was never tried against.
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, withheld int, err error)
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
type OffsiteKeyRecoverer struct {
@@ -154,9 +174,14 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
// from the unwrap alone; the only way to tell is to try. Until this existed nobody tried, and
// the screen said so out loud ("innen nem tudjuk megkülönböztetni őket") — a true sentence
// about our own incuriosity, read by the customer as a statement about their code.
if m, ok := r.tryRetained(ctx, recoveryCode); ok {
m, ok, unchecked := r.tryRetained(ctx, recoveryCode)
if ok {
return "", &RetainedOpenedError{Match: m}
}
// R-304: only when EVERY earlier package the hub holds was tried may this stay a wrong code.
if unchecked != 0 {
return "", &RetainedUncheckedError{Unchecked: unchecked}
}
return "", err // the fail-closed "the recovery code did not unwrap…" message; no secret in it
}
if bundle.ResticRepoPassword == "" {
@@ -173,23 +198,38 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
// function breaking is the behaviour we had before it existed.
//
// NOTHING IS LOGGED HERE and no return value carries the code, a bundle or a password.
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool) {
//
// R-304 (2026-10-08): it also returns how many earlier packages were NOT tried — `withheld` from the hub, plus
// the ones past the attempt cap — or -1 when the retained list could not be read at all. A nil FetchRetained
// (an agent wired without the lookup) reports 0: the pre-R-311 refusal, unchanged.
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool, int) {
if r.FetchRetained == nil {
return RetainedMatch{}, false
return RetainedMatch{}, false, 0
}
blobs, _, err := r.FetchRetained(ctx)
if err != nil || len(blobs) == 0 {
return RetainedMatch{}, false
blobs, withheld, err := r.FetchRetained(ctx)
if err != nil {
return RetainedMatch{}, false, -1
}
if withheld < 0 {
withheld = 0
}
if len(blobs) == 0 {
return RetainedMatch{}, false, withheld
}
limit := r.MaxRetainedTried
if limit <= 0 {
limit = defaultMaxRetainedTried
}
unchecked := withheld
if len(blobs) > limit {
unchecked += len(blobs) - limit
}
for i, rb := range blobs {
if i >= limit {
break
}
if len(rb.Blob) == 0 {
unchecked++
continue
}
bundle, uerr := UnwrapIdentityBundle(ctx, rb.Blob, recoveryCode)
@@ -204,7 +244,7 @@ func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode strin
// correct and must be told so — but the history behind it still cannot be reopened, and
// saying otherwise would be a promise this path cannot keep.
HasResticPassword: bundle.ResticRepoPassword != "",
}, true
}, true, 0
}
return RetainedMatch{}, false
return RetainedMatch{}, false, unchecked
}
+76 -2
View File
@@ -115,8 +115,9 @@ func TestRecover_WrongCode_StaysAPlainRefusal(t *testing.T) {
}
}
// FAIL-SAFE — if the retained lookup itself fails, the original refusal must stand UNCHANGED. The
// worst outcome of this feature breaking is the behaviour we had before it.
// FAIL-SAFE — if the retained lookup itself fails, the lookup's error never reaches the customer and no
// retained package is claimed. R-304 (2026-10-08) changed what stands instead: not the wrong-code refusal (the
// earlier packages were never tried, so „wrong" is not known) but ErrRetainedUnchecked with Unchecked = -1.
//
// RED-PROOF: make tryRetained propagate the fetch error instead of returning false → the customer
// gets a new, unexplained failure mode → this FAILS.
@@ -140,6 +141,10 @@ func TestRecover_RetainedFetchFails_OriginalRefusalStands(t *testing.T) {
if containsStr(err.Error(), "hub exploded") {
t.Error("the retained-lookup failure leaked into the customer-facing refusal — it must be silent")
}
var ue *RetainedUncheckedError
if !errors.As(err, &ue) || ue.Unchecked != -1 {
t.Errorf("err = %v, want RetainedUncheckedError{-1} — the earlier packages were never tried (R-304)", err)
}
}
// A nil FetchRetained keeps the pre-R-311 behaviour EXACTLY. An agent wired without it must be
@@ -228,3 +233,72 @@ func containsStr(hay, needle string) bool {
return false
})()
}
// ── R-304 (2026-10-08) — „wrong code" only when every earlier package was tried ──────────────────────
//
// The consequence asserted: is the refusal a WRONG CODE (the only error the screen may answer with „check your
// typing")? It may be only when the hub withheld nothing and every served package was tried.
//
// RED-PROOF: make tryRetained return 0 for `unchecked` → the three „unchecked" cases return the plain refusal
// → they FAIL; the all-tried case keeps passing.
func TestR304_WrongCodeOnlyWhenEveryEarlierPackageWasTried(t *testing.T) {
ensureAge(t)
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "4444567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
other := sealBundle(t, IdentityBundle{ResticRepoPassword: "5555567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
const code = "a code that opens nothing whatsoever in this test"
cases := []struct {
name string
blobs int
withheld int
limit int
unchecked int // 0 = must be a plain wrong code
}{
{"all tried, nothing withheld", 2, 0, 6, 0},
{"the hub withheld rows (no key material / over its cap)", 1, 3, 6, 3},
{"only withheld rows, nothing served", 0, 2, 6, 2},
{"the attempt cap stopped the loop", 5, 0, 2, 3},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
blobs := make([]RetainedBlob, 0, c.blobs)
for i := 0; i < c.blobs; i++ {
blobs = append(blobs, RetainedBlob{Blob: other, SupersededAt: "2026-08-01 00:00:00", Index: i})
}
_, err := OffsiteKeyRecoverer{
Fetch: fetcherFor(current),
FetchRetained: func(context.Context) ([]RetainedBlob, int, error) {
return blobs, c.withheld, nil
},
MaxRetainedTried: c.limit,
}.RecoverOffsiteRepoPassword(context.Background(), code)
if err == nil {
t.Fatal("a code that opens nothing succeeded")
}
var ue *RetainedUncheckedError
isUnchecked := errors.As(err, &ue)
if c.unchecked == 0 {
if isUnchecked {
t.Fatalf("every earlier package was tried — this IS a wrong code, got %v", err)
}
return
}
if !isUnchecked || ue.Unchecked != c.unchecked || !errors.Is(err, ErrRetainedUnchecked) {
t.Fatalf("err = %v, want RetainedUncheckedError{%d} — never call the code wrong when packages were not tried", err, c.unchecked)
}
})
}
}
// A malformed (empty) served package counts as not tried.
func TestR304_EmptyServedPackageCountsAsUnchecked(t *testing.T) {
ensureAge(t)
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "6666567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
_, err := OffsiteKeyRecoverer{
Fetch: fetcherFor(current),
FetchRetained: retainedFetcherFor(RetainedBlob{Blob: nil, SupersededAt: "2026-08-01 00:00:00"}),
}.RecoverOffsiteRepoPassword(context.Background(), testR2)
var ue *RetainedUncheckedError
if !errors.As(err, &ue) || ue.Unchecked != 1 {
t.Fatalf("err = %v, want RetainedUncheckedError{1}", err)
}
}
+60 -5
View File
@@ -66,6 +66,13 @@ type ProvenRestoreTestReporter interface {
ProvenRestoreTests(ctx context.Context) []RestoreTest
}
// KnownBackupReporter is the newest SUCCESSFUL backup per tier and guest kept on disk (R-894's
// backup-success-state.json). collectBackups folds it into the report so a restart does not erase the
// hub's evidence of a backup that ran minutes before it (the kernel-night shape, 2026-10-09).
type KnownBackupReporter interface {
KnownBackupSuccesses() []Backup
}
// PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern).
// Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty.
type PBSReporter interface {
@@ -105,6 +112,7 @@ type Collector struct {
backups BackupReporter
restoreTests RestoreTestReporter
provenTests ProvenRestoreTestReporter
knownBackups KnownBackupReporter // R-894 file → host report after a restart (nil → in-memory only)
pbs PBSReporter
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
@@ -115,6 +123,7 @@ type Collector struct {
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
deliberateOff func() *DeliberateOff // R-998: household power-off record of this boot (nil → omitted)
foreignKey ForeignKeyArchiveReporter // R-366 slice 2 (nil → omitted)
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
@@ -247,6 +256,12 @@ func (c *Collector) SetGuestDiskTrimReporter(r GuestDiskTrimReporter) *Collector
return c
}
// SetDeliberateOffSource wires the R-998 power-off record (poweroff.State.Current, adapted).
func (c *Collector) SetDeliberateOffSource(f func() *DeliberateOff) *Collector {
c.deliberateOff = f
return c
}
// SelfUpdateReporter is the D1 seam the selfupdate commit-manager plugs into (same consumer-side
// pattern — hub does not import selfupdate). nil (feature not wired) → pending=false on the report.
type SelfUpdateReporter interface {
@@ -423,6 +438,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
if c.oob != nil {
report.OOB = c.oob.OOBStatus(ctx)
}
// R-998: a household power-off accepted in this boot rides every report until the box is down.
if c.deliberateOff != nil {
report.DeliberateOff = c.deliberateOff()
}
return report, nil
}
@@ -576,14 +595,50 @@ func (c *Collector) collectStorage(ctx context.Context) []StorageTarget {
// collectBackups / collectRestoreTests read the agent's latest backup + restore-test state
// via the seams. Best-effort: a nil reporter or nil slice degrades to an empty (non-nil)
// list so the collection always marshals as [].
//
// The saved successes (SetKnownBackupReporter) are added for each tier and guest the in-memory list has
// no success for at or after the saved time. Why: the in-memory list is empty after an agent restart,
// and the hub's backup-freshness check reads only what the reports carried. Measured 2026-10-09 on
// demo-felhom: the night backup landed 02:40 UTC, the kernel step restarted the host at 02:44, no report
// fell in between, so two kernel nights in a row left no trace and the hub alarmed „newest backup is 48h
// old" at 03:00. Only SUCCESSES are saved, so a failure is never hidden and never invented.
// Pinned by TestCollectBackups_SavedSuccessSurvivesARestart.
func (c *Collector) collectBackups(ctx context.Context) []Backup {
if c.backups == nil {
return []Backup{}
out := []Backup{}
if c.backups != nil {
if b := c.backups.Backups(ctx); b != nil {
out = append(out, b...)
}
}
if b := c.backups.Backups(ctx); b != nil {
return b
if c.knownBackups == nil {
return out
}
return []Backup{}
for _, k := range c.knownBackups.KnownBackupSuccesses() {
kt, err := time.Parse(time.RFC3339, k.StartedAt)
if err != nil {
continue
}
covered := false
for _, b := range out {
if !b.Success || b.TargetID != k.TargetID || b.VMID != k.VMID {
continue
}
if bt, err := time.Parse(time.RFC3339, b.StartedAt); err == nil && !bt.Before(kt) {
covered = true
break
}
}
if !covered {
out = append(out, k)
}
}
return out
}
// SetKnownBackupReporter wires the R-894 saved successes into the host report (nil-safe → in-memory only).
func (c *Collector) SetKnownBackupReporter(r KnownBackupReporter) *Collector {
c.knownBackups = r
return c
}
// collectRestoreTests merges the in-memory result with the PERSISTED per-tier proofs (R-189).
+62
View File
@@ -0,0 +1,62 @@
package hub
import (
"context"
"testing"
)
type fakeMemBackups []Backup
func (f fakeMemBackups) Backups(context.Context) []Backup { return f }
type fakeKnownBackups []Backup
func (f fakeKnownBackups) KnownBackupSuccesses() []Backup { return f }
// The consequence, not the mechanism: after a restart (empty in-memory list) the host report still carries
// the night's backup, so the hub's freshness check sees it. Measured 2026-10-09 on demo-felhom: without this
// the report carried backups: [] and the hub alarmed „newest backup is 48h old" an hour after a good backup.
// RED-PROOF: return before the saved-success loop in collectBackups → the first case reports nothing → fails.
func TestCollectBackups_SavedSuccessSurvivesARestart(t *testing.T) {
saved := Backup{TargetID: "felhom-backup", VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z"}
t.Run("restart: memory empty, the saved success is reported", func(t *testing.T) {
c := &Collector{backups: fakeMemBackups(nil), knownBackups: fakeKnownBackups{saved}}
got := c.collectBackups(context.Background())
if len(got) != 1 || got[0].StartedAt != saved.StartedAt || !got[0].Success || got[0].TargetID != "felhom-backup" {
t.Fatalf("want the saved success, got %+v", got)
}
})
t.Run("memory holds the same or a newer success: no second entry", func(t *testing.T) {
mem := Backup{TargetID: "felhom-backup", VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z", Archive: "a"}
c := &Collector{backups: fakeMemBackups{mem}, knownBackups: fakeKnownBackups{saved}}
if got := c.collectBackups(context.Background()); len(got) != 1 || got[0].Archive != "a" {
t.Fatalf("want only the in-memory record, got %+v", got)
}
})
t.Run("a newer FAILURE in memory never hides the saved success, and is kept", func(t *testing.T) {
fail := Backup{TargetID: "felhom-backup", VMID: 9201, Success: false, StartedAt: "2026-10-10T02:40:00Z", Error: "x"}
c := &Collector{backups: fakeMemBackups{fail}, knownBackups: fakeKnownBackups{saved}}
got := c.collectBackups(context.Background())
if len(got) != 2 || got[0].Success || !got[1].Success {
t.Fatalf("want the failure and the saved success, got %+v", got)
}
})
t.Run("another tier's success does not cover this tier", func(t *testing.T) {
other := Backup{TargetID: "felhom-pbs", VMID: 9201, Success: true, StartedAt: "2026-10-10T00:00:00Z"}
c := &Collector{backups: fakeMemBackups{other}, knownBackups: fakeKnownBackups{saved}}
if got := c.collectBackups(context.Background()); len(got) != 2 {
t.Fatalf("want both tiers, got %+v", got)
}
})
t.Run("no saved state wired: the in-memory list as before, never nil", func(t *testing.T) {
c := &Collector{}
if got := c.collectBackups(context.Background()); got == nil || len(got) != 0 {
t.Fatalf("want an empty non-nil list, got %#v", got)
}
})
}
+29
View File
@@ -0,0 +1,29 @@
package hub
import (
"context"
"errors"
"testing"
"time"
)
// R-998: the power-off path waits on SetReportedHook, so the hook must fire only for a report the hub
// ACCEPTED — a failed push must not let the box power off believing the hub knows.
func TestLoop_ReportedHookOnlyAfterAcceptedReport(t *testing.T) {
var cn, rn int32
rep := &HostReport{DeliberateOff: &DeliberateOff{RequestedAt: "2026-10-13T18:00:00Z"}}
loop := NewLoop(&fakeCollector{report: rep, n: &cn},
&fakeReporter{env: &ControlEnvelope{}, errSeq: []error{errors.New("hub 5xx")}, n: &rn},
900*time.Second, quietLogger())
var seen []*HostReport
loop.SetReportedHook(func(r *HostReport) { seen = append(seen, r) })
loop.cycle(context.Background(), 900*time.Second) // the push fails
if len(seen) != 0 {
t.Fatalf("hook fired for a refused report: %d", len(seen))
}
loop.cycle(context.Background(), 900*time.Second) // the push is accepted
if len(seen) != 1 || seen[0].DeliberateOff == nil || seen[0].DeliberateOff.RequestedAt != "2026-10-13T18:00:00Z" {
t.Fatalf("hook after an accepted report = %+v", seen)
}
}
+10
View File
@@ -66,8 +66,15 @@ type Loop struct {
// retry, no duplicate shipping). Loop state is single-goroutine (cycle only).
logTailSource func(maxBytes int) []string
logTailPending bool
// reported (R-998) is called after every report the hub ACCEPTED, with that report. The power-off
// path waits on it so the hub has the deliberate-off record before the box goes down.
reported func(r *HostReport)
}
// SetReportedHook wires the R-998 after-report observer. Optional.
func (l *Loop) SetReportedHook(f func(r *HostReport)) { l.reported = f }
// logTailMaxBytes caps the heartbeat log tail (newest lines kept).
const logTailMaxBytes = 128 * 1024
@@ -156,6 +163,9 @@ func (l *Loop) cycle(ctx context.Context, current time.Duration) time.Duration {
l.logger.Warn("hub: report failed; keeping current interval", "err", err)
return current
}
if l.reported != nil {
l.reported(report)
}
if report.LogTail != nil {
// Transparency: the pull is visible in the box's own log (and thus in the ring).
l.logger.Info("operator log pull served", "component", "agent", "lines", len(report.LogTail.Lines))
+22
View File
@@ -144,6 +144,12 @@ type HostReport struct {
// in the steady state, so the cross-repo host-report golden stays byte-stable.
LogTail *LogTail `json:"log_tail,omitempty"`
// DeliberateOff (R-998, doc 14): the household asked this box to power off, in THIS boot. Present on
// every report from the request until the box goes down; absent again after the next boot — that
// absence is how the hub learns the box is back. The hub holds its silence alarms while it is set
// (14 days). `omitempty`: absent in the steady state, so the cross-repo host-report golden stays stable.
DeliberateOff *DeliberateOff `json:"deliberate_off,omitempty"`
// OOB is the operator-access health stanza (TASK H1). It answers the operator's question — "can I
// get into this box right now, and if not, why" — from the hub: felhom-sshd up + on which port,
// locally reachable, the tunnel handshake age (the OOB path rides wg-felhom), whether the operator
@@ -636,6 +642,17 @@ type WireOSUpdate struct {
// HostRelease is the newest approved HOST release (hub v0.131.0, `11` §8 step 3) — a separate set: a version
// approved for the guest is not approved for the host by that fact alone.
HostRelease *WireOSRelease `json:"host_release,omitempty"`
// Kernel is the kernel lane's instruction for THIS box (R-836, `09` §3 decision 172, `11` §5.11): the kernel the
// household was told about, and whether tonight is a told night. Nil (an older hub, or no kernel due) = no kernel
// step. The hub sets Tonight only after the household's mail the day before went out — no mail, no step.
Kernel *WireKernelStep `json:"kernel,omitempty"`
}
// WireKernelStep is the hub's kernel-lane instruction (hub osupdates.KernelBlock — field-exact, cross-repo).
type WireKernelStep struct {
Kver string `json:"kver"` // e.g. "7.0.14-22-pve" — the wrapper refuses any other (R23)
Tonight bool `json:"tonight"` // the household was mailed the day before: tonight's leg may reboot
NotifiedAt string `json:"notified_at,omitempty"` // when that mail went out (RFC 3339), for the log
}
// WireOSRelease is an approved version set; Snapshot is the approval time (YYYYMMDDTHHMMSSZ) the wrapper uses
@@ -747,3 +764,8 @@ type ForeignKeyArchives struct {
Oldest string `json:"oldest"`
Newest string `json:"newest"`
}
// DeliberateOff is the household-requested power-off record (R-998).
type DeliberateOff struct {
RequestedAt string `json:"requested_at"` // RFC3339 UTC, the agent's clock
}
+17 -4
View File
@@ -15,7 +15,7 @@ import (
// Red-proof: drop the `b.Success &&` guard and the failed-backup sub-case fails; move the call after release()
// and the gate sub-case fails.
func TestAfterPrimaryBackup(t *testing.T) {
run := func(t *testing.T, failErr string) (calls []int, gateHeld bool) {
run := func(t *testing.T, failErr, path string) (calls []int, gateHeld bool) {
gate := &backup.InFlight{}
b := &fakeBackups{failErr: failErr}
srv := newTestServerS(t, &fakeGuests{}, b, &fakeStore{}, nil)
@@ -34,7 +34,7 @@ func TestAfterPrimaryBackup(t *testing.T) {
done <- struct{}{}
})
h := srv.Handler()
if do(t, h, "POST", "/backup", "A", "").Code != http.StatusAccepted {
if do(t, h, "POST", path, "A", "").Code != http.StatusAccepted {
t.Fatal("POST /backup not accepted")
}
select {
@@ -47,7 +47,7 @@ func TestAfterPrimaryBackup(t *testing.T) {
return calls, gateHeld
}
t.Run("success runs the leg under the gate", func(t *testing.T) {
calls, held := run(t, "")
calls, held := run(t, "", "/backup")
if len(calls) != 1 {
t.Fatalf("the leg ran %d time(s), want 1", len(calls))
}
@@ -56,8 +56,21 @@ func TestAfterPrimaryBackup(t *testing.T) {
}
})
t.Run("a failed backup runs nothing", func(t *testing.T) {
if calls, _ := run(t, "vzdump exploded"); len(calls) != 0 {
if calls, _ := run(t, "vzdump exploded", "/backup"); len(calls) != 0 {
t.Fatalf("the leg ran after a FAILED backup: %v", calls)
}
})
// R-899: a household press is not the night's backup — no OS leg after it. Same fake, same successful backup as
// the first sub-case; only the query differs. Red-proof: make handleBackup ignore `trigger` and this sub-case
// fails with the leg run once.
t.Run("a manual press runs nothing", func(t *testing.T) {
if calls, _ := run(t, "", "/backup?trigger=manual"); len(calls) != 0 {
t.Fatalf("the OS leg ran after a manual press: %v", calls)
}
})
t.Run("the scheduled path with the new query still runs the leg", func(t *testing.T) {
if calls, _ := run(t, "", "/backup?trigger=night"); len(calls) != 1 {
t.Fatalf("the leg ran %d time(s) after a non-manual backup, want 1", len(calls))
}
})
}
+4 -1
View File
@@ -226,6 +226,9 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
if base == nil {
base = context.Background()
}
// The reply's fields are read HERE, before the job goroutine may write job.Phase (a data race the race
// detector caught, 2026-10-13 — the reply could show the finished phase or a torn value).
jobID, phase := job.JobID, job.Phase
done := make(chan struct{})
go func() {
defer close(done)
@@ -274,7 +277,7 @@ func (s *Server) handleEscrowCeremonyStart(w http.ResponseWriter, r *http.Reques
"claim_ttl_s", int(escrowClaimTTL.Seconds()), "duration_ms", time.Since(start).Milliseconds())
}()
s.escrowDone = done // tests wait on it; production polls the status endpoint
writeStatus(w, http.StatusAccepted, true, map[string]any{"vmid": vmid, "job_id": job.JobID, "phase": job.Phase}, "")
writeStatus(w, http.StatusAccepted, true, map[string]any{"vmid": vmid, "job_id": jobID, "phase": phase}, "")
}
// handleEscrowCeremonyStatus is GET /escrow/ceremony/status: phase + the NON-SECRET summary.
+16
View File
@@ -127,6 +127,22 @@ func (s *Server) handleRecoverOffsitePassword(w http.ResponseWriter, r *http.Req
"retained_has_restic_pw": match.HasResticPassword,
},
"the recovery code is correct, but it belongs to an EARLIER sealed package (superseded "+match.SupersededAt+"), not the one currently held")
// ── R-304 (2026-10-08) — NOT EVERY EARLIER PACKAGE WAS CHECKED. ─────────────────────────
//
// The current package refused the code, no retained package opened it — and at least one earlier
// package the hub holds was never tried (or the list could not be read). Saying „the code is
// wrong" here would claim a check that did not happen. 424 (Failed Dependency): the verdict
// depends on packages we could not try. The controller classifies on the status, never on this
// sentence; an older controller maps an unknown status to its neutral „we do not know why".
case errors.Is(err, escrow.ErrRetainedUnchecked):
n := -1
var ue *escrow.RetainedUncheckedError
if errors.As(err, &ue) {
n = ue.Unchecked
}
s.logger.Warn("local-api: offsite key recovery: the code did not open the current package and earlier packages were NOT all checked — not reported as a wrong code (R-304)", "vmid", vmid, "unchecked", n)
writeStatus(w, http.StatusFailedDependency, false, map[string]any{"older_unchecked": n},
"the recovery code did not open the current sealed package, and earlier packages the hub holds were not all checked — the code may belong to one of them; nothing was written")
case errors.Is(err, escrow.ErrNoResticPassword):
s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid)
writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)")
@@ -54,6 +54,13 @@ func TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus(t *testing.T) {
wantStatus: 404,
mustNotSay: []string{"did not open"},
},
{
// R-304: earlier packages were not all tried — never „did not open the sealed bundle" (the wrong-code words).
name: "earlier packages not all checked — not a wrong code",
err: &escrow.RetainedUncheckedError{Unchecked: 2},
wantStatus: 424,
mustNotSay: []string{"did not open the sealed bundle", "could not be fetched"},
},
{
name: "the bundle predates the repository-password field",
err: escrow.ErrNoResticPassword,
+21
View File
@@ -1,6 +1,7 @@
package localapi
import (
"time"
"context"
"fmt"
"log/slog"
@@ -104,6 +105,26 @@ func (b *GuestBinder) RebootGuest(ctx context.Context, vmid int) error {
return nil
}
// ShutdownGuest stops the guest cleanly with a time limit (R-998 host power-off, step 2). `pct shutdown`
// blocks until the guest is down or the limit passes; then it fails and the guest is still running.
func (b *GuestBinder) ShutdownGuest(ctx context.Context, vmid int, timeout time.Duration) error {
secs := int(timeout / time.Second)
if secs < 1 {
secs = 1
}
b.logger.Warn("host-poweroff: shutting the guest down", "vmid", vmid, "timeout_s", secs)
if err := b.run(ctx, "pct", "shutdown", strconv.Itoa(vmid), "--timeout", strconv.Itoa(secs)); err != nil {
return fmt.Errorf("host-poweroff: pct shutdown %d: %w", vmid, err)
}
return nil
}
// PowerOffHost powers the Proxmox host off (R-998, step 3). The one fixed vector `/usr/bin/systemctl poweroff`.
func (b *GuestBinder) PowerOffHost(ctx context.Context) error {
b.logger.Warn("host-poweroff: powering the host off")
return b.run(ctx, "/usr/bin/systemctl", "poweroff")
}
func (b *GuestBinder) run(ctx context.Context, name string, args ...string) error {
_, stderr, err := b.runner.Run(ctx, name, args...)
if err != nil {
+362
View File
@@ -0,0 +1,362 @@
package localapi
import (
"context"
"errors"
"net/http"
"os"
"path/filepath"
"strconv"
"strings"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
)
// ── POST /host/poweroff (R-998, documentation/architecture/14-host-power-off.md) ──────────────────────
//
// The household's „A gép kikapcsolása": shut THIS guest down cleanly, then power the whole host off. The
// first local-API route whose effect reaches beyond the caller's own guest — a compromised controller can
// now switch the box off, and nobody can switch it on remotely. The mitigations are the limits below:
//
// - refused in the first 10 minutes after boot (no power-off loop);
// - at most ONE accepted request per boot (poweroff.State, persisted, keyed by the boot);
// - refused while a backup, restore test, OS update step, controller update, drive format, NAS verify
// or escrow ceremony runs, or a vzdump of the guest is in flight — each named in the 409;
// - refused while ANY OTHER guest or VM runs on the host (security review 2026-10-13): one guest's
// controller must not switch off another household's guest, a scratch guest or a test VM (demo-hp
// carries the Tester 1 VM). Read from /proc (`lxc-start -n N`, `kvm -id N`), fail closed;
// - the heavy-operation gate is TAKEN on acceptance and never given back (the box is going down), so no
// backup, restore test or OS step can start between the checks and the power-off (security review).
// Released only if the power-off command itself fails.
//
// Every check fails CLOSED: an unwired gate, an unreadable task list or /proc refuses.
// On acceptance the record is written FIRST, the hub is given a report carrying it (bounded wait), and
// only then: guest shutdown (time-limited), host power-off. Pinned by hostpoweroff_test.go.
// HostPowerer is the privileged half: both run as fixed sudo vectors (configs/felhom-agent.sudoers).
type HostPowerer interface {
ShutdownGuest(ctx context.Context, vmid int, timeout time.Duration) error
PowerOffHost(ctx context.Context) error
}
// HostPowerOffConfig wires the route. A nil config leaves it answering 503.
type HostPowerOffConfig struct {
State *poweroff.State
Power HostPowerer
ReportNow func(ctx context.Context) bool // push a report carrying the record; true = the hub accepted it
Uptime func() (time.Duration, bool) // time since host boot; false = unknown (refuse)
Running func() ([]int, error) // vmids of every running guest and VM on the host; default /proc
MinUptime time.Duration // default 10 min
GuestStop time.Duration // pct shutdown --timeout; default 120 s
ReportMax time.Duration // longest wait for the hub; default 60 s
JobWait time.Duration // longest wait for a job that started after acceptance; default 15 min
JobPoll time.Duration // how often that wait looks again; default 10 s
Now func() time.Time
}
// Refusal codes — the controller maps each to a household sentence. Stable wire values.
const (
PowerOffTooSoon = "too_soon_after_boot"
PowerOffAlreadyAsked = "already_requested_this_boot"
PowerOffBackup = "backup_running"
PowerOffRestoreTest = "restore_test_running"
PowerOffOSUpdate = "os_update_running"
PowerOffCtrlUpdate = "controller_update_running"
PowerOffDrive = "drive_operation_running"
PowerOffEscrow = "escrow_ceremony_running"
PowerOffBusy = "host_busy"
PowerOffOtherGuests = "other_guests_running"
PowerOffUnknown = "state_unknown"
)
// SetHostPowerOff enables POST /host/poweroff.
func (s *Server) SetHostPowerOff(c HostPowerOffConfig) {
if c.MinUptime == 0 {
c.MinUptime = 10 * time.Minute
}
if c.GuestStop == 0 {
c.GuestStop = 120 * time.Second
}
if c.ReportMax == 0 {
c.ReportMax = 60 * time.Second
}
if c.Now == nil {
c.Now = time.Now
}
if c.JobWait == 0 {
c.JobWait = 15 * time.Minute
}
if c.JobPoll == 0 {
c.JobPoll = 10 * time.Second
}
if c.Running == nil {
c.Running = func() ([]int, error) { return runningGuestsFromProc("/proc") }
}
s.powerOff = &c
}
type hostPowerOffRequest struct {
VMID int `json:"vmid"`
}
func (s *Server) handleHostPowerOff(w http.ResponseWriter, r *http.Request, vmid int) {
c := s.powerOff
if c == nil || c.State == nil || c.Power == nil {
writeErr(w, http.StatusServiceUnavailable, "host power-off not configured on this host")
return
}
if r.ContentLength != 0 {
var req hostPowerOffRequest
if !decodeBody(w, r, &req) {
return
}
if !s.scopedFromBody(w, req.VMID, vmid, r.URL.Path) {
return
}
}
// One decision at a time: two requests racing must not both pass the guards before either marks.
s.powerOffMu.Lock()
defer s.powerOffMu.Unlock()
if code, detail := s.powerOffBlocker(r.Context(), vmid); code != "" {
s.logger.Info("local-api: host power-off refused", "vmid", vmid, "reason", code, "detail", detail)
writeStatus(w, http.StatusConflict, false, map[string]any{"reason": code, "detail": detail}, "host power-off refused: "+code)
return
}
// Take the heavy-operation gate for good: nothing heavy may start between these checks and the power-off.
release, busy, free := s.inFlight.TryAcquire("host-poweroff")
if !free {
writeStatus(w, http.StatusConflict, false, map[string]any{"reason": PowerOffBusy, "detail": busy}, "host power-off refused: "+PowerOffBusy)
return
}
rec, err := c.State.Mark(c.Now(), vmid)
if err != nil {
release()
code := PowerOffUnknown
if errors.Is(err, poweroff.ErrAlreadyThisBoot) {
code = PowerOffAlreadyAsked
}
s.logger.Warn("local-api: host power-off refused (record)", "vmid", vmid, "reason", code, "err", err)
writeStatus(w, http.StatusConflict, false, map[string]any{"reason": code, "detail": err.Error()}, "host power-off refused: "+code)
return
}
s.logger.Warn("local-api: HOST POWER-OFF accepted (household request)", "vmid", vmid,
"requested_at", rec.RequestedAt.Format(time.RFC3339))
base := s.baseCtx
if base == nil {
base = context.Background()
}
go s.runHostPowerOff(base, vmid, release)
writeStatus(w, http.StatusAccepted, true, map[string]any{
"vmid": vmid, "powering_off": true, "requested_at": rec.RequestedAt.Format(time.RFC3339),
}, "")
}
// runHostPowerOff: hub first (bounded), then the guest (time-limited), then the host. A failed guest
// shutdown does not stop the power-off — the household asked for it, and Proxmox's own shutdown stops
// any guest still running (pve-guests) before the host goes down.
func (s *Server) runHostPowerOff(base context.Context, vmid int, release func()) {
c := s.powerOff
if c.ReportNow != nil {
ctx, cancel := context.WithTimeout(base, c.ReportMax)
ok := c.ReportNow(ctx)
cancel()
s.logger.Warn("host-poweroff: step 1/3 — hub told", "vmid", vmid, "hub_accepted", ok)
}
if !s.waitJobsClear(base, vmid, "before the guest shutdown") {
release()
return
}
start := time.Now()
ctx, cancel := context.WithTimeout(base, c.GuestStop+60*time.Second)
err := c.Power.ShutdownGuest(ctx, vmid, c.GuestStop)
cancel()
if err != nil {
s.logger.Error("host-poweroff: step 2/3 — guest shutdown failed; powering off anyway", "vmid", vmid,
"took", time.Since(start).Round(time.Second), "err", err)
} else {
s.logger.Warn("host-poweroff: step 2/3 — guest stopped", "vmid", vmid, "took", time.Since(start).Round(time.Second))
}
if !s.waitJobsClear(base, vmid, "before the host power-off") {
release()
return
}
s.logger.Warn("host-poweroff: step 3/3 — powering the host off")
ctx, cancel = context.WithTimeout(base, 60*time.Second)
defer cancel()
if err := c.Power.PowerOffHost(ctx); err != nil {
s.logger.Error("host-poweroff: host power-off command failed — heavy-operation gate released", "err", err)
release()
}
}
// powerOffBlocker returns the first reason to refuse ("" = none) and a detail for the log/operator.
func (s *Server) powerOffBlocker(ctx context.Context, vmid int) (string, string) {
c := s.powerOff
if c.State.Current() != nil {
return PowerOffAlreadyAsked, "a power-off was already accepted in this boot"
}
if c.Uptime == nil {
return PowerOffUnknown, "uptime not wired"
}
up, ok := c.Uptime()
if !ok {
return PowerOffUnknown, "cannot read the host uptime"
}
if up < c.MinUptime {
return PowerOffTooSoon, "host up " + up.Round(time.Second).String() + ", minimum " + c.MinUptime.String()
}
if s.inFlight == nil || s.staleLock == nil {
return PowerOffUnknown, "the heavy-operation gate or the Proxmox task reader is not wired"
}
if code, detail := s.otherGuestsBlocker(vmid); code != "" {
return code, detail
}
if busy := s.inFlight.Busy(); busy != "" {
switch {
case strings.HasPrefix(busy, "backup:"):
return PowerOffBackup, busy
case busy == "restore-test":
return PowerOffRestoreTest, busy
case strings.HasPrefix(busy, "os-"):
return PowerOffOSUpdate, busy
default:
return PowerOffBusy, busy
}
}
return s.jobBlocker(ctx, vmid)
}
// jobBlocker is the part of the checks that can change after acceptance — a backup job, a controller update, a drive
// or NAS job, an escrow ceremony, a Proxmox vzdump. runHostPowerOff asks it again before each step (security review
// 2026-10-13: the checks ran only once). The heavy-operation gate is not here: the power-off holds it.
func (s *Server) jobBlocker(ctx context.Context, vmid int) (string, string) {
s.jobsMu.Lock()
for k, j := range s.jobs {
if j != nil && backupInFlight(j.Phase) {
s.jobsMu.Unlock()
return PowerOffBackup, "backup job " + j.JobID + " on target " + k.target
}
}
s.jobsMu.Unlock()
s.swapMu.Lock()
for v, on := range s.swapInFlight {
if on {
s.swapMu.Unlock()
return PowerOffCtrlUpdate, "controller swap in flight for guest " + strconv.Itoa(v)
}
}
s.swapMu.Unlock()
if s.formatJobs != nil {
if j := s.formatJobs.get(); j != nil && j.Phase == formatPhaseRunning {
return PowerOffDrive, "drive format running"
}
}
s.netVerifyMu.Lock()
nv := s.netVerifyCur != nil && s.netVerifyCur.Phase == netVerifyPhaseRunning
s.netVerifyMu.Unlock()
if nv {
return PowerOffDrive, "network storage verify running"
}
s.escrowMu.Lock()
esc := s.escrowJob != nil && s.escrowJob.Phase == escrowPhaseRunning
s.escrowMu.Unlock()
if esc {
return PowerOffEscrow, "escrow ceremony running"
}
vz, err := s.staleLock.BackupRunning(ctx, vmid)
if err != nil {
return PowerOffUnknown, "cannot list running Proxmox tasks: " + err.Error()
}
if vz {
return PowerOffBackup, "vzdump running for guest " + strconv.Itoa(vmid)
}
return "", ""
}
// runningGuestsFromProc lists the vmids of running containers (`lxc-start … -n N`) and VMs (`kvm -id N`) from
// the process table — readable by the non-root agent, unlike /run/qemu-server (measured on demo-hp 2026-10-13).
// An unreadable /proc is an error (fail closed); an unreadable single process is skipped (it exited).
func runningGuestsFromProc(procDir string) ([]int, error) {
ents, err := os.ReadDir(procDir)
if err != nil {
return nil, err
}
seen := map[int]bool{}
var out []int
for _, e := range ents {
if _, err := strconv.Atoi(e.Name()); err != nil {
continue
}
raw, err := os.ReadFile(filepath.Join(procDir, e.Name(), "cmdline"))
if err != nil || len(raw) == 0 {
continue
}
args := strings.Split(strings.TrimRight(string(raw), "\x00"), "\x00")
flag := ""
switch filepath.Base(args[0]) {
case "lxc-start":
flag = "-n"
case "kvm", "qemu-system-x86_64":
flag = "-id"
default:
continue
}
for i := 1; i+1 < len(args); i++ {
if args[i] == flag {
if v, err := strconv.Atoi(args[i+1]); err == nil && !seen[v] {
seen[v] = true
out = append(out, v)
}
}
}
}
return out, nil
}
// otherGuestsBlocker refuses while any guest or VM other than the caller's runs (or the list cannot be read).
func (s *Server) otherGuestsBlocker(vmid int) (string, string) {
running, err := s.powerOff.Running()
if err != nil {
return PowerOffUnknown, "cannot list the running guests: " + err.Error()
}
var others []string
for _, v := range running {
if v != vmid {
others = append(others, strconv.Itoa(v))
}
}
if len(others) > 0 {
return PowerOffOtherGuests, "other guests or VMs running: " + strings.Join(others, ", ")
}
return "", ""
}
// waitJobsClear waits until jobBlocker finds nothing, at most JobWait. false = still busy (or unreadable) at the end:
// the power-off is CANCELLED — the box stays on, the gate is released, the log says why. Better a box that did not
// switch off than a backup, update or format cut in the middle.
func (s *Server) waitJobsClear(base context.Context, vmid int, when string) bool {
c := s.powerOff
ctx, cancel := context.WithTimeout(base, c.JobWait)
defer cancel()
for {
// Another guest or VM started after acceptance (security review 2026-10-13): no waiting — cancel at once.
if code, detail := s.otherGuestsBlocker(vmid); code != "" {
s.logger.Error("host-poweroff: CANCELLED — another guest is running now; the box stays on", "when", when,
"reason", code, "detail", detail)
return false
}
code, detail := s.jobBlocker(ctx, vmid)
if code == "" {
return true
}
s.logger.Warn("host-poweroff: waiting — a job started after the request", "when", when, "reason", code, "detail", detail)
select {
case <-ctx.Done():
s.logger.Error("host-poweroff: CANCELLED — still busy after the wait; the box stays on", "when", when,
"reason", code, "detail", detail, "waited", c.JobWait)
return false
case <-time.After(c.JobPoll):
}
}
}
+353
View File
@@ -0,0 +1,353 @@
package localapi
import (
"context"
"encoding/json"
"errors"
"net/http"
"os"
"path/filepath"
"sort"
"strings"
"sync"
"sync/atomic"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/backup"
"gitea.dooplex.hu/admin/felhom-agent/internal/poweroff"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// fakePower records the power-off steps in order.
type fakePower struct {
mu sync.Mutex
steps []string
}
func (f *fakePower) ShutdownGuest(_ context.Context, vmid int, timeout time.Duration) error {
f.mu.Lock()
defer f.mu.Unlock()
f.steps = append(f.steps, "shutdown:"+itoaT(vmid)+":"+timeout.String())
return nil
}
func (f *fakePower) PowerOffHost(context.Context) error {
f.mu.Lock()
defer f.mu.Unlock()
f.steps = append(f.steps, "poweroff")
return nil
}
func (f *fakePower) got() []string {
f.mu.Lock()
defer f.mu.Unlock()
return append([]string(nil), f.steps...)
}
func itoaT(n int) string { b, _ := json.Marshal(n); return string(b) }
type fakeVzdump struct{ running bool }
func (f fakeVzdump) Guests(context.Context) ([]proxmox.Guest, error) { return nil, nil }
func (f fakeVzdump) Lock(context.Context, int) (string, bool, error) { return "", true, nil }
func (f fakeVzdump) BackupRunning(context.Context, int) (bool, error) { return f.running, nil }
func (f fakeVzdump) Unlock(context.Context, int) error { return nil }
func (f fakeVzdump) Start(context.Context, int) error { return nil }
func (f fakeVzdump) HasVzdumpSnapshot(context.Context, int) (bool, error) { return false, nil }
func (f fakeVzdump) DeleteVzdumpSnapshot(context.Context, int) error { return nil }
type powerRig struct {
srv *Server
h http.Handler
power *fakePower
state *poweroff.State
report *[]string
}
func newPowerRig(t *testing.T, uptime time.Duration) *powerRig {
t.Helper()
srv := newTestServerS(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
srv.inFlight = &backup.InFlight{}
srv.staleLock = fakeVzdump{}
p := &fakePower{}
st := poweroff.New(filepath.Join(t.TempDir(), "poweroff.json"), func() string { return "1700000000" })
var reports []string
srv.SetHostPowerOff(HostPowerOffConfig{
State: st,
Power: p,
ReportNow: func(context.Context) bool {
// The record must already exist when the hub is told — that is the point of telling it.
if st.Current() == nil {
reports = append(reports, "report-without-record")
} else if len(p.got()) != 0 {
reports = append(reports, "report-after-"+p.got()[0])
} else {
reports = append(reports, "report")
}
return true
},
Uptime: func() (time.Duration, bool) { return uptime, true },
Running: func() ([]int, error) { return []int{8200}, nil },
Now: func() time.Time { return testNow },
})
return &powerRig{srv: srv, h: srv.Handler(), power: p, state: st, report: &reports}
}
func (r *powerRig) waitSteps(t *testing.T, n int) []string {
t.Helper()
for i := 0; i < 200 && len(r.power.got()) < n; i++ {
time.Sleep(time.Millisecond)
}
return r.power.got()
}
func reasonOf(t *testing.T, body []byte) string {
t.Helper()
var resp struct {
Data struct {
Reason string `json:"reason"`
} `json:"data"`
}
if err := json.Unmarshal(body, &resp); err != nil {
t.Fatalf("body %s: %v", body, err)
}
return resp.Data.Reason
}
// The whole path: 202, the record written, the hub told FIRST, then guest shutdown (120 s), then host off.
func TestHostPowerOff_AcceptedOrder(t *testing.T) {
r := newPowerRig(t, time.Hour)
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusAccepted {
t.Fatalf("got %d want 202 (%s)", w.Code, w.Body.String())
}
steps := r.waitSteps(t, 2)
want := []string{"shutdown:8200:2m0s", "poweroff"}
if len(steps) != 2 || steps[0] != want[0] || steps[1] != want[1] {
t.Fatalf("steps = %v, want %v", steps, want)
}
// The heavy-operation gate stays taken: no backup can start between the checks and the power-off.
if _, busy, ok := r.srv.inFlight.TryAcquire("backup:primary"); ok || busy != "host-poweroff" {
t.Fatalf("after acceptance a backup could take the gate (ok=%v busy=%q)", ok, busy)
}
if len(*r.report) != 1 || (*r.report)[0] != "report" {
t.Fatalf("hub report = %v, want one report carrying the record", *r.report)
}
if rec := r.state.Current(); rec == nil || rec.VMID != 8200 {
t.Fatalf("record = %+v", rec)
}
}
// Refused during a backup — the shared heavy-operation gate held by a backup tier.
func TestHostPowerOff_RefusedDuringBackup(t *testing.T) {
r := newPowerRig(t, time.Hour)
release, _, ok := r.srv.inFlight.TryAcquire("backup:primary")
if !ok {
t.Fatal("could not hold the gate")
}
defer release()
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffBackup {
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffBackup)
}
time.Sleep(5 * time.Millisecond)
if s := r.power.got(); len(s) != 0 {
t.Fatalf("power steps ran during a backup: %v", s)
}
if r.state.Current() != nil {
t.Fatal("a refused request must not use up the boot's one request")
}
// The same guard names a restore test and an OS step.
release()
for gate, want := range map[string]string{"restore-test": PowerOffRestoreTest, "os-pve-step": PowerOffOSUpdate} {
rel, _, _ := r.srv.inFlight.TryAcquire(gate)
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
rel()
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != want {
t.Fatalf("%s: got %d %s, want 409 %s", gate, w.Code, w.Body.String(), want)
}
}
}
// A vzdump of the guest in flight (a Proxmox task, not the agent's own gate) refuses too.
func TestHostPowerOff_RefusedDuringVzdump(t *testing.T) {
r := newPowerRig(t, time.Hour)
r.srv.staleLock = fakeVzdump{running: true}
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffBackup {
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffBackup)
}
}
// One accepted request per boot: the second is refused and nothing runs twice.
func TestHostPowerOff_SecondRequestSameBootRefused(t *testing.T) {
r := newPowerRig(t, time.Hour)
if w := do(t, r.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
t.Fatalf("first: %d", w.Code)
}
r.waitSteps(t, 2)
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffAlreadyAsked {
t.Fatalf("second: got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffAlreadyAsked)
}
time.Sleep(5 * time.Millisecond)
if s := r.power.got(); len(s) != 2 {
t.Fatalf("steps = %v, want exactly one run", s)
}
}
// Refused in the first 10 minutes after boot; an unknown uptime refuses too (fail closed).
func TestHostPowerOff_TooSoonAfterBoot(t *testing.T) {
r := newPowerRig(t, 9*time.Minute)
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffTooSoon {
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffTooSoon)
}
r.srv.powerOff.Uptime = func() (time.Duration, bool) { return 0, false }
w = do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffUnknown {
t.Fatalf("unknown uptime: got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffUnknown)
}
}
// Not configured → 503; another guest's vmid in the body → 403; no token → 401.
func TestHostPowerOff_ScopeAndConfig(t *testing.T) {
plain := newTestServer(t, &fakeGuests{}, &fakeBackups{}, &fakeStore{}, nil)
if w := do(t, plain, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusServiceUnavailable {
t.Fatalf("unconfigured: %d", w.Code)
}
r := newPowerRig(t, time.Hour)
if w := do(t, r.h, "POST", "/host/poweroff", "A", `{"vmid":9300}`); w.Code != http.StatusForbidden {
t.Fatalf("cross-guest: %d", w.Code)
}
if w := do(t, r.h, "POST", "/host/poweroff", "", ""); w.Code != http.StatusUnauthorized {
t.Fatalf("no token: %d", w.Code)
}
if r.state.Current() != nil || len(r.power.got()) != 0 {
t.Fatal("a refused request changed state")
}
}
// Another guest or VM on the host refuses: one controller must not switch off someone else's guest (security
// review 2026-10-13; demo-hp runs a scratch guest and the Tester 1 VM). An unreadable process list refuses too.
func TestHostPowerOff_RefusedWhileOtherGuestsRun(t *testing.T) {
r := newPowerRig(t, time.Hour)
r.srv.powerOff.Running = func() ([]int, error) { return []int{8200, 9202, 341}, nil }
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffOtherGuests {
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffOtherGuests)
}
r.srv.powerOff.Running = func() ([]int, error) { return nil, errors.New("no /proc") }
w = do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffUnknown {
t.Fatalf("unreadable: got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffUnknown)
}
if len(r.power.got()) != 0 || r.state.Current() != nil {
t.Fatal("a refused request changed state")
}
}
// Fail closed: an unwired vzdump reader refuses instead of skipping the check.
func TestHostPowerOff_FailsClosedWithoutTaskReader(t *testing.T) {
r := newPowerRig(t, time.Hour)
r.srv.staleLock = nil
w := do(t, r.h, "POST", "/host/poweroff", "A", "")
if w.Code != http.StatusConflict || reasonOf(t, w.Body.Bytes()) != PowerOffUnknown {
t.Fatalf("got %d %s, want 409 %s", w.Code, w.Body.String(), PowerOffUnknown)
}
}
// The /proc reader finds containers and VMs by their real command lines (as measured on demo-hp).
func TestRunningGuestsFromProc(t *testing.T) {
dir := t.TempDir()
write := func(pid, cmd string) {
_ = os.MkdirAll(filepath.Join(dir, pid), 0o755)
_ = os.WriteFile(filepath.Join(dir, pid, "cmdline"), []byte(strings.ReplaceAll(cmd, " ", "\x00")+"\x00"), 0o644)
}
write("1927", "/usr/bin/lxc-start -F -n 9201")
write("1900", "/usr/bin/lxc-start -F -n 9202")
write("1823", "/usr/bin/kvm -id 341 -name night1004-tester1 -no-shutdown")
write("974", "/usr/libexec/lxc/lxc-monitord --daemon")
write("50", "/usr/bin/bash -c echo -n 5")
_ = os.MkdirAll(filepath.Join(dir, "self"), 0o755)
got, err := runningGuestsFromProc(dir)
if err != nil {
t.Fatal(err)
}
sort.Ints(got)
if len(got) != 3 || got[0] != 341 || got[1] != 9201 || got[2] != 9202 {
t.Fatalf("got %v, want [341 9201 9202]", got)
}
if _, err := runningGuestsFromProc(filepath.Join(dir, "missing")); err == nil {
t.Fatal("an unreadable /proc must be an error")
}
}
// switchVzdump is a vzdump reader the test can flip while the power-off runs.
type switchVzdump struct {
fakeVzdump
on *atomic.Bool
}
func (f switchVzdump) BackupRunning(context.Context, int) (bool, error) { return f.on.Load(), nil }
// A job that starts AFTER acceptance (security review 2026-10-13): the power-off waits; if the job is still running
// when the wait ends, the power-off is cancelled — no guest shutdown, no host power-off, the gate given back.
func TestHostPowerOff_JobStartedAfterAcceptance(t *testing.T) {
// (a) it clears during the wait → the power-off goes ahead after it
r := newPowerRig(t, time.Hour)
var on atomic.Bool
r.srv.staleLock = switchVzdump{on: &on}
r.srv.powerOff.JobWait, r.srv.powerOff.JobPoll = 2*time.Second, 2*time.Millisecond
r.srv.powerOff.ReportNow = func(context.Context) bool { on.Store(true); return true } // a vzdump starts now
if w := do(t, r.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
t.Fatalf("accept: %d %s", w.Code, w.Body.String())
}
time.Sleep(30 * time.Millisecond)
if s := r.power.got(); len(s) != 0 {
t.Fatalf("a step ran while the vzdump ran: %v", s)
}
on.Store(false)
if s := r.waitSteps(t, 2); len(s) != 2 {
t.Fatalf("after the vzdump ended the steps = %v, want shutdown + poweroff", s)
}
// (b) it never clears → cancelled, nothing switched off, the heavy gate free again
r2 := newPowerRig(t, time.Hour)
var on2 atomic.Bool
r2.srv.staleLock = switchVzdump{on: &on2}
r2.srv.powerOff.JobWait, r2.srv.powerOff.JobPoll = 40*time.Millisecond, 2*time.Millisecond
r2.srv.powerOff.ReportNow = func(context.Context) bool { on2.Store(true); return true }
if w := do(t, r2.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
t.Fatalf("accept: %d", w.Code)
}
time.Sleep(150 * time.Millisecond)
if s := r2.power.got(); len(s) != 0 {
t.Fatalf("cancelled power-off still ran: %v", s)
}
if busy := r2.srv.inFlight.Busy(); busy != "" {
t.Fatalf("the heavy gate is still held after a cancel: %q", busy)
}
}
// A guest or VM started AFTER acceptance cancels the power-off at once (security review 2026-10-13).
func TestHostPowerOff_OtherGuestStartedAfterAcceptance(t *testing.T) {
r := newPowerRig(t, time.Hour)
var started atomic.Bool
r.srv.powerOff.Running = func() ([]int, error) {
if started.Load() {
return []int{8200, 341}, nil
}
return []int{8200}, nil
}
r.srv.powerOff.ReportNow = func(context.Context) bool { started.Store(true); return true } // a VM starts now
if w := do(t, r.h, "POST", "/host/poweroff", "A", ""); w.Code != http.StatusAccepted {
t.Fatalf("accept: %d %s", w.Code, w.Body.String())
}
time.Sleep(50 * time.Millisecond)
if s := r.power.got(); len(s) != 0 {
t.Fatalf("power-off ran although a VM started: %v", s)
}
if busy := r.srv.inFlight.Busy(); busy != "" {
t.Fatalf("the heavy gate is still held after a cancel: %q", busy)
}
}
+13 -1
View File
@@ -394,6 +394,10 @@ type Server struct {
// detached pipeline runs through (tests inject; production defaults set in NewServer).
netVerifyMu sync.Mutex
netVerifyCur *netVerifyJob
// R-998: POST /host/poweroff (nil = not configured → 503); one guard-and-mark decision at a time.
powerOff *HostPowerOffConfig
powerOffMu sync.Mutex
// netTrigger performs the mount-waking directory read through the automount path (Q1).
netTrigger func(where string) error
// netMounted judges mount success from /proc/mounts — the §8 truth source (never readability).
@@ -550,6 +554,8 @@ func (s *Server) Handler() http.Handler {
mux.HandleFunc("POST /disks/guest-attach", s.withGuest(s.handleDiskGuestAttach))
// Guest reboot (slice 10 P2 activation): user-triggered restart to activate pending drive binds.
mux.HandleFunc("POST /guest/reboot", s.withGuest(s.handleGuestReboot))
// Host power-off (R-998, doc 14): the household's „A gép kikapcsolása". Guarded in hostpoweroff.go.
mux.HandleFunc("POST /host/poweroff", s.withGuest(s.handleHostPowerOff))
// Guest RAM resize (v0.90.0, R-24): read current allocation + bounds, and apply a bounded resize
// (live cgroup apply, no reboot). Self-scoped; the agent enforces every bound fresh per request.
mux.HandleFunc("GET /guest/memory", s.withGuest(s.handleGuestMemory))
@@ -825,6 +831,10 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
return
}
key := backupJobKey{vmid: vmid, target: tier.TargetID}
// R-899 (operator ruling 2026-10-08): a household press („Mentés most") is not the night's backup. A controller
// that knows sends `trigger=manual`; then the OS leg does not follow (it belongs to the night, after the night's
// own copy). An older controller sends nothing and keeps the old behaviour.
manual := r.URL.Query().Get("trigger") == "manual"
// ONE BACKUP AT A TIME PER GUEST, ACROSS ALL TIERS (operator ruling 2026-07-26: "other backup
// shouldn't start until finished"). vzdump takes a guest lock, so a concurrent second backup
@@ -926,7 +936,9 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
}
s.finishJob(key, jobID, b)
// OS leg (agent v0.140.0): after the night's whole-guest copy exists, still holding the heavy-op gate.
if b.Success && tier.Primary && s.afterPrimaryBackup != nil {
if b.Success && tier.Primary && s.afterPrimaryBackup != nil && manual {
s.logger.Info("local-api: no OS leg after a manual backup — it follows the night's own backup (R-899)", "vmid", vmid, "job", jobID)
} else if b.Success && tier.Primary && s.afterPrimaryBackup != nil {
s.afterPrimaryBackup(base, vmid)
}
}()
+446
View File
@@ -0,0 +1,446 @@
package osupdate
// The kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11). The root half is felhom-os-apply's layer "kernel"
// (configs/, its own tests); this file decides WHEN and judges the boot:
//
// - the night leg (Run → runKernel): after a healthy host step, on a night the hub marks as told (the household was
// mailed the day before — no mail, no step): ring 0 STAGES the pending kernel (select pending-kernel, the root-owned
// ring-0 mark) and reboots; ring 1 reboots only a step a signed os_kernel_step staged earlier (KernelStepExecutor).
// - after a boot (KernelAfterBoot, at daemon start): the wrapper says what became of the step. On the new kernel the
// agent JUDGES the boot — the host health rule (`11` §8.2: the Proxmox daemons, the guest running and healthy, the
// tunnel) AND the box reaching the hub — for KernelJudgeWait. Healthy → kernel-good (the new kernel becomes the
// default). Not healthy by the deadline → ONE self-revert (kernel-revert: a reboot into the old kernel, still the
// default). A crash on the new kernel needs nothing from the agent: GRUB already boots the old default.
//
// The host is rebooted by this file only through the wrapper (kernel-reboot, kernel-revert), and only for a staged step.
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"log/slog"
"regexp"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
)
// OpKernelStep is the signed op class that STAGES a kernel set on a ring-1 box (it never reboots: the night leg does,
// once the household was told). CC may sign it until the first paying customer (R-530 ruling).
const OpKernelStep = "os_kernel_step"
// DefaultKernelJudgeWait is how long a one-shot boot may take to come back healthy before the agent reverts it ONCE.
// Measured 2026-10-07 (`audits/kernel-lane-2026-10-07/B/`): every container healthy 68 s after a reboot on demo-felhom,
// 272 s on demo-hp; 20 minutes stays under the hub's 45-minute host_stale (a box that never comes back alarms after it).
const DefaultKernelJudgeWait = 20 * time.Minute
var kverRE = regexp.MustCompile(`^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$`)
// KernelView is the wrapper's kernel object (felhom-os-apply Kernel.view).
type KernelView struct {
Running string `json:"running"`
Default string `json:"default"`
Flag *string `json:"flag"`
Phase string `json:"phase"`
From string `json:"from"`
To string `json:"to"`
SelfRevertUsed bool `json:"self_revert_used"`
Reason string `json:"reason"`
VMID int `json:"vmid"` // the customer guest the step was staged for (the health rule's guest)
}
// kernelReportRing is the ring an after-boot report carries. In the first second after a boot the agent has not
// fetched the hub's block yet, and Block() then answers ring 1 — so a ring-0 box's „judging" report said ring 1
// (seen on demo-felhom, 2026-10-08 night, `audits/kernel-night-2026-10-07/readback/`). Order: the fetched block; the
// block the daemon saved on disk before the reboot (R-866); ring 1 as before. A label only: the hub's approval reads its
// own ring list. Pinned by TestKernelReportRing_BeforeFirstFetch.
func (l *Leg) kernelReportRing() int {
l.mu.Lock()
fetched := l.block
l.mu.Unlock()
if fetched != nil {
return fetched.Ring
}
if b, _, ok := LoadSavedBlock(l.planDir()); ok && b != nil {
return b.Ring
}
return 1
}
func parseKernel(raw json.RawMessage) KernelView {
var v KernelView
_ = json.Unmarshal(raw, &v)
return v
}
// kernelPlan is one kernel-layer wrapper call.
func kernelPlan(mode string, vmid int, extra map[string]any) map[string]any {
p := map[string]any{"release_id": "kernel", "layer": LayerKernel, "lane": "slow", "vmid": vmid, "mode": mode,
"packages": []Package{}}
for k, v := range extra {
p[k] = v
}
return p
}
// KernelStatus reads the kernel lane's state (read only).
func (l *Leg) KernelStatus(ctx context.Context, vmid int) (KernelView, error) {
wr, err := l.call(ctx, "kstatus"+l.now().UTC().Format("150405"), kernelPlan("kernel-status", vmid, nil))
if err != nil {
return KernelView{}, err
}
if wr.refused() {
return KernelView{}, fmt.Errorf("kernel-status refused: %s", wr.Refused)
}
return parseKernel(wr.Kernel), nil
}
// kernelDue reports whether tonight's leg may take a kernel step, and why not.
func kernelDue(blk hub.WireOSUpdate, trigger string) (bool, string) {
switch {
case trigger != "night":
return false, "a kernel step runs only in the night leg (never a debug pass)"
case blk.Kernel == nil:
return false, "the hub names no kernel step for this box"
case !blk.Enabled:
return false, "OS updates are switched off for this box"
case !kverRE.MatchString(blk.Kernel.Kver):
return false, "the hub's kernel " + blk.Kernel.Kver + " is not a kernel version"
case !blk.Kernel.Tonight:
return false, "the household has not been told about tonight (no mail, no step — `09` §3 decision 172)"
}
return true, ""
}
// runKernel is the night leg's last step. It returns the stage report (Layer "" when nothing ran). On success the box
// is rebooting when it returns.
func (l *Leg) runKernel(ctx context.Context, runID string, vmid int, trigger string, blk hub.WireOSUpdate) Report {
lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid, "trigger", trigger, "ring", blk.Ring)
if ok, why := kernelDue(blk, trigger); !ok {
lg.Info("osupdate: kernel step skipped — " + why)
return Report{}
}
want := blk.Kernel.Kver
st, err := l.KernelStatus(ctx, vmid)
if err != nil {
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid,
Mode: "apply", Outcome: "failed", HealthReason: "kernel status unreadable: " + err.Error()})
}
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid, Mode: "apply", ReleaseID: want}
switch {
case st.Phase == "staged" && st.To == want:
lg.Info("osupdate: kernel step — a staged kernel waits for tonight", "from", st.From, "to", st.To)
rep.Outcome, rep.Healthy = "staged", true
case blk.Ring != 0:
lg.Info("osupdate: kernel step skipped — ring 1 boots only a kernel a signed os_kernel_step staged", "phase", st.Phase, "staged", st.To, "want", want)
return Report{}
default:
// R-898: EXACTLY the kernel the household was told about — never "whatever is pending tonight" (the sources can
// offer a newer one by night; the step then refused, R23, and the night was lost). A version no longer
// installable is refused by the wrapper before any change (R7) and the hub tells the household again.
wr, cerr := l.call(ctx, runID, kernelPlan("apply", vmid, map[string]any{"release_id": "ring0-" + runID,
"select": "listed", "packages": KernelSet(want), "expect_kver": want, "run_id": runID, "trigger": trigger,
"ring": blk.Ring}))
rep.unsent = reportFile(l.planDir(), runID, LayerKernel, "apply")
rep.Kernel = rawOrNil(wr.Kernel)
switch {
case cerr != nil:
rep.Outcome, rep.HealthReason = "failed", cerr.Error()
return l.finish(ctx, lg, rep)
case wr.refused():
rep.Outcome, rep.Refused = "refused", wr.Refused
return l.finish(ctx, lg, rep)
case wr.failed():
rep.Outcome, rep.Refused = "failed", wr.Failed
return l.finish(ctx, lg, rep)
case wr.OutcomeHint == "nothing" || len(wr.Upgraded) == 0:
rep.Outcome, rep.Healthy = "nothing", true
return l.finish(ctx, lg, rep)
}
rep.Outcome, rep.Healthy, rep.Upgraded, rep.Authority, rep.PassSeconds = "staged", true, wr.Upgraded, wr.Authority, wr.PassSeconds
rep.RebootNeeded = true
}
rep = l.finish(ctx, lg, rep) // the hub hears "staged" BEFORE the box goes down
wr, err := l.call(ctx, runID, kernelPlan("kernel-reboot", vmid, nil))
switch {
case err != nil:
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid,
Mode: "kernel-reboot", ReleaseID: want, Outcome: "failed", HealthReason: "kernel-reboot: " + err.Error()})
case wr.refused() || wr.failed():
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid,
Mode: "kernel-reboot", ReleaseID: want, Outcome: "refused", Refused: firstRaw(wr.Refused, wr.Failed),
Kernel: rawOrNil(wr.Kernel)})
}
lg.Warn("osupdate: kernel step — the box restarts now for its one-shot boot", "to", want)
return rep
}
func firstRaw(a, b json.RawMessage) json.RawMessage {
if r := rawOrNil(a); r != nil {
return r
}
return rawOrNil(b)
}
// KernelJudge is what KernelAfterBoot needs besides the leg: the hub reachability probe is the "judging" report itself.
type KernelJudge struct {
Wait time.Duration // default DefaultKernelJudgeWait
Poll time.Duration // default 30 s
}
// KernelAfterBoot runs once at daemon start: what became of a kernel step across the boot. On the new kernel it judges
// the boot (blocking up to the wait — run it in a goroutine). vmid 0 = the guest the step recorded (it may not run yet).
func (l *Leg) KernelAfterBoot(ctx context.Context, vmid int, j KernelJudge) Report {
runID := "boot-" + l.now().UTC().Format("20060102T150405Z")
lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid)
wr, err := l.call(ctx, runID, kernelPlan("kernel-boot", vmid, nil))
if err != nil {
lg.Warn("osupdate: kernel after-boot check failed", "err", err)
return Report{}
}
if wr.refused() {
lg.Info("osupdate: kernel after-boot check refused (an older wrapper, or a BYO host)", "refused", string(wr.Refused))
return Report{}
}
v := parseKernel(wr.Kernel)
if vmid <= 0 {
vmid = v.VMID // after a boot the guest may not run yet — the step's own record names it
}
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid, Mode: "kernel-boot",
ReleaseID: v.To, Kernel: rawOrNil(wr.Kernel)}
switch wr.KernelEvent {
case "fell_back":
rep.Outcome, rep.HealthReason = "fell_back", v.Reason
lg.Warn("osupdate: kernel step FELL BACK — the new kernel did not come up; the box runs the old one", "from", v.From, "to", v.To)
return l.finish(ctx, lg, rep)
case "self_reverted":
rep.Outcome, rep.HealthReason = "self_reverted", v.Reason
lg.Warn("osupdate: kernel step SELF-REVERTED — back on the old kernel", "from", v.From, "to", v.To, "reason", v.Reason)
return l.finish(ctx, lg, rep)
case "revert_failed":
rep.Outcome, rep.HealthReason = "revert_failed", v.Reason
lg.Error("osupdate: kernel self-revert came back on the NEW kernel — no second revert; the operator decides", "to", v.To)
return l.finish(ctx, lg, rep)
case "judging":
return l.judgeKernel(ctx, runID, vmid, v, wr.HealthBefore, j, lg)
}
return Report{}
}
// KernelVerdict is THE one-shot boot rule (R-836; pinned by TestKernelVerdict): the host health rule (`11` §8.2 —
// the Proxmox daemons and the agent active, the customer guest running and its own rule passing, the tunnel running)
// AND the box reached the hub since this boot.
func KernelVerdict(before, after *Health, tunnel string, hubReached bool) (bool, string) {
if ok, why := HostHealthVerdict(before, after, tunnel); !ok {
return false, why
}
if !hubReached {
return false, "the box has not reached the hub since the boot"
}
return true, ""
}
func (l *Leg) judgeKernel(ctx context.Context, runID string, vmid int, v KernelView, before *Health, j KernelJudge, lg *slog.Logger) Report {
wait, poll := j.Wait, j.Poll
if wait <= 0 {
wait = DefaultKernelJudgeWait
}
if poll <= 0 {
poll = 30 * time.Second
}
lg.Info("osupdate: kernel step — judging the one-shot boot", "from", v.From, "to", v.To, "wait", wait.String())
start := l.now()
deadline := start.Add(wait)
hubReached := false
var why string
for {
if !hubReached && l.Hub != nil {
// the hub's reachability IS this report reaching it (and the operator sees the box is back on the new kernel)
body, _ := json.Marshal(Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
Mode: "kernel-boot", ReleaseID: v.To, Outcome: "judging", Kernel: mustRaw(v)})
rctx, cancel := context.WithTimeout(ctx, 30*time.Second)
if err := l.Hub.PostOSReport(rctx, body); err == nil {
hubReached = true
lg.Info("osupdate: kernel step — the box reached the hub on the new kernel", "after", l.now().Sub(start).Round(time.Second).String())
}
cancel()
}
var h *Health
hr, err := l.call(ctx, runID, kernelPlan("health", vmid, nil))
switch {
case err != nil:
why = "no health reading: " + err.Error()
case hr.refused():
why = "no health reading: " + string(hr.Refused)
default:
h = hr.Health
}
if h != nil {
t := hub.TunnelUnknown
if l.Tunnel != nil {
t, _ = l.Tunnel.Status(ctx)
}
var ok bool
ok, why = KernelVerdict(before, h, t, hubReached)
if ok {
return l.kernelGood(ctx, runID, vmid, v, start, lg)
}
}
if !l.now().Before(deadline) || ctx.Err() != nil {
break
}
l.sleep(ctx, poll)
}
if ctx.Err() != nil {
lg.Warn("osupdate: kernel judging stopped (the agent is stopping) — the next start judges again", "reason", why)
return Report{}
}
// not healthy by the deadline: tell the hub (best effort), then ONE self-revert into the old kernel
rep := l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
Mode: "kernel-revert", ReleaseID: v.To, Outcome: "health_failed", HealthReason: why + " — reverting to " + v.From,
Kernel: mustRaw(v)})
lg.Error("osupdate: kernel step — the one-shot boot is NOT healthy; restarting ONCE into the old kernel", "reason", why,
"waited", wait.String(), "from", v.From, "to", v.To)
wr, err := l.call(ctx, runID, kernelPlan("kernel-revert", vmid, map[string]any{"reason": truncate(why, 280)}))
if err != nil || wr.refused() || wr.failed() {
lg.Error("osupdate: kernel self-revert did not start — the box stays on the new kernel; the operator decides",
"err", err, "refused", string(firstRaw(wr.Refused, wr.Failed)))
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
Mode: "kernel-revert", ReleaseID: v.To, Outcome: "revert_failed", Refused: firstRaw(wr.Refused, wr.Failed),
HealthReason: "the self-revert did not start"})
}
return rep
}
func (l *Leg) kernelGood(ctx context.Context, runID string, vmid int, v KernelView, start time.Time, lg *slog.Logger) Report {
wr, err := l.call(ctx, runID, kernelPlan("kernel-good", vmid, nil))
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid, Mode: "kernel-good",
ReleaseID: v.To}
switch {
case err != nil:
rep.Outcome, rep.HealthReason = "failed", "kernel-good: "+err.Error()
case wr.refused() || wr.failed():
rep.Outcome, rep.Refused, rep.HealthReason = "failed", firstRaw(wr.Refused, wr.Failed), "kernel-good did not move the default"
default:
rep.Outcome, rep.Healthy = "applied", true
rep.HealthReason = fmt.Sprintf("healthy %s after the agent started; the new kernel is the default", l.now().Sub(start).Round(time.Second))
}
rep.Kernel = rawOrNil(wr.Kernel)
lg.Info("osupdate: kernel step — "+rep.Outcome, "to", v.To, "reason", rep.HealthReason)
return l.finish(ctx, lg, rep)
}
func mustRaw(v any) json.RawMessage {
b, _ := json.Marshal(v)
return b
}
func truncate(s string, n int) string {
if len(s) <= n {
return s
}
return s[:n]
}
// KernelSet is the package set that installs exactly kver (R-898; the hub's kernelSet, field-exact): the series
// meta-package and the signed image, both at the kernel's own version. Proxmox keeps old kernel versions in its archive.
// nil for a string that is not a kernel version.
func KernelSet(kver string) []Package {
m := kverSeriesRE.FindStringSubmatch(kver)
if m == nil {
return nil
}
v := kver[:len(kver)-len("-pve")]
return []Package{{Name: "proxmox-kernel-" + m[1], Version: v, Origin: PVEOrigin},
{Name: "proxmox-kernel-" + kver + "-signed", Version: v, Origin: PVEOrigin}}
}
var kverSeriesRE = regexp.MustCompile(`^([0-9]+\.[0-9]+)\.[0-9]+-[0-9]+-pve$`)
// KernelStepParams are a signed os_kernel_step's params: the exact kernel set (the wrapper compares it with the plan).
type KernelStepParams struct {
ReleaseID string `json:"release_id"`
Packages []Package `json:"packages"`
Kver string `json:"kver"`
VMID int `json:"vmid,omitempty"`
}
// KernelStepExecutor STAGES a verified os_kernel_step (signedjobs.Executor) under the heavy-op gate. It never reboots:
// the night leg reboots a staged kernel on a night the household was told about.
type KernelStepExecutor struct {
Leg *Leg
Guest func(ctx context.Context) (int, error)
Gate func(ctx context.Context) (release func(), err error)
}
// Execute implements signedjobs.Executor.
func (e KernelStepExecutor) Execute(ctx context.Context, op string, params json.RawMessage) error {
if op != OpKernelStep {
return signedjobs.ErrNoExecutor
}
so, ok := signedjobs.SignedOpFrom(ctx)
if !ok {
return fmt.Errorf("os_kernel_step: no signed envelope in the context — the wrapper could not verify it")
}
var p KernelStepParams
if err := json.Unmarshal(params, &p); err != nil || len(p.Packages) == 0 || !kverRE.MatchString(p.Kver) {
return fmt.Errorf("os_kernel_step: params must name the kernel set and its kver: %v", err)
}
vmid := p.VMID
if vmid == 0 {
if e.Guest == nil {
return fmt.Errorf("os_kernel_step: no vmid and no guest finder")
}
v, err := e.Guest(ctx)
if err != nil {
return fmt.Errorf("os_kernel_step: find the customer guest: %w", err)
}
vmid = v
}
if e.Gate != nil {
release, err := e.Gate(ctx)
if err != nil {
return fmt.Errorf("os_kernel_step: heavy-op gate busy (a backup or restore-test runs): %w", err)
}
defer release()
}
rep := e.Leg.StageKernelSigned(ctx, vmid, p, so.Blob, string(so.Sig))
if rep.Outcome == "staged" || rep.Outcome == "nothing" {
return nil
}
return fmt.Errorf("os_kernel_step: %s (%s) %s", rep.Outcome, rep.HealthReason, string(rep.Refused))
}
// StageKernelSigned stages a signed kernel set (ring 1): install + flag, no reboot.
func (l *Leg) StageKernelSigned(ctx context.Context, vmid int, p KernelStepParams, blob []byte, sig string) Report {
unlock := l.lockPass(true)
defer unlock()
l.sendUnsentLocked(ctx) // R-868
runID := l.now().UTC().Format("20060102T150405Z")
rid := p.ReleaseID
if rid == "" {
rid = "signed-" + runID
}
lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid, "trigger", "signed", "release", rid)
wr, err := l.call(ctx, runID, kernelPlan("apply", vmid, map[string]any{"release_id": rid, "select": "listed",
"packages": p.Packages, "expect_kver": p.Kver, "run_id": runID, "trigger": "signed", "ring": l.Block().Ring,
"signed": map[string]string{"blob_b64": base64.StdEncoding.EncodeToString(blob), "sig": sig}}))
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "signed", Ring: l.Block().Ring, VMID: vmid, Mode: "apply",
ReleaseID: rid, Kernel: rawOrNil(wr.Kernel), unsent: reportFile(l.planDir(), runID, LayerKernel, "apply")}
switch {
case err != nil:
rep.Outcome, rep.HealthReason = "failed", err.Error()
case wr.refused():
rep.Outcome, rep.Refused = "refused", wr.Refused
case wr.failed():
rep.Outcome, rep.Refused = "failed", wr.Failed
case wr.OutcomeHint == "nothing" || len(wr.Upgraded) == 0:
rep.Outcome, rep.Healthy = "nothing", true
default:
rep.Outcome, rep.Healthy, rep.Upgraded, rep.Authority, rep.PassSeconds = "staged", true, wr.Upgraded, wr.Authority, wr.PassSeconds
rep.RebootNeeded = true
}
return l.finish(ctx, lg, rep)
}
+328
View File
@@ -0,0 +1,328 @@
package osupdate
import (
"context"
"encoding/base64"
"encoding/json"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
)
// ---- the kernel lane (R-836, `09` §3 decision 172, `11` §5.11) ----
const kOld, kNew = "7.0.2-6-pve", "7.0.14-22-pve"
func kview(phase string) json.RawMessage {
return mustRaw(KernelView{Running: kOld, Default: kOld, Phase: phase, From: kOld, To: kNew, VMID: 9201})
}
func tonight(ring int) *hub.WireOSUpdate {
return &hub.WireOSUpdate{Ring: ring, Enabled: true, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: true}}
}
func kernelCalls(w *fakeWrapper) []string {
var m []string
for _, p := range w.plans {
if p["layer"] == LayerKernel {
m = append(m, p["mode"].(string))
}
}
return m
}
// Ring 0, a told night: after the healthy host step the leg stages the pending kernel (select pending-kernel, the
// kernel the household was told about), tells the hub "staged", THEN reboots — the kernel step ends the night.
func TestKernel_Ring0ToldNightStagesThenReboots(t *testing.T) {
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{
"kernel-status": {{Kernel: kview("none")}},
"apply": {{Upgraded: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22"}}, Authority: "ring0", Kernel: kview("staged")}},
"kernel-reboot": {{Kernel: kview("oneshot")}},
}}
l, h := newLeg(t, w, tonight(0))
p := l.Run(context.Background(), 9201, "night")
if got := strings.Join(kernelCalls(w), ","); got != "kernel-status,apply,kernel-reboot" {
t.Fatalf("kernel calls = %s", got)
}
var ap map[string]any
for _, x := range w.plans {
if x["layer"] == LayerKernel && x["mode"] == "apply" {
ap = x
}
}
// R-898: EXACTLY the told kernel — the listed set derived from it, never "pending" (red before the fix: select was
// pending-kernel, so a newer kernel in the sources by night was refused R23 and the night was lost)
pk, _ := json.Marshal(ap["packages"])
if ap["select"] != "listed" || ap["expect_kver"] != kNew || ap["lane"] != "slow" ||
string(pk) != `[{"name":"proxmox-kernel-7.0","origin":"Proxmox Debian Repository","version":"7.0.14-22"},{"name":"proxmox-kernel-7.0.14-22-pve-signed","origin":"Proxmox Debian Repository","version":"7.0.14-22"}]` {
t.Fatalf("stage plan = %v (packages %s)", ap, pk)
}
if p.Kernel.Outcome != "staged" || !p.Kernel.Healthy {
t.Fatalf("kernel report = %+v", p.Kernel)
}
last := h.reports[len(h.reports)-1]
if last.Layer != LayerKernel || last.Outcome != "staged" {
t.Fatalf("the hub must hear 'staged' before the reboot: %+v", h.reports)
}
// the kernel step is the LAST wrapper call of the night
if lp := w.plans[len(w.plans)-1]; lp["layer"] != LayerKernel || lp["mode"] != "kernel-reboot" {
t.Fatalf("the reboot must end the night, last call = %v", lp)
}
}
// No mail, no step: a kernel the household was NOT told about never runs; nor in a debug pass; nor without a block.
// COMPANION RED-PROOF (observed): drop the `!blk.Kernel.Tonight` case in kernelDue → the first sub-case fails.
func TestKernel_NoMailNoStep(t *testing.T) {
cases := map[string]struct {
blk *hub.WireOSUpdate
trigger string
}{
"not told": {&hub.WireOSUpdate{Ring: 0, Enabled: true, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: false}}, "night"},
"debug pass": {tonight(0), "debug"},
"no block": {&hub.WireOSUpdate{Ring: 0, Enabled: true}, "night"},
"switch off": {&hub.WireOSUpdate{Ring: 0, Enabled: false, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: true}}, "night"},
"bad kver": {&hub.WireOSUpdate{Ring: 0, Enabled: true, Kernel: &hub.WireKernelStep{Kver: "7.0; reboot", Tonight: true}}, "night"},
}
for name, c := range cases {
w := &fakeWrapper{t: t}
l, _ := newLeg(t, w, c.blk)
p := l.Run(context.Background(), 9201, c.trigger)
if len(kernelCalls(w)) != 0 || p.Kernel.Layer != "" {
t.Fatalf("%s: a kernel step ran: %v", name, kernelCalls(w))
}
}
}
// The kernel step needs a healthy host step on an appliance, and a healthy Proxmox step when one ran.
func TestKernel_SkippedWithoutHealthyEarlierSteps(t *testing.T) {
w := &fakeWrapper{t: t}
l, _ := newLeg(t, w, tonight(0))
l.Appliance = false
l.Run(context.Background(), 9201, "night")
if len(kernelCalls(w)) != 0 {
t.Fatalf("a BYO box took a kernel step: %v", kernelCalls(w))
}
w2 := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerPVE: {Upgraded: []Package{{Name: "pve-manager", Version: "9.2.21"}},
PVEManager: "9.2.2"}}} // pveversion still old → the pve step is unhealthy
l2, _ := newLeg(t, w2, tonight(0))
l2.Run(context.Background(), 9201, "night")
if len(kernelCalls(w2)) != 0 {
t.Fatalf("a kernel step ran after an unhealthy Proxmox step: %v", kernelCalls(w2))
}
}
// Ring 1 reboots only a kernel a signed job staged — never stages one itself in the night leg.
func TestKernel_Ring1RebootsOnlyASignedStage(t *testing.T) {
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}}}}
l, _ := newLeg(t, w, tonight(1))
l.Run(context.Background(), 9201, "night")
if got := strings.Join(kernelCalls(w), ","); got != "kernel-status" {
t.Fatalf("ring 1 without a staged kernel: calls = %s", got)
}
w2 := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("staged")}},
"kernel-reboot": {{Kernel: kview("oneshot")}}}}
l2, _ := newLeg(t, w2, tonight(1))
p := l2.Run(context.Background(), 9201, "night")
if got := strings.Join(kernelCalls(w2), ","); got != "kernel-status,kernel-reboot" || p.Kernel.Outcome != "staged" {
t.Fatalf("ring 1 with a staged kernel: calls = %s report = %+v", got, p.Kernel)
}
}
// A refused stage never reboots.
func TestKernel_RefusedStageNeverReboots(t *testing.T) {
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}},
"apply": {{Refused: json.RawMessage(`{"code":"R20","reason":"/boot/efi is not a mounted vfat ESP"}`)}}}}
l, h := newLeg(t, w, tonight(0))
p := l.Run(context.Background(), 9201, "night")
if got := strings.Join(kernelCalls(w), ","); got != "kernel-status,apply" || p.Kernel.Outcome != "refused" {
t.Fatalf("calls = %s report = %+v", got, p.Kernel)
}
if last := h.reports[len(h.reports)-1]; last.Layer != LayerKernel || last.Outcome != "refused" {
t.Fatalf("the hub must hear the refusal: %+v", last)
}
}
// THE one-shot boot rule: the host rule AND the hub reached. COMPANION RED-PROOF (observed): drop the hubReached
// check in KernelVerdict → the second case fails.
func TestKernelVerdict(t *testing.T) {
if ok, why := KernelVerdict(hostOK(), hostOK(), hub.TunnelRunning, true); !ok {
t.Fatalf("a healthy boot read unhealthy: %s", why)
}
if ok, why := KernelVerdict(hostOK(), hostOK(), hub.TunnelRunning, false); ok || !strings.Contains(why, "hub") {
t.Fatalf("a box that has not reached the hub must not pass: ok=%v %q", ok, why)
}
down := hostOK()
down.GuestRunning = new(bool)
if ok, _ := KernelVerdict(hostOK(), down, hub.TunnelRunning, true); ok {
t.Fatal("a guest that does not run must fail")
}
if ok, _ := KernelVerdict(hostOK(), hostOK(), hub.TunnelUnknown, true); ok {
t.Fatal("an unknown tunnel must fail (the host rule)")
}
}
func judgingLeg(t *testing.T, w *fakeWrapper) (*Leg, *fakeHub) {
if w.kernelRep == nil {
w.kernelRep = map[string][]WrapperReport{}
}
if _, ok := w.kernelRep["kernel-boot"]; !ok {
w.kernelRep["kernel-boot"] = []WrapperReport{{KernelEvent: "judging", Kernel: mustRaw(KernelView{Running: kNew,
Default: kOld, Phase: "judging", From: kOld, To: kNew, VMID: 9201}), HealthBefore: hostOK()}}
}
return newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
}
// A healthy one-shot boot: the hub hears "judging", then kernel-good, then "applied".
func TestKernelAfterBoot_HealthyBecomesTheDefault(t *testing.T) {
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-good": {{Kernel: kview("good")}}}}
l, h := judgingLeg(t, w)
r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 10 * time.Minute, Poll: 30 * time.Second})
if got := strings.Join(kernelCalls(w), ","); got != "kernel-boot,health,kernel-good" {
t.Fatalf("calls = %s", got)
}
if r.Outcome != "applied" || !r.Healthy {
t.Fatalf("report = %+v", r)
}
if len(h.reports) != 2 || h.reports[0].Outcome != "judging" || h.reports[1].Outcome != "applied" {
t.Fatalf("hub reports = %+v", h.reports)
}
if w.plans[1]["vmid"] != float64(9201) {
t.Fatalf("the health reading must use the step's own guest, got %v", w.plans[1]["vmid"])
}
}
// An unhealthy one-shot boot: wait the full judge time, tell the hub, then ONE kernel-revert.
// COMPANION RED-PROOF (observed): return before the kernel-revert call in judgeKernel → "calls" fails.
func TestKernelAfterBoot_UnhealthyRevertsOnceAfterTheWait(t *testing.T) {
down := hostOK()
down.GuestRunning = new(bool)
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"health": {{Health: down}},
"kernel-revert": {{Kernel: kview("reverting")}}}}
l, h := judgingLeg(t, w)
start := l.now()
r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 10 * time.Minute, Poll: time.Minute})
calls := kernelCalls(w)
if calls[len(calls)-1] != "kernel-revert" || strings.Count(strings.Join(calls, ","), "kernel-revert") != 1 {
t.Fatalf("calls = %v", calls)
}
if waited := l.now().Sub(start); waited < 10*time.Minute {
t.Fatalf("reverted after %s — before the judge wait", waited)
}
if r.Outcome != "health_failed" || !strings.Contains(r.HealthReason, "not running") {
t.Fatalf("report = %+v", r)
}
if last := h.reports[len(h.reports)-1]; last.Outcome != "health_failed" {
t.Fatalf("the hub must hear health_failed before the revert reboot: %+v", h.reports)
}
for _, p := range w.plans {
if p["mode"] == "kernel-revert" && !strings.Contains(p["reason"].(string), "not running") {
t.Fatalf("the revert must carry the reason: %v", p)
}
}
}
// A box that never reaches the hub is not "healthy" — it reverts too.
func TestKernelAfterBoot_NoHubMeansRevert(t *testing.T) {
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-revert": {{Kernel: kview("reverting")}}}}
l, _ := judgingLeg(t, w)
l.Hub = unreachableHub{}
l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 5 * time.Minute, Poll: time.Minute})
if c := kernelCalls(w); c[len(c)-1] != "kernel-revert" {
t.Fatalf("calls = %v", c)
}
}
type unreachableHub struct{}
func (unreachableHub) PostOSReport(context.Context, []byte) error { return context.DeadlineExceeded }
// What kernel-boot found becomes the hub's outcome, with no judging and no reboot.
func TestKernelAfterBoot_FallBackAndRevertResultsAreReported(t *testing.T) {
for ev, want := range map[string]string{"fell_back": "fell_back", "self_reverted": "self_reverted", "revert_failed": "revert_failed"} {
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-boot": {{KernelEvent: ev,
Kernel: mustRaw(KernelView{Running: kOld, Default: kOld, Phase: ev, From: kOld, To: kNew, Reason: "r", VMID: 9201})}}}}
l, h := judgingLeg(t, w)
r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{})
if r.Outcome != want || len(h.reports) != 1 || h.reports[0].Outcome != want {
t.Fatalf("%s: report %+v hub %+v", ev, r, h.reports)
}
if got := strings.Join(kernelCalls(w), ","); got != "kernel-boot" {
t.Fatalf("%s: calls = %s", ev, got)
}
}
// nothing to do → nothing reported
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-boot": {{KernelEvent: "none", Kernel: kview("good")}}}}
l, h := judgingLeg(t, w)
if r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{}); r.Layer != "" || len(h.reports) != 0 {
t.Fatalf("an ordinary boot must report nothing: %+v %+v", r, h.reports)
}
}
// The signed executor STAGES (listed + the raw envelope + the kver) and never reboots.
func TestKernelStepExecutor_StagesNeverReboots(t *testing.T) {
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"apply": {{Upgraded: []Package{{Name: "proxmox-kernel-7.0",
Version: "7.0.14-22"}}, Authority: "signed", Kernel: kview("staged")}}}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
e := KernelStepExecutor{Leg: l, Guest: func(context.Context) (int, error) { return 9201, nil }}
params, _ := json.Marshal(KernelStepParams{ReleaseID: "os-kernel-1", Kver: kNew,
Packages: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22", Origin: PVEOrigin}}})
ctx := signedjobs.WithSignedOp(context.Background(), &reconcile.SignedOp{Blob: []byte(`{"op":"os_kernel_step"}`), Sig: []byte("SIG")})
if err := e.Execute(ctx, OpKernelStep, params); err != nil {
t.Fatal(err)
}
pp := w.plans[len(w.plans)-1]
sg, _ := pp["signed"].(map[string]any)
if pp["mode"] != "apply" || pp["select"] != "listed" || pp["expect_kver"] != kNew || sg == nil ||
sg["blob_b64"] != base64.StdEncoding.EncodeToString([]byte(`{"op":"os_kernel_step"}`)) {
t.Fatalf("plan = %v", pp)
}
if got := strings.Join(kernelCalls(w), ","); got != "apply" {
t.Fatalf("a signed stage must never reboot: %s", got)
}
if len(h.reports) != 1 || h.reports[0].Outcome != "staged" {
t.Fatalf("hub = %+v", h.reports)
}
if err := e.Execute(context.Background(), OpKernelStep, params); err == nil {
t.Fatal("no envelope must refuse")
}
bad, _ := json.Marshal(KernelStepParams{Kver: "x", Packages: []Package{{Name: "a"}}})
if err := e.Execute(ctx, OpKernelStep, bad); err == nil {
t.Fatal("a bad kver must refuse")
}
if err := e.Execute(ctx, OpPVEStep, params); err != signedjobs.ErrNoExecutor {
t.Fatalf("another op must pass through the chain: %v", err)
}
}
// os_kernel_step is never benign.
func TestKernelStep_IsDestructiveClass(t *testing.T) {
if reconcile.Classify(reconcile.ClassOSKernelStep, reconcile.Provenance{}) != reconcile.Destructive {
t.Fatal("os_kernel_step must be destructive-class (signed, operational key)")
}
}
// A kept stage report (the agent was killed mid-stage) reaches the hub as "staged" with its kernel view.
func TestKernel_KeptStageReportIsStaged(t *testing.T) {
w := &fakeWrapper{t: t}
l, _ := newLeg(t, w, tonight(0))
ring := 0
rep := l.reportFromKept(context.Background(), WrapperReport{Layer: LayerKernel, Mode: "apply", Ring: &ring,
Upgraded: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22"}}, Kernel: kview("staged")}, "/x/report-r-kernel-apply.json")
if rep.Outcome != "staged" || !rep.Healthy || len(rep.Kernel) == 0 {
t.Fatalf("kept = %+v", rep)
}
}
func TestKernelSet(t *testing.T) {
if got := KernelSet("7.0.14-20-pve"); len(got) != 2 || got[0].Name != "proxmox-kernel-7.0" || got[0].Version != "7.0.14-20" ||
got[1].Name != "proxmox-kernel-7.0.14-20-pve-signed" {
t.Fatalf("%+v", got)
}
if KernelSet("7.0; reboot") != nil || KernelSet("") != nil {
t.Fatal("a non-kernel string must give no set")
}
}
+23 -1
View File
@@ -45,6 +45,9 @@ const (
// LayerPVE is the HOST's Proxmox userspace packages — slow lane (R-812 option A, `09` §3 decision 163, `11` §5.10):
// ring 0 in the night leg after a healthy host step, ring 1 only inside a signed os_pve_step. Never a kernel.
LayerPVE = "pve"
// LayerKernel is the HOST's kernel — the kernel lane (R-836, `09` §3 decision 172, `11` §5.11): a one-shot boot of
// the new kernel through the ESP flag, the default moved only after a healthy boot (kernel.go).
LayerKernel = "kernel"
)
// PVEOrigin is the origin apt prints for download.proxmox.com (the wrapper's PVE_ORIGIN); InstalledPVEOrigin is how
@@ -126,6 +129,11 @@ type WrapperReport struct {
OOMCheck json.RawMessage `json:"oom_check"`
// PVEManager: the pve layer — pveversion's pve-manager version after the step ("unknown" = unreadable).
PVEManager string `json:"pve_manager"`
// Kernel (R-836): the kernel layer's view {running, default, flag, phase, from, to, …}; KernelEvent what kernel-boot
// found after a boot; OutcomeHint "nothing" when no kernel was pending.
Kernel json.RawMessage `json:"kernel"`
KernelEvent string `json:"kernel_event"`
OutcomeHint string `json:"outcome_hint"`
// R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the
// agent process that started the pass. ReleaseID / VMID were always in the report.
RunID string `json:"run_id"`
@@ -168,6 +176,10 @@ type Report struct {
OOMCheck json.RawMessage `json:"oom_check,omitempty"`
// PVEManager: pve layer — pve-manager's version after the step (the hub's System page; R-812 option A).
PVEManager string `json:"pve_manager,omitempty"`
// Kernel: kernel layer — the wrapper's kernel view, byte for byte (running, default, flag, phase, from, to). The
// outcomes of this layer: staged | applied (the new kernel is the default) | fell_back | health_failed (self-revert
// started) | self_reverted | revert_failed | judging | nothing | refused | failed.
Kernel json.RawMessage `json:"kernel,omitempty"`
unsent string // R-868: the wrapper's kept copy of this pass's report — deleted once the hub has it
}
@@ -500,7 +512,7 @@ func (l *Leg) call(ctx context.Context, runID string, plan map[string]any) (Wrap
// Pass is one leg's reports; an empty Layer means the step did not run.
type Pass struct {
Guest, Host, Docker, PVE Report
Guest, Host, Docker, PVE, Kernel Report
}
// Run is one pass: the guest layer, then (on an appliance, after a good guest step) the host layer, then (ring 0
@@ -542,6 +554,16 @@ func (l *Leg) Run(ctx context.Context, vmid int, trigger string) Pass {
default:
p.PVE = l.runPVE(ctx, g.RunID, vmid, trigger, blk, dockerOpts{})
}
// R-836 (`09` §3 decision 172): the kernel step ENDS the night — an appliance, after a healthy host step (and a
// healthy Proxmox step when one ran), only on a night the hub marks as told. It reboots the box. Pinned by TestKernel_*.
switch {
case !l.Appliance || h.Layer == "" || !okStep(h):
lg.Info("osupdate: kernel step skipped — no healthy host step this pass (an appliance only)", "appliance", l.Appliance, "host_outcome", h.Outcome)
case p.PVE.Layer != "" && !okStep(p.PVE):
lg.Warn("osupdate: kernel step skipped — the Proxmox step did not end healthy", "pve_outcome", p.PVE.Outcome)
default:
p.Kernel = l.runKernel(ctx, g.RunID, vmid, trigger, blk)
}
return p
}
+12 -1
View File
@@ -25,6 +25,7 @@ type fakeWrapper struct {
plans []map[string]any
keep bool // R-868: like the real wrapper, keep an apply report beside the plan
pveGateHeld bool
kernelRep map[string][]WrapperReport // R-836: per kernel-layer mode, successive answers (the last one repeats)
}
func yes() *bool { b := true; return &b }
@@ -52,9 +53,19 @@ func (f *fakeWrapper) Run(_ context.Context, name string, args ...string) ([]byt
f.plans = append(f.plans, plan)
layer := plan["layer"].(string)
ok := guestOK()
if layer == LayerHost || layer == LayerPVE {
if layer == LayerHost || layer == LayerPVE || layer == LayerKernel {
ok = hostOK()
}
if layer == LayerKernel {
if seq := f.kernelRep[plan["mode"].(string)]; len(seq) > 0 {
rep := seq[0]
if len(seq) > 1 {
f.kernelRep[plan["mode"].(string)] = seq[1:]
}
out, _ := json.Marshal(rep)
return []byte("OSAPPLY-REPORT " + string(out) + "\n"), []byte("os-apply: DONE rc=0\n"), nil
}
}
if layer == LayerPVE && plan["mode"] == "apply" {
f.pveGateHeld = pvegate.Stepping() // R-812: the /etc/pve write gate must be held while the pve step runs
}
+32
View File
@@ -0,0 +1,32 @@
package osupdate
import (
"os"
"path/filepath"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// After a boot the agent has not fetched the hub's block yet; the after-boot kernel report must still carry the
// box's real ring (seen 2026-10-08: a ring-0 box's „judging" report said ring 1). Red-proof: return Block().Ring
// from kernelReportRing and the saved-block case fails with 1.
func TestKernelReportRing_BeforeFirstFetch(t *testing.T) {
dir := t.TempDir()
l := &Leg{PlanDir: dir}
if got := l.kernelReportRing(); got != 1 {
t.Fatalf("nothing fetched, nothing saved: ring %d, want 1 (the old default)", got)
}
l.saveBlock(&hub.WireOSUpdate{Ring: 0, Enabled: true})
l2 := &Leg{PlanDir: dir} // a fresh daemon after the reboot: no block fetched yet
if got := l2.kernelReportRing(); got != 0 {
t.Fatalf("ring-0 block saved before the reboot: ring %d, want 0", got)
}
l2.SetBlock(&hub.WireOSUpdate{Ring: 1, Enabled: true})
if got := l2.kernelReportRing(); got != 1 {
t.Fatalf("a fetched block wins over the saved one: ring %d, want 1", got)
}
if _, err := os.Stat(filepath.Join(dir, SavedBlockFile)); err != nil {
t.Fatalf("saved block missing: %v", err)
}
}
+11
View File
@@ -142,6 +142,17 @@ func (l *Leg) reportFromKept(ctx context.Context, wr WrapperReport, path string)
default:
rep.Outcome = "applied"
}
if wr.Layer == LayerKernel {
// a kernel STAGE changes nothing the box runs (the new kernel only boots once, at the night's reboot), so its
// kept copy needs no fresh health reading (R-836)
rep.Kernel = rawOrNil(wr.Kernel)
rep.Upgraded, rep.PassSeconds, rep.Authority = wr.Upgraded, wr.PassSeconds, wr.Authority
if rep.Outcome == "applied" {
rep.Outcome = "staged"
}
rep.Healthy, rep.HealthReason = rep.Outcome == "staged" || rep.Outcome == "nothing", prefix
return rep
}
rep.Upgraded, rep.PassSeconds = wr.Upgraded, wr.PassSeconds
rep.DockerEngine, rep.Authority, rep.Undo = wr.DockerEngine, wr.Authority, wr.Undo
rep.OOMCheck = rawOrNil(wr.OOMCheck)
+117
View File
@@ -0,0 +1,117 @@
// Package poweroff keeps the box's record of a household-requested power-off (R-998, doc 14).
//
// One small JSON file in the agent's state dir. A record belongs to the boot it was made in (the
// kernel's btime from /proc/stat), so:
// - "at most one accepted request per boot": Mark refuses when this boot already has a record;
// - the host report carries the record only while the box is still in that boot (Current), and
// the next boot's reports carry nothing — that absence is how the hub learns the box is back.
//
// Pinned by state_test.go.
package poweroff
import (
"encoding/json"
"errors"
"fmt"
"os"
"path/filepath"
"strings"
"sync"
"time"
)
// ErrAlreadyThisBoot is Mark's refusal: a power-off was already accepted in the current boot.
var ErrAlreadyThisBoot = errors.New("a power-off was already accepted in this boot")
// ErrBootUnknown is Mark's refusal when the boot cannot be identified — fail closed: without it the
// per-boot limit could not hold.
var ErrBootUnknown = errors.New("cannot read the host's boot time")
// Record is one accepted request.
type Record struct {
RequestedAt time.Time `json:"requested_at"`
BootBtime string `json:"boot_btime"` // /proc/stat btime of the boot it was accepted in
VMID int `json:"vmid"` // the guest whose controller asked
}
// State is the persisted record.
type State struct {
path string
btime func() string
mu sync.Mutex
}
// New returns the state at path. btime returns the current boot's /proc/stat btime ("" = unknown).
func New(path string, btime func() string) *State {
if btime == nil {
btime = ProcBtime
}
return &State{path: path, btime: btime}
}
// Current is the record made in THIS boot, or nil (none, another boot's, unreadable).
func (s *State) Current() *Record {
if s == nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
r := s.load()
b := s.btime()
if r == nil || b == "" || r.BootBtime != b {
return nil
}
return r
}
// Mark records an accepted request for this boot. It refuses a second one in the same boot.
func (s *State) Mark(now time.Time, vmid int) (*Record, error) {
s.mu.Lock()
defer s.mu.Unlock()
b := s.btime()
if b == "" {
return nil, ErrBootUnknown
}
if r := s.load(); r != nil && r.BootBtime == b {
return nil, ErrAlreadyThisBoot
}
r := &Record{RequestedAt: now.UTC().Truncate(time.Second), BootBtime: b, VMID: vmid}
data, _ := json.Marshal(r)
if err := os.MkdirAll(filepath.Dir(s.path), 0o750); err != nil {
return nil, fmt.Errorf("poweroff: state dir: %w", err)
}
tmp := s.path + ".tmp"
if err := os.WriteFile(tmp, data, 0o600); err != nil {
return nil, fmt.Errorf("poweroff: write: %w", err)
}
if err := os.Rename(tmp, s.path); err != nil {
return nil, fmt.Errorf("poweroff: rename: %w", err)
}
return r, nil
}
func (s *State) load() *Record {
data, err := os.ReadFile(s.path)
if err != nil {
return nil
}
var r Record
if json.Unmarshal(data, &r) != nil || r.BootBtime == "" {
return nil
}
return &r
}
// ProcBtime reads the boot time (seconds since the epoch, as text) from /proc/stat; "" when unreadable.
func ProcBtime() string {
data, err := os.ReadFile("/proc/stat")
if err != nil {
return ""
}
for _, line := range strings.Split(string(data), "\n") {
if strings.HasPrefix(line, "btime ") {
return strings.TrimSpace(strings.TrimPrefix(line, "btime "))
}
}
return ""
}
+41
View File
@@ -0,0 +1,41 @@
package poweroff
import (
"errors"
"path/filepath"
"testing"
"time"
)
// One accepted request per boot; the record is "current" only in the boot it was made in.
func TestOnePerBootAndCurrentFollowsTheBoot(t *testing.T) {
boot := "1000"
s := New(filepath.Join(t.TempDir(), "poweroff.json"), func() string { return boot })
if s.Current() != nil {
t.Fatal("no record yet, Current must be nil")
}
now := time.Date(2026, 10, 13, 18, 0, 0, 0, time.UTC)
if _, err := s.Mark(now, 9201); err != nil {
t.Fatalf("first Mark: %v", err)
}
if r := s.Current(); r == nil || !r.RequestedAt.Equal(now) || r.VMID != 9201 {
t.Fatalf("Current after Mark = %+v", r)
}
if _, err := s.Mark(now.Add(time.Minute), 9201); !errors.Is(err, ErrAlreadyThisBoot) {
t.Fatalf("second Mark in the same boot: err = %v, want ErrAlreadyThisBoot", err)
}
boot = "2000" // the machine was switched on again
if s.Current() != nil {
t.Fatal("a record from the previous boot must not be current")
}
if _, err := s.Mark(now.Add(time.Hour), 9201); err != nil {
t.Fatalf("Mark in a new boot: %v", err)
}
}
func TestUnknownBootFailsClosed(t *testing.T) {
s := New(filepath.Join(t.TempDir(), "p.json"), func() string { return "" })
if _, err := s.Mark(time.Now(), 1); !errors.Is(err, ErrBootUnknown) {
t.Fatalf("err = %v, want ErrBootUnknown", err)
}
}
+6 -1
View File
@@ -56,6 +56,11 @@ const (
// key) like os_docker_step; the root wrapper re-verifies the same signature itself.
ClassOSPVEStep OpClass = "os_pve_step"
// A kernel step on the host (R-836, `09` §3 decision 172, `11` §5.11) — ring 1: it STAGES a kernel (install + the
// one-shot flag; the night leg reboots it). Destructive-class (signed, operational key) like os_pve_step; the root
// wrapper re-verifies the same signature itself.
ClassOSKernelStep OpClass = "os_kernel_step"
// The config bundle (agent v0.143.0, R-840, `11` §5.4.2): the box's ROOT-OWNED files (sudoers, wrappers, units).
// Destructive-class (signed, operational key) like agent_update; the root wrapper re-verifies the signature itself.
ClassAgentConfigUpdate OpClass = "agent_config_update"
@@ -127,7 +132,7 @@ func Classify(class OpClass, prov Provenance) Disposition {
return Destructive
case ClassKeyRotation:
return Destructive
case ClassAgentUpdate, ClassOSDockerStep, ClassOSPVEStep, ClassAgentConfigUpdate:
case ClassAgentUpdate, ClassOSDockerStep, ClassOSPVEStep, ClassOSKernelStep, ClassAgentConfigUpdate:
// Never benign — no agent-internal provenance can make replacing the agent binary
// unsigned-safe (a compromised process must not be able to self-bless an update).
return Destructive