Compare commits

...

18 Commits

Author SHA1 Message Date
admin db25b469ba Host report keeps the last backup across a restart; Secure Boot meta-package leaves the Proxmox lane
gates / gates (push) Successful in 1m7s
Found in the 2026-10-09 kernel-night read-back:
- demo-felhom: the kernel step restarted the host 4 minutes after the night
  backup; the in-memory backup list was empty after the restart, so the hub
  alarmed "host tier: newest backup is 48h old". The report now adds R-894's
  saved newest success per tier (one shared instance).
- demo-hp: proxmox-secure-boot-support pulled shim-signed-common into the
  ring-0 Proxmox plan; the step was refused R6 and the kernel step skipped.
  It joins HOST_SLOW_RE with shim and GRUB.

Red-proved: TestCollectBackups_SavedSuccessSurvivesARestart,
TestR894_LastKnownBackupsIsWiredIntoTheDaemon,
test_ring0_pending_pve_leaves_the_secure_boot_meta_out.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-09 06:47:58 +02:00
admin b228b44f0e REPORT/CONTEXT: 2026-10-08 day (R-899, R-304 on main, unreleased)
gates / gates (push) Successful in 52s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-08 08:08:53 +02:00
admin 91b9405c81 R-304: no 'wrong code' when earlier sealed packages were not all checked (424 older_unchecked)
gates / gates (push) Successful in 55s
Unreleased; ships with tomorrow's release.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-08 07:53:01 +02:00
admin 4c69c25b48 R-899: no OS leg after a household press (trigger=manual); after-boot kernel reports carry the saved ring
gates / gates (push) Successful in 50s
Unreleased; ships with tomorrow's release.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-08 07:34:19 +02:00
admin c9013bb47d CHANGELOG/REPORT: v0.153.0 released and delivered
gates / gates (push) Successful in 1m33s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 19:06:04 +02:00
admin 2d1e5d0774 R-898: ring 0 stages exactly the told kernel (select listed, KernelSet(kver))
gates / gates (push) Successful in 1m6s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 18:45:28 +02:00
admin f09c53efc1 REPORT/CONTEXT: v0.152.0 the kernel lane
gates / gates (push) Successful in 1m9s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 18:20:11 +02:00
admin 92d647a6f6 CHANGELOG: v0.152.0 released (shas, step bundle); host_stale is 45 min (comment fix)
gates / gates (push) Successful in 1m0s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 15:42:26 +02:00
admin d03ab7f1f5 R-836: the kernel lane — one-shot boot through the ESP flag, boot good / one self-revert, night step on a told night
gates / gates (push) Successful in 44s
Wrapper layer kernel (stage / reboot / boot / good / revert / cancel / status;
R20-R23), the two GRUB generators in the bundle (option C on the one-shot
entry), the agent's night step and after-boot judge (host health rule + hub
reached, 20 min measured), the signed os_kernel_step (stage only).
Red-proofs: felhom.eu audits/kernel-lane-2026-10-07/A/redproof.txt.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 15:18:24 +02:00
admin b17d1c597d REPORT/CHANGELOG: 2026-10-07 day
gates / gates (push) Successful in 59s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 14:21:01 +02:00
admin 0ae01dfb52 CHANGELOG: v0.151.0 released (R-861, R-812 A, R-366, R-105)
gates / gates (push) Successful in 58s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 13:14:47 +02:00
admin dd7cdc09e7 R-366 slice 2 (decision 168): the host report carries the archives the restore-test skipped as another key's
gates / gates (push) Successful in 1m6s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 10:19:38 +02:00
admin 7b0a8b234b R-105 (decision 169): remove the escrow-create -directive flag and the upload's directive field
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 10:19:38 +02:00
admin ce1a4b4758 R-812 option A: the Proxmox package lane (layer pve) + the /etc/pve write gate
The wrapper gains layer "pve" (slow lane): the host's Proxmox userspace
packages only — origin "Proxmox Debian Repository", never a kernel / boot /
firmware / microcode name (R14), no removal, no undo, a new package only from
an allow-list; authority = a signed os_pve_step or the root-owned ring-0 mark.
The night leg runs it in ring 0 after a healthy host step; ring 1 only by a
signed job (PVEStepExecutor). While it runs, the agent's own /etc/pve writes
(every non-GET API call, pct config verbs, pvesm, pveum, felhom-pbs-apply)
wait on internal/pvegate. Health = the host rule + unchanged container ids +
pveversion reads the installed pve-manager.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 10:19:38 +02:00
admin ac90169a5d R-861 (a) A1 + (b) B2: the image ref goes to a root verb that checks it; the agent's in-guest tee grant is gone; felhom-op's pct lines are exact (09 §3 decision 165)
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 10:19:28 +02:00
admin 154d6dcaa9 Shared rule file: no hub image build or deploy in a session the operator does not attend (09 §3 decision 162)
gates / gates (push) Successful in 56s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 09:55:02 +02:00
admin 2f7072050f REPORT: released and delivered (2026-10-07)
gates / gates (push) Successful in 1m4s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 09:31:58 +02:00
admin 85e799f360 CHANGELOG: v0.150.0 released (R-528, R-894, R-330)
gates / gates (push) Successful in 46s
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_0159rPz1ZhFKsS53msqPYxtS
2026-10-07 08:59:08 +02:00
54 changed files with 4058 additions and 215 deletions
+2 -1
View File
@@ -23,7 +23,8 @@ unconditional: true
customer data; anything that changes a promise the product makes to a customer; anything that
reverses a documented design decision (`documentation/architecture/` — a design decision is not a
defect, R-370); anything on DooPlex or ep0; baking or vouching a golden; promoting a
catalog version; a new external dependency.
catalog version; a new external dependency; **a hub image build or hub deploy in a session the operator does not
attend** (operator ruling 2026-10-07, `09` §3 decision 162).
## 2. When you may decide instead of ask (operator grant, 2026-09-14)
+157 -4
View File
@@ -1,4 +1,157 @@
## Unreleased (2026-10-06 night, later) — after a restart the agent remembers the last backup per tier (R-894); three more SMART counters on the wire (R-330)
## Unreleased (2026-10-08) — no OS leg after a household press; the after-boot kernel report carries the real ring (R-899); no „wrong code" when older packages were not checked (R-304); the last backup survives a restart in the host report; the Secure Boot meta-package leaves the Proxmox lane
**Delivery: the agent binary AND the config bundle** — the wrapper `felhom-os-apply` changed (2026-10-09).
- **The hub's backup evidence survives a restart (2026-10-09, found in the kernel-night read-back).** The host report's
`backups` list came only from memory, which is empty after an agent restart. On demo-felhom the night backup landed
02:40 UTC and the kernel step restarted the host at 02:44 — no report fell in between, two nights running — so the hub
alarmed „host tier: newest backup is 48h old" at 03:00 after a good backup. The report now adds R-894's saved newest
success per tier and guest (`backup-success-state.json`, ONE instance shared with the local API) where memory holds no
success at or after it. Only successes are saved, so a failure is never hidden. `hub.KnownBackupReporter`,
`BackupSuccessState.KnownBackupSuccesses`; tests `TestCollectBackups_SavedSuccessSurvivesARestart` (red-proved: the
restart case reported nothing), `TestBackupSuccessState_KnownBackupSuccessesSurviveAReopen`, and
`TestR894_LastKnownBackupsIsWiredIntoTheDaemon` now also asserts the collector gets the same instance (red-proved).
- **`proxmox-secure-boot-support` is a boot-chain package (2026-10-09).** On demo-hp (Secure Boot on) its upgrade pulled
`shim-signed-common`; in the ring-0 Proxmox plan that refused the whole step (R6) and skipped the night's kernel step,
after the household had been told „tonight". It joins `HOST_SLOW_RE` with shim and GRUB (they stay out of every lane,
as before). `test_ring0_pending_pve_leaves_the_secure_boot_meta_out` (red-proved).
- **R-899 (operator ruling 2026-10-08, option A):** `POST /backup?trigger=manual` (a „Mentés most" press, sent by the
controller from its next release) runs no OS leg after it: the leg belongs to the night, after the night's own copy.
Before, a press ran the leg at once (in the day; on 2026-10-07 08:51 demo-hp skipped it only by the 20-hour rule). A
request without the parameter (an older controller, the scheduled path) behaves as before.
`internal/localapi/server.go`; `TestAfterPrimaryBackup` gained two sub-cases (red-proved: ignoring `trigger`, the leg
ran once after a press).
- **R-304 — „wrong code" only when every earlier package was tried.** `POST /escrow/recover-offsite-password` answered
400 („the recovery code did not open the sealed bundle") whenever the current package and the TRIED earlier packages
refused the code — even when the hub withheld earlier packages (rows with no key material, rows over its serve cap), a
served package was malformed, the 6-attempt cap stopped the loop, or the retained list could not be read at all. Now
those cases answer **424** with `older_unchecked` (the count, -1 = unknown) and a sentence that does not call the code
wrong. `escrow.ErrRetainedUnchecked` / `RetainedUncheckedError`; the retained fetcher's second value is now every
withheld package (unopenable + truncated + malformed). Tests `TestR304_*` (real age crypto; red-proved: without the
check, four cases returned the wrong-code error) and a 424 row in `TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus`.
An older controller maps the unknown 424 to its neutral „we do not know why" sentence.
- **The „ring 1" label after a boot:** in the first second after a reboot the agent has not fetched the hub's block, and
the after-boot kernel reports (`judging`, `good`, `revert`, `fell_back`…) said ring 1 on a ring-0 box (demo-felhom,
2026-10-08 night). Now they read the fetched block, else the block the daemon saved on disk before the reboot (R-866),
else ring 1 as before. A label only — the hub's approval reads its own ring list. `kernelReportRing`,
`TestKernelReportRing_BeforeFirstFetch` (red-proved: ring 1, want 0).
## v0.153.0 — ring 0 stages exactly the told kernel (R-898; `09` §3 decision 176) (2026-10-07)
Released by `scripts/release-agent.sh`: binary sha256 `b204ebe6d65944f43b70dcfa4ae0dfb90da38c92e2ba6a38a1826e488eeb6630`, bundle
`6db216275eb0dd39188d93481a2045998a69e8cb7838ad908a58d65d9a6a9b57` (tag `v0.153.0` = `2d1e5d0`). Delivered (binary only — the
bundle files are unchanged from 0.152.0) to demo-hp, demo-felhom, Tester 1 on 2026-10-07 19:03.
**Delivery: the agent binary only** — no root file changed (the wrapper is unchanged; its tests gained two cases).
- `internal/osupdate/kernel.go`: ring 0's night kernel step stages EXACTLY the kernel the household was told about
(select `listed`, `KernelSet(kver)` = the series meta-package and the signed image at the kernel's own version) instead
of "whatever is pending tonight". Seen 2026-10-07: demo-felhom was told about 7.0.14-20 while its sources offered
7.0.14-22 by night — the old code staged `pending-kernel` and the wrapper refused it (R23), losing the night. A told
version that is no longer installable is refused by the wrapper before any change (R7) and the hub tells the household
again for the newer kernel (hub v0.143.1). Tests `TestKernel_Ring0ToldNightStagesThenReboots` (red-proved against the
old select), `TestKernelSet`; wrapper `test_ring0_listed_installs_the_told_kernel_not_the_newest`,
`test_ring0_told_kernel_gone_is_refused_before_any_change`.
## v0.152.0 — the kernel lane (R-836; `09` §3 decisions 164, 172; `11` §5.11) (2026-10-07)
Released by `scripts/release-agent.sh`: binary sha256 `95ff42208e36ba49b6e2b09a97e81a6fa11562ecc8042f1ed18d378b8b1f88b1`,
config bundle sha256 `f0c2cec374b711b3c131c955012b33fd0ad495337d049b7a743c3eef9e85c20b` (tag `v0.152.0` = `d03ab7f`).
Step bundle `0.152.0-step1` sha256 `0b71d32b054cf3b7ade0234ffcbb0df159901f542cde540adaee411db466f48e` (the 0.151.0 bundle
with only `felhom-os-apply` replaced; `scripts/build-step-bundle.py`), published as package version `0.152.0-step1`.
**Delivery: agent binary, then the STEP bundle `0.152.0-step1`, then the bundle `0.152.0`** — the bundle ADDS two paths
(the GRUB generators), and an installed `felhom-os-apply` refuses a path its own table lacks (R16, R-880).
A new kernel boots ONCE; if it crashes the box comes back on the old kernel by itself; it becomes the default only after
a healthy boot; a booted-but-unhealthy kernel is reverted ONCE by the agent with no person. Built on the spike's
candidate 2 (`audits/kernel-spike-2026-10-07/`), with option C on the one-shot entry.
- `configs/felhom-grub-oneshot.sh` → `/etc/grub.d/01_felhom_oneshot` (bundle): reads `felhom_next` from a GRUB env block
on the ESP (`EFI/felhom/oneshot.env`), clears and saves it BEFORE the menu, and sets the default to that kernel's
one-shot entry only when the name is an installed kernel. No vfat ESP → prints nothing.
- `configs/felhom-grub-oneshot-entries.sh` → `/etc/grub.d/42_felhom_oneshot` (bundle): one entry per installed kernel,
id `felhom-oneshot-<ver>`, the normal entry plus `softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10`
(option C). Sorted after `10_linux`: never entry 0, never the default.
- `configs/felhom-os-apply`: layer `kernel` (lane slow; an appliance; authority = a signed `os_kernel_step` or the
root-owned ring-0 mark). Modes: `apply` STAGES (select `pending-kernel` or a signed `listed` set; `expect_kver` = the
kernel the household was told about): pins the GRUB default to the RUNNING kernel in
`/etc/default/grub.d/zz-felhom-kernel-default.cfg` and proves it from grub.cfg, installs, proves the default did not
move and the one-shot entry exists, writes the flag; never reboots. `kernel-reboot` (a staged step only),
`kernel-boot` (judging | fell_back | self_reverted | revert_failed), `kernel-good` (the new kernel becomes the
default, proved), `kernel-revert` (ONE per step; refused when the default is not the old kernel), `kernel-cancel`,
`kernel-status`. New refusals: R20 (the box cannot do a one-shot: not UEFI, no vfat ESP, a separate /boot, GRUB
without fat/loadenv, the generators missing, a hand pin), R21 (the crash guard tripped or an unclean boot in its
window), R22 (the phase does not allow the mode; never two steps within 20 h), R23 (not exactly one newer kernel, or
not the one signed / told). State `/var/lib/felhom-kernel/state.json`. Facts carry `kernel_lane`; the next-boot
kernel reads the flag and the grub.cfg default. Tests: `KernelLane` (27), red-proof
`felhom.eu/documentation/audits/kernel-lane-2026-10-07/A/redproof.txt`.
- `configs/felhom-crash-guard` unchanged; `KernelStepCannotLeaveTheBoxOff` (3 tests) pins that a step's planned reboot,
one crash and one self-revert add ONE unclean boot (a panic before userspace adds none), so the box cannot stay off.
- `internal/osupdate/kernel.go`: the night leg ends with the kernel step — after a healthy host step (and a healthy
Proxmox step when one ran), trigger `night` only, on a night the hub's `os_update.kernel` block marks `tonight` (the
household was mailed the day before — no mail, no step). Ring 0 stages + reboots; ring 1 reboots only a kernel a
signed `os_kernel_step` staged (`KernelStepExecutor`: stage only, under the heavy-op gate). The hub hears `staged`
BEFORE the reboot. At every start `KernelAfterBoot`: on the new kernel it JUDGES the boot — `KernelVerdict` = the
host health rule (`11` §8.2) AND the box reached the hub (the `judging` report itself) — for 20 minutes (measured:
everything healthy 68 s after the reboot on demo-felhom, 272 s on demo-hp; under the hub's 45-minute `host_stale` (`alerting.stale_threshold`)).
Healthy → `kernel-good`, outcome `applied`; not healthy → outcome `health_failed`, then ONE `kernel-revert`.
Tests: `TestKernel*` (13); red-proofs in the same file.
- `internal/hub`: `WireOSUpdate.Kernel` {kver, tonight, notified_at}. `internal/reconcile`: `os_kernel_step` is
destructive-class. `cmd/felhom-opsign`: the op is listed.
## v0.151.0 — the agent can no longer hand the guest any image; the Proxmox package lane; the other-key archives reported; the DR directive retired (R-861, R-812 A, R-366, R-105; `09` §3 163, 165, 168, 169) (2026-10-07)
Released by `scripts/release-agent.sh`: binary sha256 `0464354f2cdf452a7c5d2a74d9191fe91415fcfa244154480d26d5b30e10b194`
config bundle sha256 `bacd1d175a9392bc1755341d01a106abbd72aba48ab575e7a32dd819d8f5da4c` (tag `v0.151.0` = `dd7cdc0`).
**Deliver the binary FIRST, then the bundle:** the bundle's sudoers removes the `tee` grant the 0.150.0 binary still uses.
No path added (26 → 26), so no step bundle.
### Part of v0.151.0 — the agent can no longer hand the guest any image; felhom-op's pct lines are exact (R-861 (a) A1, (b) B2; `09` §3 decision 165)
**Delivery order: agent binary FIRST, then the config bundle.** The new sudoers drops the agent's in-guest `tee`
grant; an older binary still calls `tee`, so a bundle that lands before the binary would stop managed controller
updates (and the old binary's capability probe would read `controllerswap-write` degraded). No bundle path is added
(`felhom-priv-apply` and `/etc/sudoers.d/felhom-op` are already bundle files), so no step bundle.
- `configs/felhom-priv-apply`: new verb `controller-image <vmid>` — reads the ref on stdin (≤ 256 bytes, ASCII, one
optional trailing newline), requires `^gitea\.dooplex\.hu/admin/felhom-controller:[0-9]+\.[0-9]+\.[0-9]+$` (the agent's
own `controllerImageRe`), then runs `pct exec <vmid> -- tee /etc/felhom-controller-image` AS ROOT; refusal rule `I1`
(rc 3), a bad vmid `A1` (rc 2); listed in `--self-check`.
- `configs/felhom-agent.sudoers` `FELHOM_CONTROLLERSWAP`: `pct ^exec [0-9]+ -- tee /etc/felhom-controller-image$`
REMOVED; `/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$` added.
- `internal/localapi`: `GuestExecutor.GuestExecStdin` replaced by `WriteControllerImage`; `GuestBinder.WriteControllerImage`
pipes `ref\n` to the verb through the fenced runner; the swap's `writeImage` calls it. Capability `controllerswap-write`
now probes the verb.
- `configs/felhom-op.sudoers` (B2, hygiene): `pct start|stop|unlock [0-9]*` → `pct ^start [0-9]+$` etc. (the glob's `*`
matched spaces: `pct stop 9201 --skiplock 1` passed).
- Tests: `ControllerImage` (5, `configs/test_felhom_priv_apply.py`), `TestSudoersRefusesTheR861Injections` (+3 lines),
`TestSudoersAllowsTheControllerImageVerb`, `TestFelhomOpSudoersPctIsExact`, `TestR861_WriteControllerImageUsesTheRootVerb`,
`TestControllerSwap_WriteViaRootVerb_NoShell`. Red-proofs: `felhom.eu/documentation/audits/day-2026-10-07/C/`.
- `README.md`: the controller-swap paragraph described the removed `tee` path — corrected.
### Part of v0.151.0 — the Proxmox package lane (R-812 option A, `09` §3 decision 163)
**MinAgent impact: none** (a new layer; an older hub ignores the pve report). **The bundle carries the new
`felhom-os-apply` — deliver it with the binary** (signed `agent_update`, then signed `agent_config_update`).
- `configs/felhom-os-apply`: new layer `pve`, lane `slow` only — the host's Proxmox USERSPACE packages: origin `Proxmox Debian Repository` only (R2), never a kernel / boot / firmware / microcode name (R14, `HOST_SLOW_RE` — the kernel is R-836's lane), no removal (R4), no undo (R5), a new package only from `PVE_NEW_ALLOW` (`proxmox-firewall-data`, measured on demo-felhom; R6 otherwise), an appliance only (R12), authority = a signed `os_pve_step` or the root-owned ring-0 mark (R3). Select `pending-pve` (ring 0): installed Proxmox-origin packages with a pending upgrade. The report carries `pve_manager` (pveversion after the step).
- `internal/pvegate` (new): the agent's own writes to /etc/pve wait while a pve step runs (pmxcfs restarts); the step waits for writes in flight (bounded, 2 min — then it fails and does not run). Wired at `proxmox.Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile).
- `internal/osupdate`: `LayerPVE`; the night leg runs the pve step in ring 0 after a healthy host step (an appliance; ring 1 never in the night leg); `PVEHealthVerdict` = the host rule + every running container keeps its id + pveversion reads the installed pve-manager; the pve report carries Proxmox userspace only (the hub's candidate set). `PVEStepExecutor` (signed `os_pve_step`, ring 1, under the heavy-op gate and the /etc/pve gate); `reconcile.ClassOSPVEStep` (destructive-class); `felhom-opsign -op os_pve_step` (params by `-params`).
- Tests: wrapper `PVELane` (17; red first — the `pve-manager` plan was refused R12 on the old code), `pvegate` (5), `TestPVEGate_*` + `TestWritesEtcPVE`, `TestPVE_*`, `TestPVEHealthVerdict`, `TestPVEStepExecutor_*`. Red-proofs: `felhom.eu/documentation/audits/day-2026-10-07/B/`.
### Part of v0.151.0
- R-366 slice 2 (`09` §3 decision 168): the restore-test pick records, per tier, the archives it skipped as written with another key (count, oldest, newest — no key material) in a `ForeignKeyLedger`; the host report carries it as `foreign_key_archives.tiers` (the stanza absent until a tier was evaluated since start, `tiers: []` when none — no null on the wire, the report contract forbids it). The hub turns a change into one operator line. Tests `TestR366_PickRecordsArchivesWrittenWithAnotherKey`, `TestR366_EvaluatedWithNoneIsAnEmptyList` (red-proved, `felhom.eu/documentation/audits/day-2026-10-07/E/`).
- R-105 option A (`09` §3 decision 169): the `--selftest=escrow-create -directive <file>` flag and the escrow upload's `directive` field are removed — nothing read the directive; the DR path reads the recipe, tenantsync and the escrow blob. The hub ignores a `directive` from an older agent.
## v0.150.0 — the Docker step proves the engine reports a memory kill; after a restart the agent remembers the last backup per tier; three more SMART counters on the wire (R-528, R-894, R-330; `09` §3 decisions 157, 161) (2026-10-07)
Released by `scripts/release-agent.sh`: binary sha256 `a23d1c9085bc7fd4fc48fe0327f6504aa83e6331510fb4a3d23e042dddb26f9c`
config bundle sha256 `88456b386d9b1027bd22861cac8c23df004bf9fd9f67644d6595bfca8c94498e` (tag `v0.150.0` = `3a72a48`).
**The bundle carries the new `felhom-os-apply` (the memory-kill check) — deliver it with the binary:** signed
`agent_update`, then signed `agent_config_update`. No path added (26 → 26), so no step bundle.
### Part of v0.150.0 (2026-10-06 night, later) — after a restart the agent remembers the last backup per tier (R-894); three more SMART counters on the wire (R-330)
Ships with the memory-kill check below as v0.150.0, AFTER the 2026-10-07 night read-back. Nothing delivered tonight.
@@ -9,7 +162,7 @@ Ships with the memory-kill check below as v0.150.0, AFTER the 2026-10-07 night r
- Tests: `TestBackupDue_R894_*` (restart = a new server and a new state from the same file; fresh / old / none / storage answers / failed backup not saved), `TestBackupSuccessState_*`, `TestR894_LastKnownBackupsIsWiredIntoTheDaemon` (AST). Four red-proofs observed (`felhom.eu/documentation/audits/night-burndown-2026-10-06/s4/`).
- **R-330 (disk health Phase 2, the wire only):** the SMART summary carries three more SATA raw counters — `reported_uncorrect` (187), `command_timeout` (188, carried as the vendor reports it; some pack several counters), `udma_crc_errors` (199). Pointer + omitempty: an attribute the drive does not report is OMITTED (unknown), never 0. No verdict reads them yet. Tests `TestParseSMART_R330_*` (two red-proofs, `felhom.eu/documentation/audits/night-burndown-2026-10-06/r330/`).
## Unreleased (2026-10-06 night) — the Docker step proves the engine reports a memory kill (`09` §3 decision 157, R-528)
### Part of v0.150.0 (2026-10-06 night) — the Docker step proves the engine reports a memory kill (`09` §3 decision 157, R-528)
To be released as v0.150.0 with its config bundle AFTER the 2026-10-07 night read-back (the night of 2026-10-06 runs v0.149.0 on purpose).
@@ -18,11 +171,11 @@ To be released as v0.150.0 with its config bundle AFTER the 2026-10-07 night rea
- `configs/test_felhom_os_apply.py`: its `unittest.main()` sat in the middle of the file, so 11 tests (UnsentReport, SaveReportOnDisk, AgentDiesMidPass, CrashLeftTheJournal) never ran — moved to the end; all pass.
- Tests: the OOMCheck class (pass, OOMKilled=false, no event, unreadable image, removal on an inspect error, not on other layers or in health mode, the events window after the settle wait, mode oom-check alone and its refusals); TestDocker_OOMCheckReachesTheHubUnchanged, TestR868_KeptCopyCarriesTheOOMCheck. 12 red-proofs in `felhom.eu/documentation/audits/readback-2026-10-07/F/`.
## Unreleased (2026-10-06 evening) — the shared rule file (`09` §3 decision 152); no code change
### Part of v0.150.0 (2026-10-06 evening) — the shared rule file (`09` §3 decision 152); no code change
- `.claude/rules/unprompted-work.md` added, byte-identical to the copies in felhom.eu, felhom-controller, app-catalog-felhom.eu and the workspace root (checked with `diff` against the controller's copy and one md5 across all five). Its copies line names five copies.
## Unreleased (2026-10-06 afternoon) — instruction files kept true (`09` §3 decision 150); no code change
### Part of v0.150.0 (2026-10-06 afternoon) — instruction files kept true (`09` §3 decision 150); no code change
- `CLAUDE.md` „Gates — ONE entry point": the runner runs every gate in its `GATES` table (five: three shared, `published`, `release-complete`); `--fast` skips `published` (network). It said two gates and „all of them".
- `CLAUDE.md`: the decoy gate and its audit are named with their `felhom.eu/` prefix (they do not exist in this repo).
+7
View File
@@ -1,5 +1,12 @@
# CONTEXT — felhom-agent working state
> **2026-10-08 (day) — UNRELEASED on main, ships with tomorrow's release (decision 178).** R-899: `POST /backup?trigger=manual`
> runs no OS leg (`localapi/server.go`); after-boot kernel reports carry the saved block's ring (`kernelReportRing`). R-304:
> `escrow.ErrRetainedUnchecked` → HTTP 424 `older_unchecked` when not every earlier package was tried (withheld, caps,
> malformed, unreadable list). Binary-only delivery; no wrapper or root file changed.
> **2026-10-07 (evening) — v0.152.0, the kernel lane (R-836, decision 172, `11` §5.11).** Wrapper layer `kernel` + two GRUB generators in the bundle (delivered with step bundle `0.152.0-step1` — the bundle adds paths, R-880); `osupdate/kernel.go`: night step on told nights only, after-boot judge (host rule + hub reached, 20 min), ONE self-revert, `os_kernel_step` stages only. Proven on Tester 1 (panic → fell_back, held guest → self_reverted, healthy → 7.0.14-22 default). Open: R-898, R-897; the ring-0 night run.
> **2026-10-04 night — v0.143.0 RELEASED + vouched (R-840, decision 96): the config bundle.** `felhom-os-apply` mode
> `bundle` (signed `agent_config_update`, verified by the wrapper itself; trust files never bundle paths) +
> `--install-bundle` (installer 1.31.0); `BUNDLE_FILES` is the one table; `scripts/build-config-bundle.py`;
+8 -7
View File
@@ -44,13 +44,14 @@ unnoticed until a user hit them. `internal/capability` makes that loud:
(`HostCapabilityChecker`) alerts the operator on a Critical capability going degraded. Serve-degraded
— the probe never blocks startup. (Next self-health slice: the controller↔agent channel check.)
**Controller-swap under non-root (v0.45.0).** The agent-owned controller image swap
(`internal/localapi/controllerswap.go`) no longer shells out: `writeImage` pipes the image ref on
**stdin** into an in-guest `tee /etc/felhom-controller-image` (via `GuestExecStdin` →
`Runner.RunStdin`, the same fenced `sudo -n` runner) — no `bash -c`, no interpolation. Its 5 narrow
grants live in the `FELHOM_CONTROLLERSWAP` sudoers alias (all read-only or fixed-target; the `tee`
target is the FIXED image path, content stdin-fed) and in the capability manifest (Critical), so a
dropped grant is a build failure + a live degraded signal. No general `pct exec` is granted.
**Controller-swap under non-root (v0.45.0; the write since R-861 (a) A1).** The agent-owned controller image swap
(`internal/localapi/controllerswap.go`) no longer shells out. The write goes on **stdin** to the ROOT verb
`felhom-priv-apply controller-image <vmid>` (`GuestBinder.WriteControllerImage` → `Runner.RunStdin`, the same fenced
`sudo -n` runner), which re-checks the ref against our registry + repository + an x.y.z tag and writes
`/etc/felhom-controller-image` inside the guest itself; the agent has no in-guest `tee` grant any more (before, a
compromised agent could feed any image — sudo cannot see stdin). Its grants live in the `FELHOM_CONTROLLERSWAP` sudoers
alias (read-only or fixed-target) and in the capability manifest (Critical), so a dropped grant is a build failure + a
live degraded signal. No general `pct exec` is granted.
## The `storage` package — observe + watchdog (slice 5)
+14 -12
View File
@@ -1,14 +1,16 @@
# REPORT — the second burn-down night (2026-10-06 → 07): R-894 and the R-330 wire half, on main, unreleased
# REPORT — agent, 2026-10-08 (day): R-899 and R-304 on main, unreleased
Nothing was released or delivered (brief fence). Both changes ship with the memory-kill check as **v0.150.0**, after the
2026-10-07 night read-back. Session report: `felhom.eu/REPORT-night-burndown-2026-10-06.md`; night log
`felhom.eu/documentation/audits/night-burndown-2026-10-06/NIGHT-LOG.md`.
**No release, no delivery today** (kernel night 8→9; `09` §3 decision 178). Binary-only; ships with tomorrow's release.
- **R-894** (`74b5eae`): the newest successful backup per tier is kept on disk (`backup-success-state.json` in the OOB
state dir) and read by `/backup/due` **only when the tier's storage cannot be read**: fresh → not due, older than the
cadence → due, none → due with age unknown (as before). A storage that answers stays the ground truth. Tests
`TestBackupDue_R894_*`, `TestBackupSuccessState_*`, `TestR894_LastKnownBackupsIsWiredIntoTheDaemon`; four red-proofs
`felhom.eu/documentation/audits/night-burndown-2026-10-06/s4/`. Architecture: `07` §6.1 updated.
- **R-330, wire only** (`adaf86a`): the host report's SMART summary carries 187 `reported_uncorrect`, 188
`command_timeout`, 199 `udma_crc_errors` (pointers, omitted when unknown). No verdict reads them.
- Suite `go test ./...` rc 0; `scripts/agent_gates.py` (full) all OK; CI green on each push (runs 1451, 1467).
| Item | Commit | Tests | Red-proof |
|---|---|---|---|
| R-899 — `trigger=manual` (a press) runs no OS leg | `4c69c25` | `TestAfterPrimaryBackup` + 2 sub-cases | ignoring `trigger` → the leg ran once after a press (`[8200]`) |
| „ring 1" label in after-boot kernel reports (seen 2026-10-08 night, demo-felhom) | `4c69c25` | `TestKernelReportRing_BeforeFirstFetch` | `Block().Ring` → `ring 1, want 0` |
| R-304 — 424 `older_unchecked` instead of „did not open the sealed bundle" when earlier packages were not all tried | `91b9405` | `TestR304_*` (real age crypto), 424 row in `TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus` | check off → four cases returned the wrong-code error |
**Read before the change (read-only, demo-hp):** on 2026-10-07 08:51 the morning press DID reach the OS leg
(`osupdate: skipped — already ran tonight`, saved by the 20-hour rule only); `os-update-block.json` on demo-hp holds
`ring 0` (the fallback the label fix reads).
**Gates:** `go build/vet/test ./...` rc 0; `agent_gates.py --fast` rc 0. **CI:** job 1529 (`4c69c25`) success; job 1530
(`91b9405`) success.
+4 -2
View File
@@ -15,7 +15,7 @@
| `SudoHostOps.run` | internal/storage/hostops.go | `run(ctx, name, args...) error` | allowlisted exec with stderr-wrapped error | Every arg pre-validated via validate.go before this is called |
| `Prober.Probe` | internal/capability/probe.go | `Probe(ctx) []Status` | live sudo-policy capability check (`sudo -n -l --`) | Needs a DIRECT runner (never the sudo-prefixing one — double-sudo); never executes probed cmds. v0.86.0: config-gated caps (`Capability.GatedBy` + `Prober.GateActive`) report `inactive`/"disabled by configuration" ONLY when healthy — broken plumbing stays degraded; the pbsdr-* gate answers from `pbsdr.Manager.DRConfigured` (marker-backed across restarts) |
| ~~`stageTemp`~~ (REMOVED v0.146.0, R-861) | — | — | — | Nothing the agent writes is `install`ed where root reads it any more: use `felhom-priv-apply` (below) or ship a fixed file in the bundle |
| `felhom-priv-apply` (v0.146.0, R-861) | configs/felhom-priv-apply | `felhom-priv-apply unit <name> \| dnsmasq <tmp> <name> \| wg \| sshd-config \| sshd-key` | ANY agent-rendered file a root program reads (systemd unit, dnsmasq drop-in, wg-quick conf, OOB sshd) — fixed source + destination, CONTENT checked against the agent's own renderers | A new renderer needs a verb + a contract test (`internal/privapplytest.Check`) feeding its REAL output; never a new `install` sudoers line |
| `felhom-priv-apply` (v0.146.0, R-861) | configs/felhom-priv-apply | `felhom-priv-apply unit <name> \| dnsmasq <tmp> <name> \| wg \| sshd-config \| sshd-key \| controller-image <vmid>` (the last reads the ref on stdin, R-861 (a) A1) | ANY agent-rendered file a root program reads (systemd unit, dnsmasq drop-in, wg-quick conf, OOB sshd) — fixed source + destination, CONTENT checked against the agent's own renderers | A new renderer needs a verb + a contract test (`internal/privapplytest.Check`) feeding its REAL output; never a new `install` sudoers line |
| `privapplytest.Check` | internal/privapplytest/check.go | `Check(t, verb, name, content) string` | the Go↔root-checker contract: a renderer's real output must read `OK` | Skips without python3; one call per rendered shape + one refused control |
| `BUNDLE_FILES` + `Bundle` (mode `bundle`, `--install-bundle`; mode `agent_update` v0.146.0, R-861) | configs/felhom-os-apply | the ONE table of root-owned paths + the installer of them | ANY new root-owned file the installer writes (sudoers line, wrapper, unit) — add it to the table, never a new installer fetch (R-840) | The builder (`scripts/build-config-bundle.py`) and the installer read the same table; `test_every_root_file_the_installer_writes_is_in_the_bundle` fails on a path the bundle lacks. Trust files (`/etc/felhom/os-trust.json`, `operator-signers`) are NEVER bundle paths (R17) |
| `osupdate.ConfigUpdateExecutor` | internal/osupdate/bundle.go | signed op `agent_config_update` {agent_version, bundle_sha256} | delivering the bundle to an installed box | a courier only: the root wrapper re-verifies signature, host, nonce and sha itself |
@@ -88,6 +88,8 @@
| Symbol | File | Short signature | Use for | Gotchas |
|---|---|---|---|---|
| `pvegate.Write` / `pvegate.Step` | internal/pvegate/pvegate.go | `Write(ctx) (release, waited, err)` / `Step(ctx) (end, err)` | R-812 option A: keep the agent's own /etc/pve writes out of a Proxmox package step (pmxcfs restarts) | Already wired at the two chokepoints — `Client.doBody` (every non-GET) and `ExecRunner.RunStdin` (`WritesEtcPVE`: pct config verbs, pvesm, pveum, felhom-pbs-apply create/reconcile). A new root CLI that writes /etc/pve goes into `WritesEtcPVE`, never its own lock. Never take `Step` around anything but the wrapper call (`Leg.runPVE`) — a `Write` inside a `Step` deadlocks until its context ends. |
| `osupdate.KernelVerdict` / `Leg.KernelAfterBoot` | internal/osupdate/kernel.go | `KernelVerdict(before, after, tunnel, hubReached) (ok, why)` | R-836: THE one-shot-boot rule — the host health rule (`HostHealthVerdict`) AND the box reached the hub since the boot | The only judge of a new kernel. Never reboot the host from Go: every reboot is the wrapper's (`kernel-reboot`, `kernel-revert`), and only for a staged step. A new kernel-lane state lives in the wrapper's `/var/lib/felhom-kernel/state.json`, never in the agent's own files (the agent can write those). |
| `Client.WaitTask` | internal/proxmox/task.go | `WaitTask(ctx, upid, opts) (TaskStatus, error)` | asserting EVERY mutating op | POST 200 ≠ success; authz can fail at task exec; `AllowWarnings` opt-in |
| `Client.Pool` | internal/proxmox/query.go | `Pool(ctx, name) (PoolInfo, error)` | felhom-pool membership (the ownership registry, A1) | Needs `Pool.Audit` at `/pool/<name>` (host-install v1.9.0+); `Pool.Allocate` does NOT satisfy the read; members can be storages (type `storage`, vmid 0) — filter them |
| `Client` mutate wrappers (`RestoreLXC/Vzdump/DestroyLXC/Snapshot/Rollback/SetConfig/ResizeLXC/Start/Stop`) | internal/proxmox/mutate.go | return `(upid, error)` | all API mutations | Async → always pair with WaitTask; route via gate/queue, not ad-hoc |
@@ -168,7 +170,7 @@
| `backup.SpecBuilder` / `backup.TierPicker` / `(*BackupRunner).PickSettledRestoreCandidateOn` | internal/backup/schedule.go, runner.go | `func(ctx,archive) RestoreTestSpec`; `func(ctx,target,notAfter) (archive,landed,error)` | The per-run restore-test spec + per-tier **settled** candidate lookup (R-85, widened by R-86) | The spec is built **PER RUN**, never frozen at construction — the pre-R-85 immediately-invoked value made the offsite tier unschedulable AND went stale on any config change. `SourceTier` comes from **the archive**, never the configured target (the v0.100.0 rule). A tier with no archive returns `("", zero, nil)` — **`""` is NOT an error**, or every fresh box looks broken for its first week. **R-86: `notAfter` is the settle cutoff** (zero = no cutoff, which is what keeps `PickRestoreCandidateOn` a one-line call into it), and the picker now skips entries failing `archivePlausiblyComplete` — under per-archive due-ness an incomplete phantom would be picked forever, fail forever, never earn proof, and make the tier due at EVERY evaluation. |
| `localapi.BackupTier` + `normalizeBackupTiers` / `config.BackupConfig.BackupTiers` | internal/localapi/backup_tiers.go, internal/config/config.go | `normalizeBackupTiers(tiers, legacy, cadence) []BackupTier`; `BackupTiers() ([]BackupTier, []string)` | THE R-82 multi-tier resolution — one runner per tier, primary first | **The untargeted local-API contract is FROZEN**: no `?target=` ⇒ primary tier ⇒ pre-R-82 response BYTES (Target is `omitempty` and stays empty). Never default a missing cadence — reject it and log the warning at ERROR. Never share one retention knob between tiers. Jobs are keyed by (vmid,target). |
| `localapi.StaleLockController` | internal/localapi/stalelock.go | `*staleLockController` (Client + Runner + pool) | `fakeStaleLock` (Server-level) stalelock_test.go; `fakeStaleLockAPI` (controller-level, tests the A1 pool intersect) stalelock_pool_test.go |
| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
| `localapi.GuestExecutor` | internal/localapi/controllerswap.go | `*GuestBinder` (pct exec; the image write via `felhom-priv-apply controller-image`) | `fakeGuestExec` internal/localapi/controllerswap_test.go |
| `guestnet.Runner` / `guestnet.GuestSource` (R-54, v0.92.0) | internal/guestnet/{probe,watchdog}.go | `*proxmox.ExecRunner`; the POOL-VERIFIED `localapi.StaleLockController.Guests` (ListLXC ∩ felhom pool, audit A1) | `scriptedRunner` + `fakeGuests` internal/guestnet/watchdog_test.go. **Never wire a bare `ListLXC` here** — under a broad token that would run dhclient inside a co-tenant's container. Every assertion is an exec COUNT, and the load-bearing ones are the negatives: a static guest, an unprobeable guest, a boot-race guest and an unproven guest list must record **zero** heal calls |
| `guestnet.Watchdog.SetDampers` / `now` (clock seam) | internal/guestnet/watchdog.go | config `guest_net.*`; `now` defaults to `time.Now` | tests advance a manual clock (the storage-watchdog pattern) and assert the heal ceilings EXACTLY — ≥10 min apart, ≤3/hour, and ≤30 over a scripted 10 hours of permanent failure. A damper with no test is a comment |
| `hub.GuestNetReporter` (R-54) | internal/hub/collect.go | `*guestnet.Watchdog` (`GuestNetStatus`) | internal/hub/collect_guestnet_test.go asserts the stanza through the PRODUCTION `Collect` path AND that the `guest_net` key is ABSENT from the wire when no reporter is wired — an always-present empty stanza would make "not wired" and "found nothing" the same signal, which is the shape v0.91.0 hid behind |
+90 -61
View File
@@ -135,39 +135,38 @@ func main() {
return
}
var (
cfgPath string
selftest selftestFlag
vmid int
watch time.Duration
archive string
mode string
hostname string
keep bool
rootfsGrow int
dataVolGrow int
dataVolMount string
sysDataGrow int
sysDataMount string
cores int
memoryMB int
pbsStorage string
paperkey bool
offline bool
upload bool
custID string
custDomain string
custName string
custEmail string
hubPassword string
blobPath string
expectedFP string
keyDest string
installWGKey bool
idBundlePath string
directivePath string
swapImage string
outputMode string
showVersion bool
cfgPath string
selftest selftestFlag
vmid int
watch time.Duration
archive string
mode string
hostname string
keep bool
rootfsGrow int
dataVolGrow int
dataVolMount string
sysDataGrow int
sysDataMount string
cores int
memoryMB int
pbsStorage string
paperkey bool
offline bool
upload bool
custID string
custDomain string
custName string
custEmail string
hubPassword string
blobPath string
expectedFP string
keyDest string
installWGKey bool
idBundlePath string
swapImage string
outputMode string
showVersion bool
)
flag.StringVar(&cfgPath, "config", envOr("FELHOM_AGENT_CONFIG", "/etc/felhom-agent/agent.json"), "path to the agent config file (JSON)")
flag.Var(&selftest, "selftest", "run a self-test and exit: bare/`read` = read-only queries; `task` = reversible mutating exercise (needs -vmid); `hub` = one collect+report; `storage` = observe storage (+ -watch); `backup` = one-shot backup of -vmid; `restore-test` = restore→boot→verify→teardown of -archive (or newest backup); `restore-test-due` = READ-ONLY: print the per-tier due verdict the scheduler would act on, with its cost; `pbs-verify` = trigger a PBS verify + print snapshot records; `bring-up` = restore→reset identity→size→start link-up of -archive into -vmid (needs -mode/-archive/-vmid; optional -cores/-memory cap; tears down unless -keep); `provision` = full slice-8A chain: bring-up provision + mint token + populate bootstrap config mount (needs -archive/-vmid/-customer-id/-hub-password; optional -rootfs-grow/-datavol-grow/-cores/-memory (-sysdata-grow is deprecated: folded into -datavol-grow); keeps the guest)")
@@ -198,7 +197,6 @@ func main() {
flag.StringVar(&keyDest, "keydest", "", "for --selftest=escrow-consume: where to install the recovered key (0600)")
flag.BoolVar(&installWGKey, "install-wg-key", false, "for --selftest=identity-consume: ALSO install the recovered wg_private_key into wgtunnel's key file (S5 DR; create-only, refuses to overwrite)")
flag.StringVar(&idBundlePath, "identity-bundle", "", "for --selftest=escrow-create: a 0600 JSON file {tunnel_token,pbs_token} to ALSO escrow under R (10D)")
flag.StringVar(&directivePath, "directive", "", "for --selftest=escrow-create: a JSON file with the non-secret DR directive (pbs repo/ns, expected fingerprint, tunnel id)")
flag.StringVar(&custID, "customer-id", "", "for --selftest=provision: the customer id — the hub config-pull target, baked into the guest's bootstrap")
flag.StringVar(&hubPassword, "hub-password", "", "for --selftest=provision: the customer's hub RETRIEVAL PASSPHRASE (SECRET) — baked into bootstrap.json so the controller pulls its config (and the customer-scoped hub key) from the hub. The customer must already exist in the hub.")
flag.StringVar(&swapImage, "image", "", "for --selftest=controller-swap: the target controller image ref (gitea.dooplex.hu/admin/felhom-controller:<semver>) — must already be pulled in the guest")
@@ -269,7 +267,7 @@ func main() {
SysDataGrowGB: sysDataGrow, SysDataMount: sysDataMount, Cores: cores, MemoryMB: memoryMB},
}))
case "escrow-create":
os.Exit(runSelftestEscrowCreate(context.Background(), cfg, logger, pbsStorage, paperkey, offline, upload, idBundlePath, directivePath, outputMode))
os.Exit(runSelftestEscrowCreate(context.Background(), cfg, logger, pbsStorage, paperkey, offline, upload, idBundlePath, outputMode))
case "escrow-consume":
os.Exit(runSelftestEscrowConsume(context.Background(), logger, blobPath, expectedFP, keyDest))
case "identity-consume":
@@ -792,6 +790,9 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
pbsTargets := pbsTargetsFromPVE(cfg, px, logger)
pbsReporter := pbs.NewLiveSnapshotReporter(pbsTargets, pbsStore, pbs.DefaultLiveSnapshotTimeout, logger)
collector := hub.NewCollector(px, newTunnelProber(cfg, px), observer, backupStore, backupStore, pbsReporter, cfg.Hub.HostID, version, logger)
// R-366 slice 2: the restore-test's ledger of archives written with another key → the host report.
foreignKeys := backup.NewForeignKeyLedger()
collector.SetForeignKeyArchiveReporter(foreignKeys)
collector.SetBackupTargetResolver(primaryBackupTargetOf(cfg)) // R-109: the recipe names the live target
// Privileged-capability self-check (v0.44.0): probe the sudoers grants the non-root agent
// depends on. The probe runs `sudo -n -l` LITERALLY (a policy LIST, never executing the
@@ -876,6 +877,13 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
go osLeg.SendUnsentLoop(ctx, 5*time.Minute, func(n int) {
logger.Info("osupdate: sent kept report(s)", "count", n)
})
// R-836 (`09` §3 decision 172): what became of a kernel step across this boot; on a one-shot boot of a new kernel,
// judge it (the host health rule + the hub reached) for KernelJudgeWait, then make it the default or revert ONCE.
// The wait: measured 2026-10-07 (`audits/kernel-lane-2026-10-07/B/`) — every container healthy 68 s after the
// reboot on demo-felhom and 272 s on demo-hp (the hub reached at 63 s / 189 s); 20 minutes leaves room for a slow
// network and stays under the hub's 45-minute host_stale (its
// alerting.stale_threshold; host_down at 90). On a box without the kernel lane (an older wrapper, a BYO host) the check is refused and logged.
go osLeg.KernelAfterBoot(ctx, 0, osupdate.KernelJudge{Wait: osupdate.DefaultKernelJudgeWait})
// Reconcile (slice 4) runs alongside the hub loop, sharing the per-guest queue
// (doc 03 §10). At slice 4 the desired-state provider is empty (no hub serving
@@ -1017,7 +1025,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
// with the local API so a backup and a restore-test can never run together.
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
heavyOps := &backup.InFlight{}
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, logger)
scheduler := buildRestoreTestScheduler(cfg, px, engine, backupStore, rtState, heavyOps, foreignKeys, logger)
// R-189: the host report's restore_tests[] must survive an agent restart. The in-memory store
// holds only this process's latest run, and under per-archive due-ness the agent will not
// re-test an archive it has already proven — so without this the hub can report a tier unproven
@@ -1109,6 +1117,25 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
}
return release, nil
}}
// R-812 option A: a signed Proxmox package step (`11` §5.10) — ring 1; under the heavy-op gate and the /etc/pve gate.
pveExec := osupdate.PVEStepExecutor{Leg: osLeg, Guest: firstGuest(px),
Gate: func(ctx context.Context) (func(), error) {
release, busy, ok := heavyOps.TryAcquire("os-pve-step")
if !ok {
return nil, fmt.Errorf("busy: %s", busy)
}
return release, nil
}}
// R-836 (`09` §3 decision 172): a signed kernel step STAGES a kernel on a ring-1 box (install + the one-shot flag,
// never a reboot — the night leg reboots it on a night the household was told about); under the heavy-op gate.
kernelExec := osupdate.KernelStepExecutor{Leg: osLeg, Guest: firstGuest(px),
Gate: func(ctx context.Context) (func(), error) {
release, busy, ok := heavyOps.TryAcquire("os-kernel-step")
if !ok {
return nil, fmt.Errorf("busy: %s", busy)
}
return release, nil
}}
// Agent v0.143.0 (R-840): the config bundle — the box's root-owned files by a signed job; the wrapper verifies it.
bundleExec := osupdate.ConfigUpdateExecutor{Leg: osLeg, URLTemplate: suCfg.URLTemplate, Username: suCfg.Username, Token: suCfg.Token,
// The capability probe confirms from the agent's side: `sudo -l` lists every command the new sudoers grants.
@@ -1120,7 +1147,7 @@ func runDaemon(cfg config.Config, logger *slog.Logger, logRing *applog.Ring) int
}
logger.Warn("osupdate: capability probe after the config bundle", "ok", ok, "total", total, "degraded", strings.Join(names, ","))
}}
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, bundleExec}, cfg.Hub.HostID, logger)
jobsRunner := signedjobs.NewRunner(client, gate, signedjobs.ExecutorChain{wipeExec, decommExec, updateExec, dockerExec, pveExec, kernelExec, bundleExec}, cfg.Hub.HostID, logger)
loop.SetEnvelopeObserver(hub.MultiObserver(desiredSyncer, jobsRunner))
// Controller-driven escrow ceremony (v0.88.0): static config facts + the LATE-BOUND DR gate —
@@ -1700,7 +1727,7 @@ func primaryBackupTargetOf(cfg config.Config) func() hub.ConfiguredBackupTarget
// disables the cadence (returns a scheduler that just waits) when the cadence is off or the
// scratch band / restore storage is invalid — a misconfig must not crash the daemon, and the
// machinery still works on-demand via --selftest=restore-test.
func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *reconcile.Engine, store *backup.Store, rtState *backup.RestoreTestState, inFlight *backup.InFlight, logger *slog.Logger) *backup.Scheduler {
func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *reconcile.Engine, store *backup.Store, rtState *backup.RestoreTestState, inFlight *backup.InFlight, foreign *backup.ForeignKeyLedger, logger *slog.Logger) *backup.Scheduler {
// R-86: this is the EVALUATION interval, not the trigger. What decides a test happens is the
// per-archive due-check in internal/backup/restoretest_due.go.
cadence := cfg.Backup.RestoreTestEvalInterval()
@@ -1719,6 +1746,9 @@ func buildRestoreTestScheduler(cfg config.Config, px *proxmox.Client, engine *re
min, max := cfg.Backup.ScratchBand()
target := cfg.Backup.BackupTarget()
runner := backup.NewBackupRunner(px, target, "", "felhom restore-test", "", logger)
if foreign != nil {
runner.SetForeignKeyLedger(foreign) // R-366 slice 2: the pick records archives written with another key
}
// Every configured tier is a rotation candidate, not just the primary.
cfgTiers, _ := cfg.Backup.BackupTiers() // warnings already logged where the tiers are armed
tierIDs := make([]string, 0, len(cfgTiers))
@@ -1874,11 +1904,15 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
return nil, 0, ferr
}
out := make([]escrow.RetainedBlob, 0, len(resp.Packages))
// R-304: every package the hub holds and the code will NOT be tried against — no key material,
// over the hub's cap, or malformed here. A refusal may call the code wrong only when this is 0.
withheld := resp.UnopenableCount + resp.TruncatedCount
for _, p := range resp.Packages {
blob, derr := base64.StdEncoding.DecodeString(p.IdentityEscrowB64)
if derr != nil || len(blob) == 0 {
// One malformed package must not sink the rest — the customer's code may open a
// later one, and a skipped entry is strictly better than a refusal we cannot justify.
withheld++
continue
}
out = append(out, escrow.RetainedBlob{
@@ -1888,9 +1922,13 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
Index: p.Index,
})
}
return out, resp.UnopenableCount, nil
return out, withheld, nil
},
}
// R-894: ONE instance — the local API writes it, the host report reads it (2026-10-09: a restart right
// after the night backup erased the hub's evidence of it).
lastKnownBackups := backup.NewBackupSuccessState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "backup-success-state.json"))
collector.SetKnownBackupReporter(lastKnownBackups)
srv, err := localapi.NewServer(localapi.Options{
EscrowRecovery: escrowRecoverer,
ListenAddr: cfg.LocalAPI.ListenAddr,
@@ -1903,7 +1941,7 @@ func buildLocalAPIServer(cfg config.Config, px *proxmox.Client, store *backup.St
Store: store,
// R-894: the newest success per tier on disk — the due-check's fallback when the storage cannot be
// read right after a restart. Same state dir as restore-test-state.json.
LastKnownBackups: backup.NewBackupSuccessState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "backup-success-state.json")),
LastKnownBackups: lastKnownBackups,
Storage: observer,
DriveTargets: driveTargets, // Impl-2a: registry+units drives for the /disks view (union w/ Observe storages)
Smart: storage.NewSmartReader(hostOps), // v0.95.0 Fix B: SMART for the union-path drives
@@ -2264,7 +2302,7 @@ func runSelftestRestoreTestDue(ctx context.Context, cfg config.Config, logger *s
return 1
}
rtState := backup.NewRestoreTestState(filepath.Join(cfg.OOB.WithDefaults().StateDir, "restore-test-state.json"))
sched := buildRestoreTestScheduler(cfg, px, nil, backup.NewStore(), rtState, &backup.InFlight{}, logger)
sched := buildRestoreTestScheduler(cfg, px, nil, backup.NewStore(), rtState, &backup.InFlight{}, nil, logger)
fmt.Printf("eval_interval=%s settle=%s\n", cfg.Backup.RestoreTestEvalInterval(), cfg.Backup.RestoreTestSettle())
start := time.Now()
@@ -2690,7 +2728,6 @@ type escrowCeremonyOpts struct {
offline bool
upload bool
identityBundlePath string
directivePath string
}
// escrowCeremonyOutcome is the shared core's result. R is the ONLY secret; Sum mirrors the
@@ -2749,11 +2786,10 @@ func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger,
return out, &escrowCeremonyErr{kind: "setup", err: fmt.Errorf("PBS key for %q not found (%s): %v", storage, keyPath, err)}
}
// Slice 10D.1: optionally ALSO wrap the identity bundle under the same R, and carry the non-secret
// directive for the hub. The bundle file is a 0600 secret (tunnel/pbs tokens); the directive is
// non-secret (pbs repo/ns, expected fingerprint, tunnel id).
// Slice 10D.1: optionally ALSO wrap the identity bundle under the same R. The bundle file is a 0600 secret
// (tunnel/pbs tokens). The non-secret "directive" that used to ride along is retired (R-105, `09` §3 decision 169:
// nothing read it; the DR path reads the recipe, tenantsync and the escrow blob).
var identity *escrow.IdentityBundle
var directive json.RawMessage
if opts.identityBundlePath != "" {
raw, err := os.ReadFile(opts.identityBundlePath)
if err != nil {
@@ -2764,11 +2800,6 @@ func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger,
return out, &escrowCeremonyErr{kind: "setup", err: fmt.Errorf("identity bundle is not valid JSON {tunnel_token,pbs_token}: %v", err)}
}
identity = &b
if opts.directivePath != "" {
if d, err := os.ReadFile(opts.directivePath); err == nil && json.Valid(d) {
directive = d
}
}
}
// S3: auto-inject the offsite WG private key into the escrowed identity when the key file
// exists — a NEW escrow run should always capture the live tunnel identity. Field NAME only
@@ -2847,7 +2878,7 @@ func escrowCeremony(ctx context.Context, cfg config.Config, logger *slog.Logger,
ResticPwSealed: resticStaged,
}
if opts.upload {
if err := uploadEscrowBlob(ctx, cfg, res, directive, resticPwSHA256); err != nil {
if err := uploadEscrowBlob(ctx, cfg, res, resticPwSHA256); err != nil {
// R is minted and the blob self-verified — only the hub leg failed. kind "upload" lets
// the text shell keep the pre-extraction order (R surfaced, THEN the failure).
return out, &escrowCeremonyErr{kind: "upload", err: err}
@@ -2904,7 +2935,7 @@ func printEscrowTextRBlock(out *escrowCeremonyOutcome) {
// NOTHING else there; every human/info line goes to stderr; failures exit non-zero with no
// partial JSON. This is the controller-driven ceremony's parse surface (spike §2.3: the text
// banner is positionally brittle).
func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slog.Logger, storage string, paperkey, offline, upload bool, identityBundlePath, directivePath, outputMode string) int {
func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slog.Logger, storage string, paperkey, offline, upload bool, identityBundlePath, outputMode string) int {
switch outputMode {
case "", "text", "json":
default:
@@ -2922,7 +2953,7 @@ func runSelftestEscrowCreate(ctx context.Context, cfg config.Config, logger *slo
out, cerr := escrowCeremony(ctx, cfg, logger, escrowCeremonyOpts{
storage: storage, paperkey: paperkey, offline: offline, upload: upload,
identityBundlePath: identityBundlePath, directivePath: directivePath,
identityBundlePath: identityBundlePath,
})
if cerr != nil {
switch cerr.kind {
@@ -3105,20 +3136,19 @@ type escrowUploadRequest struct {
BlobB64 string `json:"blob_b64"` // base64 of the opaque R-wrapped blob (ciphertext)
KeyFingerprint string `json:"key_fingerprint"` // for operator display only
Posture string `json:"posture"` // e.g. "zero_knowledge"
// Slice 10D.1 — optional DR bundle (identity escrow + non-secret directive). Omitted in slice-7.
IdentityBlobB64 string `json:"identity_blob_b64,omitempty"`
DirectiveJSON json.RawMessage `json:"directive,omitempty"`
CreatedAt string `json:"created_at"` // RFC3339
// Slice 10D.1 — optional identity escrow. Omitted in slice-7. (The `directive` is retired — R-105.)
IdentityBlobB64 string `json:"identity_blob_b64,omitempty"`
CreatedAt string `json:"created_at"` // RFC3339
// SLICE 3 — sha256 hex of the offsite restic repo password sealed in the identity blob (present only
// when a staged password was folded in). Non-reversible hash of a 256-bit random secret — safe to
// store/serve; lets the controller VERIFY "the escrow covers the CURRENT key" and auto-confirm.
ResticPwSHA256 string `json:"restic_pw_sha256,omitempty"`
}
// uploadEscrowBlob PUTs the opaque blob (and, for 10D, the identity blob + non-secret directive) to
// uploadEscrowBlob PUTs the opaque blob (and, for 10D, the identity blob) to
// the hub, authed with the per-host key. The hub stores ciphertext + non-secret fields; no usable
// secret leaves the agent.
func uploadEscrowBlob(ctx context.Context, cfg config.Config, res escrow.CreateResult, directive json.RawMessage, resticPwSHA256 string) error {
func uploadEscrowBlob(ctx context.Context, cfg config.Config, res escrow.CreateResult, resticPwSHA256 string) error {
if cfg.Hub.URL == "" || cfg.Hub.HostID == "" || cfg.Hub.APIKey == "" {
return fmt.Errorf("hub not configured (url/host_id/api_key)")
}
@@ -3131,7 +3161,6 @@ func uploadEscrowBlob(ctx context.Context, cfg config.Config, res escrow.CreateR
}
if len(res.IdentityBlob) > 0 {
upReq.IdentityBlobB64 = base64.StdEncoding.EncodeToString(res.IdentityBlob)
upReq.DirectiveJSON = directive
}
body, _ := json.Marshal(upReq)
url := strings.TrimRight(cfg.Hub.URL, "/") + "/api/v1/hosts/" + cfg.Hub.HostID + "/escrow"
+31
View File
@@ -12,12 +12,18 @@ import (
//
// COMPANION RED-PROOF (observed): delete the `LastKnownBackups:` line from buildLocalAPIServer → this
// fails with "localapi.Options in buildLocalAPIServer has no LastKnownBackups field". Restored.
//
// 2026-10-09: the SAME instance also feeds the host report (collector.SetKnownBackupReporter), so a
// restart right after a backup no longer erases the hub's evidence of it. The field may name a local
// variable; the variable must be built by backup.NewBackupSuccessState and be the one handed to the
// collector. RED-PROOF (observed): drop the SetKnownBackupReporter call → "not handed to the collector".
func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
_, f := parseMain(t)
if !callsWithin(f, "main")["runDaemon"] || !callsWithin(f, "runDaemon")["buildLocalAPIServer"] {
t.Fatal("main → runDaemon → buildLocalAPIServer is broken — the path this test asserts is not the live one")
}
var field, built bool
var fieldVar, handed string
for _, d := range f.Decls {
fd, ok := d.(*ast.FuncDecl)
if !ok || fd.Name == nil || fd.Name.Name != "buildLocalAPIServer" || fd.Body == nil {
@@ -45,6 +51,28 @@ func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
if callsIn(kv.Value)["backup.NewBackupSuccessState"] {
built = true
}
if id, ok := kv.Value.(*ast.Ident); ok {
fieldVar = id.Name
}
}
}
return true
})
// a local variable: built by NewBackupSuccessState, and handed to the collector
ast.Inspect(fd.Body, func(n ast.Node) bool {
switch x := n.(type) {
case *ast.AssignStmt:
for i, l := range x.Lhs {
if id, ok := l.(*ast.Ident); ok && fieldVar != "" && id.Name == fieldVar && i < len(x.Rhs) &&
callsIn(x.Rhs[i])["backup.NewBackupSuccessState"] {
built = true
}
}
case *ast.CallExpr:
if fn, ok := x.Fun.(*ast.SelectorExpr); ok && fn.Sel.Name == "SetKnownBackupReporter" && len(x.Args) == 1 {
if id, ok := x.Args[0].(*ast.Ident); ok {
handed = id.Name
}
}
}
return true
@@ -56,6 +84,9 @@ func TestR894_LastKnownBackupsIsWiredIntoTheDaemon(t *testing.T) {
if !built {
t.Fatal("LastKnownBackups is not built by backup.NewBackupSuccessState")
}
if handed == "" || handed != fieldVar {
t.Fatalf("the saved backups (%q) are not handed to the collector (SetKnownBackupReporter got %q)", fieldVar, handed)
}
}
func callsIn(n ast.Node) map[string]bool {
+1 -1
View File
@@ -43,7 +43,7 @@ func main() {
func run() error {
var (
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | agent_config_update")
op = flag.String("op", "", "op class to sign, e.g. storage_wipe | guest_destroy | decommission | agent_update | os_docker_step | os_pve_step | os_kernel_step | agent_config_update")
host = flag.String("host", "", "target host_id (anti-retarget — the op runs ONLY on this host)")
guest = flag.String("guest", "", "target guest_id (\"\" = host-scoped op)")
keyID = flag.String("key-id", "", "key id of the signing key (must match a pinned agent signer)")
+5 -3
View File
@@ -111,15 +111,17 @@ Cmnd_Alias FELHOM_INTERMEDIARY = \
# docker inspect -f * — container running/health/image (read-only; `*` spans the -f template
# + container across spaces, spike-confirmed)
# systemctl restart <fixed unit> — re-run the golden's bootstrap (the only state change)
# tee <FIXED image file> — WRITE the ref; content is fed on STDIN (no shell, no interpolation),
# the agent strict-validates the ref (controllerImageRe) before the write.
# felhom-priv-apply controller-image <vmid> — WRITE the ref (R-861 (a) A1, `09` §3 decision 165): the ref goes on
# STDIN to the ROOT wrapper, which requires our registry + repository + an x.y.z tag
# and writes the guest file itself. The agent's own `tee` grant is GONE: before, a
# compromised agent could hand the guest's bootstrap ANY image (sudo cannot see stdin).
# Validated GO: felhom.eu/documentation/audits/SPIKE-controllerswap-narrow-grants-2026-06-29.md.
Cmnd_Alias FELHOM_CONTROLLERSWAP = \
/usr/sbin/pct ^exec [0-9]+ -- cat /etc/felhom-controller-image$, \
/usr/sbin/pct ^exec [0-9]+ -- docker image inspect gitea\.dooplex\.hu/admin/felhom-controller\:[0-9]+\.[0-9]+\.[0-9]+$, \
/usr/sbin/pct ^exec [0-9]+ -- docker inspect -f .+ (felhom-controller|cloudflared)$, \
/usr/sbin/pct ^exec [0-9]+ -- systemctl restart felhom-controller-bootstrap\.service$, \
/usr/sbin/pct ^exec [0-9]+ -- tee /etc/felhom-controller-image$
/usr/local/sbin/felhom-priv-apply ^controller-image [0-9]+$
# Stale-lock recovery (F2-b, v0.49.0). A host reboot DURING a vzdump backup leaves the guest with a
# `snapshot-delete`/`backup` lock + `onboot:1` then can't start it → the customer box stays DOWN. The
+39
View File
@@ -0,0 +1,39 @@
#!/bin/sh
# /etc/grub.d/42_felhom_oneshot — the kernel lane's one-shot ENTRIES (R-836, `09` §3 decision 172, `11` §5.11).
# Installed by the config bundle (felhom-os-apply BUNDLE_FILES), 0755 root. update-grub runs it.
#
# One menu entry per installed Proxmox kernel, id `felhom-oneshot-<version>`, booted ONLY when 01_felhom_oneshot found
# the flag naming it. It is the normal entry plus option C (decision 172): softlockup_panic=1 hardlockup_panic=1
# hung_task_panic=1 panic=10 — a lockup the kernel can detect becomes a panic, and a panic restarts the box in 10 s into
# the default (the old kernel). A true dead freeze still needs a person (spike candidate 3 failed on all three boxes).
# It sorts AFTER 10_linux, so it is never entry 0 and never the default.
#
# No vfat ESP at /boot/efi → prints nothing (no flag can name these entries).
set -e
prefix="/usr"
exec_prefix="/usr"
datarootdir="/usr/share"
. "$datarootdir/grub/grub-mkconfig_lib"
esp_uuid=$(findmnt -n -o UUID,FSTYPE /boot/efi 2>/dev/null | awk '$2 == "vfat" { print $1 }')
[ -n "$esp_uuid" ] || exit 0
kernels=$(ls /boot/vmlinuz-*-pve 2>/dev/null | sed 's#^/boot/vmlinuz-##' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$' || true)
[ -n "$kernels" ] || exit 0
case "${GRUB_DEVICE}" in
/dev/mapper/*|/dev/dm-*|"") root_arg="root=${GRUB_DEVICE}" ;;
*) if [ -n "${GRUB_DEVICE_UUID}" ]; then root_arg="root=UUID=${GRUB_DEVICE_UUID}"; else root_arg="root=${GRUB_DEVICE}"; fi ;;
esac
[ -n "${GRUB_DEVICE}" ] || root_arg="root=$(findmnt -n -o SOURCE /)"
rel=$(make_system_path_relative_to_its_root /boot)
prep=$(prepare_grub_to_access_device "$(${grub_probe:-grub-probe} --target=device /boot)" | sed 's/^/ /')
for k in $kernels; do
[ -f "/boot/initrd.img-$k" ] || continue
cat <<EOF
menuentry 'Felhom one-shot: $k' --class proxmox --id felhom-oneshot-$k {
insmod gzio
$prep
echo 'Loading Linux $k (felhom one-shot) ...'
linux $rel/vmlinuz-$k $root_arg ro ${GRUB_CMDLINE_LINUX} ${GRUB_CMDLINE_LINUX_DEFAULT} softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10
initrd $rel/initrd.img-$k
}
EOF
done
+36
View File
@@ -0,0 +1,36 @@
#!/bin/sh
# /etc/grub.d/01_felhom_oneshot — the kernel lane's ONE-SHOT boot (R-836, `09` §3 decisions 164 + 172, `11` §5.11).
# Installed by the config bundle (felhom-os-apply BUNDLE_FILES), 0755 root. update-grub runs it; it prints GRUB script.
#
# At boot, GRUB reads `felhom_next` from an environment block on the ESP (vfat — GRUB can rewrite a file there; it
# cannot on the LVM /boot, R-836), CLEARS it, and — only if it names an installed kernel — boots that kernel's one-shot
# entry (42_felhom_oneshot) instead of the default. The next boot uses the default again whatever happens: a new kernel
# that panics comes back on the old one by itself. felhom-os-apply writes the flag (mode apply) and never the default
# for a new kernel. Measured on the Tester 1 VM, demo-felhom and demo-hp (Secure Boot on):
# `audits/kernel-spike-2026-10-07/` (candidate 2).
#
# No vfat ESP at /boot/efi, or no Proxmox kernel → prints nothing (the box boots exactly as before).
set -e
esp_uuid=$(findmnt -n -o UUID,FSTYPE /boot/efi 2>/dev/null | awk '$2 == "vfat" { print $1 }')
[ -n "$esp_uuid" ] || exit 0
kernels=$(ls /boot/vmlinuz-*-pve 2>/dev/null | sed 's#^/boot/vmlinuz-##' | grep -E '^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$' || true)
[ -n "$kernels" ] || exit 0
cat <<EOF
# felhom kernel lane: a one-shot kernel named on the ESP, read and cleared before the menu
insmod part_gpt
insmod fat
search --no-floppy --fs-uuid --set=felhom_esp $esp_uuid
if [ -f (\$felhom_esp)/EFI/felhom/oneshot.env ]; then
load_env -f (\$felhom_esp)/EFI/felhom/oneshot.env felhom_next
if [ "\${felhom_next}" ]; then
set felhom_boot="\${felhom_next}"
set felhom_next=
save_env -f (\$felhom_esp)/EFI/felhom/oneshot.env felhom_next
EOF
for k in $kernels; do
printf ' if [ "${felhom_boot}" = "%s" ]; then set default="felhom-oneshot-%s"; fi\n' "$k" "$k"
done
cat <<EOF
fi
fi
EOF
+5 -3
View File
@@ -16,8 +16,10 @@ Cmnd_Alias FELHOM_OP_REPAIR = \
/usr/bin/systemctl reset-failed felhom-sshd, \
/usr/bin/systemctl restart felhom-sshd, \
/usr/sbin/pct list, \
/usr/sbin/pct start [0-9]*, \
/usr/sbin/pct stop [0-9]*, \
/usr/sbin/pct unlock [0-9]*
/usr/sbin/pct ^start [0-9]+$, \
/usr/sbin/pct ^stop [0-9]+$, \
/usr/sbin/pct ^unlock [0-9]+$
# R-861 (b) B2 (`09` §3 decision 165, hygiene): one numeric vmid per pct verb, anchored — the old glob `[0-9]*` also
# matched spaces, so `pct stop 9201 --skiplock 1` passed. Pinned by TestFelhomOpSudoersPctIsExact.
felhom-op ALL=(root) NOPASSWD: FELHOM_OP_REPAIR
+554 -19
View File
@@ -21,6 +21,11 @@
# or, for an unsigned ring-0 step, the root-owned TRUST_FILE saying `"ring0_slow_lane": true` (set by hand on the demo
# boxes only). The agent's own config is NOT trusted for either: the agent can write it. A Docker step also needs
# `live-restore` ON (R15) — without it every container restarts.
# Agent v0.151.0 (R-812 option A, `09` §3 decision 163) adds the layer "pve": the HOST's Proxmox USERSPACE packages,
# lane "slow" only, origin "Proxmox Debian Repository" only, never a kernel / boot / firmware / microcode name (R14 —
# the kernel is R-836's lane), no removal, no undo, a NEW package only from PVE_NEW_ALLOW, an appliance only (R12), and
# the same authority as the Docker step (R3: a signed `os_pve_step`, or the root-owned ring-0 mark). Select
# "pending-pve" (ring 0): every installed Proxmox-origin package with a pending upgrade, minus HOST_SLOW_RE.
#
# Modes (plan field "mode"):
# inventory `apt-get update`, then report what is installed (with origin), what is pending, and health.
@@ -88,8 +93,13 @@ OOMCHECK_TIMEOUTS = {"image": 10, "clock": 5, "run": 30, "inspect": 10, "events"
OOMCHECK_SETTLE = 2
# Kernel, boot and firmware packages are the SLOW lane on the host whatever their origin (`11` C3, §5.2): a host
# reboot is needed for them to take effect, and a bad one can stop the box from booting.
# proxmox-secure-boot-support is the Secure Boot meta-package: its only job is to pull shim-signed and the signed GRUB,
# so it belongs with them. Measured 2026-10-09 on demo-hp (Secure Boot on): left in the pve lane, its upgrade pulled
# shim-signed-common, the whole pve step was refused R6, and the night's kernel step was skipped with it
# (`audits/kernel-night-2026-10-08/`). Pinned by test_ring0_pending_pve_leaves_the_secure_boot_meta_out.
HOST_SLOW_RE = re.compile(r"^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|"
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr)")
r"pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr|"
r"proxmox-secure-boot-support)")
# restart_needed() leaves out processes whose cgroup line matches (grep basic regex). Host: the LXC guests' own
# processes (`0::/lxc/<vmid>/...`) -- NOT lxc-start itself, whose cgroup is `0::/lxc.monitor/<vmid>` (measured
# 2026-10-04 on demo-felhom: the old pattern "lxc" hid lxc-start with 20 deleted maps, so "reboot needed" stayed false
@@ -100,6 +110,12 @@ HOST_SERVICES = ["pveproxy", "pvedaemon", "pvestatd", "pve-cluster", "felhom-age
DOCKER_NAMES = ("containerd.io", "docker-buildx-plugin", "docker-ce", "docker-ce-cli", "docker-ce-rootless-extras",
"docker-compose-plugin")
DOCKER_ORIGIN = "Docker CE"
# The Proxmox package lane (R-812 option A): the origin apt prints for download.proxmox.com, and the ONLY new packages
# a pve step may add (measured on demo-felhom 2026-10-07: a full upgrade adds proxmox-firewall-data and the kernel;
# the kernel is refused by R14 whatever this list says).
PVE_ORIGIN = "Proxmox Debian Repository"
PVE_NEW_ALLOW = ("proxmox-firewall-data",)
PVE_SIGNED_OP = "os_pve_step"
# ROOT-OWNED trust anchors (the installer writes them; the demo boxes got them by hand, R-840). Never the agent's config.
TRUST_FILE = "/etc/felhom/os-trust.json" # {"host_id": "...", "ring0_slow_lane": false}
TRUST_SIGNERS = "/etc/felhom/operator-signers" # ssh allowed_signers: <key_id> namespaces="felhom-op-v1" <key>
@@ -113,6 +129,32 @@ DAEMON_JSON = "/etc/docker/daemon.json"
# wrapper restarts exactly the containers that mount one of these paths (never the apps, never the engine).
DOCKER_SOCKETS = ("/var/run/docker.sock", "/run/docker.sock")
CRASH_GUARD_STATE = "/var/lib/felhom-crash-guard/state.json"
# ---------- the kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11) ----------
# A new kernel boots ONCE through a flag in a GRUB environment block on the ESP (vfat — GRUB can rewrite it there; on
# the LVM /boot it cannot, R-836). The bundle's two GRUB generators read and clear the flag (01_felhom_oneshot) and
# give each installed kernel a one-shot entry with the lockup-to-panic options (42_felhom_oneshot, option C). The GRUB
# default is the kernel the box RUNS, pinned in KERNEL_DEFAULT_CFG; only "kernel-good" (after a healthy one-shot boot)
# moves it to the new kernel. Measured in the spike on the Tester 1 VM, demo-felhom and demo-hp (Secure Boot on):
# `audits/kernel-spike-2026-10-07/`.
KERNEL_OP = "os_kernel_step"
KERNEL_STATE = "/var/lib/felhom-kernel/state.json" # 0644 root: the step's phase (the agent reads it)
KERNEL_DEFAULT_CFG = "/etc/default/grub.d/zz-felhom-kernel-default.cfg" # sourced last: GRUB_DEFAULT = this kernel
ONESHOT_SNIPPET = "/etc/grub.d/01_felhom_oneshot" # bundle-owned: read + clear the flag, pick the entry
ONESHOT_ENTRIES = "/etc/grub.d/42_felhom_oneshot" # bundle-owned: the one-shot entries (option C options)
GRUB_CFG = "/boot/grub/grub.cfg"
ESP_MOUNT = "/boot/efi"
ONESHOT_ENV = ESP_MOUNT + "/EFI/felhom/oneshot.env"
ONESHOT_ARGS = "softlockup_panic=1 hardlockup_panic=1 hung_task_panic=1 panic=10"
KERNEL_PIN_FILE = "/etc/kernel/proxmox-boot-pin" # an operator's `proxmox-boot-tool kernel pin` — never fought
KVER_RE = re.compile(r"^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$")
KERNEL_IMAGE_RE = re.compile(r"^proxmox-kernel-([0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve)(-signed)?$")
# the packages a kernel step may UPGRADE (never add): the kernel series meta-package, the default-kernel meta, the boot
# helper and the firmware the kernel loads. Everything else in HOST_SLOW_RE (grub, shim, microcode, efibootmgr) stays out.
KERNEL_UPGRADE_RE = re.compile(r"^(proxmox-default-kernel|proxmox-kernel-[0-9]+\.[0-9]+|proxmox-kernel-helper|pve-firmware)$")
KERNEL_MIN_GAP = 20 * 3600 # never two kernel steps in one night
KERNEL_MODES = ("kernel-status", "kernel-reboot", "kernel-boot", "kernel-good", "kernel-revert", "kernel-cancel")
# phases: staged → oneshot → judging → good | reverting → self_reverted | revert_failed; oneshot → fell_back; staged → cancelled
KERNEL_ACTIVE = ("staged", "oneshot", "judging", "reverting")
# ---------- the config bundle (R-840, agent v0.143.0, `11` §5.4.2) ----------
# A signed `agent_config_update` job carries {agent_version, bundle_sha256}; the bundle is ONE JSON file built from this
@@ -159,6 +201,10 @@ BUNDLE_FILES = [
("/etc/systemd/system/felhom-crash-guard-check.service", "felhom-crash-guard-check.service", 0o644, "unit", "replace"),
("/etc/systemd/system/felhom-crash-guard-check.timer", "felhom-crash-guard-check.timer", 0o644, "unit", "replace"),
("/etc/felhom/crash-guard.conf", "crash-guard.conf", 0o644, "plain", "if-absent"),
# the kernel lane's two GRUB generators (R-836, `11` §5.11): they take effect at the next update-grub, which the
# first kernel step runs itself; with no flag on the ESP they change nothing about how the box boots.
("/etc/grub.d/01_felhom_oneshot", "felhom-grub-oneshot.sh", 0o755, "sh", "replace"),
("/etc/grub.d/42_felhom_oneshot", "felhom-grub-oneshot-entries.sh", 0o755, "sh", "replace"),
("/etc/systemd/system/felhom-agent.service", "felhom-agent.service", 0o644, "agent-unit", "replace"),
("/etc/systemd/system/felhom-agent-rollback.service", "felhom-agent-rollback.service", 0o644, "unit", "replace"),
("/etc/systemd/system/felhom-agent.service.d/felhom-agent-limits.conf", "felhom-agent-limits.conf", 0o644, "dropin", "replace"),
@@ -423,7 +469,8 @@ class Apply:
def check_plan(self, plan):
mode = plan.get("mode", "apply")
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update", "oom-check"):
if mode not in ("apply", "inventory", "health", "facts", "live-restore-on", "bundle", "agent_update", "oom-check") \
+ KERNEL_MODES:
raise Refused("R11", f"unknown mode {mode!r}")
if mode == "agent_update":
if plan.get("layer") != "host":
@@ -434,12 +481,18 @@ class Apply:
raise Refused("R11", "bundle is a host-layer mode")
return mode, "host", 0, "bundle"
layer = plan.get("layer")
if layer not in ("guest", "host", "docker"):
raise Refused("R12", f"layer {layer!r} is not guest, host or docker")
if layer not in ("guest", "host", "docker", "pve", "kernel"):
raise Refused("R12", f"layer {layer!r} is not guest, host, docker, pve or kernel")
if (mode in KERNEL_MODES) != (layer == "kernel" and mode not in ("apply", "health")):
raise Refused("R11", f"mode {mode!r} does not fit layer {layer!r}")
lane = plan.get("lane", "fast")
if layer == "docker" and lane != "slow":
raise Refused("R3", "the Docker engine is the slow lane (`11` §5.8); a fast-lane Docker plan is refused")
if layer != "docker" and lane != "fast":
if layer == "pve" and lane != "slow":
raise Refused("R3", "the Proxmox packages are the slow lane (`11` §5.10); a fast-lane pve plan is refused")
if layer == "kernel" and lane != "slow":
raise Refused("R3", "the kernel is the slow lane (`11` §5.11); a fast-lane kernel plan is refused")
if layer not in ("docker", "pve", "kernel") and lane != "fast":
raise Refused("R3", f"the {layer} layer has no slow lane in this release (kernel, Proxmox: `11` §8 step 6)")
if mode == "facts" and layer != "host":
raise Refused("R11", "facts is a host-layer mode (it reads the host and the guest)")
@@ -447,10 +500,12 @@ class Apply:
raise Refused("R11", "live-restore-on is a guest-layer mode")
if mode == "oom-check" and layer != "docker":
raise Refused("R11", "oom-check is a docker-layer mode (it checks the guest's Docker engine)")
if plan.get("undo") and layer != "docker":
if plan.get("undo") and layer != "docker": # the pve layer has no undo in this release (R-812 option A)
raise Refused("R5", "an undo (downgrade) exists only for the Docker layer, inside a signed job")
vmid = plan.get("vmid")
if not isinstance(vmid, int) or isinstance(vmid, bool) or vmid <= 0:
if layer == "kernel" and mode in KERNEL_MODES and mode != "kernel-reboot" and vmid == 0:
pass # after a boot the guest may not run (that is what is judged); these modes never touch it
elif not isinstance(vmid, int) or isinstance(vmid, bool) or vmid <= 0:
raise Refused("R11", f"vmid must be a positive integer, got {vmid!r}")
rid = plan.get("release_id", "")
if not isinstance(rid, str) or not re.match(r"^[A-Za-z0-9._:-]{1,80}$", rid):
@@ -458,17 +513,24 @@ class Apply:
if plan.get("allow_new"):
raise Refused("R6", "allow_new is a slow-lane field; the fast lane never adds a package")
select = plan.get("select", "listed")
if select not in ("listed", "pending-fast", "pending-docker"):
if select not in ("listed", "pending-fast", "pending-docker", "pending-pve", "pending-kernel"):
raise Refused("R11", f"unknown select {select!r}")
if (select == "pending-docker") != (layer == "docker" and select != "listed"):
if select == "pending-docker" or layer == "docker":
raise Refused("R11", f"select {select!r} does not fit layer {layer!r}")
if (select == "pending-pve") != (layer == "pve" and select != "listed"):
raise Refused("R11", f"select {select!r} does not fit layer {layer!r}")
if (select == "pending-kernel") != (layer == "kernel" and select != "listed"):
raise Refused("R11", f"select {select!r} does not fit layer {layer!r}")
ek = plan.get("expect_kver")
if ek is not None and (layer != "kernel" or not isinstance(ek, str) or not KVER_RE.match(ek)):
raise Refused("R11", f"expect_kver {ek!r} is not a kernel version of the kernel layer")
pk = plan.get("packages", [])
if not isinstance(pk, list):
raise Refused("R11", "packages must be a list")
if mode == "apply" and select == "listed" and not pk:
raise Refused("R11", "packages must be a non-empty list in apply mode (select listed)")
if select in ("pending-fast", "pending-docker") and pk:
if select in ("pending-fast", "pending-docker", "pending-pve", "pending-kernel") and pk:
raise Refused("R11", f"select {select} takes no package list")
seen = set()
for e in pk:
@@ -488,6 +550,19 @@ class Apply:
continue
if n in DOCKER_NAMES:
raise Refused("R2", f"{n} is a Docker package — the slow lane (`11` §5.8), never in a {layer} plan")
if layer == "kernel":
if o != PVE_ORIGIN:
raise Refused("R2", f"{n}: origin {o!r} is not {PVE_ORIGIN!r} (the kernel layer)")
if not (KERNEL_UPGRADE_RE.match(n) or KERNEL_IMAGE_RE.match(n)):
raise Refused("R23", f"{n} is not a kernel package (a kernel image, the kernel meta-packages, "
f"proxmox-kernel-helper or pve-firmware)")
continue
if layer == "pve":
if o != PVE_ORIGIN:
raise Refused("R2", f"{n}: origin {o!r} is not {PVE_ORIGIN!r} (the pve layer)")
if HOST_SLOW_RE.match(n):
raise Refused("R14", f"{n} is a kernel / boot / firmware package — never the pve lane (R-836)")
continue
if o not in FAST_ORIGINS:
raise Refused("R2", f"{n}: origin {o!r} is not Debian / Debian-Security (the fast lane, `11` C3)")
if layer == "host" and HOST_SLOW_RE.match(n):
@@ -630,12 +705,13 @@ class Apply:
now = self.r.now()
self.r.write_nonces({k: v for k, v in seen.items() if v > now})
def docker_authority(self, plan):
"""R3 for the docker layer: returns (who, undo). A signed job binds the EXACT package list and the undo flag."""
def docker_authority(self, plan, op_name=SIGNED_OP):
"""R3 for the docker (and, op_name os_pve_step, the pve) layer: returns (who, undo). A signed job binds the EXACT
package list and the undo flag."""
trust = self.load_trust()
signed = plan.get("signed")
if signed:
params = self.verify_signed(signed, trust)
params = self.verify_signed(signed, trust, op_name=op_name)
want = sorted(f"{e.get('name')}={e.get('version')}" for e in params.get("packages") or [])
got = sorted(f"{e['name']}={e['version']}" for e in plan.get("packages", []))
if not want or want != got:
@@ -649,7 +725,7 @@ class Apply:
raise Refused("R3", "an undo (downgrade) needs a signed operator job")
if trust.get("ring0_slow_lane") is True:
return "ring0", False
raise Refused("R3", "a Docker step needs a signed operator job (ring 1) or this box's root-owned ring-0 mark")
raise Refused("R3", f"a {self.layer} slow-lane step needs a signed operator job (ring 1) or this box's root-owned ring-0 mark")
def live_restore(self):
rc, out, _ = self.g(["docker", "info", "--format", "{{.LiveRestoreEnabled}}"], timeout=60)
@@ -737,6 +813,17 @@ class Apply:
return v or "unknown"
h = {"debian": first(["cat", "/etc/debian_version"]), "kernel_running": first(["uname", "-r"])}
h["kernel_next_boot"], h["kernel_next_boot_source"] = self.kernel_next_boot()
try:
# the kernel lane (R-836): the default and the one-shot flag, read from grub.cfg and the ESP themselves
kv = Kernel(self, {}).view()
kv["setup_problems"] = Kernel(self, {}).setup_problems()
h["kernel_lane"] = kv
if kv.get("flag"):
h["kernel_next_boot"], h["kernel_next_boot_source"] = kv["flag"], "felhom one-shot flag (once; then the default)"
elif kv.get("default") not in (None, "unknown"):
h["kernel_next_boot"], h["kernel_next_boot_source"] = kv["default"], "grub.cfg default"
except Exception as e: # never cost the System page its other facts
h["kernel_lane"] = {"error": str(e)[:200]}
rc, out, _ = self.r.host(["apt-mark", "showhold"], 60)
h["held"] = sorted(out.split()) if rc == 0 else None
try:
@@ -795,8 +882,8 @@ class Apply:
# ---------- target helpers ----------
def x(self, argv, timeout=1800):
"""Run in the TARGET layer: the guest via pct exec, or the host directly."""
if self.layer == "host":
"""Run in the TARGET layer: the guest via pct exec, or the host directly (host, pve and kernel)."""
if self.layer in ("host", "pve", "kernel"):
return self.r.host(argv, timeout)
return self.r.guest(self.vmid, argv, timeout) # guest and docker both live in the customer guest
@@ -891,7 +978,7 @@ class Apply:
def restart_needed(self):
"""Processes still mapping deleted files, OUTSIDE containers (C11). Guest: outside docker; host: outside the
LXC guests (the host's /proc shows guest processes too)."""
skip = RESTART_SKIP_CGROUP["host" if self.layer == "host" else "guest"]
skip = RESTART_SKIP_CGROUP["host" if self.layer in ("host", "pve", "kernel") else "guest"]
script = ('for p in /proc/[0-9]*; do grep -q "(deleted)" $p/maps 2>/dev/null || continue; '
'grep -q "%s" $p/cgroup 2>/dev/null && continue; echo "${p#/proc/} $(cat $p/comm 2>/dev/null)"; done' % skip)
rc, out, _ = self.x(["sh", "-c", script], timeout=120)
@@ -965,8 +1052,12 @@ class Apply:
return Bundle(self).from_plan(plan)
if self.mode == "agent_update":
return self.agent_update(plan)
if self.layer == "host":
if self.layer in ("host", "pve", "kernel"):
self.check_appliance()
if self.layer == "kernel" and self.mode != "health":
# the kernel lane's own modes (R-836): most of them run while the guest is still starting after a boot, so
# the guest check is the stage's and the reboot's own (Kernel.run), not every mode's
return Kernel(self, plan).run()
self.check_guest(self.vmid)
log = self.r.log
if self.mode == "live-restore-on":
@@ -976,6 +1067,9 @@ class Apply:
self.report["oom_check"] = self.oom_check()
return 0
self.who, self.allow_downgrade = ("fast", False)
if self.layer == "pve" and self.mode == "apply":
self.who, self.allow_downgrade = self.docker_authority(plan, op_name=PVE_SIGNED_OP)
self.report["authority"] = self.who
if self.layer == "docker" and self.mode == "apply":
self.who, self.allow_downgrade = self.docker_authority(plan)
if self.live_restore() != "true":
@@ -988,7 +1082,7 @@ class Apply:
log(f"os-apply: START release={plan.get('release_id')} layer={self.layer}" +
(f":{self.vmid}" if self.layer != "host" else "") +
f" lane={plan.get('lane', 'fast')} mode={self.mode} select={self.select} packages={len(plan.get('packages', []))}" +
(f" authority={self.who}{' UNDO' if self.allow_downgrade else ''}" if self.layer == "docker" else ""))
(f" authority={self.who}{' UNDO' if self.allow_downgrade else ''}" if self.layer in ("docker", "pve") else ""))
if self.apt_lock_held():
raise Refused("R9", f"another apt/dpkg holds the lock on the {self.layer}")
self.report["health_before"] = self.health()
@@ -1012,6 +1106,8 @@ class Apply:
if self.layer == "docker":
rc_v, out_v, _ = self.g(["docker", "version", "--format", "{{.Server.Version}}"], timeout=60)
self.report["docker_engine"] = out_v.strip() if rc_v == 0 and out_v.strip() else "unknown"
if self.layer == "pve":
self.report["pve_manager"] = self.pve_manager()
self.report["reboot_scanned"] = "reboot_needed" in self.report
self.report["health_after"] = self.health()
if self.layer == "docker" and self.mode == "apply":
@@ -1161,10 +1257,26 @@ class Apply:
return [{"name": p["name"], "version": p["to"], "origin": DOCKER_ORIGIN} for p in pend
if p["from"] is not None and p["name"] in DOCKER_NAMES and self.origin_name(p["origin"]) == {DOCKER_ORIGIN}]
def pending_pve(self):
"""Ring 0 (select pending-pve): the newest pending version of each INSTALLED Proxmox-origin package, never a
kernel / boot / firmware / microcode name (HOST_SLOW_RE, R-836's lane), never another origin."""
rc, pend, remv, _ = self.simulate(["dist-upgrade"])
return [{"name": p["name"], "version": p["to"], "origin": PVE_ORIGIN} for p in pend
if p["from"] is not None and not HOST_SLOW_RE.match(p["name"]) and p["name"] not in DOCKER_NAMES
and self.origin_name(p["origin"]) == {PVE_ORIGIN}]
def pve_manager(self):
"""pveversion's pve-manager version ("unknown" when it cannot be read)."""
rc, out, _ = self.r.host(["pveversion"], 60)
m = re.match(r"^pve-manager/([^/\s]+)", out.strip()) if rc == 0 else None
return m.group(1) if m else "unknown"
def origin_ok(self, origin):
o = self.origin_name(origin)
if self.layer == "docker":
return o == {DOCKER_ORIGIN}
if self.layer == "pve":
return o == {PVE_ORIGIN}
return bool(o & set(FAST_ORIGINS))
def apply(self, plan):
@@ -1173,6 +1285,8 @@ class Apply:
packages = plan["packages"]
elif self.select == "pending-docker":
packages = self.pending_docker()
elif self.select == "pending-pve":
packages = self.pending_pve()
else:
packages = self.pending_fast()
cmp_op = "ne" if self.allow_downgrade else "gt"
@@ -1221,6 +1335,11 @@ class Apply:
raise Refused("R4", f"the plan would remove {', '.join(remv[:5])}")
want = dict(upgrade)
for p in sim:
if p["from"] is None and self.layer == "pve" and p["name"] in PVE_NEW_ALLOW and self.origin_ok(p["origin"]) \
and not HOST_SLOW_RE.match(p["name"]):
self.r.log(f"os-apply: NEW {p['name']}={p['to']} (on the pve lane's allow-list)")
self.report.setdefault("added", []).append({"name": p["name"], "version": p["to"]})
continue
if p["from"] is None:
raise Refused("R6", f"the plan would add a package that is not installed: {p['name']}")
if p["name"] not in want:
@@ -1231,7 +1350,7 @@ class Apply:
raise Refused("R5", f"{p['name']} would be downgraded {p['from']} -> {p['to']}")
if not self.origin_ok(p["origin"]):
raise Refused("R2", f"{p['name']} would come from {p['origin']}, not the {self.layer} layer's origin")
if self.layer == "host" and HOST_SLOW_RE.match(p["name"]):
if self.layer in ("host", "pve") and HOST_SLOW_RE.match(p["name"]):
raise Refused("R14", f"{p['name']} is a kernel / boot / firmware package — the host's slow lane")
need = self.download_bytes(args)
free = self.free_bytes()
@@ -1314,6 +1433,422 @@ class Apply:
self.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
def _iso(t):
return time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime(t))
def _parse_iso(s):
try:
return calendar.timegm(time.strptime(s, "%Y-%m-%dT%H:%M:%SZ"))
except (TypeError, ValueError):
return None
class Kernel:
"""The kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11). Refusal codes: R3 (authority), R4 (removal),
R6 (a package outside the step), R8 (space), R9 (locks), R12 (appliance), R20 (the box's boot setup cannot do a
one-shot), R21 (the crash guard is tripped or saw an unclean boot within its window), R22 (the step's phase does not
allow this mode), R23 (the kernel set is not one exact new kernel).
Modes (plan "mode", layer "kernel", lane "slow"):
apply STAGE: install the kernel set, keep the GRUB default on the kernel the box runs, write the flag.
Never reboots. select "pending-kernel" (ring 0, the root-owned mark) or "listed" (a signed
os_kernel_step). expect_kver: the kernel the hub told the household about — any other is R23.
kernel-reboot a STAGED step's reboot (the night leg, after the household was told): phase oneshot, then reboot.
kernel-boot after a boot: what became of the step (judging | fell_back | self_reverted | revert_failed).
kernel-good the one-shot boot was healthy: the new kernel becomes the GRUB default.
kernel-revert the one-shot boot was NOT healthy: reboot ONCE into the old kernel (still the default).
kernel-cancel drop a staged step: clear the flag (the package stays installed, the default never moved).
kernel-status read only."""
def __init__(self, apply, plan):
self.a, self.r, self.plan = apply, apply.r, plan
self.report = apply.report
self.log = apply.r.log
# ---------- reading the box ----------
def running(self):
rc, out, _ = self.r.host(["uname", "-r"], 30)
v = out.strip() if rc == 0 else ""
return v if KVER_RE.match(v) else ""
def state(self):
try:
s = json.loads(self.r.read_file(KERNEL_STATE))
return s if isinstance(s, dict) else {}
except (OSError, ValueError):
return {}
def save_state(self, s):
s["updated_at"] = _iso(self.r.now())
self.r.put_file(KERNEL_STATE, (json.dumps(s, indent=2, sort_keys=True) + "\n").encode(), 0o644)
def flag(self):
"""The one-shot flag: the kernel named, "" when the env block holds none, None when there is no env block."""
rc, out, _ = self.r.host(["grub-editenv", ONESHOT_ENV, "list"], 30)
if rc != 0:
return None
for l in out.splitlines():
if l.startswith("felhom_next="):
return l.split("=", 1)[1].strip()
return ""
def grub_cfg(self):
try:
return self.r.read_file(GRUB_CFG)
except OSError:
return ""
@staticmethod
def default_kver(cfg):
"""The kernel grub.cfg boots by default (00_header's `set default=`), or "unknown"."""
m = re.search(r'^\s*set default="(?:gnulinux-advanced-[^>"]*>)?gnulinux-([0-9][^"]*?-pve)-advanced-[^"]*"', cfg, re.M)
return m.group(1) if m else "unknown"
@staticmethod
def entry_id(cfg, kver):
"""The GRUB_DEFAULT value that names kver's normal entry, read from grub.cfg itself (10_linux's ids)."""
sub = re.search(r"\$menuentry_id_option '(gnulinux-advanced-[^']+)'", cfg)
m = re.search(r"\$menuentry_id_option '(gnulinux-" + re.escape(kver) + r"-advanced-[^']+)'", cfg)
if not m:
return None
return f"{sub.group(1)}>{m.group(1)}" if sub else m.group(1)
def setup_problems(self):
"""R20: why this box cannot do a one-shot boot (empty = it can). Measured shape: UEFI, a vfat ESP at /boot/efi,
/boot on the root filesystem, GRUB with fat + loadenv, the bundle's two generators, no hand pin."""
why = []
if not self.r.lexists("/sys/firmware/efi"):
why.append("the box does not boot UEFI")
rc, out, _ = self.r.host(["findmnt", "-n", "-o", "FSTYPE", ESP_MOUNT], 30)
if rc != 0 or out.strip() != "vfat":
why.append(f"{ESP_MOUNT} is not a mounted vfat ESP ({out.strip() or 'not mounted'})")
rc, out, _ = self.r.host(["findmnt", "-n", "-o", "TARGET", "/boot"], 30)
if rc == 0 and out.strip():
why.append("/boot is a separate filesystem (the one-shot entries assume /boot on the root filesystem)")
for m in ("fat", "loadenv"):
if not self.r.lexists(f"/usr/lib/grub/x86_64-efi/{m}.mod"):
why.append(f"GRUB has no {m} module")
for p in (ONESHOT_SNIPPET, ONESHOT_ENTRIES):
if not self.r.lexists(p):
why.append(f"{p} is missing (the config bundle installs it)")
if self.r.lexists(KERNEL_PIN_FILE):
why.append("a kernel is pinned by hand (proxmox-boot-tool kernel pin) — the lane never fights it")
return why
def guard(self):
try:
return json.loads(self.r.read_file(CRASH_GUARD_STATE))
except (OSError, ValueError):
return None
def check_guard(self):
"""R21: a kernel step only on a box whose crash guard is armed and saw no unclean boot within its window — so
the step's own reboots (clean), one crash and one self-revert (clean) can never reach the 3rd unclean boot that
leaves the box off (`11` §5.9). Pinned by test_felhom_crash_guard KernelStepCannotLeaveTheBoxOff."""
g = self.guard()
if not isinstance(g, dict):
raise Refused("R21", f"no crash guard state ({CRASH_GUARD_STATE}) — a kernel step needs the guard")
if g.get("tripped") or not g.get("armed"):
raise Refused("R21", "the crash guard is tripped — no kernel step until it re-arms")
if (g.get("unclean_boots_in_window") or 0) > 0:
raise Refused("R21", f"{g.get('unclean_boots_in_window')} unclean boot(s) within the guard's window — wait")
def view(self, st=None, cfg=None):
st = self.state() if st is None else st
cfg = self.grub_cfg() if cfg is None else cfg
return {"running": self.running() or "unknown", "default": self.default_kver(cfg), "flag": self.flag(),
"phase": st.get("phase", "none"), "from": st.get("from"), "to": st.get("to"),
"step_id": st.get("step_id"), "self_revert_used": bool(st.get("self_revert_used")), "vmid": st.get("vmid"),
"staged_at": st.get("staged_at"), "rebooted_at": st.get("rebooted_at"),
"result_at": st.get("result_at"), "reason": st.get("reason")}
# ---------- writing the box ----------
def write_default(self, kver):
"""Pin the GRUB default to kver's normal entry, regenerate grub.cfg, and PROVE it (the default read back)."""
cfg = self.grub_cfg()
eid = self.entry_id(cfg, kver)
if not eid:
raise Refused("R20", f"grub.cfg has no normal entry for {kver}")
body = ("# felhom kernel lane (R-836, `11` §5.11) — written by felhom-os-apply; the kernel that booted healthily\n"
f'GRUB_DEFAULT="{eid}"\n')
self.r.put_file(KERNEL_DEFAULT_CFG, body.encode(), 0o644)
rc, out, err = self.r.host(["update-grub"], 300)
got = self.default_kver(self.grub_cfg())
if rc != 0 or got != kver:
raise Refused("R20", f"update-grub rc={rc}: the default reads {got}, not {kver}: {(out + err).strip()[-200:]}")
self.log(f"os-apply: KERNEL default = {kver} (proved from grub.cfg)")
def set_flag(self, kver):
self.r.host(["mkdir", "-p", os.path.dirname(ONESHOT_ENV)], 30)
if self.flag() is None:
rc, out, err = self.r.host(["grub-editenv", ONESHOT_ENV, "create"], 30)
if rc != 0:
raise Refused("R20", f"cannot create the one-shot env block on the ESP: {(out + err).strip()[-200:]}")
rc, out, err = self.r.host(["grub-editenv", ONESHOT_ENV, "set", f"felhom_next={kver}"], 30)
if rc != 0 or self.flag() != kver:
raise Refused("R20", f"the one-shot flag did not read back as {kver}: {(out + err).strip()[-200:]}")
def clear_flag(self):
if self.flag():
self.r.host(["grub-editenv", ONESHOT_ENV, "unset", "felhom_next"], 30)
def reboot(self, why):
self.log(f"os-apply: KERNEL REBOOT — {why}")
rc, out, err = self.r.host(["systemctl", "reboot"], 60)
self.report["reboot_rc"] = rc
if rc != 0:
self.report["failed"] = {"rc": 3, "step": "reboot", "reason": (out + err).strip()[-200:]}
return 3
return 0
# ---------- the modes ----------
def run(self):
mode = self.a.mode
self.report["kernel_mode"] = mode
if mode == "kernel-status":
v = self.view()
v["setup_problems"] = self.setup_problems()
self.report["kernel"] = v
return 0
fn = {"apply": self.stage, "kernel-reboot": self.reboot_staged, "kernel-boot": self.after_boot,
"kernel-good": self.good, "kernel-revert": self.revert, "kernel-cancel": self.cancel}[mode]
rc = fn()
self.report["kernel"] = self.view()
return rc
def phase_is(self, st, *phases):
if st.get("phase") not in phases:
raise Refused("R22", f"the kernel step is {st.get('phase', 'none')!r}, not {' or '.join(phases)} — "
f"{self.a.mode} does not apply")
def stage(self):
a = self.a
st = self.state()
if st.get("phase") in KERNEL_ACTIVE:
raise Refused("R22", f"a kernel step is already {st['phase']} ({st.get('from')} -> {st.get('to')})")
last = _parse_iso(st.get("staged_at"))
if last is not None and self.r.now() - last < KERNEL_MIN_GAP:
raise Refused("R22", "a kernel step was staged within the last 20 hours — never two in one night")
why = self.setup_problems()
if why:
raise Refused("R20", "; ".join(why))
self.check_guard()
a.check_guest(a.vmid)
who, _ = a.docker_authority(self.plan, op_name=KERNEL_OP)
self.report["authority"] = who
old = self.running()
if not old:
raise Refused("R20", "the running kernel is not a Proxmox kernel version")
self.log(f"os-apply: START release={self.plan.get('release_id')} layer=kernel lane=slow mode=apply "
f"select={a.select} authority={who} running={old}")
if a.apt_lock_held():
raise Refused("R9", "another apt/dpkg holds the lock on the host")
self.report["health_before"] = a.health()
a.repair()
rc, out, err = a.x(APT_ENV + ["apt-get", "-q", "update"], timeout=600)
if rc != 0:
raise Refused("R7", f"apt-get update failed on the host: {(out + err).strip().splitlines()[-1:]}")
inst = a.installed()
if a.select == "pending-kernel":
_, pend, _, _ = a.simulate(["dist-upgrade"])
want = [(p["name"], p["to"]) for p in pend if p["from"] is not None and KERNEL_UPGRADE_RE.match(p["name"])
and a.origin_name(p["origin"]) == {PVE_ORIGIN}]
else:
want = [(e["name"], e["version"]) for e in self.plan["packages"]]
# upgrades of installed names (never a downgrade), and at most the listed new kernel image
args, upg = [], {}
for n, v in want:
if n in inst:
if a.dpkg_cmp(v, "gt", inst[n]):
upg[n] = v
args.append(f"{n}={v}")
elif KERNEL_IMAGE_RE.match(n):
args.append(f"{n}={v}")
else:
raise Refused("R6", f"{n} is not installed and is not a kernel image")
if not args:
self.report["upgraded"], self.report["outcome_hint"] = [], "nothing"
self.log("os-apply: DONE rc=0 upgraded=0 (no pending kernel)")
return 0
rc, sim, remv, text = a.simulate(["install", "--no-install-recommends"] + args)
if rc != 0:
raise Refused("R7", "the simulation failed: " + (text.strip().splitlines()[-1] if text.strip() else ""))
if remv:
raise Refused("R4", f"the kernel step would remove {', '.join(remv[:5])}")
images = []
listed = dict(want)
for p in sim:
if a.origin_name(p["origin"]) != {PVE_ORIGIN}:
raise Refused("R2", f"{p['name']} would come from {p['origin']}, not {PVE_ORIGIN!r}")
m = KERNEL_IMAGE_RE.match(p["name"])
if p["from"] is None:
if not m:
raise Refused("R6", f"the kernel step would add {p['name']}, which is not a kernel image")
if a.select == "listed" and listed.get(p["name"]) != p["to"]:
raise Refused("R23", f"the kernel step would add {p['name']}={p['to']}, not the signed set")
images.append((m.group(1), p["to"]))
continue
if p["name"] not in upg or p["to"] != upg[p["name"]]:
raise Refused("R6", f"the kernel step would touch {p['name']} ({p['to']}), which is not in the step")
if not a.dpkg_cmp(p["to"], "gt", p["from"]):
raise Refused("R5", f"{p['name']} would be downgraded {p['from']} -> {p['to']}")
if len(images) > 1:
raise Refused("R23", f"the kernel step would add {len(images)} kernels — one at a time")
# the target kernel: the new image, else the series meta-package's version (its image is already installed)
if images:
new = images[0][0]
if images[0][1] + "-pve" != new:
raise Refused("R23", f"the image version {images[0][1]} does not name the kernel {new}")
else:
metas = [(n, v) for n, v in upg.items() if re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n)]
if len(metas) != 1:
self.report["upgraded"], self.report["outcome_hint"] = [], "nothing"
self.log("os-apply: DONE rc=0 upgraded=0 (the pending set names no kernel to boot)")
return 0
new = metas[0][1] + "-pve"
if not KVER_RE.match(new) or not a.dpkg_cmp(new[:-4], "gt", old[:-4]):
raise Refused("R23", f"the kernel {new} is not newer than the running {old}")
ek = self.plan.get("expect_kver")
if ek and ek != new:
raise Refused("R23", f"the step would boot {new}, but the household was told about {ek}")
need = a.download_bytes(["install", "--no-install-recommends"] + args)
free = a.free_bytes()
if free >= 0 and free < max(MIN_FREE, 3 * need):
raise Refused("R8", f"free space {free} B is below max(500 MB, 3 x download {need} B)")
# 1. the default = the kernel the box RUNS (it booted healthily), proved from grub.cfg BEFORE the install
default_before = self.default_kver(self.grub_cfg())
self.write_default(old)
# 2. install (the kernel's own postinst runs update-grub; our default file keeps the default on `old`)
t0 = time.time()
rc, out, err = a.x(APT_ENV + ["apt-get", "-y", "-q"] + DPKG_OPTS + ["install", "--no-install-recommends"] + args)
a.x(["apt-get", "clean"])
if rc != 0:
_, aud, _ = a.x(["dpkg", "--audit"])
self.report["failed"] = {"rc": rc, "step": "install", "dpkg_audit": (aud.strip().splitlines() or ["clean"])[0],
"tail": (out + err).strip().splitlines()[-3:]}
self.log(f"os-apply: FAILED rc={rc} step=install (the default stays {old}; no flag written)")
return 3
self.report["upgraded"] = [{"name": x.split("=", 1)[0], "version": x.split("=", 1)[1]} for x in args]
self.report["seconds"] = round(time.time() - t0, 1)
# 3. prove: the image is there, the default is still `old`, the one-shot entry for `new` exists
cfg = self.grub_cfg()
if f"felhom-oneshot-{new}" not in cfg or self.default_kver(cfg) != old:
self.r.host(["update-grub"], 300)
cfg = self.grub_cfg()
for f in (f"/boot/vmlinuz-{new}", f"/boot/initrd.img-{new}"):
if not self.r.lexists(f):
self.report["failed"] = {"rc": 3, "step": "verify", "reason": f"{f} is missing after the install"}
return 3
if f"felhom-oneshot-{new}" not in cfg or "felhom_next" not in cfg:
self.report["failed"] = {"rc": 3, "step": "verify", "reason": f"grub.cfg has no one-shot entry for {new}"}
return 3
if self.default_kver(cfg) != old:
self.report["failed"] = {"rc": 3, "step": "verify",
"reason": f"the install moved the default to {self.default_kver(cfg)} — no flag written"}
return 3
# 4. the flag — the ONLY thing that makes the next boot use `new`, and only once
self.set_flag(new)
self.save_state({"phase": "staged", "step_id": self.plan.get("release_id"), "from": old, "to": new, "vmid": a.vmid,
"staged_at": _iso(self.r.now()), "authority": who, "default_before": default_before,
"packages": self.report["upgraded"], "self_revert_used": False})
self.report["reboot_needed"] = True
self.log(f"os-apply: KERNEL STAGED {old} -> {new} (default {old}, one-shot flag {new}); upgraded={len(args)} "
f"seconds={self.report['seconds']}")
return 0
def reboot_staged(self):
st = self.state()
self.phase_is(st, "staged")
if self.flag() != st.get("to"):
raise Refused("R22", f"the one-shot flag reads {self.flag()!r}, not {st.get('to')!r}")
if self.running() != st.get("from"):
raise Refused("R22", f"the box runs {self.running()!r}, not the step's old kernel {st.get('from')!r}")
cfg = self.grub_cfg()
if self.default_kver(cfg) != st["from"] or f"felhom-oneshot-{st['to']}" not in cfg:
raise Refused("R20", "grub.cfg no longer keeps the old default with a one-shot entry for the new kernel")
if self.setup_problems():
raise Refused("R20", "; ".join(self.setup_problems()))
self.check_guard()
self.a.check_guest(self.a.vmid)
st["health_before"] = self.a.health()
st["phase"], st["rebooted_at"] = "oneshot", _iso(self.r.now())
self.save_state(st)
return self.reboot(f"one-shot boot of {st['to']} (the default stays {st['from']})")
def after_boot(self):
st = self.state()
ph, run = st.get("phase"), self.running()
old, new = st.get("from"), st.get("to")
event = "none"
if ph in ("oneshot", "staged", "judging") and run == new and new:
if ph != "judging":
st["phase"], st["judging_since"], event = "judging", _iso(self.r.now()), "judging"
else:
event = "judging"
elif ph in ("oneshot", "judging") and run == old:
# the new kernel did not come up, or crashed: GRUB already booted the default (the old kernel)
self.clear_flag()
st["phase"], st["result_at"], event = "fell_back", _iso(self.r.now()), "fell_back"
st["reason"] = "the box came back on the old kernel by itself (the new one did not boot, or crashed)"
elif ph == "reverting" and run == old:
st["phase"], st["result_at"], event = "self_reverted", _iso(self.r.now()), "self_reverted"
elif ph == "reverting" and run == new:
st["phase"], st["result_at"], event = "revert_failed", _iso(self.r.now()), "revert_failed"
st["reason"] = "the self-revert came back on the NEW kernel — never retried (one self-revert per step)"
if event not in ("none",) and st.get("phase") != ph:
self.save_state(st)
self.log(f"os-apply: KERNEL AFTER-BOOT {ph} -> {st['phase']} running={run} ({old} -> {new})")
self.report["kernel_event"] = event
if event == "judging":
self.report["health_before"] = st.get("health_before") # the agent judges the boot against it
return 0
def good(self):
st = self.state()
self.phase_is(st, "judging")
if self.running() != st.get("to"):
raise Refused("R22", f"the box runs {self.running()!r}, not the new kernel {st.get('to')!r}")
try:
self.write_default(st["to"])
except Refused:
# put the old default back — the box must never be left without a proved default
self.write_default(st["from"])
raise
self.clear_flag()
st["phase"], st["result_at"] = "good", _iso(self.r.now())
self.save_state(st)
self.log(f"os-apply: KERNEL GOOD {st['to']} is the default now (was {st['from']})")
return 0
def revert(self):
st = self.state()
self.phase_is(st, "judging")
if st.get("self_revert_used"):
raise Refused("R22", "this kernel step already used its one self-revert")
if self.running() != st.get("to"):
raise Refused("R22", f"the box runs {self.running()!r}, not the new kernel {st.get('to')!r}")
self.clear_flag()
cfg = self.grub_cfg()
if self.default_kver(cfg) != st.get("from"):
raise Refused("R20", f"the GRUB default reads {self.default_kver(cfg)}, not the old {st.get('from')} — "
f"no self-revert into an unknown kernel")
reason = self.plan.get("reason")
st["reason"] = reason[:300] if isinstance(reason, str) else "the one-shot boot was not healthy"
st["self_revert_used"], st["phase"], st["reverted_at"] = True, "reverting", _iso(self.r.now())
self.save_state(st)
return self.reboot(f"self-revert to {st['from']}: {st['reason']}")
def cancel(self):
st = self.state()
self.phase_is(st, "staged")
self.clear_flag()
st["phase"], st["result_at"], st["reason"] = "cancelled", _iso(self.r.now()), "cancelled before the reboot"
self.save_state(st)
self.log(f"os-apply: KERNEL CANCELLED {st.get('to')} (installed, never the default; flag cleared)")
return 0
class Bundle:
"""The config bundle (R-840, `11` §5.4.2): every root-owned file the installer's step 5 writes, installed as ONE
signed unit. Every check runs before the first write; a failed write or a failed self-check puts every previous
+50 -1
View File
@@ -17,6 +17,8 @@ Verbs (each one sudoers line, exact-match pattern):
wg /var/lib/felhom-agent/wg/wg-felhom.conf -> /etc/wireguard/wg-felhom.conf (0600)
sshd-config /var/lib/felhom-agent/felhom-sshd/sshd_config -> /etc/felhom-sshd/sshd_config
sshd-key /var/lib/felhom-agent/felhom-sshd/authorized_keys.felhom-op -> /etc/felhom-sshd/authorized_keys/felhom-op
controller-image <vmid> the ref on STDIN -> /etc/felhom-controller-image INSIDE guest <vmid> (R-861 (a) A1): only
our registry + our repository + an x.y.z tag; the agent no longer has a `tee` grant
--self-check prints "felhom-priv-apply ok verbs=..." (the bundle's self-check)
Exit codes: 0 installed (or already identical), 2 usage, 3 refused (content or source), 4 install failed.
@@ -39,7 +41,14 @@ WG_SRC, WG_DEST = STATE + "/wg/wg-felhom.conf", "/etc/wireguard/wg-felhom.conf"
SSHD_SRC, SSHD_DEST = STATE + "/felhom-sshd/sshd_config", "/etc/felhom-sshd/sshd_config"
KEY_SRC, KEY_DEST = STATE + "/felhom-sshd/authorized_keys.felhom-op", "/etc/felhom-sshd/authorized_keys/felhom-op"
MAX_BYTES = 64 * 1024
VERBS = ("unit", "dnsmasq", "wg", "sshd-config", "sshd-key")
VERBS = ("unit", "dnsmasq", "wg", "sshd-config", "sshd-key", "controller-image")
# R-861 (a) A1 (`09` §3 decision 165): the SAME pattern as the agent's controllerImageRe (internal/localapi/
# controllerswap.go) — a compromised agent cannot hand the guest's bootstrap any other image. Pinned by
# configs/test_felhom_priv_apply.py ControllerImage.
CONTROLLER_IMAGE_RE = re.compile(r"^gitea\.dooplex\.hu/admin/felhom-controller:[0-9]+\.[0-9]+\.[0-9]+$")
CONTROLLER_IMAGE_FILE = "/etc/felhom-controller-image"
CONTROLLER_IMAGE_MAX = 256
VMID_RE = re.compile(r"^[0-9]{1,9}$")
UNIT_NAME_RE = re.compile(r"^mnt-[A-Za-z0-9_.\\-]+\.(mount|automount)$")
DNSMASQ_TMP_RE = re.compile(r"^/tmp/felhom-resolver-[0-9]+\.conf$")
@@ -123,6 +132,16 @@ class Host:
pass
raise
def read_stdin(self, limit):
return sys.stdin.buffer.read(limit + 1)
def write_guest_image(self, vmid, data):
"""As root: `pct exec <vmid> -- tee <the fixed file>` with the checked ref on stdin (no shell)."""
r = subprocess.run(["/usr/sbin/pct", "exec", str(vmid), "--", "tee", CONTROLLER_IMAGE_FILE],
input=data, stdout=subprocess.DEVNULL, stderr=subprocess.PIPE, timeout=60)
if r.returncode != 0:
raise OSError(f"pct exec {vmid} tee exited {r.returncode}: {r.stderr.decode(errors='replace').strip()[:200]}")
def log(self, line):
print(line, file=sys.stderr)
try:
@@ -377,6 +396,34 @@ def plan(argv):
raise Refused("A1", f"wrong arguments for {v}")
def controller_image(rest, host):
"""R-861 (a) A1: read the ref on stdin, check it, write it INSIDE the guest as root."""
try:
if len(rest) != 1 or not VMID_RE.match(rest[0]):
raise Refused("A1", "usage: felhom-priv-apply controller-image <vmid> (the ref on stdin)")
raw = host.read_stdin(CONTROLLER_IMAGE_MAX)
if len(raw) > CONTROLLER_IMAGE_MAX:
raise Refused("I1", f"the image ref is longer than {CONTROLLER_IMAGE_MAX} bytes")
try:
text = raw.decode("ascii")
except UnicodeDecodeError:
raise Refused("I1", "the image ref is not ASCII")
ref = text[:-1] if text.endswith("\n") else text
if not CONTROLLER_IMAGE_RE.match(ref) or "\n" in ref:
raise Refused("I1", "the image ref is not gitea.dooplex.hu/admin/felhom-controller:<x.y.z>")
except Refused as e:
host.log(f"felhom-priv-apply: REFUSED [{e.rule}] controller-image {' '.join(rest)[:40]}: {e.reason}")
return 2 if e.rule == "A1" else 3
vmid = int(rest[0])
try:
host.write_guest_image(vmid, (ref + "\n").encode())
except (OSError, subprocess.SubprocessError) as e:
host.log(f"felhom-priv-apply: FAILED controller-image {vmid}: {e}")
return 4
host.log(f"felhom-priv-apply: WROTE controller-image {vmid} {ref}")
return 0
def main(argv, host=None):
host = host or Host()
if argv == ["--self-check"]:
@@ -396,6 +443,8 @@ def main(argv, host=None):
return 3
print("OK")
return 0
if argv and argv[0] == "controller-image":
return controller_image(argv[1:], host)
try:
verb, src, dest, mode, checker = plan(argv)
data = host.read_source(src)
+57
View File
@@ -139,5 +139,62 @@ class Guard(unittest.TestCase):
self.assertEqual(mode, 0o644)
class KernelStepCannotLeaveTheBoxOff(unittest.TestCase):
"""R-836 / `11` §5.11 Part B 4: a kernel step's planned reboot, one crash and one self-revert cannot add up to the
box staying off. The wrapper starts a step only when the guard is armed with NO unclean boot in its window
(felhom-os-apply Kernel.check_guard, R21); the planned reboot and the self-revert are orderly (`systemctl reboot` —
the clean-stop marker); so the step adds at most ONE unclean boot, and the box stays off only after the LIMIT-th
(3rd) within the hour. Red-proof: audits/kernel-lane-2026-10-07/A/redproof.txt."""
def setUp(self):
self.d = tempfile.TemporaryDirectory()
self.e = FakeEnv(self.d.name)
cg.main(["x", "boot"], self.e)
s = self.e.state()
self.assertTrue(s["armed"])
self.assertEqual(s["unclean_boots_in_window"], 0, "the wrapper's precondition (R21)")
def tearDown(self):
self.d.cleanup()
def planned(self, minutes):
cg.main(["x", "clean-stop"], self.e)
self.e.t += minutes * 60
cg.main(["x", "boot"], self.e)
def crash(self, minutes):
self.e.t += minutes * 60
cg.main(["x", "boot"], self.e)
def test_planned_reboot_one_crash_one_self_revert(self):
self.planned(2) # the step's one-shot reboot (orderly)
self.crash(3) # the new kernel crashes after the guard ran; the box restarts (panic=10)
self.planned(2) # the self-revert (orderly)
s = self.e.state()
self.assertFalse(s["tripped"], s)
self.assertTrue(s["armed"])
self.assertEqual(self.e.panic(), 10, "the box still restarts after a crash")
self.assertEqual(s["unclean_boots_in_window"], 1, "the step added exactly one unclean boot")
def test_a_panic_before_userspace_is_not_even_counted(self):
# the one-shot kernel panics before the guard's unit runs (measured: rdinit= and init= missing): the planned
# reboot's clean-stop marker is still there when the old kernel boots, so this boot counts as clean.
cg.main(["x", "clean-stop"], self.e)
self.e.t += 120 # the panicking boot: no userspace, the guard never ran
cg.main(["x", "boot"], self.e)
s = self.e.state()
self.assertEqual(s["unclean_boots_in_window"], 0, s)
self.assertEqual(self.e.panic(), 10)
def test_the_box_stays_off_only_after_two_more_crashes_than_the_step_makes(self):
self.planned(2)
self.crash(3) # the step's one crash
self.planned(2) # the self-revert
self.crash(5) # an UNRELATED crash within the hour: the guard trips (the 3rd would leave it off)
s = self.e.state()
self.assertTrue(s["tripped"])
self.assertEqual(s["unclean_boots_in_window"], 2, "two unclean boots: one from the step, one not")
if __name__ == "__main__":
unittest.main()
+682 -1
View File
@@ -232,6 +232,8 @@ class Fake:
return (0, self.daemon_json, "") if self.daemon_json is not None else (1, "", "No such file")
if cmd == "cat" and a[1] == "/etc/debian_version":
return 0, "13.7\n", ""
if cmd == "pveversion":
return 0, f"pve-manager/{self.installed.get('pve-manager', '9.2.2')}/abcdef (running kernel: 7.0.14-20-pve)\n", ""
if cmd == "uname":
return 0, "7.0.14-20-pve\n", ""
if cmd == "apt-mark":
@@ -284,7 +286,7 @@ class Fake:
n, v = x.split("=", 1)
if v not in self.avail(n):
return 100, "", f"E: Version '{v}' for '{n}' was not found"
origin = "Docker CE:trixie" if n in osapply.DOCKER_NAMES else SEC if n == "openssl" else DEB
origin = getattr(self, "origins", {}).get(n) or ("Docker CE:trixie" if n in osapply.DOCKER_NAMES else SEC if n == "openssl" else DEB)
out += f"Inst {n} [{self.installed[n]}] ({v} {origin} [amd64])\n"
out += "".join(l + "\n" for l in self.extra_sim)
return 0, out, ""
@@ -1324,5 +1326,684 @@ class OOMCheck(unittest.TestCase):
self.assertEqual((rc, rep["refused"]["code"]), (2, "R11"), rep)
PVE = "Proxmox Debian Repository:stable"
PVE_SET = [{"name": "pve-manager", "version": "9.2.21", "origin": "Proxmox Debian Repository"},
{"name": "libpve-common-perl", "version": "9.1.9", "origin": "Proxmox Debian Repository"}]
def pve_fake(signed=None, ring0=False):
f = Fake()
f.installed.update({"pve-manager": "9.2.2", "libpve-common-perl": "9.1.1", "proxmox-kernel-helper": "9.0.4"})
f.live["pve-manager"] = {"9.2.21", "9.2.2"}
f.live["libpve-common-perl"] = {"9.1.9", "9.1.1"}
f.origins = {"pve-manager": PVE, "libpve-common-perl": PVE, "proxmox-kernel-helper": PVE, "shim-signed": PVE,
"proxmox-firewall-data": PVE}
f.plan = {"release_id": "os-pve-t1", "layer": "pve", "lane": "slow", "vmid": 9201, "mode": "apply",
"packages": [dict(p) for p in PVE_SET]}
if ring0:
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": True})
if signed is not None:
f.plan["signed"] = signed
return f
class PVELane(unittest.TestCase):
"""R-812 option A (`09` §3 decision 163): the host's Proxmox USERSPACE packages, slow lane, no kernel / boot /
firmware / microcode (R14), no removal, a new package only from PVE_NEW_ALLOW. Red-proof: audits/day-2026-10-07/B/."""
def refused(self, f, code):
rc, rep = run(f)
self.assertEqual(rc, 2, rep)
self.assertEqual(rep["refused"]["code"], code, rep)
self.assertEqual(f.installed["pve-manager"], "9.2.2", "nothing may be installed on a refusal")
return rep
# THE RED TEST (design-R-812 §5): before the pve layer existed this plan was refused R12.
def test_pve_manager_plan_is_installed_on_the_host(self):
f = pve_fake(signed=signed_job(packages=PVE_SET, op="os_pve_step"))
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(f.installed["pve-manager"], "9.2.21")
self.assertEqual(rep["authority"], "signed")
self.assertEqual(rep["pve_manager"], "9.2.21", "pveversion after the step is reported")
inst = [c for c in f.calls if c[0] == "host" and "install" in c[1] and "-s" not in c[1] and "-f" not in c[1]
and "--print-uris" not in c[1]]
self.assertTrue(inst, "the pve layer installs on the HOST")
self.assertFalse([c for c in f.calls if c[0] == "guest" and "install" in c[2]], "nothing installed in the guest")
self.assertEqual(sorted(rep["health_after"]["host_services"]), sorted(osapply.HOST_SERVICES))
def test_kernel_in_a_pve_plan_is_refused(self):
f = pve_fake(ring0=True)
f.plan["packages"].append({"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"})
self.refused(f, "R14")
def test_shim_in_a_pve_plan_is_refused(self):
f = pve_fake(ring0=True)
f.plan["packages"].append({"name": "shim-signed", "version": "1.47+pmx1", "origin": "Proxmox Debian Repository"})
self.refused(f, "R14")
def test_kernel_pulled_in_by_the_simulation_is_refused(self):
f = pve_fake(ring0=True)
f.installed["proxmox-kernel-helper"] = "9.0.4"
f.extra_sim = ["Inst proxmox-kernel-helper [9.0.4] (9.0.6 Proxmox Debian Repository:stable [all])"]
self.refused(f, "R6") # not in the plan — refused before the name check; R14 below when it IS in the plan
g = pve_fake(ring0=True)
g.live["proxmox-kernel-helper"] = {"9.0.6"}
g.plan["packages"] = [dict(PVE_SET[0])]
g.extra_sim = ["Inst proxmox-kernel-helper [9.0.4] (9.0.6 Proxmox Debian Repository:stable [all])"]
# a kernel-helper the plan did not name is R6; the R14 name check covers a listed one (test above)
self.refused(g, "R6")
def test_debian_package_in_a_pve_plan_is_refused(self):
f = pve_fake(ring0=True)
f.plan["packages"].append({"name": "libc6", "version": "2.41-12+deb13u4", "origin": "Debian"})
self.refused(f, "R2")
def test_debian_origin_in_the_pve_simulation_is_refused(self):
f = pve_fake(ring0=True)
f.origins["libpve-common-perl"] = DEB
self.refused(f, "R2")
def test_docker_package_in_a_pve_plan_is_refused(self):
f = pve_fake(ring0=True)
f.plan["packages"].append({"name": "docker-ce", "version": "5:29.8.2-1~debian.13~trixie", "origin": "Proxmox Debian Repository"})
self.refused(f, "R2")
def test_pve_in_the_fast_lane_is_refused(self):
f = pve_fake(ring0=True)
f.plan["lane"] = "fast"
self.refused(f, "R3")
def test_no_authority_is_refused(self):
self.refused(pve_fake(), "R3")
def test_docker_signed_op_does_not_authorize_a_pve_step(self):
self.refused(pve_fake(signed=signed_job(packages=PVE_SET, op="os_docker_step")), "R3")
def test_pve_on_a_byo_box_is_refused(self):
f = pve_fake(ring0=True)
f.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
self.refused(f, "R12")
def test_unlisted_new_package_is_refused(self):
f = pve_fake(ring0=True)
f.extra_sim = ["Inst proxmox-new-thing (1.0 Proxmox Debian Repository:stable [all])"]
self.refused(f, "R6")
def test_allow_listed_new_package_is_accepted(self):
f = pve_fake(ring0=True)
f.extra_sim = ["Inst proxmox-firewall-data (0.1 Proxmox Debian Repository:stable [all])"]
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(f.installed["pve-manager"], "9.2.21")
def test_allow_new_is_never_an_open_door_in_the_fast_lane(self):
f = Fake()
f.plan["layer"] = "host"
f.extra_sim = ["Inst proxmox-firewall-data (0.1 Proxmox Debian Repository:stable [all])"]
rc, rep = run(f)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R6"), rep)
def test_undo_is_refused(self):
f = pve_fake(signed=signed_job(packages=PVE_SET, op="os_pve_step"))
f.plan["undo"] = True
self.refused(f, "R5")
def test_ring0_pending_pve_takes_only_installed_proxmox_userspace(self):
f = pve_fake(ring0=True)
f.plan["select"], f.plan["packages"] = "pending-pve", []
f.installed.update({"linux-image-amd64": "6.12.1", "tailscale": "1.102.2"})
f.pending_sim = [
"Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
"Inst libpve-common-perl [9.1.1] (9.1.9 Proxmox Debian Repository:stable [all])",
"Inst proxmox-kernel-helper [9.0.4] (9.0.6 Proxmox Debian Repository:stable [all])",
"Inst proxmox-kernel-7.0.14-20-pve-signed (7.0.14-20 Proxmox Debian Repository:stable [amd64])",
"Inst proxmox-firewall-data (0.1 Proxmox Debian Repository:stable [all])",
"Inst libc6 [2.41-12+deb13u3] (2.41-12+deb13u4 Debian:13.7/stable [amd64])",
"Inst tailscale [1.102.2] (1.102.5 Tailscale:pkgs.tailscale.com [amd64])",
]
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["authority"], "ring0")
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["libpve-common-perl", "pve-manager"],
"pending-pve: installed, Proxmox-origin, never kernel/boot/firmware, never Debian or other origins")
self.assertEqual(f.installed["libc6"], "2.41-12+deb13u3")
# 2026-10-09, demo-hp (Secure Boot on): proxmox-secure-boot-support's upgrade pulls shim-signed-common; in the pve
# plan that refused the whole step (R6) and skipped the night's kernel step. It is a boot-chain package: left out.
# RED-PROOF: drop proxmox-secure-boot-support from HOST_SLOW_RE -> it is in the plan -> this test fails.
def test_ring0_pending_pve_leaves_the_secure_boot_meta_out(self):
f = pve_fake(ring0=True)
f.plan["select"], f.plan["packages"] = "pending-pve", []
f.installed.update({"proxmox-secure-boot-support": "9.0.1"})
f.pending_sim = [
"Inst pve-manager [9.2.2] (9.2.21 Proxmox Debian Repository:stable [amd64])",
"Inst proxmox-secure-boot-support [9.0.1] (9.0.2 Proxmox Debian Repository:stable [all])",
"Inst shim-signed-common [1.48+pmx1+16.1-1+pmx1] (1.51+pmx1+16.1-2+pmx1 Proxmox Debian Repository:stable [all])",
]
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["pve-manager"])
self.assertEqual(f.installed["proxmox-secure-boot-support"], "9.0.1")
def test_select_pending_pve_needs_the_pve_layer(self):
f = Fake()
f.plan["layer"], f.plan["select"], f.plan["packages"] = "host", "pending-pve", []
rc, rep = run(f)
self.assertEqual((rc, rep["refused"]["code"]), (2, "R11"), rep)
# ---------- the kernel lane (R-836, `09` §3 decision 172, `11` §5.11) ----------
U = "1af1fcc6-639c-416b-a7e5-c4470d41a502"
OLD, NEW = "7.0.2-6-pve", "7.0.14-22-pve"
KERNEL_SET = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-22", "origin": "Proxmox Debian Repository"},
{"name": "proxmox-kernel-7.0.14-22-pve-signed", "version": "7.0.14-22", "origin": "Proxmox Debian Repository"}]
class KFake(Fake):
"""A Proxmox host with GRUB on UEFI: /boot on the root LV, a vfat ESP, the bundle's two generators. update-grub is
emulated like the real 10_linux: WITHOUT the felhom default file the NEWEST kernel becomes the default (measured,
R-836 — that is the defect the lane exists for); with it, the kernel it names."""
def __init__(self):
super().__init__()
self.running = OLD
self.boot = {OLD}
self.efi, self.esp_fs, self.boot_mount, self.mods, self.snippets, self.pin = True, "vfat", "", True, True, False
self.env = None # None = no env block on the ESP; else {"felhom_next": ...}
self.tree = {} # files put_file wrote
self.reboots = []
self.update_grubs = 0
self.grub_runs_in_postinst = True
self.installed.update({"proxmox-kernel-7.0": "7.0.2-6", "proxmox-default-kernel": "2.1.0",
"pve-firmware": "3.18-3", "proxmox-kernel-7.0.2-6-pve-signed": "7.0.2-6",
"pve-manager": "9.2.21"})
self.live.update({"proxmox-kernel-7.0": {"7.0.14-22", "7.0.2-6"},
"proxmox-kernel-7.0.14-22-pve-signed": {"7.0.14-22"},
"proxmox-kernel-7.0.14-23-pve-signed": {"7.0.14-23"},
"pve-firmware": {"3.18-7", "3.18-3"}})
self.origins = {}
self.kernel_pending = ["Inst pve-firmware [3.18-3] (3.18-7 Proxmox Debian Repository:stable [all])",
"Inst proxmox-kernel-7.0.14-22-pve-signed (7.0.14-22 Proxmox Debian Repository:stable [amd64])",
"Inst proxmox-kernel-7.0 [7.0.2-6] (7.0.14-22 Proxmox Debian Repository:stable [amd64])",
"Inst pve-manager [9.2.21] (9.2.22 Proxmox Debian Repository:stable [amd64])",
"Inst libc6 [2.41-12+deb13u3] (2.41-12+deb13u4 Debian:13.7/stable [amd64])"]
self.plan = {"release_id": "kernel-t1", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": "apply",
"select": "pending-kernel", "packages": []}
self.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": True})
self.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": True, "tripped": False, "unclean_boots_in_window": 0})
self.cfg = self.render()
# -- GRUB --
def newest(self):
best = None
for k in self.boot:
if best is None or dpkg_cmp(k[:-4], "gt", best[:-4]):
best = k
return best
def render(self):
d = self.tree.get(osapply.KERNEL_DEFAULT_CFG)
if d:
dflt = re.search(r'GRUB_DEFAULT="([^"]+)"', d).group(1)
else:
dflt = f"gnulinux-advanced-{U}>gnulinux-{self.newest()}-advanced-{U}"
cfg = ('if [ "${next_entry}" ] ; then\n set default="${next_entry}"\nelse\n'
f' set default="{dflt}"\nfi\n'
f"submenu 'Advanced options' $menuentry_id_option 'gnulinux-advanced-{U}' {{\n")
for k in sorted(self.boot):
cfg += f" menuentry 'Proxmox VE, with Linux {k}' $menuentry_id_option 'gnulinux-{k}-advanced-{U}' {{ }}\n"
cfg += "}\n"
if self.snippets:
cfg += "### BEGIN /etc/grub.d/01_felhom_oneshot ###\nload_env felhom_next\n"
cfg += "".join(f"menuentry 'Felhom one-shot: {k}' --id felhom-oneshot-{k} {{ }}\n" for k in sorted(self.boot))
return cfg
def lexists(self, p):
if p == "/sys/firmware/efi":
return self.efi
if p.startswith("/usr/lib/grub/x86_64-efi/"):
return self.mods
if p in (osapply.ONESHOT_SNIPPET, osapply.ONESHOT_ENTRIES):
return self.snippets
if p == osapply.KERNEL_PIN_FILE:
return self.pin
m = re.match(r"^/boot/(vmlinuz|initrd\.img)-(.+)$", p)
if m:
return m.group(2) in self.boot
return p in self.tree
def put_file(self, path, data, mode):
self.tree[path] = data.decode()
def read_file(self, p):
if p == osapply.GRUB_CFG:
return self.cfg
if p in self.tree:
return self.tree[p]
return super().read_file(p)
def host(self, argv, timeout=600, stdin=None):
if argv[0] == "uname":
self.calls.append(("host", argv))
return 0, self.running + "\n", ""
if argv[0] == "findmnt":
self.calls.append(("host", argv))
if argv[-1] == osapply.ESP_MOUNT:
return (0, self.esp_fs + "\n", "") if self.esp_fs else (1, "", "")
return (0, self.boot_mount + "\n", "") if self.boot_mount else (1, "", "")
if argv[0] == "grub-editenv":
self.calls.append(("host", argv))
if argv[2] == "list":
return (1, "", "no such file") if self.env is None else \
(0, "".join(f"{k}={v}\n" for k, v in self.env.items()), "")
if argv[2] == "create":
self.env = {}
return 0, "", ""
if argv[2] == "set":
k, v = argv[3].split("=", 1)
self.env[k] = v
return 0, "", ""
if argv[2] == "unset":
self.env.pop(argv[3], None)
return 0, "", ""
if argv[0] == "update-grub":
self.calls.append(("host", argv))
self.update_grubs += 1
self.cfg = self.render()
return 0, "", ""
if argv[0] == "mkdir":
self.calls.append(("host", argv))
return 0, "", ""
if argv[:2] == ["systemctl", "reboot"]:
self.calls.append(("host", argv))
self.reboots.append(self.running)
return 0, "", ""
return super().host(argv, timeout, stdin)
def emulate(self, argv):
a = [x for x in argv if not re.match(r"^[A-Z_]+=", x) and x != "env"]
if a[0] == "apt-get" and "install" in a and "-s" not in a and "--print-uris" not in a and "-f" not in a:
if self.install_rc:
return self.install_rc, "", "E: boom"
for x in a:
if "=" in x and not x.startswith("-") and "::" not in x:
n, v = x.split("=", 1)
self.installed[n] = v
m = re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n)
if m:
self.installed[f"proxmox-kernel-{v}-pve-signed"] = v
if m or osapply.KERNEL_IMAGE_RE.match(n):
self.boot.add(v + "-pve")
if self.grub_runs_in_postinst:
self.cfg = self.render() # the kernel's postinst runs update-grub (zz-update-grub)
return 0, "Setting up proxmox-kernel ...\n", ""
return super().emulate(argv)
def sim(self, a):
if "--print-uris" in a:
return 0, "", ""
if "dist-upgrade" in a:
return 0, "\n".join(self.kernel_pending) + "\n", ""
out, named = "", set()
for x in a:
if "=" in x and not x.startswith("-") and "::" not in x:
named.add(x.split("=", 1)[0])
for x in a:
if "=" in x and not x.startswith("-") and "::" not in x:
n, v = x.split("=", 1)
if v not in self.avail(n):
return 100, "", f"E: Version '{v}' for '{n}' was not found"
o = self.origins.get(n, PVE)
out += f"Inst {n} [{self.installed[n]}] ({v} {o} [amd64])\n" if n in self.installed else f"Inst {n} ({v} {o} [amd64])\n"
if re.match(r"^proxmox-kernel-[0-9]+\.[0-9]+$", n):
img = f"proxmox-kernel-{v}-pve-signed"
if img not in self.installed and img not in named:
out += f"Inst {img} ({v} {PVE} [amd64])\n"
out += "".join(l + "\n" for l in self.extra_sim)
return 0, out, ""
def kfake(**kw):
f = KFake()
for k, v in kw.items():
setattr(f, k, v)
return f
def kmode(f, mode, **extra):
f.plan = {"release_id": "kernel-t1", "layer": "kernel", "lane": "slow", "vmid": 9201, "mode": mode, "packages": []}
f.plan.update(extra)
return run(f)
def staged(f=None):
f = f or kfake()
rc, rep = run(f)
assert rc == 0, rep
return f
class KernelLane(unittest.TestCase):
"""R-836 / `09` §3 decision 172: stage a kernel once through the ESP flag; the default moves only after a healthy
one-shot boot. Red-proof: audits/kernel-lane-2026-10-07/A/redproof.txt."""
def refused(self, f, code, mode=None, **extra):
rc, rep = kmode(f, mode, **extra) if mode else run(f)
self.assertEqual(rc, 2, rep)
self.assertEqual(rep["refused"]["code"], code, rep)
return rep
# --- the four red tests the brief names (Part A 3) ---
def test_installing_a_kernel_does_not_change_the_grub_default(self):
f = kfake()
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertIn(NEW, f.boot, "the new kernel was installed")
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD,
"the default must stay the kernel the box runs (R-836: an install made the new one default)")
self.assertIn(f"gnulinux-{OLD}-advanced-{U}", f.tree[osapply.KERNEL_DEFAULT_CFG])
self.assertEqual(f.env, {"felhom_next": NEW}, "the ONLY way to the new kernel is the one-shot flag")
self.assertEqual(f.reboots, [], "staging never reboots")
st = json.loads(f.tree[osapply.KERNEL_STATE])
self.assertEqual((st["phase"], st["from"], st["to"]), ("staged", OLD, NEW))
self.assertEqual(rep["kernel"]["default"], OLD)
def test_a_box_without_the_esp_flag_is_refused(self):
for attr, val, frag in (("esp_fs", "ext4", "vfat"), ("efi", False, "UEFI"), ("snippets", False, "bundle"),
("mods", False, "module"), ("boot_mount", "/boot", "separate"), ("pin", True, "pinned")):
f = kfake(**{attr: val})
rep = self.refused(f, "R20")
self.assertIn(frag, rep["refused"]["reason"], attr)
self.assertNotIn(NEW, f.boot, f"{attr}: nothing may be installed on a refusal")
self.assertIsNone(f.env, f"{attr}: no flag on a refusal")
def test_a_reboot_without_the_flag_is_refused(self):
f = staged()
f.env = {"felhom_next": ""}
self.refused(f, "R22", mode="kernel-reboot")
self.assertEqual(f.reboots, [])
def test_a_kernel_outside_the_approved_set_is_refused(self):
# a signed set that names only the meta-package: the image the sources pull in was never signed for
f = kfake()
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
f.plan.update(select="listed", packages=[dict(KERNEL_SET[0])],
signed=signed_job(packages=[KERNEL_SET[0]], op="os_kernel_step"))
self.refused(f, "R23")
self.assertNotIn(NEW, f.boot)
# a signed set names 7.0.14-22; the sources would ALSO bring 7.0.14-23
f2 = kfake()
f2.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
f2.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
signed=signed_job(packages=KERNEL_SET, op="os_kernel_step"))
f2.extra_sim = ["Inst proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23 Proxmox Debian Repository:stable [amd64])"]
self.refused(f2, "R23")
# a name that is not a kernel package at all
g = kfake()
g.plan.update(select="listed", packages=[{"name": "pve-manager", "version": "9.2.22", "origin": "Proxmox Debian Repository"}])
self.refused(g, "R23")
# the household was told about another kernel
h = kfake()
h.plan["expect_kver"] = "7.0.14-23-pve"
self.refused(h, "R23")
self.assertNotIn(NEW, h.boot)
def test_the_snippet_clears_the_flag_on_use(self):
"""01_felhom_oneshot: the flag is copied, CLEARED and SAVED before any `set default`, all inside the one branch
that runs only when the flag is set; a default is set only for an installed kernel's own entry."""
text = (HERE / "felhom-grub-oneshot.sh").read_text()
body = text[text.index("cat <<EOF"):]
i_copy = body.index('set felhom_boot="\\${felhom_next}"')
i_clear = body.index("set felhom_next=\n")
i_save = body.index("save_env -f (\\$felhom_esp)/EFI/felhom/oneshot.env felhom_next")
i_default = body.index('set default="felhom-oneshot-%s"')
self.assertLess(i_copy, i_clear)
self.assertLess(i_clear, i_save)
self.assertLess(i_save, i_default, "the flag must be cleared on disk BEFORE GRUB boots anything")
self.assertIn('if [ "${felhom_boot}" = "%s" ]', body, "a default only for a kernel that is installed")
self.assertIn(osapply.ONESHOT_ENV[len(osapply.ESP_MOUNT):], text, "the wrapper and GRUB name the same env file")
# --- the rest of the stage ---
def test_option_c_options_are_on_the_one_shot_entry_only(self):
text = (HERE / "felhom-grub-oneshot-entries.sh").read_text()
self.assertIn(osapply.ONESHOT_ARGS, text)
self.assertIn("--id felhom-oneshot-$k", text)
self.assertNotIn("set default", text, "the entries file never sets the default")
def test_both_generators_ride_the_bundle(self):
self.assertEqual(osapply.BUNDLE_DESTS[osapply.ONESHOT_SNIPPET][1:4], ("felhom-grub-oneshot.sh", 0o755, "sh"))
self.assertEqual(osapply.BUNDLE_DESTS[osapply.ONESHOT_ENTRIES][1:4], ("felhom-grub-oneshot-entries.sh", 0o755, "sh"))
def test_ring0_pending_kernel_takes_only_the_kernel_set(self):
f = kfake()
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["authority"], "ring0")
self.assertEqual(sorted(u["name"] for u in rep["upgraded"]), ["proxmox-kernel-7.0", "pve-firmware"])
self.assertEqual(f.installed["pve-manager"], "9.2.21", "a Proxmox userspace package is the pve lane's")
self.assertEqual(f.installed["libc6"], "2.41-12+deb13u3", "a Debian package is the fast lane's")
def test_signed_listed_set_is_installed(self):
f = kfake()
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
f.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
signed=signed_job(packages=KERNEL_SET, op="os_kernel_step"))
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["authority"], "signed")
self.assertEqual(f.env, {"felhom_next": NEW})
def test_no_authority_and_the_wrong_op_are_refused(self):
f = kfake()
f.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
self.refused(f, "R3")
g = kfake()
g.files[osapply.TRUST_FILE] = json.dumps({"host_id": "demo-hp-bb76ea", "ring0_slow_lane": False})
g.plan.update(select="listed", packages=[dict(p) for p in KERNEL_SET],
signed=signed_job(packages=KERNEL_SET, op="os_pve_step"))
self.refused(g, "R3")
def test_fast_lane_byo_and_wrong_select_are_refused(self):
f = kfake()
f.plan["lane"] = "fast"
self.refused(f, "R3")
g = kfake()
g.files[osapply.INSTALL_STATE] = json.dumps({"mode": "byo"})
self.refused(g, "R12")
h = kfake()
h.plan["select"] = "pending-pve"
self.refused(h, "R11")
k = kfake()
k.plan.update(layer="host", lane="fast", mode="kernel-status")
self.refused(k, "R11")
def test_the_crash_guard_must_be_armed_and_quiet(self):
f = kfake()
f.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": False, "tripped": True, "unclean_boots_in_window": 2})
self.refused(f, "R21")
g = kfake()
g.files[osapply.CRASH_GUARD_STATE] = json.dumps({"armed": True, "tripped": False, "unclean_boots_in_window": 1})
self.refused(g, "R21")
h = kfake()
del h.files[osapply.CRASH_GUARD_STATE]
self.refused(h, "R21")
self.assertNotIn(NEW, h.boot)
def test_never_two_steps_in_one_night(self):
f = staged()
self.refused(f, "R22") # still staged
g = staged()
st = json.loads(g.tree[osapply.KERNEL_STATE])
st["phase"] = "good"
g.tree[osapply.KERNEL_STATE] = json.dumps(st)
g.clock += 3600
self.refused(g, "R22") # done, but within 20 h
def test_removal_new_non_kernel_two_kernels_and_older_are_refused(self):
f = kfake(extra_sim=["Remv pve-firmware [3.18-3]"])
self.refused(f, "R4")
g = kfake(extra_sim=["Inst proxmox-new-thing (1.0 Proxmox Debian Repository:stable [all])"])
self.refused(g, "R6")
h = kfake(extra_sim=["Inst proxmox-kernel-7.0.14-23-pve-signed (7.0.14-23 Proxmox Debian Repository:stable [amd64])"])
self.refused(h, "R23")
k = kfake(running="7.0.14-23-pve")
k.boot = {"7.0.14-23-pve"}
k.cfg = k.render()
self.refused(k, "R23")
m = kfake()
m.origins = {"proxmox-kernel-7.0": DEB}
self.refused(m, "R2")
# R-898: ring 0 stages EXACTLY the told kernel (select listed, the set derived from it) — even when the sources
# offer a newer one by night; a told version that can no longer be installed is refused BEFORE any change (R7).
def test_ring0_listed_installs_the_told_kernel_not_the_newest(self):
f = kfake()
f.live["proxmox-kernel-7.0"] = {"7.0.14-20", "7.0.14-22", "7.0.2-6"}
f.live["proxmox-kernel-7.0.14-20-pve-signed"] = {"7.0.14-20"}
told = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"},
{"name": "proxmox-kernel-7.0.14-20-pve-signed", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"}]
f.plan.update(select="listed", packages=told, expect_kver="7.0.14-20-pve")
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["authority"], "ring0")
self.assertEqual(f.env, {"felhom_next": "7.0.14-20-pve"})
self.assertEqual(f.installed["proxmox-kernel-7.0"], "7.0.14-20")
self.assertNotIn("7.0.14-22-pve", f.boot)
def test_ring0_told_kernel_gone_is_refused_before_any_change(self):
f = kfake()
f.live["proxmox-kernel-7.0"] = {"7.0.14-22"} # 7.0.14-20 is no longer in the archive
told = [{"name": "proxmox-kernel-7.0", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"},
{"name": "proxmox-kernel-7.0.14-20-pve-signed", "version": "7.0.14-20", "origin": "Proxmox Debian Repository"}]
f.plan.update(select="listed", packages=told, expect_kver="7.0.14-20-pve")
rep = self.refused(f, "R7")
self.assertIsNone(f.env)
self.assertNotIn(osapply.KERNEL_DEFAULT_CFG, f.tree, "refused before the default was even pinned")
self.assertEqual(f.installed["proxmox-kernel-7.0"], "7.0.2-6")
def test_nothing_pending_changes_nothing(self):
f = kfake(kernel_pending=[])
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertEqual(rep["upgraded"], [])
self.assertIsNone(f.env)
self.assertNotIn(osapply.KERNEL_STATE, f.tree)
def test_a_failed_install_writes_no_flag(self):
f = kfake(install_rc=100)
rc, rep = run(f)
self.assertEqual(rc, 3, rep)
self.assertIsNone(f.env)
self.assertNotIn(osapply.KERNEL_STATE, f.tree)
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
def test_a_postinst_without_update_grub_is_regenerated_and_proved(self):
f = kfake(grub_runs_in_postinst=False)
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
self.assertIn(f"felhom-oneshot-{NEW}", f.cfg)
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
# --- the reboot, the boot, good / revert ---
def test_reboot_of_a_staged_step(self):
f = staged()
rc, rep = kmode(f, "kernel-reboot")
self.assertEqual(rc, 0, rep)
self.assertEqual(f.reboots, [OLD])
st = json.loads(f.tree[osapply.KERNEL_STATE])
self.assertEqual(st["phase"], "oneshot")
self.assertIn("host_services", st["health_before"])
def test_reboot_needs_a_staged_step(self):
self.refused(kfake(), "R22", mode="kernel-reboot")
def boot_into(self, f, kver):
f.running = kver
if f.env and f.env.get("felhom_next"):
f.env["felhom_next"] = "" # GRUB cleared it (01_felhom_oneshot)
return kmode(f, "kernel-boot")
def test_healthy_one_shot_becomes_the_default(self):
f = staged()
kmode(f, "kernel-reboot")
rc, rep = self.boot_into(f, NEW)
self.assertEqual((rc, rep["kernel_event"], rep["kernel"]["phase"]), (0, "judging", "judging"), rep)
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD, "judging does not move the default")
rc, rep = kmode(f, "kernel-good")
self.assertEqual(rc, 0, rep)
self.assertEqual(osapply.Kernel.default_kver(f.cfg), NEW)
self.assertEqual(rep["kernel"]["phase"], "good")
def test_a_panic_falls_back_and_is_recorded(self):
f = staged()
kmode(f, "kernel-reboot")
rc, rep = self.boot_into(f, OLD)
self.assertEqual((rc, rep["kernel_event"]), (0, "fell_back"), rep)
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD)
self.refused(f, "R22", mode="kernel-good")
def test_one_self_revert_then_never_again(self):
f = staged()
kmode(f, "kernel-reboot")
self.boot_into(f, NEW)
rc, rep = kmode(f, "kernel-revert", reason="the customer guest is not running")
self.assertEqual(rc, 0, rep)
self.assertEqual(f.reboots, [OLD, NEW])
self.assertEqual(osapply.Kernel.default_kver(f.cfg), OLD, "the self-revert boots the default, the old kernel")
rc, rep = self.boot_into(f, OLD)
self.assertEqual(rep["kernel_event"], "self_reverted")
# the same step can never revert again
st = json.loads(f.tree[osapply.KERNEL_STATE])
st["phase"] = "judging"
f.tree[osapply.KERNEL_STATE] = json.dumps(st)
f.running = NEW
self.refused(f, "R22", mode="kernel-revert")
self.assertEqual(len(f.reboots), 2)
def test_a_revert_that_comes_back_new_is_never_retried(self):
f = staged()
kmode(f, "kernel-reboot")
self.boot_into(f, NEW)
kmode(f, "kernel-revert")
rc, rep = self.boot_into(f, NEW)
self.assertEqual(rep["kernel_event"], "revert_failed")
self.refused(f, "R22", mode="kernel-revert")
def test_revert_refuses_an_unknown_default(self):
f = staged()
kmode(f, "kernel-reboot")
self.boot_into(f, NEW)
f.tree[osapply.KERNEL_DEFAULT_CFG] = f'GRUB_DEFAULT="gnulinux-advanced-{U}>gnulinux-9.9.9-1-pve-advanced-{U}"\n'
f.cfg = f.render()
self.refused(f, "R20", mode="kernel-revert")
self.assertEqual(len(f.reboots), 1)
def test_cancel_clears_the_flag(self):
f = staged()
rc, rep = kmode(f, "kernel-cancel")
self.assertEqual(rc, 0, rep)
self.assertFalse(f.env.get("felhom_next"), "the flag is gone")
self.assertEqual(rep["kernel"]["phase"], "cancelled")
def test_status_is_read_only_and_names_setup_problems(self):
f = kfake(esp_fs="ext4")
rc, rep = kmode(f, "kernel-status")
self.assertEqual(rc, 0, rep)
self.assertTrue(rep["kernel"]["setup_problems"])
self.assertEqual(f.tree, {})
self.assertFalse([c for c in f.calls if c[1][0] in ("update-grub", "systemctl", "apt-get")])
def test_facts_carry_the_kernel_lane(self):
f = staged()
f.plan = {"release_id": "facts", "layer": "host", "lane": "fast", "vmid": 9201, "mode": "facts", "packages": []}
rc, rep = run(f)
self.assertEqual(rc, 0, rep)
h = rep["facts"]["host"]
self.assertEqual(h["kernel_lane"]["phase"], "staged")
self.assertEqual(h["kernel_next_boot"], NEW)
self.assertIn("one-shot", h["kernel_next_boot_source"])
if __name__ == "__main__":
unittest.main()
+60
View File
@@ -56,6 +56,18 @@ class FakeHost:
def log(self, line):
self.logs.append(line)
# controller-image (R-861 (a) A1): the ref arrives on stdin and is written INSIDE the guest by root.
stdin = b""
guest_writes = None
def read_stdin(self, limit):
return self.stdin[:limit + 1]
def write_guest_image(self, vmid, data):
if self.guest_writes is None:
self.guest_writes = []
self.guest_writes.append((vmid, data))
LOCAL_UNIT = """# Managed by felhom-agent — do not edit by hand.
[Unit]
@@ -316,5 +328,53 @@ class Refuses(unittest.TestCase):
self.refused(h, ["wg", "/etc/shadow"], "A1")
class ControllerImage(unittest.TestCase):
"""R-861 (a) A1 (`09` §3 decision 165): the agent can no longer `tee` any image ref into the guest. The root verb
reads the ref on stdin, requires our registry + our repository + an x.y.z tag, and writes the guest file itself.
RED-PROOF: on the pre-A1 wrapper `controller-image` is not a verb (A1 usage, rc 2) — the accepted case fails."""
def go(self, ref, *argv):
h = FakeHost()
h.stdin = ref.encode() if isinstance(ref, str) else ref
return h, run(h, *(argv or ("controller-image", "9201")))
def test_our_controller_ref_is_written_in_the_guest(self):
h, rc = self.go("gitea.dooplex.hu/admin/felhom-controller:0.301.0\n")
self.assertEqual(rc, 0, h.logs)
self.assertEqual(h.guest_writes, [(9201, b"gitea.dooplex.hu/admin/felhom-controller:0.301.0\n")])
def test_a_foreign_image_is_refused(self):
for ref in ("docker.io/library/alpine:latest\n", "alpine\n",
"gitea.dooplex.hu/admin/felhom-controller:latest\n",
"gitea.dooplex.hu/admin/other:0.1.0\n",
"evil.example/admin/felhom-controller:0.301.0\n",
"gitea.dooplex.hu/admin/felhom-controller:0.301.0\nalpine\n",
"gitea.dooplex.hu/admin/felhom-controller:0.301.0 x\n",
"", "\n"):
h, rc = self.go(ref)
self.assertEqual(rc, 3, f"{ref!r} was accepted")
self.assertFalse(h.guest_writes, f"{ref!r} wrote the guest file")
self.assertTrue(any("[I1]" in l for l in h.logs), h.logs)
def test_oversize_stdin_is_refused(self):
h, rc = self.go("gitea.dooplex.hu/admin/felhom-controller:0.301.0" + " " * 300)
self.assertEqual(rc, 3)
self.assertFalse(h.guest_writes)
def test_vmid_must_be_numeric(self):
for argv in (("controller-image", "9201;id"), ("controller-image", "-1"), ("controller-image",),
("controller-image", "9201", "9202")):
h, rc = self.go("gitea.dooplex.hu/admin/felhom-controller:0.301.0\n", *argv)
self.assertIn(rc, (2, 3), argv)
self.assertFalse(h.guest_writes, argv)
def test_self_check_names_the_verb(self):
import io, contextlib
buf = io.StringIO()
with contextlib.redirect_stdout(buf):
pa.main(["--self-check"])
self.assertIn("controller-image", buf.getvalue())
if __name__ == "__main__":
unittest.main(verbosity=2)
+23
View File
@@ -103,6 +103,29 @@ func (s *BackupSuccessState) LastKnownSuccess(target string, vmid int) (time.Tim
return e.at, ok
}
// KnownBackupSuccesses returns every saved success as a host-report record (the collector's
// KnownBackupReporter, so the hub keeps its evidence across a restart). Only the tier, guest and start
// time are known; the archive name is not saved and stays empty. Sorted for a stable report.
func (s *BackupSuccessState) KnownBackupSuccesses() []hub.Backup {
if s == nil {
return nil
}
s.mu.Lock()
defer s.mu.Unlock()
out := make([]hub.Backup, 0, len(s.last))
for _, e := range s.last {
out = append(out, hub.Backup{TargetID: e.target, VMID: e.vmid, Success: true, CrashConsistent: true,
StartedAt: e.at.Format(time.RFC3339), UncoveredVolumes: []string{}})
}
sort.Slice(out, func(i, j int) bool {
if out[i].TargetID != out[j].TargetID {
return out[i].TargetID < out[j].TargetID
}
return out[i].VMID < out[j].VMID
})
return out
}
func (s *BackupSuccessState) saveLocked() error {
entries := make([]backupSuccessJSON, 0, len(s.last))
for _, e := range s.last {
+17
View File
@@ -57,3 +57,20 @@ func TestBackupSuccessState_CorruptFileIsEmpty(t *testing.T) {
t.Fatal("a corrupt file must read as nothing known")
}
}
// The saved successes come back as host-report records: success, tier, guest, start time (R-894 → the
// host report, 2026-10-09). A new state read from the same file gives the same records (the restart case).
func TestBackupSuccessState_KnownBackupSuccessesSurviveAReopen(t *testing.T) {
path := filepath.Join(t.TempDir(), "backup-success-state.json")
s := NewBackupSuccessState(path)
if err := s.RecordBackupSuccess("felhom-backup", hub.Backup{VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z"}); err != nil {
t.Fatal(err)
}
got := NewBackupSuccessState(path).KnownBackupSuccesses()
if len(got) != 1 || !got[0].Success || got[0].TargetID != "felhom-backup" || got[0].VMID != 9201 || got[0].StartedAt != "2026-10-09T02:40:16Z" {
t.Fatalf("got %+v", got)
}
if got[0].UncoveredVolumes == nil {
t.Fatal("uncovered_volumes must marshal as [], not null")
}
}
+60
View File
@@ -0,0 +1,60 @@
package backup
import (
"context"
"sort"
"sync"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// ForeignKeyLedger (R-366 slice 2, `09` §3 decision 168) holds, per backup tier, the whole-guest archives the
// restore-test pick skipped because another key wrote them (R-727 — an earlier install of this box). Since R-727 the
// skip was one INFO log line per archive and nothing else, so after a reinstall the operator was never told that the
// box's older whole-guest copies are unreadable to it. The host report carries this ledger; the hub raises ONE
// operator event when it changes.
//
// It reports nil until a tier has been evaluated since the agent started, so a restart does not read as "the set
// changed to empty" (the hub keeps its last state for an absent field).
type ForeignKeyLedger struct {
mu sync.Mutex
byTarget map[string]hub.ForeignKeyArchives
}
// NewForeignKeyLedger builds an empty ledger.
func NewForeignKeyLedger() *ForeignKeyLedger {
return &ForeignKeyLedger{}
}
func (l *ForeignKeyLedger) set(target string, n int, oldest, newest int64) {
l.mu.Lock()
defer l.mu.Unlock()
if l.byTarget == nil {
l.byTarget = map[string]hub.ForeignKeyArchives{}
}
e := hub.ForeignKeyArchives{Target: target, Count: n}
if n > 0 {
e.Oldest = time.Unix(oldest, 0).UTC().Format(time.RFC3339)
e.Newest = time.Unix(newest, 0).UTC().Format(time.RFC3339)
}
l.byTarget[target] = e
}
// ForeignKeyArchives implements hub.ForeignKeyArchiveReporter: nil before any evaluation; otherwise the tiers that
// hold such archives (`Tiers` empty, never nil, when none do), sorted by tier.
func (l *ForeignKeyLedger) ForeignKeyArchives(context.Context) *hub.ForeignKeyArchivesStanza {
l.mu.Lock()
defer l.mu.Unlock()
if l.byTarget == nil {
return nil
}
out := []hub.ForeignKeyArchives{}
for _, e := range l.byTarget {
if e.Count > 0 {
out = append(out, e)
}
}
sort.Slice(out, func(i, j int) bool { return out[i].Target < out[j].Target })
return &hub.ForeignKeyArchivesStanza{Tiers: out}
}
@@ -0,0 +1,69 @@
package backup
import (
"context"
"encoding/json"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
)
// R-366 slice 2 (`09` §3 decision 168) — the restore-test's skip of an archive written with another key stops being
// silent: the pick records, per tier, how many it skipped and their time range, and the host report carries it.
//
// COMPANION RED-PROOF (observed): remove the `r.foreign.set(...)` call from PickSettledRestoreCandidateOn → this fails
// with "after one evaluation the ledger must report felhom-pbs: 2 archives …; got []". Restored.
func TestR366_PickRecordsArchivesWrittenWithAnotherKey(t *testing.T) {
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", EncryptionKey: thisBoxKey}},
content: []proxmox.StorageContent{
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T17:27:32Z", Content: "backup", VMID: 9201, Size: 4774114206, CTime: 1789579652, Encrypted: earlierBox2},
{VolID: "felhom-pbs:backup/ct/9201/2026-09-16T21:59:54Z", Content: "backup", VMID: 9201, Size: 20811501236, CTime: 1789595994, Encrypted: earlierBox1},
{VolID: "felhom-pbs:backup/ct/9201/2026-09-29T19:37:07Z", Content: "backup", VMID: 9201, Size: 3490689830, CTime: 1790710627, Encrypted: thisBoxKey},
},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
l := NewForeignKeyLedger()
r.SetForeignKeyLedger(l)
if got := l.ForeignKeyArchives(context.Background()); got != nil {
t.Fatalf("before any evaluation the ledger must be nil (the hub keeps its state); got %v", got)
}
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Time{}); err != nil {
t.Fatal(err)
}
st := l.ForeignKeyArchives(context.Background())
want := hub.ForeignKeyArchives{Target: "felhom-pbs", Count: 2, Oldest: "2026-09-16T17:27:32Z", Newest: "2026-09-16T21:59:54Z"}
var got []hub.ForeignKeyArchives
if st != nil {
got = st.Tiers
}
if len(got) != 1 || got[0] != want {
t.Fatalf("after one evaluation the ledger must report felhom-pbs: 2 archives 2026-09-16T17:27:32Z…21:59:54Z; got %v", got)
}
}
// Evaluated and none found → the stanza with `tiers: []`; not evaluated → no stanza at all. Never a null on the wire.
func TestR366_EvaluatedWithNoneIsAnEmptyList(t *testing.T) {
api := &fakeBackupAPI{
storages: []proxmox.Storage{{Storage: "felhom-pbs", Type: "pbs", EncryptionKey: thisBoxKey}},
content: []proxmox.StorageContent{{VolID: "felhom-pbs:backup/ct/9201/2026-09-29T19:37:07Z", Content: "backup", VMID: 9201, Size: 3490689830, CTime: 1790710627, Encrypted: thisBoxKey}},
}
r := NewBackupRunner(api, "local", proxmox.ModeSnapshot, "", "keep-last=1", quiet())
l := NewForeignKeyLedger()
r.SetForeignKeyLedger(l)
b, _ := json.Marshal(hub.HostReport{ForeignKeyArchives: l.ForeignKeyArchives(context.Background())})
if strings.Contains(string(b), "foreign_key_archives") {
t.Fatalf("not evaluated must omit the stanza; got %s", b)
}
if _, _, err := r.PickSettledRestoreCandidateOn(context.Background(), "felhom-pbs", time.Time{}); err != nil {
t.Fatal(err)
}
b, _ = json.Marshal(hub.HostReport{ForeignKeyArchives: l.ForeignKeyArchives(context.Background())})
if !strings.Contains(string(b), `"foreign_key_archives":{"tiers":[]}`) {
t.Fatalf("evaluated with none must report tiers: []; got %s", b)
}
}
+17
View File
@@ -66,8 +66,13 @@ type BackupRunner struct {
// due-check is served from the local-API handler goroutines.
rejectedMu sync.Mutex
rejected map[string]struct{}
// foreign (R-366 slice 2) records, per tier, the archives the pick skipped as another key's. nil = not wired.
foreign *ForeignKeyLedger
}
// SetForeignKeyLedger wires the R-366 slice-2 ledger the host report reads.
func (r *BackupRunner) SetForeignKeyLedger(l *ForeignKeyLedger) { r.foreign = l }
// NewBackupRunner builds a runner. mode defaults to snapshot (works for a stopped guest and
// for lvm-thin); the caller may pass ModeStop for storages without snapshot support. retention is the
// per-run prune spec ("keep-last=N", or "" to never prune) — only the periodic local backup sets it.
@@ -370,6 +375,8 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
var best string
var bestCTime int64 = -1
known := map[int]bool{} // vmid → the guest exists on this node (asked once per vmid per pick)
var foreignN int // R-366 slice 2: archives skipped as another key's, and their time range
var foreignMin, foreignMax int64
for _, e := range contents {
if e.Content != "backup" {
continue
@@ -384,6 +391,13 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
}
if ownKey != "" && !strings.EqualFold(e.Encrypted, ownKey) {
r.noteNotAGuestBackupOnce(e, fmt.Sprintf("written by another box (key %s, this box's key %s) — not this box's proof", shortFP(e.Encrypted), shortFP(ownKey)))
foreignN++
if foreignMin == 0 || e.CTime < foreignMin {
foreignMin = e.CTime
}
if e.CTime > foreignMax {
foreignMax = e.CTime
}
continue
}
// R-689 (v0.136.0): … OF A GUEST THAT STILL EXISTS here. Measured on demo-hp 2026-09-27 right after
@@ -420,6 +434,9 @@ func (r *BackupRunner) PickSettledRestoreCandidateOn(ctx context.Context, target
bestCTime, best = e.CTime, e.VolID
}
}
if r.foreign != nil {
r.foreign.set(target, foreignN, foreignMin, foreignMax)
}
if best == "" {
return "", time.Time{}, nil
}
+2 -1
View File
@@ -145,7 +145,8 @@ var manifest = []Capability{
{"controllerswap-image-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "image", "inspect", "gitea.dooplex.hu/admin/felhom-controller:0.0.0"}, true, ""},
{"controllerswap-inspect", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "docker", "inspect", "-f", "{{.State.Running}}", "felhom-controller"}, true, ""},
{"controllerswap-restart", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "systemctl", "restart", "felhom-controller-bootstrap.service"}, true, ""},
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/sbin/pct", []string{"exec", "9201", "--", "tee", "/etc/felhom-controller-image"}, true, ""},
// R-861 (a) A1 (decision 165): the write goes through the ROOT verb that checks the ref; the agent has no `tee` grant.
{"controllerswap-write", "controller-swap / managed auto-update", "/usr/local/sbin/felhom-priv-apply", []string{"controller-image", "9201"}, true, ""},
// ---- Stale-lock recovery (FELHOM_STALELOCK, v0.49.0; Critical: a guest stuck behind a stale
// reboot-during-backup lock can't start → the customer box stays DOWN until this clears it) ----
+4 -3
View File
@@ -200,7 +200,8 @@ func TestRedProof_DroppedGrantFailsCheck(t *testing.T) {
}
// TestRedProof_DroppedControllerSwapTeeFailsCheck is the companion red-proof for the v0.45.0
// FELHOM_CONTROLLERSWAP grants: with the `tee /etc/felhom-controller-image` line removed, the
// FELHOM_CONTROLLERSWAP grants: with the write grant removed (since R-861 (a) A1 the `felhom-priv-apply controller-image`
// line; before it, an agent `tee /etc/felhom-controller-image`), the
// controllerswap-write capability MUST be reported uncovered. Proves the build gate watches the new
// swap write grant (so dropping it can't ship a non-root agent that silently can't auto-update).
func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
@@ -210,7 +211,7 @@ func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
}
var kept []string
for _, ln := range strings.Split(string(data), "\n") {
if strings.Contains(ln, "tee /etc/felhom-controller-image") {
if strings.Contains(ln, "felhom-priv-apply ^controller-image") { // R-861 (a) A1: the write's grant
continue
}
kept = append(kept, ln)
@@ -229,7 +230,7 @@ func TestRedProof_DroppedControllerSwapTeeFailsCheck(t *testing.T) {
}
cmdline := write.Binary + " " + strings.Join(write.ReprArgs, " ")
if matchesAny(cmdline, entries) {
t.Errorf("red-proof FAILED: controllerswap-write still matches after dropping the tee grant")
t.Errorf("red-proof FAILED: controllerswap-write still matches after dropping its grant")
}
if full := parseSudoersEntries(t, string(data)); !matchesAny(cmdline, full) {
t.Errorf("controllerswap-write should be covered by the real sudoers")
@@ -49,6 +49,10 @@ var r861Injections = []string{
"/usr/local/sbin/felhom-priv-apply unit ../../etc/x.mount",
"/usr/local/sbin/felhom-priv-apply dnsmasq /etc/shadow felhom-x.conf",
"/usr/local/sbin/felhom-priv-apply wg /etc/shadow",
// R-861 (a) A1 (decision 165): the agent wrote ANY image ref into the guest by `tee` — now only the root verb may
"/usr/sbin/pct exec 9201 -- tee /etc/felhom-controller-image",
"/usr/local/sbin/felhom-priv-apply controller-image 9201 9202",
"/usr/local/sbin/felhom-priv-apply controller-image 9201;id",
}
func TestSudoersRefusesTheR861Injections(t *testing.T) {
@@ -93,3 +97,42 @@ func TestSudoersFstrimRuleIsExact(t *testing.T) {
}
}
}
// R-861 (a) A1: the managed controller update still has its route — the root verb, one numeric vmid.
func TestSudoersAllowsTheControllerImageVerb(t *testing.T) {
data, err := os.ReadFile(sudoersPath)
if err != nil {
t.Fatal(err)
}
if !matchesAny("/usr/local/sbin/felhom-priv-apply controller-image 9201", parseSudoersEntries(t, string(data))) {
t.Fatal("the sudoers does not allow `felhom-priv-apply controller-image 9201` — a managed controller update cannot write its image")
}
}
// R-861 (b) B2 (decision 165, hygiene): felhom-op's `pct start|stop|unlock` grants are ONE numeric vmid each. The old
// glob `[0-9]*` eats spaces, so `pct stop 9201 --skiplock 1` and two vmids matched.
// RED-PROOF: on the pre-B2 felhom-op.sudoers (`/usr/sbin/pct stop [0-9]*`) the decoys match.
func TestFelhomOpSudoersPctIsExact(t *testing.T) {
data, err := os.ReadFile("../../configs/felhom-op.sudoers")
if err != nil {
t.Fatal(err)
}
entries := parseSudoersEntries(t, string(data))
for _, ok := range []string{"/usr/sbin/pct start 9201", "/usr/sbin/pct stop 9201", "/usr/sbin/pct unlock 9201", "/usr/sbin/pct list"} {
if !matchesAny(ok, entries) {
t.Errorf("felhom-op lost a repair verb: %s", ok)
}
}
for _, bad := range []string{
"/usr/sbin/pct stop 9201 --skiplock 1",
"/usr/sbin/pct start 9201 9202",
"/usr/sbin/pct unlock 9201 --whatever",
"/usr/sbin/pct start 92a1",
"/usr/sbin/pct stop ",
"/usr/sbin/pct destroy 9201",
} {
if matchesAny(bad, entries) {
t.Errorf("felhom-op's sudoers allows %q", bad)
}
}
}
+50 -10
View File
@@ -59,8 +59,26 @@ var (
// It carries no material and no code: only WHICH earlier package opened, by its supersession date,
// which is the one fact the customer needs to recognise it.
ErrCodeOpensRetained = errors.New("escrow: the recovery code did not open the CURRENT sealed package, but it DID open a retained earlier one")
// ErrRetainedUnchecked — the code did NOT open the current package, and NOT every earlier package the hub
// holds for this host was tried (R-304, 2026-10-08): the retained list could not be fetched, the hub
// withheld rows (over its serve cap, or rows with no key material), a package was malformed, or the
// attempt cap stopped the loop. So „the code is wrong" is NOT known — it may be right for a package
// nobody tried. Distinct from a mistype for exactly the R-224 reason: never accuse the customer of
// something we did not check.
ErrRetainedUnchecked = errors.New("escrow: the recovery code did not open the current sealed package, and some earlier packages were NOT checked")
)
// RetainedUncheckedError wraps ErrRetainedUnchecked with how many earlier packages went unchecked
// (-1 = unknown: the retained list itself could not be read). No secret.
type RetainedUncheckedError struct {
Unchecked int
}
func (e *RetainedUncheckedError) Error() string {
return fmt.Sprintf("%s (unchecked=%d)", ErrRetainedUnchecked.Error(), e.Unchecked)
}
func (e *RetainedUncheckedError) Unwrap() error { return ErrRetainedUnchecked }
// RetainedMatch says which retained package a code opened. Returned inside RetainedOpenedError; it
// carries no secret — not the code, not the bundle, not the repository password.
type RetainedMatch struct {
@@ -102,8 +120,10 @@ type RetainedBlob struct {
}
// RetainedFetcher yields this host's RETAINED sealed packages, newest-superseded first. An empty
// slice is a clean "none". R-311.
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, unopenable int, err error)
// slice is a clean "none". R-311. `withheld` counts the earlier packages the hub holds that are NOT in
// blobs — rows with no key material, rows over the hub's serve cap, and packages dropped as malformed
// (R-304): each is a package the code was never tried against.
type RetainedFetcher func(ctx context.Context) (blobs []RetainedBlob, withheld int, err error)
// OffsiteKeyRecoverer is the assembled links 6→8. Construct it with a fetcher; call it with R.
type OffsiteKeyRecoverer struct {
@@ -154,9 +174,14 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
// from the unwrap alone; the only way to tell is to try. Until this existed nobody tried, and
// the screen said so out loud ("innen nem tudjuk megkülönböztetni őket") — a true sentence
// about our own incuriosity, read by the customer as a statement about their code.
if m, ok := r.tryRetained(ctx, recoveryCode); ok {
m, ok, unchecked := r.tryRetained(ctx, recoveryCode)
if ok {
return "", &RetainedOpenedError{Match: m}
}
// R-304: only when EVERY earlier package the hub holds was tried may this stay a wrong code.
if unchecked != 0 {
return "", &RetainedUncheckedError{Unchecked: unchecked}
}
return "", err // the fail-closed "the recovery code did not unwrap…" message; no secret in it
}
if bundle.ResticRepoPassword == "" {
@@ -173,23 +198,38 @@ func (r OffsiteKeyRecoverer) RecoverOffsiteRepoPassword(ctx context.Context, rec
// function breaking is the behaviour we had before it existed.
//
// NOTHING IS LOGGED HERE and no return value carries the code, a bundle or a password.
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool) {
//
// R-304 (2026-10-08): it also returns how many earlier packages were NOT tried — `withheld` from the hub, plus
// the ones past the attempt cap — or -1 when the retained list could not be read at all. A nil FetchRetained
// (an agent wired without the lookup) reports 0: the pre-R-311 refusal, unchanged.
func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode string) (RetainedMatch, bool, int) {
if r.FetchRetained == nil {
return RetainedMatch{}, false
return RetainedMatch{}, false, 0
}
blobs, _, err := r.FetchRetained(ctx)
if err != nil || len(blobs) == 0 {
return RetainedMatch{}, false
blobs, withheld, err := r.FetchRetained(ctx)
if err != nil {
return RetainedMatch{}, false, -1
}
if withheld < 0 {
withheld = 0
}
if len(blobs) == 0 {
return RetainedMatch{}, false, withheld
}
limit := r.MaxRetainedTried
if limit <= 0 {
limit = defaultMaxRetainedTried
}
unchecked := withheld
if len(blobs) > limit {
unchecked += len(blobs) - limit
}
for i, rb := range blobs {
if i >= limit {
break
}
if len(rb.Blob) == 0 {
unchecked++
continue
}
bundle, uerr := UnwrapIdentityBundle(ctx, rb.Blob, recoveryCode)
@@ -204,7 +244,7 @@ func (r OffsiteKeyRecoverer) tryRetained(ctx context.Context, recoveryCode strin
// correct and must be told so — but the history behind it still cannot be reopened, and
// saying otherwise would be a promise this path cannot keep.
HasResticPassword: bundle.ResticRepoPassword != "",
}, true
}, true, 0
}
return RetainedMatch{}, false
return RetainedMatch{}, false, unchecked
}
+76 -2
View File
@@ -115,8 +115,9 @@ func TestRecover_WrongCode_StaysAPlainRefusal(t *testing.T) {
}
}
// FAIL-SAFE — if the retained lookup itself fails, the original refusal must stand UNCHANGED. The
// worst outcome of this feature breaking is the behaviour we had before it.
// FAIL-SAFE — if the retained lookup itself fails, the lookup's error never reaches the customer and no
// retained package is claimed. R-304 (2026-10-08) changed what stands instead: not the wrong-code refusal (the
// earlier packages were never tried, so „wrong" is not known) but ErrRetainedUnchecked with Unchecked = -1.
//
// RED-PROOF: make tryRetained propagate the fetch error instead of returning false → the customer
// gets a new, unexplained failure mode → this FAILS.
@@ -140,6 +141,10 @@ func TestRecover_RetainedFetchFails_OriginalRefusalStands(t *testing.T) {
if containsStr(err.Error(), "hub exploded") {
t.Error("the retained-lookup failure leaked into the customer-facing refusal — it must be silent")
}
var ue *RetainedUncheckedError
if !errors.As(err, &ue) || ue.Unchecked != -1 {
t.Errorf("err = %v, want RetainedUncheckedError{-1} — the earlier packages were never tried (R-304)", err)
}
}
// A nil FetchRetained keeps the pre-R-311 behaviour EXACTLY. An agent wired without it must be
@@ -228,3 +233,72 @@ func containsStr(hay, needle string) bool {
return false
})()
}
// ── R-304 (2026-10-08) — „wrong code" only when every earlier package was tried ──────────────────────
//
// The consequence asserted: is the refusal a WRONG CODE (the only error the screen may answer with „check your
// typing")? It may be only when the hub withheld nothing and every served package was tried.
//
// RED-PROOF: make tryRetained return 0 for `unchecked` → the three „unchecked" cases return the plain refusal
// → they FAIL; the all-tried case keeps passing.
func TestR304_WrongCodeOnlyWhenEveryEarlierPackageWasTried(t *testing.T) {
ensureAge(t)
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "4444567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
other := sealBundle(t, IdentityBundle{ResticRepoPassword: "5555567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
const code = "a code that opens nothing whatsoever in this test"
cases := []struct {
name string
blobs int
withheld int
limit int
unchecked int // 0 = must be a plain wrong code
}{
{"all tried, nothing withheld", 2, 0, 6, 0},
{"the hub withheld rows (no key material / over its cap)", 1, 3, 6, 3},
{"only withheld rows, nothing served", 0, 2, 6, 2},
{"the attempt cap stopped the loop", 5, 0, 2, 3},
}
for _, c := range cases {
t.Run(c.name, func(t *testing.T) {
blobs := make([]RetainedBlob, 0, c.blobs)
for i := 0; i < c.blobs; i++ {
blobs = append(blobs, RetainedBlob{Blob: other, SupersededAt: "2026-08-01 00:00:00", Index: i})
}
_, err := OffsiteKeyRecoverer{
Fetch: fetcherFor(current),
FetchRetained: func(context.Context) ([]RetainedBlob, int, error) {
return blobs, c.withheld, nil
},
MaxRetainedTried: c.limit,
}.RecoverOffsiteRepoPassword(context.Background(), code)
if err == nil {
t.Fatal("a code that opens nothing succeeded")
}
var ue *RetainedUncheckedError
isUnchecked := errors.As(err, &ue)
if c.unchecked == 0 {
if isUnchecked {
t.Fatalf("every earlier package was tried — this IS a wrong code, got %v", err)
}
return
}
if !isUnchecked || ue.Unchecked != c.unchecked || !errors.Is(err, ErrRetainedUnchecked) {
t.Fatalf("err = %v, want RetainedUncheckedError{%d} — never call the code wrong when packages were not tried", err, c.unchecked)
}
})
}
}
// A malformed (empty) served package counts as not tried.
func TestR304_EmptyServedPackageCountsAsUnchecked(t *testing.T) {
ensureAge(t)
current := sealBundle(t, IdentityBundle{ResticRepoPassword: "6666567890abcdef0123456789abcdef0123456789abcdef0123456789abcdef"}, testR)
_, err := OffsiteKeyRecoverer{
Fetch: fetcherFor(current),
FetchRetained: retainedFetcherFor(RetainedBlob{Blob: nil, SupersededAt: "2026-08-01 00:00:00"}),
}.RecoverOffsiteRepoPassword(context.Background(), testR2)
var ue *RetainedUncheckedError
if !errors.As(err, &ue) || ue.Unchecked != 1 {
t.Fatalf("err = %v, want RetainedUncheckedError{1}", err)
}
}
+66 -5
View File
@@ -66,6 +66,13 @@ type ProvenRestoreTestReporter interface {
ProvenRestoreTests(ctx context.Context) []RestoreTest
}
// KnownBackupReporter is the newest SUCCESSFUL backup per tier and guest kept on disk (R-894's
// backup-success-state.json). collectBackups folds it into the report so a restart does not erase the
// hub's evidence of a backup that ran minutes before it (the kernel-night shape, 2026-10-09).
type KnownBackupReporter interface {
KnownBackupSuccesses() []Backup
}
// PBSReporter is the slice-6-Phase-B seam the pbs verify loop plugs into (same pattern).
// Returns the agent's latest-known PBS snapshot inventory + verify-state. nil → empty.
type PBSReporter interface {
@@ -105,6 +112,7 @@ type Collector struct {
backups BackupReporter
restoreTests RestoreTestReporter
provenTests ProvenRestoreTestReporter
knownBackups KnownBackupReporter // R-894 file → host report after a restart (nil → in-memory only)
pbs PBSReporter
temp TempReader // slice 9: host CPU/chassis temp (nil-safe → nil temp)
capProbe func(ctx context.Context) []capability.Status // v0.44.0: privileged-capability self-check (nil → empty)
@@ -115,6 +123,7 @@ type Collector struct {
ctrlSup ControllerSupervisorReporter // R-523: in-guest controller supervisor (nil → stanza omitted)
guestNet GuestNetReporter // R-54: per-guest network watchdog (nil → stanza omitted)
diskTrim GuestDiskTrimReporter // R-444: weekly guest disk trim (nil → stanza omitted)
foreignKey ForeignKeyArchiveReporter // R-366 slice 2 (nil → omitted)
selfUpdate SelfUpdateReporter // D1: agent self-update pending status (nil → false)
mgmtPlane MgmtPlaneReporter // G1: management-plane health (nil → stanza omitted)
oob OOBReporter // H1: operator-access health (nil → stanza omitted)
@@ -228,6 +237,18 @@ func (c *Collector) SetGuestNetReporter(g GuestNetReporter) *Collector {
return c
}
// ForeignKeyArchiveReporter is the R-366 slice-2 seam: the restore-test's ledger of archives written with another
// key. nil = not evaluated yet (the stanza is omitted and the hub keeps its state).
type ForeignKeyArchiveReporter interface {
ForeignKeyArchives(ctx context.Context) *ForeignKeyArchivesStanza
}
// SetForeignKeyArchiveReporter wires the restore-test's foreign-key ledger (R-366 slice 2; nil-safe → omitted).
func (c *Collector) SetForeignKeyArchiveReporter(r ForeignKeyArchiveReporter) *Collector {
c.foreignKey = r
return c
}
// SetGuestDiskTrimReporter wires the R-444 weekly trim job as a report source (nil-safe → stanza omitted).
func (c *Collector) SetGuestDiskTrimReporter(r GuestDiskTrimReporter) *Collector {
c.diskTrim = r
@@ -394,6 +415,10 @@ func (c *Collector) Collect(ctx context.Context) (*HostReport, error) {
if c.diskTrim != nil {
report.GuestDiskTrim = c.diskTrim.GuestDiskTrimStatus(ctx)
}
// R-366 slice 2: archives the restore-test skipped as another key's (nil → not evaluated yet → omitted).
if c.foreignKey != nil {
report.ForeignKeyArchives = c.foreignKey.ForeignKeyArchives(ctx)
}
// D1: agent self-update pending status (nil reporter → pending=false, the steady state).
if c.selfUpdate != nil {
report.SelfUpdatePending, report.SelfUpdatePendingVersion = c.selfUpdate.SelfUpdatePending()
@@ -559,14 +584,50 @@ func (c *Collector) collectStorage(ctx context.Context) []StorageTarget {
// collectBackups / collectRestoreTests read the agent's latest backup + restore-test state
// via the seams. Best-effort: a nil reporter or nil slice degrades to an empty (non-nil)
// list so the collection always marshals as [].
//
// The saved successes (SetKnownBackupReporter) are added for each tier and guest the in-memory list has
// no success for at or after the saved time. Why: the in-memory list is empty after an agent restart,
// and the hub's backup-freshness check reads only what the reports carried. Measured 2026-10-09 on
// demo-felhom: the night backup landed 02:40 UTC, the kernel step restarted the host at 02:44, no report
// fell in between, so two kernel nights in a row left no trace and the hub alarmed „newest backup is 48h
// old" at 03:00. Only SUCCESSES are saved, so a failure is never hidden and never invented.
// Pinned by TestCollectBackups_SavedSuccessSurvivesARestart.
func (c *Collector) collectBackups(ctx context.Context) []Backup {
if c.backups == nil {
return []Backup{}
out := []Backup{}
if c.backups != nil {
if b := c.backups.Backups(ctx); b != nil {
out = append(out, b...)
}
}
if b := c.backups.Backups(ctx); b != nil {
return b
if c.knownBackups == nil {
return out
}
return []Backup{}
for _, k := range c.knownBackups.KnownBackupSuccesses() {
kt, err := time.Parse(time.RFC3339, k.StartedAt)
if err != nil {
continue
}
covered := false
for _, b := range out {
if !b.Success || b.TargetID != k.TargetID || b.VMID != k.VMID {
continue
}
if bt, err := time.Parse(time.RFC3339, b.StartedAt); err == nil && !bt.Before(kt) {
covered = true
break
}
}
if !covered {
out = append(out, k)
}
}
return out
}
// SetKnownBackupReporter wires the R-894 saved successes into the host report (nil-safe → in-memory only).
func (c *Collector) SetKnownBackupReporter(r KnownBackupReporter) *Collector {
c.knownBackups = r
return c
}
// collectRestoreTests merges the in-memory result with the PERSISTED per-tier proofs (R-189).
+62
View File
@@ -0,0 +1,62 @@
package hub
import (
"context"
"testing"
)
type fakeMemBackups []Backup
func (f fakeMemBackups) Backups(context.Context) []Backup { return f }
type fakeKnownBackups []Backup
func (f fakeKnownBackups) KnownBackupSuccesses() []Backup { return f }
// The consequence, not the mechanism: after a restart (empty in-memory list) the host report still carries
// the night's backup, so the hub's freshness check sees it. Measured 2026-10-09 on demo-felhom: without this
// the report carried backups: [] and the hub alarmed „newest backup is 48h old" an hour after a good backup.
// RED-PROOF: return before the saved-success loop in collectBackups → the first case reports nothing → fails.
func TestCollectBackups_SavedSuccessSurvivesARestart(t *testing.T) {
saved := Backup{TargetID: "felhom-backup", VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z"}
t.Run("restart: memory empty, the saved success is reported", func(t *testing.T) {
c := &Collector{backups: fakeMemBackups(nil), knownBackups: fakeKnownBackups{saved}}
got := c.collectBackups(context.Background())
if len(got) != 1 || got[0].StartedAt != saved.StartedAt || !got[0].Success || got[0].TargetID != "felhom-backup" {
t.Fatalf("want the saved success, got %+v", got)
}
})
t.Run("memory holds the same or a newer success: no second entry", func(t *testing.T) {
mem := Backup{TargetID: "felhom-backup", VMID: 9201, Success: true, StartedAt: "2026-10-09T02:40:16Z", Archive: "a"}
c := &Collector{backups: fakeMemBackups{mem}, knownBackups: fakeKnownBackups{saved}}
if got := c.collectBackups(context.Background()); len(got) != 1 || got[0].Archive != "a" {
t.Fatalf("want only the in-memory record, got %+v", got)
}
})
t.Run("a newer FAILURE in memory never hides the saved success, and is kept", func(t *testing.T) {
fail := Backup{TargetID: "felhom-backup", VMID: 9201, Success: false, StartedAt: "2026-10-10T02:40:00Z", Error: "x"}
c := &Collector{backups: fakeMemBackups{fail}, knownBackups: fakeKnownBackups{saved}}
got := c.collectBackups(context.Background())
if len(got) != 2 || got[0].Success || !got[1].Success {
t.Fatalf("want the failure and the saved success, got %+v", got)
}
})
t.Run("another tier's success does not cover this tier", func(t *testing.T) {
other := Backup{TargetID: "felhom-pbs", VMID: 9201, Success: true, StartedAt: "2026-10-10T00:00:00Z"}
c := &Collector{backups: fakeMemBackups{other}, knownBackups: fakeKnownBackups{saved}}
if got := c.collectBackups(context.Background()); len(got) != 2 {
t.Fatalf("want both tiers, got %+v", got)
}
})
t.Run("no saved state wired: the in-memory list as before, never nil", func(t *testing.T) {
c := &Collector{}
if got := c.collectBackups(context.Background()); got == nil || len(got) != 0 {
t.Fatalf("want an empty non-nil list, got %#v", got)
}
})
}
+32
View File
@@ -129,6 +129,12 @@ type HostReport struct {
// wired; an empty `guests` list means the job runs and no guest has been trimmed yet. No secret.
GuestDiskTrim *GuestDiskTrimStatus `json:"guest_disk_trim,omitempty"`
// ForeignKeyArchives (R-366 slice 2, `09` §3 decision 168): per backup tier, the whole-guest archives the
// restore-test SKIPPED because they were written with another key (an earlier install of this box). This box
// cannot open them; the hub turns a CHANGE of this list into one operator event. Absent = not evaluated yet since
// the agent started (the hub keeps its last state); `tiers: []` = evaluated, none found.
ForeignKeyArchives *ForeignKeyArchivesStanza `json:"foreign_key_archives,omitempty"`
// LogTail is the agent's on-demand debug-ring tail (v0.83.0 observability) — the agent
// mirror of the controller's report log_tails channel. Present ONLY on the heartbeat
// right after the control envelope requested it (log_tail_requested); consume-once on
@@ -630,6 +636,17 @@ type WireOSUpdate struct {
// HostRelease is the newest approved HOST release (hub v0.131.0, `11` §8 step 3) — a separate set: a version
// approved for the guest is not approved for the host by that fact alone.
HostRelease *WireOSRelease `json:"host_release,omitempty"`
// Kernel is the kernel lane's instruction for THIS box (R-836, `09` §3 decision 172, `11` §5.11): the kernel the
// household was told about, and whether tonight is a told night. Nil (an older hub, or no kernel due) = no kernel
// step. The hub sets Tonight only after the household's mail the day before went out — no mail, no step.
Kernel *WireKernelStep `json:"kernel,omitempty"`
}
// WireKernelStep is the hub's kernel-lane instruction (hub osupdates.KernelBlock — field-exact, cross-repo).
type WireKernelStep struct {
Kver string `json:"kver"` // e.g. "7.0.14-22-pve" — the wrapper refuses any other (R23)
Tonight bool `json:"tonight"` // the household was mailed the day before: tonight's leg may reboot
NotifiedAt string `json:"notified_at,omitempty"` // when that mail went out (RFC 3339), for the log
}
// WireOSRelease is an approved version set; Snapshot is the approval time (YYYYMMDDTHHMMSSZ) the wrapper uses
@@ -726,3 +743,18 @@ type WireRestoreDirective struct {
Archive string `json:"archive,omitempty"` // source archive/snapshot to restore from
VMID int `json:"vmid,omitempty"`
}
// ForeignKeyArchivesStanza wraps the per-tier list so "evaluated, none" (`tiers: []`) differs from "not evaluated"
// (the stanza absent) without a null on the wire.
type ForeignKeyArchivesStanza struct {
Tiers []ForeignKeyArchives `json:"tiers"`
}
// ForeignKeyArchives is one tier's count of archives written with another key (R-366 slice 2): the count and the
// newest/oldest archive time (RFC3339, UTC). No key material — the fingerprints stay on the box.
type ForeignKeyArchives struct {
Target string `json:"target"`
Count int `json:"count"`
Oldest string `json:"oldest"`
Newest string `json:"newest"`
}
+17 -4
View File
@@ -15,7 +15,7 @@ import (
// Red-proof: drop the `b.Success &&` guard and the failed-backup sub-case fails; move the call after release()
// and the gate sub-case fails.
func TestAfterPrimaryBackup(t *testing.T) {
run := func(t *testing.T, failErr string) (calls []int, gateHeld bool) {
run := func(t *testing.T, failErr, path string) (calls []int, gateHeld bool) {
gate := &backup.InFlight{}
b := &fakeBackups{failErr: failErr}
srv := newTestServerS(t, &fakeGuests{}, b, &fakeStore{}, nil)
@@ -34,7 +34,7 @@ func TestAfterPrimaryBackup(t *testing.T) {
done <- struct{}{}
})
h := srv.Handler()
if do(t, h, "POST", "/backup", "A", "").Code != http.StatusAccepted {
if do(t, h, "POST", path, "A", "").Code != http.StatusAccepted {
t.Fatal("POST /backup not accepted")
}
select {
@@ -47,7 +47,7 @@ func TestAfterPrimaryBackup(t *testing.T) {
return calls, gateHeld
}
t.Run("success runs the leg under the gate", func(t *testing.T) {
calls, held := run(t, "")
calls, held := run(t, "", "/backup")
if len(calls) != 1 {
t.Fatalf("the leg ran %d time(s), want 1", len(calls))
}
@@ -56,8 +56,21 @@ func TestAfterPrimaryBackup(t *testing.T) {
}
})
t.Run("a failed backup runs nothing", func(t *testing.T) {
if calls, _ := run(t, "vzdump exploded"); len(calls) != 0 {
if calls, _ := run(t, "vzdump exploded", "/backup"); len(calls) != 0 {
t.Fatalf("the leg ran after a FAILED backup: %v", calls)
}
})
// R-899: a household press is not the night's backup — no OS leg after it. Same fake, same successful backup as
// the first sub-case; only the query differs. Red-proof: make handleBackup ignore `trigger` and this sub-case
// fails with the leg run once.
t.Run("a manual press runs nothing", func(t *testing.T) {
if calls, _ := run(t, "", "/backup?trigger=manual"); len(calls) != 0 {
t.Fatalf("the OS leg ran after a manual press: %v", calls)
}
})
t.Run("the scheduled path with the new query still runs the leg", func(t *testing.T) {
if calls, _ := run(t, "", "/backup?trigger=night"); len(calls) != 1 {
t.Fatalf("the leg ran %d time(s) after a non-manual backup, want 1", len(calls))
}
})
}
@@ -4,7 +4,6 @@ import (
"context"
"encoding/json"
"errors"
"io"
"log/slog"
"os"
"path/filepath"
@@ -54,8 +53,8 @@ func (f *supExec) GuestExec(_ context.Context, vmid int, args ...string) (string
}
return "", errors.New("supExec: unexpected args")
}
func (f *supExec) GuestExecStdin(context.Context, int, io.Reader, ...string) (string, error) {
return "", errors.New("supExec: no stdin exec expected")
func (f *supExec) WriteControllerImage(context.Context, int, string) error {
return errors.New("supExec: no image write expected")
}
func (f *supExec) count(vmid int) int {
f.mu.Lock()
+9 -12
View File
@@ -4,7 +4,6 @@ import (
"context"
"encoding/json"
"fmt"
"io"
"log/slog"
"net/http"
"os"
@@ -41,9 +40,10 @@ func ValidControllerImage(ref string) bool { return controllerImageRe.MatchStrin
// faked in tests. The single seam the swap composes over (no hand-rolled pct).
type GuestExecutor interface {
GuestExec(ctx context.Context, vmid int, args ...string) (string, error)
// GuestExecStdin is GuestExec with the command's stdin fed from stdin — the swap write pipes the
// image ref into an in-guest `tee` (no shell vector).
GuestExecStdin(ctx context.Context, vmid int, stdin io.Reader, args ...string) (string, error)
// WriteControllerImage writes the image ref into the guest's /etc/felhom-controller-image through the ROOT
// verb `felhom-priv-apply controller-image <vmid>` (R-861 (a) A1, `09` §3 decision 165), which re-checks the
// ref against our registry + repository + x.y.z. The agent no longer holds a `tee` grant into the guest.
WriteControllerImage(ctx context.Context, vmid int, image string) error
}
// ControllerSwapState is the durable record of a swap (crash-safety + status). Written before the swap
@@ -141,14 +141,11 @@ func (c *ControllerSwapper) imagePresent(ctx context.Context, vmid int, image st
}
func (c *ControllerSwapper) writeImage(ctx context.Context, vmid int, image string) error {
// Non-root path: pipe the image ref into an in-guest `tee` over stdin — no shell, no
// interpolation, no `bash -c` (the only swap vector that would have needed an arbitrary-exec
// grant). The trailing "\n" makes the on-disk bytes byte-identical to the golden's
// `printf '%s\n'`; the bootstrap reads `IMAGE=$(cat …)` so the newline is stripped on read
// (spike SPIKE-controllerswap-narrow-grants-2026-06-29). image is strict-validated
// (controllerImageRe) upstream in Swap; defence-in-depth, the stdin path can't smuggle anyway.
_, err := c.exec.GuestExecStdin(ctx, vmid, strings.NewReader(image+"\n"), "tee", controllerImageFile)
return err
// R-861 (a) A1: the ROOT verb writes the file (it re-checks the ref — a compromised agent cannot hand the guest's
// bootstrap another image). The bytes are `image\n`, byte-identical to the golden's `printf '%s\n'`; the bootstrap
// reads `IMAGE=$(cat …)` so the newline is stripped on read. image is also strict-validated (controllerImageRe)
// upstream in Swap.
return c.exec.WriteControllerImage(ctx, vmid, image)
}
func (c *ControllerSwapper) restartBootstrap(ctx context.Context, vmid int) error {
+14 -28
View File
@@ -23,7 +23,7 @@ type fakeGuestExec struct {
present map[string]bool // images pulled into the guest
good map[string]bool // images that report healthy when running
containerImg string // image the running container currently has
teeStdin []string // raw bytes piped into each `tee` write (the swap's write vector)
teeStdin []string // image refs handed to WriteControllerImage (the root verb, R-861 (a) A1)
failRestart bool
noHealthBlock bool // if set, .State.Health is absent ("none")
restartCount int // F1: .RestartCount reported by docker inspect (a crash-looper has >0)
@@ -75,20 +75,15 @@ func (f *fakeGuestExec) GuestExec(_ context.Context, _ int, args ...string) (str
return "", fmt.Errorf("fake: unexpected exec %v", args)
}
// GuestExecStdin models the swap's write vector: `tee /etc/felhom-controller-image` with the image
// piped on stdin. It records the raw stdin bytes and sets the modeled file content (newline-stripped,
// as the bootstrap's `IMAGE=$(cat …)` read would see it).
func (f *fakeGuestExec) GuestExecStdin(_ context.Context, _ int, stdin io.Reader, args ...string) (string, error) {
// WriteControllerImage models the swap's write: the root verb `felhom-priv-apply controller-image <vmid>` (R-861
// (a) A1). It records the ref and sets the modeled file content, as the bootstrap's `IMAGE=$(cat …)` would read it.
func (f *fakeGuestExec) WriteControllerImage(_ context.Context, _ int, image string) error {
f.mu.Lock()
defer f.mu.Unlock()
f.calls = append(f.calls, args)
b, _ := io.ReadAll(stdin)
if len(args) >= 2 && args[0] == "tee" && args[1] == controllerImageFile {
f.teeStdin = append(f.teeStdin, string(b))
f.imageFile = strings.TrimSpace(string(b))
return string(b), nil // tee echoes stdin to stdout
}
return "", fmt.Errorf("fake: unexpected exec-stdin args=%v stdin=%q", args, string(b))
f.calls = append(f.calls, []string{"felhom-priv-apply", "controller-image", image})
f.teeStdin = append(f.teeStdin, image+"\n")
f.imageFile = image
return nil
}
// wrote reports whether the image was written via the stdin `tee` vector with the exact `image\n`
@@ -162,7 +157,7 @@ func TestControllerSwap_Happy(t *testing.T) {
// The write vector must be the stdin `tee` with byte-identical `image\n` and NO shell — the
// controllerswap.go writeImage rewrite. This would FAIL on the pre-change `bash -c "printf … >"` impl.
func TestControllerSwap_WriteViaStdinTee_NoShell(t *testing.T) {
func TestControllerSwap_WriteViaRootVerb_NoShell(t *testing.T) {
fe := &fakeGuestExec{
imageFile: prevImg,
present: map[string]bool{newImg: true},
@@ -173,22 +168,15 @@ func TestControllerSwap_WriteViaStdinTee_NoShell(t *testing.T) {
t.Fatalf("state = %q, want done", st.State)
}
if !fe.wrote(newImg) {
t.Errorf("expected a tee write of %q+\\n; teeStdin=%q", newImg, fe.teeStdin)
t.Errorf("expected the root verb to write %q; writes=%q", newImg, fe.teeStdin)
}
sawTee := false
for _, c := range fe.calls {
if len(c) >= 2 && c[0] == "tee" {
sawTee = true
if c[1] != controllerImageFile {
t.Errorf("tee target = %q, want fixed %q", c[1], controllerImageFile)
}
if len(c) >= 1 && c[0] == "tee" {
t.Errorf("the swap still uses an in-guest tee (R-861 (a) A1 removed that grant): %v", c)
}
}
if !sawTee {
t.Error("no tee call recorded — writeImage did not use the stdin tee vector")
}
if fe.usedShell() {
t.Errorf("swap used a shell vector (bash/-c/printf) — must be stdin tee only; calls=%v", fe.calls)
t.Errorf("swap used a shell vector (bash/-c/printf); calls=%v", fe.calls)
}
}
@@ -300,9 +288,7 @@ func (s *inspectScript) GuestExec(_ context.Context, _ int, args ...string) (str
}
return "", nil
}
func (s *inspectScript) GuestExecStdin(_ context.Context, _ int, _ io.Reader, _ ...string) (string, error) {
return "", nil
}
func (s *inspectScript) WriteControllerImage(context.Context, int, string) error { return nil }
func fastSwapper(exec GuestExecutor) *ControllerSwapper {
s := NewControllerSwapper(exec, "", discardLogger())
+16
View File
@@ -127,6 +127,22 @@ func (s *Server) handleRecoverOffsitePassword(w http.ResponseWriter, r *http.Req
"retained_has_restic_pw": match.HasResticPassword,
},
"the recovery code is correct, but it belongs to an EARLIER sealed package (superseded "+match.SupersededAt+"), not the one currently held")
// ── R-304 (2026-10-08) — NOT EVERY EARLIER PACKAGE WAS CHECKED. ─────────────────────────
//
// The current package refused the code, no retained package opened it — and at least one earlier
// package the hub holds was never tried (or the list could not be read). Saying „the code is
// wrong" here would claim a check that did not happen. 424 (Failed Dependency): the verdict
// depends on packages we could not try. The controller classifies on the status, never on this
// sentence; an older controller maps an unknown status to its neutral „we do not know why".
case errors.Is(err, escrow.ErrRetainedUnchecked):
n := -1
var ue *escrow.RetainedUncheckedError
if errors.As(err, &ue) {
n = ue.Unchecked
}
s.logger.Warn("local-api: offsite key recovery: the code did not open the current package and earlier packages were NOT all checked — not reported as a wrong code (R-304)", "vmid", vmid, "unchecked", n)
writeStatus(w, http.StatusFailedDependency, false, map[string]any{"older_unchecked": n},
"the recovery code did not open the current sealed package, and earlier packages the hub holds were not all checked — the code may belong to one of them; nothing was written")
case errors.Is(err, escrow.ErrNoResticPassword):
s.logger.Warn("local-api: offsite key recovery: the bundle opened but predates the repository-password field", "vmid", vmid)
writeErr(w, http.StatusConflict, "the recovery code opened the bundle, but it carries NO offsite repository password (sealed before that field existed; it cannot be retro-fitted)")
@@ -54,6 +54,13 @@ func TestRecoverOffsitePassword_EachSituationGetsItsOwnStatus(t *testing.T) {
wantStatus: 404,
mustNotSay: []string{"did not open"},
},
{
// R-304: earlier packages were not all tried — never „did not open the sealed bundle" (the wrong-code words).
name: "earlier packages not all checked — not a wrong code",
err: &escrow.RetainedUncheckedError{Unchecked: 2},
wantStatus: 424,
mustNotSay: []string{"did not open the sealed bundle", "could not be fetched"},
},
{
name: "the bundle predates the repository-password field",
err: escrow.ErrNoResticPassword,
+10 -9
View File
@@ -3,7 +3,6 @@ package localapi
import (
"context"
"fmt"
"io"
"log/slog"
"strconv"
"strings"
@@ -126,14 +125,16 @@ func (b *GuestBinder) GuestExec(ctx context.Context, vmid int, args ...string) (
return string(out), nil
}
// GuestExecStdin is GuestExec with the in-guest command's stdin fed from stdin. The controller-swap
// write uses it to pipe the image ref into an in-guest `tee` (no shell vector, no interpolation),
// through the same fenced runner so the `sudo -n` prefix stays in one place.
func (b *GuestBinder) GuestExecStdin(ctx context.Context, vmid int, stdin io.Reader, args ...string) (string, error) {
pctArgs := append([]string{"exec", strconv.Itoa(vmid), "--"}, args...)
out, stderr, err := b.runner.RunStdin(ctx, stdin, "pct", pctArgs...)
// privApplyBin is the root content checker (R-861); its `controller-image` verb writes the guest's image file.
const privApplyBin = "/usr/local/sbin/felhom-priv-apply"
// WriteControllerImage pipes `image\n` to `felhom-priv-apply controller-image <vmid>` through the same fenced runner
// (the `sudo -n` prefix stays in one place). The verb checks the ref as root and writes the guest file itself
// (R-861 (a) A1, `09` §3 decision 165). Pinned by TestR861_WriteControllerImageUsesTheRootVerb.
func (b *GuestBinder) WriteControllerImage(ctx context.Context, vmid int, image string) error {
_, stderr, err := b.runner.RunStdin(ctx, strings.NewReader(image+"\n"), privApplyBin, "controller-image", strconv.Itoa(vmid))
if err != nil {
return string(out), fmt.Errorf("pct exec %d %v: %w: %s", vmid, args, err, strings.TrimSpace(string(stderr)))
return fmt.Errorf("felhom-priv-apply controller-image %d: %w: %s", vmid, err, strings.TrimSpace(string(stderr)))
}
return string(out), nil
return nil
}
@@ -0,0 +1,48 @@
package localapi
import (
"context"
"io"
"testing"
)
// stdinRecorder is a proxmox.Runner that records each call and the stdin it was handed.
type stdinRecorder struct {
name string
args []string
stdin string
}
func (r *stdinRecorder) Run(_ context.Context, name string, args ...string) ([]byte, []byte, error) {
r.name, r.args = name, args
return nil, nil, nil
}
func (r *stdinRecorder) RunStdin(_ context.Context, stdin io.Reader, name string, args ...string) ([]byte, []byte, error) {
b, _ := io.ReadAll(stdin)
r.name, r.args, r.stdin = name, args, string(b)
return nil, nil, nil
}
// R-861 (a) A1 (`09` §3 decision 165): the managed controller update writes the guest's image file through the ROOT
// verb, never through an in-guest `tee` the agent could feed any image.
//
// COMPANION RED-PROOF (observed): restore the pre-A1 body (`b.runner.RunStdin(ctx, …, "pct", "exec", vmid, "--",
// "tee", controllerImageFile)`) → this fails with "the image write ran pct …, want felhom-priv-apply". Restored.
func TestR861_WriteControllerImageUsesTheRootVerb(t *testing.T) {
rec := &stdinRecorder{}
b := NewGuestBinder(rec, discardLogger())
const img = "gitea.dooplex.hu/admin/felhom-controller:0.302.0"
if err := b.WriteControllerImage(context.Background(), 9201, img); err != nil {
t.Fatal(err)
}
if rec.name != privApplyBin {
t.Fatalf("the image write ran %s %v, want felhom-priv-apply", rec.name, rec.args)
}
if len(rec.args) != 2 || rec.args[0] != "controller-image" || rec.args[1] != "9201" {
t.Fatalf("argv = %v, want [controller-image 9201] (the sudoers line `^controller-image [0-9]+$`)", rec.args)
}
if rec.stdin != img+"\n" {
t.Fatalf("stdin = %q, want the ref plus one newline", rec.stdin)
}
}
+7 -1
View File
@@ -825,6 +825,10 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
return
}
key := backupJobKey{vmid: vmid, target: tier.TargetID}
// R-899 (operator ruling 2026-10-08): a household press („Mentés most") is not the night's backup. A controller
// that knows sends `trigger=manual`; then the OS leg does not follow (it belongs to the night, after the night's
// own copy). An older controller sends nothing and keeps the old behaviour.
manual := r.URL.Query().Get("trigger") == "manual"
// ONE BACKUP AT A TIME PER GUEST, ACROSS ALL TIERS (operator ruling 2026-07-26: "other backup
// shouldn't start until finished"). vzdump takes a guest lock, so a concurrent second backup
@@ -926,7 +930,9 @@ func (s *Server) handleBackup(w http.ResponseWriter, r *http.Request, vmid int)
}
s.finishJob(key, jobID, b)
// OS leg (agent v0.140.0): after the night's whole-guest copy exists, still holding the heavy-op gate.
if b.Success && tier.Primary && s.afterPrimaryBackup != nil {
if b.Success && tier.Primary && s.afterPrimaryBackup != nil && manual {
s.logger.Info("local-api: no OS leg after a manual backup — it follows the night's own backup (R-899)", "vmid", vmid, "job", jobID)
} else if b.Success && tier.Primary && s.afterPrimaryBackup != nil {
s.afterPrimaryBackup(base, vmid)
}
}()
+446
View File
@@ -0,0 +1,446 @@
package osupdate
// The kernel lane (R-836, `09` §3 decisions 164 + 172, `11` §5.11). The root half is felhom-os-apply's layer "kernel"
// (configs/, its own tests); this file decides WHEN and judges the boot:
//
// - the night leg (Run → runKernel): after a healthy host step, on a night the hub marks as told (the household was
// mailed the day before — no mail, no step): ring 0 STAGES the pending kernel (select pending-kernel, the root-owned
// ring-0 mark) and reboots; ring 1 reboots only a step a signed os_kernel_step staged earlier (KernelStepExecutor).
// - after a boot (KernelAfterBoot, at daemon start): the wrapper says what became of the step. On the new kernel the
// agent JUDGES the boot — the host health rule (`11` §8.2: the Proxmox daemons, the guest running and healthy, the
// tunnel) AND the box reaching the hub — for KernelJudgeWait. Healthy → kernel-good (the new kernel becomes the
// default). Not healthy by the deadline → ONE self-revert (kernel-revert: a reboot into the old kernel, still the
// default). A crash on the new kernel needs nothing from the agent: GRUB already boots the old default.
//
// The host is rebooted by this file only through the wrapper (kernel-reboot, kernel-revert), and only for a staged step.
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"log/slog"
"regexp"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
)
// OpKernelStep is the signed op class that STAGES a kernel set on a ring-1 box (it never reboots: the night leg does,
// once the household was told). CC may sign it until the first paying customer (R-530 ruling).
const OpKernelStep = "os_kernel_step"
// DefaultKernelJudgeWait is how long a one-shot boot may take to come back healthy before the agent reverts it ONCE.
// Measured 2026-10-07 (`audits/kernel-lane-2026-10-07/B/`): every container healthy 68 s after a reboot on demo-felhom,
// 272 s on demo-hp; 20 minutes stays under the hub's 45-minute host_stale (a box that never comes back alarms after it).
const DefaultKernelJudgeWait = 20 * time.Minute
var kverRE = regexp.MustCompile(`^[0-9]+\.[0-9]+\.[0-9]+-[0-9]+-pve$`)
// KernelView is the wrapper's kernel object (felhom-os-apply Kernel.view).
type KernelView struct {
Running string `json:"running"`
Default string `json:"default"`
Flag *string `json:"flag"`
Phase string `json:"phase"`
From string `json:"from"`
To string `json:"to"`
SelfRevertUsed bool `json:"self_revert_used"`
Reason string `json:"reason"`
VMID int `json:"vmid"` // the customer guest the step was staged for (the health rule's guest)
}
// kernelReportRing is the ring an after-boot report carries. In the first second after a boot the agent has not
// fetched the hub's block yet, and Block() then answers ring 1 — so a ring-0 box's „judging" report said ring 1
// (seen on demo-felhom, 2026-10-08 night, `audits/kernel-night-2026-10-07/readback/`). Order: the fetched block; the
// block the daemon saved on disk before the reboot (R-866); ring 1 as before. A label only: the hub's approval reads its
// own ring list. Pinned by TestKernelReportRing_BeforeFirstFetch.
func (l *Leg) kernelReportRing() int {
l.mu.Lock()
fetched := l.block
l.mu.Unlock()
if fetched != nil {
return fetched.Ring
}
if b, _, ok := LoadSavedBlock(l.planDir()); ok && b != nil {
return b.Ring
}
return 1
}
func parseKernel(raw json.RawMessage) KernelView {
var v KernelView
_ = json.Unmarshal(raw, &v)
return v
}
// kernelPlan is one kernel-layer wrapper call.
func kernelPlan(mode string, vmid int, extra map[string]any) map[string]any {
p := map[string]any{"release_id": "kernel", "layer": LayerKernel, "lane": "slow", "vmid": vmid, "mode": mode,
"packages": []Package{}}
for k, v := range extra {
p[k] = v
}
return p
}
// KernelStatus reads the kernel lane's state (read only).
func (l *Leg) KernelStatus(ctx context.Context, vmid int) (KernelView, error) {
wr, err := l.call(ctx, "kstatus"+l.now().UTC().Format("150405"), kernelPlan("kernel-status", vmid, nil))
if err != nil {
return KernelView{}, err
}
if wr.refused() {
return KernelView{}, fmt.Errorf("kernel-status refused: %s", wr.Refused)
}
return parseKernel(wr.Kernel), nil
}
// kernelDue reports whether tonight's leg may take a kernel step, and why not.
func kernelDue(blk hub.WireOSUpdate, trigger string) (bool, string) {
switch {
case trigger != "night":
return false, "a kernel step runs only in the night leg (never a debug pass)"
case blk.Kernel == nil:
return false, "the hub names no kernel step for this box"
case !blk.Enabled:
return false, "OS updates are switched off for this box"
case !kverRE.MatchString(blk.Kernel.Kver):
return false, "the hub's kernel " + blk.Kernel.Kver + " is not a kernel version"
case !blk.Kernel.Tonight:
return false, "the household has not been told about tonight (no mail, no step — `09` §3 decision 172)"
}
return true, ""
}
// runKernel is the night leg's last step. It returns the stage report (Layer "" when nothing ran). On success the box
// is rebooting when it returns.
func (l *Leg) runKernel(ctx context.Context, runID string, vmid int, trigger string, blk hub.WireOSUpdate) Report {
lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid, "trigger", trigger, "ring", blk.Ring)
if ok, why := kernelDue(blk, trigger); !ok {
lg.Info("osupdate: kernel step skipped — " + why)
return Report{}
}
want := blk.Kernel.Kver
st, err := l.KernelStatus(ctx, vmid)
if err != nil {
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid,
Mode: "apply", Outcome: "failed", HealthReason: "kernel status unreadable: " + err.Error()})
}
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid, Mode: "apply", ReleaseID: want}
switch {
case st.Phase == "staged" && st.To == want:
lg.Info("osupdate: kernel step — a staged kernel waits for tonight", "from", st.From, "to", st.To)
rep.Outcome, rep.Healthy = "staged", true
case blk.Ring != 0:
lg.Info("osupdate: kernel step skipped — ring 1 boots only a kernel a signed os_kernel_step staged", "phase", st.Phase, "staged", st.To, "want", want)
return Report{}
default:
// R-898: EXACTLY the kernel the household was told about — never "whatever is pending tonight" (the sources can
// offer a newer one by night; the step then refused, R23, and the night was lost). A version no longer
// installable is refused by the wrapper before any change (R7) and the hub tells the household again.
wr, cerr := l.call(ctx, runID, kernelPlan("apply", vmid, map[string]any{"release_id": "ring0-" + runID,
"select": "listed", "packages": KernelSet(want), "expect_kver": want, "run_id": runID, "trigger": trigger,
"ring": blk.Ring}))
rep.unsent = reportFile(l.planDir(), runID, LayerKernel, "apply")
rep.Kernel = rawOrNil(wr.Kernel)
switch {
case cerr != nil:
rep.Outcome, rep.HealthReason = "failed", cerr.Error()
return l.finish(ctx, lg, rep)
case wr.refused():
rep.Outcome, rep.Refused = "refused", wr.Refused
return l.finish(ctx, lg, rep)
case wr.failed():
rep.Outcome, rep.Refused = "failed", wr.Failed
return l.finish(ctx, lg, rep)
case wr.OutcomeHint == "nothing" || len(wr.Upgraded) == 0:
rep.Outcome, rep.Healthy = "nothing", true
return l.finish(ctx, lg, rep)
}
rep.Outcome, rep.Healthy, rep.Upgraded, rep.Authority, rep.PassSeconds = "staged", true, wr.Upgraded, wr.Authority, wr.PassSeconds
rep.RebootNeeded = true
}
rep = l.finish(ctx, lg, rep) // the hub hears "staged" BEFORE the box goes down
wr, err := l.call(ctx, runID, kernelPlan("kernel-reboot", vmid, nil))
switch {
case err != nil:
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid,
Mode: "kernel-reboot", ReleaseID: want, Outcome: "failed", HealthReason: "kernel-reboot: " + err.Error()})
case wr.refused() || wr.failed():
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: trigger, Ring: blk.Ring, VMID: vmid,
Mode: "kernel-reboot", ReleaseID: want, Outcome: "refused", Refused: firstRaw(wr.Refused, wr.Failed),
Kernel: rawOrNil(wr.Kernel)})
}
lg.Warn("osupdate: kernel step — the box restarts now for its one-shot boot", "to", want)
return rep
}
func firstRaw(a, b json.RawMessage) json.RawMessage {
if r := rawOrNil(a); r != nil {
return r
}
return rawOrNil(b)
}
// KernelJudge is what KernelAfterBoot needs besides the leg: the hub reachability probe is the "judging" report itself.
type KernelJudge struct {
Wait time.Duration // default DefaultKernelJudgeWait
Poll time.Duration // default 30 s
}
// KernelAfterBoot runs once at daemon start: what became of a kernel step across the boot. On the new kernel it judges
// the boot (blocking up to the wait — run it in a goroutine). vmid 0 = the guest the step recorded (it may not run yet).
func (l *Leg) KernelAfterBoot(ctx context.Context, vmid int, j KernelJudge) Report {
runID := "boot-" + l.now().UTC().Format("20060102T150405Z")
lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid)
wr, err := l.call(ctx, runID, kernelPlan("kernel-boot", vmid, nil))
if err != nil {
lg.Warn("osupdate: kernel after-boot check failed", "err", err)
return Report{}
}
if wr.refused() {
lg.Info("osupdate: kernel after-boot check refused (an older wrapper, or a BYO host)", "refused", string(wr.Refused))
return Report{}
}
v := parseKernel(wr.Kernel)
if vmid <= 0 {
vmid = v.VMID // after a boot the guest may not run yet — the step's own record names it
}
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid, Mode: "kernel-boot",
ReleaseID: v.To, Kernel: rawOrNil(wr.Kernel)}
switch wr.KernelEvent {
case "fell_back":
rep.Outcome, rep.HealthReason = "fell_back", v.Reason
lg.Warn("osupdate: kernel step FELL BACK — the new kernel did not come up; the box runs the old one", "from", v.From, "to", v.To)
return l.finish(ctx, lg, rep)
case "self_reverted":
rep.Outcome, rep.HealthReason = "self_reverted", v.Reason
lg.Warn("osupdate: kernel step SELF-REVERTED — back on the old kernel", "from", v.From, "to", v.To, "reason", v.Reason)
return l.finish(ctx, lg, rep)
case "revert_failed":
rep.Outcome, rep.HealthReason = "revert_failed", v.Reason
lg.Error("osupdate: kernel self-revert came back on the NEW kernel — no second revert; the operator decides", "to", v.To)
return l.finish(ctx, lg, rep)
case "judging":
return l.judgeKernel(ctx, runID, vmid, v, wr.HealthBefore, j, lg)
}
return Report{}
}
// KernelVerdict is THE one-shot boot rule (R-836; pinned by TestKernelVerdict): the host health rule (`11` §8.2 —
// the Proxmox daemons and the agent active, the customer guest running and its own rule passing, the tunnel running)
// AND the box reached the hub since this boot.
func KernelVerdict(before, after *Health, tunnel string, hubReached bool) (bool, string) {
if ok, why := HostHealthVerdict(before, after, tunnel); !ok {
return false, why
}
if !hubReached {
return false, "the box has not reached the hub since the boot"
}
return true, ""
}
func (l *Leg) judgeKernel(ctx context.Context, runID string, vmid int, v KernelView, before *Health, j KernelJudge, lg *slog.Logger) Report {
wait, poll := j.Wait, j.Poll
if wait <= 0 {
wait = DefaultKernelJudgeWait
}
if poll <= 0 {
poll = 30 * time.Second
}
lg.Info("osupdate: kernel step — judging the one-shot boot", "from", v.From, "to", v.To, "wait", wait.String())
start := l.now()
deadline := start.Add(wait)
hubReached := false
var why string
for {
if !hubReached && l.Hub != nil {
// the hub's reachability IS this report reaching it (and the operator sees the box is back on the new kernel)
body, _ := json.Marshal(Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
Mode: "kernel-boot", ReleaseID: v.To, Outcome: "judging", Kernel: mustRaw(v)})
rctx, cancel := context.WithTimeout(ctx, 30*time.Second)
if err := l.Hub.PostOSReport(rctx, body); err == nil {
hubReached = true
lg.Info("osupdate: kernel step — the box reached the hub on the new kernel", "after", l.now().Sub(start).Round(time.Second).String())
}
cancel()
}
var h *Health
hr, err := l.call(ctx, runID, kernelPlan("health", vmid, nil))
switch {
case err != nil:
why = "no health reading: " + err.Error()
case hr.refused():
why = "no health reading: " + string(hr.Refused)
default:
h = hr.Health
}
if h != nil {
t := hub.TunnelUnknown
if l.Tunnel != nil {
t, _ = l.Tunnel.Status(ctx)
}
var ok bool
ok, why = KernelVerdict(before, h, t, hubReached)
if ok {
return l.kernelGood(ctx, runID, vmid, v, start, lg)
}
}
if !l.now().Before(deadline) || ctx.Err() != nil {
break
}
l.sleep(ctx, poll)
}
if ctx.Err() != nil {
lg.Warn("osupdate: kernel judging stopped (the agent is stopping) — the next start judges again", "reason", why)
return Report{}
}
// not healthy by the deadline: tell the hub (best effort), then ONE self-revert into the old kernel
rep := l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
Mode: "kernel-revert", ReleaseID: v.To, Outcome: "health_failed", HealthReason: why + " — reverting to " + v.From,
Kernel: mustRaw(v)})
lg.Error("osupdate: kernel step — the one-shot boot is NOT healthy; restarting ONCE into the old kernel", "reason", why,
"waited", wait.String(), "from", v.From, "to", v.To)
wr, err := l.call(ctx, runID, kernelPlan("kernel-revert", vmid, map[string]any{"reason": truncate(why, 280)}))
if err != nil || wr.refused() || wr.failed() {
lg.Error("osupdate: kernel self-revert did not start — the box stays on the new kernel; the operator decides",
"err", err, "refused", string(firstRaw(wr.Refused, wr.Failed)))
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid,
Mode: "kernel-revert", ReleaseID: v.To, Outcome: "revert_failed", Refused: firstRaw(wr.Refused, wr.Failed),
HealthReason: "the self-revert did not start"})
}
return rep
}
func (l *Leg) kernelGood(ctx context.Context, runID string, vmid int, v KernelView, start time.Time, lg *slog.Logger) Report {
wr, err := l.call(ctx, runID, kernelPlan("kernel-good", vmid, nil))
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "boot", Ring: l.kernelReportRing(), VMID: vmid, Mode: "kernel-good",
ReleaseID: v.To}
switch {
case err != nil:
rep.Outcome, rep.HealthReason = "failed", "kernel-good: "+err.Error()
case wr.refused() || wr.failed():
rep.Outcome, rep.Refused, rep.HealthReason = "failed", firstRaw(wr.Refused, wr.Failed), "kernel-good did not move the default"
default:
rep.Outcome, rep.Healthy = "applied", true
rep.HealthReason = fmt.Sprintf("healthy %s after the agent started; the new kernel is the default", l.now().Sub(start).Round(time.Second))
}
rep.Kernel = rawOrNil(wr.Kernel)
lg.Info("osupdate: kernel step — "+rep.Outcome, "to", v.To, "reason", rep.HealthReason)
return l.finish(ctx, lg, rep)
}
func mustRaw(v any) json.RawMessage {
b, _ := json.Marshal(v)
return b
}
func truncate(s string, n int) string {
if len(s) <= n {
return s
}
return s[:n]
}
// KernelSet is the package set that installs exactly kver (R-898; the hub's kernelSet, field-exact): the series
// meta-package and the signed image, both at the kernel's own version. Proxmox keeps old kernel versions in its archive.
// nil for a string that is not a kernel version.
func KernelSet(kver string) []Package {
m := kverSeriesRE.FindStringSubmatch(kver)
if m == nil {
return nil
}
v := kver[:len(kver)-len("-pve")]
return []Package{{Name: "proxmox-kernel-" + m[1], Version: v, Origin: PVEOrigin},
{Name: "proxmox-kernel-" + kver + "-signed", Version: v, Origin: PVEOrigin}}
}
var kverSeriesRE = regexp.MustCompile(`^([0-9]+\.[0-9]+)\.[0-9]+-[0-9]+-pve$`)
// KernelStepParams are a signed os_kernel_step's params: the exact kernel set (the wrapper compares it with the plan).
type KernelStepParams struct {
ReleaseID string `json:"release_id"`
Packages []Package `json:"packages"`
Kver string `json:"kver"`
VMID int `json:"vmid,omitempty"`
}
// KernelStepExecutor STAGES a verified os_kernel_step (signedjobs.Executor) under the heavy-op gate. It never reboots:
// the night leg reboots a staged kernel on a night the household was told about.
type KernelStepExecutor struct {
Leg *Leg
Guest func(ctx context.Context) (int, error)
Gate func(ctx context.Context) (release func(), err error)
}
// Execute implements signedjobs.Executor.
func (e KernelStepExecutor) Execute(ctx context.Context, op string, params json.RawMessage) error {
if op != OpKernelStep {
return signedjobs.ErrNoExecutor
}
so, ok := signedjobs.SignedOpFrom(ctx)
if !ok {
return fmt.Errorf("os_kernel_step: no signed envelope in the context — the wrapper could not verify it")
}
var p KernelStepParams
if err := json.Unmarshal(params, &p); err != nil || len(p.Packages) == 0 || !kverRE.MatchString(p.Kver) {
return fmt.Errorf("os_kernel_step: params must name the kernel set and its kver: %v", err)
}
vmid := p.VMID
if vmid == 0 {
if e.Guest == nil {
return fmt.Errorf("os_kernel_step: no vmid and no guest finder")
}
v, err := e.Guest(ctx)
if err != nil {
return fmt.Errorf("os_kernel_step: find the customer guest: %w", err)
}
vmid = v
}
if e.Gate != nil {
release, err := e.Gate(ctx)
if err != nil {
return fmt.Errorf("os_kernel_step: heavy-op gate busy (a backup or restore-test runs): %w", err)
}
defer release()
}
rep := e.Leg.StageKernelSigned(ctx, vmid, p, so.Blob, string(so.Sig))
if rep.Outcome == "staged" || rep.Outcome == "nothing" {
return nil
}
return fmt.Errorf("os_kernel_step: %s (%s) %s", rep.Outcome, rep.HealthReason, string(rep.Refused))
}
// StageKernelSigned stages a signed kernel set (ring 1): install + flag, no reboot.
func (l *Leg) StageKernelSigned(ctx context.Context, vmid int, p KernelStepParams, blob []byte, sig string) Report {
unlock := l.lockPass(true)
defer unlock()
l.sendUnsentLocked(ctx) // R-868
runID := l.now().UTC().Format("20060102T150405Z")
rid := p.ReleaseID
if rid == "" {
rid = "signed-" + runID
}
lg := l.log().With("run", runID, "layer", LayerKernel, "vmid", vmid, "trigger", "signed", "release", rid)
wr, err := l.call(ctx, runID, kernelPlan("apply", vmid, map[string]any{"release_id": rid, "select": "listed",
"packages": p.Packages, "expect_kver": p.Kver, "run_id": runID, "trigger": "signed", "ring": l.Block().Ring,
"signed": map[string]string{"blob_b64": base64.StdEncoding.EncodeToString(blob), "sig": sig}}))
rep := Report{RunID: runID, Layer: LayerKernel, Trigger: "signed", Ring: l.Block().Ring, VMID: vmid, Mode: "apply",
ReleaseID: rid, Kernel: rawOrNil(wr.Kernel), unsent: reportFile(l.planDir(), runID, LayerKernel, "apply")}
switch {
case err != nil:
rep.Outcome, rep.HealthReason = "failed", err.Error()
case wr.refused():
rep.Outcome, rep.Refused = "refused", wr.Refused
case wr.failed():
rep.Outcome, rep.Refused = "failed", wr.Failed
case wr.OutcomeHint == "nothing" || len(wr.Upgraded) == 0:
rep.Outcome, rep.Healthy = "nothing", true
default:
rep.Outcome, rep.Healthy, rep.Upgraded, rep.Authority, rep.PassSeconds = "staged", true, wr.Upgraded, wr.Authority, wr.PassSeconds
rep.RebootNeeded = true
}
return l.finish(ctx, lg, rep)
}
+328
View File
@@ -0,0 +1,328 @@
package osupdate
import (
"context"
"encoding/base64"
"encoding/json"
"strings"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
)
// ---- the kernel lane (R-836, `09` §3 decision 172, `11` §5.11) ----
const kOld, kNew = "7.0.2-6-pve", "7.0.14-22-pve"
func kview(phase string) json.RawMessage {
return mustRaw(KernelView{Running: kOld, Default: kOld, Phase: phase, From: kOld, To: kNew, VMID: 9201})
}
func tonight(ring int) *hub.WireOSUpdate {
return &hub.WireOSUpdate{Ring: ring, Enabled: true, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: true}}
}
func kernelCalls(w *fakeWrapper) []string {
var m []string
for _, p := range w.plans {
if p["layer"] == LayerKernel {
m = append(m, p["mode"].(string))
}
}
return m
}
// Ring 0, a told night: after the healthy host step the leg stages the pending kernel (select pending-kernel, the
// kernel the household was told about), tells the hub "staged", THEN reboots — the kernel step ends the night.
func TestKernel_Ring0ToldNightStagesThenReboots(t *testing.T) {
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{
"kernel-status": {{Kernel: kview("none")}},
"apply": {{Upgraded: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22"}}, Authority: "ring0", Kernel: kview("staged")}},
"kernel-reboot": {{Kernel: kview("oneshot")}},
}}
l, h := newLeg(t, w, tonight(0))
p := l.Run(context.Background(), 9201, "night")
if got := strings.Join(kernelCalls(w), ","); got != "kernel-status,apply,kernel-reboot" {
t.Fatalf("kernel calls = %s", got)
}
var ap map[string]any
for _, x := range w.plans {
if x["layer"] == LayerKernel && x["mode"] == "apply" {
ap = x
}
}
// R-898: EXACTLY the told kernel — the listed set derived from it, never "pending" (red before the fix: select was
// pending-kernel, so a newer kernel in the sources by night was refused R23 and the night was lost)
pk, _ := json.Marshal(ap["packages"])
if ap["select"] != "listed" || ap["expect_kver"] != kNew || ap["lane"] != "slow" ||
string(pk) != `[{"name":"proxmox-kernel-7.0","origin":"Proxmox Debian Repository","version":"7.0.14-22"},{"name":"proxmox-kernel-7.0.14-22-pve-signed","origin":"Proxmox Debian Repository","version":"7.0.14-22"}]` {
t.Fatalf("stage plan = %v (packages %s)", ap, pk)
}
if p.Kernel.Outcome != "staged" || !p.Kernel.Healthy {
t.Fatalf("kernel report = %+v", p.Kernel)
}
last := h.reports[len(h.reports)-1]
if last.Layer != LayerKernel || last.Outcome != "staged" {
t.Fatalf("the hub must hear 'staged' before the reboot: %+v", h.reports)
}
// the kernel step is the LAST wrapper call of the night
if lp := w.plans[len(w.plans)-1]; lp["layer"] != LayerKernel || lp["mode"] != "kernel-reboot" {
t.Fatalf("the reboot must end the night, last call = %v", lp)
}
}
// No mail, no step: a kernel the household was NOT told about never runs; nor in a debug pass; nor without a block.
// COMPANION RED-PROOF (observed): drop the `!blk.Kernel.Tonight` case in kernelDue → the first sub-case fails.
func TestKernel_NoMailNoStep(t *testing.T) {
cases := map[string]struct {
blk *hub.WireOSUpdate
trigger string
}{
"not told": {&hub.WireOSUpdate{Ring: 0, Enabled: true, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: false}}, "night"},
"debug pass": {tonight(0), "debug"},
"no block": {&hub.WireOSUpdate{Ring: 0, Enabled: true}, "night"},
"switch off": {&hub.WireOSUpdate{Ring: 0, Enabled: false, Kernel: &hub.WireKernelStep{Kver: kNew, Tonight: true}}, "night"},
"bad kver": {&hub.WireOSUpdate{Ring: 0, Enabled: true, Kernel: &hub.WireKernelStep{Kver: "7.0; reboot", Tonight: true}}, "night"},
}
for name, c := range cases {
w := &fakeWrapper{t: t}
l, _ := newLeg(t, w, c.blk)
p := l.Run(context.Background(), 9201, c.trigger)
if len(kernelCalls(w)) != 0 || p.Kernel.Layer != "" {
t.Fatalf("%s: a kernel step ran: %v", name, kernelCalls(w))
}
}
}
// The kernel step needs a healthy host step on an appliance, and a healthy Proxmox step when one ran.
func TestKernel_SkippedWithoutHealthyEarlierSteps(t *testing.T) {
w := &fakeWrapper{t: t}
l, _ := newLeg(t, w, tonight(0))
l.Appliance = false
l.Run(context.Background(), 9201, "night")
if len(kernelCalls(w)) != 0 {
t.Fatalf("a BYO box took a kernel step: %v", kernelCalls(w))
}
w2 := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerPVE: {Upgraded: []Package{{Name: "pve-manager", Version: "9.2.21"}},
PVEManager: "9.2.2"}}} // pveversion still old → the pve step is unhealthy
l2, _ := newLeg(t, w2, tonight(0))
l2.Run(context.Background(), 9201, "night")
if len(kernelCalls(w2)) != 0 {
t.Fatalf("a kernel step ran after an unhealthy Proxmox step: %v", kernelCalls(w2))
}
}
// Ring 1 reboots only a kernel a signed job staged — never stages one itself in the night leg.
func TestKernel_Ring1RebootsOnlyASignedStage(t *testing.T) {
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}}}}
l, _ := newLeg(t, w, tonight(1))
l.Run(context.Background(), 9201, "night")
if got := strings.Join(kernelCalls(w), ","); got != "kernel-status" {
t.Fatalf("ring 1 without a staged kernel: calls = %s", got)
}
w2 := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("staged")}},
"kernel-reboot": {{Kernel: kview("oneshot")}}}}
l2, _ := newLeg(t, w2, tonight(1))
p := l2.Run(context.Background(), 9201, "night")
if got := strings.Join(kernelCalls(w2), ","); got != "kernel-status,kernel-reboot" || p.Kernel.Outcome != "staged" {
t.Fatalf("ring 1 with a staged kernel: calls = %s report = %+v", got, p.Kernel)
}
}
// A refused stage never reboots.
func TestKernel_RefusedStageNeverReboots(t *testing.T) {
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-status": {{Kernel: kview("none")}},
"apply": {{Refused: json.RawMessage(`{"code":"R20","reason":"/boot/efi is not a mounted vfat ESP"}`)}}}}
l, h := newLeg(t, w, tonight(0))
p := l.Run(context.Background(), 9201, "night")
if got := strings.Join(kernelCalls(w), ","); got != "kernel-status,apply" || p.Kernel.Outcome != "refused" {
t.Fatalf("calls = %s report = %+v", got, p.Kernel)
}
if last := h.reports[len(h.reports)-1]; last.Layer != LayerKernel || last.Outcome != "refused" {
t.Fatalf("the hub must hear the refusal: %+v", last)
}
}
// THE one-shot boot rule: the host rule AND the hub reached. COMPANION RED-PROOF (observed): drop the hubReached
// check in KernelVerdict → the second case fails.
func TestKernelVerdict(t *testing.T) {
if ok, why := KernelVerdict(hostOK(), hostOK(), hub.TunnelRunning, true); !ok {
t.Fatalf("a healthy boot read unhealthy: %s", why)
}
if ok, why := KernelVerdict(hostOK(), hostOK(), hub.TunnelRunning, false); ok || !strings.Contains(why, "hub") {
t.Fatalf("a box that has not reached the hub must not pass: ok=%v %q", ok, why)
}
down := hostOK()
down.GuestRunning = new(bool)
if ok, _ := KernelVerdict(hostOK(), down, hub.TunnelRunning, true); ok {
t.Fatal("a guest that does not run must fail")
}
if ok, _ := KernelVerdict(hostOK(), hostOK(), hub.TunnelUnknown, true); ok {
t.Fatal("an unknown tunnel must fail (the host rule)")
}
}
func judgingLeg(t *testing.T, w *fakeWrapper) (*Leg, *fakeHub) {
if w.kernelRep == nil {
w.kernelRep = map[string][]WrapperReport{}
}
if _, ok := w.kernelRep["kernel-boot"]; !ok {
w.kernelRep["kernel-boot"] = []WrapperReport{{KernelEvent: "judging", Kernel: mustRaw(KernelView{Running: kNew,
Default: kOld, Phase: "judging", From: kOld, To: kNew, VMID: 9201}), HealthBefore: hostOK()}}
}
return newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
}
// A healthy one-shot boot: the hub hears "judging", then kernel-good, then "applied".
func TestKernelAfterBoot_HealthyBecomesTheDefault(t *testing.T) {
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-good": {{Kernel: kview("good")}}}}
l, h := judgingLeg(t, w)
r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 10 * time.Minute, Poll: 30 * time.Second})
if got := strings.Join(kernelCalls(w), ","); got != "kernel-boot,health,kernel-good" {
t.Fatalf("calls = %s", got)
}
if r.Outcome != "applied" || !r.Healthy {
t.Fatalf("report = %+v", r)
}
if len(h.reports) != 2 || h.reports[0].Outcome != "judging" || h.reports[1].Outcome != "applied" {
t.Fatalf("hub reports = %+v", h.reports)
}
if w.plans[1]["vmid"] != float64(9201) {
t.Fatalf("the health reading must use the step's own guest, got %v", w.plans[1]["vmid"])
}
}
// An unhealthy one-shot boot: wait the full judge time, tell the hub, then ONE kernel-revert.
// COMPANION RED-PROOF (observed): return before the kernel-revert call in judgeKernel → "calls" fails.
func TestKernelAfterBoot_UnhealthyRevertsOnceAfterTheWait(t *testing.T) {
down := hostOK()
down.GuestRunning = new(bool)
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"health": {{Health: down}},
"kernel-revert": {{Kernel: kview("reverting")}}}}
l, h := judgingLeg(t, w)
start := l.now()
r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 10 * time.Minute, Poll: time.Minute})
calls := kernelCalls(w)
if calls[len(calls)-1] != "kernel-revert" || strings.Count(strings.Join(calls, ","), "kernel-revert") != 1 {
t.Fatalf("calls = %v", calls)
}
if waited := l.now().Sub(start); waited < 10*time.Minute {
t.Fatalf("reverted after %s — before the judge wait", waited)
}
if r.Outcome != "health_failed" || !strings.Contains(r.HealthReason, "not running") {
t.Fatalf("report = %+v", r)
}
if last := h.reports[len(h.reports)-1]; last.Outcome != "health_failed" {
t.Fatalf("the hub must hear health_failed before the revert reboot: %+v", h.reports)
}
for _, p := range w.plans {
if p["mode"] == "kernel-revert" && !strings.Contains(p["reason"].(string), "not running") {
t.Fatalf("the revert must carry the reason: %v", p)
}
}
}
// A box that never reaches the hub is not "healthy" — it reverts too.
func TestKernelAfterBoot_NoHubMeansRevert(t *testing.T) {
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-revert": {{Kernel: kview("reverting")}}}}
l, _ := judgingLeg(t, w)
l.Hub = unreachableHub{}
l.KernelAfterBoot(context.Background(), 0, KernelJudge{Wait: 5 * time.Minute, Poll: time.Minute})
if c := kernelCalls(w); c[len(c)-1] != "kernel-revert" {
t.Fatalf("calls = %v", c)
}
}
type unreachableHub struct{}
func (unreachableHub) PostOSReport(context.Context, []byte) error { return context.DeadlineExceeded }
// What kernel-boot found becomes the hub's outcome, with no judging and no reboot.
func TestKernelAfterBoot_FallBackAndRevertResultsAreReported(t *testing.T) {
for ev, want := range map[string]string{"fell_back": "fell_back", "self_reverted": "self_reverted", "revert_failed": "revert_failed"} {
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-boot": {{KernelEvent: ev,
Kernel: mustRaw(KernelView{Running: kOld, Default: kOld, Phase: ev, From: kOld, To: kNew, Reason: "r", VMID: 9201})}}}}
l, h := judgingLeg(t, w)
r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{})
if r.Outcome != want || len(h.reports) != 1 || h.reports[0].Outcome != want {
t.Fatalf("%s: report %+v hub %+v", ev, r, h.reports)
}
if got := strings.Join(kernelCalls(w), ","); got != "kernel-boot" {
t.Fatalf("%s: calls = %s", ev, got)
}
}
// nothing to do → nothing reported
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"kernel-boot": {{KernelEvent: "none", Kernel: kview("good")}}}}
l, h := judgingLeg(t, w)
if r := l.KernelAfterBoot(context.Background(), 0, KernelJudge{}); r.Layer != "" || len(h.reports) != 0 {
t.Fatalf("an ordinary boot must report nothing: %+v %+v", r, h.reports)
}
}
// The signed executor STAGES (listed + the raw envelope + the kver) and never reboots.
func TestKernelStepExecutor_StagesNeverReboots(t *testing.T) {
w := &fakeWrapper{t: t, kernelRep: map[string][]WrapperReport{"apply": {{Upgraded: []Package{{Name: "proxmox-kernel-7.0",
Version: "7.0.14-22"}}, Authority: "signed", Kernel: kview("staged")}}}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
e := KernelStepExecutor{Leg: l, Guest: func(context.Context) (int, error) { return 9201, nil }}
params, _ := json.Marshal(KernelStepParams{ReleaseID: "os-kernel-1", Kver: kNew,
Packages: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22", Origin: PVEOrigin}}})
ctx := signedjobs.WithSignedOp(context.Background(), &reconcile.SignedOp{Blob: []byte(`{"op":"os_kernel_step"}`), Sig: []byte("SIG")})
if err := e.Execute(ctx, OpKernelStep, params); err != nil {
t.Fatal(err)
}
pp := w.plans[len(w.plans)-1]
sg, _ := pp["signed"].(map[string]any)
if pp["mode"] != "apply" || pp["select"] != "listed" || pp["expect_kver"] != kNew || sg == nil ||
sg["blob_b64"] != base64.StdEncoding.EncodeToString([]byte(`{"op":"os_kernel_step"}`)) {
t.Fatalf("plan = %v", pp)
}
if got := strings.Join(kernelCalls(w), ","); got != "apply" {
t.Fatalf("a signed stage must never reboot: %s", got)
}
if len(h.reports) != 1 || h.reports[0].Outcome != "staged" {
t.Fatalf("hub = %+v", h.reports)
}
if err := e.Execute(context.Background(), OpKernelStep, params); err == nil {
t.Fatal("no envelope must refuse")
}
bad, _ := json.Marshal(KernelStepParams{Kver: "x", Packages: []Package{{Name: "a"}}})
if err := e.Execute(ctx, OpKernelStep, bad); err == nil {
t.Fatal("a bad kver must refuse")
}
if err := e.Execute(ctx, OpPVEStep, params); err != signedjobs.ErrNoExecutor {
t.Fatalf("another op must pass through the chain: %v", err)
}
}
// os_kernel_step is never benign.
func TestKernelStep_IsDestructiveClass(t *testing.T) {
if reconcile.Classify(reconcile.ClassOSKernelStep, reconcile.Provenance{}) != reconcile.Destructive {
t.Fatal("os_kernel_step must be destructive-class (signed, operational key)")
}
}
// A kept stage report (the agent was killed mid-stage) reaches the hub as "staged" with its kernel view.
func TestKernel_KeptStageReportIsStaged(t *testing.T) {
w := &fakeWrapper{t: t}
l, _ := newLeg(t, w, tonight(0))
ring := 0
rep := l.reportFromKept(context.Background(), WrapperReport{Layer: LayerKernel, Mode: "apply", Ring: &ring,
Upgraded: []Package{{Name: "proxmox-kernel-7.0", Version: "7.0.14-22"}}, Kernel: kview("staged")}, "/x/report-r-kernel-apply.json")
if rep.Outcome != "staged" || !rep.Healthy || len(rep.Kernel) == 0 {
t.Fatalf("kept = %+v", rep)
}
}
func TestKernelSet(t *testing.T) {
if got := KernelSet("7.0.14-20-pve"); len(got) != 2 || got[0].Name != "proxmox-kernel-7.0" || got[0].Version != "7.0.14-20" ||
got[1].Name != "proxmox-kernel-7.0.14-20-pve-signed" {
t.Fatalf("%+v", got)
}
if KernelSet("7.0; reboot") != nil || KernelSet("") != nil {
t.Fatal("a non-kernel string must give no set")
}
}
+153 -6
View File
@@ -20,6 +20,7 @@ import (
"log/slog"
"os"
"path/filepath"
"regexp"
"sort"
"strings"
"sync"
@@ -27,6 +28,7 @@ import (
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/proxmox"
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
)
// WrapperPath is the pinned sudoers vector (configs/felhom-agent.sudoers FELHOM_OSAPPLY).
@@ -40,8 +42,24 @@ const (
LayerGuest = "guest"
LayerHost = "host"
LayerDocker = "docker" // the guest's Docker engine set — slow lane (`11` §5.8)
// LayerPVE is the HOST's Proxmox userspace packages — slow lane (R-812 option A, `09` §3 decision 163, `11` §5.10):
// ring 0 in the night leg after a healthy host step, ring 1 only inside a signed os_pve_step. Never a kernel.
LayerPVE = "pve"
// LayerKernel is the HOST's kernel — the kernel lane (R-836, `09` §3 decision 172, `11` §5.11): a one-shot boot of
// the new kernel through the ESP flag, the default moved only after a healthy boot (kernel.go).
LayerKernel = "kernel"
)
// PVEOrigin is the origin apt prints for download.proxmox.com (the wrapper's PVE_ORIGIN); InstalledPVEOrigin is how
// the wrapper's inventory names the same source.
const (
PVEOrigin = "Proxmox Debian Repository"
InstalledPVEOrigin = "Proxmox"
)
// hostSlowRE mirrors the wrapper's HOST_SLOW_RE: kernel, boot, firmware and microcode names never ride the pve lane.
var hostSlowRE = regexp.MustCompile(`^(linux-(image|headers|kbuild|modules|base)|proxmox-kernel|proxmox-default-kernel|pve-kernel|pve-firmware|firmware-|grub|shim|systemd-boot|intel-microcode|amd64-microcode|efibootmgr)`)
// DockerNames are the six packages of the Docker engine set (the wrapper's DOCKER_NAMES).
var DockerNames = map[string]bool{"containerd.io": true, "docker-buildx-plugin": true, "docker-ce": true,
"docker-ce-cli": true, "docker-ce-rootless-extras": true, "docker-compose-plugin": true}
@@ -109,6 +127,13 @@ type WrapperReport struct {
// OOMCheck (R-528, `09` decision 157): the docker layer's memory-kill check, {result, oom_killed, oom_event,
// exit_code, image, detail}. Carried to the hub UNCHANGED; the agent never reads it.
OOMCheck json.RawMessage `json:"oom_check"`
// PVEManager: the pve layer — pveversion's pve-manager version after the step ("unknown" = unreadable).
PVEManager string `json:"pve_manager"`
// Kernel (R-836): the kernel layer's view {running, default, flag, phase, from, to, …}; KernelEvent what kernel-boot
// found after a boot; OutcomeHint "nothing" when no kernel was pending.
Kernel json.RawMessage `json:"kernel"`
KernelEvent string `json:"kernel_event"`
OutcomeHint string `json:"outcome_hint"`
// R-868 (v0.144.0): the agent's ids, echoed from the plan, so a report kept on disk can be sent without the
// agent process that started the pass. ReleaseID / VMID were always in the report.
RunID string `json:"run_id"`
@@ -149,6 +174,12 @@ type Report struct {
// OOMCheck: docker layer — the wrapper's oom_check object, byte-for-byte (R-528; the hub decides approval on it).
// Pinned by TestDocker_OOMCheckReachesTheHubUnchanged and TestR868_KeptCopyCarriesTheOOMCheck.
OOMCheck json.RawMessage `json:"oom_check,omitempty"`
// PVEManager: pve layer — pve-manager's version after the step (the hub's System page; R-812 option A).
PVEManager string `json:"pve_manager,omitempty"`
// Kernel: kernel layer — the wrapper's kernel view, byte for byte (running, default, flag, phase, from, to). The
// outcomes of this layer: staged | applied (the new kernel is the default) | fell_back | health_failed (self-revert
// started) | self_reverted | revert_failed | judging | nothing | refused | failed.
Kernel json.RawMessage `json:"kernel,omitempty"`
unsent string // R-868: the wrapper's kept copy of this pass's report — deleted once the hub has it
}
@@ -386,6 +417,36 @@ func EngineOf(pkgVersion string) string {
return v
}
// PVEHealthVerdict is THE Proxmox-package-step health rule (R-812 option A; pinned by TestPVEHealthVerdict): the host
// rule (every host service active, the guest running and healthy, the tunnel running), plus every container running at
// the start still runs as the SAME container (a Proxmox step must not restart the household's apps), plus pveversion
// now reports the pve-manager the step installed (wantPVE "" = pve-manager was not in the step).
func PVEHealthVerdict(before, after *Health, tunnel, wantPVE, gotPVE string) (bool, string) {
if ok, why := HostHealthVerdict(before, after, tunnel); !ok {
return false, why
}
if before != nil && before.Guest != nil && after.Guest != nil {
names := make([]string, 0, len(before.Guest.Containers))
for n := range before.Guest.Containers {
names = append(names, n)
}
sort.Strings(names)
for _, n := range names {
b := before.Guest.Containers[n]
if b.State != "running" || b.ID == "" {
continue
}
if a := after.Guest.Containers[n]; a.ID != b.ID {
return false, n + " is a new container (id changed) — the Proxmox step restarted the household's app"
}
}
}
if wantPVE != "" && gotPVE != wantPVE {
return false, "pveversion reads pve-manager " + gotPVE + ", not " + wantPVE
}
return true, ""
}
// DockerHealthVerdict is THE Docker-step health rule (`11` §5.8; pinned by TestDockerHealthVerdict): the guest rule,
// plus every container running at the start still runs as the SAME container (same id — a changed id means the
// household's apps restarted, which `live-restore` exists to prevent), plus the engine now reports the version the step
@@ -451,7 +512,7 @@ func (l *Leg) call(ctx context.Context, runID string, plan map[string]any) (Wrap
// Pass is one leg's reports; an empty Layer means the step did not run.
type Pass struct {
Guest, Host, Docker Report
Guest, Host, Docker, PVE, Kernel Report
}
// Run is one pass: the guest layer, then (on an appliance, after a good guest step) the host layer, then (ring 0
@@ -479,13 +540,54 @@ func (l *Leg) Run(ctx context.Context, vmid int, trigger string) Pass {
if err := l.EnsureLiveRestore(ctx, g.RunID, vmid); err != nil {
p.Docker = l.finish(ctx, lg, Report{RunID: g.RunID, Layer: LayerDocker, Trigger: trigger, Ring: 0, VMID: vmid,
Mode: "apply", Outcome: "failed", HealthReason: "live-restore could not be turned on: " + err.Error()})
return p
} else {
p.Docker = l.runLayer(ctx, g.RunID, LayerDocker, vmid, trigger, blk, dockerOpts{})
}
p.Docker = l.runLayer(ctx, g.RunID, LayerDocker, vmid, trigger, blk, dockerOpts{})
}
// R-812 option A: the Proxmox package step — ring 0, an appliance, after a HEALTHY host step (it is a host change).
// A Docker step's outcome does not gate it (the Docker set lives in the guest). Pinned by TestPVE_*.
switch {
case blk.Ring != 0 || !blk.Enabled:
lg.Info("osupdate: pve step skipped — ring 1 takes a Proxmox set only inside a signed operator job (`11` §5.10)", "ring", blk.Ring, "enabled", blk.Enabled)
case !l.Appliance || h.Layer == "" || !okStep(h):
lg.Info("osupdate: pve step skipped — no healthy host step this pass (an appliance only)", "appliance", l.Appliance, "host_outcome", h.Outcome)
default:
p.PVE = l.runPVE(ctx, g.RunID, vmid, trigger, blk, dockerOpts{})
}
// R-836 (`09` §3 decision 172): the kernel step ENDS the night — an appliance, after a healthy host step (and a
// healthy Proxmox step when one ran), only on a night the hub marks as told. It reboots the box. Pinned by TestKernel_*.
switch {
case !l.Appliance || h.Layer == "" || !okStep(h):
lg.Info("osupdate: kernel step skipped — no healthy host step this pass (an appliance only)", "appliance", l.Appliance, "host_outcome", h.Outcome)
case p.PVE.Layer != "" && !okStep(p.PVE):
lg.Warn("osupdate: kernel step skipped — the Proxmox step did not end healthy", "pve_outcome", p.PVE.Outcome)
default:
p.Kernel = l.runKernel(ctx, g.RunID, vmid, trigger, blk)
}
return p
}
// pveDrainWait bounds how long a pve step waits for the agent's own /etc/pve writes in flight (pvegate).
var pveDrainWait = 2 * time.Minute
// runPVE runs the pve layer while holding pvegate: the agent's own /etc/pve writes wait until it ends.
func (l *Leg) runPVE(ctx context.Context, runID string, vmid int, trigger string, blk hub.WireOSUpdate, do dockerOpts) Report {
lg := l.log().With("run", runID, "layer", LayerPVE, "vmid", vmid, "trigger", trigger)
dctx, cancel := context.WithTimeout(ctx, pveDrainWait)
end, err := pvegate.Step(dctx)
cancel()
if err != nil {
return l.finish(ctx, lg, Report{RunID: runID, Layer: LayerPVE, Trigger: trigger, Ring: blk.Ring, VMID: vmid, Mode: "apply",
ReleaseID: do.releaseID, Outcome: "failed", HealthReason: "an agent write to /etc/pve did not finish in time (pvegate): " + err.Error()})
}
lg.Info("osupdate: pve step holds the /etc/pve write gate — the agent's own writes wait until it ends")
defer func() {
end()
lg.Info("osupdate: pve step released the /etc/pve write gate")
}()
return l.runLayer(ctx, runID, LayerPVE, vmid, trigger, blk, do)
}
// dockerOpts is a signed Docker step (DockerStepExecutor); the zero value is ring 0's unsigned "pending-docker".
type dockerOpts struct {
releaseID string
@@ -571,7 +673,7 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
wire = blk.HostRelease
}
lane := "fast"
if layer == LayerDocker {
if layer == LayerDocker || layer == LayerPVE {
lane = "slow"
if do.signed != nil {
rel = hub.WireOSRelease{ID: do.releaseID}
@@ -604,6 +706,13 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
}
case layer == LayerDocker:
plan["select"] = "pending-docker" // ring 0: the wrapper checks the box's ROOT-OWNED ring-0 mark itself
case layer == LayerPVE && do.signed != nil:
plan["packages"], plan["signed"] = do.packages, do.signed
for _, p := range do.packages {
planned[p.Name] = true
}
case layer == LayerPVE:
plan["select"] = "pending-pve" // ring 0: the same root-owned mark; the wrapper picks installed Proxmox userspace
case !blk.Enabled:
plan["mode"] = "inventory"
lg.Info("osupdate: switched OFF for this box — reporting only")
@@ -637,6 +746,7 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
rep.Upgraded, rep.PassSeconds = wr.Upgraded, wr.PassSeconds
rep.DockerEngine, rep.Authority, rep.Undo = wr.DockerEngine, wr.Authority, wr.Undo
rep.OOMCheck = rawOrNil(wr.OOMCheck)
rep.PVEManager = wr.PVEManager
if rep.Outcome == "" {
switch {
case rep.Mode == "inventory" && !blk.Enabled:
@@ -654,21 +764,27 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
}
// Health: compare with the start of the pass; give restarted services time (only after an install).
cur := wr.HealthAfter
wantEngine := ""
wantEngine, wantPVE := "", ""
for _, u := range wr.Upgraded {
if u.Name == "docker-ce" {
wantEngine = EngineOf(u.Version)
}
if u.Name == "pve-manager" {
wantPVE = u.Version
}
}
verdict := func(h *Health) (bool, string) {
if layer == LayerDocker {
return DockerHealthVerdict(wr.HealthBefore, h, wantEngine, wr.DockerEngine)
}
if layer == LayerHost {
if layer == LayerHost || layer == LayerPVE {
t := hub.TunnelUnknown
if l.Tunnel != nil {
t, _ = l.Tunnel.Status(ctx)
}
if layer == LayerPVE {
return PVEHealthVerdict(wr.HealthBefore, h, t, wantPVE, wr.PVEManager)
}
return HostHealthVerdict(wr.HealthBefore, h, t)
}
return HealthVerdict(wr.HealthBefore, h)
@@ -708,6 +824,10 @@ func (l *Leg) runLayer(ctx context.Context, runID, layer string, vmid int, trigg
// the docker report carries the engine set only (the guest report already carries the Debian packages)
rep.Installed, rep.Pending = onlyDocker(wr.Installed), onlyDockerPending(wr.Pending)
rep.NotCovered = nil
} else if layer == LayerPVE {
// the pve report carries the Proxmox userspace set only — the hub's candidate is built from it
rep.Installed, rep.Pending = onlyPVE(wr.Installed), onlyPVEPending(wr.Pending)
rep.NotCovered = nil
} else {
rep.NotCovered = notCovered(wr.Pending, blk.Ring, planned)
}
@@ -732,6 +852,33 @@ func onlyDocker(in []Package) []Package {
return out
}
// onlyPVE keeps the installed Proxmox-origin packages the pve lane may touch (never a kernel / boot / firmware name).
func onlyPVE(in []Package) []Package {
var out []Package
for _, p := range in {
if (p.Origin == InstalledPVEOrigin || p.Origin == PVEOrigin) && !hostSlowRE.MatchString(p.Name) && !DockerNames[p.Name] {
out = append(out, p)
}
}
return out
}
func onlyPVEPending(in []Pending) []Pending {
var out []Pending
for _, p := range in {
if p.From == "" || hostSlowRE.MatchString(p.Name) || DockerNames[p.Name] {
continue
}
for _, o := range p.Origin {
if o == PVEOrigin {
out = append(out, p)
break
}
}
}
return out
}
func onlyDockerPending(in []Pending) []Pending {
var out []Pending
for _, p := range in {
+33 -12
View File
@@ -13,16 +13,19 @@ import (
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
)
// fakeWrapper plays /usr/local/sbin/felhom-os-apply: it reads the plan the leg wrote and answers per layer and mode.
type fakeWrapper struct {
t *testing.T
pending []Pending
applyRep map[string]WrapperReport // per layer
healthSeq map[string][]*Health // per layer: answers to successive "health" calls
plans []map[string]any
keep bool // R-868: like the real wrapper, keep an apply report beside the plan
t *testing.T
pending []Pending
applyRep map[string]WrapperReport // per layer
healthSeq map[string][]*Health // per layer: answers to successive "health" calls
plans []map[string]any
keep bool // R-868: like the real wrapper, keep an apply report beside the plan
pveGateHeld bool
kernelRep map[string][]WrapperReport // R-836: per kernel-layer mode, successive answers (the last one repeats)
}
func yes() *bool { b := true; return &b }
@@ -50,9 +53,22 @@ func (f *fakeWrapper) Run(_ context.Context, name string, args ...string) ([]byt
f.plans = append(f.plans, plan)
layer := plan["layer"].(string)
ok := guestOK()
if layer == LayerHost {
if layer == LayerHost || layer == LayerPVE || layer == LayerKernel {
ok = hostOK()
}
if layer == LayerKernel {
if seq := f.kernelRep[plan["mode"].(string)]; len(seq) > 0 {
rep := seq[0]
if len(seq) > 1 {
f.kernelRep[plan["mode"].(string)] = seq[1:]
}
out, _ := json.Marshal(rep)
return []byte("OSAPPLY-REPORT " + string(out) + "\n"), []byte("os-apply: DONE rc=0\n"), nil
}
}
if layer == LayerPVE && plan["mode"] == "apply" {
f.pveGateHeld = pvegate.Stepping() // R-812: the /etc/pve write gate must be held while the pve step runs
}
var rep WrapperReport
switch plan["mode"] {
case "inventory":
@@ -160,8 +176,8 @@ func TestRing0_OneCallPerLayer(t *testing.T) {
if g.Outcome != "applied" || !g.Healthy || ho.Outcome != "applied" || !ho.Healthy {
t.Fatalf("guest %+v\nhost %+v", g, ho)
}
if calls(w) != "guest:apply,host:apply,guest:live-restore-on,docker:apply" {
t.Fatalf("calls = %s, want one apply per layer, guest first, then live-restore and the ring-0 docker step", calls(w))
if calls(w) != "guest:apply,host:apply,guest:live-restore-on,docker:apply,pve:apply" {
t.Fatalf("calls = %s, want one apply per layer, guest first, then live-restore, the ring-0 docker step and the pve step", calls(w))
}
for _, p := range w.plans[:2] {
if p["select"] != "pending-fast" || p["snapshot"] != "" || len(p["packages"].([]any)) != 0 {
@@ -171,7 +187,7 @@ func TestRing0_OneCallPerLayer(t *testing.T) {
if len(g.NotCovered) != 1 || g.NotCovered[0] != "docker-ce" {
t.Fatalf("not covered = %v", g.NotCovered)
}
if len(h.reports) != 3 || h.reports[0].Layer != LayerGuest || h.reports[1].Layer != LayerHost || h.reports[2].Layer != LayerDocker {
if len(h.reports) != 4 || h.reports[0].Layer != LayerGuest || h.reports[1].Layer != LayerHost || h.reports[2].Layer != LayerDocker || h.reports[3].Layer != LayerPVE {
t.Fatalf("hub got %+v", h.reports)
}
}
@@ -389,7 +405,7 @@ func TestHostReport_CarriesRebootScanned(t *testing.T) {
}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
run2(l, "night")
if len(h.reports) != 3 || !h.reports[1].RebootScanned || !h.reports[1].RebootNeeded || h.reports[0].RebootScanned {
if len(h.reports) != 4 || !h.reports[1].RebootScanned || !h.reports[1].RebootNeeded || h.reports[0].RebootScanned {
t.Fatalf("hub got %+v", h.reports)
}
}
@@ -436,7 +452,12 @@ func TestDocker_Ring0PlanAndReport(t *testing.T) {
DockerEngine: "29.8.2", Authority: "ring0"}}}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
p := l.Run(context.Background(), 9201, "night")
dp := w.plans[len(w.plans)-1]
var dp map[string]any
for _, x := range w.plans {
if x["layer"] == "docker" {
dp = x
}
}
if dp["layer"] != "docker" || dp["lane"] != "slow" || dp["select"] != "pending-docker" {
t.Fatalf("docker plan = %v", dp)
}
+152
View File
@@ -0,0 +1,152 @@
package osupdate
import (
"context"
"encoding/base64"
"encoding/json"
"strings"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
"gitea.dooplex.hu/admin/felhom-agent/internal/reconcile"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
)
// ---- the Proxmox package step (R-812 option A, `09` §3 decision 163, `11` §5.10) ----
// Ring 0: after a healthy host step the leg runs the pve layer — slow lane, select pending-pve — while holding the
// /etc/pve write gate; the report carries only Proxmox userspace packages and pve-manager's version.
//
// COMPANION RED-PROOF (observed): call runLayer instead of runPVE in Run → "the /etc/pve write gate was not held".
func TestPVE_Ring0PlanGateAndReport(t *testing.T) {
w := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerPVE: {
Upgraded: []Package{{Name: "pve-manager", Version: "9.2.21"}},
Installed: []Package{{Name: "pve-manager", Version: "9.2.21", Origin: "Proxmox"},
{Name: "proxmox-kernel-helper", Version: "9.0.4", Origin: "Proxmox"}, {Name: "libc6", Version: "u4", Origin: "Debian"}},
Pending: []Pending{{Name: "qemu-server", From: "9.0.1", To: "9.0.9", Origin: []string{PVEOrigin}},
{Name: "proxmox-kernel-7.0", From: "7.0.2", To: "7.0.14", Origin: []string{PVEOrigin}},
{Name: "libc6", From: "u3", To: "u4", Origin: []string{"Debian"}}},
PVEManager: "9.2.21", Authority: "ring0"}}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
p := l.Run(context.Background(), 9201, "night")
var pp map[string]any
for _, x := range w.plans {
if x["layer"] == LayerPVE {
pp = x
}
}
if pp == nil || pp["lane"] != "slow" || pp["select"] != "pending-pve" {
t.Fatalf("pve plan = %v (calls %s)", pp, calls(w))
}
if !w.pveGateHeld {
t.Fatal("the /etc/pve write gate was not held while the pve step ran")
}
if pvegate.Stepping() {
t.Fatal("the gate must be released after the step")
}
r := p.PVE
if r.Outcome != "applied" || !r.Healthy || r.PVEManager != "9.2.21" {
t.Fatalf("pve report = %+v", r)
}
if len(r.Installed) != 1 || r.Installed[0].Name != "pve-manager" || len(r.Pending) != 1 || r.Pending[0].Name != "qemu-server" {
t.Fatalf("the pve report must carry Proxmox userspace only: installed=%v pending=%v", r.Installed, r.Pending)
}
if h.reports[len(h.reports)-1].Layer != LayerPVE {
t.Fatalf("the hub must get the pve report: %+v", h.reports)
}
}
// Ring 1 never takes a Proxmox step in the night leg.
func TestPVE_Ring1NightLegNeverSteps(t *testing.T) {
w := &fakeWrapper{t: t}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
if p := l.Run(context.Background(), 9201, "night"); p.PVE.Layer != "" || strings.Contains(calls(w), "pve") {
t.Fatalf("ring 1 took a pve step: %s", calls(w))
}
}
// No healthy host step (a BYO box, or an unhealthy host step) → no pve step.
func TestPVE_SkippedWithoutAHealthyHostStep(t *testing.T) {
w := &fakeWrapper{t: t}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
l.Appliance = false
if p := l.Run(context.Background(), 9201, "night"); p.PVE.Layer != "" || strings.Contains(calls(w), "pve") {
t.Fatalf("a BYO box took a pve step: %s", calls(w))
}
w2 := &fakeWrapper{t: t}
l2, _ := newLeg(t, w2, &hub.WireOSUpdate{Ring: 0, Enabled: true})
l2.Tunnel = fakeTunnel{"stopped"} // the host step reads unhealthy
w2.applyRep = map[string]WrapperReport{LayerHost: {Upgraded: []Package{{Name: "libc6", Version: "u4"}}}}
if p := l2.Run(context.Background(), 9201, "night"); p.PVE.Layer != "" || strings.Contains(calls(w2), "pve") {
t.Fatalf("a pve step ran after an unhealthy host step: %s", calls(w2))
}
}
// A write in flight that never finishes makes the pve step give up (failed), never run without the gate.
func TestPVE_GivesUpWhenAWriteDoesNotFinish(t *testing.T) {
old := pveDrainWait
pveDrainWait = 50_000_000 // 50 ms
defer func() { pveDrainWait = old }()
rel, _, _ := pvegate.Write(context.Background())
defer rel()
w := &fakeWrapper{t: t}
l, _ := newLeg(t, w, &hub.WireOSUpdate{Ring: 0, Enabled: true})
p := l.Run(context.Background(), 9201, "night")
if p.PVE.Outcome != "failed" || strings.Contains(calls(w), "pve") {
t.Fatalf("the pve step must fail without a wrapper call: %+v calls=%s", p.PVE, calls(w))
}
}
// THE pve health rule. COMPANION RED-PROOF (observed): drop the container-id loop or the pve-manager check in
// PVEHealthVerdict → the matching case below fails.
func TestPVEHealthVerdict(t *testing.T) {
before, after := hostOK(), hostOK()
before.Guest.Containers["app"] = Container{State: "running", Health: "healthy", ID: "a1"}
after.Guest.Containers["app"] = Container{State: "running", Health: "healthy", ID: "a1"}
if ok, why := PVEHealthVerdict(before, after, hub.TunnelRunning, "9.2.21", "9.2.21"); !ok {
t.Fatalf("healthy step read unhealthy: %s", why)
}
if ok, _ := PVEHealthVerdict(before, after, hub.TunnelRunning, "9.2.21", "9.2.2"); ok {
t.Fatal("pveversion still on the old pve-manager must fail")
}
after.Guest.Containers["app"] = Container{State: "running", Health: "healthy", ID: "b2"}
if ok, why := PVEHealthVerdict(before, after, hub.TunnelRunning, "", "9.2.2"); ok || !strings.Contains(why, "id changed") {
t.Fatalf("an app restarted by the Proxmox step must fail, got ok=%v %q", ok, why)
}
}
// The signed executor hands the RAW envelope and the exact list to the wrapper's pve layer.
func TestPVEStepExecutor_PassesTheSignedEnvelope(t *testing.T) {
w := &fakeWrapper{t: t, applyRep: map[string]WrapperReport{LayerPVE: {
Upgraded: []Package{{Name: "pve-manager", Version: "9.2.21"}}, PVEManager: "9.2.21", Authority: "signed"}}}
l, h := newLeg(t, w, &hub.WireOSUpdate{Ring: 1, Enabled: true})
e := PVEStepExecutor{Leg: l, Guest: func(context.Context) (int, error) { return 9201, nil }}
params, _ := json.Marshal(PVEStepParams{ReleaseID: "os-pve-1", Packages: []Package{{Name: "pve-manager", Version: "9.2.21", Origin: PVEOrigin}}})
ctx := signedjobs.WithSignedOp(context.Background(), &reconcile.SignedOp{Blob: []byte(`{"op":"os_pve_step"}`), Sig: []byte("SIG")})
if err := e.Execute(ctx, OpPVEStep, params); err != nil {
t.Fatal(err)
}
pp := w.plans[len(w.plans)-1]
sg, _ := pp["signed"].(map[string]any)
if pp["layer"] != LayerPVE || pp["lane"] != "slow" || pp["release_id"] != "os-pve-1" || sg == nil ||
sg["blob_b64"] != base64.StdEncoding.EncodeToString([]byte(`{"op":"os_pve_step"}`)) || sg["sig"] != "SIG" {
t.Fatalf("pve plan = %v", pp)
}
if !w.pveGateHeld || calls(w) != "pve:apply" || len(h.reports) != 1 || h.reports[0].Trigger != "signed" {
t.Fatalf("gate=%v calls=%s reports=%+v", w.pveGateHeld, calls(w), h.reports)
}
if err := e.Execute(context.Background(), OpPVEStep, params); err == nil {
t.Fatal("no envelope must refuse")
}
if err := e.Execute(context.Background(), OpDockerStep, params); err != signedjobs.ErrNoExecutor {
t.Fatalf("another op must pass through the chain: %v", err)
}
}
// os_pve_step is never benign.
func TestPVEStep_IsDestructiveClass(t *testing.T) {
if reconcile.Classify(reconcile.ClassOSPVEStep, reconcile.Provenance{}) != reconcile.Destructive {
t.Fatal("os_pve_step must be destructive-class (signed, operational key)")
}
}
+85
View File
@@ -0,0 +1,85 @@
package osupdate
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"gitea.dooplex.hu/admin/felhom-agent/internal/signedjobs"
)
// OpPVEStep is the signed op class of a Proxmox package step (R-812 option A, `11` §5.10): a ring-1 box takes an
// approved Proxmox set only through it. No undo in this release. CC may sign it until the first paying customer.
const OpPVEStep = "os_pve_step"
// PVEStepParams are the signed params. The wrapper compares Packages with the plan byte-for-byte.
type PVEStepParams struct {
ReleaseID string `json:"release_id"`
Packages []Package `json:"packages"`
VMID int `json:"vmid,omitempty"`
}
// PVEStepExecutor runs a verified os_pve_step (signedjobs.Executor) under the host-wide heavy-op gate (Gate) and the
// /etc/pve write gate (inside runPVE).
type PVEStepExecutor struct {
Leg *Leg
Guest func(ctx context.Context) (int, error)
Gate func(ctx context.Context) (release func(), err error)
}
// Execute implements signedjobs.Executor.
func (e PVEStepExecutor) Execute(ctx context.Context, op string, params json.RawMessage) error {
if op != OpPVEStep {
return signedjobs.ErrNoExecutor
}
so, ok := signedjobs.SignedOpFrom(ctx)
if !ok {
return fmt.Errorf("os_pve_step: no signed envelope in the context — the wrapper could not verify it")
}
var p PVEStepParams
if err := json.Unmarshal(params, &p); err != nil || len(p.Packages) == 0 {
return fmt.Errorf("os_pve_step: params must name the Proxmox set: %v", err)
}
vmid := p.VMID
if vmid == 0 {
if e.Guest == nil {
return fmt.Errorf("os_pve_step: no vmid and no guest finder")
}
v, err := e.Guest(ctx)
if err != nil {
return fmt.Errorf("os_pve_step: find the customer guest: %w", err)
}
vmid = v
}
if e.Gate != nil {
release, err := e.Gate(ctx)
if err != nil {
return fmt.Errorf("os_pve_step: heavy-op gate busy (a backup or restore-test runs): %w", err)
}
defer release()
}
rep := e.Leg.RunPVESigned(ctx, vmid, p, so.Blob, string(so.Sig))
switch rep.Outcome {
case "applied", "nothing":
if rep.Healthy {
return nil
}
}
return fmt.Errorf("os_pve_step: %s (%s) %s", rep.Outcome, rep.HealthReason, string(rep.Refused))
}
// RunPVESigned is one signed Proxmox step (ring 1): the pve layer with the signed envelope, which the wrapper verifies
// itself, holding the /etc/pve write gate.
func (l *Leg) RunPVESigned(ctx context.Context, vmid int, p PVEStepParams, blob []byte, sig string) Report {
unlock := l.lockPass(true)
defer unlock()
l.sendUnsentLocked(ctx) // R-868
runID := l.now().UTC().Format("20060102T150405Z")
rid := p.ReleaseID
if rid == "" {
rid = "signed-" + runID
}
return l.runPVE(ctx, runID, vmid, "signed", l.Block(), dockerOpts{releaseID: rid, packages: p.Packages,
signed: map[string]string{"blob_b64": base64.StdEncoding.EncodeToString(blob), "sig": sig}})
}
+32
View File
@@ -0,0 +1,32 @@
package osupdate
import (
"os"
"path/filepath"
"testing"
"gitea.dooplex.hu/admin/felhom-agent/internal/hub"
)
// After a boot the agent has not fetched the hub's block yet; the after-boot kernel report must still carry the
// box's real ring (seen 2026-10-08: a ring-0 box's „judging" report said ring 1). Red-proof: return Block().Ring
// from kernelReportRing and the saved-block case fails with 1.
func TestKernelReportRing_BeforeFirstFetch(t *testing.T) {
dir := t.TempDir()
l := &Leg{PlanDir: dir}
if got := l.kernelReportRing(); got != 1 {
t.Fatalf("nothing fetched, nothing saved: ring %d, want 1 (the old default)", got)
}
l.saveBlock(&hub.WireOSUpdate{Ring: 0, Enabled: true})
l2 := &Leg{PlanDir: dir} // a fresh daemon after the reboot: no block fetched yet
if got := l2.kernelReportRing(); got != 0 {
t.Fatalf("ring-0 block saved before the reboot: ring %d, want 0", got)
}
l2.SetBlock(&hub.WireOSUpdate{Ring: 1, Enabled: true})
if got := l2.kernelReportRing(); got != 1 {
t.Fatalf("a fetched block wins over the saved one: ring %d, want 1", got)
}
if _, err := os.Stat(filepath.Join(dir, SavedBlockFile)); err != nil {
t.Fatalf("saved block missing: %v", err)
}
}
+23 -3
View File
@@ -142,24 +142,42 @@ func (l *Leg) reportFromKept(ctx context.Context, wr WrapperReport, path string)
default:
rep.Outcome = "applied"
}
if wr.Layer == LayerKernel {
// a kernel STAGE changes nothing the box runs (the new kernel only boots once, at the night's reboot), so its
// kept copy needs no fresh health reading (R-836)
rep.Kernel = rawOrNil(wr.Kernel)
rep.Upgraded, rep.PassSeconds, rep.Authority = wr.Upgraded, wr.PassSeconds, wr.Authority
if rep.Outcome == "applied" {
rep.Outcome = "staged"
}
rep.Healthy, rep.HealthReason = rep.Outcome == "staged" || rep.Outcome == "nothing", prefix
return rep
}
rep.Upgraded, rep.PassSeconds = wr.Upgraded, wr.PassSeconds
rep.DockerEngine, rep.Authority, rep.Undo = wr.DockerEngine, wr.Authority, wr.Undo
rep.OOMCheck = rawOrNil(wr.OOMCheck)
wantEngine := ""
rep.PVEManager = wr.PVEManager
wantEngine, wantPVE := "", ""
for _, u := range wr.Upgraded {
if u.Name == "docker-ce" {
wantEngine = EngineOf(u.Version)
}
if u.Name == "pve-manager" {
wantPVE = u.Version
}
}
verdict := func(h *Health) (bool, string) {
switch wr.Layer {
case LayerDocker:
return DockerHealthVerdict(wr.HealthBefore, h, wantEngine, wr.DockerEngine)
case LayerHost:
case LayerHost, LayerPVE:
t := hub.TunnelUnknown
if l.Tunnel != nil {
t, _ = l.Tunnel.Status(ctx)
}
if wr.Layer == LayerPVE {
return PVEHealthVerdict(wr.HealthBefore, h, t, wantPVE, wr.PVEManager)
}
return HostHealthVerdict(wr.HealthBefore, h, t)
}
return HealthVerdict(wr.HealthBefore, h)
@@ -167,7 +185,7 @@ func (l *Leg) reportFromKept(ctx context.Context, wr WrapperReport, path string)
ok, why := verdict(wr.HealthAfter)
if !ok && len(wr.Upgraded) > 0 && wr.VMID > 0 {
lane := "fast"
if wr.Layer == LayerDocker {
if wr.Layer == LayerDocker || wr.Layer == LayerPVE {
lane = "slow"
}
if hr, err := l.call(ctx, "kept-"+runID, map[string]any{"release_id": "kept", "layer": wr.Layer, "lane": lane,
@@ -187,6 +205,8 @@ func (l *Leg) reportFromKept(ctx context.Context, wr WrapperReport, path string)
rep.RebootScanned = wr.RebootScanned
if wr.Layer == LayerDocker {
rep.Installed, rep.Pending = onlyDocker(wr.Installed), onlyDockerPending(wr.Pending)
} else if wr.Layer == LayerPVE {
rep.Installed, rep.Pending = onlyPVE(wr.Installed), onlyPVEPending(wr.Pending)
} else {
planned := map[string]bool{}
for _, u := range wr.Upgraded {
+10
View File
@@ -5,6 +5,7 @@ import (
"context"
"encoding/json"
"fmt"
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
"io"
"net/http"
"net/url"
@@ -105,6 +106,15 @@ func (c *Client) do(ctx context.Context, method, path string, body io.Reader, ou
// doBody is the single HTTP chokepoint: builds the request, sets auth, executes,
// maps non-2xx to APIError, and decodes the data envelope.
func (c *Client) doBody(ctx context.Context, method, path string, body io.Reader, contentType string, out any) error {
// R-812 option A: every non-GET call may write /etc/pve — it waits while a Proxmox package step restarts pmxcfs
// (pvegate). Pinned by TestPVEGate_ClientWriteWaitsGetDoesNot.
if method != http.MethodGet {
release, _, gerr := pvegate.Write(ctx)
if gerr != nil {
return fmt.Errorf("proxmox: %s %s held back by a Proxmox package step: %w", method, path, gerr)
}
defer release()
}
req, err := http.NewRequestWithContext(ctx, method, c.base+path, body)
if err != nil {
return fmt.Errorf("proxmox: building request: %w", err)
+33
View File
@@ -4,8 +4,10 @@ import (
"context"
"encoding/json"
"fmt"
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
"io"
"os/exec"
"path/filepath"
"strconv"
)
@@ -59,6 +61,15 @@ func (r *ExecRunner) Run(ctx context.Context, name string, args ...string) ([]by
// RunStdin is Run with the process stdin fed from stdin (nil = no stdin). The sudo-prefix/mode
// handling is identical to Run — kept here so both paths share one place.
func (r *ExecRunner) RunStdin(ctx context.Context, stdin io.Reader, name string, args ...string) ([]byte, []byte, error) {
// R-812 option A: a root CLI that writes /etc/pve waits while a Proxmox package step runs (pvegate).
// Pinned by TestPVEGate_ExecRunnerPctSetWaits / TestWritesEtcPVE.
if WritesEtcPVE(name, args) {
release, _, gerr := pvegate.Write(ctx)
if gerr != nil {
return nil, nil, fmt.Errorf("proxmox: %s held back by a Proxmox package step: %w", name, gerr)
}
defer release()
}
var cmd *exec.Cmd
if r.Mode == RunnerSudo {
sudo := r.SudoPath
@@ -77,6 +88,28 @@ func (r *ExecRunner) RunStdin(ctx context.Context, stdin io.Reader, name string,
return stdout.b, stderr.b, err
}
// WritesEtcPVE reports whether a root command writes /etc/pve: `pct` with a config-changing verb, `pvesm`, `pveum`,
// and the PBS storage wrapper's create / reconcile verbs. `pct exec|status|list|config` and every other command do not
// (the os-update wrapper itself must never wait on the gate its own step holds). Pinned by TestWritesEtcPVE.
func WritesEtcPVE(name string, args []string) bool {
base := filepath.Base(name)
switch base {
case "pvesm", "pveum":
return true
case "pct":
if len(args) == 0 {
return false
}
switch args[0] {
case "set", "create", "destroy", "restore", "unlock", "resize", "snapshot", "delsnapshot", "rollback", "move-volume", "start", "stop", "reboot", "shutdown":
return true
}
case "felhom-pbs-apply":
return len(args) > 0 && (args[0] == "create" || args[0] == "reconcile")
}
return false
}
// Privileged is the root-CLI backend.
type Privileged struct {
runner Runner
+92
View File
@@ -0,0 +1,92 @@
package proxmox
import (
"context"
"net/http"
"testing"
"time"
"gitea.dooplex.hu/admin/felhom-agent/internal/pvegate"
)
// R-812 option A: while a Proxmox package step holds the gate, a non-GET API call waits and a GET does not.
//
// COMPANION RED-PROOF (observed): delete the pvegate.Write block in doBody → this fails with "a PUT reached the API
// while the Proxmox step held the gate". Restored. (audits/day-2026-10-07/B/red-pvegate-chokepoints.txt)
func TestPVEGate_ClientWriteWaitsGetDoesNot(t *testing.T) {
d := &mockDoer{fn: func(*http.Request) (*http.Response, error) { return jsonResp(200, `{"data":null}`), nil }}
c := newTestClient(d)
end, err := pvegate.Step(context.Background())
if err != nil {
t.Fatal(err)
}
if err := c.get(context.Background(), "/nodes", nil); err != nil {
t.Fatalf("a GET must not wait on the gate: %v", err)
}
if d.calls != 1 {
t.Fatalf("the GET must reach the API, calls=%d", d.calls)
}
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
err = c.postForm(ctx, http.MethodPut, "/nodes/x/lxc/9201/config", nil, nil)
if d.calls != 1 {
end()
t.Fatal("a PUT reached the API while the Proxmox step held the gate")
}
if err == nil {
end()
t.Fatal("a PUT held back past its deadline must fail")
}
end()
if err := c.postForm(context.Background(), http.MethodPut, "/nodes/x/lxc/9201/config", nil, nil); err != nil || d.calls != 2 {
t.Fatalf("after the step the PUT must go through (err=%v calls=%d)", err, d.calls)
}
}
// A root `pct set` waits on the gate; `pct exec` does not.
//
// COMPANION RED-PROOF (observed): delete the WritesEtcPVE block in RunStdin → this fails with "pct set ran while the
// Proxmox step held the gate". Restored.
func TestPVEGate_ExecRunnerPctSetWaits(t *testing.T) {
r := &ExecRunner{Mode: RunnerDirect}
end, err := pvegate.Step(context.Background())
if err != nil {
t.Fatal(err)
}
defer end()
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
start := time.Now()
_, _, err = r.Run(ctx, "/nonexistent/pct", "set", "9201", "-mp8", "/x")
if err == nil || time.Since(start) < 90*time.Millisecond {
t.Fatalf("pct set ran while the Proxmox step held the gate (err=%v after %s)", err, time.Since(start))
}
start = time.Now()
_, _, _ = r.Run(context.Background(), "/nonexistent/pct", "exec", "9201", "--", "true")
if time.Since(start) > 80*time.Millisecond {
t.Fatal("pct exec must not wait on the gate")
}
}
func TestWritesEtcPVE(t *testing.T) {
for _, c := range []struct {
name string
args []string
want bool
}{
{"pct", []string{"set", "9201", "-mp8", "x"}, true},
{"/usr/sbin/pct", []string{"create", "9201"}, true},
{"pct", []string{"exec", "9201", "--", "true"}, false},
{"pct", []string{"status", "9201"}, false},
{"pvesm", []string{"add", "dir", "x"}, true},
{"pveum", []string{"acl", "modify"}, true},
{"/usr/local/sbin/felhom-pbs-apply", []string{"reconcile"}, true},
{"/usr/local/sbin/felhom-pbs-apply", []string{"read"}, false},
{"/usr/local/sbin/felhom-os-apply", []string{"--plan", "x"}, false},
{"pct", nil, false},
} {
if got := WritesEtcPVE(c.name, c.args); got != c.want {
t.Errorf("WritesEtcPVE(%s %v) = %v, want %v", c.name, c.args, got, c.want)
}
}
}
+104
View File
@@ -0,0 +1,104 @@
// Package pvegate keeps the agent's own writes to /etc/pve out of the way of a Proxmox package step (R-812 option A,
// `09` §3 decision 163, `11` §5.10).
//
// WHY. A `pve` step upgrades pve-cluster / pve-manager / qemu-server / pve-container; their postinst scripts restart
// pmxcfs (the FUSE filesystem behind /etc/pve) and the API daemons. A write that lands while pmxcfs restarts fails or,
// worse, half-lands (design-R-812 §3 A, "can go wrong"). Backups and restore-tests are already kept out by the
// host-wide heavy-op gate; this gate covers everything else the agent writes: every non-GET Proxmox API call
// (proxmox.Client.doBody) and every root CLI that writes /etc/pve (proxmox.ExecRunner — `pct set|create|…`, `pvesm`,
// `pveum`, `felhom-pbs-apply create|reconcile`).
//
// THE RULE. Write waits while a step runs (bounded by its own context). Step marks the step and then waits until every
// write already in flight has finished; it never waits forever (its context bounds it, and the caller gives up and
// does not run the step). One step at a time. Pinned by pvegate_test.go and, at the two chokepoints, by
// proxmox TestPVEGate_*.
package pvegate
import (
"context"
"errors"
"sync"
"time"
)
var (
mu sync.Mutex
inFlight int
stepping bool
stepDone chan struct{}
)
// ErrStepRunning is returned by Step when another step already holds the gate.
var ErrStepRunning = errors.New("pvegate: a Proxmox package step is already running")
// Write marks one /etc/pve write in flight, first waiting while a Proxmox package step runs. The returned release must
// be called when the write has finished. waited reports how long the write was held back.
func Write(ctx context.Context) (release func(), waited time.Duration, err error) {
start := time.Now()
for {
mu.Lock()
if !stepping {
inFlight++
mu.Unlock()
var once sync.Once
return func() {
once.Do(func() {
mu.Lock()
inFlight--
mu.Unlock()
})
}, time.Since(start), nil
}
ch := stepDone
mu.Unlock()
select {
case <-ch:
case <-ctx.Done():
return nil, time.Since(start), ctx.Err()
}
}
}
// Step marks a Proxmox package step and waits until every /etc/pve write already in flight has finished. On error the
// gate is released again and the step must not run. end releases the gate and lets the held-back writes go.
func Step(ctx context.Context) (end func(), err error) {
mu.Lock()
if stepping {
mu.Unlock()
return nil, ErrStepRunning
}
stepping = true
done := make(chan struct{})
stepDone = done
mu.Unlock()
var once sync.Once
end = func() {
once.Do(func() {
mu.Lock()
stepping = false
close(done)
mu.Unlock()
})
}
for {
mu.Lock()
n := inFlight
mu.Unlock()
if n == 0 {
return end, nil
}
select {
case <-ctx.Done():
end()
return nil, ctx.Err()
case <-time.After(50 * time.Millisecond):
}
}
}
// Stepping reports whether a Proxmox package step holds the gate (for logs).
func Stepping() bool {
mu.Lock()
defer mu.Unlock()
return stepping
}
+104
View File
@@ -0,0 +1,104 @@
package pvegate
import (
"context"
"testing"
"time"
)
// A write that starts while a step runs waits until the step ends.
//
// COMPANION RED-PROOF (observed): make Write ignore `stepping` → this fails with "the write went through while the
// Proxmox step held the gate". Restored. (audits/day-2026-10-07/B/red-pvegate.txt)
func TestWrite_WaitsWhileAStepRuns(t *testing.T) {
end, err := Step(context.Background())
if err != nil {
t.Fatal(err)
}
got := make(chan time.Time, 1)
go func() {
rel, _, err := Write(context.Background())
if err == nil {
rel()
}
got <- time.Now()
}()
select {
case <-got:
end()
t.Fatal("the write went through while the Proxmox step held the gate")
case <-time.After(150 * time.Millisecond):
}
ended := time.Now()
end()
select {
case at := <-got:
if at.Before(ended) {
t.Fatal("the write finished before the step ended")
}
case <-time.After(2 * time.Second):
t.Fatal("the write never went through after the step ended")
}
}
// A step waits for a write already in flight before it starts.
func TestStep_WaitsForAWriteInFlight(t *testing.T) {
rel, _, err := Write(context.Background())
if err != nil {
t.Fatal(err)
}
started := make(chan struct{})
go func() {
end, err := Step(context.Background())
if err == nil {
close(started)
end()
}
}()
select {
case <-started:
rel()
t.Fatal("the step started while a write was in flight")
case <-time.After(150 * time.Millisecond):
}
rel()
select {
case <-started:
case <-time.After(2 * time.Second):
t.Fatal("the step never started after the write finished")
}
}
// A step that cannot drain the writes in time gives up and releases the gate (it never waits forever).
func TestStep_GivesUpAndReleases(t *testing.T) {
rel, _, _ := Write(context.Background())
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
if _, err := Step(ctx); err == nil {
t.Fatal("the step must give up while a write is in flight past its deadline")
}
if Stepping() {
t.Fatal("a step that gave up must release the gate")
}
rel()
}
// A write held back past its own deadline returns the context's error.
func TestWrite_HonoursItsContext(t *testing.T) {
end, _ := Step(context.Background())
defer end()
ctx, cancel := context.WithTimeout(context.Background(), 50*time.Millisecond)
defer cancel()
if _, _, err := Write(ctx); err == nil {
t.Fatal("a write held back past its deadline must fail")
}
}
// One step at a time.
func TestStep_OneAtATime(t *testing.T) {
end, _ := Step(context.Background())
defer end()
if _, err := Step(context.Background()); err != ErrStepRunning {
t.Fatalf("a second step must be refused, got %v", err)
}
}
+10 -1
View File
@@ -52,6 +52,15 @@ const (
// (signed, operational key) like agent_update; the root wrapper re-verifies the same signature itself.
ClassOSDockerStep OpClass = "os_docker_step"
// A Proxmox package step on the host (R-812 option A, `11` §5.10) — ring 1. Destructive-class (signed, operational
// key) like os_docker_step; the root wrapper re-verifies the same signature itself.
ClassOSPVEStep OpClass = "os_pve_step"
// A kernel step on the host (R-836, `09` §3 decision 172, `11` §5.11) — ring 1: it STAGES a kernel (install + the
// one-shot flag; the night leg reboots it). Destructive-class (signed, operational key) like os_pve_step; the root
// wrapper re-verifies the same signature itself.
ClassOSKernelStep OpClass = "os_kernel_step"
// The config bundle (agent v0.143.0, R-840, `11` §5.4.2): the box's ROOT-OWNED files (sudoers, wrappers, units).
// Destructive-class (signed, operational key) like agent_update; the root wrapper re-verifies the signature itself.
ClassAgentConfigUpdate OpClass = "agent_config_update"
@@ -123,7 +132,7 @@ func Classify(class OpClass, prov Provenance) Disposition {
return Destructive
case ClassKeyRotation:
return Destructive
case ClassAgentUpdate, ClassOSDockerStep, ClassAgentConfigUpdate:
case ClassAgentUpdate, ClassOSDockerStep, ClassOSPVEStep, ClassOSKernelStep, ClassAgentConfigUpdate:
// Never benign — no agent-internal provenance can make replacing the agent binary
// unsigned-safe (a compromised process must not be able to self-bless an update).
return Destructive